LivingSocial Tech Team

JRuby "include?" 2x faster than 1.8.6?

2009-07-07T00:00:00+00:00

Its been a few months since I’ve looked at JRuby and 1.8.6 can be annoying slow at simple things. So this morning I took another look at Ruby1.9 and JRuby.

See gist below:

6.587s (1.8.6) vs. 5.687s (1.9) vs. 3.556s (JRuby 1.4)… and that’s just the include method. JRuby is almost 2x 1.8.6 without any optimizations. I’ve read this this awesome article about JRuby perf optimizations, but its unclear which flags are safe to run in a production environment beyond ‘–server’, which didn’t seem to impact local testing of a handful of iterations.

Does anyone have Rails benchmarks running nightly comparing 1.8.6 against jruby? I guess thats a more comprehensive test.

UPDATE: Eddie guessed PHP was much faster… Apparently not.

UPDATE 2: Warren suggested a different method other than include, that would work. (a & [i]).empty?

Interesting to see it much faster on 1.8.6, consistent on JRuby, and slower on 1.9

UPDATE 3 : Responses to comments from dmilith: Sorry.The original gist had version numbers, if you look at its history. I put them back. I do mention –server above, and even adding warmup to the script, it doesn't appear to impact numbers as all. But for it to be warmed up, you’d have to run the benchmark 10k times? Whats an acceptable warmup period? Also, –fast seems strange to me. Didn't help the numbers as well.

I hate when people say uBenchmarks suck. The ruby lang specs should include a suite of these types benchmarks as well. Its important to keep tabs on them to help find route cause when macro performance suites improve/degrade over time.

Ruby Scoping Bug?

2009-01-28T00:00:00+00:00

class Foo
def title
"Oh hai."
end

def buggy
puts title

if false
title = 'Oh noes!'
end

puts title
end
end

Foo.new.buggy

This code outputs:

Oh hai.
nil

Isn’t it odd that the variable being created in an if-false block is still being created and set to nil? Hopefully this is fixed in Ruby 1.9.

Merb routing in Rails – Today!

2008-12-29T00:00:00+00:00

We’ve been hearing great things about Merb routing so a few weeks ago, we wanted to see if we could get it working in a Rails application as a proof of concept. The goal of this project was to use the Merb routing engine along with router.rb and without touching any of our existing Rails application (i.e. anything in app/*). Pre-existing URL route recognition needed to continue to work and all named routes and url_for logic needed to generate the same URLs as before.

With the announcement that Merb and Rails are merging, we figured that it’d be a great time to share a little bit of what we learned.

Background

We’re running Rails 2.2.2
When we started messing with routes, we had 2500 generated routes (i.e. “rake routes | wc -l”). After Aaron’s formatted routes patch, we had 1250. We currently have over 1500.
Routing accounted for around 4 seconds out of 6 for ./script/console and mongrel_rails to start up.
Each mongrel_rails process was running at a minimum of 250mb in production.

Step 1: compiling

Compiling is the first part of routing and involves loading an application’s routes.rb or router.rb and storing it in some way that allows for generation and recognition.

The merb_routing plugin contains a subset of merb-core… just the classes related to routes. It shouldn’t be a surprise that these classes assume they are running in a Merb environment and not a Rails environment. To successfully load these merb routing classes without drastically modifying the source, we wrote two small compatibility layers (mini-merb and mini-extlib) that provide some basic functionality that Merb and ExtLib provide (i.e. Merb.logger, Merb.root, etc). This enabled us to successfully load the Merb::Router classes in our environment without any errors.

We then rewrote our routes.rb by hand using router.rb syntax. We had to modify Merb routing slightly because we were getting “memory exhausted” compile errors. It turned out that Merb routing uses a single if/elsif structure to recognize routes and due to our large number of routes, we hit a ruby quirk where you can’t have more than 2498 branches of logic in a single if-elsif statement.

irb(main):001:0&gt; eval("if 1; #{"elsif 1;" * 2498} end")
=&gt; nil
irb(main):002:0&gt; eval("if 1; #{"elsif 1;" * 2499} end")
SyntaxError: (eval):1:in `irb_binding': compile error
(eval):1: memory exhausted

The quick fix was using many single if statement with a return inside rather than one giant if-elsif. After a few hooks into ActionController::Routing we had our environment using Merb/router.rb instead of Rails/routes.rb.

We had to make a few other minor changes to Merb routing to be compatible with rails:

added support for { :method => :any } in routing conditions
added support for BLAH_index as a named route for singular resources (Rails creates a named route called “blog_index” instead of just “blog” for singular resources)

Lastly, we wrote a rake task that overrides the default “rake routes” to pull from Merb routes instead of Rails routes. At this point, we had the routes loaded and could see the objects in script/console.

Step 2: recognizing

Recognition is the second part of routing and happens at the beginning of every request. It translates a URL into various parameters and figures out which controller/action to invoke for that request.

Once we had router.rb working correctly, route recognition and parameter loading was surprisingly easy to wire in.

Step 3: generating

Generation is the third part of routing and translates structured options into a URL string. This happens every time you use url_for, a named route helper or redirect_to.

Generation was the trickiest part of this project and had the most edge cases. Philosophically speaking, Merb and Rails route generation are very different. Rails gives you named routes as helpers (person_path, person_url, etc) that you can use as well as a url_for() method which will search through all of the routes and find the best route given the options you provide. Merb on the other hand provides a single method, url(), which all generation goes through. If you don’t explicitly provide a named route, url() will use the default route (i.e. :controller/:action/:id) rather than looking for the “best route”.

Getting the named routes to work was pretty easy and only required a single method_missing catch-all for ActionController::Base and ActionView::Base. Using a simple regexp, if the requested method ends in _url or _path, it uses Merb::Router#url and passes in the named route.

There was a bit of trickery involved in getting the path vs. url as well as the :only_path stuff working correctly, but overall not too hard.

The last tricky piece was returning the best route instead of the default route when no named route was provided. This is accomplished by looping through all of the available routes and determining which of the routes satisfies the most options passed in. With 1500 routes, this turned out to be a bit slow, so some optimization ideas were borrowed from Rails and a cache is maintained of available routes given a controller/action.

Conclusion

We’ve been using this plugin successfully in production for the last month. Our environment startup time as well as our memory overhead were both reduced drastically as soon as we put it in to production. We started this development when Rails 2.1 was the latest stable release and benchmarks against Rails 2.1 put Merb routing way ahead in just about every metric we tested.

The routing in Rails 2.2 was sped up substantially and is now comparable to Merb (Rails wins in some benchmarks, Merb in others).BUT…. Merb still blows away rails in startup time, by 2-3x.We thought we could take out our merb-routing hacks and reduce pre complexity, but after watching production restarts, we decided to put it back.

This plugin can be found on github: merb_routing

UPDATE: after talking to Carl Lerche, it sounds like the new router refactoring he is working on will support both syntaxes on the same prebase. That’ll be very cool.

Modular routing in rails and merb (acts_as_routing)

2008-11-26T00:00:00+00:00

Here’s a proof of concept plugin that will monkeypatch Rails or Merb routing to allow you to define “acts as blocks” anywhere throughout your application (i.e. a plugin) and then use them in your routes file.

Imagine the plugin acts_as_commentable defines the following in its init.rb:

ActionController::Routing.routes_for_acts_as(:commentable) do |map|
  map.resources :comments
  map.best_comment '/best-comment', :controller => 'comments', :action => 'best'
end

If you added these :acts_as to your config/routes.rb:

ActionController::Routing::Routes.draw do |map|

  map.resources :people, :acts_as => [:commentable]

  map.resources :posts, :acts_as => [:commentable]

end

You could then use these routes throughout your application:

  <%=person_comments_path(Person.first)%>

  <%=post_comments_path(Post.first)%>

  <%=person_best_comment_path(Person.first)%>

This is equivalent to doing:

ActionController::Routing::Routes.draw do |map|

  map.resources :people do |people_map|
    people_map.resources :comments

    people_map.best_comment '/best-comment', :controller => 'comments', :action => 'best'

  end

  map.resources :posts do |posts_map|

    posts_map.resources :comments

    posts_map.best_comment '/best-comment', :controller => 'comments', :action => 'best'

  end

end

The plugin is available on github: http://github.com/hungrymachine/acts_as_routing/tree/master

P.S. At some point, I’ll submit patches to Rails and Merb so this functionality is native rather than provided via a monkeypatching plugin.

JRuby and why it might be nice to be back on the JVM – Part 2

2008-11-21T00:00:00+00:00

In the previous post, I did some JRuby testing and noticed perf improvements over time.

Mark Imbriaco, of 37Signals, asked how it compared to MRI. I was curious too.

I can’t promise a “clean” comparison, your mileage may vary, but… MRI was:

Completed in 552ms (View: 104, DB: 14) | 200 OK [http://someurl.com/people/3]
Completed in 345ms (View: 60, DB: 3) | 200 OK [http://someurl.com/people/3]
Completed in 346ms (View: 60, DB: 4) | 200 OK [http://someurl.com/people/3]
Completed in 347ms (View: 56, DB: 4) | 200 OK [http://someurl.com/people/3]
Completed in 349ms (View: 58, DB: 5) | 200 OK [http://someurl.com/people/3]

Some stats on my runtime

$ java -version
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-284)
Java HotSpot(TM) Client VM (build 1.5.0_16-133, mixed mode, sharing)
$ ruby -v
ruby 1.8.6 (2008-08-11 patchlevel 287) [i686-darwin9.5.0]

Both ran on a MacBook 2.53GHz. Likely the perf is better on Java6?

Installed Java6 and re-ran. Similar but better results.

Completed in 8815ms (View: 1632, DB: 429) | 200 OK [http://someurl.com/people/3]
Completed in 1017ms (View: 656, DB: 25) | 200 OK [http://someurl.com/people/3]
Completed in 998ms (View: 618, DB: 18) | 200 OK [http://someurl.com/people/3]
Completed in 1300ms (View: 437, DB: 35) | 200 OK [http://someurl.com/people/3]
Completed in 315ms (View: 119, DB: 21) | 200 OK [http://someurl.com/people/3]
Completed in 361ms (View: 140, DB: 20) | 200 OK [http://someurl.com/people/3]

Wow! Thats damn close to MRI…eventually… JRuby warmup time sucks though. Maybe warm it up during deploy before putting it back into the load balancer? Not exactly sure how to warm it correctly though.

If you just hit a URL on your app, you’ll only warm up THAT execution path… and you cant hit all actions, so…. thoughts?

JRuby and why it might be nice to be back on the JVM

2008-11-19T00:00:00+00:00

We ported our Rails application to JRuby in order to do some GC comparisons yesterday. Not a ton of changes required. Mostly replacing gems with native dependencies. YAML is a bit more strict than its MRI counterpart…. but all told, thats pretty amazing! Nice job JRuby crew!

That said…. one interesting, entirely un-scientific, test case. After requesting the same URL 6 times, the VM got a lot better. 3.4s to 480ms

Completed in 3380ms (View: 2739, DB: 65) | 200 OK [http://someurl.com/people/3]
Completed in 1299ms (View: 1211, DB: 18) | 200 OK [http://someurl.com/people/3]
Completed in 1020ms (View: 949, DB: 20) | 200 OK [http://someurl.com/people/3]
Completed in 852ms (View: 732, DB: 18) | 200 OK [http://someurl.com/people/3]
Completed in 743ms (View: 628, DB: 31) | 200 OK [http://someurl.com/people/3]
Completed in 485ms (View: 374, DB: 14) | 200 OK [http://someurl.com/people/3]

It doesnt appear to be the DB caching the data. That time stayed consistent.

I didnt use any crazy runtime flags.. Just jetty, a war file, and -server….

If it was something we were memcaching, then after the 2nd request, it would have been consistent.

The amount of time spent in the view improved dramatically.

I didnt think JRuby leveraged JIT’ing yet. Anyone care to guess/explain why it was significantly better per request?

Note: It did trail off in the 300ms range with further testing...

Are your mongrels growing by 20MB/request on Rails 2.2? Blame AssetTag!

2008-11-19T00:00:00+00:00

After porting our production application to Rails 2.2, we noticed a major memory leak.

Beforehand, monit would restart instances a handful of times a day. After Rails 2.2, monit restarted instances THOUSANDS of times a day.

This is a graph of one of our haproxy instances a couple days ago.

We looked at everything, including time spent rewriting Routes, thinking that was the culprit.

This morning, we all sat around and fought the issue old school style. binary debugging… and found it: AssetTagHelper. See the patch here.

The new thread-safe asset tag code keeps a static AssetTag::Cache = {} of all asset_tags created (css,jss, and all images).

Internally, each AssetTag object keeps a reference to the controller and template objects, and in turn all instance variables you created in your request.

What does that mean? Say you have a people controller, that loads a person and their stuff, and you show images of their stuff via image_tag().

 class PeopleController  30)
   end
 end

When image_tag() is called, it does rails magic to append file extensions, asset_ids, and the like. To be smart, it caches those objects so it doesnt hit the file system
to figure all that out on every request. The problem is it puts it in a static cache, AssetTag::Cache.

So each PeopleController instance has a reference to 1 person and 30 Stuffs. After 1000 people look at their pages, or better yet google crawls your site, you have 1k @controllers with a total of 1000 People Objects, and 1000*30 Stuff objects. This would normally be fine. The objects leave scope and get GC’ed. But, if you generate an image tag to an unique asset, AssetTag puts that into a cache, AssetTag::Cache, with a reference to the @controller of the request. So All People and their Stuff are kept around forever, unable to be GC’ed…. every time a unique image is rendered via AssetTag. Eventually monit has to kill the process.

The patch we just submitted does 2 things.

1) It now keeps a cache of just the modified path strings, caching the file access stuff. If you have tons of local images, reference them by fully qualified host. Thats better for lots of reasons. Cookie-less asset hosts with multiple subdomains FTW!

2) It stops caching absolute URL paths. You cant do anything on the filesystem to verify them, and keeping a cache of those would also grow unbounded. We have millions of items in our system, each with a reference to an image.

Here is a graph of that haproxy today… Sleeping………

In order to do some testing of your own, here’s a simplistic after_filter you can add to application.rb (or is it now application_controller.rb?). Make sure you run this in production mode or with cache_classes = true. As you click around your site, you should see that the cache retains references to controller instances, just to name a few. After you apply the patch, you’ll see the cache is just strings.

 def assettag_cache
    puts "-"*80
    puts "[AssetTag::Cache] Now #{ActionView::Helpers::AssetTagHelper::AssetTag::Cache.size} items"
    ActionView::Helpers::AssetTagHelper::AssetTag::Cache.values.each do |asset_tag|
      if asset_tag === ActionView::Helpers::AssetTagHelper::AssetTag
        puts "   [Asset] #{asset_tag.instance_variable_get("@source")}  #{asset_tag.instance_variable_get("@controller").class.to_s}"
      else
        puts "   [Asset] #{asset_tag}"
      end
    end
  end

Ohh… and we havent given up on routes… Warren is working on some very interesting enhancements to rails routing. Looking forward to blogging about that soon.

How to save 100m of RAM per mongrel (Part 2)

2008-11-11T00:00:00+00:00

UPDATE I’ve added a patch to Rails Edge for this fix, which is much different than the patch below. See here

In a previous article, I called out the massive memory usage of the default rails resource behavior, and it seems others have as well. In an attempt to decrease the number of routes, I commented out the “formatted_*” routes, and manually entered them back by hand.

But after some internal discussion/testing with Warren, we realized that was sloppy and error prone. Instead, I hacked Routing segments to allow for an optional format segment, so that formatted routes and normal routes are shared. The one downside, from what I can tell so far, is you lose “formatted_*” named routes + url helpers for those routes, but passing format to a url_for still works

person_path(:id => 1, :format => "json") =>  /people/1.json

In the below gist, you’ll see a OptionalFormatSegment, which sneakily gets around the ‘.’ regex separator, removes the formatted_* named routes added by default, and should be the same solution as the previous post, but without the need to manually put all the routes back in.

I’m still testing this approach, but am interested in what other folks think.

Note: This monkey-patch only works on Rails 2.2

How to save 100m of RAM per mongrel

2008-11-08T00:00:00+00:00

UPDATE: See Part 2 of this for a better solution

Note: This monkey-patch only works on Rails 2.2

We recently noticed our mongrels, upon startup, were 244M. Eek.

PID USER            PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
53749 some_user     15   0  337m 244m 2912 S    0  2.0   0:17.49 mongrel_rails

Running Bleakhouse on our rails app, i was amazed at the number of objects created. over 2M!

$ BLEAK_HOUSE=1 ruby-bleak-house ./script/runner 'puts 1'
** Bleakhouse: installed
1
** BleakHouse: working...
** BleakHouse: complete
** Bleakhouse: Run 'bleak /tmp/bleak.53749.0.dump' to analyze.

$ bleak /tmp/bleak.53749.0.dump
Displaying top 20 most common line/class pairs
2,047,025 total objects
2,047,025 filled heap slots
2,403,764 free heap slots
1779063 __null__:__null__:__node__
197423 __null__:__null__:String
16519 __null__:__null__:Array
12728 __null__:__null__:ActionController::Routing::DividerSegment
10495 __null__:__null__:Hash
5676 __null__:__null__:ActionController::Routing::StaticSegment
5436 __null__:__null__:Class
5132 __null__:__null__:Regexp
4525 __null__:__null__:ActionController::Routing::DynamicSegment
2524 __null__:__null__:ActionController::Routing::Route
1307 __null__:__null__:Gem::Version

Whats up with all those routing objects! So i tried a blank routes.rb file….

519,307 total objects
519,307 filled heap slots
233,503 free heap slots

Arghh…. 1.5M objects just because of routes.rb. I know our application is large, lots of resources and nested resources, but thats crazy.

$ rake routes | wc -l
2516

Ok, so taking a closer look, I realized something. Each resource route has a corresponding formatted_* route, lots of which we dont use. So now for some nasty monkey-patching.

ActionController::Routing::RouteSet::Mapper.class_eval do
protected

  def map_unnamed_routes(map, path_without_format, options)
    map.connect(path_without_format, options)
    #map.connect("#{path_without_format}.:format", options)
  end

  def map_named_routes(map, name, path_without_format, options)
    map.named_route(name, path_without_format, options)
    #map.named_route("formatted_#{name}", "#{path_without_format}.:format", options)
  end
end

I then manually added back the formatted routes we needed, careful to use named routes as “map.formatted_foo_bar”

$ rake routes | wc -l
1490

$ bleak /tmp/bleak.56554.0.dump
1,242,796 total objects
1,242,796 filled heap slots
1,224,309 free heap slots

Awesome. I no longer create almost 800k objects.

PID USER            PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
56554 someuser     15   0  249m 156m 2912 S    0  1.3   0:09.92 mongrel_rails

244m > 156m

This is not a very elegant solution… but I saved almost 100m per mongrel. Ohh.. and it reduced startup time by 50%.

How big are your mongrels?

How does Merb’s router differ from Rails routing? Worth investigating?

MRI Ruby + MySQL + Threads == Stop the world… JRuby doesn’t

2008-08-27T00:00:00+00:00

As we have been internally discussing how to scale our databases from 10’s of millions of rows to 100’s of millions, database sharding came up.

Depending on your data model and your application, sharding data into tables by some natural key is great if any given request uses only one shard. FiveRun’s DataFabric seems to help with that. Its obviously best to shard the data in the most used way, but occasionally you’ll need to write broadcast queries across shards. It’d be even better if those broadcast queries were executed concurrently. Well, apparently, its not that simple in MRI Ruby.

The mysql (2.7) gem stop’s the world while executing a query. A sloppy but reproducible test is here. This is not a mutex lock on the re-use of a single connection object. Eddie also reached out to the Sequel maintainer, who agrees it is likely due “to the fact that the C drivers don’t release the interpreter lock while they wait for a response from the server.” JRuby, or more accurately JDBC, acts as expected. We even tested DataMapper’s DataObjects::MySQL, as it appears they’ve re-implemented the mysql gem. Unfortunately it suffers from the same stop-the-world issue.

$ ruby mysql_locking_test.rb
Loading MRI MySQL
Serial: 4.00969815254211s
Multi-threaded: 4.00785183906555s

$ data_mapper=true ruby mysql_locking_test.rb
Loading DataMapper
Serial: 4.01330804824829s
Multi-threaded: 4.01132893562317s

$ jruby mysql_locking_test.rb
Loading JDBC-MySQL
Serial: 4.2802369594573975s
Multi-threaded: 1.0499329566955566s

Only in JRuby was the multi-threaded != sequential time, as expected.

Potentially unrelated, no one has touched the mysql gem in 3 years?!?!

$ gem spec -v 2.7 mysql | grep date
date: 2005-10-09 00:00:00 +00:00

So even Ruby ORM frameworks (Sequel, DataMapper) that say they’re thread-safe, are not concurrent on MRI… at least for mysql. For folks not using Rails, which already has a mutex lock higher in the stack, this must be a performance issue. For example, Merb + DataMapper + MySQL. If there is a 2s SQL query, all threads in that process stop for 2s.

Can others verify? “select sleep(2) from dual;” is a great way to test for this.

UPDATE: Multiple have asked, so to clarify. The sample code here creates a new connection PER thread. The mysql docs states: “Two threads can’t send a query to the MySQL server at the same time on the same connection”, but the test is creating a new connection object per thread, so that should not apply.

UPDATE 2: A couple folks have mentioned asynch drivers. One for mysql and one for postgres, but I believe they’re based on NeverBlock, which is Ruby 1.9 only. That sounds like awesome progress. What’s the realistic ETA for folks running 1.9 in a production environment? At least until Christmas for a 1.9 official release?

UPDATE 3: Looks like lots is happening here. There now is a mysql driver that supports async/threaded operations on 1.8! See the NeverBlock MySQL project. Looking forward to testing this in a production environment.

Using helpers in a controller: with_helpers

2008-08-07T00:00:00+00:00

Although there is supposed to be a clear separation between views and controllers, often when it comes to helper functions, there is a small bit of overlap and there are situations where it’d be nice to simply use a few helpers from inside an action.

ActionController::Base.class_eval do
  def with_helpers(&block)
    template = ActionView::Base.new([],{},self)
    template.extend self.class.master_helper_module
    add_variables_to_assigns
    template.assigns = @assigns
    template.send(:assign_variables_from_controller)
    forget_variables_added_to_assigns
    template.instance_eval(&block)
  end
end

Here is what it looks like in a controller. Although this is a non-sensical example, it shows off how you can use a helper (i.e. link_to) in an action. It also shows how the instance variables (i.e. @person) set in the action are available in the with_helpers block… the same way that instance variables are available in views.

class MyController < ApplicationController
  def my_action
    @person = Person.find(params[:id])
    render :text => with_helpers { link_to(@person.full_name, person_path(@person)) }
  end
end

Breaking the Rails static asset timestamp cache in development mode

2008-06-21T00:00:00+00:00

Rails automatically adds the File.mtime to static assets when using stylesheet_link_tag and javascript_include_tag. The file’s mtime is cached to prevent excessive file system access… even in development mode. This is problematic in a Facebook canvas during development mode because often you won’t immediately see the changes you make to your stylesheets and javascripts.

Here is a monkey patch you can throw into a config/initializers/break_asset_cache_in_dev_mode.rb to fix this:

if RAILS_ENV == 'development'
  require 'action_controller/dispatcher'
  ActionController::Dispatcher.before_dispatch do
    ActionView::Base.computed_public_paths.clear
  end
end

Rake task for syntax checking a Ruby on Rails project

2008-06-04T00:00:00+00:00

Here’s a quick rake file that will crawl through your Rails project and syntax check ruby, erb, and yml files. You should always run this before doing a “cap deploy” and it even doesn’t hurt to run it before a “svn commit”.

require 'erb'
require 'open3'
require 'yaml'

task :check_syntax => [:check_ruby, :check_erb, :check_yaml]

task :check_erb do
  (Dir["**/*.erb"] + Dir["**/*.rhtml"]).each do |file|
    next if file.match("vendor/rails")
    Open3.popen3('ruby -c') do |stdin, stdout, stderr|
      stdin.puts(ERB.new(File.read(file), nil, '-').src)
      stdin.close
      if error = ((stderr.readline rescue false))
        puts file + error[1..-1]
      end
      stdout.close rescue false
      stderr.close rescue false
    end
  end
end

task :check_ruby do
  Dir['**/*.rb'].each do |file|
    next if file.match("vendor/rails")
    next if file.match("vendor/plugins/.*/generators/.*/templates")
    Open3.popen3("ruby -c #{file}") do |stdin, stdout, stderr|
      if error = ((stderr.readline rescue false))
        puts error
      end
      stdin.close rescue false
      stdout.close rescue false
      stderr.close rescue false
    end
  end
end

task :check_yaml do
  Dir['**/*.yml'].each do |file|
    next if file.match("vendor/rails")
    begin
      YAML.load_file(file)
    rescue => e
      puts "#{file}:#{(e.message.match(/on line (\d+)/)[1] + ':') rescue nil} #{e.message}"
    end
  end
end

And here’s what the output looks like:

warren:tmp_project$ rake check_syntax
(in /Users/warren/tmp_project)
app/controllers/application.rb:37: syntax error, unexpected '='
app/views/people/home.html.erb:60: syntax error, unexpected '<', expecting $end
vendor/plugins/will_paginate/test/fixtures/users.yml:13: syntax error on line 13, col 0: `dev_<%= digit %>:'

This could probably be expanded to support lots of other types of files as well (css, javascripts, etc).

Updates:

Exclude all files in vendor/rails (thanks szeryf)

ESI & Mongrel-ESI.. Request for Feedback

2008-06-04T00:00:00+00:00

The Railsconf08 talk on ESI & Rails has sparked some interest in the community, and Todd, the core mongrel-esi maintainer, is asking for feedback on the mongrel-esi mailing list.

The latest rumor is he is working on a nginx port of mongrel-esi, which I have to admit sounds very interesting…

How are people planning to use ESI? Can anyone provide Todd with some feedback?

  I'm wondering what is the main set of features preventing folks
today from using mongrel-esi in production?  Is it :

* performance
* documentation
* stability

If it's performance, can someone do some load testing maybe provide
some numbers and even some target numbers?  I am working on an
improved concurrency model, that should help improve page performance
for pages with lots of esi:include tags

if it's documentation, what's the missing bits, and can any of you
help out by filling in the gaps?

if it's stability, can you provide some samples that fail?

-Todd

Cleaning up old releases

2008-06-04T00:00:00+00:00

Instead of relaying on running cleanup of old releases via capistrano, we have a cron job to only keep releases for last two days (but at least three latest).

#!/usr/bin/env ruby

require 'fileutils'

KEEP_RELEASES = 3
KEEP_DAYS = 2
EXCLUDE_APPS = %W(uploadr)

cut_time = (Time.now.utc - KEEP_DAYS*24*60*60).strftime("%Y%m%d%H%M%S").to_i

Dir['/u/apps/*'].each do |app|
  next if EXCLUDE_APPS.include?(File.basename(app))
  dirs = Dir["#{ app }/releases/*"]
  fresh = dirs.select { |dir| (dir.split('/').last).to_i > cut_time }
  latest = dirs.sort.last(KEEP_RELEASES )

  (dirs - fresh - latest).each do |dir|
    FileUtils.rm_rf dir
  end
end

RailsConf 2008 Presentation – Assembling Pages Last

2008-05-31T00:00:00+00:00

I presented this morning at RailsConf about Edge Caching and ESI. You can download my presentation below.

Assembling Pages Last: Edge Caching, ESI, and Rails

People asked some awesome questions, caught the edge cases I didnt want to cover which were complicated, like authorization/security, multi-level ESI includes, per page include limits, etc. The people at RailsConf are awesome. Thats totally why I enjoy these events.

The End of Slideshows: Animoto

2008-05-13T00:00:00+00:00

UPDATE: Animoto just raised a round of investment from Amazon! Congrats Guys!

Animoto is a great idea. They take your photos and create a production quality video to the music of your choice. Its the end of those boring slide shows, for good.

(From a recent Techcrunch article here)

We had the pleasure to work with the Animoto guys to launch their Facebook application, “Animoto Videos”, which leveraged all of the existing photos on Facebook. The growth was amazing.

(From a recent AllFacebook article here)

Scaling an application from a few hundred users to over a million in just a few days isnt easy, but we had a great team. Their backend rendering farm lived in Amazon’s Cloud, and the growth was so impressive, Jeff Bezos even spoke about them at Y Combinator’s Startup School just a few weeks ago. From 50 EC2 instances to over 4k in only a few days. See the video below.

<div> <a href="http://www.omnisio.com">Share and annotate your videos</a> with Omnisio!</div> <p></p>

It was a pleasure working with the entire team from Animoto, RightScale, and Amazon. See their blog posts about the application here, here, and here.

I’m sure I’ll cross paths with many of you at RailsConf. First round of beers is on me.

Advanced Rails Recipes

2008-05-13T00:00:00+00:00

Congratulations to Mike Clark and the Pragmatic Programmers team for shipping Advanced Rails Recipes. I highly recommend this book of 70+ recipes on topics ranging from deployment to UI to security and performance, just to list a few.

Val, Warren and I are honored to have our recipes included.

Recipe 34: Play Nice with Facebook

Recipe 62: Profile in the Browser

Recipe 67: Encrypt Sensitive Data

It was a pleasure to work with Mike Clark on this project. See his screencast below.

Advanced Rails Recipes Screencast (33Mb, Quicktime)

ActsAsInsertOrUpdate

2008-04-23T00:00:00+00:00

Problem

With high volume Rails applications, entities with unique constraints are expensive and error prone to create/update. ActsAsInsertOrUpdate helps solve that problem (if you’re using MySQL), by leveraging the “INSERT … ON DUPLICATE KEY UPDATE” functionality.

Scenario

Lets say you have a Person, and Entity, and a Rating. Each user can rate each entity only once, and if they re-rate the entity, it should update the value.

class Entity < ActiveRecord::Base
  has_many :ratings
end

class Person < ActiveRecord::Base
 has_many :ratings
end

class Rating < ActiveRecord::Base
 belongs_to   erson
 belongs_to :Entity
end

Here is the table that back’s Rating. Notice the Unique Key constraint on (entity_id, person_id).

CREATE TABLE `ratings` (
  `id` int(11) NOT NULL auto_increment,
  `rating` tinyint(4) default '0',
  `person_id` int(11) default NULL,
  `entity_id` int(11) default NULL,
  `created_at` datetime default NULL,
  `updated_at` datetime default NULL,
  PRIMARY KEY  (`id`),
  UNIQUE KEY `index_ratings_on_entity_id_and_person_id` (`entity_id`,`person_id`),
)

Previously, the logic would be something like:

1) Check if a rating exists for the User + Entity
2) If so, update
3) If not, insert
4) rescue the insert in case there is a unqiue constraint error
5) retrieve the record (and/or update with the new rating)

If the table is MyISAM, Steps 1-5 aren’t transactionally safe. If you’re using InnoDB, and experience heavy volumes of traffic, you’re prone to Deadlock’s. This is even more of a concern is the unique entity is shared across multiple users, as seen with a recent client of ours.

Solution:

class Rating < ActiveRecord::Base
 belongs_to   erson
 belongs_to :Entity
 acts_as_insert_or_update :field_to_update => "rating"
end

Now Steps 1-5 above become, just one. Rating.create(..)

In the background, ActsAsInsertOrUpdate overwrites the implementation of ActionRecord:Base#create, to leverage an often unsed feature of MySQL called INSERT … ON DUPLICATE KEY UPDATE. As configured above, if a duplicate record is found for the unique constraint, the rating field will be updated with the new value.

Caution

This is a brute force hack on ActiveRecord::Base#create. Use at your own risk.

Code

Waiting for a rubyforge account. Will post more info soon.

MySQL Stored Function: parsing a JSON encoded string

2008-04-14T00:00:00+00:00

For analytics purposes, we ended up storing JSON-encoded data as a column in a mysql table. Although we don’t often need to query it directly, from time to time, it makes things a bit easier/faster. Below is a MySQL stored function that takes two parameters (a JSON encoded string, and the name of a key) and returns the value associated with that key.

CREATE FUNCTION JSON(`json` TEXT, `search_key` VARCHAR(255)) RETURNS TEXT DETERMINISTIC BEGIN

  DECLARE i INT DEFAULT 1;
  DECLARE json_length INT DEFAULT LENGTH(json);
  DECLARE state ENUM('reading_key','done_reading_key','reading_string', 'reading_array');
  DECLARE tmp_key TEXT;
  DECLARE tmp_value TEXT;
  DECLARE current_char VARCHAR(1);

  WHILE i
Examples
Here's a few examples of how it can be used:
SELECT JSON('{"key1":"val\\"ue1","key2":"value2","key3":["array1","array2"],"key4":"value4"}', 'key1');
# returns 'val"ue1'

SELECT JSON('{"key1":"val\\"ue1","key2":"value2","key3":["array1","array2"],"key4":"value4"}', 'key2');
# returns 'value2'

SELECT JSON('{"key1":"val\\"ue1","key2":"value2","key3":["array1","array2"],"key4":"value4"}', 'key3');
# returns '"array1","array2"'

SELECT JSON('{"key1":"val\\"ue1","key2":"value2","key3":["array1","array2"],"key4":"value4"}', 'key4');
# returns 'value4'

SELECT JSON('{"key1":"val\\"ue1","key2":"value2","key3":["array1","array2"],"key4":"value4"}', 'key5');
# returns NULL

Notes:
If you're trying to run this in the MySQL console, you'll need to set the DELIMITER to be something other than a semi-colon.  Before executing the above, run "DELIMITER $$" and after executing it, run "$$" and then "DELIMITER ;" to set your delimiter back to semi-colon.
If you're trying to run this in a Rails migration, don't forget to escape the back-slashes (i.e. '\\' should become '\\\\')
Although this handles several simple use cases of extracting JSON-encoded data, it is by no means comprehensive.   There are many JSON-encoded structures that this will not work on.  This will not work correctly with nested arrays, or with named hashes.
The performance of this is pretty slow.  A better approach would be to create a UDF that plugs into MySQL.  Here's a UDF to encode JSON data, but not decode: http://www.mysqludf.org/lib_mysqludf_json/index.php
Faster Implementation:
Here's a faster version, but it's not quite as robust:
CREATE FUNCTION JSON_FAST(`json` TEXT, `search_key` VARCHAR(255)) RETURNS TEXT DETERMINISTIC BEGIN
  IF INSTR(json, CONCAT('"', search_key, '":"')) THEN
    RETURN SUBSTRING_INDEX(SUBSTRING(json, INSTR(json, CONCAT('"', search_key, '":"')) +
           LENGTH(search_key) + 4), '"', 1);
  ELSEIF INSTR(json, CONCAT('"', search_key, '": "')) THEN
    RETURN SUBSTRING_INDEX(SUBSTRING(json, INSTR(json, CONCAT('"', search_key, '": "')) +
           LENGTH(search_key) + 5), '"', 1);
  ELSE
    RETURN NULL;
  END IF;
END

Here's some key differences:
SELECT JSON('{"key":"value \"plus quotes\""}', 'key');
# returns 'value "plus quotes"'
SELECT JSON_FAST('{"key":"value \"plus quotes\""}', 'key');
# returns 'value \'

SELECT JSON('{"key":["value1","value2"]}', 'key');
# returns '"value1","value2"'
SELECT JSON_FAST('{"key":["value1","value2"]}', 'key');
# returns NULL

Curb your Net::HTTP

2008-04-08T00:00:00+00:00

Curb is a ruby binding for libcurl. We’ve had sporadic issues with Net::HTTP, which this might aleviate via native dns, native timeouts, performance improvements, etc. It wouldnt be hard to re-implement ActiveResource, rfacebook, myspace-ruby, etc to use it instead. Anyone using this already?

sudo gem install curb

require 'rubygems'
require 'curb'
require "net/http"
require 'benchmark'

iterations = 40
Benchmark.bm do |x|
  x.report("curb") do
    iterations.times do
      c = Curl::Easy.perform("http://www.google.com")
      #puts c.body_str
    end
  end
  x.report("net/http")  do
    iterations.times do
      http = Net::HTTP.start("www.google.com")
      req = Net::HTTP::Get.new("/")
      res = http.request(req)
      #puts res.body
    end
  end
end

             user     system      total        real
curb      0.010000   0.030000   0.040000 (  4.019197)
net/http  0.140000   0.110000   0.250000 (  4.155106)

Agressive Timeouts On External API Calls

2008-03-31T00:00:00+00:00

One of the challenges with writing a Facebook or Bebo application is staying within a limit it gives you to respond with data before it shows the Application Did Not Respond page to a user. Having a content reach application calling external APIs, like Amazon or YouTube, with response times beyond your control, forces you to keep such calls short to allow extra time for processing. We usually wrap them in aggressive timeouts with a retry. As an example is this code from the Ruby Amazon E-Commerce REST Service API gem rewritten to limit a single call attempt to two seconds with one more retry.

Original Code

module Amazon
  class Ecs

    def self.send_request(opts)
      request_url = prepare_url(opts)

      res = Net::HTTP.get_response(URI::parse(request_url))
      unless res.kind_of? Net::HTTPSuccess
        raise Amazon::RequestError, "HTTP Response: #{res.code} #{res.message}"
      end
      Response.new(res.body)
    end

  end
end

Modified Code

module Amazon
  class Ecs

    class EmptyResponse
      def items; []; end
      def total_pages; 0; end
    end

    def self.send_request(opts)

      res = timed_try(request_url, 2) do |url|

        uri = URI::parse(url)
        req = Net::HTTP.new(uri.host, uri.port)

        # Agressive timeouts
        req.open_timeout = 1
        req.read_timeout = 2

        req.start { |http| http.request_get(url) }

      end

      res.kind_of?(Net::HTTPSuccess) ? Response.new(res.body) : EmptyResponse.new

    end

private

     def timed_try(url, attempts, &block)

       attempt = 1
       begin
         block.call(url)
       rescue Timeout::Error
         if attempt >= attempts
           RAILS_DEFAULT_LOGGER.warn "[amazon_api] gave up after attempt ##{ attempt } to get data from #{ url }"
           nil
         else
           RAILS_DEFAULT_LOGGER.warn "[amazon_api] attempt ##{ attempt } timed out on getting data from #{ url }"
           attempt += 1
           retry
         end
       end

     end

  end
end

Reviewing Application Health with HAProxy Stats

2008-03-27T00:00:00+00:00

One of the methods we use for checking the health of our applications is stats collected from HAProxy. We utilize it to see how many requests are scheduled for execution on mongrel instances. The graph is one indication of how our applications perform. When we launched the new version of the site three weeks ago, the graph for a single vertical (ReadingSocial) on a typical Tuesday looked like this:

So, between porting all verticals to Myspace, Orkut, Bebo, and enhancing the functionality, we spent some time on optimization. In addition to analyzing slow-query logs with mysqlsla, Aaron wrapped all external API calls (and we do a lot of them – to Amazon, Facebook, Myspace, etc) in slow monitoring so we could see where the latest external bottleneck was so we could fix it one by one. Three weeks later the graph became much more peaceful:

Viva LivingSocial!

2008-03-15T00:00:00+00:00

You haven’t heard from us for a while because we were working on a new project – LivingSocial – the web-site that spreads across all major social networks (Facebook, Myspace, Bebo, Orkut) allowing people to talk to each other about their interests without barriers. We are going to speak more about it in a dedicated blog while keeping this one for posts on Rails and other technologies.

Simple Google Pie Chart Graph in Rails

2008-02-16T00:00:00+00:00

Although there are many comprehensive libraries out there for google graphs in Rails, all we needed was a quick and simple pie chart. Here’s what we came up with:

module ApplicationHelper
  def google_pie_chart(data, options = {})
    options[:width] ||= 250
    options[:height] ||= 100
    options[:colors] = %w(0DB2AC F5DD7E FC8D4D FC694D FABA32 704948 968144 C08FBC ADD97E)
    dt = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-."
    options[:divisor] ||= 1

    while (data.map { |k,v| v }.max / options[:divisor] >= 4096) do
      options[:divisor] *= 10
    end

    opts = {
      :cht => "p",
      :chd => "e:#{data.map{|k,v|v=v/options[:divisor];dt[v/64..v/64]+dt[v%64..v%64]}}",
      :chl => "#{data.map { |k,v| CGI::escape(k + " (#{v})")}.join('|')}",
      :chs => "#{options[:width]}x#{options[:height]}",
      :chco => options[:colors].slice(0, data.length).join(',')
    }

    image_tag("http://chart.apis.google.com/chart?#{opts.map{|k,v|"#{k}=#{v}"}.join('&')}")
  rescue
  end
end

Here’s an example of how to use it:

<%=google_pie_chart([["Very Liberal", 8], ["Liberal", 22], ["Moderate", 9], ["Conservative", 4], ["Very Conservative", 1]], :width => 626, :height => 300) %>

And what it renders:

Leverage Rails Resource Routes on Facebook

2008-01-07T00:00:00+00:00

Since all canvas page views are proxied through POSTs, resource routes were hopelessly broken. The Facebook platform team was kind enough to add a new feature just for us rails folks: a new signed parameter that indicates the original request type (i.e. POST v. GET) against canvas pages.

Here’s a small patch you can stick at the end of environment.rb to restore REST-ful routes.

class ActionController::Routing::RouteSet
  def extract_request_environment(request)
    contents = request.body.read
    facebook_method_override = nil
    if contents =~ /fb_sig_request_method=([A-Z]+)/
      facebook_method_override = $1
    end
    request.body.string = contents
    { :method => facebook_method_override ? facebook_method_override.downcase.intern : request.method}
  end
end

UPDATE: Here’s a slightly better implementation that allows the .get? and .post? helpers as well as a few other things to work (thanks Chris Nolan for reminding us to update this blog entry):

ActionController::AbstractRequest.class_eval do
  def request_method_with_facebook_overrides
    @request_method ||= begin
      case
        when parameters[:_method]
          parameters[:_method].downcase.to_sym
        when parameters[:fb_sig_request_method]
          parameters[:fb_sig_request_method].downcase.to_sym
        else
          request_method_without_facebook_overrides
      end
    end
  end
  alias_method_chain :request_method, :facebook_overrides
end

Pretty SVN commit emails

2007-11-05T00:00:00+00:00

So i know it may be old school, but I like getting svn commit messages, especially when working on a small team. I found ElliotH’s post about A better subversion post-commit hook than commit-email.pl, but it wasnt entirely working.

Most of the credit goes to him, I just fixed the colorization.

[UPDATED]: for some reason, new lines werent looking good in all emails

#!/usr/bin/ruby -w

# A Subversion post-commit hook. Edit the configurable stuff below, and
# copy into your repository's hooks/ directory as "post-commit". Don't
# forget to "chmod a+x post-commit".

# ------------------------------------------------------------------------

# You *will* need to change these.

address="FOO@SOME_DOMAIN.com"
sendmail="/usr/sbin/sendmail"
svnlook="/usr/bin/svnlook"

# ------------------------------------------------------------------------

require 'cgi'

# Subversion's commit-email.pl suggests that svnlook might create files.
Dir.chdir("/tmp")

# What revision in what repository?
repo = ARGV.shift()
rev = ARGV.shift()

# Get the overview information.
info=`#{svnlook} info #{repo} -r #{rev}`
info_lines=info.split("\n")
author=info_lines.shift
date=info_lines.shift
info_lines.shift
comment=info_lines

# Output the overview.
body = "<p><b>#{author}</b> #{date}</p>"
body << "<p>"
comment.each { |line|  body << "#{CGI.escapeHTML(line)}<br/>\n" }
body << "</p>"
body << "<hr noshade>"

# Get and output the patch.
changes=`#{svnlook} diff #{repo} -r #{rev}`
body << "<pre>"
changes.each do |top_line|
  top_line.split("\n").each do |line|
    color = case
      when line =~ /^Modified: / || line =~ /^=+$/ || line =~ /^@@ /: "gray"
      when line =~ /^-/: "red"
      when line =~ /^\+/: "blue"
      else "black"
    end
    body << %Q{<font style="color:#{color}">#{CGI.escapeHTML(line)}</font><br/>\n}
 end
end
body << "</pre>"

# Write the header.
header = ""
header << "To: #{address}\n"
header << "From: #{address}\n"
header << "Subject: [SVN] #{repo} revision #{rev}\n"
header << "Reply-to: #{address}\n"
header << "MIME-Version: 1.0\n"
header << "Content-Type: text/html; charset=UTF-8\n"
header << "Content-Transfer-Encoding: 8bit\n"
header << "\n"

# Send the mail.
begin
    fd = open("|#{sendmail} #{address}", "w")
    fd.print(header)
    fd.print(body)
rescue
    exit(1)
end
fd.close

# We're done.
exit(0)

JRuby trunk == better/faster Rails performance

2007-11-03T00:00:00+00:00

So I remember a couple months ago playing with JRuby, and while fibonacci was super fast, Rails was way off.. ActiveRecord performance 6x-10x slower than MRI…

Looks like its getting better. Disclaimer: These are really really simple non-scientific tests.

Local mysql database, MYISAM, table people, with 2 columns, (id, name). 100k rows

> jruby -J-server -O script/console production
  >> Benchmark.measure {10000.times {Person.find :first}}.total
  => 2.286

> ./script/console production
  >>  Benchmark.measure {10000.times {Person.find :first}}.total
  => 1.7

Mongrel
  > ab -n 1000 http://localhost:3001/people/1
  Requests per second:    95.02 [#/sec] (mean)

On Glassfish v3 with: RAILS_ENV=production jruby -J-server -O -S glassfish_rails glass2, after a bit of warmup.

> ab -n 1000 http://localhost:8080/glass2/people/1
  Requests per second:    48.25 [#/sec] (mean)

turning logging mostly off

> ab -n 1000 http://localhost:8080/glass2/people/1
  Requests per second:    56.70 [#/sec] (mean)

All in all, thats impressive. Congrats to Charles, Ola, and the whole JRuby crew. I dont think i’ll be putting this in production yet, but I’m very interested to hear from others that have.

Super fast IP to lat/lng in Rails – Part 2

2007-10-31T00:00:00+00:00

In Super fast IP to lat/lng in Rails, I showed a solution for fast IP to lat/lng resolution in rails. I called it “Super fast” because it performed orders of magnitude faster the the RESTful interface, but it was also “Super fast” to implement. That being said, Kyle made a comment to check out the GeoIP gem. I had heard of MaxMind before, but I didnt want to spend hundreds of dollars to solve this problem. What I didnt know was they also have a free download of their “lite” datasource. They have a GeoLiteCountry and GeoLiteCity version, although only the City version has lat/lng info. They provide wrappers in most languages (including Ruby), and while Kyle’s suggested geoip, I found geoip_city on RubyForge which I like a bit better.

[UPDATED]: I had forgotten to include the install instructions for getting the GeoIP C library.

Install the C bindings, the gem (which isnt packaged as a gem for easy download) and get the data.

wget http://www.maxmind.com/download/geoip/api/c/GeoIP.tar.gz
tar -zxvf GeoIP.tar.gz
cd GeoIP
./configure && make && sudo make install

wget http://rubyforge.org/frs/download.php/27077/geoip_city-0.1.gem
sudo gem install geoip_city-0.1.gem
wget http://www.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz
sudo mkdir /usr/local/share/GeoIP
sudo mv GeoLiteCity.dat /usr/local/share/GeoIP/GeoLiteCity.dat

Then the ruby part:

>> require 'geoip_city'
>> g = GeoIPCity::Database.new('/usr/local/share/GeoIP/GeoLiteCity.dat')
>> res = g.look_up('4.2.2.2')
>> puts "lat: #{res[:latitude]} lng: #{res[:longitude]}"
lat: 38.0 lng: -97.0

So the question is (although its probably obvious), which is faster?

>>  Benchmark.measure { 1000.times {Hostip.geocode('4.2.2.2')}}.total
=> 1.02
>>   Benchmark.measure { 100000.times {g.look_up('4.2.2.2')}}.total
=> 0.5

Conclusion? The C library is MUCH faster than my database version. 200k req/s vs. 1k req/s. As a side note, I also tested the “geoip” gem, and it was about 3x faster than my database version.

But all in all, for the 5 minutes it took me to write the original version, it was fast enough… but MaxMind GeoLiteCity + the geoip_city gem is Super fast-er.

OpenSocial is here!

2007-10-31T00:00:00+00:00

As you may have seen on TechCrunch and in the New York Times, Google is days away from the developer launch of Open Social.

As a trusted tester/launch partner, we’ve been busy porting our social shopping suite (such as Visual Bookshelf) to OpenSocial.

I’ll follow up tomorrow with more thoughts, but I wanted to congratulate the OpenSocial team on the upcoming launch. We’re definitely excited!

Monitoring Rails Apps: Pulse + More

2007-10-26T00:00:00+00:00

Paul Gross from Thoughtworks recently created a pulse gem. The gem adds a simple action to your rails app, “/pulse”, which acts as a heartbeat.

The pulse gem currently defines the method as:

def pulse
     render :text => "OK"
  end

Then you can configure haproxy to monitor your application by hitting http://server/pulse and verifying the response is “OK”.

We’ve had a similar action in our applications for quite some time, although I really like the idea of externalizing it to a gem. Our implementation is slightly different though. Since so many rails applications depend on a database, I instead added:

def pulse
      rows = ActiveRecord::Base.connection.execute("select 1 from dual").num_rows rescue 0
      render :text => rows == 1 ? "OK" : "Error!"
  end

IIRC, “select 1 from dual” is the fastest query you can run against a database, in MySQL, Oracle, and Postgres. And if you have multiple databases, you can add union in the one pulse request, or have multiple actions.

Now you are not only testing your application is live, but that it can connect to the database.

Super fast IP to lat/lng in Rails

2007-10-22T00:00:00+00:00

In building the eye candy demo for the Graphing Social conference, I needed a quick way to geo-locate users by IP address. For Rails, GeoKit is an awesome plugin. It supports a list of providers, and overlays distance calculations, before_filter helpers, all sorts of good stuff.

It uses hostip.info to do IP to lat/lng, using their RESTful interface:

http://api.hostip.info/get_html.php?ip=12.215.42.19&position=true
  Country: UNITED STATES (US)
  City: Sugar Grove, IL
  Latitude: 41.7696
  Longitude: -88.4588

This is great, but its just too darn slow to geolocate 12 users a second. All I needed was a FAST ip to lat/long lookup mechanism. Luckily, HostInfo provides the raw data. Here’s how I built super fast GeoLoc by IP method.

Download, create a DB, and import the data.

wget http://hostip.ww.com/hostip_current.sql.gz
gunzip hostip_current.sql.gz
mysqladmin -uroot create hostip
mysql -uroot hostip < hostip_current.sql

Create a simple HostIp class. Note: I need to dynamically build this query b/c the data is sharded across tables by the A class of the IP.

class Hostip < ActiveRecord::Base
  def self.geocode(ip)
    a,b,c,d = ip.split(".")
    self.set_table_name "hostip.ip4_#{a}"
    ip = find(:first, :select => "lat,lng",
                :joins => %Q{INNER JOIN hostip.cityByCountry
                               ON hostip.ip4_#{a}.city = hostip.cityByCountry.city
                               AND hostip.ip4_#{a}.country = hostip.cityByCountry.country},
                :conditions => ["b = ? and c = ?", b,c])
    if ip
      [ip.lat, ip.lng]
    else
      ["",""]
    end
  end
end

Usage

>> Hostip.geocode("4.2.2.2")
=> ["39.944", "-105.062"]

Thats it! Now you can geolocate by IP in your own datacenter.

Thanks again to the guys at HostIp for sharing this data!

The Importance of Eye Candy (and no, not the kind you’re thinking)

2007-10-19T00:00:00+00:00

We just got back from several conferences around the social media space. CommunityNext was a developer-centric event that allowed for many interactions and trading of best practices with several top Facebook developers. GraphingSocial was a slightly larger gathering that brought together an impressive list of speakers including Reid Hoffman, Michael Arrington, Danny Sullivan and others. It was a great opportunity to network, see some of the latest Facebook applications, and just generally talk shop with investors and developers in the social media space. Sometimes at these events it’s so easy to get lost in the shuffle, that it is tough to talk with the people you want to talk to. We did, however, have what we determined to be a strategic advantage: Eye Candy

Here’s how it went down:

Step 1: Several weeks before event ~ Randomly sign up for booth and to be a conference sponsor without really having a plan other than “A booth seems like a good idea. We can talk to people. And stuff.”

Step 2: 7 days and 3 minutes before event ~ Contemplate whether “Hungry Machine Hula Hoops” send the right message.

Step 3: 7 days and 2 minutes before event ~ Return to fetal position in corner

Step 4: 3 days before event ~ Eureka! An idea is born. Everybody loves Twittervision and everybody loves money – let’s combine the two!

Step 5: 2 days before the event ~ Begin building application that shows the geographic distribution of impressions and clicks within the Hungry Machine ad network (pic below)

Step 6: 1 day before event ~ Take break from building application because In & Out Burger is just too damn good.

Step 7: Later in that day ~ Eat 3rd double double

Step 8: Day of event ~ Deploy application; set up Apple 30 inch monitor that seems to emit a radioactive hue. Notice third arm grow out of small child as he passes monitor

Step 9: Booth now open for business! Potential clients, partners and just plain interesting folks start to roll in. They cannot resist the lure that is a 30 inch monitor with people’s heads popping up on a world map.

In all seriousness, the above process was only slightly embellished, but really did serve a purpose in how successful and how much we got out of the GraphingSocial conference. Having an initial, built-in talking point is a great way to break the ice with people you are interested in talking with, and generally helps reveal someone’s area of interest within the first minute or two of conversation. We would not have had nearly as successful of a conference without a little extra pizzazz.

Marketing your Facebook App

2007-09-28T00:00:00+00:00

In the good ol’ days of, let’s say…. three months ago, life was simple. The birds chirped a bit louder, the grass was a little greener, and, best of all, you could build an app on Facebook, submit it to the directory, tag it with a category or two, and see it nestle in among a smattering of other applications and watch your traffic grow at an obscene pace. Those days are over. Kuput. Finito. Building an app on Facebook and letting it “do its thing” is an increasingly difficult, if not impossible thing to do. The good news is that there are a host of traditional and non-traditional ways to gather good, quality traffic for your application. The bad news is that, very likely, it’s going to cost you:

Buy an initial seed number of users: If you can generate 10/20/30 thousand users on your app, your installed user base should be large enough at that point in time to see if it sinks or swims. You’ll likely have crossed a wide-demographic range by that point in time and have been able to see if you’ve built your app to properly leverage the viral touch points and social graph.

Generate “off-Facebook” buzz: Sure, Facebook pages aren’t crawlable. And sure, your app can’t live outside of Facebook. But that doesn’t mean there isn’t a universe (you know that thing… what’s it called… that’s right, the world wide web) that does still exist outside of Facebook. If you issue a web-based press release, it could get picked up by a variety of news sources who could reference it and Google will definitely crawl it. If you know any bloggers that would have an interest in your application, reach out to them and give them a sneak peak under the hood. If you have an existing destination site, leverage it! 40+ish million people are on Facebook, which means there is a reasonable chance the viewer of your destination site is also a Facebook member.

It’s in the name: This is certainly just a hypothesis, but I think a very reasonable one. If someone sends me an app, and I can immediately understand the concept of the app pre-install, I can make a decision point prior to installation about whether or not I think the app would be useful. I may not install it, but at least there is a chance that I will. If I don’t immediately grok how the app will work prior to install, there is no chance I will install it – what is it going to ask of me? Will I have to go out of Facebook to register for a site?

Just a few thoughts to consider when figuring out how to make your app go from 0 to 60(thousand users, that is) in 5 seconds flat.

And so it begins

2007-08-25T00:00:00+00:00

And so it begins. Welcome to the maiden posting for the Hungry Machine blog. We’re 4 guys bringing our expertise in Ruby On Rails and experience from AOL, Revolution Health, and Blockbuster together to build some great new products and services. For those of you that read us at RevolutionOnRails, we have a new addition, Tim O’Shaughnessy, who will be handling a bit more of the product-side of the house for us.

We plan to continue exploring Rails, sharing our expertise with others via this blog and by releasing more plugins, and diving into new areas and applications, ranging from our continued development of Facebook applications to entirely new fields.

Lastly… why Hungry Machine? Well, we think it represents our mentality and effort we’ll put into our everyday activities… we’re hungry for success. Check back often to see what we’re up to.

Who we are:

Aaron Batalion: Aaron has been banging the Rails drum since 2005 and has been deep in the Rails community ever since. A lead portal architect for RevolutionHealth.com, Aaron was also a portal architect for Blockbuster.com.

Eddie Frederick: Eddie was a RoR lead at Revolution Health and has become one of the leading developers in the Facebook space, authoring applications with well over 1M users. He specializes in creating end-to-end user experiences from deployment to application UI.

Tim O’Shaughnessy: An entrepreneur at heart, Tim led the Product Development Department – Consumer Products at Revolution Health before helping found Hungry Machine. Tim also has has launched successful products at AOL and enjoys long walks on the beach.

Val Aleksenko: Val brings over 17 years of software development experience to Hungry Machine. Val’s like MacGyver but with software; give him a stack, a deployment script, and a Red Bull and watch him go.

Reconfiguring the whole rails stack via a central YAML file

2007-08-20T00:00:00+00:00

The challenge with hosting of multiple Rails-based Facebook applications is that the amount of users grow quickly. To address this problem we are using EC2 nodes that we can expand/shrink as the demand grows. The price/performance ratio isn’t quite what we first expected, so we are moving toward having a few dedicated boxes instead. Another problem that we add at least a couple of applications a week. On each box that hosts them, we need to reconfigure monit, haproxy, nginx, logrotate and nagios.

To mitigate both issues on dedicated boxes, we resolved to have a central configuration definition in svn with individual box configurations keyed on localhost name. A ruby script regenerates all those aforementioned configuration files from ERB-processed templates when it is run on a box and bounces the services. A sample config looks like:

dedicated-1:

    description: "The dedicated box #1"
    ip: 64.233.167.99
    failover: dedicated-2

    apps:

        bookshelf:
            port: 5000
            instances: 20
            response: Book

        ljconnect:
            port: 6000
            instances: 7
            virtual: ljconnect.hungrymachine.com
            response: Journal

That definition would generate a monit config with 20 instances of the bookshelf application and 7 instances of the ljconnect application plus all other configurations (including nagios health checks expecting the response value) . It is all possible because we adopt a fixed application deployment file structure and port numbering conventions (via offsets) for all servers.

Using a ruby-based AIM notifier in nagios

2007-08-14T00:00:00+00:00

If you use nagios for monitoring of your rails instances, you might want to get notification not only via email or SMS-messages but to your AIM when you are online. The script (libexec/aim_notifier.rb) utilizes the Net::TOC gem for sending out notifications:

#!/usr/bin/env ruby

 require 'rubygems'
 require 'net/toc'

 user = 'your_bot_name'
 password = 'bot_password'

 msg = ARGV[0].to_s.gsub('\n', "\n")

 client = Net::TOC.new(user, password)

 client.connect

 sleep 3

 buddies = []

 client.buddy_list.each_group { |g, b| buddies = b if g == 'Friends' }

 buddies.each do |b|
   b.send_im(msg) if b.available?
 end

 sleep 3

 client.disconnect

You need to add any account you want to be notified to bot’s friends (either by logging to AIM using the bot account or using Net::TOC’s ability to add friends).

The last piece is to add a new notifier in etc/objects/commands.cfg as:

 define command{
         command_name    notify-service-by-aim
         command_line    $USER1$/aim_notifier.rb $ARG1$ $ARG2$ "***** Nagios *****\n\nNotification Ty
 pe: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState:
 $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$"
         }

and to append it to the list of notifiers defined for a contact template in etc/objects/commands.cfg:

 service_notification_commands   notify-service-by-email,notify-service-by-aim

Repeat the configuration if you want to use the AIM notification for hosts as well.

Routing to the initial action after facebook application install

2007-08-14T00:00:00+00:00

Some facebook applications might have multiple entries. For example, a user might be adding an application (action – new) or replying to an invitation (action – reply, param – id). Since the UI for Facebook application configuration allows to provide only static Post-Add URL it might seem like there is no way to route users back to the original action if they tried to reach when the application has not been installed for them. Luckily, we have full control on the destination via the next paramater of the post install URL. All we need is to build a URL using the incoming call parameters with the exclusion of Facebook-specific ones.

This is an example for Facebook on Rails based code that might go to the application controller:

class ApplicationController < Facebook::FBMLController

protected

  before_filter :require_facebook_install

  def require_facebook_install
    if in_canvas? && !fbsession.is_valid?
      redirect_to fbsession.get_install_url(:next => url_for(post_install_params))
      false
    end
  end

  def post_install_params
    params.merge(:init => true).delete_if { |k, v| k.starts_with?('fb_sig') }
  end

end

Notice that the code sets the init parameter so it can be used to identify a post install call

Indentifying users who just installed your facebook apps

2007-08-14T00:00:00+00:00

Sometimes it is useful to do some action on a Facebook user right after your application has been installed by the user. For example, you might want to push some default FBML to user’s profile in case he does not complete the action you expect him to do after installation. Facebook application configuration allows to provide Post-Add URL to route users to the destination url after the application install. It could be a dedicated post_add action or, in case of a default action where you have some code in the controller and since Facebook limits amount of redirects you can use, it could be a parameter to the url, like &init=true, used to identify that it was a post-install action and execute on it.