LabRatz

One of the shocking things about going into production is to go on a little break or even get away from the house only to gasp in horror as you stare at a blank 404 where your earth shaking rails app should be when you get back online.

Integrating stuff like memcached into your rails app is mostly trivial thanks to the work of some wonderful rails hackers but keeping this moving parts moving is a task in itself. What’s the use of having a killer app that only works on your dev box right?

Naturally, the good thing about development is that everyone else has been there and done that. The other day I discovered monit thanks to igvita.com.

Sometimes a variety of bugs in plugins or your code can bring your web app down to its knees. This doesn’t even include those moments when your hosting service goes down for whatever reason whether it be maintenance or trouble. Smart developers (that’s you) will already have crucial parts of their service automatically restart along with any hardware restarting either by setting their chkconfig or whatever their system uses. Most of the libraries that we depend on are mostly reliable most of the time like the MySQL server that we only need to worry about start up initialization (although even this depends on how hard you push it).

Unfortunately, for most beginners, rails still continues to be a dream to develop in and a nightmare to deploy unless you know what you’re doing. Also random application crashes will increase the more moving parts you have such as memcached, mongrel, and acts_as_ferret.

Enter Monit

Monit is a wonderful little tool to ensure that your little web app keeps chugging along happily. It’ll monitor whatever you tell it to and ensure that they continue to run under the conditions you specify.

Using it with mongrel_cluster will ensure that users of your web app will be served a proper web page even as one of the other mongrels get culled for misbehaving.

In order to fully take advantage of mongrel_cluster and monit, you need to grab the mongrel_cluster prerelease. You can of course, use monit to serve as your cluster management but as Ezra notes, mongrel_cluster does a better job of keeping memory consumption in check as well as dealing with orphaned pids (only prerelease) that has given anyone that deploys rails a headache or two.

I can also vouch for the fact that monit has a slew of cool features to help you really keep the system humming along nicely. Remember memcached? Well, as much as it provides performace boosts to your app, it also introduces a potential headache. You see, when used in conjunction with mongrel you need to be careful to ensure that memcached is running before your mongrel starts. Otherwise you get weird behavior such as your flash[:notices] not showing up.

So, suppose your memcached crashes but mongrel stays up. Even if memcached got restarted, you’re only half way there since you need to restart mongrel immediately after that. Fortunately, this is trivial with monit because you can simply set a hierarchy by stipulating a depends on condition. This will ensure that whatever is being depended on will be restarted by monit first before the other processes get started.

  set daemon 60
  set init
  set mailserver localhost
  set mail-format { from: monit@monit.com }
  set alert who@gmail.com

  set httpd port 2812 and
      use address localhost
      allow localhost        

  check process mongrel_8000
     with pidfile /var/run/mongrel_cluster/mongrel.pid
     start program = "/usr/local/bin/mongrel_rails cluster::start -C /root/inbited.com/current/config/mongrel_cluster.yml --clean --only 8000" 
     stop program = "/usr/local/bin/mongrel_rails cluster::stop -C /root/current/config/mongrel_cluster.yml --clean --only 8000" 

     if totalmem is greater than 60.0 MB for 5 cycles then restart
     if cpu is greater than 50% for 2 cycles then alert                  
     if cpu is greater than 80% for 3 cycles then restart                
     if loadavg(5min) greater than 10 for 8 cycles then restart          
     if 3 restarts within 5 cycles then timeout
     depends on memcached                          

     if failed port 8000 protocol http                
         with timeout 10 seconds
         then restart
     group mongrel

  check process mongrel_8001
     with pidfile /var/run/mongrel_cluster/mongrel.pid
     start program = "/usr/local/bin/mongrel_rails cluster::start -C /root/inbited.com/current/config/mongrel_cluster.yml --clean --only 8001" 
     stop program = "/usr/local/bin/mongrel_rails cluster::stop -C /root/current/config/mongrel_cluster.yml --clean --only 8001" 

     if totalmem is greater than 60.0 MB for 5 cycles then restart       
     if cpu is greater than 50% for 2 cycles then alert                  
     if cpu is greater than 80% for 3 cycles then restart                
     if loadavg(5min) greater than 10 for 8 cycles then restart          
     if 3 restarts within 5 cycles then timeout  
     depends on memcached                               

     if failed port 8001 protocol http                
         with timeout 10 seconds
         then restart
     group mongrel

  check process mongrel_8002
     with pidfile /var/run/mongrel_cluster/mongrel.pid
     start program = "/usr/local/bin/mongrel_rails cluster::start -C /root/inbited.com/current/config/mongrel_cluster.yml --clean --only 8002" 
     stop program = "/usr/local/bin/mongrel_rails cluster::stop -C /root/current/config/mongrel_cluster.yml --clean --only 8002" 

     if totalmem is greater than 60.0 MB for 5 cycles then restart       
     if cpu is greater than 50% for 2 cycles then alert                  
     if cpu is greater than 80% for 3 cycles then restart                
     if loadavg(5min) greater than 10 for 8 cycles then restart          
     if 3 restarts within 5 cycles then timeout 
     depends on memcached                         

     if failed port 8002 protocol http              
         with timeout 10 seconds
         then restart
     group mongrel

  check process memcached
     with pidfile /tmp/memcached.pid
     start program = "/etc/init.d/memcached start" 
     stop program =  "/etc/init.d/memcached stop" 
     if failed host 127.0.0.1 port 11211 then restart

This sample config is based on igvita and Ezra’s. As you can see there’s a “depends on memcached” call. What this does is set a hierarchy so if memcached gets restarted, mongrels will get restarted afterwards. On initialization, memcached will get started first so your web app can function properly.

I plan on tweaking this so that I can be extra sure to have essential services like mysql and nginx humming along properly as well (although I’ve yet to see these go out, the load on my service is currently close to nil).

I’ve only touched on the tip of the iceberg really and I just think I might even check out how to use this to keep rogue apps from eating up too many resources on my mac.

Now, apps should never ever be crashing to start but things happen and rails will always have its kinks until well it’s… enterprise ready™. But hey, it’s more important that your app stays humming especially if you have moving parts like memcached or ferret that form an integral part of your service.

Monit will also dutifully email you of any incidences that trigger it. This combined with exception notification can provide you with valuable debugging info.

Monit makes Mongrel play nice! – igvita.com

Mongrel Mongrel Cluster 1.0.1.1 Prerelease: Healing power of the pack!