Throughout the duration of the Dojo Dashboard project we have contended with a persistent bug. When hitting the URL for the app on the staging server, not all of the widgets will be displayed, and when they do not show up, a small div the width of a column shows up where the a widget should be. We noticed a pattern though -- whenever this happened, it was always five widgets that showed up.

We mentioned this to Micah, and he said something to the effect of, “Oh, yeah, I know why that is happening.” Great. Do enlighten us? “It has to do with the throttling of our servers.” One line of defense against a DOS attack is to throttle your server so that it only responds to requests at a fixed rate (average requests per second) but also has some leeway to handle extra requests in a given time frame (burst limit). A good way to think about this is to envision the process with the Leaky Bucket metaphor.

One of the stories for this week is to make it so that an admin for 8th Light can alter the nginx (pronounced “engine ex”) configuration to any of the servers individually. In order to do this we needed to dig into Boucher, a custom-built open-source tool for provisioning servers. Put plainly, Boucher is “a suite of Rake tasks that simplify your AWS deployment strategy,” and we simply needed to add to the recipe a few lines of code. Just three simple steps!

1) nginx ‘limit_req_zone’ module

This module is what allows us to throttle requests and is built into nginx. A little bit of googling helped us figure out where in our nginx_site.config.erb file to put our embedded ruby.

2) cookbook definition

Next up we needed add some params to our NGINX definition file like so:

:avg_request_rate => 1
:burst_limit => 5

 First in the params, then then also in the template block.

3) cookbook recipe

Finally we needed to add the correct configuration to our server recipe file so that all of the widgets will be loaded at once. Because we have 9 widgets plus 2 addition weather widgets for other cities and 1 extra metra widget, 15 seems like a good burst limit. We decided to stick with an average of 1 request per second though. To the nginx app “dojo do block we add:

avg_request_rate 1
burst_limit 15

Voila! After deploying and restarting our dojo server, all widgets load immediately! And because we use Boucher for all of our AWS deployments, adjusting throttling for any of the other servers is as simple as adding a couple lines to the cookbook definition and recipe for that particular server. For further reading on the topic you can check out this blog post by Micah about servers in source control.

P.S. If you haven’t figured it out yet, the reason why only five widgets showed up is because the nginx throttling defaults to a burst size of 5 and 1 request per second. Thus, the first request to the URL counts as one request, and five subsequent requests cause the widgets to show up, reaching the limit of six requests in one second.