Tribune DataViz

Matters of interest, from the data reporters and developers across Tribune Publishing

Bees with machine guns! Low-cost, distributed load-testing using EC2.

with 36 comments

The challenge

You don’t get to start optimizing until you’ve made the servers fucking cry.
Brian Boyer

In June we launched Chicago Breaking Business, which is, at least in terms of traffic, our largest site so far. Both before and during that project I had been iterating toward a load-testing solution that would meet the challenge of the above quote and provide us with a reasonable certainty that sites of this size (and larger) would not collapse under sudden waves of traffic. The traditional load-testing tools we tried (ab, flood, jmeter, etc.), while often useful, suffered from a number of problems when trying to simulate this sort of event:

  • Testing clients may have an operating system imposed maximum number of out-bound connections.
  • Testing a staging environment from a development workstation introduces network latency, firewalling, and traffic shaping as variables in the tests.
  • Testing from a single client will cause Amazon’s Elastic Load Balancer to route all connections to a single application server, no matter how many are available. (so-called “sticky sessions”)

What we needed was a solution that allowed us to use multiple, topographically-close clients to all simultaneously assault our servers with traffic.

Put it another way, what we needed were bees with machine guns.

The solution

Bees with machines guns is a fabric script that I put together to aggressively load-test our sites. It provides methods to spin up an army of EC2 instances (the bees), assault a target server with a configurable amount of traffic, and then spin those instances back down.

Here is an example of what a Bees session might look like:

$ fab -f up:2
Connecting to the hive.
Attempting to call up 2 bees.
Waiting for bees to load their machine guns...
Bee i-df450ab4 is ready for the attack.
Bee i-dd450ab6 is ready for the attack.
The swarm has assembled 2 bees.

$ fab -f attack:,1000,100
Read 2 bees from the roster.
Connecting to the hive.
Assembling bees.
Each of 2 bees will fire 500 rounds, 50 at a time.
Stinging URL so it will be cached for the attack.
[localhost] run: curl >> /dev/null
Organizing the swarm.
Bee 0 is joining the swarm.
Bee 1 is joining the swarm.
Bee 0 is firing his machine gun. Bang bang!
Bee 1 is firing his machine gun. Bang bang!
Bee 0 is out of ammo.
Bee 1 lost sight of the target (connection timed out).
Offensive complete.
     Target failed to fully respond to 1 bees.
     Complete requests:		500
     Requests per second:	81.670000 [#/sec] (mean)
     Time per request:		612.188000 [ms] (mean)
     50% response time:		358.000000 [ms] (mean)
     90% response time:		1541.000000 [ms] (mean)
Mission Assessment: Target successfully fended off the swarm.
The swarm is awaiting new orders.

$ fab -f down
Read 2 bees from the roster.
Connecting to the hive.
Calling off the swarm.
Stood down 2 bees.

(Note: Bees assumes you have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set as environment variables–just as you would if you were using the Amazon command line tools.)

Using the Bees I was able to verify that our new load-balanced and Varnished servers could handle a sustained load significantly beyond our requirements. Obviously this testing strategy doesn’t attempt to address all scenarios and it should be used in conjunction with traditional testing tools such as jmeter. That being said, it also provides a pretty interesting example of using EC2 to solve problems that can really only be properly addressed with distributed computing. This particular example is very well-suited to validating caching and scaling strategies.

The caveat

If you decide to use the Bees, please keep in mind the following important caveat: they are, more-or-less a distributed denial-of-service attack in a fancy package and, therefore, if you point them at any server you don’t own you will behaving unethically, have your Amazon Web Services account locked-out, and be liable in a court of law for any downtime you cause.

You have been warned.

Now then–how does your server handle thousands of simultaneous connections?


Written by Christopher Groskopf

July 8, 2010 at 9:12 am

36 Responses

Subscribe to comments with RSS.

  1. And here I’ve been using Siege like a sucker.

    Jeremy Bowers

    July 8, 2010 at 9:23 am

  2. I approve of the mental image this conjured up. Oh, and the code. But mostly the mental image.

    Niran Babalola

    July 8, 2010 at 3:55 pm

  3. Did you look at The Grinder? It’s got some nice tools for this sort of stuff, and is Pythonic. Curious to know if you evaluated it an rejected, and if so why.

    Josh Koenig

    September 21, 2010 at 12:45 pm

    • The short answer: I hadn’t heard of that load testing tool until you mentioned it.

      That said, in just glancing at it I’m struck by the intuition that–like most Java apps–it might be a little heavy for our needs. In addition to our distributed testing needs, part of the reason I eschew JMeter is because its just too many options and too much configuration for our immediate concerns.

      I look forward to giving it a more thorough look next time I have the opportunity to survey the available tools.

      Christopher Groskopf

      September 21, 2010 at 2:49 pm

  4. Great name and a useful script. Though I’m wondering if using EC2 instances to test other EC2 instances or other Amazon Web Services is not going to result in lots of internal Amazon traffic and not real world external traffic? I guess that doesn’t matter for some tests but I imagine it could skew things for other tests?

    Paul M. Watson

    October 29, 2010 at 4:58 am

    • Paul,

      I was wondering about that same thing, so I did some quick ping tests to see what kind of latency you might expect:

      Basically, it looks like if your bees are in the same region as your ec2 app, your network is going to be unrealistically fast. If you put your bees in another region though (eg. West vs East), the latency is what you’d expect for the internet and your tests will be pretty reasonable. Of course there are tools that fake network latency and such, but since Bees With Machine Guns is all about simple, just switch out your boto config to use the different region before calling up, and you should be good to go.

      Wes Winham

      November 9, 2010 at 1:50 pm

      • Thanks for the replies Wes and Christopher. Good points about the purpose of Bees, I’ll keep it in mind.

        Paul Watson

        November 9, 2010 at 3:18 pm

    • Paul, Wes’s recommendation is certainly a good one if this is your use case, however, the point of the Bees is not to measure realistic throughput with network constraints, but to measure the theoretical maximum performance of the target server. To that end, I actually believe its an advantage to have the load testing boxes as close to the target server as possible. When I use the bees I want to know how much traffic we would need before the server fell over, not what real world traffic might look like.

      Of course, you could probably modify the bees to suit other purposes, and I’d be curious to see someone try that.

      Christopher Groskopf

      November 9, 2010 at 2:35 pm

  5. […] Bees with machine guns! Low-cost, distributed load-testing using EC2 Bees with machine guns. Awesome name, awesome log messages, and a great looking tool. (tags: perftesting ec2 aws loadtesting) […]

  6. […] Bees with machine guns! Low-cost, distributed load-testing using EC2. […]

  7. […] Bees with machine guns! Low-cost, distributed load-testing using EC2. « News Apps Blog This will probably prove to be handy at some point for me. Using the cloud to load test servers and stuff. What a great idea! (tags: cloud EC2 amazon loadtesting) […]

  8. […] only surprising that somebody didn’t bring it down earlier, especially when things like Bees with machine guns! are […]

  9. I did some load testing for which we just released but nothing as elaborate as this. (So no-one should be surprised when it buckles.) Great job (and when do you have time for sleep?)

    BTW, I think Bees With Machine Guns should be franchised into a game.

    Nick Oba

    December 12, 2010 at 10:41 am

  10. I like the products of your blog, maybe u will like these things of my blog!

    custom baseball jerseys

    February 23, 2011 at 9:14 pm

  11. […] say the least. Someone linked me to a amazing little script written by the Chicago Tribune called Bees with Machine Guns . Its essentially a script that will smash your site with Amazon EC2 Instances . Using your creds it […]

  12. […] can learn more about bees from the author of the script, Chicago Tribute. It’s designed to be light-weight and easy to use, and I agree! I had a bit of trouble using […]

  13. […] Bees with machine guns! Low-cost, distributed load-testing using EC2 at the Chicago Tribune for the write-up. This entry was posted in Rambling. Bookmark the […]

  14. You guys rock. I was failing miserably to load test my application from my laptop — but the swarm of bees was able to do it. Thank you!

    Jason Padvorac

    August 3, 2011 at 5:30 pm

  15. […] van de New York Times,…k-your-server/ en de Chicago Tribune,…-machine-guns/ over het gebruik van oa Varnish. Als je gaat zoeken kun je hierover nog veel meer succesverhalen […]

  16. One thing you never discuss is how in instrumented the system under test when you did this? How did you know what the load was doing to the server? How close it was to failure and what was the primary constraint? (cpu, memory, network, disk, etc) What did you use to monitor that and what stats did you monitor?


    March 28, 2012 at 12:09 pm

  17. If you need something more powerful, try Load Tester LITE. It also features EC2 integration, but provides a friendly UI and can test many scenarios containing many pages along with all the required resources for each page. You get live graphs during the test as well as an extensive post-test report.

    Chris M

    June 8, 2012 at 6:09 am

  18. It really looks like a usefull tool. Is there any limitation in the number of total requests and concurrent requests that the utility can handle?


    June 27, 2012 at 8:12 am

  19. Has anyone created an AWS IAM policy for a bees run? I’m trying to cobble one together, but it’s slow going (basically waiting to see where boto throws an exception each time I change it). It’d be nice to not have to have the load tester have full access.


    Chris Hecker

    August 6, 2012 at 5:42 pm

    • can some one put steps together needed to execute load test using Beeswithmachineguns in windows environment. Thanks.


      October 23, 2012 at 3:09 pm

  20. Very cool. We are going to try it out at @cinchcast

  21. […] only has an awesome name but meets requirements 1-4.  Here's newsapps's blog post: Bees with machine guns! Low-cost, distributed load-testing using EC2.  Source: beeswithmachinegunsEmbed QuoteComment Loading… • Tue    Benjamin […]

  22. I am a noob Linux user here. Just wondering..
    I received a message your bees are peace-loving hippies. My first thought I have no apache ab on my instance.
    I login as root to perform yum install httpd-tools -y – got the message already installed and latest version.

    hmmm bad feeling – launched an attack again… and again my bees still being a swarm of hippies….

    Any suggestion ?

    Kloud Gate

    May 9, 2013 at 9:29 pm

  23. Even I got a message “…your bees are peace-loving hippies”.
    Does anyone have a clue to this issue ?


    May 13, 2013 at 9:36 am

  24. Great tool!
    I have noticed that the bees seem to ignore ‘-g group’ and use the default security group instead. If this is happening for you, be sure to open up port 22 on the default security group instead of the security group you assigned the bees.

    Graham Be Kiind

    June 26, 2013 at 12:39 pm

  25. Great little piece of software. I got one question though. Given its using this command:

    “ab -r -n %(num_requests)s -c %(concurrent_requests)s %(options)s “%(url)s”

    for each “bee”, than technically if you spawn 5 bees with 10 concurrency, the concurrency becomes actually 50 or I am missing something?

    Feras Salim

    October 20, 2013 at 2:39 pm

  26. What’s “low cost”? This is a very relative term. To a large company low cost load testing could be a thousand dollars or more.


    October 26, 2013 at 11:55 pm

  27. […] idea for bees with machine guns originates back to the Chicago Tribune, as they launched one of their largest sites in terms of traffic and had to find a suitable load […]

    | Happy Cloud Solutions

    December 14, 2014 at 4:36 am

  28. […] you could get away with a nice little tool like this. The tool is a few years old based on this blog. It’s gone through 2 releases now. It might be a worthy project to make a pull request […]

  29. […] generate traffic to your staging environment—the Paper magazine team used the evocatively named Bees with Machine Guns—then view the Scalability Analysis Report to see how your site […]

  30. […] under maximum connection load before rolling it out to prod. If you’re in AWS, you can use Bees With MachineGuns to put it under heavy concurrent […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: