Benchmarking and Load Testing with Siege

Posted on

When launching a web application, you want to ensure that it will be able to handle the load of users accessing the site. We can get an idea of how it will perform by using a benchmarking tool. While there are many popular tools such as ab (Apache HTTP server benchmarking tool) used by developers today, I want to introduce you to Siege.

Why Siege

Siege is an HTTP load testing and benchmarking utility. It has a similar interface to ab, which will make transitioning to the tool almost seamless. Also, instead of testing against a single URL, Siege allows you to test against multiple. This allows for a more real-world simulation of how a user would use your system.

Install

If you are running Mac OS X, you can install Siege via homebrew by running the following:

brew install siege

Benchmarking

To benchmark URLs in Siege, set the -b option. This will run your tests with no delay for throughput benchmarking. Without the -b benchmark option, Siege adds a one second delay between requests for each simulated user.

You can have this benchmark run forever, or you can specify a specific period of time. To set the duration of the test, use the -t NUMm option. The NUMm format is a representation of time. Here are some time format examples:

  • -t60S (60 seconds)
  • -t1H (1 hour)
  • -t120M (120 minutes)

To demonstrate running a benchmark, let's run Siege against this blog:

siege -b -t60S  http://blog.remarkablelabs.com 

Running the above command will output the following performance statistics:

** SIEGE 2.72
** Preparing 15 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.

Transactions:               4066 hits
Availability:             100.00 %
Elapsed time:              60.00 secs
Data transferred:          13.73 MB
Response time:              0.22 secs
Transaction rate:          67.77 trans/sec
Throughput:             0.23 MB/sec
Concurrency:               14.95
Successful transactions:        4066
Failed transactions:               0
Longest transaction:            3.28

Based on the results, the server was always available and had an average response time of 230ms. The average amount of requests per second was 67.77.

Load Testing

Combined with a tool like New Relic, Siege's load testing features will give you insight into your application's weak spots. When creating a load test, you should set the -d and -c NUM options. The -d NUM option sets a random interval between 0 and NUM that each "user" will sleep for between requests. While the -c NUM option sets the concurrent number of simulated users.

Here is an example of a load test of a Heroku application:

$ siege -c50 -d10 -t3M http://some.application.com

** SIEGE 2.72
** Preparing 50 concurrent users for battle.
The server is now under siege...[alert] socket: 100081664 select timed out: Operation timed out

Lifting the server siege...      done.

Transactions:               1398 hits
Availability:              99.93 %
Elapsed time:             179.94 secs
Data transferred:           4.70 MB
Response time:              1.21 secs
Transaction rate:           7.77 trans/sec
Throughput:             0.03 MB/sec
Concurrency:                9.40
Successful transactions:        1398
Failed transactions:               1
Longest transaction:           29.47
Shortest transaction:           0.10

During that test, 50 concurrent connections bombarding the server resulted in an average response time of 1.21 requests per seconds and a single failed transaction. The Longest transaction statistic was 29.47, which allows me to assume the failure was due to a Heroku timeout.

Multiple URLS

To really simulate an experience a user may take, you can provide a file of URLs separated by line to the -f option. Another recommendation is to set the -i option that tells Siege to randomly select URLs from the text file, resulting in a more real-world test.

siege -c50 -d10 -i -f site.txt

Comments

comments powered by Disqus