Scotish Ruby Conference - Rocket Fuelled Cucumbers
Posted on March 26th, 2010 in Uncategorized
Joseph Wilks did an excellent talk on how to speed up and scale your Cucumber integration tests.
What is Cucumber?
Cucumber is a Behavior Driven Development (http://bit.ly/r7L4) framework for integration tests.
Cucumber uses plain-text descriptions as automated tests. The language used is called Gherkin.
The plain-text descriptions can be written by anyone, but usually they are written by the management team of the project who will do a role play for each of the users who use the system. The QA team would usually be involved in every step of creating the descriptions.
The honey moon period
You have started to write your application, writing a few cucumber descriptions, a little RSpec. You have your tests running with AutoTest and even have a notification from Growl (on a mac) to tell you that your tests have passed, but this can be short lived.
As your application grows, so do the amount of cucumber tests. This means it takes more time for the tests to run, and every time you make a change to your code, autotest will run all of your tests again. This can take a lot of time! Autotest is good, but it is pretty dumb!
Lazy Loading
But wait! There is a tool to solve the problem that Autotest has, its called Cucover (http://bit.ly/dg3Je3).
Cucover is a think wrapper for Cucumber, that well makes it lazy. It works by using RCov to detect which parts of the code have changed, Cucover will then only rerun the scenarios that are affected by the code changes.
You can install Cucover by using this command:
sudo gem install mattwynne-cucover --source http://gems.github.com
Lazy loading can also be used for loading web services, this is so time is not wasted loading up services that a feature does not require.
Handy Cucumber features
Another way to speed up your tests during development is to use the handy features built into cucumber.
Tagging: $ cucumber –tags @media
Tagging is used to run certain scenarios.
Filenames: $ cucumber visitor_*
You can name files with a naming convention and only run those files using a regex.
Subsystems: $ cucumber admin_suite/*
Put each of your different features into folders and get cucumber to run all features in that folder.
Continuous Integration
Another way to rocket fuel your cucumbers is to use Continuous Integration. There are many continuous integration systems out there, here are some:
- Hudson
- Team City
- Integrity
- Bamboo
- Run Code Run
- CI Joe
- CruiseControl.rb
- Cerberus
Spork
Spork (http://bit.ly/1zUlXg) is Tim Harper’s implementation of test server (similar to the script/spec_server provided by rspec-rails), except rather than using the Rails constant unloading to reload your files, it forks a copy of the server each time you run your tests. The result? Spork runs more solid: it doesn’t get corrupted over time, and it properly handles modules and any voo-doo meta programming you may have put in your app.
Divide and Conquer
One way to divide your application is to use Rails Engines. Rails Engines are mini Rails applications that are run from the vendor/plugins directory, you can write separate cucumber tests and run these against your rails engine.
Another way is to use Service Oriented Architecture (http://bit.ly/PTT-SOA) by using this design principle, you can separate your cucumber tests for each service of your application.
Axing Treetop
Treetop is the parser that Cucumber uses to read the plain-text descriptions, but it is well….a bit slow.
This is where Ragel comes in.
Here are some statistics comparing Regal and Treetop:
To parse 1257 Scenarios when using tags:
Treetop took 60 seconds
Regal took 5 seconds.
Regal is currently being integrated with Cucumber, but you can get it now by running:
sudo gem install gherkin
Looking to the Cloud
Amazon EC2 is a good way to use cloud computing to run your cucumber tests.
To run 1257 steps for a http://www.songkick.com/ on one build server took 4 hours.
On 19 Amazon S3 instances it took 13minutes and 40 seconds.
The key is to have an image created so that you can just startup as many instances as you need, so that they are all setup and ready to receive tests.
To use multiple servers to run your tests you can use a tool called Testjour.
Testjour can distribute your tests over multiple servers or if you like, multiple cores of your build machine.
To run 254 Features, and 1257 steps, took 4 hours on one build server.
When using 3 servers and Testjour, it took 2.
The only problem with Testjour is that its poorly documented, it says it can do things that it can’t.
It is also very fragile!
This is how Testjour works, it first uploads your code to all of the slave servers.
Next it uploads the tests to a Redis Work Queue, this then sends the tests to which ever slave servers are free to be tested against the application.
Joseph’s company spends around $80 a day on Amazon EC2 instances, but this saves dividends
Useful tip
To see how long your cucumber tests take to run, you can run the following command:
cucumber -format usage features/
