Seeing Geronimo 2.0 pass the Sun TCK tests brought a smile to the faces of the team here -- our guys helped the Geronimo team achieve shorter test cycles by distributing tests with AnthillPro. So what’s the trick to distributing tests?

Automate the Deployment

The first step to distributing functional tests is to make your deployment process push button and repeatable across your test farm. This is pretty much AnthillPro’s bread and butter, so we’ll leave that for another post.

If you want to distribute unit tests, you can always just move the test binaries and the compiled code out to build farm machines using the same mechanism.

Seperate Test Suites

Next ensure that you have stand-alone tests suites that can be run without any other test suite being run first. With functional tests, this can often require some changes to how you work now. Some simple set-up that gets the system to a known base state that is shared across the test suites is one approach. But each test suite could be responsible for taking your application from freshly installed, to a configuration for that suite, then through tests against that configuration.

Parallelize

If after that process, you have two or three test suites life is easy. Create a job for each one and set them in a parallel in your workflow. The downside is that you’ll only be able to distribute across two or three machines and the process will take as long as the longest suite. That isn’t ideal if you have many hours of functional tests.

If you have 20 or 100 test suites we can take better advantage of a test farm with Anthill’s load balancing, and job distribution. That said, configuring 20 or 100 test jobs is not fun. Instead, we want to use the Iterate option in the workflow definition. What Iterate allows us to do is run a single job many times with some rules for how many machines to use and how many iterations a single machine may run simultaneously.

So we create a single test job. That job will call your Run Test Suite script and will pass in the iteration number. Your test script (generally) will provide the mapping that says if the iteration number is 1, run test suite A. If it is iteration 2, run test suite B. The workflow instructs that job to run iteration for every suite that we have and limits each machine run one suite at a time. For stress testing, we might run many suites on each client machine.

When one suite finishes, Anthill provides that machine with the next test. This supports high utilization, and shorter test cycles without cumbersome configuration. Congratulations to the Geronimo team and thanks for the valuable insights that have made job iteration as useful as it is today.