Testing Koans

Koans
Koans have been proposed as an effective way to learn a new programming language. But what exactly are Koans? According to Wikipedia a Koan (where the o has a macron, a straight bar placed above it - which my text editor refuses to produce) is a "case, story, dialogue, question or statement in the history and lore of Zen Buddhism". Huh? Reading the Wikipedia article did not help me at all. All I understand is that a Koan is something the Buddhist monks would work with, a mystical sentence or maybe a kind of poem, which does not make any sense, but somehow helps them on their way to enlightenment. It seems the metaphor has been transferred from Buddhism to software, e.g. Hacker Koans and Koans are related to the TAO of Programming. (Again no idea what TAO is supposed to mean here. This is like a recursive definition.)

Ruby Koans
As far as I know, the first Programming Koans were available in Ruby, created by the late Jim Weirich, a popular Ruby hacker. Ruby Koans consists of several little exercises, starting with basic things and building on each other to move to more advanced topics in the end. The goal is to learn Ruby, to walk the "path to enlightenment" as Jim put it. He also wanted to teach the Ruby culture. The Ruby community has a strong focus on testing, which is considered essential to "do great things in the language". In fact the exercises are a list of failing test cases, where tiny pieces of code have to be filled in to make them pass. For example, here is the exercise to learn accessing array elements,

def test_accessing_array_elements
  array = [:peanut, :butter, :and, :jelly]

  assert_equal __(:peanut), array[0]
  assert_equal __(:peanut), array.first
  assert_equal __(:jelly), array[3]
  assert_equal __(:jelly), array.last
  assert_equal __(:jelly), array[-1]
  assert_equal __(:butter), array[-3]
end

The double underscore marks the place where the code has to be changed to make it work and pass the test. These tests are very simple and there is not much explanation. Maybe this is the connection to the Zen Koans: The Language Koans are a list of exercises to work through, to master the language (i.e. reach enlightenment). Each one is very small (i.e. a sentence) but does not make much sense on its own. The exercises are sorted by increasing difficulty (i.e. the path to walk). Following Jim's example, Koans are usually based on unit tests which you make succeed. Language Koans are available for many programming languages, see a list of Koans by Laura Diane Hamilton.

Testing Koans
I took the idea for Testing Koans from Carlos Blé's training JavaScript for Testers. He created some Koans for JavaScript with inverted work-flow. The code was already in place, but the assertions were missing. That was reasonable as the training was created for tester.

xUnit Koans
Earlier this year I ran an introductory unit testing workshop for the local PHP community. I expected a junior audience and aimed for the most basic exercise for xUnit assertions and life cycle methods. I wanted the participants to focus on PHPUnit alone. I created some sample code, together with unit tests, and then deleted the assertion statements. The first test looked similar to the following Java code:

import org.junit.Test;

public class Session1_GreeterTest {

  @Test
  public void shouldReturnHelloName() {
    Greeter greeter = new Greeter();
    // TODO check that "Hello Peter" is greeter.greet("Peter")
  }

  @Test
  public void shouldReturnHelloForNull() {
    Greeter greeter = new Greeter();
    // TODO check that "Hello" is greeter.greet(null)
  }

  // more tests skipped...

}

The participants went through the tests one by one, adding assertions or fixing incomplete statements, making the tests pass. While this looked like a very basic and short exercise, developers unfamiliar to PHPUnit (and xUnit in general) needed several hours to complete all my PHPUnit Testing Koans.

Due to the uniform nature of all xUnit ports, the style and structure of the exercise can be used for other programming languages. I ported the exercise to Java using JUnit, creating Java/JUnit Koans. Both Koans cover the basic functionality of PHPUnit and JUnit, e.g. assertions, testing for exceptions and before- and after-methods. More advanced features could be added. I will port the Koans to C#/NUnit and Ruby/minitest as soon as I will need them.

Conclusion
Koans are a great way to partition the process of knowledge acquisition into a series of little exercises. They verify themselves, giving you fast feedback but you can still learn at your own pace. Language Koans are established and available for many languages. These can be extended to any library or public API you want to master. Testing Koans work similar, just inverted. They are available for PHPUnit and JUnit for now. I would love to see more ports and also Koans for different testing styles, e.g. RSpec or Jasmine Testing Koans.

Assert or MatcherAssert

I regularly meet online with friends and practice remote pair programming. Last time, while working with my friend Thomas, one of us mistyped the static import for assertThat and we ended with an import of org.hamcrest.MatcherAssert instead of org.junit.Assert. I had never seen MatcherAssert before and Thomas asked me if I knew what the actual difference between them would be. I did not know but we are going to find out right now.

A little bit of history
The Hamcrest Matchers are part of JUnit 4 since version 4.4. JUnit up to version 4.10 shipped with Hamcrest 1.1 and on release 4.11 it switched to Hamcrest 1.3. Looking at the history of MatcherAssert it seems that it had been in the core already but had been moved out again before version 1.1 was released.

• May 22, 2007 - Moved assertThat() into hamcrest-core.
• May 23, 2007 - Moved assertThat() back into hamcrest-integration (really this time).
• July 19, 2007 - JUnit 4.4 shipped including Hamcrest 1.1
• November 25, 2008 - Moved MatcherAssert to core.
• July 9, 2012 - Hamcrest version 1.3 released
• November 14, 2012 - shipped JUnit 4.11

Show me the code!
Now let us compare the actual source code. JUnit's Assert.assertThat looks like

public static <T> void assertThat(T actual, Matcher<T> matcher) {
   assertThat("", actual, matcher);
}

public static <T> void assertThat(String reason, T actual,
                                    Matcher<T> matcher) {
   if (!matcher.matches(actual)) {
      Description description = new StringDescription();
      description.appendText(reason);
      description.appendText("\nExpected: ");
      description.appendDescriptionOf(matcher);
      description.appendText("\n     got: ");
      description.appendValue(actual);
      description.appendText("\n");

      throw new AssertionError(description.toString());
   }
}

whereas Hamcrest's MatcherAssert code is

public static <T> void assertThat(T actual, Matcher<? super T> matcher) {
   assertThat("", actual, matcher);
}

public static <T> void assertThat(String reason, T actual,
                                    Matcher<? super T> matcher) {
   if (!matcher.matches(actual)) {
      Description description = new StringDescription();
      description.appendText(reason)
                 .appendText("\nExpected: ")
                 .appendDescriptionOf(matcher)
                 .appendText("\n     but: ");
      matcher.describeMismatch(actual, description);

      throw new AssertionError(description.toString());
   }
}

public static void assertThat(String reason, boolean assertion) {
   if (!assertion) {
      throw new AssertionError(reason);
   }
}

assertThat(String, boolean)
The most obvious difference is that Hamcrest defines an additional assertThat method which accepts a boolean expression. The method is identical to Assert.assertTrue(String, boolean). No big deal.

<? super T>
The signatures of both classes are almost identical. Almost because Hamcrest has a variation of the generic type T. JUnit accepts matchers that have the same type as the actual value in the argument list T actual, Matcher<T> matcher, while Hamcrest also accepts them for super types of T in T actual, Matcher<? super T> matcher. Allowing parent types of T makes sense because what matches the parents of T will always match T too. Here is an example of a Matcher<Number> matching against an Integer which extends Number.

import org.hamcrest.CustomMatcher;
...

@Test
public void shouldAllowSuperTypeMatcherInAssert() {
   Integer actual = 3;
   Matcher<Number> matcher = new CustomMatcher<Number>("anything") {
      @Override
      public boolean matches(Object anything) {
         return true;
      }
   };
   MatcherAssert.assertThat(actual, matcher); // (1)
   Assert.assertThat(actual, matcher); // (2)
}

Line (1) binds <T> to <Integer> because <? super Integer> allows Number as type of the given matcher. On the other hand, line (2) compiles as well, setting <T> directly to <Number>. So the two signatures are equivalent.

Description of the Mismatch
The third difference is the way the two assertions describe the mismatch. While JUnit just appends the actual value with description.appendValue(actual), Hamcrest asks the matcher to describe its mismatch by matcher.describeMismatch(actual, description), which might give more information in case of failure. So let us dig into some more code.

@Test
public void shouldDisplaySimilarMessageForIsMatcher() {
   int actual = 1;
   Matcher<Integer> matcher = is(2);
   assertJUnitMessage(actual, matcher,
      "Message\nExpected: is <2>\n  got: <1>\n");
   assertHamcrMessage(actual, matcher,
      "Message\nExpected: is <2>\n  but: was <1>");
}

As the BaseMatcher's describeMismatch just returns "was " + actual there is no real difference in the output of both assertions. What about other matchers? Let us look for implementations of describeMismatch that do more than just return the actual value. The only matcher I found was org.hamcrest.TypeSafeMatcher, which has several subclasses in the Hamcrest library, e.g. HasProperty, IsEmptyIterable and IsCloseTo.

import org.hamcrest.beans.HasProperty;
import org.hamcrest.collection.IsEmptyIterable;
import org.hamcrest.number.IsCloseTo;
...

@Test
public void shouldDisplaySimilarMessageForHasPropertyMatcher() {
   Object actual = "abc";
   Matcher<Object> matcher = HasProperty.hasProperty("prop");

   assertJUnitMessage(actual, matcher,
      "Message\nExpected: hasProperty(\"prop\")\n  got: \"abc\"\n");
   assertHamcrMessage(actual, matcher,
      "Message\nExpected: hasProperty(\"prop\")\n  but: no \"prop\" in \"abc\"");
}

@Test
public void shouldDisplaySimilarMessageForIsEmptyIterableMatcher() {
   Iterable<String> actual = Collections.<String> singletonList("a");
   Matcher<Iterable<String>> matcher = IsEmptyIterable.<String> emptyIterable();

   assertJUnitMessage(actual, matcher,
      "Message\nExpected: an empty iterable\n  got: <[a]>\n");
   assertHamcrMessage(actual, matcher,
      "Message\nExpected: an empty iterable\n  but: [\"a\"]");
}

@Test
public void shouldDisplaySimilarMessageForIsCloseToMatcher() {
   double actual = 2.0;
   Matcher<Double> matcher = IsCloseTo.closeTo(1, 0.1);

   assertJUnitMessage(actual, matcher,
      "Message\nExpected: a numeric value within <0.1> of <1.0>\n  got: <2.0>\n");
   assertHamcrMessage(actual, matcher,
      "Message\nExpected: a numeric value within <0.1> of <1.0>\n  but: <2.0> differed by <0.9>");
}

Hamcrest creates slightly more detailed error messages, but only for these three cases.

Conclusion
org.hamcrest.MatcherAssert is not a replacement for org.junit.Assert because it does not come with all the assertions which we are used to. But MatcherAssert contains a slightly different assertThat. Using that method could potentially give better error messages because the matcher is called to describe the mismatch. When using a complex, custom matcher this could improve the error messages and speed up error diagnosis. Currently only three matchers implement their own descriptions. These come with the Hamcrest library which is not included with JUnit right now.

T-shirt Update

I love t-shirts and keep wearing them regardless of my employer's dress code. I had only two complaints about them in the last year, so things worked out quite well ;-) The last time I wrote about Code Cop t-shirts was more than two years ago, so it is definitely time for an update.

Don't Repeat Yourself
The Don't Repeat Yourself or DRY principle states that every piece of knowledge must have a single, unambiguous, authoritative representation within a system. In simple terms this means that there should be no duplicates or repetition. The dark blue DRY shirt reminds me to write things once and only once. It is also my best selling shirt ever, as a friend bought one after he was exposed to horrible duplicated code at the client's site. Unfortunately I did not get rich as I bought him a beer in return.

Keep the bar green to keep the code clean
Keeping the bar green is the primary motto of JUnit, the popular unit-testing framework for Java. This grey shirt displays a simplified image of the Eclipse JUnit runner together with a green bar. I had problems finding the right dimensions for the characters and the green check marks and threw away at least three versions until I got it right. One day I wore it in the office and one of my colleagues, a project manager, asked me "green is good, right?" I just love them.

Red-Green-Refactor
Red-Green-Refactor is the cycle of Test Driven Development. You write a failing test which results in a red bar. Then you write some code until the bar is green. Finally you clean up the code you just wrote, e.g. improve the names or remove duplication. (Remember DRY?) For the white shirt I used typefaces according to these phases: Red is scary, I do not like it, so I used a Halloween typeface for it. Then I go for a green bar as fast as possible, so I used a dynamic typeface for that. My result after refactoring is sorted and tidy symbolized by a regular typeface.

(Buy Code Cop shirts)

Eclipse Plugin Unit Testing

I will describe the set-up of a unit testing infrastructure for a big RCP application consisting of several plugins. In the first part I will write about how to run JUnit tests inside Eclipse.

Test Plugins
First we need a place to put the tests. We do not want to put the tests in the same project as the production code because we do not want to ship the tests. So for every plugin which we want to write tests for, we create a separate plugin with the same name suffixed with ".test". For example let's assume we want to write a JUnit test for a class in the company.product.general plugin. First we look for a plugin named company.product.general.test. Does it exist? Maybe we never used it and need to check it out from version control. Still nothing? OK, so we need to create it.

For a new test plugin we create a fragment project. We do that by selecting File > New > Project > Plug-in Development > Fragment Project. A fragment allows us to add code to another plug-in, known as the host. The fragment is merged with its host and has full access to all the packages declared in its host, not only the exported ones. Among other ways to set up tests for plugins, fragments generally are the best solution. So our unit tests have access to all classes of the host plugin. The fragment inherits all dependencies from the host. The only explicit required dependency of the test plugin is org.junit4. (Remember that various files in the build system might need to be updated for new (test) plugins to be part of the build. These might include some kind of loading rules, certain Ant build files or test features.)

The above set-up is enough to run simple JUnit tests which do not use any feature of Eclipse's OSGI container. To enable PDE JUnit tests the host plugin needs to define Eclipse-ExtensibleAPI: true in its MANIFEST.MF file. This allows the fragment to access classes from the host plugin and tells the PDE tooling to add the fragment to the classpath of the host plugin.

Test Packages
We use the plugin's name as base package of the plugin's source code to avoid name collisions. For example all packages of the plugin company.product.general are sub packages of company.product.general. This is not the case for fragments, as fragments are merged with the host. All packages of the company.product.general.test fragment are still sub packages of company.product.general.

The package with the same name as the test fragment is used for common test set-up or custom assertion logic that might be used for more than one test. For example the package company.product.general.test might contain a class called UsLocale, which sets the current locale to US, as needed to test some business logic. Common test code can be reused by other test plugins if it is exported from the fragment. This works the same way as for normal plugins.

Warning about "Add Required Plugins"
Sometimes it is necessary to add new plugins to a launch configuration in order for the application to start. When you do that by using the "Add Required Plugins" button in the launch configuration, make sure that all the test plugins are not selected afterwards. We had some issues when test-only functionality somehow "leaked" into production.

Simple JUnit Tests
When the test plugin is ready, we can start writing simple JUnit tests. They will not necessarily be "simple", they are just called like that to distinguish them from PDE JUnit tests which have a more complex set-up. Even when developing RCP applications, these simple JUnit tests have their use. We can use them to test any class that is not dependent on the Eclipse Platform API. They are particularly useful for testing business logic and model classes.

JUnit tests go into the same Java package as the class under test. So the test plugin has the same packages as the host plugin. Test classes are named like the class under test suffixed with "Test". For example let's assume we want to test the class company.product.general.api.CatchHandlers in the plugin company.product.general. The related test is company.product.general.api.CatchHandlersTest in the company.product.general.test plugin.

Mocking
Unit testing is supposed to test classes or methods in isolation but classes usually depend on other classes. For more complex scenarios we want to use different objects than the live ones to get rid of these dependencies. These objects are provided to the class under test. By using them we can verify logic independently of the depended-on objects. Such objects are called Test Doubles.

Dummy objects, fakes and stubs are created easily with custom classes. But true mocks need some more logic to verify the order of method invocations (which needs some kind of state machine). Fortunately there are several frameworks that bring mocks to Java, for example EasyMock. To use it in our RCP application we need to create on OSGI bundle (i.e. an Eclipse plugin) from its jars. Let the plugin company.product.test.mock contain the mocking framework's jars. To use its classes in tests add it to the dependencies of the test fragment.

PDE JUnit Tests
PDE or Plugin JUnit tests are executed by a special test runner that launches another Eclipse instance and executes the test methods within that workbench. So PDE JUnit tests can be used for classes that are dependent on Eclipse Platform API. Parts of the Eclipse platform are available outside a running workbench as well, so a normal JUnit test might still be an option. Additionally some other parts, for example the extension registry can be mocked. But there are cases where a running workbench is necessary. For example you may assume a particular behaviour of Plugin.getStateLocation(), which would not be available if you were just using an ordinary JUnit test. Besides that the tests themselves look exactly like any other test.

When automating the build these PDE tests should be distinguished from basic ones. So we follow a different naming convention and call them *PlatformTests. Also note that the usual class extension mechanism might not work inside the workbench. You need to introduce interfaces for the types you want to mock.

JUnit Tutorials

I prepared a short list of tips and links for some colleagues to get started with JUnit.

• A good start for an absolute beginner is the JUnit Cookbook.
• JUnit is integrated in Eclipse. Lars Vogel's JUnit Tutorial shows how to write and execute tests using Eclipse. Talking about Eclipse, I always use the keyboard short-cut ALT-SHIFT-X and then key T to launch the test open in the editor window. After that I use the keyboard short-cut CTRL-F11, which runs the last launched test or application, to re-run my JUnit test until it succeeds.
• JUnit comes pre-packaged with Hamcrest, a framework for writing matcher objects. These matchers improve the readability of tests and provide better failure messages. Consider writing a custom Matcher if you need to compare large objects with one another based on complex state.
• A mistake that I see quite often is the handling of expected exceptions in tests. Szczepan Faber has written down five rules how to do that and to avoid any problems.
• To explore more advanced topics of JUnit see my presentation on Practical Unit Testing (June 2009).
• Also make sure that your tests are deterministic. Non-deterministic tests are a serious threat to the discipline of unit testing.

All these links focus on using the JUnit technology, but do not explain how to write good tests. If you have some spare time and would like to know more about JUnit and unit testing in general I recommend the book Pragmatic Unit Testing in Java with JUnit.

Behaviour - the "New" Way
By the rise of BDD the common understanding of unit tests has changed. Even if you do not do TDD or BDD these things apply to any unit test.

• They are now called Micro tests to distinguish them from traditional unit tests. Tests with a larger scope, e.g. integration or end-to-end tests, are no unit tests even if they make use of JUnit.
• Further the names of test methods should be full sentences focusing on behaviour. JUnit 4.x removes the need to prefix test methods with test, and usually the sentence of the expected behaviour starts with the word should.
• The test methods should be created using the Arrange-Act-Assert or even better the Given-When-Then pattern.

Recommended Watching
Finally I highly recommend the recording of The Deep Synergy Between Testability and Good Design by Michael Feathers. It's an excellent talk examining relationship among test-pains, code smells and design principles. Go, watch it!

Custom PMD Rules

Last month I submitted my fourth article from the 'Code Cop' series. After Daily Build, Daily Code Analysis and Automated Testing I wrote about checking architecture rules. One approach described there used PMD. In case you don't know what PMD is, it's a static code analysis tool for Java. It was first developed in 2002 and is updated regularly. It checks the abstract syntax tree (AST) of Java code. (PMD is a great tool. You should definitely use it!)

PMD vs. AST (I just love TLAs)
PMD comes with many predefined rules. Some of them are real life-savers, e.g. EmptyCatchBlock. It's possible to define your own rules. The easiest way to do this is to use XPath expressions to match patterns in the AST. For example, the following trivial class

package org.codecop.myapp.db;

import java.sql.Connection;

public class DBSearch {
   ...
}

is represented inside PMD by the following AST

+ PackageDeclaration
|    + Name:org.codecop.myapp.db
+ ImportDeclaration
|    + Name:java.sql.Connection
+ TypeDeclaration
     + ClassOrInterfaceDeclaration:DBSearch
          + ClassOrInterfaceBody
               + ...

An XPath expression to find imports from the java.sql package looks like

//ImportDeclaration[starts-with(Name/@Image, 'java.sql.')]

It is quite simple (at least when you are familiar with XPath ;-) For further details read the PMD XPath Rule Tutorial.

I've always been enthusiastic about static code analysis and PMD in particular and have been using it successfully since 2004. (For example see my very first presentation on static code analysis with PMD.) I love it; It's a great tool. I have created several custom rules to enforce consistent coding conventions and to prevent bad coding practices. Today I am going to share some of these rules with you in this post.

First Rule
One common bug is public boolean equals(MyClass o) instead of public boolean equals(Object o) which shows that the developer is trying (and failing) to override the equals(Object) method of Object in MyClass. It will work correctly when invoked directly with a MyClass instance but will fail otherwise, e.g. when used inside collections. Such suspiciously close (but still different) equals methods are matched by

//ClassDeclaration//MethodDeclarator
  [@Image = 'equals']
  [count(FormalParameters/*)=1]
  [not ( FormalParameters//Type/Name[
           @Image='Object' or @Image='java.lang.Object' ] ) ]

This rule explained in plain English matches all the method declarations that are named equals, and have one parameter which type is neither Object nor java.lang.Object. (Object is mentioned twice because PMD analyses the source and therefore can't know about simple and full qualified class names.) Later, Tom included this SuspiciousEqualsMethodName rule into PMD's default set of rules.

Copy and Paste
The easiest way to define your own rule is to take an existing one and tweak it. Like SuspiciousEqualsMethodName was derived from SuspiciousHashcodeMethodName, the next JumbledIterator is quite similar to JumbledIncrementer. (JumbledIterator was created by Richard Beitelmair, one of my colleagues who appointed me "Code Cop" and presented me with my first Code Cop T-shirt.) So what's wrong with the following line?

for (Iterator it1 = iterator(); it2.hasNext(); ) { ... }

Most likely it2 should be it1, shouldn't it. Richard created the following rule to pick up on these kinds of errors:

//ForStatement[
  ( ForInit//ClassOrInterfaceType/@Image='Iterator' or
    ForInit//ClassOrInterfaceType/@Image='Enumeration'
  ) and
  ( ends-with(Expression//Name/@Image, '.hasNext') or
    ends-with(Expression//Name/@Image, '.hasMoreElements')
  ) and not (
    starts-with(Expression//Name/@Image,
      concat(ForInit//VariableDeclaratorId/@Image, '.'))
  )
]

A Real Environment
After the last two sections we are warmed up and finally ready for some real stuff. On several occasions I have met developers who wondered why their code Long.getLong(stringContainingANumber) would not work. Well it worked, but it did not parse the String as they expected. This is because the Long.getLong() is a shortcut to access System.getProperty(). What they really wanted was Long.parseLong(). Here is the UnintendedEnvUsage rule:

//PrimaryExpression/PrimaryPrefix/Name[
  @Image='Boolean.getBoolean' or
  @Image='Integer.getInteger' or
  @Image='Long.getLong'
]

Care for Your Tests
PMD provides several rules to check JUnit tests. One of my favourite rules is JUnitTestsShouldIncludeAssert which avoids (the common) tests that do not assert anything. (Such tests just make sure that no Exception is thrown during their execution. This is fair enough but why bother to write them and not add some assert statements to make sure the code is behaving correctly.) Unfortunately, one "quick fix" for that problem is to add assertTrue(true). Rule UnnecessaryBooleanAssertion will protect your tests from such abominations.

A mistake I keep finding in JUnit 3.x tests is not calling super in test fixtures. The framework methods setUp() and tearDown() of class TestCase must always call super.setUp() and super.tearDown(). This is similar to constructor chaining to enable the proper preparation and cleaning up of resources. Whereas Findbugs defines a rule for that, PMD does not. So here is a simple XPath for JunitSetupDoesNotCallSuper rule:

//MethodDeclarator[
  ( @Image='setUp' and count(FormalParameters/*)=0 and
    count(../Block//PrimaryPrefix[@Image='setUp'])=0
  ) or
  ( @Image='tearDown' and count(FormalParameters/*)=0 and
    count(../Block//PrimaryPrefix[@Image='tearDown'])=0
  )
]

Obviously this expression catches more than is necessary: If you have a setUp() method outside of test cases this will also be flagged. Also it does not check the order of invocations, i.e. super.setUp() must be the first and super.tearDown() the last statement. For Spring's AbstractSingleSpringContextTests the methods onSetUp() and onTearDown() would have to be checked instead. So it's far from perfect, but it has still found a lot of bugs for me.

That's all for now. There are more rules that I could share with you, but this blog entry is already far too long. I've set up a repository pmd-rules containing the source code of the rules described here (and many more).

Running JUnit in Parallel

Back in 2008 I had to speed up our daily build. (I should have posted about it since long, but I just didn't make it. Recently when I saw a related post on a similar topic my bad conscience overwhelmed me.) The first thing was to get a faster machine, something with four 3 GHz cores. It worked excellent! All file based operations like compile performed 3 times faster just out of the box, thanks to the included RAID 0+1 disk array. As our automated tests took half of the total build time, I dealt with them first: I applied the usual optimisations as told in my talk about practical JUnit testing, tuning section. So I managed to halve JUnit execution time.

Good, but still not fast enough. The problem was how to utilise all the shiny new cores during one build to speed it up as much as possible. So test execution needed to run in parallel. Some commercial build servers promised to be able to spread build targets over several agents. Unfortunately I had no opportunity to check them out, they cost quite beyond my budget. The only free distributed JUnit runner I found was using ComputeFarm JINI in a research project which did not look mature enough for production usage. Worth mentioning is GridGain’s JunitSuiteAdapter. It's able to distribute JUnit tests across a cluster of nodes. GridGain is a free cloud implementation, it's really hot stuff. But it's not a build solution, so integrating it into the existing build would have been difficult.

As I did not find anything useful had to come up with a minimalist home grown solution. I started with a plain JUnit target junitSequential which ran all tests in sequence:

<target name="junitSequential">

  <junit fork="yes" failureproperty="failed"
         haltonfailure="false" forkmode="perBatch">

    <classpath>
      <fileset dir="${lib.dir}" includes="*.jar" />
      <pathelement location="${classes.dir}" />
    </classpath>

    <batchtest>
      <fileset dir="${classes.dir}"
               includes="**/*Test.class" />
    </batchtest>
  </junit>

  <fail message="JUnit test FAILED" if="failed" />

</target>

I used haltonfailure="false" to execute all tests regardless if some failded or not. Otherwise <batchtest> would stop after the first broken test. With failureproperty="failed" and <fail if="failed" /> the build still failed if necessary. There is nothing special here.

Ant is able to run tasks in parallel using the <parallel> tag. (See my related post about forking several Ant calls in parallel.) A parallel running target would look like

<target name="junitParallelIdea">
  <parallel>
    <antcall target="testSomeJUnit" />
    <antcall target="testOtherJUnit" />
  </parallel>
</target>

Good, but how to split the set of tests into Some and Other? My first idea was to separate them by their names, i.e. by the first letter of the test's class name, using the inclusion pattern **/${junit.letter}*Test.class in the <batchtest>'s fileset. So I got 26 groups of tests running in parallel.

<target name="junitParallelNamedGroups">
  <parallel>

    <antcall target="-junitForLetter">
      <param name="junit.letter" value="A" />
    </antcall>
    <antcall target="-junitForLetter">
      <param name="junit.letter" value="B" />
    </antcall>
    <antcall target="-junitForLetter">
      <param name="junit.letter" value="C" />
    </antcall>
    <!-- continue with D until Z -->

  </parallel>
</target>

<target name="-junitForLetter">

  <junit fork="yes" forkmode="perBatch">

    <!-- classpath as above -->

    <batchtest>
      <fileset dir="${classes.dir}"
               includes="**/${junit.letter}*Test.class" />
    </batchtest>
  </junit>

</target>

forkmode="perBatch" created a new JVM for each group. Without forking each test class would get it's own class loader, filling up the perm space. Setting reloading="false" made things even worse. All those singletons started clashing even without considering race conditions. So I took the overhead of creating additional Java processes.

Unfortunately the grouping by letter approach had some problems. First the number of threads needed to be specified with <parallel>'s threadsperprocessor or threadcount attribute, else there would be 26 parallel processes competing for four cores. My experiments showed that two threads per processor performed best for the given set of JUnit tests. (Those JUnit tests were not "strictly unit", some tests called the database or web services, freeing the CPU during blocking. For tests with very little IO it might have looked different.)

Also my haltonfailure approach did not work because <antcall> does not return any properties set inside the called -junitForLetter target. There was no Ant command that supported that. But AntCallBack of the Antelope Ant extensions was able to do the trick: After registering the custom task with name="antcallback" I replaced the plain <antcall>s with <antcallback target="..." return="failed">.

Separating JUnit test cases by their names produced unbalanced and therefore unpredictable results regarding overall execution time. Depending on naming conventions some groups would run much longer than others. Ant's Custom Selectors are a much better way to split a fileset into a given number of parts producing a few balanced filesets with roughly the same number of test classes.

import java.io.File;

import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.types.Parameter;
import org.apache.tools.ant.types.selectors.BaseExtendSelector;

public class DividingSelector extends BaseExtendSelector {

  private int counter;
  /** Number of total parts to split. */
  private int divisor;
  /** Current part to accept. */
  private int part;

  public void setParameters(Parameter[] pParameters) {
    super.setParameters(pParameters);
    for (int j = 0; j < pParameters.length; j++) {
      Parameter p = pParameters[j];
      if ("divisor".equalsIgnoreCase(p.getName())) {
        divisor = Integer.parseInt(p.getValue());
      }
      else if ("part".equalsIgnoreCase(p.getName())) {
        part = Integer.parseInt(p.getValue());
      }
      else {
        throw new BuildException("unknown " + p.getName());
      }
    }
  }

  public void verifySettings() {
    super.verifySettings();
    if (divisor <= 0 || part <= 0) {
      throw new BuildException("part or divisor not set");
    }
    if (part > divisor) {
      throw new BuildException("part must be <= divisor");
    }
  }

  public boolean isSelected(File dir, String name, File path) {
    counter = counter % divisor + 1;
    return counter == part;
  }
}

One of the four available cores was used for static code analysis, which was very CPU intensive and one was used for integration testing. The remaining two cores were dedicated to unit tests. Using 4 balanced groups of tests executing in parallel, the time spent for JUnit tests was halved again: Yippee

<target name="junitParallel4Groups">
  <parallel threadcount="4">
    <antcallback target="-junitForDivided" return="failed">
      <param name="junit.division.total" value="4" />
      <param name="junit.division.num" value="1" />
    </antcallback>
    <antcallback target="-junitForDivided" return="failed">
      <param name="junit.division.total" value="4" />
      <param name="junit.division.num" value="2" />
    </antcallback>
    <antcallback target="-junitForDivided" return="failed">
      <param name="junit.division.total" value="4" />
      <param name="junit.division.num" value="3" />
    </antcallback>
    <antcallback target="-junitForDivided" return="failed">
      <param name="junit.division.total" value="4" />
      <param name="junit.division.num" value="4" />
    </antcallback>
  </parallel>
  <fail message="JUnit test FAILED" if="failed" />
</target>

<target name="-junitForDivided">

  <junit fork="true" failureproperty="failed"
         haltonfailure="false" forkmode="perBatch">

    <!-- classpath as above -->

    <batchtest>
      <fileset dir="${classes.dir}">
        <include name="**/*Test.class" />
        <custom classname="DividingSelector" classpath="classes">
          <param name="divisor" value="${junit.division.total}" />
          <param name="part" value="${junit.division.num}" />
        </custom>
      </fileset>
    </batchtest>

  </junit>

</target>

(Download source code of DividingSelector.)

Epiloge
Using this approach I kept the option to execute the tests one after another with num=1 of total=1 providing an easy way to switch between normal and parallel execution. This was useful when debugging the build script...

<target name="junitSequential">
  <antcallback target="-junitForDivided" return="failed">
    <param name="junit.division.total" value="1" />
    <param name="junit.division.num" value="1" />
  </antcallback>
  <fail message="JUnit test FAILED" if="failed" />
</target>

Practical Unit Testing

In the beginning of June I gave a presentation about practical unit testing with JUnit at the Java Student User Group in Vienna. JSUG is a small group of dedicated students that formed last year. The scope was practical as well as pragmatic - in fact just bits of information I considered useful for the daily development. It was good to see young developers that are eager to write tests. I hope when they are thrown into legacy code later in their career that their principles will not just crumble.

Download the Practical Unit Testing presentation slides.

Resources used in the slides:
This section contains links to topics I talked about. I give a list of all my sources. Hopefully Google does not punish me for creating this link-farm like page ;-)

JUnit basics used sources from Unit testing (Wikipedia), JUnit.org & JUnit, Early look at JUnit 4, Design to Unit Test, Test-driven design, Part 1 & Part 2; Checkstyle, FindBugs and PMD.

The mocking chapter used sources from jMock, EasyMock (Easier testing with EasyMock), Initializing bean using EasyMock, Mocking & Spring, Oh no, we're testing the Mock! and the Law Of Demeter.

Singletons are a pain for testing: Patterns I Hate, Singletons Are Evil and Why Singletons are Evil, Refactor singleton, Test flexibly with AspectJ and mock objects with AspectJ and the Google Singleton Detector.

There are some tools to help testing J2EE apps: HtmlUnit/HttpUnit, HttpClient, Testing Servlets and ServletUnit, Jetty, Cactus, Simple-JNDI, MockEJB and ActiveMQ.

Tuning the tests means often tuning the database: DbUnit, H2 and HSQLDB.

The Code Coverage chapter talks about EMMA (EclEmma), Cobertura, Agitar but Don't be fooled by the coverage report, Crap4j, Testability Explorer.

A new trend is testing with scripts: Unit test your Java code faster with Groovy or Using JRuby for Java testing, RSpec, JRuby, JtestR and ScalaCheck/Specs.

Finally some cool tools for testing and the build are JUnitPerf, SWTBot, XmlUnit, Distributed JUnit and GridGain.