Log in

Converting OpenGamma From JUnit to TestNG

4 April 2011 By

I've known about TestNG personally for some time, and use it in JSR-310/ThreeTen. However, recently we at OpenGamma decided to move the platform from JUnit 4 to TestNG 6.0.1, so it seemed like a good opportunity to document the experience.

The OpenGamma application stack is a broad platform for the financial risk analytics industry. In structure terms, OpenGamma contains many of the elements of a typical enterprise system - utility code, persistence in multiple databases, integration with message oriented middleware, a binary protocol (Fudge), authorization, a compute grid and business logic. Testing these different components is obviously a complex affair, with a combination of unit and integration tests being needed.

For example, I've helped develop a number of REST-style data access objects, known as the "masters", which are coded in such a way that they can be used against multiple different databases. Our default database choice is Postgres, however we have also used HSQL, Apache Derby and Vertica, with a simple approach to add code to support others. Ensuring that the code is tested against each database requires the same, or very similar, tests to be run against each.

The testing issues are compounded as we are coming up to the open source release of OpenGamma, where anyone will be able to take a look and see what we have been up to. There won't be an instance of Postgres or Vertica available for downloaders to connect to, thus some tests would fail (HSQL is a local database, so will work fine). However, what this means is that we need to have a greater configurability in the tests, allowing OpenGamma employees to run against Postgres and HSQL, while allowing open source downloaders to only run HSQL.

The choice to use JUnit 4 as the test framework occurred before I joined. In many ways, JUnit is still the "standard" choice which is often taken without considering the alternative. However, OpenGamma has clearly reached the point now where TestNG makes sense.

The main reason to use TestNG is the greater focus on broader testing, rather than just "unit" testing. This support is shown in parameterization (running tests based on external data), groups of tests (allowing a subset of all the tests to be run) and parallelization (to test concurrency and to reduce the length of testing).

TestNG also uses the modern Apache License v2 rather than the outdated and slightly controversial CPL.

Conversion

To switch OpenGamma to TestNG involved converting all the existing test classes and integrating with the broader toolset. For the basic features, the two frameworks, JUnit 4 and TestNG, are relatively similar (JUnit 4 adopted the annotation approach from TestNG). However, there are subtleties in the conversion. This table includes the main annotations that a typical conversion will see.

Use case JUnit 4 TestNG Comment
Method is a test @Test @Test Simply change the import
Class is a test N/A @Test We decided to stick with per method annotations in the end
Test is disabled @Ignore @Test(enabled=false) @Ignore on the class disables all tests in the class, whereas @Test(enabled = false) on the class doesn't override @Test on a method
An exception is expected @Test(expected=IOException.class) @Test(expectedExceptions=IOException.class) Simple spelling change
Test timeout @Test(timeout=20000) @Test(timeOut=20000) Simple spelling change
Run before/after each test method @Before/@After @BeforeMethod/@AfterMethod Clearer annotation name
Run before/after each test method @BeforeClass/@AfterClass @BeforeClass/@AfterClass JUnit requires a static method, TestNG allows either a static or an instance method
Run before/after each entire test suite N/A @BeforeSuite/@AfterSuite  

With @Test, I decided after experimentation to stick with using it at the method level. This is because any @Test(enabled = false) at the class level does not override an @Test at the method level. This makes it quite hard to disable a whole class. The latest Eclipse plugin features an quick fix (Ctrl-F1) that can pull the @Test up to the class level or push it down to each method which is useful if you want to try both approaches.

A key feature of TestNG for the conversion task is an Eclipse-based automatic conversion tool. You simply right-click on part of your codebase and let the tool do the work. During the process of converting OpenGamma, I worked with Cedric Beust to refine the tool. With the recent enhancements it successfully converted the vast majority of the OpenGamma codebase without issues. Obviously no automatic conversion tool is perfect, but this one is now good enough to be genuinely useful.

Parameterization

A key goal of the conversion was better support for parameterization, such as running different kinds of database tests. TestNG supports two types of parameterization - fixed values from the setup and dynamically created ones from code.

File-based parameters can be supplied by name using @Parameters({"fooFile","barFile"}). The values are read from the file that controls the tests, typically testng.xml, but this may also be a YAML format file now.

Code-based parameters are suppiled by data providers. A method, annotated with @DataProvider(name="foo"), is written that returns an Object[][]. This is then used by one or more test methods @Test(dataProvider="foo"). The data provider may be shared between multiple classes by making it a static method and using an additional atribute @Test(dataProvider="foo", dataProviderClass=org.baz.Bar.class).

During the OpenGamma conversion and based on my suggestion, Cedric enhanced the parameterization support. The previous @Factory annotation was a little clumsy requiring extra classes and methods. However, @Factory can now be used on a constructor where it takes a data provider. Bringing these together, here is an example:

public class CachingTest {

  // the data provider, returning an array of object arrays,
  // where the inner array is the arguments to a method call
  @DataProvider(name = "cacheHints")
  public static Object[][] data_cacheHints() {
    return new Object[][] {
        {CacheSelectHint.allPrivate()},
        {CacheSelectHint.allShared()},
        {CacheSelectHint.mixed()},
    };
  }

  // setup by @Factory, new instance created for each factory value
  private final CacheSelectHint _filter;
  
  // setup by @BeforeMethod before each test
  private WriteBehindViewComputationCache _cache;

  // a new instance is created for each value returned by the data provider
  @Factory(dataProvider = "cacheHints")
  public WriteBehindViewComputationCacheTest(final CacheSelectHint filter) {
    _filter = filter;
  }

  // this method is called before each test method
  @BeforeMethod
  public void init() {
    _cache = new WriteBehindViewComputationCache(_filter);
  }

  @Test
  public void getNew() { ... }

  @Test
  public void getExisting() { ... }
}

Thus, three instances of the CachingTest class are created, each with a different value of CachingSelectHint. The two test methods, each preceeded by a call to init() are called three times each, once for each instance of the class. That gives six invocations in total. This approach is useful when sharing a single larger object, such as a more heavyweight object, between multiple tests.

Assertions

A key area of difference between JUnit and TestNG is the assertions. JUnit specifies the expected value followed by the actual value, whereas this is reversed in TestNG. However, TestNG supplies a class called AssertJUnit where the methods match the order of JUnit. This class is used in the auto-conversion, and OpenGamma intends to continue writing assert statements this way around.

Personally, I find the different ordering of the frameworks confusing. As I switch between projects I frequently get it wrong! Thus for OpenGamma it definitely made sense to keep the JUnit style, so other team members didn't have to change their test writing habits.

Tool integration

A further key feature of TestNG is the ability to integrate with other tools. TestNG itself supplies a full-featured Eclipse plugin, with test runner, quick fixes and the auto-conversion described above. TestNG also includes integration with Ant.

The integration also includes the ability to generate an XML file in the same format as that which JUnit would produce. This can then be used as input into a variety of other tools, including the JUnit report runner (TestNG has its own HTML report as well).

At OpenGamma, we successfully integrated TestNG with Ant, Bamboo and Clover.

Advanced features

Now that the conversion is complete, the whole OpenGamma team can start using the more advanced features and reap the benefits of the conversion.

Tests can have dependencies. It is possible to setup the tests so that a failure in one test causes other tests to be skipped. This avoids having the tests take a long time to run when a key element has been broken.

Tests can be grouped. The groups attribute allows a test to be allocated to a group, such as "fast" or "database". The configuration can choose which groups to run, allowing a subset of the full suite to be easily accessed. Dependencies between groups can be setup allowing a quick "smoke test" to be created that is run before the longer more complex tests.

Tests can be run multiple times. The same test can be run multiple times, potentially in parallel. This can be linked to a success percentage, which could be used to handle a test occasionally failing due to a network issue.

Summary

The conversion from JUnit to TestNG was successful and relatively painless. This was greatly helped by the rapid response of Cedric to questions and issues that were raised. That assistance was vital to the conversion - thanks again Cedric!

Hopefully all users of TestNG will benefit from the bug fixes and enhancements that were recently added. I certainly think the @Factory changes are very useful! I'd recommend other projects that have a need for large-scale testing beyond simple unit tests to consider whether switching to TestNG might be beneficial. And if you're starting a new project, perhaps you should consider using TestNG from the start!

This is the developer blog of OpenGamma. For more posts by the OpenGamma team, check out our main blog.

About the Author

Stephen Colebourne

Stephen Colebourne is Engineering Lead, Platform at OpenGamma.

Follow us on Twitter