a little madness

As a part of our continued efforts to support every unit testing library we can in Pulse, I have been looking into CUnit, the imaginitively-named framework for plain C.

Initial Impressions

The first thing I noticed about CUnit was the clean website and full documentation. This is a rarity amongst the C-based libraries I have seen, and very welcome to a first-time user. The structure supported by the library is a fairly standard suite-based grouping of individual test functions. Notably, CUnit comes with a few reporting interfaces out of the box – including the all important XML reports (typically the easiest way to integrate into a continuous build).

Setup on Windows

I built the library using Visual C++ 2008 Express. Although the provided VC files were from an earlier version, 2008 converted and built the solution without issues. The static library appeared at:

Note: I first tried my luck with an unsupported configuration by building the library under Cygwin using ftjam. This didn’t work, although I suspect porting the build would not be hard.

The First Test

Next I started with a contrived test program to see how it all fits together. The program has a single suite with setup and teardown functions, with one passing an one failing case.

The code illustrates how to write test functions with assertions, as well as how to assemble a test suite and run it with the built-in basic UI. The output from this program is shown below:

  CUnit - A Unit testing framework for C - Version 2.1-0
http://cunit.sourceforge.net/
Suite: First Suite
  Test: Test Pass ... passed
  Test: Test Fail ... FAILED
    1. c:\projects\cunitplay\firsttest\main.cpp:30  -
        CU_ASSERT_STRING_EQUAL(gFoo,"no soup for you!")
--Run Summary: Type      Total     Ran  Passed  Failed
               suites        1       1     n/a       0
               tests         2       2       1       1
               asserts       2       2       1       1

Generating XML Reports

Generating an XML report required a very small change. Rather than using the basic UI, I switched to the automated UI and set the output filename prefix:

Running this program produces two XML output files: CUnit-Listing.xml and CUnit-Results.xml. The former contains data about the test suites and cases themselves, the latter contains the results of running the tests. This latter report is useful for integration with other systems; e.g. it is the report that we now post-process to pull test results into Pulse.

Note that these XML files come with a DTD and stylesheet, which is useful, but also akward. As the DTD and XSL file are not present alongside the reports where they are generated, opening the files will fail.

Improvements

Starting from this vey basic test setup leaves plenty of room for improvement. For example, if you look at the sample code above, you will notice that a lot of time is spent assembling the suite and checking for errors. Thankfully the CUnit authors have noticed the same and offer simplifications:

• There are shortcuts for managing tests. For a non-trivial suite this method of describing the test suites in nested arrays and registering in one hit would be a lot simpler.
• You can optionally tell CUnit to abort on error. Since any error at the CUnit level is likely to be fatal, this should reduce the need for error handling.

Another improvement that could perhaps be added in the future is better reporting for assertion failures. Looking back at the assertion failure output:

  Test: Test Fail ... FAILED
    1. c:\projects\cunitplay\firsttest\main.cpp:30  -
        CU_ASSERT_STRING_EQUAL(gFoo,"no soup for you!")

you can see that it pulls out the assertion expression as-is. However, given that I am using a specific string equals assertion, CUnit should also be able to show me the actual values that were compared at runtime, and even the difference. This can avoid the need to run the test again under a debugger to see the values.

Conclusion

Overall I would rate CUnit as a decent choice for CUnit testing. In terms of features all the basics are covered, and important improvements have been made. There is not much new, however, and there is still room for improvement. Where this library really shines is in the out of the box experience; thanks to excellent documentation and multiple included UIs.

If you’re looking for a plain C unit testing solution this is the best open source alternative I have seen. While you’re at it, why not go for a full continuous integeration setup with the CUnit support now in Pulse .

This entry was posted on Wednesday, March 19th, 2008 at 12:37 am and is filed under Agile, Continuous Integration, Technology, Testing. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.