Just a quick note to mention that Daniel will be carrying the torch for Zutubi at CITCON Asia-Pacific 2008 in Melbourne this weekend. After enjoying the event last year we signed up to sponsor this time around. Catch up with Daniel and ask him about Pulse 2.0 beta!
Sadly I won’t make it, on account of being on the other side of the planet…
Maybe my brain just doesn’t work in a way that appreciates them, but I really don’t see the point of tag clouds. Sure, adding a cloud makes your site feel oh-so 2.0, but does it really accomplish anything? Is there some advantage over a sorted list? Do people actually use tag clouds?
I find them annoying because they:
When the information is useful I would prefer it to be presented as sorted and/or coloured list of a uniform font size. I for one would be quite happy to see these useless jumbles of letters disappear!
Ever need to stub just a method or two of some interface to nicely isolate a unit to test? In the past I have used classic mock object libraries to help, but these suffered from a few problems:
Some time ago EasyMock came along and solved the first two problems. By using a new record-replay interface for defining expected calls, EasyMock tests were both more intuitive and more maintainable. However the third problem remained: sometimes I just want to stub the methods, and not verify that they are called in any specific order or number of times. Some contraints can be relaxed with EasyMock but it always seemed a pain to do extra work when I just wanted the library to do less!
Well, last week I discovered Mockito, a more recent library inspired in part by EasyMock. The difference with Mockito is that stubbing and verification are separated. Instead of recording expectations, you:
This separation works brilliantly when you don’t want to verify everything by default. When the test is unconcerned with exactly how the stub is called, there is no need to verify at all.
You can read more about how Mockito came about on the author’s blog. I highly recommend checking it out!
Anyone who does web development, particularly involving a significant amount of JavaScript, knows what a boon Firebug has been. The ability to debug your code in Firefox is a huge time saver. Unfortunately, those doing web development also know that they need to support Internet Explorer. Not only does IE have a host of quirks, it also has almost no support for debugging JavaScript.
For a long time I have been relying on Microsoft Script Debugger to debug IE-specific JavaScript problems. By installing this debugger and configuring IE to allow external script debugging (by unchecking Tools > Internet Options > Advanced > Disable Script Debugging (Other)), you could trap errors in an external debugger window. The biggest gain from this was the ability to determine the exact line where the error occurred - something that could be impossible to decipher from the built-in error dialog in IE. The script debugger also provides a call stack and a rudimentary command window allowing some inspection of the program state. However, the interface is clunky at best, making navigation through the code and inspection difficult.
Now, though, I have finally stumbled upon a Better Way. For some time Microsoft have made limited “Express” versions of Visual Studio tools available for free (with registration). One such free tool is Visual Web Developer Express. Like the other Visual Studio tools, Web Developer includes a decent debugger - in this case one that understands JavaScript. The only trick is getting the Express version of Web Developer to debug an external IE process for you. Thankfully, someone has figured out how to do just that and blogged about it. Essentially, you create a dummy project in Web Developer and configure it to point at your external web application. Now when you “Debug” the project Web Developer will launch IE and any errors will be trapped by the debugger. Happy debugging!
Note for FireFox users: If, like me, FireFox is your default browser, you may also need to set IE as your default browser in Web Developer. You can do this by opening an HTML file, then going to “File > Browse With …” and setting IE to the default.
Ubuntu 8.04, the Hardy Heron, was released last week. I eagerly upgraded to see if my previous wireless networking issues had been resolved, and joy of joys they had! This allowed me to finally switch back to Ubuntu as my primary development environment.
Most of the switch went without a hitch, however I did encounter one problem. The acceptance tests for Pulse rely heavily on Selenium. When trying to launch Selenium on Hardy, Selenium RC always got stuck at:
Preparing Firefox profile…
The default version of Firefox installed with Hardy is Firefox 3 beta 5 - as opposed to Firefox 2 in previous releases. So it seems that Selenium and Firefox 3 don’t agree with each other - or at least not when using the current release of Selenium (1.0-b1). My best guess is that Selenium is trying to install extensions into Firefox but is failing as the compatible versions for the extensions don’t include Firefox 3. A better diagnostic would be nice!
In any case, the easiest resolution was to install Firefox 2, which can sit alongside Firefox 3 on Hardy. Just install the firefox-2 package:
Then make sure Selenium uses this version. One way to do this is to create a link named firefox-bin in your PATH that points to the Firefox 2 binary. However, I prefer to leave my system clean and make an exception just for Selenium. To allow this, our tests support an optional SELENIUM_BROWSER environment variable that specifies in Selenium format the browser to use (this comes in handy in other cases too). So before running tests, I ensure that SELENIUM_BROWSER is set:
To make this my default with no extra effort, I have actually just added it to my .bashrc.
For many years, step 2 has eluded the slashdot community:
1: Start your project.
2: …
3. Profit!
Now, however, Ext, the company behind the popular ExtJS Javascript library, seem to believe they have pinned down this step. In the latest 2.1 release, Ext has switched from a dual LGPL/commercial license model to a dual GPL/commercial license model. Despite claims from the company that this was done due to community concerns about ExtJS not being fully open-source, the majority have seen what this really is:
1: Start your project under a LGPL/commercial license model.
2: Switch to a GPL/commercial license model.
3: Profit!
The model works because step 1 allows you to build a community around the more liberal LGPL license. In particular, as the LGPL is commercial-friendly, the community will include many people building commercial applications. Once the community is suckered in and committed, the license is changed, leaving them high and dry. Well, not quite: they can continue to use new versions of the library by buying a commercial license. Hence the profit!
Not surprisingly, this sudden license switch has caused a large community backlash. People don’t like to feel that they have been baited in this way. The defense from Ext runs along a couple of lines:
The backlash has been all the greater because of how the switch has been handled:
The saddest part about this is that the Ext team really have built a fantastic library, and a vibrant community around it. The library had all the hallmarks of an open source success story. Now, however, Ext have committed the cardinal sin of an open source project: they have undermined the trust of their own community. I can see the strategy making a short term profit, and perhaps even a long term business. I cannot, however, see the community returning to full strength. In the long run, although Ext may have a successful commercial offering, they will lose a large part of their current and potential community to competing and still-open projects. The power of the community will see those competitors flourish.
A major focus of agile development is fast feedback - of many kinds. As this idea is taken to its extreme, it is worthwhile asking if feedback can be too fast. That is, is there a point at which shrinking the feedback loop becomes counterproductive?
The benefit of feedback is knowing if you are heading in the right direction. The faster the feedback, the less time you spend heading towards dead ends. So the benefit increases as feedback gets faster. But what of the cost? It is useful to split the cost into two categories:
Setup cost is not usually a big issue except in extreme cases, as it can be paid off over time. More interesting is the ongoing cost of receiving and processing feedback. Naturally receiving, understanding and acting on feedback takes time. The important part is distinguishing when this time is productive versus wasted. Processing feedback can be wasteful in various ways:
How do these problems relate to the speed of feedback? The cost of false alarms is roughly inversely proportional to the length of the feedback cycle. As the relationship is linear, false alarms are not compounded by making feedback faster. In any case, accuracy of feedback is always paramount: false alarms must be eliminated.
More interesting is the relationship between feedback speed and the other two potential problems: distraction and repetition. In both these cases, shrinking the feedback cycle may produce a disproportionate increase in the occurrence of problems. Thus, when shrinking the feedback cycle, care must be taken to address both of these areas.
Distraction can be addressed in multiple ways. Firstly, receipt of feedback should be optional. Users must be able to pause the feedback mechanism (or possibly switch it off altogether). Then when a user needs to for deep thought, they can ensure that they are not interrupted. Secondly, the method of feedback needs to be in keeping with both the importance and frequency of the message to convey. For example, an emergency that requires immediate attention may use a bold and hard to ignore mechanism, whereas a small status update should be less intrusive. Continuous feedback must be modeless — as an example think of the real time spell-checking functionality in many word processors. Lastly, feedback should be categorised to allow smart filtering. Different users often require different levels of feedback — what is important for some is a trivial detail to others.
Repetition is usually simpler to address: feedback can be suppressed when there is no change. Note that in some cases a change for one user may not represent a change for another user. Thus, as above, filtering that is based on individual users may be necessary.
The value of feedback, and indeed fast feedback, is unquestioned. Taking this principle to the extreme, it follows that faster feedback is better. I feel this is true provided that the crucial issues of distraction and repetition are addressed.
As our new acceptance test suite for Pulse 2.0 has grown, naturally it has taken longer to execute. More problematicly, however, several tests towards the end of the suite were taking longer and longer to run. This clearly highlighted a scalability issue: as more data was piled into the testing installation, tests took longer to execute. The situation became so bad that a single save in the configuration UI could take tens of seconds to complete.
A simple way to solve this would be to blow away the data after each test (or at regular intervals) so that we don’t see the build up. This would improve the performance of the test suite, but would also hide a clear performance issue in the application. Better to keep on eating our own dog food and fix the underlying problem itself. This way not only would the tests execute more quickly, but (more importantly) the product would also be improved.
Basic profiling identified the major bootleneck was many repeated calls to Introspector.getBeanInfo. This call was known by us to be slow, but according the documentation the results should be cached internally. It turns out that the cache is ignored when a certain argument is used (quite common!), so we were triggering a large amount of reflection many times over. By adding a simple cache on top of this method, it practically disappeared from the profiles and the time taken to run our test suite was better than halved! More importantly, the latency in the UI was decreased by several times to an acceptable level.
Naturally optimisation is not always this easy. The more tuned your application, the harder it becomes to find such dramatic improvements. However, it pays to keep an eye on your test suite performance, and profile if things don’t feel right. You might just find a simple win-win situation like the one described.
As a part of our continued efforts to support every unit testing library we can in Pulse, I have been looking into CUnit, the imaginitively-named framework for plain C.
The first thing I noticed about CUnit was the clean website and full documentation. This is a rarity amongst the C-based libraries I have seen, and very welcome to a first-time user. The structure supported by the library is a fairly standard suite-based grouping of individual test functions. Notably, CUnit comes with a few reporting interfaces out of the box - including the all important XML reports (typically the easiest way to integrate into a continuous build).
I built the library using Visual C++ 2008 Express. Although the provided VC files were from an earlier version, 2008 converted and built the solution without issues. The static library appeared at:
Note: I first tried my luck with an unsupported configuration by building the library under Cygwin using ftjam. This didn’t work, although I suspect porting the build would not be hard.
Next I started with a contrived test program to see how it all fits together. The program has a single suite with setup and teardown functions, with one passing an one failing case.
char *gFoo;
int setup(void)
{
gFoo = strdup(“i can has cheezburger”);
if(gFoo == NULL)
{
return 1;
}
return 0;
}
int teardown(void)
{
free(gFoo);
return 0;
}
void testPass(void)
{
CU_ASSERT_STRING_EQUAL(gFoo, “i can has cheezburger”);
}
void testFail(void)
{
CU_ASSERT_STRING_EQUAL(gFoo, “no soup for you!”);
}
int main()
{
CU_pSuite pSuite = NULL;
if(CU_initialize_registry() != CUE_SUCCESS)
{
return CU_get_error();
}
pSuite = CU_add_suite(“First Suite”, setup, teardown);
if(pSuite == NULL)
{
goto exit;
}
if(CU_add_test(pSuite, “Test Pass”, testPass) == NULL)
{
goto exit;
}
if(CU_add_test(pSuite, “Test Fail”, testFail) == NULL)
{
goto exit;
}
CU_basic_set_mode(CU_BRM_VERBOSE);
CU_basic_run_tests();
exit:
CU_cleanup_registry();
return CU_get_error();
}
The code illustrates how to write test functions with assertions, as well as how to assemble a test suite and run it with the built-in basic UI. The output from this program is shown below:
CUnit - A Unit testing framework for C - Version 2.1-0
http://cunit.sourceforge.net/
Suite: First Suite
Test: Test Pass ... passed
Test: Test Fail ... FAILED
1. c:projectscunitplayfirsttestmain.cpp:30 -
CU_ASSERT_STRING_EQUAL(gFoo,"no soup for you!")
--Run Summary: Type Total Ran Passed Failed
suites 1 1 n/a 0
tests 2 2 1 1
asserts 2 2 1 1
Generating an XML report required a very small change. Rather than using the basic UI, I switched to the automated UI and set the output filename prefix:
/* Test functions the same as previous example. */
…
int main()
{
/* All setup code remains the same */
…
if(CU_add_test(pSuite, “Test Fail”, testFail) == NULL)
{
goto exit;
}
CU_set_output_filename(“CUnit”);
CU_automated_run_tests();
CU_list_tests_to_file();
exit:
CU_cleanup_registry();
return CU_get_error();
}
Running this program produces two XML output files: CUnit-Listing.xml and CUnit-Results.xml. The former contains data about the test suites and cases themselves, the latter contains the results of running the tests. This latter report is useful for integration with other systems; e.g. it is the report that we now post-process to pull test results into Pulse.
Note that these XML files come with a DTD and stylesheet, which is useful, but also akward. As the DTD and XSL file are not present alongside the reports where they are generated, opening the files will fail.
Starting from this vey basic test setup leaves plenty of room for improvement. For example, if you look at the sample code above, you will notice that a lot of time is spent assembling the suite and checking for errors. Thankfully the CUnit authors have noticed the same and offer simplifications:
Another improvement that could perhaps be added in the future is better reporting for assertion failures. Looking back at the assertion failure output:
Test: Test Fail ... FAILED
1. c:projectscunitplayfirsttestmain.cpp:30 -
CU_ASSERT_STRING_EQUAL(gFoo,"no soup for you!")
you can see that it pulls out the assertion expression as-is. However, given that I am using a specific string equals assertion, CUnit should also be able to show me the actual values that were compared at runtime, and even the difference. This can avoid the need to run the test again under a debugger to see the values.
Overall I would rate CUnit as a decent choice for CUnit testing. In terms of features all the basics are covered, and important improvements have been made. There is not much new, however, and there is still room for improvement. Where this library really shines is in the out of the box experience; thanks to excellent documentation and multiple included UIs.
If you’re looking for a plain C unit testing solution this is the best open source alternative I have seen. While you’re at it, why not go for a full continuous integeration setup with the CUnit support now in Pulse ;).
Checked exceptions is one of those Java features that tends to work up a lively discussion. So forgive me for rehashing an old debate, but I am dismayed to see continued rejection of them as a failed feature, and even more so to see proposals to remove them from the language (not that I ever think this will happen).
Most would agree that mistakes were made in the way checked exceptions were used in the early Java years. Some libraries even go so far as doing something about it. However, an argument against the feature itself this does not make. Like anything new, people just needed time in the trenches to learn how to get the most out of checked exceptions. A key point here is that checked exceptions are available but not compulsory. This leads to a reasonable conclusion, as made in Effective Java:
Use checked exceptions for recoverable conditions and runtime exceptions for programming errors
The key point here is recoverable. If you can and should handle the exception, and usually close to the source, then a checked exception is exactly what you need. This will encourage proper handling of an error case - and make it impossible to not realise the case exists. For unrecoverable problems, handling is unlikely to be close to the source, and handling (if any) is likely to be more generic. In this case an unchecked exception is used, to avoid maintenance problems and the leakage of implementation details.
None of the above is new, and to me it is not contraversial. But the continued attacks on checked exceptions indicate that some still disagree. Let’s take a look at some common arguments against checked exceptions that still get airtime:
Some argue that the proliferation of code that either swallows checked exceptions (catches them without handling them) or just rethrows them all (”throws Exception”) is evidence that the feature is flawed. To me this argument carries no weight. If we look at why exception handling is abused in this way, it boils down to two possibilities:
You need well-designed APIs and good programmers to get quality software, whether you are using checked exceptions or not.
A common complaint is how changes to the exception specification for a method bubbles outwards, creating a maintenance nightmare. There are two responses to this:
Switching every last exception to unchecked won’t make maintenance problems go away, it will just make it easier to ignore them - to your own detriment.
Some argue that all the try/catch blocks that checked exceptions force on us clutter up the code. Is the alternative, however, to elide the error handling? If you need to handle an error, the code needs to go somewhere. In an exception-based language, that is a try-catch block, whether your exception is checked or not. At least the exception mechanism lets you take the error handling code out of line from your actual functionality. You could argue that the Java’s exception handling syntax and abstraction capabilities make error handling more repetitive than it should be - and I would agree with you. This is orthogonal to the checked vs unchecked issue, though.
Some would argue that instead of the complier enforcing exception handling in a rigid way, it would be better to rely on tests catching problems at runtime. This is analogous to the debate re: static vs dynamic typing, and is admittedly not a clear cut issue. My response to this is that earlier and more reliable feedback (i.e. from the compiler) is clearly beneficial. Measuring the relative cost of maintaining exception specifications versus maintaining tests is more difficult. In cases where an exception almost certainly needs to be recovered from (i.e. the only case where a checked exception should be used), however, I would argue that the testing would be at least as expensive, and less reliable.
I am yet to hear an argument against checked exceptions that I find convincing. To me, they are a valuable language feature that, when used correctly, makes programs more robust. As such, it is nonsense to suggest throwing them out. Instead, the focus should be on encouraging their appropriate use.
A little madness is the blog of Daniel Ostermeier and Jason Sankey. We are the founders of zutubi, a company based in Sydney, Australia that makes software for software developers.
