a little madness

Earlier in the year I put Selenium in a cage using Xnest. This allows me to run browser-popping tests in the background without disturbing my desktop or (crucially) stealing my focus.

On that post Rohan stopped by to mention a nice alternative to Xnest: Xephyr. As the Xephyr homepage will tell you:

Xephyr is a kdrive based X Server which targets a window on a host X Server as its framebuffer. Unlike Xnest it supports modern X extensions ( even if host server doesn’t ) such as Composite, Damage, randr etc (no GLX support now). It uses SHM Images and shadow framebuffer updates to provide good performance. It also has a visual debugging mode for observing screen updates.

It sounded sweet, but I hadn’t tried it out until recently, on a newer box where I didn’t already have Xnest setup. The good news is the setup is as simple as with Xnest in my prior post:

1. Install Xephyr: which runs an X server inside a window:
   $ sudo apt-get install xserver-xephyr
2. Install a simple window manager: again, for old times’ sake, I’ve gone for fvwm:
   $ sudo apt-get install fvwm
3. Start Xephyr: choose an unused display number (most standard setups will already be using 0) — I chose 1. As with Xnest, the -ac flag turns off access control, which you might want to be more careful about. My choice of window size is largely arbitrary:
   $ Xephyr :1 -screen 1024×768 -ac &
4. Set DISPLAY: so that subsequent X programs connect to Xephyr, you need to set the environment variable DISPLAY to whatever you passed as the first argument to Xephyr above:
   $ export DISPLAY=:1
5. Start your window manager: to manage windows in your nested X instance:
   $ fvwm &
6. Run your tests: however you normally would:
   $ ant accept.master

Then just sit back and watch the browsers launched by Selenium trapped in the Xephyr window. Let’s see them take your focus now!

Lately one of our Pulse agents has been bogged down, to the extent that some of our heavier acceptance tests started to genuinely time out. Tests failing due to environmental factors can lead to homicidal mania, so I’ve been trying to diagnose what is going on before someone gets hurt!

The box in question runs Windows Vista, and I noticed while poking around that some disk operations were very slow. In fact, deleting even a handful of files via Explorer took so long that I gave up (we’re talking hours here). About this time I fired up the Reliability and Performance Manager that comes with Vista (Control Panel > Administrative Tools). I noticed that there was constant disk activity, and a lot of it centered around C:\$MFT — the NTFS Master File Table.

I had already pared back the background tasks on this machine: the Recycle Bin was disabled, Search Indexing was turned off and Defrag ran on a regular schedule. So why was my file system so dog slow? The answer came when I looked into the AppData\Local\Temp directory for the user running the Pulse agent. The directory was filled with tens of thousands of entries, many of which were directories that themselves contained many files.

The junk that had built up in this directory was quite astounding. Although some of it can be explained by tests that don’t clean up after themselves, I believe a lot of the junk came from tests that had to be killed forcefully without time to clean up. It was also evident that every second component we were using was part of the problem – Selenium, JFreechart, JNA, Ant and Ivy all joined the party.

So, how to resolve this? Of course any tests that don’t clean up after themselves should be fixed. But in reality this won’t always work — especially given the fact that Windows will not allow open files to be deleted. So the practical solution is to regularly clean out the temporary directory. In fact, it’s quite easy to set up a little Pulse project that will do just that, and let Pulse do the work of scheduling it via a cron trigger. With Pulse in control of the scheduling there is no risk the cleanup will overlap with another build.

A more general solution is to start with a guaranteed-clean environment in the first place. After all, acceptance tests have a habit of messing with a machine in other ways too. Re-imaging the machine after each build, or using a virtual machine that can be restored to a clean state, is a more reliable way to avoid the junk. Pulse is actually designed to allow reimaging/rebooting of agents to be done in a post-stage hook — the agent management code on the master allows for agents to go offline at this point, and not try to reuse them until their status can be confirmed by a later ping.

Many developers seem to look upon software build tools (Ant/Make/Maven/Rake/SCons/etc) with disdain. They see them as painful creatures put on the earth to gnaw at their patience. The developer just wants to get on with Real Work (coding), and not have to deal with such annoying details. This attitude arises for a few reasons:

• Someone Else’s Problem: some don’t see writing/maintaining build scripts as part of their job. They’re programmers, not build/release engineers.
• Yet Another Language: build tools (frequently) use a custom language for their scripts. Some resist having to learn Yet Another Language.
• Real Pain: writing, and particularly debugging build scripts can be a right pain. The language might be restrictive, debugging support limited, the behaviour arcane.
• It’s Good Enough: at the start of the project someone slapped together a copy-pasted build script that just, feebly, manages to output something resembling a binary. Nobody will ever bother looking at the build system again.
• Lack Of Time: improving the build system is not a customer requirement. Hence, nobody will ever get around to doing it.

I would argue that this attitude is harmful. I could even refute each point above, one by one. But it doesn’t matter. Even if all these points were valid, there is a single overriding factor: the build system is dog food you have to eat. If you don’t like the taste, why are you putting up with it? So: learn your build tool, and learn it well. Invest a day in improving the build system, and reap the benefits every day thereafter. Keep the knowledge with you, and refactor your build scripts just like your code to make everyone’s life easier.

I can remember many a time that I put off making such improvements for an age. When I finally make the change, the first thought is always: I rock! The next thought is: I’m a fool for not doing this sooner!

In this article, we look into automating the release process. We begin by reviewing the benefits of automated releases. We then take a look at common steps involved in the automation process, and some of the challenges they may present.

Read the full article at zutubi.com.

Recently I’ve been cooking up some new acceptance-level tests for Pulse. The sort of tests that need to take a Pulse release package, unpack it, start the server and poke it with a stick. This is the sort of automation that I would typically use bash for: since a lot of the work involves executing and managing subprocess. However, I had learned from earlier testing that process control from bash is not very portable. More specifically, the behaviour of Cygwin bash on Windows differs considerably from *nix bash. At the time, I worked around the problem by doing some process control using Ant, but there had to be a better way.

Enter Python. Or, more specifically, the new(ish) subprocess module (introduced in Python 2.4). With this module Python finally has a concise but powerful way to launch and control subprocesses. It also includes convenient ways to handle subprocess I/O, including setting up pipelines. What’s more, I have so far found it to be quite portable between Linux and Windows.

Starting with something simple, let’s say you just want to launch a command synchronously (much as you would in a shell script):

Simple. Want to set up a pipeline like `dmesg | grep hda` (capturing the final output)? Here’s an example straight from the doco:

For my specific case, I used the ability to launch a Pulse server processes in the background and capture the output extensively. When done, I shut down the server using the standard shutdown script (testing of said script being a nice side-effect). An asynchronous child is started simply by creating a Popen object:

I have simplified this example a bit, but the real work is all there, and it’s pleasantly straightforward. This combined well with Python’s simple XML-RPC API, which I used to manipulate the running server.

So far the switch to from bash/Ant to Python has been a great success. If you need a cross-platform way to automate your acceptance testing, give it a go.

Each day, every developer on a software team interacts with the build system several times. Despite this, many teams underplay the importance of an effective build system. Is your build system working for you, or do you merely tolerate its existence? In this article, we will journey through build system requirements starting from the most basic through to those that make the whole development process more efficient.

Read the The Road To Build Enlightenment at zutubi.com.

Well, if they try to do anything useful anyway. Last week I was working on improvements to the Windows service installation for Pulse. The existing service installation was handled by a pretty simple batch file wrapping the service executable. However, this method had some major limitations, mostly related to location of a compatible JVM. I needed to expand the logic in the batch script considerably as part of the solution to these limitations.

That’s when I realised: logic and batch scripts don’t really go together. In fact, scripting and batch scripts don’t really go together. Even something as simple as reading the output of an executed command is bizarre in batch (there is an obscure for loop syntax to do it). Fed up with the archaic nature and limitations of batch files, I went looking for an alternative. I had heard of powershell (aka Monad), but of course it is not yet widely installed. So I turned to Windows Script Host, which has been around since Windows 98. I hadn’t used it before, but I discovered you could write scripts in JScript (Javascript-like) and easily access the filesystem and registry, so it seemed like a good fit.

In fact, apart from the pain of navigating MSDN documentation online, my foray into WSH started quite promisingly. Then, just as I was really starting to get somewhere, Norton Antivirus chimed in. Apparently, this script that I was running was trying to do something “malicious”. Whenever I tried to do something moderately useful in the script, like access file or registry information, Norton would warn me in no uncertain terms. Brilliant. No matter that I was running the script directly, just as I could run any executable that could do equally “malicious” things. I suppose Symantec doesn’t care about false positives; that might take effort. Instead, WSH has been rendered useless to software distributors, except for those willing to have their customers warned that their software is “malicious” by a virus scanner.

In the end I implemented the solution as a standalone C++ program, because at least that way I knew I could get it to work. It’s a sad state of affairs when I am forced to write a few hundred lines of C++ when a 50 line script should do. That’s what happens when a platform is completely devoid of useful scripting options.

Well, it’s finally here! The first public beta of pulse 1.1. This is a huge milestone for us, just take a look at what’s new to see what I mean. The biggest single feature is powerful distributed building, but as you’ll see the list is long!

Thanks so much to our private pre-release testers, your feedback has been invaluable. Now for the rest of you: bring it on!

Pulse version 1.0.6 has been released. This release fixes some minor issues in 1.0.5, see the release notes for details.

Zutubi is proud to announce the availability of the Pulse automated build (or continuous integration) server for sale from today. This is the culmination of many months of development and beta testing. We would like to thank all of our beta testers for their feedback during this period.

If you haven’t tried it yet, download pulse today and let us know what you think!