Feeding The Right Sources To Your Builds

I was recently asked the best approach for how to control the input to one's builds, and while there is no single solution to this that fits all projects, here are two ways you might want to go.

Freeze Then Branch

The current approach for JBoss Tools development involves continuous nightly builds from trunk until such time as the project is deemed frozen in prep for an upcoming milestone or release candidate. At that time, the trunk is copied to branches/JBossTools-3.1.0.M4, and very little is done to that branch - only urgent tweaks and fixes. A new Hudson job is cloned from the current one, and adjusted to point at the branched sources instead of trunk. The maps are also updated to point at the new source location in SVN.

This allows more nightly builds towards an upcoming stable development milestone, while new development can continue in parallel in trunk. When the milestone is released, the unneeded job is disabled or removed.

The only issue with this approach is that all plugins, built from the head of a branch (or trunk) are re-versioned w/ the latest timestamp every time they're compiled. Thus upgrading from one 80M update to the next requires another 80M download. To mitigate this, milestones are released only once every month or two.

Tag And Release

The current approach for numerous Eclipse.org projects, such as GEF, is to develop continuously in HEAD and when a number of changes are ready to be released, a tool such as the org.eclipse.releng.tools plugin can be used to, in a single operation, both tag the sources and release those tags into the map file.

This permits a granular approach to plugin versioning wherein only the plugins which have actually changed are renumbered, and thus incremental updates between releases is possible, and if only a single plugin changes from week to week, only that plugin will be upgraded.

This approach also allows your adopters to get more stable, weekly updates rather than hourly or ad hoc nightlies which may include incomplete changes. Sure, you can only watch for good builds in Hudson, but a more predictable schedule makes inter-build communication easier.

The only issue with this approach is that it introduces extra overhead, unless the tag&release process can be automated and run on a (weekly?) schedule. For CVS sources, there is the org.eclipse.releng.tools.tagandrelease crontab script available; for SVN, no such script (yet) exists. Want to help? See bug 264713.

Hybrid Approach

With the Athena Common Build, you can set up a Hudson job to run from tagged maps, but also override those tags on demand to force a build from head (or trunk) instead of from the specific repo tags.

To do so, pass in the following flags to EXTRAFLAGS via your build's job configuration / parameters. To build from the workspace's sources instead of fetching anything new from the repo (bug 252774):

-localSourceCheckoutDir /opt/users/hudsonbuild/.hudson/jobs/${JOB_NAME}/workspace/

To fetch from the repo, but ignore tags (bug 251926):

-forceContextQualifier -fetchTag HEAD

Or, like many projects on build.eclipse.org have done, set up two jobs: one for nightly from trunk, and one for weekly integration from maps. Then instead of doing CI builds when you remember to, they will run automatically when the repo changes so you'll have immediate feedback when something has broken. Add email, IRC, or Twitter notification to the job, and everyone else will know too.

Tracking Build Status With Hudson Data APIs

A number of people have been twittering recently about Hudson Helper, and the fact that it can't (yet) support http access to Hudson servers. (There's just no pleasing some people, eh David?)

UPDATE: David reports that Hudson Helper has worked with both http and https since day one. He invites direct feedback if you're having problems.

To help fill this gap, I'd like to detail some of the handy API features of Hudson I've discovered since I first started using it back in October, which cane be fetched via http (or https) in a browser or via a script.

Datum	Example
Latest Successful build number	buildNumber
Latest Successful zip (published artifact)	GEF-Update-*.zip
All checked out Project Set Files (Hudson workspace)	*.psf
XML Digest of Latest Stable Build	lastStableBuild/api/xml
SVN revision used for Latest Stable Build	//changeSet/revision/revision

For more on the APIs available to the Latest Successful, Stable, Failed, or in fact simply the Latest Build, see:

1. /lastSuccessfulBuild/api/
2. /lastStableBuild/api/
3. /lastFailedBuild/api/
4. /lastBuild/api/

Of course, should you want details on a specific build rather than the latest, you can replace the "last*Build" token with an actual build number.

Finally, because no post about APIs should be complete with out some script showing how to exploit that interface, here's a quick example of how to fetch the latest successful, and as yet unreleased Drools 5.1 runtime library zip for use in our JBoss Tools 3.1.0.M2 builds. In this example, we fetch the build number for the last successful build and compare it to a cached version. We also fetch and cache the latest SVN revision number (in a build.properties file) so that we can later fetch Drools sources from the same point in time as the precompiled Drools binaries in the zip. This guarantees we're building from trunk, but only a good build in trunk, skipping over any failed builds or intermediate states (partial commits).

#!/bin/bash
droolsSNAPSHOTnum=drools-SNAPSHOT-num.txt
droolsSNAPSHOTrev=drools-SNAPSHOT-rev.txt
droolsSNAPSHOTzip=drools-SNAPSHOT-bin.zip
droolsSNAPSHOTurl=http://jboss-hudson-server/hudson/job/drools/lastSuccessfulBuild/artifact/trunk/target/drools-5.1.0.SNAPSHOT-bin.zip

buildNumOld=0; if [[ -f $droolsSNAPSHOTnum ]]; then buildNumOld=$(cat $droolsSNAPSHOTnum); fi
buildNumNew=$(wget -q --no-clobber -O - http://jboss-hudson-server/hudson/job/drools/lastSuccessfulBuild/buildNumber)

buildRevOld=0; if [[ -f $droolsSNAPSHOTrev ]]; then buildRevOld=$(cat $droolsSNAPSHOTrev); fi
buildRevNew=$(wget -q --no-clobber -O - http://jboss-hudson-server/hudson/job/drools/lastSuccessfulBuild/api/xml?xpath=//changeSet/revision/revision)

if [[ $buildNumNew -gt $buildNumOld ]]; then
 # get: 27013; must change to 27013 
 echo $buildRevNew > $droolsSNAPSHOTrev;
 sed -i "s#\|##g" $droolsSNAPSHOTrev 
 buildRevNew="$(cat $droolsSNAPSHOTrev)"; #echo "."$buildRevNew"."
 # replace "defaultTag=trunk:\d+" with defaultTag=trunk:${buildRevNew} in build.properties
 #  defaultSvnUrl=http://anonsvn.jboss.org/repos/labs/labs/jbossrules
 #  defaultTag=trunk:27013
 sed -i "s#defaultTag=trunk:\d\+#defaultTag=trunk:$buildRevNew#g" build.properties; # grep "defaultTag=trunk:" build.properties 

 echo $buildNumNew > $droolsSNAPSHOTnum; 
 echo "Download $droolsSNAPSHOTurl ..."
 wget -q --no-clobber -O $droolsSNAPSHOTzip $droolsSNAPSHOTurl 

 # ...

fi

Oh, and BTW, if you're ever looking for the latest hudson.war executable, it's always here.