New Year's Day

Been too long since I blogged here. Time to begin again.

All is white on New Year's Day
Like a dead channel, on the TV
I want to be coding
Be coding night and day
The more things change
The more they stay the same

I will iterate again
I will iterate again

Under a neon bulb
Group of devs chatter, in @s and pings
Bugzillas logged, JIRAs are too
The blogosphere says, says 
Say it's true, it's true...
And we can break through
Though platforms too 
Many spring from one 

I... I will write some tests again
I... I will write some tests again

Oh...
Maybe the time is right
Oh... maybe tonight...

I'll debug my tests again
I'll debug my tests again

And so we're told it's the container age
Transform and deliver your apps a new way
Still I want to be coding
Be coding night and day
The more things change
The more they stay the same

U2 - New Year's Day

Build Nomenclature Conventions: What's in a name?

The following post is inspired by Mickael Istria's recent blog, Call a spade a spade, and a Nightly a Snapshot.

When I was doing builds for the Eclipse Modeling Project, I-builds were weekly published nightlies -- same level of stability as a SNAPSHOT (to use Maven parlance) or nightly, but published on a weekly schedule to bridge the gap between nightly/daily/SNAPSHOT/CI builds and the every-6-weeks milestone releases. The goal was to provide something stable enough for early adopters to grab once a week, but without the non-stop flux of nightlies. Regardless of the label on the build, the process was the same: tag CVS, then build using that tag.

The Final/GA/Release ("R") builds were done as simple renames of the last good milestone or release candidate build, so as to ensure binary-compatibility w/ the last-tested milestone/RC. The same was true for "M" and "S" builds -- they were just renamed "I" builds, and the letter was there simply to differentiate between a maintenance build (M), a stable milestone (S), or release (R).

Branching only happened when a release was done and it was time to produce the maintenance stream vs. the ongoing next-year-release. Sometimes branching would happen AFTER the x.y.1 maintenance because it saved duplication of commits in the x.y+1.0 and x.y.1 streams.

--

Now at JBoss, we publish "nightly" builds, which are keyed to SVN changes and therefore could be as often as hourly or as infrequent as weekly, depending on what's happening in the repo.

We also do milestone builds about once ever 6-8 weeks (similar to the Eclipse.org release train schedules), which is more carefully vetted, tested, and QE'd. It is produced using the same *process* as the nightlies, but are named differently and pulled from a freshly-created stable branch in the repo (so its degree of change/churn is less). (Branching happens right before every milestone or release candidate so that hardening/stabilization/documentation can happen in the branch while trunk stays open for new development.)

--

Bottom line -- I've only ever needed three types of builds, regardless of nomenclature or labelling differences. And of these 3, the last 2 are the same thing but renamed to underline the build quality/stability:

* nightly/CI/integration/weekly/SNAPSHOT build (unstable, for bleeding edge adopters)

* development milestone (probably a re-christened nightly; stable, early adopters)

* stable release / Final / GA (probably a re-christened milestone; release quality)

--

So... does it matter if it's called nightly, integration or SNAPSHOT? or Stable, Milestone, Maintenance, Final, GA or Release? As long as it's easily reproducible (yeah, Tycho!), what's in a name?

Feeding The Right Sources To Your Builds

I was recently asked the best approach for how to control the input to one's builds, and while there is no single solution to this that fits all projects, here are two ways you might want to go.

Freeze Then Branch

The current approach for JBoss Tools development involves continuous nightly builds from trunk until such time as the project is deemed frozen in prep for an upcoming milestone or release candidate. At that time, the trunk is copied to branches/JBossTools-3.1.0.M4, and very little is done to that branch - only urgent tweaks and fixes. A new Hudson job is cloned from the current one, and adjusted to point at the branched sources instead of trunk. The maps are also updated to point at the new source location in SVN.

This allows more nightly builds towards an upcoming stable development milestone, while new development can continue in parallel in trunk. When the milestone is released, the unneeded job is disabled or removed.

The only issue with this approach is that all plugins, built from the head of a branch (or trunk) are re-versioned w/ the latest timestamp every time they're compiled. Thus upgrading from one 80M update to the next requires another 80M download. To mitigate this, milestones are released only once every month or two.

Tag And Release

The current approach for numerous Eclipse.org projects, such as GEF, is to develop continuously in HEAD and when a number of changes are ready to be released, a tool such as the org.eclipse.releng.tools plugin can be used to, in a single operation, both tag the sources and release those tags into the map file.

This permits a granular approach to plugin versioning wherein only the plugins which have actually changed are renumbered, and thus incremental updates between releases is possible, and if only a single plugin changes from week to week, only that plugin will be upgraded.

This approach also allows your adopters to get more stable, weekly updates rather than hourly or ad hoc nightlies which may include incomplete changes. Sure, you can only watch for good builds in Hudson, but a more predictable schedule makes inter-build communication easier.

The only issue with this approach is that it introduces extra overhead, unless the tag&release process can be automated and run on a (weekly?) schedule. For CVS sources, there is the org.eclipse.releng.tools.tagandrelease crontab script available; for SVN, no such script (yet) exists. Want to help? See bug 264713.

Hybrid Approach

With the Athena Common Build, you can set up a Hudson job to run from tagged maps, but also override those tags on demand to force a build from head (or trunk) instead of from the specific repo tags.

To do so, pass in the following flags to EXTRAFLAGS via your build's job configuration / parameters. To build from the workspace's sources instead of fetching anything new from the repo (bug 252774):

-localSourceCheckoutDir /opt/users/hudsonbuild/.hudson/jobs/${JOB_NAME}/workspace/

To fetch from the repo, but ignore tags (bug 251926):

-forceContextQualifier -fetchTag HEAD

Or, like many projects on build.eclipse.org have done, set up two jobs: one for nightly from trunk, and one for weekly integration from maps. Then instead of doing CI builds when you remember to, they will run automatically when the repo changes so you'll have immediate feedback when something has broken. Add email, IRC, or Twitter notification to the job, and everyone else will know too.

Using Hudson for parallel Athena builds

This week I set up a multi-configuration Hudson job to try doing EMF's Query, Validation, and Transaction projects as a single job (instead of the traditional three linked jobs), then Bernd took over to do most of the work required to migrate to the new, simpler build system... and unearthed a few sticky bugs along the way. For example, Athena now supports running from jarred test plugins, as well as test plugin folders. (Yes, I'm amazed no one ever tested that use case before now either. Anyway, it works now.)

Doing these builds in parallel (rather than serially) may be a bad idea, but for now, it lets the Q,V,T developers build all three linked components via a single build button, and if the sources for any of the three changes, all three will be rebuilt and tests will be run.

Since Transaction depends on Query and Validation, this is certainly useful; however, Query and Validation do not depend on each other, so there may be some extraneous build churn with this set up.

Still, it's nice to see a the status of all three builds in one place:

What's next? Well, hooking up FindBugs, eliminating all the old SDK/runtime zips (because we have the new, shiny, p2 update site zip instead, and you can convert from repo to "runnable" SDK easily using this script), and figuring out how to make the builds use each others' newer binary output instead of building from a known entity (Query 1.4.0.N.ten.minutes.ago vs. 1.3.0.R.last.month). Luckily, Athena supports building against p2 repos and Hudson has APIs for fetching the latest good output, so this should be as simple as configuring the Transaction build to fetch from the latest successful Query zip using properties like these:

dependencyURLs=https://build.eclipse.org/hudson/.../path/to/Update.zip

Of course if you want to build against the latest PUBLISHED update, you could do this:

repositoryURLs=http://download.eclipse.org/modeling/emf/query/updates/nightly/
featureIDsToInstall=org.eclipse.emf.query.sdk

Tracking Build Status With Hudson Data APIs

A number of people have been twittering recently about Hudson Helper, and the fact that it can't (yet) support http access to Hudson servers. (There's just no pleasing some people, eh David?)

UPDATE: David reports that Hudson Helper has worked with both http and https since day one. He invites direct feedback if you're having problems.

To help fill this gap, I'd like to detail some of the handy API features of Hudson I've discovered since I first started using it back in October, which cane be fetched via http (or https) in a browser or via a script.

Datum	Example
Latest Successful build number	buildNumber
Latest Successful zip (published artifact)	GEF-Update-*.zip
All checked out Project Set Files (Hudson workspace)	*.psf
XML Digest of Latest Stable Build	lastStableBuild/api/xml
SVN revision used for Latest Stable Build	//changeSet/revision/revision

For more on the APIs available to the Latest Successful, Stable, Failed, or in fact simply the Latest Build, see:

1. /lastSuccessfulBuild/api/
2. /lastStableBuild/api/
3. /lastFailedBuild/api/
4. /lastBuild/api/

Of course, should you want details on a specific build rather than the latest, you can replace the "last*Build" token with an actual build number.

Finally, because no post about APIs should be complete with out some script showing how to exploit that interface, here's a quick example of how to fetch the latest successful, and as yet unreleased Drools 5.1 runtime library zip for use in our JBoss Tools 3.1.0.M2 builds. In this example, we fetch the build number for the last successful build and compare it to a cached version. We also fetch and cache the latest SVN revision number (in a build.properties file) so that we can later fetch Drools sources from the same point in time as the precompiled Drools binaries in the zip. This guarantees we're building from trunk, but only a good build in trunk, skipping over any failed builds or intermediate states (partial commits).

#!/bin/bash
droolsSNAPSHOTnum=drools-SNAPSHOT-num.txt
droolsSNAPSHOTrev=drools-SNAPSHOT-rev.txt
droolsSNAPSHOTzip=drools-SNAPSHOT-bin.zip
droolsSNAPSHOTurl=http://jboss-hudson-server/hudson/job/drools/lastSuccessfulBuild/artifact/trunk/target/drools-5.1.0.SNAPSHOT-bin.zip

buildNumOld=0; if [[ -f $droolsSNAPSHOTnum ]]; then buildNumOld=$(cat $droolsSNAPSHOTnum); fi
buildNumNew=$(wget -q --no-clobber -O - http://jboss-hudson-server/hudson/job/drools/lastSuccessfulBuild/buildNumber)

buildRevOld=0; if [[ -f $droolsSNAPSHOTrev ]]; then buildRevOld=$(cat $droolsSNAPSHOTrev); fi
buildRevNew=$(wget -q --no-clobber -O - http://jboss-hudson-server/hudson/job/drools/lastSuccessfulBuild/api/xml?xpath=//changeSet/revision/revision)

if [[ $buildNumNew -gt $buildNumOld ]]; then
 # get: 27013; must change to 27013 
 echo $buildRevNew > $droolsSNAPSHOTrev;
 sed -i "s#\|##g" $droolsSNAPSHOTrev 
 buildRevNew="$(cat $droolsSNAPSHOTrev)"; #echo "."$buildRevNew"."
 # replace "defaultTag=trunk:\d+" with defaultTag=trunk:${buildRevNew} in build.properties
 #  defaultSvnUrl=http://anonsvn.jboss.org/repos/labs/labs/jbossrules
 #  defaultTag=trunk:27013
 sed -i "s#defaultTag=trunk:\d\+#defaultTag=trunk:$buildRevNew#g" build.properties; # grep "defaultTag=trunk:" build.properties 

 echo $buildNumNew > $droolsSNAPSHOTnum; 
 echo "Download $droolsSNAPSHOTurl ..."
 wget -q --no-clobber -O $droolsSNAPSHOTzip $droolsSNAPSHOTurl 

 # ...

fi

Oh, and BTW, if you're ever looking for the latest hudson.war executable, it's always here.