Hudson & Virtual Box: Creating a common share with SSHFS

I've been working a recently with my own Hudson server, setting up Fedora and OpenSolaris images as slave nodes to prototype new ways to build. Tonight I found a very handy trick for mounting shared drives using SSHFS. Kudos to the man page writers -- all the tips I found via Google were outdated or just plain wrong. man sshfs to the rescue!

Step 1: create a folder on the Hudson master / Virtual Box host with files you want to share, eg., /shared

Step 2: create an empty folder on each Hudson slave / Virtual Box guest as a mount point for the shared folder, eg., /shared

Step 3: on the guest, install sshfs, eg., sudo yum install fuse-sshfs.

Step 4: mount the share: sshfs user@hudson-master-host.your-domain.com:/shared /shared

Of course you'll probably want to set up ssh keys and add this to your startup scripts so that the mount happens automatically when the guest is launched, but I'll leave that for next time.

Athena Common Builder: Thanks to the early adopters!

Here's a quick list of some of the projects @ Eclipse.org currently using Athena. If you haven't tried Athena, maybe it's time!

1. Linux Tools
2. Visual Editor (VE)
3. Voice Tools (VTP)
4. PDE Declarative Services Modeling Incubator (pde.ds.modeling.incubator)
5. Nebula Widgets Gallery
6. Faceted Project Framework (fproj)
7. EMF Query (EMF MQ)
8. EMF Validation (EMF VF)
9. EMF Transaction (EMF MT)
10. Ajax Tools Framework (ATF)

And @ JBoss.org, all available in JBoss Tools 3.1.0.M2 for Eclipse 3.5 (Galileo) ...

1. JMX Console
2. BPEL Editor*
3. jBPM 3 and 4
4. FreeMarker
5. ...to name but a few.

* - will be available in M3 or you can get a nightly build here.

What are people saying about Athena?

"[I] really like the new build. It is much less confusing then the old [Common Modeling] one."

"[C]ongrats for this builder. It is quite good and I'm eager to rely fully on it."

"We are playing around with Athena and finding it really useful. It is already deployed to one of our customer developments. Thanks a lot for the hard work! We will definitely try to contribute back anything useful we will have in the process."

So, how do *you* get started? Here's a FAQ. Here's our New and Noteworthy. And here's all the rest of the knowledge base articles.

Oh, and I'd be remiss if I didn't mention that there's a big list of requested features still waiting for contributions. If you use Athena, you're already half-way to contributing back. Want to help? Drop me a line any time and we can discuss what holes need filling that match your skills and most directly improve your use of PDE, p2 and Athena.

Using Hudson for parallel Athena builds

This week I set up a multi-configuration Hudson job to try doing EMF's Query, Validation, and Transaction projects as a single job (instead of the traditional three linked jobs), then Bernd took over to do most of the work required to migrate to the new, simpler build system... and unearthed a few sticky bugs along the way. For example, Athena now supports running from jarred test plugins, as well as test plugin folders. (Yes, I'm amazed no one ever tested that use case before now either. Anyway, it works now.)

Doing these builds in parallel (rather than serially) may be a bad idea, but for now, it lets the Q,V,T developers build all three linked components via a single build button, and if the sources for any of the three changes, all three will be rebuilt and tests will be run.

Since Transaction depends on Query and Validation, this is certainly useful; however, Query and Validation do not depend on each other, so there may be some extraneous build churn with this set up.

Still, it's nice to see a the status of all three builds in one place:

What's next? Well, hooking up FindBugs, eliminating all the old SDK/runtime zips (because we have the new, shiny, p2 update site zip instead, and you can convert from repo to "runnable" SDK easily using this script), and figuring out how to make the builds use each others' newer binary output instead of building from a known entity (Query 1.4.0.N.ten.minutes.ago vs. 1.3.0.R.last.month). Luckily, Athena supports building against p2 repos and Hudson has APIs for fetching the latest good output, so this should be as simple as configuring the Transaction build to fetch from the latest successful Query zip using properties like these:

dependencyURLs=https://build.eclipse.org/hudson/.../path/to/Update.zip

Of course if you want to build against the latest PUBLISHED update, you could do this:

repositoryURLs=http://download.eclipse.org/modeling/emf/query/updates/nightly/
featureIDsToInstall=org.eclipse.emf.query.sdk

My love-hate with SVN, Part 7: Setting svn:ignore properties

If you've inherited code where someone has accidentally committed *.class files, here's how you can find those bin/ folders, wipe them out, and hopefully never see them in the Synchronize view again.

Sure, you could use Team > Add to svn:ignore... on a file or Team > Set Property... on a folder in Eclipse using Subversive, but this is faster and less clickity-click.

for d in $(find -maxdepth 1 -mindepth 1 -type d ~/workspace/ -name "org.eclipse.*"); do
  pushd $d
  svn up
  svn propset svn:ignore "bin
bin/*" .
  svn propget svn:ignore . 
  svn propset svn:ignore "*.class
*/*.class
**/*.class" bin
  svn propget svn:ignore bin
  cd bin
  rm -fr *
  svn delete org
  svn delete model
  # find other deleteables with `svn status`
  cd ..
  svn ci -m "svn:ignore"
  popd
done

HOWTO: generate .m3u playlist from .mp3 directory

Been trying to find a solution to this one for ages. Turns out it's stupidly simple - just dump the results of a find into a file.

#!/bin/bash
dir="$1"
echo "Create playlist for $1 ..."
if [[ $2 ]]; then list="$2"; else list="$1"; fi

pushd "$dir" 2>&1 >/dev/null
find . -type f -name "*.mp3" > "$list.m3u"
echo "Found these files:"
cat "$list.m3u"
popd 2>&1 >/dev/null

With or without the "./" prefix on each found file, the resulting .m3u files work on my Blackberry (and are found automatically), including nested folders and paths with spaces. To run the above, just do this:

$ ./m3u.sh "artist folder/disc folder" "playlist name"

My love-hate with SVN, Part 6: Installation Ease Of Use

For months I've been annoyed by the fact that installation of Subversive (or Subclipse) requires fetching features and plugins from 3 or more update sites. No more!

Today, as an exercise to learn how to use the <p2.mirror/> task and provide a reproduceable, offline way to get Subversive into a virtual machine, I've created an update site zip, complete with site.xml and p2 metadata, which can be used to install Subversive from a single source. Here's the Ant script if you'd like to try this at home.

Because let's be real: you can only complain so long before it's time to roll up your sleeves and pitch in, right? That's how open source survives - thanks to people who care enough to complain AND care enough to help.

Here's the 13M update site zip, which includes the following:

Subversive 0.7.8
SVN Connector 2.2.1
SVNKit 1.2.3
JNA 3.0.9
ECF 3.0.0

Any problems, please report them in bug 284077.

HOWTO: Burn ISO image to DVD w/ linux commandline

When burning an ISO image to disc, I simply use this:

$ growisofs -dvd-compat -speed=1 -Z /dev/dvd=disc.iso

Using -speed=1 takes a little longer than the default "as fast as possible" mode, but guarantees the disc can be properly read in the pickiest of drives (eg., a Wii DVD drive). DVD-R (not DVD+R) is also recommended.

HOWTO: Be full of C.R.A.P.

In part 1 I rambled on at length about what I think needs to be done to prove yourself to a project team in order to become a committer on tht project.

So, what the crap's up with being full of C.R.A.P.? I'm not referring to the four principles of design (Contrast, Repetition, Alignment, Proximity), though there are some similarities here.

For me, being full of C.R.A.P is about the transition from one state to another:

Contributing Responsive Able Peer

to

Committed Responsible Active Participant

How do you move from one state to another?

Simple, right?

'Till next time...

HOWTO: Becoming an open source project committer

The Tweetosphere/blogosphere has been buzzing with discussions about what one needs to do to be a committer @ Eclipse.org.

I got my rights by working for IBM and being handed the keys to the Porsche when I started working at the Toronto Lab as a member of the EMF team, oh so many lunar eclipses ago. No longer with IBM, I'll retain my committerships until I manually ask to be removed, or they claw 'em from my cold dead hands. After all, what's a revised patch but a 2nd Amendment? (Aside: seriously, people, it's 2009. You don't need a gun. There's no Imperial Army coming to steal your land. LET IT GO.)

For most committers, however, you can't just be appointed to the job; you have to earn it. So, here are my tips for getting on *my* project, the Athena Common Build.

1. Easiest way to get on the project: be invited by someone already on the team by personal recommendation (see criteria below). Others can +1/-1 the suggestion based on the criteria below, but in my experience with other projects, no one ever vetoes a nomination. (I've seen it once, and it only delayed that person's committership by about a month.) So cozy up to the existing committers, and you're in. Why is this? Because it's OPEN source, and how can you be open if you exclude people who want to contribute?
2. The nominee must use the project at least weekly, if not daily. For Athena, this means you have to be actively writing Ant scripts, doing builds, or at least be active in PDE or p2 development. Why is this important?
   a) I don't want "dump and run" code which I'll then have to maintain, and
   b) if you're not a user, you can't intelligently decide what pains exist and which are important to solve
3. I'd like to see two accepted patches to prove you've got the technical skill, and that you're willing to thrown down and help with existing known issues - see 2 (b) above.
4. If you're not technical (or not *yet* technical), then you need demonstrated skills or commitment, or have worked in a related field with someone mutually known who can vouch for you.

So, what constitutes "commitment?" Lots of things...

• show up to meetings
• comment on or write bugs, blogs, wiki, articles, recipes, HOWTOs, newsgroup, mailing list, IRC
• submits patches or test cases
• help triage bugs
• mentor students (GSoc or other)
• runs contests, does viral marketing, etc.

Now, of course, these items are not all mesaurable, but if people know you're involved, and you'd like to be a committer, you'll likely be voted in. (Many people trying out Athena may have noticed I've offered them committer rights in exchange for code or doc contributions. So far, no takers, but the offer stands.)

Frankly, I'd rather have more people as committers who do little to the code base but who have the power to do so when needed. For example, (if the data is accurate) Kim's only committed 48 LOC in the past 9 months, compared to my 80,000 LOC (seriously, that can't be right) - but what she, Andrew and Andrew have done has been invaluable. And, often much more valuable, they've all helped out with with advice in bugs. Thanks!

Good planning trumps code any day.

Continued in part 2

Tracking Build Status With Hudson Data APIs

A number of people have been twittering recently about Hudson Helper, and the fact that it can't (yet) support http access to Hudson servers. (There's just no pleasing some people, eh David?)

UPDATE: David reports that Hudson Helper has worked with both http and https since day one. He invites direct feedback if you're having problems.

To help fill this gap, I'd like to detail some of the handy API features of Hudson I've discovered since I first started using it back in October, which cane be fetched via http (or https) in a browser or via a script.

Datum	Example
Latest Successful build number	buildNumber
Latest Successful zip (published artifact)	GEF-Update-*.zip
All checked out Project Set Files (Hudson workspace)	*.psf
XML Digest of Latest Stable Build	lastStableBuild/api/xml
SVN revision used for Latest Stable Build	//changeSet/revision/revision

For more on the APIs available to the Latest Successful, Stable, Failed, or in fact simply the Latest Build, see:

1. /lastSuccessfulBuild/api/
2. /lastStableBuild/api/
3. /lastFailedBuild/api/
4. /lastBuild/api/

Of course, should you want details on a specific build rather than the latest, you can replace the "last*Build" token with an actual build number.

Finally, because no post about APIs should be complete with out some script showing how to exploit that interface, here's a quick example of how to fetch the latest successful, and as yet unreleased Drools 5.1 runtime library zip for use in our JBoss Tools 3.1.0.M2 builds. In this example, we fetch the build number for the last successful build and compare it to a cached version. We also fetch and cache the latest SVN revision number (in a build.properties file) so that we can later fetch Drools sources from the same point in time as the precompiled Drools binaries in the zip. This guarantees we're building from trunk, but only a good build in trunk, skipping over any failed builds or intermediate states (partial commits).

#!/bin/bash
droolsSNAPSHOTnum=drools-SNAPSHOT-num.txt
droolsSNAPSHOTrev=drools-SNAPSHOT-rev.txt
droolsSNAPSHOTzip=drools-SNAPSHOT-bin.zip
droolsSNAPSHOTurl=http://jboss-hudson-server/hudson/job/drools/lastSuccessfulBuild/artifact/trunk/target/drools-5.1.0.SNAPSHOT-bin.zip

buildNumOld=0; if [[ -f $droolsSNAPSHOTnum ]]; then buildNumOld=$(cat $droolsSNAPSHOTnum); fi
buildNumNew=$(wget -q --no-clobber -O - http://jboss-hudson-server/hudson/job/drools/lastSuccessfulBuild/buildNumber)

buildRevOld=0; if [[ -f $droolsSNAPSHOTrev ]]; then buildRevOld=$(cat $droolsSNAPSHOTrev); fi
buildRevNew=$(wget -q --no-clobber -O - http://jboss-hudson-server/hudson/job/drools/lastSuccessfulBuild/api/xml?xpath=//changeSet/revision/revision)

if [[ $buildNumNew -gt $buildNumOld ]]; then
 # get: 27013; must change to 27013 
 echo $buildRevNew > $droolsSNAPSHOTrev;
 sed -i "s#\|##g" $droolsSNAPSHOTrev 
 buildRevNew="$(cat $droolsSNAPSHOTrev)"; #echo "."$buildRevNew"."
 # replace "defaultTag=trunk:\d+" with defaultTag=trunk:${buildRevNew} in build.properties
 #  defaultSvnUrl=http://anonsvn.jboss.org/repos/labs/labs/jbossrules
 #  defaultTag=trunk:27013
 sed -i "s#defaultTag=trunk:\d\+#defaultTag=trunk:$buildRevNew#g" build.properties; # grep "defaultTag=trunk:" build.properties 

 echo $buildNumNew > $droolsSNAPSHOTnum; 
 echo "Download $droolsSNAPSHOTurl ..."
 wget -q --no-clobber -O $droolsSNAPSHOTzip $droolsSNAPSHOTurl 

 # ...

fi

Oh, and BTW, if you're ever looking for the latest hudson.war executable, it's always here.

Workin' For The Wiikend

After acquiring my first DriveKey-powered "try before you buy" Wii game via torrent (and having a little fun fighting the Joker's minions off while occasionally blowing Robin into his component bricks with a well-placed BatBomb), I decided tonight to do a little more hacking. Thanks, CanadaMods.ca!

So, with the wife out watching some chick-flick w/ a friend, I got to spend a few hours playing with the HomeBrew Channel on my Wii. Very cool stuff available, from game emulators & ports, to new games, media players, and utilities. Complete list here.

To set up the HomeBrew Channel, follow these steps, including installation of the DVDx application so your Wii can play video DVDs.

Then, install the HomeBrew Browser, and grab some more software. After numerous tests, crashes, and reboots, I found that the best three options for playing video are these, all available through the HomeBrew Browser or via manual download from wiibrew.org.

Here's what I tested:

App	2G SD card w/ .mp3	Bus-powered 2.5" 500G USB drive w/ .avi	DVD-R w/ .avi	DVD-R Video DVD (burned w/ growisofs from dl'd .avi torrent) [1]	Video DVD (original, possibly DVD-DL? or DVD+R)
GeeXboX (embedded linux)	Y	Y	N	N	N
MPlayer CE	Y	Y	Y	Y	N
MPlayer TT			Y	Y	N

So, while I have scripted the process for easily converting .avi to DVD, I now no longer need to do so -- I can just plug my USB drive directly into the Wii and watch it on the big screen w/o having to waste hours in format conversion. Wii!

E-Fail

It's taken a while, but I've managed to get some metrics for how much mail I actually process.

Here's my inbox 3 weeks ago before I went on vacation for a week, then went without VPN access for a few days. The xkcd strip is particularly appropos.

Here's that same inbox today, sporting a newer version of Thunderbird. Note the pileup of over 1,000 emails in three weeks, in just ONE of the mailing list filter/folders I monitor.

So, other than filtering by sender & subject, automatically marking my own mailing list replies read, colourizing emails to make the more important ones stand out, and using "Show Unread Threads" view filtering ... what else can one do to manage the deluge?

Does anyone have any good, realistic strategies for dealing with 1000s of emails a month?

Simplified Win XP Pro EULA

http://www.microsoft.com/windowsxp/eula/pro.mspx
-- Reminds me of the WTFPL license...

Mac OS X - VPN vs. LAN: DNS Royal Rumble

I've been "sharing the Mac experience" for the past day trying to get access to my local LAN and VPN concurrently. So far, it's only one or the other, but never both at the same time.

I've tried the Cisco client, the Shimo client, vpnc (compiled from scratch with and without openssl support), vpnc 0.5.3 from DarwinPorts, and even this custom bit of script I wrote based on some tips about using scutil.

#!/bin/bash
# goal here is to collect the DNS entries from the active services and merge them into the Global list

tmpfile=$(mktemp)

# get IPs from services using scutil
function getIPs ()
{
        return_IPs=""
        keys=$(echo "list State:/Network/"$1 | scutil | awk '{print $4}')
        for f in $keys; do
                echo "> show $f"
                printf "get "$f"\nshow "$f | scutil | grep "\."
                echo "show $f" | scutil 2>&1 | grep "\." 2>&1 | \
                  awk '{print $3}' 2>&1 >> $tmpfile
        done
        #cat $tmpfile
        IPlist=$(cat $tmpfile | sort -r 2>&1 | uniq 2>&1)
        for i in $IPlist; do
                return_IPs=$return_IPs" "$i
        done
        #echo $return_IPs
        rm -fr $tmpfile
}

function setIPs ()
{
        label="$1"
        IPs="$2"; # echo $IPs
        printf "get State:/Network/$label\nd.add ServerAddresses *$IPs\nset State:/Network/$label" | scutil
        echo "> show State:/Network/"$label
        printf "get State:/Network/"$label"\nshow State:/Network/"$label | \
          scutil | grep "\."
}

echo "--- BEFORE ---"
getIPs "Service/.+/DNS"
IPs=$return_IPs

echo ""; echo "--- AFTER ---"
setIPs "Service/com.cisco.VPN/DNS" "$IPs"
setIPs "Global/DNS" "$IPs"

mv /etc/resolv.conf /etc/resolv.conf.bak
for i in $IPs; do echo "nameserver $i" >> /etc/resolv.conf; done

# ./dnsfix.sh 
--- BEFORE ---
> show State:/Network/Service/F1C45B82-45A1-4F44-89AC-82102F187F0B/DNS
    0 : 192.168.x.y
> show State:/Network/Service/com.cisco.VPN/DNS
    0 : a.b.c.d
    1 : e.f.g.h

--- AFTER ---
> show State:/Network/Service/com.cisco.VPN/DNS
    0 : 192.168.x.y
    1 : a.b.c.d
    2 : e.f.g.h
> show State:/Network/Global/DNS
    0 : 192.168.x.y
    1 : a.b.c.d
    2 : e.f.g.h

Obviously, since it's a Mac, there's got to be a dead-simple way for this to work. Anyone know how?

Learning to Love the Mac, Part 2: Mouse Tips & Desktop Management

I have an 8-button Logitech MX500 optical mouse, and this week is the first time I've ever successfully mapped functionality to all the buttons. Windows did a reasonable job with a few of the buttons; Linux doesn't support anything beyond the first three; Mac OS X Server just gets it done.

Out of the box, my third button (scroll wheel) is mapped to the seemingly pointless Dashboard, which is a huge pain when you're used to middle-clicking to open a link in a new tab or to copy/paste text in a console. To get that functionality back, go to Applications > System Preferences > Exposé & Spaces then remove Mouse Button 3 from the Dashboard's "Hide and Show" feature.

Next, I set Mouse buttons 5, 6, and 8 to All windows, Application windows, and Show Desktop.

But even cooler than these is Spaces, though as yet I can't find a way to replicate Gnome or XFCE's ability to move windows from from Space to Space which lets you drag open app windows from Space to Space which in the Spaces view (F8). Still, having up to 16 virtual desktops is very handy, particularly when you need to virtualize Windows and Linux. If you want to be able to have console windows on all Spaces rather than having them all collected on a single Space, uncheck the "When switching to an application, switch to a space with open windows for the application".

My love-hate with SVN, Part 5: Fedora 11 + Eclipse 3.5 + Subversion 1.6

Finally figured out how to make Eclipse 3.5 play nicely on Fedora 11 w/ Subversion, and I owe this bit of knowledge to our new MacPro. *sigh*

I also owe a great deal of gratitude to Cloudsmith for providing their Cloudsmith Galileo+ repository, which includes these features:

I still wish the version numbers would better align, in that I have to install the SVN Team Provider v0.7.8 with the SVN Connector v2.2.0 and the SVNKit 1.3.0 implementation v2.2.0 to make all this work with Subversion 1.6. Oof.

Learning to Love the Mac: 13 Tips

A month ago a very large package arrived in the mail: my first MacPro server. I at once fell in love with the case design - clean, simple, and dead-easy to take apart in order to add more drives and RAM. However, that's where the love boat ran aground.

To say it's been a gradual learning curve would be an understatement. Here are a few things I've learned over the past month of dealing with Mac hardware and OS, as well as retraining my fingers to use Mac keyboard bindings (META = Apple Key or Windows Key, depending on your keyboard).

1. Use META-TAB instead of ALT-TAB to cycle applications
2. Use META-LEFT/RIGHT instead of HOME/END to jump to start/end of a line
3. Use ALT-LEFT/RIGHT instead of CTRL-LEFT/RIGHT to jump to prev/next word on a line
4. META-A, META-X, META-C, META-V replace CTRL-A, CTRL-X, CTRL-C, CTRL-V for select all, cut, copy, & paste. META-L, META-T, META-N replace CTRL-L, CTRL-T, CTRL-N (jump to location bar, new tab, new window). But CTRL-TAB still switches tabs. However, if you have multiple Firefox windows open, there is no way to toggle between them with the keyboard. Same problem with multiple Terminal windows. META-TAB only switches between groups of applications, but not windows within an application.
5. Sometimes ESC works to dispose a dialog; sometimes only clicking the red X works.

6. Q replaces qemu, but doesn't seem to work very well for my existing vmware or Virtual Box images
7. Virtual Box rocks on Windows, Linux and Mac

8. XCode provides gcc, make, etc.
9. Fink and DarwinPorts replace Debian/Ubuntu's apt-get and Gentoo's emerge, respectively. Once XCode and DarwinPorts are installed, you can port install vpnc (to fetch deps and compile on the fly) or apt-get install curl (to fetch deps and install).
10. rEFIt replaces grub, and more or less works as I'd expect. /efi/refit/refit.conf approximately replaces /boot/grub/menu.lst at least as far as picking what partition to default-boot and how long to wait

11. Java is in /System/Library/Frameworks/JavaVM.framework/Home instead of /opt/ or /usr/lib/jvm/java
12. Subversion was easier to set up on Mac (using Fink) than on Fedora 10 (using yum), especially since there's now the Galileo+ Update Site from Cloudsmith so you don't have to download from multiple update sites to get it installed.

    However, the version of Subversion available via Fink doesn't work with projects checked out using Eclipse - seems that the commandline client (Subversion 1.4.4) and Subversive with SVNKit (SVN 1.6.1 w/ SVNKit 1.3.0.beta.r5741) are not compatible: svn: This client is too old to work with working copy '.'; please get a newer Subversion client. Using DarwinPorts to update the subversive client to 1.6.3 fixed this issue, but installed it into a different path (/opt/local/bin instead of /sw/bin or /usr/bin).

13. Eclipse looks better on Mac than on Linux; however, I recently stumbled across a great tip for making Eclipse waste less screen space under gtk on Linux. Highly recommended bit of gtk hackery - one file makes a world of difference!

Do you have any other tips for Linux or Windows people, surviving the transition to Mac OSX? Is there any way to tell OSX to use Windows or Linux keyboard defaults so I don't have to retrain myself?