Much ado about scripting, Linux & Eclipse: card subject to change

2010-03-04

Athena: p2 repos w/ associate sites

Want to insert associate update sites into your user's Eclipse when they go to install your features from an update site/zip? With an Athena build this is now a trivial exercise. For example... say you're building a plugin which supports integration with Subclipse or Subversive. In a traditional install from an SDK or runtime zip, you'd unpack into eclipse/, and wonder why the new plugins didn't work. In the new p2 install world, you point Eclipse at the update site or zip, and are told that you cannot install the SVN integration because missing requirements cannot be found. So you hunt down the required sites, add them to Eclipse, and try again; this time you are allowed to install the new features. To improve the ease of use for the end user, Athena now allows you to define associate.sites in your build.properties. If this property is set, URLs will be added into your p2 repo's metadata and Eclipse will automatically add those sites when it scans your zip or site. No more manual copy+paste from documentation needed!

associate.sites=\
http://subclipse.tigris.org/update_1.6.x,\
http://download.eclipse.org/technology/subversive/0.7/update-site/,\
http://community.polarion.com/projects/subversive/download/eclipse/2.0/update-site/
And, of course, you can reuse these sites when resolving dependencies against which to build or test:
repositoryURLs=\
http://download.eclipse.org/releases/galileo/,\
http://download.eclipse.org/athena/repos/eclipse-Update-3.5.2-201002111343.zip,\
http://download.eclipse.org/eclipse/updates/3.5/,\
${associate.sites}

IUsToInstall=org.eclipse.sdk.feature.group+org.eclipse.sdk.ide+\
org.eclipse.team.svn+org.eclipse.team.svn.core+org.eclipse.team.svn.ui+\
org.tigris.subversion.subclipse.core+org.tigris.subversion.subclipse.ui

2010-02-18

p2 Repository Association And The Fine Art Of Forceably Enabling Disabled Sites

This week I've spent some time exploring how to associate p2 repo sites with other sites. The reason for this is that we publish features which depend on other publishers' features, and so would like to be able to have our update site automatically enable sites on which we depend in the end user's Eclipse.

The first attempt to do this was by using p2.inf files. Based on examples on blogs and in newsgroups, I added this p2.inf file into our BIRT Integration feature:

instructions.configure=addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:0,name:Eclipse Galileo,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:1,name:Eclipse Galileo,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:0,name:Eclipse 3.5,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:1,name:Eclipse 3.5,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:0,name:BIRT 2.5,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:1,name:BIRT 2.5,enabled:true);
Unfortunately, this didn't work - I got this error while building:

 java.lang.IllegalArgumentException: No action found for: addRepository.

So, hoping that using a fully-qualified name for the action would help, I tried this:

instructions.configure=org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:0,name:Eclipse Galileo,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:1,name:Eclipse Galileo,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:0,name:Eclipse 3.5,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:1,name:Eclipse 3.5,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:0,name:BIRT 2.5,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:1,name:BIRT 2.5,enabled:true);

Success! The build no longer complained. Of course I wasn't getting any associate sites enabled when I added the new repo into Eclipse. Why, I wondered?

Thankfully, I was told that the now-deprecated Metadata Generator (org.eclipse.equinox.p2.metadata.generator.EclipseGenerator) doesn't support p2.inf files, so it was finally time to switch to Eclipse 3.5's Update Site Publisher (org.eclipse.equinox.p2.publisher.UpdateSitePublisher), which can be run like this as a drop-in replacement for the above application:

<echo>Run p2.publisher.UpdateSitePublisher using launcherjar = @{launcherjar}</echo>
<java jar="@{launcherjar}"
      fork="true" timeout="10800000"
      jvm="${java.home}/bin/java"
      failonerror="false" maxmemory="256m" taskname="p2">
 <classpath>
  <fileset dir="${builder.build.path}/plugins"
           includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
  />
  <fileset dir="${clean.eclipse.home}/plugins"
           includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
  />
  <pathelement location="${builder.build.path}/plugins" />
  <pathelement location="${clean.eclipse.home}/plugins" />
 </classpath>
 <arg line=" org.eclipse.equinox.launcher.Main -application org.eclipse.equinox.p2.publisher.UpdateSitePublisher" />
 <arg line=" -metadataRepository file:${updateSiteJarDir}/ -metadataRepositoryName "${update.site.product.name} ${update.site.description} Update Site"" />
 <arg line=" -artifactRepository file:${updateSiteJarDir}/ -artifactRepositoryName "${update.site.product.name} ${update.site.description} Artifacts"" />
 <arg line=" -source ${updateSiteJarDir}/" />
 <arg line=" -compress -publishArtifacts -reusePack200Files -configs *,*,*" />
</java>

Having switched out the deprecated app for its successor, I then discovered that using the instructions.configure touchpoint was way too late in the process to be of any use; this information is only used when you install the feature, not when you poll the repo for metadata. So to add the BIRT 2.5 update site I'd have to install the BIRT integration feature... which depends on the BIRT 2.5 site. I needed to inject this requirement in at an earlier point in the process.

So, the next step was to hit up #equinox-dev and ask for help, which I got from Pascal "LeNettoyeur" Rapicault, fellow IBM alumnus. He suggested that I append into the content.xml file in my repo the following information. Note that type 0 and 1 represent metadata repo and artifact repo, and options=1 or 0 is used to mark the site enabled or disabled.

  <references size='2'>
    <repository uri='http://download.eclipse.org/birt/update-site/2.5/'
url='http://download.eclipse.org/birt/update-site/2.5/' type='0' options='1'/>
    <repository uri='http://download.eclipse.org/birt/update-site/2.5/'
url='http://download.eclipse.org/birt/update-site/2.5/' type='1' options='1'/>
  </references>
</repository>

Success, I again thought, since the associated sites were finally being added and enabled into Eclipse when the repo was scanned for metadata. I even scripted this process so that it could be integrated into our build process, or simply reused, eg., to merge two update sites and add some associate site for optional feature installs.

But alas, every step forward brings another step backward, as I discovered that available but disabled sites are NOT enabled despite the instructions in the repo. In other words, I cannot force an existing, yet disabled site to become enabled because my repo requires it. I can only document the requirement and hope that users will RTFM without getting too annoyed.

So, what's the solution? Well, there currently isn't one. But if this scenario seems like something you'd like to see improved, please cast your votes & comments in one of these bugs:

2010-02-16

Ontario's capital now in the U.S.A.


Air Canada does not permit flight searches which have both an origin and destination in the United States.
-- When did Toronto (YYZ) move to the US? I suspect this is why Air Canada keeps needing federal bail out money: geography lessons.

2010-01-18

Monitor fail

Well, that's a week of my life I'll never have again. Spent most of last week playing with WinXP, LinuxMint 8, and Fedora 11 & 12 (both 32- and 64-bit) trying to find a solution to my external display issues.

Turns out that it's a software issue: the latest xorg intel driver has a bug.

Humorous Pictures

2010-01-12

Beyond Compare 3 for 64-bit ubuntu linux

Last week, I switched my xubuntu 8.04 (Hardy Heron) box to Linux Mint 8 (Helena), to see how things had evolved in the past 18 months. Unfortunately, I also decided to switch from 32- to 64-bit, and of course not everything works the way it used to.

Luckily, the good folks at Scooter Software pointed me to this hack to allow their 32-bit Linux flavour of Beyond Compare 3 to find the 32-bit libraries it needs to run.

So, here's an updated version of that hack for ubuntu 9.10 Karmic Koala or Linux Mint 8 Helena:

  1. Download the i386 libqt3-mt .deb package into /tmp
  2. Download the latest Beyond Compare 3 .deb package into /tmp
  3. $ sudo su
  4. apt-get install ia32-libs
    cd /tmp; dpkg-deb --extract /tmp/libqt3-mt*.deb libqt3-mt
    cp -R libqt3-mt/usr/lib/* /usr/lib32; rm -fr /tmp/libqt3-mt
    dpkg -i --force-architecture /tmp/BCompareLinux*.deb

2010-01-05

PDE Best Practices : A Noob's Guide To Plugin & Build Health

I've drafted a collection of tips for keeping your plugins healthy and your builds green.

Feel free to add to, change, or comment on the list here:

http://wiki.eclipse.org/Common_Build_Infrastructure/Getting_Started/FAQ/Plugin_Health_And_Best_Practices

Switching monitors with xrandr

Sure, there are applets and other GUI tools to do this, but nothing beats the purity of going commando, er, commandline. :) Thanks to this article for the inspiration.

# reset 
xrandr --auto

# turn off laptop screen
xrandr --output LVDS --off

# 24" 16x12 on
xrandr --output VGA --mode 1600x1200

# reset 
xrandr --auto

# turn off external screen
xrandr --output VGA --off

# laptop screen on
xrandr --output LVDS --mode 1280x800

UPDATE, 2010/01/05:

To enable both screens and force the larger one to include the taskbar:

# reset 
xrandr --auto

# laptop screen on
xrandr --output LVDS1 --mode 1280x800 --noprimary --pos 1600x0

# 24" 16x12 on
xrandr --output VGA1 --mode 1600x1200 --primary --pos 0x0

# echo current status
xrandr --current

2009-12-31

Discount Fail

I'm having a problem w/ a purchase.

I added a Blackberry Bold compatible 2400mAh battery to my cart, on sale for 20% off ($24.07).

On page two, the price was still 20% discounted:

On page three of the cart, the price jumped back up to the full $30.09 price.

So... is this item on sale, or not?

2009-12-04

Feeding The Right Sources To Your Builds

I was recently asked the best approach for how to control the input to one's builds, and while there is no single solution to this that fits all projects, here are two ways you might want to go.

Freeze Then Branch

The current approach for JBoss Tools development involves continuous nightly builds from trunk until such time as the project is deemed frozen in prep for an upcoming milestone or release candidate. At that time, the trunk is copied to branches/JBossTools-3.1.0.M4, and very little is done to that branch - only urgent tweaks and fixes. A new Hudson job is cloned from the current one, and adjusted to point at the branched sources instead of trunk. The maps are also updated to point at the new source location in SVN.

This allows more nightly builds towards an upcoming stable development milestone, while new development can continue in parallel in trunk. When the milestone is released, the unneeded job is disabled or removed.

The only issue with this approach is that all plugins, built from the head of a branch (or trunk) are re-versioned w/ the latest timestamp every time they're compiled. Thus upgrading from one 80M update to the next requires another 80M download. To mitigate this, milestones are released only once every month or two.

Tag And Release

The current approach for numerous Eclipse.org projects, such as GEF, is to develop continuously in HEAD and when a number of changes are ready to be released, a tool such as the org.eclipse.releng.tools plugin can be used to, in a single operation, both tag the sources and release those tags into the map file.

This permits a granular approach to plugin versioning wherein only the plugins which have actually changed are renumbered, and thus incremental updates between releases is possible, and if only a single plugin changes from week to week, only that plugin will be upgraded.

This approach also allows your adopters to get more stable, weekly updates rather than hourly or ad hoc nightlies which may include incomplete changes. Sure, you can only watch for good builds in Hudson, but a more predictable schedule makes inter-build communication easier.

The only issue with this approach is that it introduces extra overhead, unless the tag&release process can be automated and run on a (weekly?) schedule. For CVS sources, there is the org.eclipse.releng.tools.tagandrelease crontab script available; for SVN, no such script (yet) exists. Want to help? See bug 264713.

Hybrid Approach

With the Athena Common Build, you can set up a Hudson job to run from tagged maps, but also override those tags on demand to force a build from head (or trunk) instead of from the specific repo tags.

To do so, pass in the following flags to EXTRAFLAGS via your build's job configuration / parameters. To build from the workspace's sources instead of fetching anything new from the repo (bug 252774):

-localSourceCheckoutDir /opt/users/hudsonbuild/.hudson/jobs/${JOB_NAME}/workspace/

To fetch from the repo, but ignore tags (bug 251926):

-forceContextQualifier -fetchTag HEAD

Or, like many projects on build.eclipse.org have done, set up two jobs: one for nightly from trunk, and one for weekly integration from maps. Then instead of doing CI builds when you remember to, they will run automatically when the repo changes so you'll have immediate feedback when something has broken. Add email, IRC, or Twitter notification to the job, and everyone else will know too.

2009-11-27

Dash Athena: More Ant, More Tests, More Repos!

  • Infrastructure Changes
  • Cross-Platform / Ease of Use
  • bug 295670 Support running JUnit tests run from Ant script instead of Bash - tests can now be run on Linux and Windows (and to a lesser extent on Mac OSX) using new testLocal build step
  • Bug Fixes
  • bug 292486 Allow builds to fail if unit tests fail using failBuildIfTestFailuresOrErrors=true
  • bug 294678 Categories don't show up with IBM 1.6 JDK - implemented workaround so a different JDK can be used for p2 operations than for compilation of sources: use PACK200_JAVA_HOME=/path/to/different/JDK
  • bug 295773 Non-incubating projects no longer need to set "incubation=" in their build.properties
  • bug 292235 Included pre-compiled binary features/plugins are now included by default using PDE's runPackager property; can disable this behaviour with packageFeaturesIncludedBinaries=false
  • Documentation
  • Testing - Different ways to run or re-run tests, including as a secondary process after a build (so they can be run on a different platform or by a different user)
p2 does not natively support remote repo zips; to workaround this the zip is fetched and the URL is rewritten from http://path/to/foo.zip to jar:file:/tmp/path/foo.zip!/

Previous New & Noteworthy | More Athena Docs