Much ado about scripting, Linux & Eclipse: card subject to change

Showing posts with label tycho. Show all posts
Showing posts with label tycho. Show all posts

2011-10-26

HOWTO: Use Maven, Ant, and XSLT to scrub unwanted p2 metadata from an update site

Some time ago, I wrote about Using p2.inf to add/remove update sites. Tonight I found a simpler way to remove references in p2 metadata to external 3rd party sites.

For example, say you're repackaging some 3rd party features onto your own site, but don't want those features to provide references to the vendor's own update sites because you want to ensure that your product's site will only result in your sanctioned version being installed.

When you generate an update site, p2 pulls the information in the included features and will result in a section of references in the site's metadata that looks like this:

  <references size="6">
    <repository uri="http://download.eclipse.org/egit/updates" url="http://download.eclipse.org/egit/updates" type="0" options="0"/>
    <repository uri="http://subclipse.tigris.org/update_1.6.x" url="http://subclipse.tigris.org/update_1.6.x" type="1" options="0"/>
    <repository uri="http://download.eclipse.org/egit/updates" url="http://download.eclipse.org/egit/updates" type="1" options="0"/>
    <repository uri="http://subclipse.tigris.org/update_1.6.x" url="http://subclipse.tigris.org/update_1.6.x" type="0" options="0"/>
    <repository uri="http://eclipse.svnkit.com/1.3.x/" url="http://eclipse.svnkit.com/1.3.x/" type="0" options="0"/>
    <repository uri="http://eclipse.svnkit.com/1.3.x/" url="http://eclipse.svnkit.com/1.3.x/" type="1" options="0"/>
  </references>
To remove that, you can play with p2.inf directives, or you can simply perform an XSL transformation on the generated content.xml (inside content.jar, if your metadata is compressed) to remove the <references/> node:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
<xsl:template match="/">
        <xsl:apply-templates select="*"/>
</xsl:template>
<xsl:template match="*">
        <xsl:copy >
                <xsl:for-each select="@*">
                        <xsl:copy />
                </xsl:for-each>
                <xsl:apply-templates />
        </xsl:copy>
</xsl:template>
<xsl:template match="references" />
</xsl:stylesheet>
If you're generating your update site w/ Tycho, this transform can be called via a simple Ant script:

       <target name="remove.references">
                <!-- requires ant-contrib only if you like using if-then-else structures -->
                <if>
                        <available file="${update.site.source.dir}/content.jar" type="file" />
                        <then>
                                <unzip src="${update.site.source.dir}/content.jar" dest="${update.site.source.dir}" />
                                <delete file="${update.site.source.dir}/content.jar" />
                        </then>
                </if>
                <copy file="${update.site.source.dir}/content.xml" tofile="${update.site.source.dir}/content.old.xml" overwrite="true" />
                <xslt style="remove-references.xsl" in="${update.site.source.dir}/content.old.xml" out="${update.site.source.dir}/content.xml" />
                <zip destfile="${update.site.source.dir}/content.jar" basedir="${update.site.source.dir}" includes="content.xml" />
                <delete file="${update.site.source.dir}/content.xml" />
                <delete file="${update.site.source.dir}/content.old.xml" />
        </target>

Then, in your site's pom.xml, to call the Ant script, do this:

       <build>
                <plugins>
                        <plugin>
                                <groupId>org.apache.maven.plugins</groupId>
                                <artifactId>maven-antrun-plugin</artifactId>
                                <!-- make sure this variable is defined, eg., set to 1.3 -->
                                <version>${maven.antrun.plugin.version}</version>
                                <executions>
                                        <execution>
                                                <id>install</id>
                                                <phase>install</phase>
                                                <configuration>
                                                        <quiet>true</quiet>
                                                        <tasks>
                                                                <!-- called AFTER generating update site + zip to tweak content -->
                                                                <ant antfile="build.xml">
                                                                        <property name="SOME_ANT_VARIABLE" value="${SOME_MAVEN_VARIABLE}" />
                                                                </ant>
                                                        </tasks>
                                                </configuration>
                                                <goals>
                                                        <goal>run</goal>
                                                </goals>
                                        </execution>
                                </executions>
                                <dependencies>
                                        <!-- some dependencies your ant script might need -->
                                        <dependency>
                                                <groupId>commons-net</groupId>
                                                <artifactId>commons-net</artifactId>
                                                <version>1.4.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>org.apache.ant</groupId>
                                                <artifactId>ant</artifactId>
                                                <version>1.7.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>org.apache.ant</groupId>
                                                <artifactId>ant-nodeps</artifactId>
                                                <version>1.7.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>org.apache.ant</groupId>
                                                <artifactId>ant-trax</artifactId>
                                                <version>1.7.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>org.apache.ant</groupId>
                                                <artifactId>ant-commons-net</artifactId>
                                                <version>1.7.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>org.apache.ant</groupId>
                                                <artifactId>ant-apache-regexp</artifactId>
                                                <version>1.7.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>ant-contrib</groupId>
                                                <artifactId>ant-contrib</artifactId>
                                                <version>1.0b3</version>
                                        </dependency>
                                </dependencies>
                        </plugin>
                </plugins>
        </build>

I suppose there's probably a way to call a transform directly from Maven w/o the Ant wrapper, but this allows unpacking and repacking of the content.jar to get at the content.xml file.

2011-02-13

Simplifying The p2 Process, Part 3: Associate Sites

In Part 1 of this series, I looked at use of composite repos to provide a way of combining update sites into a single URL for ease of use and a single point of entry from which to do updates.

In Part 2, I discussed why we switched from using a collection of SDKs against which to build - using the now-deprecated brute-force "just unzip into eclipse root folder or dropins" approach - to using a single target platform update site so as to simplify maintenance and provide a reusable artifact for both build and workspace provisioning.


Now, let's look at the no-brainer that says that "less is more" when it comes to telling p2 from where to get updates, and that less effort for your user when installing is always a win.

If this sounds familar, I did blog about this briefly back in August. Since then, we've also added a quick XSLT script to remove the "Uncategorized" category that Tycho automatically adds for features which are listed in your site.xml but are not associated with a category. While this isn't strictly related to associate sites, it is about ease of use; while I applaud the desire to have everything belong to a category bucket (perhaps because the model Tycho's using requires it), the reason we'd rather hide these is to declutter the install view and not confuse people by suggesting features that won't work on their OS (eg., for which there's no XulRunner port).

But I digress...

Associate Sites

In the old days of yore, you could situate an associateSites.xml next to your site.xml in your "classic" update site, and Eclipse Update Manager would happily read that file and add those extra sites to your list of available update sites.

Then came p2, and while the old way still worked, it was no longer ideal. So, the new approach was to insert these associate sites directly into the p2 metadata for the site, content.xml and artifacts.xml (or content.jar and artifacts.jar).

This could be accomplished via a somewhat hacky appraoch - unpacking the existing metadata (content.jar) and shoehorning in the information at the bottom of the content.xml file, using an ant script (see "add.associate.sites" target, below) and a list of sites to be added:

<target name="add.associate.sites" if="associate.sites">
        <if>
                <and>
                        <!-- Defined in aggregateSite.properties -->
                        <isset property="associate.sites" />
                        <not>
                                <equals arg1="${associate.sites}" arg2="" />
                        </not>
                </and>
                <then>
                        <if>
                                <available file="${update.site.source.dir}/content.jar" type="file" />
                                <then>
                                        <unzip src="${update.site.source.dir}/content.jar" dest="${update.site.source.dir}" />
                                        <delete file="${update.site.source.dir}/content.jar" />
                                </then>
                        </if>
                        <!-- counter variable -->
                        <var name="associate.sites.0" value="" />
                        <for param="associate.site" list="${associate.sites}" delimiter=", 
">
                                <sequential>
                                        <var name="associate.sites.0" value="${associate.sites.0}00" />
                                </sequential>
                        </for>
                        <length property="associate.sites.length" string="${associate.sites.0}" />

                        <loadfile srcfile="${update.site.source.dir}/content.xml" property="content.xml">
                                <filterchain>
                                        <tailfilter lines="-1" skip="1" />
                                </filterchain>
                        </loadfile>
                        <echo file="${update.site.source.dir}/content.xml" message="${content.xml}" />
                        <echo file="${update.site.source.dir}/content.xml" append="true">  <references size='${associate.sites.length}'>
</echo>
                        <for param="associate.site" list="${associate.sites}" delimiter=", 
">
                                <sequential>
                                        <!-- insert into content.xml -->
                                        <echo file="${update.site.source.dir}/content.xml" append="true">    <repository uri='@{associate.site}' url='@{associate.site}' type='0' options='1'/>
<repository uri='@{associate.site}' url='@{associate.site}' type='1' options='1'/>
</echo>
                                </sequential>
                        </for>
                        <echo file="${update.site.source.dir}/content.xml" append="true">  </references>
</repository>
</echo>
          <!--  
    workaround for Tycho bug: uncategorized features in site.xml are put into
    "Uncategorized" category, rather than just being uncategorized (hidden) 
   -->
                 <copy file="${update.site.source.dir}/content.xml" tofile="${update.site.source.dir}/content.old.xml" overwrite="true" />
                        <xslt style="remove-uncategorized.xsl" in="${update.site.source.dir}/content.old.xml" out="${update.site.source.dir}/content.xml" />
                        <zip destfile="${update.site.source.dir}/content.jar" basedir="${update.site.source.dir}" includes="content.xml" />
                        <delete file="${update.site.source.dir}/content.xml" />
                        <delete file="${update.site.source.dir}/content.old.xml" />
                </then>
        </if>
</target>

So, now, instead of telling people to add multiple update sites to resolve missing potentially dependencies when installing, we can cause those extra sites to be automatically added at the same time they add the single URL for JBoss Tools. Now the additional sites need only be listed for reference, but no additional effort is required by the user.

BONUS HACK: to force a site that may already be listed (but disabled) to be added again, and this time definitely be enabled, you can add an extra slash into URL. Thus http://download.eclipse.org/birt/update-site/2.6 becomes http://download.eclipse.org//birt/update-site/2.6/, and as p2 sees a new site, it adds the new site (instead of ignoring it because it's already present but disabled. Again, a win.

Alternatively, you could cast an arcane spell using a p2.inf file in your feature's root folder or plugin's META-INF/ folder to add these additional, required sites... or do whatever processing you might need. I'm not sure if Tycho supports this yet, or how fully PDE supports reading this information. Got sample code? Send it to me as a comment below or via twitter to @nickboldt. Thanks!


In part 4, I'll talk a little about how to prevent your product build from getting updates from unofficial sources, and preload your product with the official sites from which to get updates. Because it's important to balance ease of use with prevention of unsupported features. SPOILER ALERT: may contain p2.inf instructions.

2011-02-09

Simplifying The p2 Process, Part 2: Target Platform Repos

In Part 1 of this series, I looked at use of composite repos to provide a way of combining update sites into a single URL for ease of use and a single point of entry from which to do updates.

Defining a Target

Now, I'd like to talk about how to escape the proliferation of zips needed to establish a target platform. For those unfamiliar with the term "target platform", it's either the installed base against which you're compiling your code, or it's the collection of things you have to install first before you can install something on top of that.

For the JBoss Tools case, we have at least 8 prereqs for installation. Here's what you had to install prior to JBoss Tools 3.1.1:

Now, admittedly, because there is also the Ganymede update site, you don't necessarily need to download and unpack all these zips in order to install JBoss Tools - instead, you need only enable the Ganymede site. (Same story for Helios and JBoss Tools 3.2.)

However, to do a reproduceable PDE-based build, you still need to create this base install. Traditionally, PDE's approach was to download and unpack these zips into the root of the Eclipse install running the build. Athena attempted to improve on this situation by allowing you to define a list of update sites and IUs (features and/or plugins) which were needed to define the platform. But it was far from portable, and hardly reusable.

Buckminster (later b3) also approached this problem by creating its own markup for defining what sites and what IUs to install, backed by an EMF model. But rather than dealing with a UI to create the model and populate it, I found it more useful to simply generate an instance of the aggregator model and then use the aggregator to fetch & install IUs. But as the aggregator is simply a wrapper for the underlying p2.mirror and p2.director tasks, you can use those directly too.

But as they say... "Don't bore us, get to the chorus!" So, here's some sample code for the various solutions for build-time provisioning.

  1. Using the buckminster aggregator (properties file) - stopped working for us w/ Eclipse 3.6, so we switched to b3

  2. Using the b3 aggregator (properties file) - stopped worked consistently due to network timeouts resolving deps & fetching IUs.

  3. Using p2.mirror - underlying p2 ant task for mirroring from one or more repos to local disk

  4. Using p2.director - underlying p2 ant task for installing IUs (from local or remote repo) into some target Eclipse

So, with these tools, you could create a p2 repo from other repos - mirroring and installing IUs as needed - and even script an installation. But was there a better way?

Target Platform Definition File

Enter the target platform definition file (.target). This file contains a list of IUs and the p2 repos from which to provision them. So, it's like a b3 aggregator model, or an Athena build.properties file, but abstracted away from the concept of a build, because it can be used for building but ALSO for provisioning a user's installed Eclipse base.

Unfortunately, the Target Platform Definition File editor in Eclipse 3.6 is less than optimal for large targets, or when your internet connection is suboptimal. So, after fighting with it for a while, filing bugs, and ultimately giving up, I went back to my handy-dandy XML editor (often just vim) to maintain it more simply. So rather than having Eclipse automatically install things based on a .target file, I revert to a workflow that actually works: installing by hand from an update site.

While Buckminster does support .target files (or so I've read), I didn't want to be dependent on it any more, preferring a more "pure" solution.

So, based on code from Peter Nehrer (@pnehrer), I then wrote an XSL transform to create a p2.mirror script from a .target file, wrapped with another Ant script (and optionally, a Maven pom.xml script).

And why might you care? Well, this .target file can be used to:

  • Provision a developer's Eclipse, using the Target Platform Definition Editor and a few clicks (when it doesn't time out)
  • Provision a developer's Eclipse via script for offline or multiple users (getting the team up to speed)

And yes, much (or all) of the above can be done w/ Buckminster and/or b3, if you like that approach.

But I prefer to create the .target as input to a build process, rather than being explicitly tied to one. So, as I noted above, if you have a .target file, you can easily generate a p2 repo, and use that repo to run downstream builds. Now, instead of having a half-dozen zips to download and unpack with every build (using the deprecated and unsupported "dropins" method) you can use a fully-p2-friendly repo site which contains everything you need to do your builds - whether you're a Hudson server or a developer working at home or offline.

Benefits

  • Unlike "a collection of zips" this single-source-site can be versioned with each release.

  • It only contains WHAT YOU ACTUALLY NEED rather than extraneous sources and doc and tangential plugins/features you don't. It's a bit like making muffins by first grinding your own flour, but at least you know there's nothing evil in that muffin mix, and you will be able to consistently reproduce the recipe every time, regardless of where you might be on teh interwebz.

  • f you're a keener / beta tester who likes to build against the latest milestone (or even a weekly integration build) of Eclipse 3.next or 4.future, you can use the script above to self-update. So, while the TP itself is a contained snapshot listing the explicit versions of feature groups needed, it can also be run in "get the latest available" mode in order to keep your TP current against some HEAD or trunk development / releases.

  • By splitting the TP out of the build, you can build it upstream. So, where in the past we had one "uberbuild" and an implied TP therein, now we have a TP build job, and it is then shared by the 34 downstream jobs which depend on it for their dependencies.

Shut up and show me the code!

# for the "foo.target" file, build a local target platform repo, fetching the latest versions and updating the .target file
$ ant -f build.xml -DtargetFile=foo.target -DuseLatest=true

# for the "bar.target" file, build a local target platform repo, but fetch only the stated versions of IUs
$ ant -f build.xml -DtargetFile=bar.target -DuseLatest=false

That's it. I also wrap the build.xml ant script w/ a pom which allows it to be called from an upstream Maven/Tycho process, but that's nothing more than just calling the script using the antrun plugin (and a few ant dependencies), like this:

<build>
        <plugins>
                <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-antrun-plugin</artifactId>
                        <version>1.6</version>
                        <executions>
                                <execution>
                                        <phase>validate</phase>
                                        <configuration>
                                                <tasks>
                                                        <ant antfile="build.xml">
                                                                <property name="targetFile" value="multiple.target" />
                                                                <!-- <property name="repoDir" value="/path/to/where/to/provision/repo"/> -->
                                                        </ant>
                                                </tasks>
                                        </configuration>
                                        <goals>
                                                <goal>run</goal>
                                        </goals>
                                </execution>
                     </executions>
                        <dependencies>
                                <dependency>
                                        <groupId>commons-net</groupId>
                                        <artifactId>commons-net</artifactId>
                                        <version>1.4.1</version>
                                </dependency>
                                <dependency>
                                        <groupId>org.apache.ant</groupId>
                                        <artifactId>ant-commons-net</artifactId>
                                        <version>1.7.1</version>
                                </dependency>
                                <dependency>
                                        <groupId>org.apache.ant</groupId>
                                        <artifactId>ant-trax</artifactId>
                                        <version>1.7.1</version>
                                </dependency>
                        </dependencies>
                </plugin>
 </plugins>
</build>

The rest of the code is here.


In part 3, I'll look back at the success we've had using associate sites instead of asking people to manually add 3rd party URLs when installing JBoss Tools. SPOILER ALERT: one URL is easier for people to use than 6.

In part 4, I'll talk a little about how to prevent your product build from getting updates from unofficial sources, and preload your product with the official sites from which to get updates. Because it's important to balance ease of use with prevention of unsupported features. SPOILER ALERT: may contain p2.inf instructions.

2011-02-01

HOWTO: Find osgi dependencies in features

Say you're trying to build something with Tycho & Maven 3 and while resolving dependencies before compilation, you're told:

[INFO] [Software being installed: 
  org.eclipse.tm.terminal.local.feature.group 0.1.0.v201006041240-10-7w312117152433, 
  Missing requirement: 
    org.eclipse.tm.terminal.local 0.1.0.v201006041322 
      requires 
        'bundle org.eclipse.cdt.core 5.2.0' 
          but it could not be found, 
  Cannot satisfy dependency: 
    org.eclipse.tm.terminal.local.feature.group 0.1.0.v201006041240-10-7w312117152433 
      depends on: 
        org.eclipse.tm.terminal.local [0.1.0.v201006041322]]

To quickly verify where this dependency is coming from, you can go look into the feature.xml for the org.eclipse.tm.terminal.local feature jar... but if you don't have it installed, this is somewhat more cumbersome; besides, you then have to unpack the jar before you can look inside it.

And maybe that feature contains a number of OTHER dependencies that you'll also need to resolve in your target platform when building. Sure, there are UI tools to do this within Eclipse, but when you're working on remote servers sometimes UI isn't available.

Workaround? Assuming you have a mirror of the update site(s) from which you're trying to resolve the dependency (eg., Helios) or can ssh to dev.eclipse.org, you can simply run a quick shell script to do the investigative work for you:

$ cd ~/downloads/releases/helios/201009240900/aggregate/; ~/bin/findDepInFeature "*tm*" cdt

./features/org.eclipse.tm.terminal.local_0.1.0.v201006041240-10-7w312117152433.jar
      <import feature="org.eclipse.cdt.platform" version="7.0.0" match="greaterOrEqual"/>
      <import plugin="org.eclipse.cdt.core" version="5.2.0" match="compatible"/>
      <import plugin="org.eclipse.core.runtime"/>
Where the script looks like this:
#!/bin/bash
# find plugins/feature deps by searching in some folder for feature jars, and searching through their feature.xml files for dependencies

# 1 - featurePattern - pattern of features to search (eg., "org.eclipse.tptp" or "\*" for all features)
# 2 - dependencyPattern  - pattern of plugins/feature deps for which to search (eg., "org.eclipse.tptp.platform.instrumentation.ui")
# 3 - location       - directory in which to search, if not "."

if [[ ! $1 ]]; then
        echo "Usage: $0 <featurePattern> <dependencyPattern> <location>"
        echo ""
        echo "Example: $0 tm.terminal cdt"
        exit 1
fi

# if no location, look in current dir (.)
if [[ $3 ]]; then location="$3"; else location="."; fi

# if no featurePattern, search all features for dependencyPattern
if [[ ! $2 ]]; then featurePattern="*"; dependencyPattern="$1"; else dependencyPattern="$2"; featurePattern="$1"; fi

rm -fr /tmp/findinfeature/; mkdir -p /tmp/findinfeature/features/
for f in $(find "$location" -type f -name "*${featurePattern}*" | egrep -v "pack.gz|source" | grep features | egrep "${featurePattern}"); do
        #echo "$f [$featurePattern, $dependencyPattern]"
        unzip -q $f -d /tmp/findinfeature/ feature.xml
        #       <import feature="org.eclipse.cdt.platform" version="7.0.0" match="greaterOrEqual"/>
        #       <import plugin="org.eclipse.cdt.core" version="5.2.0" match="compatible"/>
        if [[ ! $(cat /tmp/findinfeature/feature.xml | egrep "<import" -A3 | egrep "plugin=|feature=" -A1 -B1 | egrep "\".*${dependencyPattern}[^\"]*\"" -A1 -B1) ]]; then
                rm -fr /tmp/findinfeature/feature.xml
        else
                mv /tmp/findinfeature/feature.xml /tmp/findinfeature/${f}_feature.xml
                echo "${f}"
                cat /tmp/findinfeature/${f}_feature.xml | egrep "<import" -A3 | egrep "plugin=|feature=" -A1 -B1 | egrep "\".*${dependencyPattern}[^\"]*\"" -A1 -B1
                echo ""
        fi
        rm -fr /tmp/findinfeature/feature.xml
done

2010-11-15

HOWTO: Contributing to JBoss Tools using Git

If you'd like to use Git instead of SVN as your SCM tool of choice, here's how you can connect to the JBoss Tools SVN repo, pull down all the sources, work on them locally, then either commit changes back into the SVN repo (or submit a patch, if you're not already a committer).

The instructions below assume you have either Linux, Mac OSX, or Windows w/ cygwin. If you have none of those, YMMV.

Fetch sources from SVN

First, fetch sources from SVN using git-svn. If you don't want to check out all the components, use a subset of the components listed below. The complete list is here.

# create a directory into which to check out the JBoss Tools projects
mkdir ~/trunk; cd ~/trunk;

# fetch projects - this will take quite some time
# Committers, use http://svn.jboss.org/repos/jbosstools/trunk/
# Contributors, use http://anonsvn.jboss.org/repos/jbosstools/trunk/
for d in \
  archives as birt bpel bpmn build cdi common \
  deltacloud documentation download.jboss.org drools \
  esb examples flow freemarker gwt hibernatetools \
  jbpm jmx jsf jst maven modeshape portlet profiler \
  requirements runtime seam site smooks struts \
  tests tptp usage vpe ws xulrunner; do \
    git svn clone http://anonsvn.jboss.org/repos/jbosstools/trunk/${d};
done

Configure Eclipse

Next, fire up Eclipse Helios 3.6 for Java EE Developers.

Install the latest eGit from http://download.eclipse.org/egit/updates.

Install the latest m2eclipse from http://m2eclipse.sonatype.org/sites/m2e/ and optionally, http://m2eclipse.sonatype.org/sites/m2e-extras/.

Restart when prompted.

Import Git projects into Eclipse

Now, import Git projects into Eclipse using:

File > Import 
    Git > Projects from Git
        Click 'Add' then browse for ~/trunk/ 
        Enable [x] Look for nested repositories 
        Click 'Search', then click 'OK' when done

        Select a local repo from the list, click 'Next'
        (*) Import Existing Projects
        (*) Try to share automatically
        Click 'Next'
        Click 'Select All', then click 'Finish'

Repeat for other components you want to import. You can add each component to a working set to keep your workspace sorted by component.

Resolve missing dependencies

While the Eclipse Helios 3.6 for Java EE Developers contains most of the dependencies against which JBoss Tools must be compiled, it does not contain everything. For that, you need to install extra dependencies. There are two places to go:

  1. JBoss Tools Target Platform p2 Repo (also available as an archived update site zip for offline use) - contains all the Eclipse.org, google.com, and sonatype.org features needed to compile / install all of JBoss Tools. You can install everything, or just the pieces you need.
  2. JBoss Tools Nightly Repo (Update Site) - if you don't have all the source projects in your workspace, you can resolve dependencies against this site and install them from here. Once again, you can install everything, or just the pieces you need.

Build & run tests

With m2eclipse installed, you can simply right-click a project and select 'Run As > Maven Build (ALT-SHIFT-X, M)', which will prompt you complete a run configuration dialog. Here are the simplest options you need to set:

   Goals: clean install
   [x] Resolve Workspace artifacts

You can also run Maven to build your projects outside Eclipse, if you prefer.

If running outside Eclipse, you can run tests which are still tied to the Eclipse debugger.

Commit changes to master repo

Because there's no support yet for 'git svn rebase' or 'git svn dcommmit' you're stuck pushing changes to the master repo using the commandline. However, you can shorten the amount of typing needed using an .alias file. See below.

Use an .alias file

To avoid having to type the same git commands over and over, I use these shortcuts in my ~/.alias file:

# update local git-svn repo from master SVN repo
alias   gitup='for d in $(find . -maxdepth 1 -type d); do cd $d; echo $d; if [[ -d .git ]]; then git svn rebase; fi; cd -; done'

# Push local changes to master SVN repo
alias   gp='git svn rebase; git svn dcommit'

# commit local changes to local git-svn repo
alias   ci='git commit -m'

# check status of local git-svn repo
alias   stat='git status'

So, after committing changes (with eGit or via commandline) I can push those to the master SVN repo w/ a simple 'gp'. If your shell doesn't read the .alias file, make sure your .bashrc loads the file using one of these commands:

source /home/yourUserName/.alias
. /home/yourUserName/.alias
Or, put them directly in your .bashrc file.

2010-11-03

My First Maven Plugin

Having been working with Maven and Tycho for about the last 8 months, it was high time I got around to writing my first Maven plugin, because I need to break myself from my tendency to revert to ant and bash whenever I need to do something outside the normal Maven flow.

So, first I imported the git sources for Tycho 0.11.0-SNAPSHOT from https://github.com/sonatype/sonatype-tycho so I'd have something from which to learn. Sure, Hello World samples are nice, but a working example is always more fruitful.

git clone http://github.com/sonatype/sonatype-tycho.git

Next, I imported a few Tycho projects (tycho-p2-facade, tycho-p2-plugin, tycho-p2-publisher-plugin) from the git clone folder into Eclipse using m2eclipse 0.10.2. Because these are Maven projects and don't need Eclipse project clutter, there are no .project files and therefore they couldn't be imported as pre-existing projects.

So, I had to wobble through the New > Maven Project wizard a little until I figured out that I could check the 'Create a simple project (skip archetype selection)' box to create a new .project file in an existing project folder. Only wrinkle here is that if there's an existing pom.xml in a project, the New Maven Project wizard complains. The obvious workaround is to rename the existing pom.xml to pom.xml_, create the new Maven Project in w/ m2eclipse in the existing folder (as above), then replace the pom.xml that m2eclipse creates with the actual (renamed) one, pom.xml_ and refresh the project in Eclipse. Voila! I also updated the project settings to use JDK5 instead of the default JDK1.4. You need only do File > Import > Maven > Existing Maven Projects and browse for the pom files. So much easier than what I did above. Thanks to everyone for setting me straight!

So, with sample code to clone from it was time to create my own plugin project, then a second project on which that plugin depended in order to test the dependency chain thru Maven.

The first uses the "maven-archetype-plugin" archetype v1.1; the second, the "maven-archetype-quickstart" archetype v1.1. Here are the two projects:

  1. sample-plugin
  2. sample-project

The glue between these projects is in the sample-project's pom.xml:

<dependency>
 <groupId>org.jboss.maven.plugin</groupId>
 <artifactId>sample-plugin</artifactId>
 <version>0.0.1-SNAPSHOT</version>
</dependency>
And that's seemingly it. Next... adding actual useful functionality to a plugin, then using that functionality in our JBoss Tools build's parent pom.xml.

2010-10-08

HOWTO: Find the feature that contains a plugin

Tycho is awesome.

However, like all build systems, it has its limitations.

One such limitation is that when you're building against a target platform, and something's missing, you get errors such as these:

[INFO] Cannot complete the request.  Generating details.
{org.osgi.framework.executionenvironment=OSGi/Minimum-1.0,OSGi/Minimum-1.1, osgi.ws=cocoa, osgi.arch=x86, osgi.os=macosx, org.eclipse.update.install.features=true, org.osgi.framework.system.packages=}
[Software being installed: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier, Missing requirement: org.eclipse.tptp.platform.instrumentation.ui 4.4.1.v201009092123 requires 'bundle org.eclipse.hyades.probekit [4.2.0,5.0.0)' but it could not be found, Cannot satisfy dependency: org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.1.v201009092123-797908s73533D4H6D56 depends on: org.eclipse.tptp.platform.instrumentation.ui [4.4.1.v201009092123], Cannot satisfy dependency: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier depends on: org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.0]
[ERROR] Internal error: java.lang.RuntimeException: org.eclipse.equinox.p2.core.ProvisionException: No solution found because the problem is unsatisfiable. -> [Help 1]
org.apache.maven.InternalErrorException: Internal error: java.lang.RuntimeException: org.eclipse.equinox.p2.core.ProvisionException: No solution found because the problem is unsatisfiable.

The important part of that error message is as follows:

org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier
   requirement: org.eclipse.tptp.platform.instrumentation.ui 4.4.1.v201009092123 
      requires 'bundle org.eclipse.hyades.probekit [4.2.0,5.0.0)' 
         but it could not be found

org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.1.v201009092123-797908s73533D4H6D56 
   depends on: org.eclipse.tptp.platform.instrumentation.ui [4.4.1.v201009092123]
     dependency: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier 
        depends on: org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.0]

So, how do you find which feature contains that plugin, so that you can add it to your target platform?

First, you need access to the repository. If you have direct server access to the repository from which the plugin comes (eg., the TPTP Helios update site), you can run this script in the root of that repository.

If you don't have server access (eg., you can't ssh to dev.eclipse.org and look in ~/downloads/tptp/updates/helios), then you can pull down a zip of the site (or use a p2.mirror script to fetch a copy of the site to your local machine)... and then run this script in the root of that repository.

Essentially the script finds matching feature jar files, unpacks them to extract the feature.xml files therein, and then greps those files for lines which suggest an included plugin matching the pattern for which you're searching:

$ findInFeature platform.probekit
./features/org.eclipse.tptp.platform.probekit_4.5.1.v201009092123-7H7BF8PAkF7B77ZARCNEK.jar
   <plugin
         id="org.eclipse.tptp.platform.probekit"

./features/org.eclipse.tptp.platform.trace_4.5.1.v201009092123-7L7O8bBgJ9E99jAfGWEM.jar
   <plugin
         id="org.eclipse.tptp.platform.probekit.launch"
From there, it's a trivial exercise to add another line item into your target platform file. First, paste in the feature jar:
./features/org.eclipse.tptp.platform.probekit_4.5.1.v201009092123-7H7BF8PAkF7B77ZARCNEK.jar

Then use vim to pattern-replace that string:

:%s/.\+\/\(org.\+\)_\(\d\+.\+\)\.jar/\t\t\t/g

And you'll end up with a new .feature.group added to the target:

<unit version="4.5.1.v201009092123-7H7BF8PAkF7B77ZARCNEK" id="org.eclipse.tptp.platform.probekit.feature.group"/>

2010-10-01

JBoss Tools: making it easier to build against a complex target platform

So you want to be a JBoss Tools developer? Awesome. Welcome to the family. SVN sources are here, JIRA's over here and there's cold beer in the fridge*.

But you say it's a pain in the tuchus to download over 25 zips or add a whole bunch of update sites and hope you get everything you need? Yeah, no argument there. If only there was an easier way to resolve all the dependencies you need to get building, much less to even RUN this stuff.

To make this process simpler, I've created a p2 repo (update site) from our target platform file, which has been recently updated to include Helios SR1 dependencies. You can track subsequent work in progress here: JIRA JBIDE-6982. You can also report any issues there too.


So, now, just add this single site** into your vanilla Eclipse 3.6.1 Classic (or a Helios SR1 bundle), uncheck the box for 'Group Items by Category' and you can install everything listed. For great justice.


Some handy links:


Some handy HOWTOs:


* - Due to beer2peer limitations, YMMV.

** - I'm aware that the update site throws a 403 if you open it in a browser. I can't be arsed to generate an index.html just yet, nor are there categorized features. Because really, you don't need either - this site is only meant to be used by p2.

2010-08-11

p2 Repository Association And The Fine Art Of Forceably Enabling Disabled Sites 2: Tycho Edition

Back in February, I blogged about how to add associate sites to an update site generated w/ the Buckminster or B3 aggregator.

Since we've moved our build infrastructure to use Tycho + Maven, I had to port the script over to work there for our update site aggregation.

Here are the moving pieces:

  • pom.xml that builds the site, listing input sites as <repository> entries, and a shout-out to build.xml, below, to do extra stuff during the install phase
  • build.xml ant script called by pom.xml, above, to do additional work after the creation of the site_assembly.zip update site zip
  • aggregateSite.jbosstools.properties properties file listing the aggregate sites to add

2010-07-20

Troubleshooting Eclipse.org Mirrors / Using Profiles & Target Platform Definition Files With Tycho

If you look in the mirror and you say his name 5 times, he'll appear behind you breathing down your neck. - Candyman

If only troubleshooting mirrors was so simple. Have you ever been running a build or an install which stalls at the provisioning step, with a message like:

[INFO] Fetching org.eclipse.birt.integration.wtp.ui_2.6.0.v20100617-1315.jar.pack.gz (4.3MB of 46.13MB at 171.56kB/s) from http://mirrors.xmission.com/eclipse/birt/update-site/2.6/plugins/org.eclipse.birt.integration.wtp.ui_2.6.0.v20100617-1315.jar.pack.gz

The solution here is often simply to force p2 to pick a specific mirror rather than letting it choose any mirror it wants.

How, you ask?

Well, assuming you were polling this site looking for artifacts to install or update...

http://download.eclipse.org/birt/update-site/2.6/

... you would then change that URL to this, and look at the list of available mirrors:

http://www.eclipse.org/downloads/download.php?file=/birt/update-site/2.6/

Now it's a trivial matter to select a mirror that's close to you and try that instead of the download.eclipse.org mirror, such as:

ftp://mirror.csclub.uwaterloo.ca/eclipse/birt/update-site/2.6/

If you're running a Tycho build, this URL should be changed in your parent-pom.xml ...

<url>http://download.eclipse.org/birt/update-site/2.6/</url>

... or your .target platform file, depending on which way you're building.

<repository location="http://download.eclipse.org/birt/update-site/2.6/"/>

If you rely on a parent-pom.xml, make sure you're activating the profile with the revised URL...

mvn3 clean install -U -B -fae -Phelios

... or, if you're building against a .target platform file, make sure you update the URL in that file, and that your build points to the profile which will load the .target file.

mvn3 clean install -U -B -fae -P!helios,helios-no-target

UPDATE, 2010/08/11: Forgot to mention that there are a number of p2 update site zips available here to help with your offsite mirroring: http://download.eclipse.org/athena/repos/.

2010-06-30

Update Site Aggregation: B3 vs. Tycho

Last year, I used to aggregate update sites with the Buckminster Aggregator, but since that won't install into Eclipse 3.6 (Helios), I had to migrate to the B3 Aggregator. This new version of the Aggregator is greatly expanded and worked fine until recently, when it has begun to suffer from a rather nasty p2 problem: instead of the Director just installing the aggregator into an Eclipse instance prior to then running the aggregation, I get "The copies of profile SDKProfile are not in sync," and the whole process dies. http://www.blogger.com/img/blank.gif

So, in order to find a workaround, I went back to Tycho, and discovered you can merge update sites with little more than two simple files:

  • a site.xml, which lists the features to aggregate and how to categorize them, and
  • a pom.xml to list the source sites and drive the aggregation.

Unfortunately the Tycho solution doesn't include the ability to add associate sites to the metadata after generation, but that can simply be done as a downstream step (Tycho can call Ant using the maven-antrun-plugin).

2010-04-16

HOWTO: Build a XulRunner 1.9.1.2 Update Site with Tycho 0.8 + Maven 3

0. Install Maven 3 by downloading then unpacking the tar.gz:

cd /tmp; \
wget http://mirror.csclub.uwaterloo.ca/apache/maven/binaries/apache-maven-3.0-alpha-7-bin.tar.gz; \
tar xvzf apache-maven-3.0-alpha-7-bin.tar.gz; \
chmod 755 apache-maven-3.0-alpha-7/bin/mvn; \
alias mvn3='/tmp/apache-maven-3.0-alpha-7/bin/mvn'

1. Check out sources:

cd /tmp; \
svn co http://anonsvn.jboss.org/repos/jbosstools/branches/modular_build/xulrunner/; \
wget http://anonsvn.jboss.org/repos/jbosstools/branches/modular_build/parent-pom.xml

2. Run build:

cd xulrunner; mvn3 -fae clean install

3. You will get a p2 repo / update site in the target folder of the site project, from which you can install XulRunner or XPCOM into your Eclipse.

cd site/org.mozilla.xulrunner.site/target/site

Note that the parent-pom.xml used above can in fact be much simpler. You only need the following:
<?xml version="1.0" encoding="UTF-8"?>
<project>
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.jboss.tools</groupId>
  <artifactId>org.jboss.tools.parent.pom</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <name>JBoss Tools Parent</name>
  <packaging>pom</packaging>
  <properties>
    <tychoVersion>0.8.0</tychoVersion>
  </properties>
  <build>
    <plugins>
      <plugin>
        <groupId>org.sonatype.tycho</groupId>
        <artifactId>tycho-maven-plugin</artifactId>
        <version>${tychoVersion}</version>
        <extensions>true</extensions>
      </plugin>

      <plugin>
        <groupId>org.sonatype.tycho</groupId>
        <artifactId>target-platform-configuration</artifactId>
        <version>${tychoVersion}</version>
        <configuration>
          <resolver>p2</resolver>
          <environments>
            <environment>
              <os>macosx</os>
              <ws>cocoa</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>macosx</os>
              <ws>carbon</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>win32</os>
              <ws>win32</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>linux</os>
              <ws>gtk</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>linux</os>
              <ws>gtk</ws>
              <arch>x86_64</arch>
            </environment>
          </environments>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

2010-03-31

HOWTO: Build Plugin & Feature Projects, Then Run Their Unit Tests w/ Tycho :: GEF Example

Note that the instructions below are for Linux (or MacOSX). On Windows, your YMMV, but the process is the same.

1. Check out the entire source tree of your project from CVS, SVN, or Git into ~/build.

2. If needed, move plugins, features & test plugins into:

~/build/plugins/
~/build/features/
~/build/tests/

(Test features should go into features/ folder too.)

3. Install scala from http://www.scala-lang.org/downloads

4. Fetch genpom.scala and parent-pom.xml; save in ~/build or equivalent.

5. Tweak parent-pom.xml to suit your needs or use as-is.

6. Run this to generate pom.xml files for plugins, features, and tests:

cd ~/build/plugins/; scala ../genpom.scala
cd ~/build/features/; scala ../genpom.scala
cd ~/build/tests/; scala ../genpom.scala

7. Download Maven 3 from http://www.apache.org/dyn/closer.cgi?path=/maven/binaries/apache-maven-3.0-alpha-7-bin.tar.gz

8. Install Maven 3

sudo su; cd /opt; tar xvzf apache-maven-3.0-alpha-7-bin.tar.gz
ln -s apache-maven-3.0-alpha-7 maven3

9. For convenience, alias mvn3 to the new maven:

alias mvn3='/opt/maven3/bin/mvn 2>&1 clean install | tee buildlog.latest.txt'

10. Build the plugins, features, and finally tests:

cd ~/build/plugins/; mvn3
cd ~/build/features/; mvn3
cd ~/build/tests/; mvn3
Look in ~/build/plugins/org.eclipse.*/target/ for generated jars.
Look in ~/.m2/repository/org/eclipse/* for published jars.
To automate steps 6 and 10, you can run this script in the ~/build/ folder:
#!/bin/bash
for f in plugins/ features/ tests/; do \
  cd $f; scala ../genpom.scala; \
  /opt/maven3/bin/mvn 2>&1 clean install | tee buildlog.latest.txt; \
  cd ..; \
done

(The above blog is also posted here.)

2010-03-24

I'm in love with Tycho 0.8 and Maven 3

Monday:

Signed up for GitHub, downloaded apache-maven-3.0-alpha-7-bin.tar.gz, and built Tycho from source. Took a bit of path wrangling (namely figuring out that maven runs based on current directory), but it built and most of the tests passed. Not bad for my first day ever using Maven.

# First download Maven3 and unpack into /opt/
export MAVEN_OPTS="-Xmx512m"
export TYCHO_TARGET_PLATFORM=/home/nboldt/eclipse/35clean/eclipse
mvn=/opt/maven/bin/mvn
$mvn clean install -e -V -Pbootstrap-1
$mvn clean install -e -V -Pbootstrap-2 -Dtycho.targetPlatform=$TYCHO_TARGET_PLATFORM
$mvn clean test -e -V -Pits -Dtycho.targetPlatform=$TYCHO_TARGET_PLATFORM

The really confusing part was "what now? what did I just build? And how do I use it?" Answer: You can now run maven3 from .../sonatype-tycho/tycho-its/target/apache-maven-3.0-alpha-7/bin/mvn


Tuesday:

Attended the Sonatype-sponsored Tycho Build Workshop, and amazed Pascal by saying good things about Tycho and Maven3. Apparently I'm known to complain about all things p2, despite my numerous blogs and wiki contributions touting its success and usefulness. Also installed m2eclipse to see how effective the pom editor is.

During the workshop, I got the following JBoss Tools projects to build: jmx, archives, as, flow, jbpm3/4, drools, bpel, smooks, common. That's nearly half our projects! Thanks to Jason and Igor for their invaluable assistance.

The reason it was so easy was that Max had already created a scala script to generate all the pom.xml files we need to build JBoss Tools, and a parent-pom.xml on which those tiny pom.xml files depend. (For me as a Maven noob, not having to think at all about creating poms made this transition a no-brainer. Speaking of no-brainers, my slides from Monday's talk about Dash Athena and Zombies are here (PDF).

As an added bonus, Tycho, being more strict than PDE, exposes missing information in MANIFEST.MF files such as dependencies on plugins. PDE-based builds are apparently more forgiving because they generally include more crap than is actually needed in the target platform against which the build runs; Tycho builds against what you tell it you need. So, the mavenification of JBoss Tools will actually cause us to have better described plugins, and therefore make them easier to build anywhere. I'm very impressed so far.

Unlike Dash Athena which needs to be told all the IUs (features/plugins) against which your project needs to build, Tycho determines this information from your manifests, so the information need not be duplicated.

I still need to look at how to build update sites from the output of a Tycho build, and how to hook in SWTBot/JUnit4 tests. I have a suspicion this may be handled using Buckminster tools such as their Aggregator, or my own custom Ant hackery using the SWTBot wiki docs. Time will tell.


Wed

This morning on a whim I decided to see if I could port Max's scala script (aside: I've never seen scala before yesterday) to generate pom.xml files for EMF or GEF. Success! With some minor tweaks, I've managed to build GEF 3.6 into my local maven repo (~/.m2). Obviously there's more to be done here... but damn, this is easy.