Much ado about scripting, Linux & Eclipse: card subject to change

2011-02-16

Simplifying the p2 Process, Part 4: Using p2.inf to add/remove update sites

In Part 1 of this series, I looked at use of composite repos to provide a way of combining update sites into a single URL for ease of use and a single point of entry from which to do updates.

In Part 2, I discussed why we switched from using a collection of SDKs against which to build - using the now-deprecated brute-force "just unzip into eclipse root folder or dropins" approach - to using a single target platform update site so as to simplify maintenance and provide a reusable artifact for both build and workspace provisioning.

In Part 3, I looked at the idea of associating your repo with its upstream requirement sites, so that end-users need only use a single URL, rather than a half-dozen.


Finally, let's look at how you can use a p2.inf file to remove sites you don't support and add sites you do.

In JBDS 4, we include only two update sites - one for core features, and one for certified third-party extras, so that users will only get official updates from us, rather than from Spring, Eclipse, or anywhere else. Sure, they can manually add other URLs themselves, but that's a bit like pulling off the 'do not remove this tag' tag on a mattress or removing the 'warranty void if removed' sticker on your laptop.

So, first, we remove all the update site and discovery site URLs from our upstream features' feature.xml files, so they don't trickle down into the product.

Next, we use a p2.inf file:

# To explicitly remove a site, use instructions.unconfigure
instructions.configure=\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:0,location:https${#58}//www.your.server.com/,name:Core Product Updates);\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:1,location:https${#58}//www.your.server.com/,name:Core Product Updates);\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:0,location:https${#58}//www.your.server.com/extras/,name:Extra Product Updates);\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:1,location:https${#58}//www.your.server.com/extras/,name:Extra Product Updates);\
 

Then, to generate a site using that p2.inf instruction, here's a bit of Ant code:

<echo>Run p2.publisher.UpdateSitePublisher using launcherjar = @{launcherjar}</echo>
<java jar="@{launcherjar}"
      fork="true" timeout="10800000" jvm="${java.home}/bin/java" failonerror="true"
      maxmemory="256m" taskname="p2"
>
        <classpath>
                <fileset dir="${builder.build.path}/plugins"
                         includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
                />
                <fileset dir="${clean.eclipse.home}/plugins"
                         includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
                />
                <pathelement location="${builder.build.path}/plugins" />
                <pathelement location="${clean.eclipse.home}/plugins" />
        </classpath>
        <arg line=" org.eclipse.equinox.launcher.Main -consolelog -application org.eclipse.equinox.p2.publisher.UpdateSitePublisher"
        />
        <arg line=" -metadataRepository file:${updateSiteJarDir}/ -metadataRepositoryName "${update.site.product.name} ${update.site.description} Update Site""
        />
        <arg line=" -artifactRepository file:${updateSiteJarDir}/ -artifactRepositoryName "${update.site.product.name} ${update.site.description} Artifacts""
        />
        <arg line=" -source ${updateSiteJarDir}/" />
        <arg line=" -compress -publishArtifacts -reusePack200Files -configs *,*,*" />
</java>

Or, put your p2.inf file in the same directory as your build.properties ...

product=${builderDirectory}/jbds-all.product
runPackager=true

p2.gathering=true
p2.category.site=file:${builderDirectory}/site.xml
# locations.  Don't need a baseLocation, the transformedRepoLocation will have what we need
buildDirectory=${product.build.directory}/jbds-all-package
transformedRepoLocation=${product.build.directory}/jbds-all-package/transformed
repoBaseLocation=${product.build.directory}/jbds-all-package/toTransform

# The prefix that will be used in the generated archive.
archivePrefix=studio

# The location underwhich all of the build output will be collected.
collectingFolder=${archivePrefix}

# The list of {os, ws, arch} configurations to build.
configs = linux,gtk,x86 & win32,win32,x86 & linux,gtk,x86_64 & macosx,cocoa,x86 & macosx,cocoa,x86_64

buildId=${product.name}-product-${versionTag}
buildLabel=${buildId}

skipBase=true
skipMaps=true
skipFetch=true

... and your .product file ...

<?xml version="1.0" encoding="UTF-8"?>
<?pde version="3.5"?>

<product name="JBoss Developer Studio for Web and SOA Development" uid="com.jboss.jbds.all" id="com.jboss.jbds.product.product" application="org.eclipse.ui.ide.workbench" version="4.0.0.qualifier" useFeatures="true" includeLaunchers="true">

   <configIni use="default">
   </configIni>

   <launcherArgs>
      <programArgs>--launcher.XXMaxPermSize
256m
--launcher.defaultAction
openFile</programArgs>
      <vmArgs>-Xms512m
-Xmx1024m
-Dosgi.bundles=reference:file:org.eclipse.equinox.simpleconfigurator_1.0.200.v20100503.jar@1:start
-Dosgi.instance.area.default=@user.home/workspace</vmArgs>
      <vmArgsMac>-XstartOnFirstThread -Dorg.eclipse.swt.internal.carbon.smallFonts
-Xdock:icon=../Resources/JBDevStudio.icns</vmArgsMac>
   </launcherArgs>

   <windowImages/>

   <splash
      location="com.jboss.jbds.product" />
   <launcher name="jbdevstudio">
      <solaris/>
      <win useIco="true">
         <ico path="jbds.ico"/>
         <bmp/>
      </win>
   </launcher>

   <vm>
   </vm>

   <plugins>
   </plugins>

   <features>
      <feature id="com.jboss.jbds.product.feature" version="4.0.0.qualifier"/>
   </features>


</product>

... and when generating a product using PDE, that file and its instructions should be read at the correct time.

Hope this series has been helpful! If you have any examples of what you've done with .product or p2.inf files, please feel free to send me a link to your post or the file in your cvs, svn, or git repo. I'd love to see what else you can do with p2 and product builds.

See also:

2011-02-13

Simplifying The p2 Process, Part 3: Associate Sites

In Part 1 of this series, I looked at use of composite repos to provide a way of combining update sites into a single URL for ease of use and a single point of entry from which to do updates.

In Part 2, I discussed why we switched from using a collection of SDKs against which to build - using the now-deprecated brute-force "just unzip into eclipse root folder or dropins" approach - to using a single target platform update site so as to simplify maintenance and provide a reusable artifact for both build and workspace provisioning.


Now, let's look at the no-brainer that says that "less is more" when it comes to telling p2 from where to get updates, and that less effort for your user when installing is always a win.

If this sounds familar, I did blog about this briefly back in August. Since then, we've also added a quick XSLT script to remove the "Uncategorized" category that Tycho automatically adds for features which are listed in your site.xml but are not associated with a category. While this isn't strictly related to associate sites, it is about ease of use; while I applaud the desire to have everything belong to a category bucket (perhaps because the model Tycho's using requires it), the reason we'd rather hide these is to declutter the install view and not confuse people by suggesting features that won't work on their OS (eg., for which there's no XulRunner port).

But I digress...

Associate Sites

In the old days of yore, you could situate an associateSites.xml next to your site.xml in your "classic" update site, and Eclipse Update Manager would happily read that file and add those extra sites to your list of available update sites.

Then came p2, and while the old way still worked, it was no longer ideal. So, the new approach was to insert these associate sites directly into the p2 metadata for the site, content.xml and artifacts.xml (or content.jar and artifacts.jar).

This could be accomplished via a somewhat hacky appraoch - unpacking the existing metadata (content.jar) and shoehorning in the information at the bottom of the content.xml file, using an ant script (see "add.associate.sites" target, below) and a list of sites to be added:

<target name="add.associate.sites" if="associate.sites">
        <if>
                <and>
                        <!-- Defined in aggregateSite.properties -->
                        <isset property="associate.sites" />
                        <not>
                                <equals arg1="${associate.sites}" arg2="" />
                        </not>
                </and>
                <then>
                        <if>
                                <available file="${update.site.source.dir}/content.jar" type="file" />
                                <then>
                                        <unzip src="${update.site.source.dir}/content.jar" dest="${update.site.source.dir}" />
                                        <delete file="${update.site.source.dir}/content.jar" />
                                </then>
                        </if>
                        <!-- counter variable -->
                        <var name="associate.sites.0" value="" />
                        <for param="associate.site" list="${associate.sites}" delimiter=", 
">
                                <sequential>
                                        <var name="associate.sites.0" value="${associate.sites.0}00" />
                                </sequential>
                        </for>
                        <length property="associate.sites.length" string="${associate.sites.0}" />

                        <loadfile srcfile="${update.site.source.dir}/content.xml" property="content.xml">
                                <filterchain>
                                        <tailfilter lines="-1" skip="1" />
                                </filterchain>
                        </loadfile>
                        <echo file="${update.site.source.dir}/content.xml" message="${content.xml}" />
                        <echo file="${update.site.source.dir}/content.xml" append="true">  <references size='${associate.sites.length}'>
</echo>
                        <for param="associate.site" list="${associate.sites}" delimiter=", 
">
                                <sequential>
                                        <!-- insert into content.xml -->
                                        <echo file="${update.site.source.dir}/content.xml" append="true">    <repository uri='@{associate.site}' url='@{associate.site}' type='0' options='1'/>
<repository uri='@{associate.site}' url='@{associate.site}' type='1' options='1'/>
</echo>
                                </sequential>
                        </for>
                        <echo file="${update.site.source.dir}/content.xml" append="true">  </references>
</repository>
</echo>
          <!--  
    workaround for Tycho bug: uncategorized features in site.xml are put into
    "Uncategorized" category, rather than just being uncategorized (hidden) 
   -->
                 <copy file="${update.site.source.dir}/content.xml" tofile="${update.site.source.dir}/content.old.xml" overwrite="true" />
                        <xslt style="remove-uncategorized.xsl" in="${update.site.source.dir}/content.old.xml" out="${update.site.source.dir}/content.xml" />
                        <zip destfile="${update.site.source.dir}/content.jar" basedir="${update.site.source.dir}" includes="content.xml" />
                        <delete file="${update.site.source.dir}/content.xml" />
                        <delete file="${update.site.source.dir}/content.old.xml" />
                </then>
        </if>
</target>

So, now, instead of telling people to add multiple update sites to resolve missing potentially dependencies when installing, we can cause those extra sites to be automatically added at the same time they add the single URL for JBoss Tools. Now the additional sites need only be listed for reference, but no additional effort is required by the user.

BONUS HACK: to force a site that may already be listed (but disabled) to be added again, and this time definitely be enabled, you can add an extra slash into URL. Thus http://download.eclipse.org/birt/update-site/2.6 becomes http://download.eclipse.org//birt/update-site/2.6/, and as p2 sees a new site, it adds the new site (instead of ignoring it because it's already present but disabled. Again, a win.

Alternatively, you could cast an arcane spell using a p2.inf file in your feature's root folder or plugin's META-INF/ folder to add these additional, required sites... or do whatever processing you might need. I'm not sure if Tycho supports this yet, or how fully PDE supports reading this information. Got sample code? Send it to me as a comment below or via twitter to @nickboldt. Thanks!


In part 4, I'll talk a little about how to prevent your product build from getting updates from unofficial sources, and preload your product with the official sites from which to get updates. Because it's important to balance ease of use with prevention of unsupported features. SPOILER ALERT: may contain p2.inf instructions.

2011-02-09

Simplifying The p2 Process, Part 2: Target Platform Repos

In Part 1 of this series, I looked at use of composite repos to provide a way of combining update sites into a single URL for ease of use and a single point of entry from which to do updates.

Defining a Target

Now, I'd like to talk about how to escape the proliferation of zips needed to establish a target platform. For those unfamiliar with the term "target platform", it's either the installed base against which you're compiling your code, or it's the collection of things you have to install first before you can install something on top of that.

For the JBoss Tools case, we have at least 8 prereqs for installation. Here's what you had to install prior to JBoss Tools 3.1.1:

Now, admittedly, because there is also the Ganymede update site, you don't necessarily need to download and unpack all these zips in order to install JBoss Tools - instead, you need only enable the Ganymede site. (Same story for Helios and JBoss Tools 3.2.)

However, to do a reproduceable PDE-based build, you still need to create this base install. Traditionally, PDE's approach was to download and unpack these zips into the root of the Eclipse install running the build. Athena attempted to improve on this situation by allowing you to define a list of update sites and IUs (features and/or plugins) which were needed to define the platform. But it was far from portable, and hardly reusable.

Buckminster (later b3) also approached this problem by creating its own markup for defining what sites and what IUs to install, backed by an EMF model. But rather than dealing with a UI to create the model and populate it, I found it more useful to simply generate an instance of the aggregator model and then use the aggregator to fetch & install IUs. But as the aggregator is simply a wrapper for the underlying p2.mirror and p2.director tasks, you can use those directly too.

But as they say... "Don't bore us, get to the chorus!" So, here's some sample code for the various solutions for build-time provisioning.

  1. Using the buckminster aggregator (properties file) - stopped working for us w/ Eclipse 3.6, so we switched to b3

  2. Using the b3 aggregator (properties file) - stopped worked consistently due to network timeouts resolving deps & fetching IUs.

  3. Using p2.mirror - underlying p2 ant task for mirroring from one or more repos to local disk

  4. Using p2.director - underlying p2 ant task for installing IUs (from local or remote repo) into some target Eclipse

So, with these tools, you could create a p2 repo from other repos - mirroring and installing IUs as needed - and even script an installation. But was there a better way?

Target Platform Definition File

Enter the target platform definition file (.target). This file contains a list of IUs and the p2 repos from which to provision them. So, it's like a b3 aggregator model, or an Athena build.properties file, but abstracted away from the concept of a build, because it can be used for building but ALSO for provisioning a user's installed Eclipse base.

Unfortunately, the Target Platform Definition File editor in Eclipse 3.6 is less than optimal for large targets, or when your internet connection is suboptimal. So, after fighting with it for a while, filing bugs, and ultimately giving up, I went back to my handy-dandy XML editor (often just vim) to maintain it more simply. So rather than having Eclipse automatically install things based on a .target file, I revert to a workflow that actually works: installing by hand from an update site.

While Buckminster does support .target files (or so I've read), I didn't want to be dependent on it any more, preferring a more "pure" solution.

So, based on code from Peter Nehrer (@pnehrer), I then wrote an XSL transform to create a p2.mirror script from a .target file, wrapped with another Ant script (and optionally, a Maven pom.xml script).

And why might you care? Well, this .target file can be used to:

  • Provision a developer's Eclipse, using the Target Platform Definition Editor and a few clicks (when it doesn't time out)
  • Provision a developer's Eclipse via script for offline or multiple users (getting the team up to speed)

And yes, much (or all) of the above can be done w/ Buckminster and/or b3, if you like that approach.

But I prefer to create the .target as input to a build process, rather than being explicitly tied to one. So, as I noted above, if you have a .target file, you can easily generate a p2 repo, and use that repo to run downstream builds. Now, instead of having a half-dozen zips to download and unpack with every build (using the deprecated and unsupported "dropins" method) you can use a fully-p2-friendly repo site which contains everything you need to do your builds - whether you're a Hudson server or a developer working at home or offline.

Benefits

  • Unlike "a collection of zips" this single-source-site can be versioned with each release.

  • It only contains WHAT YOU ACTUALLY NEED rather than extraneous sources and doc and tangential plugins/features you don't. It's a bit like making muffins by first grinding your own flour, but at least you know there's nothing evil in that muffin mix, and you will be able to consistently reproduce the recipe every time, regardless of where you might be on teh interwebz.

  • f you're a keener / beta tester who likes to build against the latest milestone (or even a weekly integration build) of Eclipse 3.next or 4.future, you can use the script above to self-update. So, while the TP itself is a contained snapshot listing the explicit versions of feature groups needed, it can also be run in "get the latest available" mode in order to keep your TP current against some HEAD or trunk development / releases.

  • By splitting the TP out of the build, you can build it upstream. So, where in the past we had one "uberbuild" and an implied TP therein, now we have a TP build job, and it is then shared by the 34 downstream jobs which depend on it for their dependencies.

Shut up and show me the code!

# for the "foo.target" file, build a local target platform repo, fetching the latest versions and updating the .target file
$ ant -f build.xml -DtargetFile=foo.target -DuseLatest=true

# for the "bar.target" file, build a local target platform repo, but fetch only the stated versions of IUs
$ ant -f build.xml -DtargetFile=bar.target -DuseLatest=false

That's it. I also wrap the build.xml ant script w/ a pom which allows it to be called from an upstream Maven/Tycho process, but that's nothing more than just calling the script using the antrun plugin (and a few ant dependencies), like this:

<build>
        <plugins>
                <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-antrun-plugin</artifactId>
                        <version>1.6</version>
                        <executions>
                                <execution>
                                        <phase>validate</phase>
                                        <configuration>
                                                <tasks>
                                                        <ant antfile="build.xml">
                                                                <property name="targetFile" value="multiple.target" />
                                                                <!-- <property name="repoDir" value="/path/to/where/to/provision/repo"/> -->
                                                        </ant>
                                                </tasks>
                                        </configuration>
                                        <goals>
                                                <goal>run</goal>
                                        </goals>
                                </execution>
                     </executions>
                        <dependencies>
                                <dependency>
                                        <groupId>commons-net</groupId>
                                        <artifactId>commons-net</artifactId>
                                        <version>1.4.1</version>
                                </dependency>
                                <dependency>
                                        <groupId>org.apache.ant</groupId>
                                        <artifactId>ant-commons-net</artifactId>
                                        <version>1.7.1</version>
                                </dependency>
                                <dependency>
                                        <groupId>org.apache.ant</groupId>
                                        <artifactId>ant-trax</artifactId>
                                        <version>1.7.1</version>
                                </dependency>
                        </dependencies>
                </plugin>
 </plugins>
</build>

The rest of the code is here.


In part 3, I'll look back at the success we've had using associate sites instead of asking people to manually add 3rd party URLs when installing JBoss Tools. SPOILER ALERT: one URL is easier for people to use than 6.

In part 4, I'll talk a little about how to prevent your product build from getting updates from unofficial sources, and preload your product with the official sites from which to get updates. Because it's important to balance ease of use with prevention of unsupported features. SPOILER ALERT: may contain p2.inf instructions.

2011-02-02

Visualizing OSGi Dependencies

Yesterday I blogged about how to find dependencies in features on plugins or features using a shell script to rip through feature jars.

But maybe you're less commandline, and more visual? Well, it may be over three years old, but there's a way to visualize plugin interdependencies using Ian Bull's PDE Dependency View. Frankly, I'm amazed this isn't already a core feature in PDE (and correct me if it is).

To use this tool, simply install it from:

http://download.eclipse.org/eclipse/pde/incubator/visualization/site

After installing and restarting, hit CTRL-3 and type "Graph" to find the "Graph Plug-In Dependencies View". (It's also available from Window > Show View > Other... (ALT-SHIFT-Q,Q) under Plug-in Development, if you prefer to kick it old-school.)

Next, right-click in the view or hit the "Focus on" button in the view, and select the plugin on which you want to focus.

Now you can browse up or down through plugins to explore dependencies.

For example, to see what plugins depend on a given plugin, such as org.eclipse.tm.terminal, click the "Show Callers" button in the view.

Or, to see on which plugins org.jboss.ide.eclipe.as.rse.ui depends, click the "Show Callees" button in the view. You can shift-click on nodes to highlight them for emphasis, or click and drag them around.

2011-02-01

Syntax highlighting for code snippets in blogs

Where does he get those wonderful toys? - The Joker, Batman (1989)
Recently I was asked how I make code snippets on this blog prettier. Here's how.
  1. In your blog's template, install Alex Gorbatchev's Syntax Highlighter.
  2. Then, in the body of the blog post (with all occurrences of "<" HTML-escaped as "&lt"):
    <pre class="brush:shell"> ... </pre>
    or
    <pre class="brush:java"> ... </pre>
    or
    <pre class="brush:xml"> ... </pre>
  3. There are other "brushes" available too with which you can "paint" your code. I generally only need java, xml, and shell.

HOWTO: Find osgi dependencies in features

Say you're trying to build something with Tycho & Maven 3 and while resolving dependencies before compilation, you're told:

[INFO] [Software being installed: 
  org.eclipse.tm.terminal.local.feature.group 0.1.0.v201006041240-10-7w312117152433, 
  Missing requirement: 
    org.eclipse.tm.terminal.local 0.1.0.v201006041322 
      requires 
        'bundle org.eclipse.cdt.core 5.2.0' 
          but it could not be found, 
  Cannot satisfy dependency: 
    org.eclipse.tm.terminal.local.feature.group 0.1.0.v201006041240-10-7w312117152433 
      depends on: 
        org.eclipse.tm.terminal.local [0.1.0.v201006041322]]

To quickly verify where this dependency is coming from, you can go look into the feature.xml for the org.eclipse.tm.terminal.local feature jar... but if you don't have it installed, this is somewhat more cumbersome; besides, you then have to unpack the jar before you can look inside it.

And maybe that feature contains a number of OTHER dependencies that you'll also need to resolve in your target platform when building. Sure, there are UI tools to do this within Eclipse, but when you're working on remote servers sometimes UI isn't available.

Workaround? Assuming you have a mirror of the update site(s) from which you're trying to resolve the dependency (eg., Helios) or can ssh to dev.eclipse.org, you can simply run a quick shell script to do the investigative work for you:

$ cd ~/downloads/releases/helios/201009240900/aggregate/; ~/bin/findDepInFeature "*tm*" cdt

./features/org.eclipse.tm.terminal.local_0.1.0.v201006041240-10-7w312117152433.jar
      <import feature="org.eclipse.cdt.platform" version="7.0.0" match="greaterOrEqual"/>
      <import plugin="org.eclipse.cdt.core" version="5.2.0" match="compatible"/>
      <import plugin="org.eclipse.core.runtime"/>
Where the script looks like this:
#!/bin/bash
# find plugins/feature deps by searching in some folder for feature jars, and searching through their feature.xml files for dependencies

# 1 - featurePattern - pattern of features to search (eg., "org.eclipse.tptp" or "\*" for all features)
# 2 - dependencyPattern  - pattern of plugins/feature deps for which to search (eg., "org.eclipse.tptp.platform.instrumentation.ui")
# 3 - location       - directory in which to search, if not "."

if [[ ! $1 ]]; then
        echo "Usage: $0 <featurePattern> <dependencyPattern> <location>"
        echo ""
        echo "Example: $0 tm.terminal cdt"
        exit 1
fi

# if no location, look in current dir (.)
if [[ $3 ]]; then location="$3"; else location="."; fi

# if no featurePattern, search all features for dependencyPattern
if [[ ! $2 ]]; then featurePattern="*"; dependencyPattern="$1"; else dependencyPattern="$2"; featurePattern="$1"; fi

rm -fr /tmp/findinfeature/; mkdir -p /tmp/findinfeature/features/
for f in $(find "$location" -type f -name "*${featurePattern}*" | egrep -v "pack.gz|source" | grep features | egrep "${featurePattern}"); do
        #echo "$f [$featurePattern, $dependencyPattern]"
        unzip -q $f -d /tmp/findinfeature/ feature.xml
        #       <import feature="org.eclipse.cdt.platform" version="7.0.0" match="greaterOrEqual"/>
        #       <import plugin="org.eclipse.cdt.core" version="5.2.0" match="compatible"/>
        if [[ ! $(cat /tmp/findinfeature/feature.xml | egrep "<import" -A3 | egrep "plugin=|feature=" -A1 -B1 | egrep "\".*${dependencyPattern}[^\"]*\"" -A1 -B1) ]]; then
                rm -fr /tmp/findinfeature/feature.xml
        else
                mv /tmp/findinfeature/feature.xml /tmp/findinfeature/${f}_feature.xml
                echo "${f}"
                cat /tmp/findinfeature/${f}_feature.xml | egrep "<import" -A3 | egrep "plugin=|feature=" -A1 -B1 | egrep "\".*${dependencyPattern}[^\"]*\"" -A1 -B1
                echo ""
        fi
        rm -fr /tmp/findinfeature/feature.xml
done