Much ado about scripting, Linux & Eclipse: card subject to change

Showing posts with label p2. Show all posts
Showing posts with label p2. Show all posts

2011-10-26

HOWTO: Use Maven, Ant, and XSLT to scrub unwanted p2 metadata from an update site

Some time ago, I wrote about Using p2.inf to add/remove update sites. Tonight I found a simpler way to remove references in p2 metadata to external 3rd party sites.

For example, say you're repackaging some 3rd party features onto your own site, but don't want those features to provide references to the vendor's own update sites because you want to ensure that your product's site will only result in your sanctioned version being installed.

When you generate an update site, p2 pulls the information in the included features and will result in a section of references in the site's metadata that looks like this:

  <references size="6">
    <repository uri="http://download.eclipse.org/egit/updates" url="http://download.eclipse.org/egit/updates" type="0" options="0"/>
    <repository uri="http://subclipse.tigris.org/update_1.6.x" url="http://subclipse.tigris.org/update_1.6.x" type="1" options="0"/>
    <repository uri="http://download.eclipse.org/egit/updates" url="http://download.eclipse.org/egit/updates" type="1" options="0"/>
    <repository uri="http://subclipse.tigris.org/update_1.6.x" url="http://subclipse.tigris.org/update_1.6.x" type="0" options="0"/>
    <repository uri="http://eclipse.svnkit.com/1.3.x/" url="http://eclipse.svnkit.com/1.3.x/" type="0" options="0"/>
    <repository uri="http://eclipse.svnkit.com/1.3.x/" url="http://eclipse.svnkit.com/1.3.x/" type="1" options="0"/>
  </references>
To remove that, you can play with p2.inf directives, or you can simply perform an XSL transformation on the generated content.xml (inside content.jar, if your metadata is compressed) to remove the <references/> node:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
<xsl:template match="/">
        <xsl:apply-templates select="*"/>
</xsl:template>
<xsl:template match="*">
        <xsl:copy >
                <xsl:for-each select="@*">
                        <xsl:copy />
                </xsl:for-each>
                <xsl:apply-templates />
        </xsl:copy>
</xsl:template>
<xsl:template match="references" />
</xsl:stylesheet>
If you're generating your update site w/ Tycho, this transform can be called via a simple Ant script:

       <target name="remove.references">
                <!-- requires ant-contrib only if you like using if-then-else structures -->
                <if>
                        <available file="${update.site.source.dir}/content.jar" type="file" />
                        <then>
                                <unzip src="${update.site.source.dir}/content.jar" dest="${update.site.source.dir}" />
                                <delete file="${update.site.source.dir}/content.jar" />
                        </then>
                </if>
                <copy file="${update.site.source.dir}/content.xml" tofile="${update.site.source.dir}/content.old.xml" overwrite="true" />
                <xslt style="remove-references.xsl" in="${update.site.source.dir}/content.old.xml" out="${update.site.source.dir}/content.xml" />
                <zip destfile="${update.site.source.dir}/content.jar" basedir="${update.site.source.dir}" includes="content.xml" />
                <delete file="${update.site.source.dir}/content.xml" />
                <delete file="${update.site.source.dir}/content.old.xml" />
        </target>

Then, in your site's pom.xml, to call the Ant script, do this:

       <build>
                <plugins>
                        <plugin>
                                <groupId>org.apache.maven.plugins</groupId>
                                <artifactId>maven-antrun-plugin</artifactId>
                                <!-- make sure this variable is defined, eg., set to 1.3 -->
                                <version>${maven.antrun.plugin.version}</version>
                                <executions>
                                        <execution>
                                                <id>install</id>
                                                <phase>install</phase>
                                                <configuration>
                                                        <quiet>true</quiet>
                                                        <tasks>
                                                                <!-- called AFTER generating update site + zip to tweak content -->
                                                                <ant antfile="build.xml">
                                                                        <property name="SOME_ANT_VARIABLE" value="${SOME_MAVEN_VARIABLE}" />
                                                                </ant>
                                                        </tasks>
                                                </configuration>
                                                <goals>
                                                        <goal>run</goal>
                                                </goals>
                                        </execution>
                                </executions>
                                <dependencies>
                                        <!-- some dependencies your ant script might need -->
                                        <dependency>
                                                <groupId>commons-net</groupId>
                                                <artifactId>commons-net</artifactId>
                                                <version>1.4.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>org.apache.ant</groupId>
                                                <artifactId>ant</artifactId>
                                                <version>1.7.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>org.apache.ant</groupId>
                                                <artifactId>ant-nodeps</artifactId>
                                                <version>1.7.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>org.apache.ant</groupId>
                                                <artifactId>ant-trax</artifactId>
                                                <version>1.7.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>org.apache.ant</groupId>
                                                <artifactId>ant-commons-net</artifactId>
                                                <version>1.7.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>org.apache.ant</groupId>
                                                <artifactId>ant-apache-regexp</artifactId>
                                                <version>1.7.1</version>
                                        </dependency>
                                        <dependency>
                                                <groupId>ant-contrib</groupId>
                                                <artifactId>ant-contrib</artifactId>
                                                <version>1.0b3</version>
                                        </dependency>
                                </dependencies>
                        </plugin>
                </plugins>
        </build>

I suppose there's probably a way to call a transform directly from Maven w/o the Ant wrapper, but this allows unpacking and repacking of the content.jar to get at the content.xml file.

2011-04-19

vim scripting: replacing timestamps

After weeks of manually updating compositeArtifacts.xml and compositeContent.xml files' timestamps using `date +%s000`, I finally snapped and went looking for a better way.

Using the Vim Tips wiki as inspiration, I cobbled together this keymapped string replacement function for use in Vim:

" add this to your ~/.vimrc file, then type '\ts' to update timestamp to current
fun! ReplaceTimestamp()
   let tstamp = strftime("%s000")
   exe ":%s#<property name='p2.timestamp' value='[0-9]\\+'/>#<property name='p2.timestamp' value='" . tstamp . "'/>#g"
   echo "New time: " . tstamp
endfun
nnoremap <Leader>ts :call ReplaceTimestamp()<CR>

2011-02-16

Simplifying the p2 Process, Part 4: Using p2.inf to add/remove update sites

In Part 1 of this series, I looked at use of composite repos to provide a way of combining update sites into a single URL for ease of use and a single point of entry from which to do updates.

In Part 2, I discussed why we switched from using a collection of SDKs against which to build - using the now-deprecated brute-force "just unzip into eclipse root folder or dropins" approach - to using a single target platform update site so as to simplify maintenance and provide a reusable artifact for both build and workspace provisioning.

In Part 3, I looked at the idea of associating your repo with its upstream requirement sites, so that end-users need only use a single URL, rather than a half-dozen.


Finally, let's look at how you can use a p2.inf file to remove sites you don't support and add sites you do.

In JBDS 4, we include only two update sites - one for core features, and one for certified third-party extras, so that users will only get official updates from us, rather than from Spring, Eclipse, or anywhere else. Sure, they can manually add other URLs themselves, but that's a bit like pulling off the 'do not remove this tag' tag on a mattress or removing the 'warranty void if removed' sticker on your laptop.

So, first, we remove all the update site and discovery site URLs from our upstream features' feature.xml files, so they don't trickle down into the product.

Next, we use a p2.inf file:

# To explicitly remove a site, use instructions.unconfigure
instructions.configure=\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:0,location:https${#58}//www.your.server.com/,name:Core Product Updates);\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:1,location:https${#58}//www.your.server.com/,name:Core Product Updates);\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:0,location:https${#58}//www.your.server.com/extras/,name:Extra Product Updates);\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:1,location:https${#58}//www.your.server.com/extras/,name:Extra Product Updates);\
 

Then, to generate a site using that p2.inf instruction, here's a bit of Ant code:

<echo>Run p2.publisher.UpdateSitePublisher using launcherjar = @{launcherjar}</echo>
<java jar="@{launcherjar}"
      fork="true" timeout="10800000" jvm="${java.home}/bin/java" failonerror="true"
      maxmemory="256m" taskname="p2"
>
        <classpath>
                <fileset dir="${builder.build.path}/plugins"
                         includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
                />
                <fileset dir="${clean.eclipse.home}/plugins"
                         includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
                />
                <pathelement location="${builder.build.path}/plugins" />
                <pathelement location="${clean.eclipse.home}/plugins" />
        </classpath>
        <arg line=" org.eclipse.equinox.launcher.Main -consolelog -application org.eclipse.equinox.p2.publisher.UpdateSitePublisher"
        />
        <arg line=" -metadataRepository file:${updateSiteJarDir}/ -metadataRepositoryName "${update.site.product.name} ${update.site.description} Update Site""
        />
        <arg line=" -artifactRepository file:${updateSiteJarDir}/ -artifactRepositoryName "${update.site.product.name} ${update.site.description} Artifacts""
        />
        <arg line=" -source ${updateSiteJarDir}/" />
        <arg line=" -compress -publishArtifacts -reusePack200Files -configs *,*,*" />
</java>

Or, put your p2.inf file in the same directory as your build.properties ...

product=${builderDirectory}/jbds-all.product
runPackager=true

p2.gathering=true
p2.category.site=file:${builderDirectory}/site.xml
# locations.  Don't need a baseLocation, the transformedRepoLocation will have what we need
buildDirectory=${product.build.directory}/jbds-all-package
transformedRepoLocation=${product.build.directory}/jbds-all-package/transformed
repoBaseLocation=${product.build.directory}/jbds-all-package/toTransform

# The prefix that will be used in the generated archive.
archivePrefix=studio

# The location underwhich all of the build output will be collected.
collectingFolder=${archivePrefix}

# The list of {os, ws, arch} configurations to build.
configs = linux,gtk,x86 & win32,win32,x86 & linux,gtk,x86_64 & macosx,cocoa,x86 & macosx,cocoa,x86_64

buildId=${product.name}-product-${versionTag}
buildLabel=${buildId}

skipBase=true
skipMaps=true
skipFetch=true

... and your .product file ...

<?xml version="1.0" encoding="UTF-8"?>
<?pde version="3.5"?>

<product name="JBoss Developer Studio for Web and SOA Development" uid="com.jboss.jbds.all" id="com.jboss.jbds.product.product" application="org.eclipse.ui.ide.workbench" version="4.0.0.qualifier" useFeatures="true" includeLaunchers="true">

   <configIni use="default">
   </configIni>

   <launcherArgs>
      <programArgs>--launcher.XXMaxPermSize
256m
--launcher.defaultAction
openFile</programArgs>
      <vmArgs>-Xms512m
-Xmx1024m
-Dosgi.bundles=reference:file:org.eclipse.equinox.simpleconfigurator_1.0.200.v20100503.jar@1:start
-Dosgi.instance.area.default=@user.home/workspace</vmArgs>
      <vmArgsMac>-XstartOnFirstThread -Dorg.eclipse.swt.internal.carbon.smallFonts
-Xdock:icon=../Resources/JBDevStudio.icns</vmArgsMac>
   </launcherArgs>

   <windowImages/>

   <splash
      location="com.jboss.jbds.product" />
   <launcher name="jbdevstudio">
      <solaris/>
      <win useIco="true">
         <ico path="jbds.ico"/>
         <bmp/>
      </win>
   </launcher>

   <vm>
   </vm>

   <plugins>
   </plugins>

   <features>
      <feature id="com.jboss.jbds.product.feature" version="4.0.0.qualifier"/>
   </features>


</product>

... and when generating a product using PDE, that file and its instructions should be read at the correct time.

Hope this series has been helpful! If you have any examples of what you've done with .product or p2.inf files, please feel free to send me a link to your post or the file in your cvs, svn, or git repo. I'd love to see what else you can do with p2 and product builds.

See also:

2011-02-13

Simplifying The p2 Process, Part 3: Associate Sites

In Part 1 of this series, I looked at use of composite repos to provide a way of combining update sites into a single URL for ease of use and a single point of entry from which to do updates.

In Part 2, I discussed why we switched from using a collection of SDKs against which to build - using the now-deprecated brute-force "just unzip into eclipse root folder or dropins" approach - to using a single target platform update site so as to simplify maintenance and provide a reusable artifact for both build and workspace provisioning.


Now, let's look at the no-brainer that says that "less is more" when it comes to telling p2 from where to get updates, and that less effort for your user when installing is always a win.

If this sounds familar, I did blog about this briefly back in August. Since then, we've also added a quick XSLT script to remove the "Uncategorized" category that Tycho automatically adds for features which are listed in your site.xml but are not associated with a category. While this isn't strictly related to associate sites, it is about ease of use; while I applaud the desire to have everything belong to a category bucket (perhaps because the model Tycho's using requires it), the reason we'd rather hide these is to declutter the install view and not confuse people by suggesting features that won't work on their OS (eg., for which there's no XulRunner port).

But I digress...

Associate Sites

In the old days of yore, you could situate an associateSites.xml next to your site.xml in your "classic" update site, and Eclipse Update Manager would happily read that file and add those extra sites to your list of available update sites.

Then came p2, and while the old way still worked, it was no longer ideal. So, the new approach was to insert these associate sites directly into the p2 metadata for the site, content.xml and artifacts.xml (or content.jar and artifacts.jar).

This could be accomplished via a somewhat hacky appraoch - unpacking the existing metadata (content.jar) and shoehorning in the information at the bottom of the content.xml file, using an ant script (see "add.associate.sites" target, below) and a list of sites to be added:

<target name="add.associate.sites" if="associate.sites">
        <if>
                <and>
                        <!-- Defined in aggregateSite.properties -->
                        <isset property="associate.sites" />
                        <not>
                                <equals arg1="${associate.sites}" arg2="" />
                        </not>
                </and>
                <then>
                        <if>
                                <available file="${update.site.source.dir}/content.jar" type="file" />
                                <then>
                                        <unzip src="${update.site.source.dir}/content.jar" dest="${update.site.source.dir}" />
                                        <delete file="${update.site.source.dir}/content.jar" />
                                </then>
                        </if>
                        <!-- counter variable -->
                        <var name="associate.sites.0" value="" />
                        <for param="associate.site" list="${associate.sites}" delimiter=", 
">
                                <sequential>
                                        <var name="associate.sites.0" value="${associate.sites.0}00" />
                                </sequential>
                        </for>
                        <length property="associate.sites.length" string="${associate.sites.0}" />

                        <loadfile srcfile="${update.site.source.dir}/content.xml" property="content.xml">
                                <filterchain>
                                        <tailfilter lines="-1" skip="1" />
                                </filterchain>
                        </loadfile>
                        <echo file="${update.site.source.dir}/content.xml" message="${content.xml}" />
                        <echo file="${update.site.source.dir}/content.xml" append="true">  <references size='${associate.sites.length}'>
</echo>
                        <for param="associate.site" list="${associate.sites}" delimiter=", 
">
                                <sequential>
                                        <!-- insert into content.xml -->
                                        <echo file="${update.site.source.dir}/content.xml" append="true">    <repository uri='@{associate.site}' url='@{associate.site}' type='0' options='1'/>
<repository uri='@{associate.site}' url='@{associate.site}' type='1' options='1'/>
</echo>
                                </sequential>
                        </for>
                        <echo file="${update.site.source.dir}/content.xml" append="true">  </references>
</repository>
</echo>
          <!--  
    workaround for Tycho bug: uncategorized features in site.xml are put into
    "Uncategorized" category, rather than just being uncategorized (hidden) 
   -->
                 <copy file="${update.site.source.dir}/content.xml" tofile="${update.site.source.dir}/content.old.xml" overwrite="true" />
                        <xslt style="remove-uncategorized.xsl" in="${update.site.source.dir}/content.old.xml" out="${update.site.source.dir}/content.xml" />
                        <zip destfile="${update.site.source.dir}/content.jar" basedir="${update.site.source.dir}" includes="content.xml" />
                        <delete file="${update.site.source.dir}/content.xml" />
                        <delete file="${update.site.source.dir}/content.old.xml" />
                </then>
        </if>
</target>

So, now, instead of telling people to add multiple update sites to resolve missing potentially dependencies when installing, we can cause those extra sites to be automatically added at the same time they add the single URL for JBoss Tools. Now the additional sites need only be listed for reference, but no additional effort is required by the user.

BONUS HACK: to force a site that may already be listed (but disabled) to be added again, and this time definitely be enabled, you can add an extra slash into URL. Thus http://download.eclipse.org/birt/update-site/2.6 becomes http://download.eclipse.org//birt/update-site/2.6/, and as p2 sees a new site, it adds the new site (instead of ignoring it because it's already present but disabled. Again, a win.

Alternatively, you could cast an arcane spell using a p2.inf file in your feature's root folder or plugin's META-INF/ folder to add these additional, required sites... or do whatever processing you might need. I'm not sure if Tycho supports this yet, or how fully PDE supports reading this information. Got sample code? Send it to me as a comment below or via twitter to @nickboldt. Thanks!


In part 4, I'll talk a little about how to prevent your product build from getting updates from unofficial sources, and preload your product with the official sites from which to get updates. Because it's important to balance ease of use with prevention of unsupported features. SPOILER ALERT: may contain p2.inf instructions.

2011-02-09

Simplifying The p2 Process, Part 2: Target Platform Repos

In Part 1 of this series, I looked at use of composite repos to provide a way of combining update sites into a single URL for ease of use and a single point of entry from which to do updates.

Defining a Target

Now, I'd like to talk about how to escape the proliferation of zips needed to establish a target platform. For those unfamiliar with the term "target platform", it's either the installed base against which you're compiling your code, or it's the collection of things you have to install first before you can install something on top of that.

For the JBoss Tools case, we have at least 8 prereqs for installation. Here's what you had to install prior to JBoss Tools 3.1.1:

Now, admittedly, because there is also the Ganymede update site, you don't necessarily need to download and unpack all these zips in order to install JBoss Tools - instead, you need only enable the Ganymede site. (Same story for Helios and JBoss Tools 3.2.)

However, to do a reproduceable PDE-based build, you still need to create this base install. Traditionally, PDE's approach was to download and unpack these zips into the root of the Eclipse install running the build. Athena attempted to improve on this situation by allowing you to define a list of update sites and IUs (features and/or plugins) which were needed to define the platform. But it was far from portable, and hardly reusable.

Buckminster (later b3) also approached this problem by creating its own markup for defining what sites and what IUs to install, backed by an EMF model. But rather than dealing with a UI to create the model and populate it, I found it more useful to simply generate an instance of the aggregator model and then use the aggregator to fetch & install IUs. But as the aggregator is simply a wrapper for the underlying p2.mirror and p2.director tasks, you can use those directly too.

But as they say... "Don't bore us, get to the chorus!" So, here's some sample code for the various solutions for build-time provisioning.

  1. Using the buckminster aggregator (properties file) - stopped working for us w/ Eclipse 3.6, so we switched to b3

  2. Using the b3 aggregator (properties file) - stopped worked consistently due to network timeouts resolving deps & fetching IUs.

  3. Using p2.mirror - underlying p2 ant task for mirroring from one or more repos to local disk

  4. Using p2.director - underlying p2 ant task for installing IUs (from local or remote repo) into some target Eclipse

So, with these tools, you could create a p2 repo from other repos - mirroring and installing IUs as needed - and even script an installation. But was there a better way?

Target Platform Definition File

Enter the target platform definition file (.target). This file contains a list of IUs and the p2 repos from which to provision them. So, it's like a b3 aggregator model, or an Athena build.properties file, but abstracted away from the concept of a build, because it can be used for building but ALSO for provisioning a user's installed Eclipse base.

Unfortunately, the Target Platform Definition File editor in Eclipse 3.6 is less than optimal for large targets, or when your internet connection is suboptimal. So, after fighting with it for a while, filing bugs, and ultimately giving up, I went back to my handy-dandy XML editor (often just vim) to maintain it more simply. So rather than having Eclipse automatically install things based on a .target file, I revert to a workflow that actually works: installing by hand from an update site.

While Buckminster does support .target files (or so I've read), I didn't want to be dependent on it any more, preferring a more "pure" solution.

So, based on code from Peter Nehrer (@pnehrer), I then wrote an XSL transform to create a p2.mirror script from a .target file, wrapped with another Ant script (and optionally, a Maven pom.xml script).

And why might you care? Well, this .target file can be used to:

  • Provision a developer's Eclipse, using the Target Platform Definition Editor and a few clicks (when it doesn't time out)
  • Provision a developer's Eclipse via script for offline or multiple users (getting the team up to speed)

And yes, much (or all) of the above can be done w/ Buckminster and/or b3, if you like that approach.

But I prefer to create the .target as input to a build process, rather than being explicitly tied to one. So, as I noted above, if you have a .target file, you can easily generate a p2 repo, and use that repo to run downstream builds. Now, instead of having a half-dozen zips to download and unpack with every build (using the deprecated and unsupported "dropins" method) you can use a fully-p2-friendly repo site which contains everything you need to do your builds - whether you're a Hudson server or a developer working at home or offline.

Benefits

  • Unlike "a collection of zips" this single-source-site can be versioned with each release.

  • It only contains WHAT YOU ACTUALLY NEED rather than extraneous sources and doc and tangential plugins/features you don't. It's a bit like making muffins by first grinding your own flour, but at least you know there's nothing evil in that muffin mix, and you will be able to consistently reproduce the recipe every time, regardless of where you might be on teh interwebz.

  • f you're a keener / beta tester who likes to build against the latest milestone (or even a weekly integration build) of Eclipse 3.next or 4.future, you can use the script above to self-update. So, while the TP itself is a contained snapshot listing the explicit versions of feature groups needed, it can also be run in "get the latest available" mode in order to keep your TP current against some HEAD or trunk development / releases.

  • By splitting the TP out of the build, you can build it upstream. So, where in the past we had one "uberbuild" and an implied TP therein, now we have a TP build job, and it is then shared by the 34 downstream jobs which depend on it for their dependencies.

Shut up and show me the code!

# for the "foo.target" file, build a local target platform repo, fetching the latest versions and updating the .target file
$ ant -f build.xml -DtargetFile=foo.target -DuseLatest=true

# for the "bar.target" file, build a local target platform repo, but fetch only the stated versions of IUs
$ ant -f build.xml -DtargetFile=bar.target -DuseLatest=false

That's it. I also wrap the build.xml ant script w/ a pom which allows it to be called from an upstream Maven/Tycho process, but that's nothing more than just calling the script using the antrun plugin (and a few ant dependencies), like this:

<build>
        <plugins>
                <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-antrun-plugin</artifactId>
                        <version>1.6</version>
                        <executions>
                                <execution>
                                        <phase>validate</phase>
                                        <configuration>
                                                <tasks>
                                                        <ant antfile="build.xml">
                                                                <property name="targetFile" value="multiple.target" />
                                                                <!-- <property name="repoDir" value="/path/to/where/to/provision/repo"/> -->
                                                        </ant>
                                                </tasks>
                                        </configuration>
                                        <goals>
                                                <goal>run</goal>
                                        </goals>
                                </execution>
                     </executions>
                        <dependencies>
                                <dependency>
                                        <groupId>commons-net</groupId>
                                        <artifactId>commons-net</artifactId>
                                        <version>1.4.1</version>
                                </dependency>
                                <dependency>
                                        <groupId>org.apache.ant</groupId>
                                        <artifactId>ant-commons-net</artifactId>
                                        <version>1.7.1</version>
                                </dependency>
                                <dependency>
                                        <groupId>org.apache.ant</groupId>
                                        <artifactId>ant-trax</artifactId>
                                        <version>1.7.1</version>
                                </dependency>
                        </dependencies>
                </plugin>
 </plugins>
</build>

The rest of the code is here.


In part 3, I'll look back at the success we've had using associate sites instead of asking people to manually add 3rd party URLs when installing JBoss Tools. SPOILER ALERT: one URL is easier for people to use than 6.

In part 4, I'll talk a little about how to prevent your product build from getting updates from unofficial sources, and preload your product with the official sites from which to get updates. Because it's important to balance ease of use with prevention of unsupported features. SPOILER ALERT: may contain p2.inf instructions.

2011-02-01

HOWTO: Find osgi dependencies in features

Say you're trying to build something with Tycho & Maven 3 and while resolving dependencies before compilation, you're told:

[INFO] [Software being installed: 
  org.eclipse.tm.terminal.local.feature.group 0.1.0.v201006041240-10-7w312117152433, 
  Missing requirement: 
    org.eclipse.tm.terminal.local 0.1.0.v201006041322 
      requires 
        'bundle org.eclipse.cdt.core 5.2.0' 
          but it could not be found, 
  Cannot satisfy dependency: 
    org.eclipse.tm.terminal.local.feature.group 0.1.0.v201006041240-10-7w312117152433 
      depends on: 
        org.eclipse.tm.terminal.local [0.1.0.v201006041322]]

To quickly verify where this dependency is coming from, you can go look into the feature.xml for the org.eclipse.tm.terminal.local feature jar... but if you don't have it installed, this is somewhat more cumbersome; besides, you then have to unpack the jar before you can look inside it.

And maybe that feature contains a number of OTHER dependencies that you'll also need to resolve in your target platform when building. Sure, there are UI tools to do this within Eclipse, but when you're working on remote servers sometimes UI isn't available.

Workaround? Assuming you have a mirror of the update site(s) from which you're trying to resolve the dependency (eg., Helios) or can ssh to dev.eclipse.org, you can simply run a quick shell script to do the investigative work for you:

$ cd ~/downloads/releases/helios/201009240900/aggregate/; ~/bin/findDepInFeature "*tm*" cdt

./features/org.eclipse.tm.terminal.local_0.1.0.v201006041240-10-7w312117152433.jar
      <import feature="org.eclipse.cdt.platform" version="7.0.0" match="greaterOrEqual"/>
      <import plugin="org.eclipse.cdt.core" version="5.2.0" match="compatible"/>
      <import plugin="org.eclipse.core.runtime"/>
Where the script looks like this:
#!/bin/bash
# find plugins/feature deps by searching in some folder for feature jars, and searching through their feature.xml files for dependencies

# 1 - featurePattern - pattern of features to search (eg., "org.eclipse.tptp" or "\*" for all features)
# 2 - dependencyPattern  - pattern of plugins/feature deps for which to search (eg., "org.eclipse.tptp.platform.instrumentation.ui")
# 3 - location       - directory in which to search, if not "."

if [[ ! $1 ]]; then
        echo "Usage: $0 <featurePattern> <dependencyPattern> <location>"
        echo ""
        echo "Example: $0 tm.terminal cdt"
        exit 1
fi

# if no location, look in current dir (.)
if [[ $3 ]]; then location="$3"; else location="."; fi

# if no featurePattern, search all features for dependencyPattern
if [[ ! $2 ]]; then featurePattern="*"; dependencyPattern="$1"; else dependencyPattern="$2"; featurePattern="$1"; fi

rm -fr /tmp/findinfeature/; mkdir -p /tmp/findinfeature/features/
for f in $(find "$location" -type f -name "*${featurePattern}*" | egrep -v "pack.gz|source" | grep features | egrep "${featurePattern}"); do
        #echo "$f [$featurePattern, $dependencyPattern]"
        unzip -q $f -d /tmp/findinfeature/ feature.xml
        #       <import feature="org.eclipse.cdt.platform" version="7.0.0" match="greaterOrEqual"/>
        #       <import plugin="org.eclipse.cdt.core" version="5.2.0" match="compatible"/>
        if [[ ! $(cat /tmp/findinfeature/feature.xml | egrep "<import" -A3 | egrep "plugin=|feature=" -A1 -B1 | egrep "\".*${dependencyPattern}[^\"]*\"" -A1 -B1) ]]; then
                rm -fr /tmp/findinfeature/feature.xml
        else
                mv /tmp/findinfeature/feature.xml /tmp/findinfeature/${f}_feature.xml
                echo "${f}"
                cat /tmp/findinfeature/${f}_feature.xml | egrep "<import" -A3 | egrep "plugin=|feature=" -A1 -B1 | egrep "\".*${dependencyPattern}[^\"]*\"" -A1 -B1
                echo ""
        fi
        rm -fr /tmp/findinfeature/feature.xml
done

2011-01-29

Simplifying The p2 Process, Part 1: p2 Composite Repos

With the release of JBoss Tools 3.2 and JBoss Developer Studio 4.0 just around the corner, you may be thinking to yourself, "Self, how many update sites and SDK zips and runtimes will I need to download THIS time?"

Or maybe you're thinking, "Self, why is this so damn complicated?"

Well, folks, we heard your kvetching and we did something about it.

Composite Repos

While this is not a new concept to many, we embraced the composite update site this past year and it's made life a lot easier for iterative, agile development cycles. Last year, JBoss Tools 3.1 was built as a single Hudson job, with a second one for JBoss Developer Studio. This meant that any change in any of the components would cause a build to be launched, and 4-6hrs later, we'd have fresh bits. Yeah, far from ideal.

This year, we split up the monolith (and added a few new components!) so that now we have 34 update sites to compose into a single one against which builds can then be built. This composite update site looks like this:

compositeArtifacts.xml

<?xml version='1.0' encoding='UTF-8'?>
<?compositeArtifactRepository version='1.0.0'?>
<repository name='JBoss Tools Staging Repository' 
  type='org.eclipse.equinox.internal.p2.artifact.repository.CompositeArtifactRepository' 
  version='1.0.0'>
<properties size='2'>
<property name='p2.compressed' value='true'/>
<!-- get new time w/ `date +%s000` -->
<property name='p2.timestamp' value='1294205433000'/>
</properties>
<children size='34'>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-3.2_trunk.component--archives/all/repo/'/>
...
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-3.2_trunk.component--ws/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-pi4soa-3.1_trunk/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-teiid-designer-7.1_trunk/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-drools-5.2_trunk/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-savara-1.1_trunk/tools/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/xulrunner-1.9.1.2/all/repo/'/>
</children>
</repository>

compositeContent.xml

<?xml version='1.0' encoding='UTF-8'?>
<?compositeMetadataRepository version='1.0.0'?>
<repository name='JBoss Tools Staging Repository' 
  type='org.eclipse.equinox.internal.p2.metadata.repository.CompositeMetadataRepository' 
  version='1.0.0'>
<properties size='2'>
<property name='p2.compressed' value='true'/>
<!-- get new time w/ `date +%s000` -->
<property name='p2.timestamp' value='1294205433000'/>
</properties>
<children size='34'>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-3.2_trunk.component--archives/all/repo/'/>
...
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-3.2_trunk.component--ws/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-pi4soa-3.1_trunk/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-teiid-designer-7.1_trunk/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-drools-5.2_trunk/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-savara-1.1_trunk/tools/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/xulrunner-1.9.1.2/all/repo/'/>
</children>
</repository>

So, now that JBoss Tools is built in 34 pieces, the bits that haven't changed aren't rebuilt over and over and builds are faster. If that sounds insanely obvious to you, well, we used to have a lot of inter-component cyclic dependencies. We eliminated those early in the development cycle for JBoss Tools 3.2, and have been able to build smarter and faster ever since.

Added benefits to this composite site are:

  • Newly built and published bits are instantly available from the composite site - sure, the same was true under last year's PDE "uberbuild" regime, but that's because everything was built fresh every time, which was slow and near-impossible to get people to run at home.

  • Developers can use this site to install latest updates to components they're interested in testing - again, this was true before; but now using the same site and searching for updates, developers and beta testers can get incremental updates to the components that have actually changed, rather than having to pull down 160M every day to get a few K of changes.

  • Tycho can be pointed at this site (see below) in order to resolve binary p2 dependencies, so building a component deep in the dependency chain can be done w/o having to first build its upstream dependencies - this wasn't a concern before because everything was built from source every time, so by definition everything was already on disk. But now, if a developer only cares about a single component, like ModeShape or GWT, they need only have that source (and some bootstrapping code) on disk. Smaller, faster, more agile. And way more likely to be built locally before checking in code than before, making the painful "who broke what and when?" process much less painful. Fewer moving pieces and local dev builds at home mean - in theory - fewer incomplete or breaking commits.

When we first moved to Tycho, we needed to build a series of components locally in order to just get to a deep component. For example, the Struts component needs VPE, which needs JST and XulRunner. JST also needs the Common component, which in turn needs the Tests component.

So, to build Struts locally, 5 other components would have to be built locally first. This worked, but was still a fairly large barrier to entry for most developers (much less contributors!)

But with this new composite site, building Struts can be done without this lengthy bootstrapping; instead we just point Tycho at this composite site, and it pulls down the 5 upstream components' jars from this p2 repo - because the upstream deps are already built in Hudson.

Here's what we added to our parent pom.xml to have the builds find the binaries:

<repository>
        <id>jbosstools-nightly-staging-composite-trunk</id>
        <url>http://path.to.the.site/staging/_composite_/trunk/ </url>
        <layout>p2</layout>
        <snapshots>
                <enabled>true</enabled>
        </snapshots>
        <releases>
                <enabled>true</enabled>
        </releases>
</repository>

So, using this composite update site, we can use Maven 3 with Tycho 0.10 to generate a single update site (staged here, then ultimately published here).


In part 2, I'll look at why we switched from using a collection of SDKs (Eclipse, EMF, DTP, GEF, M2E, RSE, TPTP, UMl2, WTP, XSD and more) against which to build - using the now-deprecated brute-force "just unzip into eclipse root folder or dropins" approach - to using a single target platform update site. SPOILER ALERT: Easier to update and maintain.

In part 3, I'll look back at the success we've had using associate sites instead of asking people to manually add 3rd party URLs when installing JBoss Tools. SPOILER ALERT: one URL is easier for people to use than 6.

In part 4, I'll talk a little about how to prevent your product build from getting updates from unofficial sources, and preload your product with the official sites from which to get updates. Because it's important to balance ease of use with prevention of unsupported features. SPOILER ALERT: may contain p2.inf instructions.

By the way, JBoss Tools 3.2.0.CR1 and JBoss Developer Studio 4.0.0.CR1 are available. Get 'em while they're hot (and sourceforge is not).

2010-10-08

HOWTO: Find the feature that contains a plugin

Tycho is awesome.

However, like all build systems, it has its limitations.

One such limitation is that when you're building against a target platform, and something's missing, you get errors such as these:

[INFO] Cannot complete the request.  Generating details.
{org.osgi.framework.executionenvironment=OSGi/Minimum-1.0,OSGi/Minimum-1.1, osgi.ws=cocoa, osgi.arch=x86, osgi.os=macosx, org.eclipse.update.install.features=true, org.osgi.framework.system.packages=}
[Software being installed: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier, Missing requirement: org.eclipse.tptp.platform.instrumentation.ui 4.4.1.v201009092123 requires 'bundle org.eclipse.hyades.probekit [4.2.0,5.0.0)' but it could not be found, Cannot satisfy dependency: org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.1.v201009092123-797908s73533D4H6D56 depends on: org.eclipse.tptp.platform.instrumentation.ui [4.4.1.v201009092123], Cannot satisfy dependency: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier depends on: org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.0]
[ERROR] Internal error: java.lang.RuntimeException: org.eclipse.equinox.p2.core.ProvisionException: No solution found because the problem is unsatisfiable. -> [Help 1]
org.apache.maven.InternalErrorException: Internal error: java.lang.RuntimeException: org.eclipse.equinox.p2.core.ProvisionException: No solution found because the problem is unsatisfiable.

The important part of that error message is as follows:

org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier
   requirement: org.eclipse.tptp.platform.instrumentation.ui 4.4.1.v201009092123 
      requires 'bundle org.eclipse.hyades.probekit [4.2.0,5.0.0)' 
         but it could not be found

org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.1.v201009092123-797908s73533D4H6D56 
   depends on: org.eclipse.tptp.platform.instrumentation.ui [4.4.1.v201009092123]
     dependency: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier 
        depends on: org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.0]

So, how do you find which feature contains that plugin, so that you can add it to your target platform?

First, you need access to the repository. If you have direct server access to the repository from which the plugin comes (eg., the TPTP Helios update site), you can run this script in the root of that repository.

If you don't have server access (eg., you can't ssh to dev.eclipse.org and look in ~/downloads/tptp/updates/helios), then you can pull down a zip of the site (or use a p2.mirror script to fetch a copy of the site to your local machine)... and then run this script in the root of that repository.

Essentially the script finds matching feature jar files, unpacks them to extract the feature.xml files therein, and then greps those files for lines which suggest an included plugin matching the pattern for which you're searching:

$ findInFeature platform.probekit
./features/org.eclipse.tptp.platform.probekit_4.5.1.v201009092123-7H7BF8PAkF7B77ZARCNEK.jar
   <plugin
         id="org.eclipse.tptp.platform.probekit"

./features/org.eclipse.tptp.platform.trace_4.5.1.v201009092123-7L7O8bBgJ9E99jAfGWEM.jar
   <plugin
         id="org.eclipse.tptp.platform.probekit.launch"
From there, it's a trivial exercise to add another line item into your target platform file. First, paste in the feature jar:
./features/org.eclipse.tptp.platform.probekit_4.5.1.v201009092123-7H7BF8PAkF7B77ZARCNEK.jar

Then use vim to pattern-replace that string:

:%s/.\+\/\(org.\+\)_\(\d\+.\+\)\.jar/\t\t\t/g

And you'll end up with a new .feature.group added to the target:

<unit version="4.5.1.v201009092123-7H7BF8PAkF7B77ZARCNEK" id="org.eclipse.tptp.platform.probekit.feature.group"/>

2010-10-01

JBoss Tools: making it easier to build against a complex target platform

So you want to be a JBoss Tools developer? Awesome. Welcome to the family. SVN sources are here, JIRA's over here and there's cold beer in the fridge*.

But you say it's a pain in the tuchus to download over 25 zips or add a whole bunch of update sites and hope you get everything you need? Yeah, no argument there. If only there was an easier way to resolve all the dependencies you need to get building, much less to even RUN this stuff.

To make this process simpler, I've created a p2 repo (update site) from our target platform file, which has been recently updated to include Helios SR1 dependencies. You can track subsequent work in progress here: JIRA JBIDE-6982. You can also report any issues there too.


So, now, just add this single site** into your vanilla Eclipse 3.6.1 Classic (or a Helios SR1 bundle), uncheck the box for 'Group Items by Category' and you can install everything listed. For great justice.


Some handy links:


Some handy HOWTOs:


* - Due to beer2peer limitations, YMMV.

** - I'm aware that the update site throws a 403 if you open it in a browser. I can't be arsed to generate an index.html just yet, nor are there categorized features. Because really, you don't need either - this site is only meant to be used by p2.

2010-08-11

p2 Repository Association And The Fine Art Of Forceably Enabling Disabled Sites 2: Tycho Edition

Back in February, I blogged about how to add associate sites to an update site generated w/ the Buckminster or B3 aggregator.

Since we've moved our build infrastructure to use Tycho + Maven, I had to port the script over to work there for our update site aggregation.

Here are the moving pieces:

  • pom.xml that builds the site, listing input sites as <repository> entries, and a shout-out to build.xml, below, to do extra stuff during the install phase
  • build.xml ant script called by pom.xml, above, to do additional work after the creation of the site_assembly.zip update site zip
  • aggregateSite.jbosstools.properties properties file listing the aggregate sites to add

2010-07-20

Troubleshooting Eclipse.org Mirrors / Using Profiles & Target Platform Definition Files With Tycho

If you look in the mirror and you say his name 5 times, he'll appear behind you breathing down your neck. - Candyman

If only troubleshooting mirrors was so simple. Have you ever been running a build or an install which stalls at the provisioning step, with a message like:

[INFO] Fetching org.eclipse.birt.integration.wtp.ui_2.6.0.v20100617-1315.jar.pack.gz (4.3MB of 46.13MB at 171.56kB/s) from http://mirrors.xmission.com/eclipse/birt/update-site/2.6/plugins/org.eclipse.birt.integration.wtp.ui_2.6.0.v20100617-1315.jar.pack.gz

The solution here is often simply to force p2 to pick a specific mirror rather than letting it choose any mirror it wants.

How, you ask?

Well, assuming you were polling this site looking for artifacts to install or update...

http://download.eclipse.org/birt/update-site/2.6/

... you would then change that URL to this, and look at the list of available mirrors:

http://www.eclipse.org/downloads/download.php?file=/birt/update-site/2.6/

Now it's a trivial matter to select a mirror that's close to you and try that instead of the download.eclipse.org mirror, such as:

ftp://mirror.csclub.uwaterloo.ca/eclipse/birt/update-site/2.6/

If you're running a Tycho build, this URL should be changed in your parent-pom.xml ...

<url>http://download.eclipse.org/birt/update-site/2.6/</url>

... or your .target platform file, depending on which way you're building.

<repository location="http://download.eclipse.org/birt/update-site/2.6/"/>

If you rely on a parent-pom.xml, make sure you're activating the profile with the revised URL...

mvn3 clean install -U -B -fae -Phelios

... or, if you're building against a .target platform file, make sure you update the URL in that file, and that your build points to the profile which will load the .target file.

mvn3 clean install -U -B -fae -P!helios,helios-no-target

UPDATE, 2010/08/11: Forgot to mention that there are a number of p2 update site zips available here to help with your offsite mirroring: http://download.eclipse.org/athena/repos/.

2010-06-30

Update Site Aggregation: B3 vs. Tycho

Last year, I used to aggregate update sites with the Buckminster Aggregator, but since that won't install into Eclipse 3.6 (Helios), I had to migrate to the B3 Aggregator. This new version of the Aggregator is greatly expanded and worked fine until recently, when it has begun to suffer from a rather nasty p2 problem: instead of the Director just installing the aggregator into an Eclipse instance prior to then running the aggregation, I get "The copies of profile SDKProfile are not in sync," and the whole process dies. http://www.blogger.com/img/blank.gif

So, in order to find a workaround, I went back to Tycho, and discovered you can merge update sites with little more than two simple files:

  • a site.xml, which lists the features to aggregate and how to categorize them, and
  • a pom.xml to list the source sites and drive the aggregation.

Unfortunately the Tycho solution doesn't include the ability to add associate sites to the metadata after generation, but that can simply be done as a downstream step (Tycho can call Ant using the maven-antrun-plugin).

2010-05-25

My love-hate with SVN, Part 8: Installation Ease Of Use (UPDATED)

Back in July 2009, I blogged about My love-hate with SVN, Part 6: Installation Ease Of Use. With Helios just around the corner, I wanted to produce an updated repo for use with the latest & greatest Eclipse 3.6.

Now the <p2.mirror/> script will fetch and use Ant-Contrib automatically.

Here's the updated 15M update site zip, which includes the following:

Subversive 0.7.9
SVN Connector 2.2.2
SVNKit 1.3.3
JNA 3.2.3
ECF 3.1.0
Any problems, please report them in bug 284077.

2010-04-16

HOWTO: Build a XulRunner 1.9.1.2 Update Site with Tycho 0.8 + Maven 3

0. Install Maven 3 by downloading then unpacking the tar.gz:

cd /tmp; \
wget http://mirror.csclub.uwaterloo.ca/apache/maven/binaries/apache-maven-3.0-alpha-7-bin.tar.gz; \
tar xvzf apache-maven-3.0-alpha-7-bin.tar.gz; \
chmod 755 apache-maven-3.0-alpha-7/bin/mvn; \
alias mvn3='/tmp/apache-maven-3.0-alpha-7/bin/mvn'

1. Check out sources:

cd /tmp; \
svn co http://anonsvn.jboss.org/repos/jbosstools/branches/modular_build/xulrunner/; \
wget http://anonsvn.jboss.org/repos/jbosstools/branches/modular_build/parent-pom.xml

2. Run build:

cd xulrunner; mvn3 -fae clean install

3. You will get a p2 repo / update site in the target folder of the site project, from which you can install XulRunner or XPCOM into your Eclipse.

cd site/org.mozilla.xulrunner.site/target/site

Note that the parent-pom.xml used above can in fact be much simpler. You only need the following:
<?xml version="1.0" encoding="UTF-8"?>
<project>
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.jboss.tools</groupId>
  <artifactId>org.jboss.tools.parent.pom</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <name>JBoss Tools Parent</name>
  <packaging>pom</packaging>
  <properties>
    <tychoVersion>0.8.0</tychoVersion>
  </properties>
  <build>
    <plugins>
      <plugin>
        <groupId>org.sonatype.tycho</groupId>
        <artifactId>tycho-maven-plugin</artifactId>
        <version>${tychoVersion}</version>
        <extensions>true</extensions>
      </plugin>

      <plugin>
        <groupId>org.sonatype.tycho</groupId>
        <artifactId>target-platform-configuration</artifactId>
        <version>${tychoVersion}</version>
        <configuration>
          <resolver>p2</resolver>
          <environments>
            <environment>
              <os>macosx</os>
              <ws>cocoa</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>macosx</os>
              <ws>carbon</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>win32</os>
              <ws>win32</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>linux</os>
              <ws>gtk</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>linux</os>
              <ws>gtk</ws>
              <arch>x86_64</arch>
            </environment>
          </environments>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

2010-03-04

Athena: p2 repos w/ associate sites

Want to insert associate update sites into your user's Eclipse when they go to install your features from an update site/zip? With an Athena build this is now a trivial exercise. For example... say you're building a plugin which supports integration with Subclipse or Subversive. In a traditional install from an SDK or runtime zip, you'd unpack into eclipse/, and wonder why the new plugins didn't work. In the new p2 install world, you point Eclipse at the update site or zip, and are told that you cannot install the SVN integration because missing requirements cannot be found. So you hunt down the required sites, add them to Eclipse, and try again; this time you are allowed to install the new features. To improve the ease of use for the end user, Athena now allows you to define associate.sites in your build.properties. If this property is set, URLs will be added into your p2 repo's metadata and Eclipse will automatically add those sites when it scans your zip or site. No more manual copy+paste from documentation needed!

associate.sites=\
http://subclipse.tigris.org/update_1.6.x,\
http://download.eclipse.org/technology/subversive/0.7/update-site/,\
http://community.polarion.com/projects/subversive/download/eclipse/2.0/update-site/
And, of course, you can reuse these sites when resolving dependencies against which to build or test:
repositoryURLs=\
http://download.eclipse.org/releases/galileo/,\
http://download.eclipse.org/athena/repos/eclipse-Update-3.5.2-201002111343.zip,\
http://download.eclipse.org/eclipse/updates/3.5/,\
${associate.sites}

IUsToInstall=org.eclipse.sdk.feature.group+org.eclipse.sdk.ide+\
org.eclipse.team.svn+org.eclipse.team.svn.core+org.eclipse.team.svn.ui+\
org.tigris.subversion.subclipse.core+org.tigris.subversion.subclipse.ui

2010-02-18

p2 Repository Association And The Fine Art Of Forceably Enabling Disabled Sites

This week I've spent some time exploring how to associate p2 repo sites with other sites. The reason for this is that we publish features which depend on other publishers' features, and so would like to be able to have our update site automatically enable sites on which we depend in the end user's Eclipse.

The first attempt to do this was by using p2.inf files. Based on examples on blogs and in newsgroups, I added this p2.inf file into our BIRT Integration feature:

instructions.configure=addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:0,name:Eclipse Galileo,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:1,name:Eclipse Galileo,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:0,name:Eclipse 3.5,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:1,name:Eclipse 3.5,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:0,name:BIRT 2.5,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:1,name:BIRT 2.5,enabled:true);
Unfortunately, this didn't work - I got this error while building:

 java.lang.IllegalArgumentException: No action found for: addRepository.

So, hoping that using a fully-qualified name for the action would help, I tried this:

instructions.configure=org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:0,name:Eclipse Galileo,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:1,name:Eclipse Galileo,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:0,name:Eclipse 3.5,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:1,name:Eclipse 3.5,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:0,name:BIRT 2.5,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:1,name:BIRT 2.5,enabled:true);

Success! The build no longer complained. Of course I wasn't getting any associate sites enabled when I added the new repo into Eclipse. Why, I wondered?

Thankfully, I was told that the now-deprecated Metadata Generator (org.eclipse.equinox.p2.metadata.generator.EclipseGenerator) doesn't support p2.inf files, so it was finally time to switch to Eclipse 3.5's Update Site Publisher (org.eclipse.equinox.p2.publisher.UpdateSitePublisher), which can be run like this as a drop-in replacement for the above application:

<echo>Run p2.publisher.UpdateSitePublisher using launcherjar = @{launcherjar}</echo>
<java jar="@{launcherjar}"
      fork="true" timeout="10800000"
      jvm="${java.home}/bin/java"
      failonerror="false" maxmemory="256m" taskname="p2">
 <classpath>
  <fileset dir="${builder.build.path}/plugins"
           includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
  />
  <fileset dir="${clean.eclipse.home}/plugins"
           includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
  />
  <pathelement location="${builder.build.path}/plugins" />
  <pathelement location="${clean.eclipse.home}/plugins" />
 </classpath>
 <arg line=" org.eclipse.equinox.launcher.Main -application org.eclipse.equinox.p2.publisher.UpdateSitePublisher" />
 <arg line=" -metadataRepository file:${updateSiteJarDir}/ -metadataRepositoryName "${update.site.product.name} ${update.site.description} Update Site"" />
 <arg line=" -artifactRepository file:${updateSiteJarDir}/ -artifactRepositoryName "${update.site.product.name} ${update.site.description} Artifacts"" />
 <arg line=" -source ${updateSiteJarDir}/" />
 <arg line=" -compress -publishArtifacts -reusePack200Files -configs *,*,*" />
</java>

Having switched out the deprecated app for its successor, I then discovered that using the instructions.configure touchpoint was way too late in the process to be of any use; this information is only used when you install the feature, not when you poll the repo for metadata. So to add the BIRT 2.5 update site I'd have to install the BIRT integration feature... which depends on the BIRT 2.5 site. I needed to inject this requirement in at an earlier point in the process.

So, the next step was to hit up #equinox-dev and ask for help, which I got from Pascal "LeNettoyeur" Rapicault, fellow IBM alumnus. He suggested that I append into the content.xml file in my repo the following information. Note that type 0 and 1 represent metadata repo and artifact repo, and options=1 or 0 is used to mark the site enabled or disabled.

  <references size='2'>
    <repository uri='http://download.eclipse.org/birt/update-site/2.5/'
url='http://download.eclipse.org/birt/update-site/2.5/' type='0' options='1'/>
    <repository uri='http://download.eclipse.org/birt/update-site/2.5/'
url='http://download.eclipse.org/birt/update-site/2.5/' type='1' options='1'/>
  </references>
</repository>

Success, I again thought, since the associated sites were finally being added and enabled into Eclipse when the repo was scanned for metadata. I even scripted this process so that it could be integrated into our build process, or simply reused, eg., to merge two update sites and add some associate site for optional feature installs.

But alas, every step forward brings another step backward, as I discovered that available but disabled sites are NOT enabled despite the instructions in the repo. In other words, I cannot force an existing, yet disabled site to become enabled because my repo requires it. I can only document the requirement and hope that users will RTFM without getting too annoyed.

So, what's the solution? Well, there currently isn't one. But if this scenario seems like something you'd like to see improved, please cast your votes & comments in one of these bugs:

2009-11-27

Dash Athena: More Ant, More Tests, More Repos!

  • Infrastructure Changes
  • Cross-Platform / Ease of Use
  • bug 295670 Support running JUnit tests run from Ant script instead of Bash - tests can now be run on Linux and Windows (and to a lesser extent on Mac OSX) using new testLocal build step
  • Bug Fixes
  • bug 292486 Allow builds to fail if unit tests fail using failBuildIfTestFailuresOrErrors=true
  • bug 294678 Categories don't show up with IBM 1.6 JDK - implemented workaround so a different JDK can be used for p2 operations than for compilation of sources: use PACK200_JAVA_HOME=/path/to/different/JDK
  • bug 295773 Non-incubating projects no longer need to set "incubation=" in their build.properties
  • bug 292235 Included pre-compiled binary features/plugins are now included by default using PDE's runPackager property; can disable this behaviour with packageFeaturesIncludedBinaries=false
  • Documentation
  • Testing - Different ways to run or re-run tests, including as a secondary process after a build (so they can be run on a different platform or by a different user)
p2 does not natively support remote repo zips; to workaround this the zip is fetched and the URL is rewritten from http://path/to/foo.zip to jar:file:/tmp/path/foo.zip!/

Previous New & Noteworthy | More Athena Docs

2009-09-07

Re-Return to Athena

Thanks to Miles for taking the time to slap Athena into submission on his local system - frequent and regular stress testing is what's going to make this facade over PDE bridge the gap between today and the far-flung future (namely B3).

Because he took the time to itemize his concerns / problems / gotchas, I thought I'd take the time to explain why these happen... and which are bugs vs. features. Speaking of bugs and features, Athena has a handy New & Noteworthy wiki page which I update about once a month. If you've never seen the N&N, it's here.

# skipPack=true is useful if you want to test locally. I found that the update manager does not work with pak'd jars when running locally. Perhaps P2 is relying on something on the web server side..?
If p2 isn't properly unpacking pack'd jars, the problem is either: you're using a JDK with pack200 bugs (like Sun 1.6), or you have jars which should never be packed. How can you exclude jars from packing? See JarProcessor Options.
# The build scripts appear to be simply searching for the occurrence of strings to determine wether a given map entry type is being used. They do not appear to respect the comment lines. I guess I should file a bug report on this one -- it cost a bit of time because I have a dodgy SVN setup on my home laptop. I'm actually still not sure what is going on with that (and does the mysterious Java 134 error msg have anything to do with it) but I'm trying to learn not to fix things for which I have a work-around, i.e. if ti works on a different machine, just use that instead!
The purpose of the collectMapTypes task is to simply know what types of maps are being used so that warnings can be echoed to the user if appropriate. If you use SVN on linux, there's a good chance you'll get this message:
[echo] If you get a SIGSEGV or Java Result: 134 when fetching from SVN, remove 
[echo] subversion-javahl and svnkit to force PDE to use commandline svn client.
Why? Because it's a pain in the butt to debug, and at least having a message in the build log gives you some tea leaves to read. I favour documentation over performance in this case.
# There is a somewhat lengthy description at http://wiki.eclipse.org/Common_Build_Infrastructure/Getting_Started of how to setup your local build to use a local cache CVS site but I'm not sure what scenario that would really be helpful in. I just use COPY and it works fine. I suppose that using the CVS approach might excercie some issues that you wouldn't run into with a plain COPY. And I'm not sure wether the local build copies over the binary files or not. If it does, you could get different build results if say your local environment happens to place some artifacts in the copied directories that aren't cleaned out by the PDE build. Not going to worry about it!
A more useful scenario is to take a dump of a CVS, SVN, or Git tree, and point your build at that folder using localSourceCheckoutDir=/path/to/full/source/tree/root in your build.properties file. Then it'll build against that local source cache instead of having to fetch from the repo. Once you have the dump of the repo, from Eclipse do File > Import > Existing projects, and you can have those same projects compilable/editable in your workspace. You can also use Eclipse (eg., with Subversive, Subclipse, eGit) to update the cache before doing a build.
# A useful hack when building locally is to pack up your difficult dependency(s) into a zip file and refer to them in your dependencyURL. It doesn't matter what you use for the URL, because it will use the cached file instead. For example, GEF3D doesn't yet have a build or update site. (We've been collaborating a bit on this so hopefully the build I've got working will help them get one up soon.) So instead of solving the the SVN stuff right away, I put that off until everything else worked. So I created a zip and added an entry for http://download.eclipse.org/tools/gef/fake/gef3d.zip. You have to be careful to have your zip file packaged up correctly or it won't work. That's easy to get wrong..your file structure needs to conform to the standard structure, i.e. eclipse/plugins/some.plugin. On the other hand, you have to be careful to remove these fake dependencies when you begin testing the real fetched versions! The fetch part then fails silently; your build succeeds but using the previously downloaded files.
I generally add comments (TODO, FIXME) in my build.properties to mark temporary hacks, or use a new file, like build.properties.local.hack so I'm not breaking the server-friendly version. (Just make sure you tell build.xml which build.properties file to use, if not the default one -- and DON'T CHECK IN the changed file.)
# But in any case getting the SVN map file right has been a major headache. Again, the only reason I want this is to grab the GEF3D sources, but I can't seem to let this one go as it seems that it should be so easy. The Athena build seems to mangle things in the fetch script; or perhaps this is a PDE Build issue. So I tried the sourceforge style entry and after a couple of hours of fiddling (see below) I found a magic incantation that seems to work -- plugin@org.eclipse.draw3d=SVN,trunk,http://dev.eclipse.org/svnroot/technology/org.eclipse.gef3d,,org.eclipse.draw3d@HEAD
@HEAD is optional; you only need to specify a tag if it's not the HEAD of the branch or trunk. Here's a sample SVN map.
# It would be super nice if there were a built-in dependency analyzer so that one could find the root plugin issues easily. As it is, if you have a bad dependency you get the whole tree and you have to walk the tree by hand to find out the root cause. Overall I think this is another demonstration of the ultimate weakness of the standard build systems out there. They are by nature linear (batch) approaches and so usually you can't do post hoc or inline analysis of issues. There is no semantic information in logs! You end up having to parse them back if you want to do any useful analysis. In a parallel lifetime, I would love to work on a simple semantic logging system that would allow structured analysis of this sort.
I believe the Buckminster guys are doing work in this area, so this may make its way into B3.
# Here is something else that I would love to do given time. The map files are pretty well structured - it would be nice to whip up a quick XText editor that would at least check syntax and more ambitiously provide a mechanism for hooking in semantic checks -- wouldn't it be nice if you could check that that SVN url is correct while editing the map file itself?!
Add that to the wishlist for B3!
# I'm still not sure how Athena determines what goes in the sdk vs. runtime vs. example zips so for now I'm just building the examples manually. Anyone come across documentation for this?
This arbitrary allocation of files is done in common.releng/tools/scripts/buildAllHelper.xml, lines 1063-1097, and is only done if the build.steps entry 'buildZips' is enabled. This behaviour is as far as I'm concerned deprecated, but in place simply to ease transition for existing builds using the old Common Modeling Build to the Athena Common Build. IMHO, no one needs these zips - you only need an update site zip. If you don't like what's in the zips, you can omit the 'buildZips' step and package your own zips by hand, eg:
 <zip destfile="${buildDirectory}/${buildLabel}/${SDKZip}" update="true">
   <zipfileset src="${buildDirectory}/${buildLabel}/${allZip}" dirmode="775" filemode="664" excludes="**/*.pack.gz, **/pack.properties, **/features/*.jar, **/${domainNamespace}.*.all*, **/${domainNamespace}.*.all*/**" />
 </zip>

(For those unfamiliar with the story of the Re-Return, you need to watch HIMYM, S1E15.)