Much ado about scripting, Linux & Eclipse: card subject to change

Showing posts with label eclipse35. Show all posts
Showing posts with label eclipse35. Show all posts

2011-02-09

Simplifying The p2 Process, Part 2: Target Platform Repos

In Part 1 of this series, I looked at use of composite repos to provide a way of combining update sites into a single URL for ease of use and a single point of entry from which to do updates.

Defining a Target

Now, I'd like to talk about how to escape the proliferation of zips needed to establish a target platform. For those unfamiliar with the term "target platform", it's either the installed base against which you're compiling your code, or it's the collection of things you have to install first before you can install something on top of that.

For the JBoss Tools case, we have at least 8 prereqs for installation. Here's what you had to install prior to JBoss Tools 3.1.1:

Now, admittedly, because there is also the Ganymede update site, you don't necessarily need to download and unpack all these zips in order to install JBoss Tools - instead, you need only enable the Ganymede site. (Same story for Helios and JBoss Tools 3.2.)

However, to do a reproduceable PDE-based build, you still need to create this base install. Traditionally, PDE's approach was to download and unpack these zips into the root of the Eclipse install running the build. Athena attempted to improve on this situation by allowing you to define a list of update sites and IUs (features and/or plugins) which were needed to define the platform. But it was far from portable, and hardly reusable.

Buckminster (later b3) also approached this problem by creating its own markup for defining what sites and what IUs to install, backed by an EMF model. But rather than dealing with a UI to create the model and populate it, I found it more useful to simply generate an instance of the aggregator model and then use the aggregator to fetch & install IUs. But as the aggregator is simply a wrapper for the underlying p2.mirror and p2.director tasks, you can use those directly too.

But as they say... "Don't bore us, get to the chorus!" So, here's some sample code for the various solutions for build-time provisioning.

  1. Using the buckminster aggregator (properties file) - stopped working for us w/ Eclipse 3.6, so we switched to b3

  2. Using the b3 aggregator (properties file) - stopped worked consistently due to network timeouts resolving deps & fetching IUs.

  3. Using p2.mirror - underlying p2 ant task for mirroring from one or more repos to local disk

  4. Using p2.director - underlying p2 ant task for installing IUs (from local or remote repo) into some target Eclipse

So, with these tools, you could create a p2 repo from other repos - mirroring and installing IUs as needed - and even script an installation. But was there a better way?

Target Platform Definition File

Enter the target platform definition file (.target). This file contains a list of IUs and the p2 repos from which to provision them. So, it's like a b3 aggregator model, or an Athena build.properties file, but abstracted away from the concept of a build, because it can be used for building but ALSO for provisioning a user's installed Eclipse base.

Unfortunately, the Target Platform Definition File editor in Eclipse 3.6 is less than optimal for large targets, or when your internet connection is suboptimal. So, after fighting with it for a while, filing bugs, and ultimately giving up, I went back to my handy-dandy XML editor (often just vim) to maintain it more simply. So rather than having Eclipse automatically install things based on a .target file, I revert to a workflow that actually works: installing by hand from an update site.

While Buckminster does support .target files (or so I've read), I didn't want to be dependent on it any more, preferring a more "pure" solution.

So, based on code from Peter Nehrer (@pnehrer), I then wrote an XSL transform to create a p2.mirror script from a .target file, wrapped with another Ant script (and optionally, a Maven pom.xml script).

And why might you care? Well, this .target file can be used to:

  • Provision a developer's Eclipse, using the Target Platform Definition Editor and a few clicks (when it doesn't time out)
  • Provision a developer's Eclipse via script for offline or multiple users (getting the team up to speed)

And yes, much (or all) of the above can be done w/ Buckminster and/or b3, if you like that approach.

But I prefer to create the .target as input to a build process, rather than being explicitly tied to one. So, as I noted above, if you have a .target file, you can easily generate a p2 repo, and use that repo to run downstream builds. Now, instead of having a half-dozen zips to download and unpack with every build (using the deprecated and unsupported "dropins" method) you can use a fully-p2-friendly repo site which contains everything you need to do your builds - whether you're a Hudson server or a developer working at home or offline.

Benefits

  • Unlike "a collection of zips" this single-source-site can be versioned with each release.

  • It only contains WHAT YOU ACTUALLY NEED rather than extraneous sources and doc and tangential plugins/features you don't. It's a bit like making muffins by first grinding your own flour, but at least you know there's nothing evil in that muffin mix, and you will be able to consistently reproduce the recipe every time, regardless of where you might be on teh interwebz.

  • f you're a keener / beta tester who likes to build against the latest milestone (or even a weekly integration build) of Eclipse 3.next or 4.future, you can use the script above to self-update. So, while the TP itself is a contained snapshot listing the explicit versions of feature groups needed, it can also be run in "get the latest available" mode in order to keep your TP current against some HEAD or trunk development / releases.

  • By splitting the TP out of the build, you can build it upstream. So, where in the past we had one "uberbuild" and an implied TP therein, now we have a TP build job, and it is then shared by the 34 downstream jobs which depend on it for their dependencies.

Shut up and show me the code!

# for the "foo.target" file, build a local target platform repo, fetching the latest versions and updating the .target file
$ ant -f build.xml -DtargetFile=foo.target -DuseLatest=true

# for the "bar.target" file, build a local target platform repo, but fetch only the stated versions of IUs
$ ant -f build.xml -DtargetFile=bar.target -DuseLatest=false

That's it. I also wrap the build.xml ant script w/ a pom which allows it to be called from an upstream Maven/Tycho process, but that's nothing more than just calling the script using the antrun plugin (and a few ant dependencies), like this:

<build>
        <plugins>
                <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-antrun-plugin</artifactId>
                        <version>1.6</version>
                        <executions>
                                <execution>
                                        <phase>validate</phase>
                                        <configuration>
                                                <tasks>
                                                        <ant antfile="build.xml">
                                                                <property name="targetFile" value="multiple.target" />
                                                                <!-- <property name="repoDir" value="/path/to/where/to/provision/repo"/> -->
                                                        </ant>
                                                </tasks>
                                        </configuration>
                                        <goals>
                                                <goal>run</goal>
                                        </goals>
                                </execution>
                     </executions>
                        <dependencies>
                                <dependency>
                                        <groupId>commons-net</groupId>
                                        <artifactId>commons-net</artifactId>
                                        <version>1.4.1</version>
                                </dependency>
                                <dependency>
                                        <groupId>org.apache.ant</groupId>
                                        <artifactId>ant-commons-net</artifactId>
                                        <version>1.7.1</version>
                                </dependency>
                                <dependency>
                                        <groupId>org.apache.ant</groupId>
                                        <artifactId>ant-trax</artifactId>
                                        <version>1.7.1</version>
                                </dependency>
                        </dependencies>
                </plugin>
 </plugins>
</build>

The rest of the code is here.


In part 3, I'll look back at the success we've had using associate sites instead of asking people to manually add 3rd party URLs when installing JBoss Tools. SPOILER ALERT: one URL is easier for people to use than 6.

In part 4, I'll talk a little about how to prevent your product build from getting updates from unofficial sources, and preload your product with the official sites from which to get updates. Because it's important to balance ease of use with prevention of unsupported features. SPOILER ALERT: may contain p2.inf instructions.

2010-06-30

Update Site Aggregation: B3 vs. Tycho

Last year, I used to aggregate update sites with the Buckminster Aggregator, but since that won't install into Eclipse 3.6 (Helios), I had to migrate to the B3 Aggregator. This new version of the Aggregator is greatly expanded and worked fine until recently, when it has begun to suffer from a rather nasty p2 problem: instead of the Director just installing the aggregator into an Eclipse instance prior to then running the aggregation, I get "The copies of profile SDKProfile are not in sync," and the whole process dies. http://www.blogger.com/img/blank.gif

So, in order to find a workaround, I went back to Tycho, and discovered you can merge update sites with little more than two simple files:

  • a site.xml, which lists the features to aggregate and how to categorize them, and
  • a pom.xml to list the source sites and drive the aggregation.

Unfortunately the Tycho solution doesn't include the ability to add associate sites to the metadata after generation, but that can simply be done as a downstream step (Tycho can call Ant using the maven-antrun-plugin).

2010-04-16

HOWTO: Build a XulRunner 1.9.1.2 Update Site with Tycho 0.8 + Maven 3

0. Install Maven 3 by downloading then unpacking the tar.gz:

cd /tmp; \
wget http://mirror.csclub.uwaterloo.ca/apache/maven/binaries/apache-maven-3.0-alpha-7-bin.tar.gz; \
tar xvzf apache-maven-3.0-alpha-7-bin.tar.gz; \
chmod 755 apache-maven-3.0-alpha-7/bin/mvn; \
alias mvn3='/tmp/apache-maven-3.0-alpha-7/bin/mvn'

1. Check out sources:

cd /tmp; \
svn co http://anonsvn.jboss.org/repos/jbosstools/branches/modular_build/xulrunner/; \
wget http://anonsvn.jboss.org/repos/jbosstools/branches/modular_build/parent-pom.xml

2. Run build:

cd xulrunner; mvn3 -fae clean install

3. You will get a p2 repo / update site in the target folder of the site project, from which you can install XulRunner or XPCOM into your Eclipse.

cd site/org.mozilla.xulrunner.site/target/site

Note that the parent-pom.xml used above can in fact be much simpler. You only need the following:
<?xml version="1.0" encoding="UTF-8"?>
<project>
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.jboss.tools</groupId>
  <artifactId>org.jboss.tools.parent.pom</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <name>JBoss Tools Parent</name>
  <packaging>pom</packaging>
  <properties>
    <tychoVersion>0.8.0</tychoVersion>
  </properties>
  <build>
    <plugins>
      <plugin>
        <groupId>org.sonatype.tycho</groupId>
        <artifactId>tycho-maven-plugin</artifactId>
        <version>${tychoVersion}</version>
        <extensions>true</extensions>
      </plugin>

      <plugin>
        <groupId>org.sonatype.tycho</groupId>
        <artifactId>target-platform-configuration</artifactId>
        <version>${tychoVersion}</version>
        <configuration>
          <resolver>p2</resolver>
          <environments>
            <environment>
              <os>macosx</os>
              <ws>cocoa</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>macosx</os>
              <ws>carbon</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>win32</os>
              <ws>win32</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>linux</os>
              <ws>gtk</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>linux</os>
              <ws>gtk</ws>
              <arch>x86_64</arch>
            </environment>
          </environments>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

2010-02-18

p2 Repository Association And The Fine Art Of Forceably Enabling Disabled Sites

This week I've spent some time exploring how to associate p2 repo sites with other sites. The reason for this is that we publish features which depend on other publishers' features, and so would like to be able to have our update site automatically enable sites on which we depend in the end user's Eclipse.

The first attempt to do this was by using p2.inf files. Based on examples on blogs and in newsgroups, I added this p2.inf file into our BIRT Integration feature:

instructions.configure=addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:0,name:Eclipse Galileo,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:1,name:Eclipse Galileo,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:0,name:Eclipse 3.5,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:1,name:Eclipse 3.5,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:0,name:BIRT 2.5,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:1,name:BIRT 2.5,enabled:true);
Unfortunately, this didn't work - I got this error while building:

 java.lang.IllegalArgumentException: No action found for: addRepository.

So, hoping that using a fully-qualified name for the action would help, I tried this:

instructions.configure=org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:0,name:Eclipse Galileo,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:1,name:Eclipse Galileo,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:0,name:Eclipse 3.5,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:1,name:Eclipse 3.5,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:0,name:BIRT 2.5,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:1,name:BIRT 2.5,enabled:true);

Success! The build no longer complained. Of course I wasn't getting any associate sites enabled when I added the new repo into Eclipse. Why, I wondered?

Thankfully, I was told that the now-deprecated Metadata Generator (org.eclipse.equinox.p2.metadata.generator.EclipseGenerator) doesn't support p2.inf files, so it was finally time to switch to Eclipse 3.5's Update Site Publisher (org.eclipse.equinox.p2.publisher.UpdateSitePublisher), which can be run like this as a drop-in replacement for the above application:

<echo>Run p2.publisher.UpdateSitePublisher using launcherjar = @{launcherjar}</echo>
<java jar="@{launcherjar}"
      fork="true" timeout="10800000"
      jvm="${java.home}/bin/java"
      failonerror="false" maxmemory="256m" taskname="p2">
 <classpath>
  <fileset dir="${builder.build.path}/plugins"
           includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
  />
  <fileset dir="${clean.eclipse.home}/plugins"
           includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
  />
  <pathelement location="${builder.build.path}/plugins" />
  <pathelement location="${clean.eclipse.home}/plugins" />
 </classpath>
 <arg line=" org.eclipse.equinox.launcher.Main -application org.eclipse.equinox.p2.publisher.UpdateSitePublisher" />
 <arg line=" -metadataRepository file:${updateSiteJarDir}/ -metadataRepositoryName "${update.site.product.name} ${update.site.description} Update Site"" />
 <arg line=" -artifactRepository file:${updateSiteJarDir}/ -artifactRepositoryName "${update.site.product.name} ${update.site.description} Artifacts"" />
 <arg line=" -source ${updateSiteJarDir}/" />
 <arg line=" -compress -publishArtifacts -reusePack200Files -configs *,*,*" />
</java>

Having switched out the deprecated app for its successor, I then discovered that using the instructions.configure touchpoint was way too late in the process to be of any use; this information is only used when you install the feature, not when you poll the repo for metadata. So to add the BIRT 2.5 update site I'd have to install the BIRT integration feature... which depends on the BIRT 2.5 site. I needed to inject this requirement in at an earlier point in the process.

So, the next step was to hit up #equinox-dev and ask for help, which I got from Pascal "LeNettoyeur" Rapicault, fellow IBM alumnus. He suggested that I append into the content.xml file in my repo the following information. Note that type 0 and 1 represent metadata repo and artifact repo, and options=1 or 0 is used to mark the site enabled or disabled.

  <references size='2'>
    <repository uri='http://download.eclipse.org/birt/update-site/2.5/'
url='http://download.eclipse.org/birt/update-site/2.5/' type='0' options='1'/>
    <repository uri='http://download.eclipse.org/birt/update-site/2.5/'
url='http://download.eclipse.org/birt/update-site/2.5/' type='1' options='1'/>
  </references>
</repository>

Success, I again thought, since the associated sites were finally being added and enabled into Eclipse when the repo was scanned for metadata. I even scripted this process so that it could be integrated into our build process, or simply reused, eg., to merge two update sites and add some associate site for optional feature installs.

But alas, every step forward brings another step backward, as I discovered that available but disabled sites are NOT enabled despite the instructions in the repo. In other words, I cannot force an existing, yet disabled site to become enabled because my repo requires it. I can only document the requirement and hope that users will RTFM without getting too annoyed.

So, what's the solution? Well, there currently isn't one. But if this scenario seems like something you'd like to see improved, please cast your votes & comments in one of these bugs:

2009-07-24

Athena Common Builder: Thanks to the early adopters!

Here's a quick list of some of the projects @ Eclipse.org currently using Athena. If you haven't tried Athena, maybe it's time!

  1. Linux Tools
  2. Visual Editor (VE)
  3. Voice Tools (VTP)
  4. PDE Declarative Services Modeling Incubator (pde.ds.modeling.incubator)
  5. Nebula Widgets Gallery
  6. Faceted Project Framework (fproj)
  7. EMF Query (EMF MQ)
  8. EMF Validation (EMF VF)
  9. EMF Transaction (EMF MT)
  10. Ajax Tools Framework (ATF)
And @ JBoss.org, all available in JBoss Tools 3.1.0.M2 for Eclipse 3.5 (Galileo) ...
  1. JMX Console
  2. BPEL Editor*
  3. jBPM 3 and 4
  4. FreeMarker
  5. ...to name but a few.

* - will be available in M3 or you can get a nightly build here.

What are people saying about Athena?

"[I] really like the new build. It is much less confusing then the old [Common Modeling] one."

"[C]ongrats for this builder. It is quite good and I'm eager to rely fully on it."

"We are playing around with Athena and finding it really useful. It is already deployed to one of our customer developments. Thanks a lot for the hard work! We will definitely try to contribute back anything useful we will have in the process."

So, how do *you* get started? Here's a FAQ. Here's our New and Noteworthy. And here's all the rest of the knowledge base articles.

Oh, and I'd be remiss if I didn't mention that there's a big list of requested features still waiting for contributions. If you use Athena, you're already half-way to contributing back. Want to help? Drop me a line any time and we can discuss what holes need filling that match your skills and most directly improve your use of PDE, p2 and Athena.

2009-07-20

My love-hate with SVN, Part 6: Installation Ease Of Use

For months I've been annoyed by the fact that installation of Subversive (or Subclipse) requires fetching features and plugins from 3 or more update sites. No more!

Today, as an exercise to learn how to use the <p2.mirror/> task and provide a reproduceable, offline way to get Subversive into a virtual machine, I've created an update site zip, complete with site.xml and p2 metadata, which can be used to install Subversive from a single source. Here's the Ant script if you'd like to try this at home.

Because let's be real: you can only complain so long before it's time to roll up your sleeves and pitch in, right? That's how open source survives - thanks to people who care enough to complain AND care enough to help.

Here's the 13M update site zip, which includes the following:

Subversive 0.7.8
SVN Connector 2.2.1
SVNKit 1.2.3
JNA 3.0.9
ECF 3.0.0

Any problems, please report them in bug 284077.

2009-07-07

My love-hate with SVN, Part 5: Fedora 11 + Eclipse 3.5 + Subversion 1.6

Finally figured out how to make Eclipse 3.5 play nicely on Fedora 11 w/ Subversion, and I owe this bit of knowledge to our new MacPro. *sigh*

I also owe a great deal of gratitude to Cloudsmith for providing their Cloudsmith Galileo+ repository, which includes these features:

I still wish the version numbers would better align, in that I have to install the SVN Team Provider v0.7.8 with the SVN Connector v2.2.0 and the SVNKit 1.3.0 implementation v2.2.0 to make all this work with Subversion 1.6. Oof.

2009-06-23

Workspace #fail

Another obscure and unhelpful error message that now pops up about once every five minutes while I'm working in Eclipse. Upgraded to Eclipse 3.5 a day before the official release (thanks to my Friends of Eclipse membership), but to no avail. Evidently my workspace is pooched somehow.

Why can't more error messages tell you *HOW TO SOLVE THE PROBLEM*, rather than just reporting that something went wrong? Surely as software devs we should be able to do better... if for no better reason that to avoid having to listen to end users like myself complain? :P