Much ado about scripting, Linux & Eclipse: card subject to change

Showing posts with label update. Show all posts
Showing posts with label update. Show all posts

2011-02-16

Simplifying the p2 Process, Part 4: Using p2.inf to add/remove update sites

In Part 1 of this series, I looked at use of composite repos to provide a way of combining update sites into a single URL for ease of use and a single point of entry from which to do updates.

In Part 2, I discussed why we switched from using a collection of SDKs against which to build - using the now-deprecated brute-force "just unzip into eclipse root folder or dropins" approach - to using a single target platform update site so as to simplify maintenance and provide a reusable artifact for both build and workspace provisioning.

In Part 3, I looked at the idea of associating your repo with its upstream requirement sites, so that end-users need only use a single URL, rather than a half-dozen.


Finally, let's look at how you can use a p2.inf file to remove sites you don't support and add sites you do.

In JBDS 4, we include only two update sites - one for core features, and one for certified third-party extras, so that users will only get official updates from us, rather than from Spring, Eclipse, or anywhere else. Sure, they can manually add other URLs themselves, but that's a bit like pulling off the 'do not remove this tag' tag on a mattress or removing the 'warranty void if removed' sticker on your laptop.

So, first, we remove all the update site and discovery site URLs from our upstream features' feature.xml files, so they don't trickle down into the product.

Next, we use a p2.inf file:

# To explicitly remove a site, use instructions.unconfigure
instructions.configure=\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:0,location:https${#58}//www.your.server.com/,name:Core Product Updates);\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:1,location:https${#58}//www.your.server.com/,name:Core Product Updates);\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:0,location:https${#58}//www.your.server.com/extras/,name:Extra Product Updates);\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:1,location:https${#58}//www.your.server.com/extras/,name:Extra Product Updates);\
 

Then, to generate a site using that p2.inf instruction, here's a bit of Ant code:

<echo>Run p2.publisher.UpdateSitePublisher using launcherjar = @{launcherjar}</echo>
<java jar="@{launcherjar}"
      fork="true" timeout="10800000" jvm="${java.home}/bin/java" failonerror="true"
      maxmemory="256m" taskname="p2"
>
        <classpath>
                <fileset dir="${builder.build.path}/plugins"
                         includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
                />
                <fileset dir="${clean.eclipse.home}/plugins"
                         includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
                />
                <pathelement location="${builder.build.path}/plugins" />
                <pathelement location="${clean.eclipse.home}/plugins" />
        </classpath>
        <arg line=" org.eclipse.equinox.launcher.Main -consolelog -application org.eclipse.equinox.p2.publisher.UpdateSitePublisher"
        />
        <arg line=" -metadataRepository file:${updateSiteJarDir}/ -metadataRepositoryName "${update.site.product.name} ${update.site.description} Update Site""
        />
        <arg line=" -artifactRepository file:${updateSiteJarDir}/ -artifactRepositoryName "${update.site.product.name} ${update.site.description} Artifacts""
        />
        <arg line=" -source ${updateSiteJarDir}/" />
        <arg line=" -compress -publishArtifacts -reusePack200Files -configs *,*,*" />
</java>

Or, put your p2.inf file in the same directory as your build.properties ...

product=${builderDirectory}/jbds-all.product
runPackager=true

p2.gathering=true
p2.category.site=file:${builderDirectory}/site.xml
# locations.  Don't need a baseLocation, the transformedRepoLocation will have what we need
buildDirectory=${product.build.directory}/jbds-all-package
transformedRepoLocation=${product.build.directory}/jbds-all-package/transformed
repoBaseLocation=${product.build.directory}/jbds-all-package/toTransform

# The prefix that will be used in the generated archive.
archivePrefix=studio

# The location underwhich all of the build output will be collected.
collectingFolder=${archivePrefix}

# The list of {os, ws, arch} configurations to build.
configs = linux,gtk,x86 & win32,win32,x86 & linux,gtk,x86_64 & macosx,cocoa,x86 & macosx,cocoa,x86_64

buildId=${product.name}-product-${versionTag}
buildLabel=${buildId}

skipBase=true
skipMaps=true
skipFetch=true

... and your .product file ...

<?xml version="1.0" encoding="UTF-8"?>
<?pde version="3.5"?>

<product name="JBoss Developer Studio for Web and SOA Development" uid="com.jboss.jbds.all" id="com.jboss.jbds.product.product" application="org.eclipse.ui.ide.workbench" version="4.0.0.qualifier" useFeatures="true" includeLaunchers="true">

   <configIni use="default">
   </configIni>

   <launcherArgs>
      <programArgs>--launcher.XXMaxPermSize
256m
--launcher.defaultAction
openFile</programArgs>
      <vmArgs>-Xms512m
-Xmx1024m
-Dosgi.bundles=reference:file:org.eclipse.equinox.simpleconfigurator_1.0.200.v20100503.jar@1:start
-Dosgi.instance.area.default=@user.home/workspace</vmArgs>
      <vmArgsMac>-XstartOnFirstThread -Dorg.eclipse.swt.internal.carbon.smallFonts
-Xdock:icon=../Resources/JBDevStudio.icns</vmArgsMac>
   </launcherArgs>

   <windowImages/>

   <splash
      location="com.jboss.jbds.product" />
   <launcher name="jbdevstudio">
      <solaris/>
      <win useIco="true">
         <ico path="jbds.ico"/>
         <bmp/>
      </win>
   </launcher>

   <vm>
   </vm>

   <plugins>
   </plugins>

   <features>
      <feature id="com.jboss.jbds.product.feature" version="4.0.0.qualifier"/>
   </features>


</product>

... and when generating a product using PDE, that file and its instructions should be read at the correct time.

Hope this series has been helpful! If you have any examples of what you've done with .product or p2.inf files, please feel free to send me a link to your post or the file in your cvs, svn, or git repo. I'd love to see what else you can do with p2 and product builds.

See also:

2010-10-01

JBoss Tools: making it easier to build against a complex target platform

So you want to be a JBoss Tools developer? Awesome. Welcome to the family. SVN sources are here, JIRA's over here and there's cold beer in the fridge*.

But you say it's a pain in the tuchus to download over 25 zips or add a whole bunch of update sites and hope you get everything you need? Yeah, no argument there. If only there was an easier way to resolve all the dependencies you need to get building, much less to even RUN this stuff.

To make this process simpler, I've created a p2 repo (update site) from our target platform file, which has been recently updated to include Helios SR1 dependencies. You can track subsequent work in progress here: JIRA JBIDE-6982. You can also report any issues there too.


So, now, just add this single site** into your vanilla Eclipse 3.6.1 Classic (or a Helios SR1 bundle), uncheck the box for 'Group Items by Category' and you can install everything listed. For great justice.


Some handy links:


Some handy HOWTOs:


* - Due to beer2peer limitations, YMMV.

** - I'm aware that the update site throws a 403 if you open it in a browser. I can't be arsed to generate an index.html just yet, nor are there categorized features. Because really, you don't need either - this site is only meant to be used by p2.

2010-08-11

p2 Repository Association And The Fine Art Of Forceably Enabling Disabled Sites 2: Tycho Edition

Back in February, I blogged about how to add associate sites to an update site generated w/ the Buckminster or B3 aggregator.

Since we've moved our build infrastructure to use Tycho + Maven, I had to port the script over to work there for our update site aggregation.

Here are the moving pieces:

  • pom.xml that builds the site, listing input sites as <repository> entries, and a shout-out to build.xml, below, to do extra stuff during the install phase
  • build.xml ant script called by pom.xml, above, to do additional work after the creation of the site_assembly.zip update site zip
  • aggregateSite.jbosstools.properties properties file listing the aggregate sites to add

2010-07-20

Troubleshooting Eclipse.org Mirrors / Using Profiles & Target Platform Definition Files With Tycho

If you look in the mirror and you say his name 5 times, he'll appear behind you breathing down your neck. - Candyman

If only troubleshooting mirrors was so simple. Have you ever been running a build or an install which stalls at the provisioning step, with a message like:

[INFO] Fetching org.eclipse.birt.integration.wtp.ui_2.6.0.v20100617-1315.jar.pack.gz (4.3MB of 46.13MB at 171.56kB/s) from http://mirrors.xmission.com/eclipse/birt/update-site/2.6/plugins/org.eclipse.birt.integration.wtp.ui_2.6.0.v20100617-1315.jar.pack.gz

The solution here is often simply to force p2 to pick a specific mirror rather than letting it choose any mirror it wants.

How, you ask?

Well, assuming you were polling this site looking for artifacts to install or update...

http://download.eclipse.org/birt/update-site/2.6/

... you would then change that URL to this, and look at the list of available mirrors:

http://www.eclipse.org/downloads/download.php?file=/birt/update-site/2.6/

Now it's a trivial matter to select a mirror that's close to you and try that instead of the download.eclipse.org mirror, such as:

ftp://mirror.csclub.uwaterloo.ca/eclipse/birt/update-site/2.6/

If you're running a Tycho build, this URL should be changed in your parent-pom.xml ...

<url>http://download.eclipse.org/birt/update-site/2.6/</url>

... or your .target platform file, depending on which way you're building.

<repository location="http://download.eclipse.org/birt/update-site/2.6/"/>

If you rely on a parent-pom.xml, make sure you're activating the profile with the revised URL...

mvn3 clean install -U -B -fae -Phelios

... or, if you're building against a .target platform file, make sure you update the URL in that file, and that your build points to the profile which will load the .target file.

mvn3 clean install -U -B -fae -P!helios,helios-no-target

UPDATE, 2010/08/11: Forgot to mention that there are a number of p2 update site zips available here to help with your offsite mirroring: http://download.eclipse.org/athena/repos/.

2010-06-30

Update Site Aggregation: B3 vs. Tycho

Last year, I used to aggregate update sites with the Buckminster Aggregator, but since that won't install into Eclipse 3.6 (Helios), I had to migrate to the B3 Aggregator. This new version of the Aggregator is greatly expanded and worked fine until recently, when it has begun to suffer from a rather nasty p2 problem: instead of the Director just installing the aggregator into an Eclipse instance prior to then running the aggregation, I get "The copies of profile SDKProfile are not in sync," and the whole process dies. http://www.blogger.com/img/blank.gif

So, in order to find a workaround, I went back to Tycho, and discovered you can merge update sites with little more than two simple files:

  • a site.xml, which lists the features to aggregate and how to categorize them, and
  • a pom.xml to list the source sites and drive the aggregation.

Unfortunately the Tycho solution doesn't include the ability to add associate sites to the metadata after generation, but that can simply be done as a downstream step (Tycho can call Ant using the maven-antrun-plugin).

2010-04-16

HOWTO: Build a XulRunner 1.9.1.2 Update Site with Tycho 0.8 + Maven 3

0. Install Maven 3 by downloading then unpacking the tar.gz:

cd /tmp; \
wget http://mirror.csclub.uwaterloo.ca/apache/maven/binaries/apache-maven-3.0-alpha-7-bin.tar.gz; \
tar xvzf apache-maven-3.0-alpha-7-bin.tar.gz; \
chmod 755 apache-maven-3.0-alpha-7/bin/mvn; \
alias mvn3='/tmp/apache-maven-3.0-alpha-7/bin/mvn'

1. Check out sources:

cd /tmp; \
svn co http://anonsvn.jboss.org/repos/jbosstools/branches/modular_build/xulrunner/; \
wget http://anonsvn.jboss.org/repos/jbosstools/branches/modular_build/parent-pom.xml

2. Run build:

cd xulrunner; mvn3 -fae clean install

3. You will get a p2 repo / update site in the target folder of the site project, from which you can install XulRunner or XPCOM into your Eclipse.

cd site/org.mozilla.xulrunner.site/target/site

Note that the parent-pom.xml used above can in fact be much simpler. You only need the following:
<?xml version="1.0" encoding="UTF-8"?>
<project>
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.jboss.tools</groupId>
  <artifactId>org.jboss.tools.parent.pom</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <name>JBoss Tools Parent</name>
  <packaging>pom</packaging>
  <properties>
    <tychoVersion>0.8.0</tychoVersion>
  </properties>
  <build>
    <plugins>
      <plugin>
        <groupId>org.sonatype.tycho</groupId>
        <artifactId>tycho-maven-plugin</artifactId>
        <version>${tychoVersion}</version>
        <extensions>true</extensions>
      </plugin>

      <plugin>
        <groupId>org.sonatype.tycho</groupId>
        <artifactId>target-platform-configuration</artifactId>
        <version>${tychoVersion}</version>
        <configuration>
          <resolver>p2</resolver>
          <environments>
            <environment>
              <os>macosx</os>
              <ws>cocoa</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>macosx</os>
              <ws>carbon</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>win32</os>
              <ws>win32</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>linux</os>
              <ws>gtk</ws>
              <arch>x86</arch>
            </environment>
            <environment>
              <os>linux</os>
              <ws>gtk</ws>
              <arch>x86_64</arch>
            </environment>
          </environments>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

2010-03-05

Beyond CVS 0.8.9 Released!

It's true that the shoemaker's children often go shoeless; so to my little Sourceforge project, Beyond CVS/SVN - maintained tirelessly over the years by Chris Callendar - has had to wait *years* to finally get its own update site.

Sad, I know. But thanks to a recent breakthough in being able to insert associate sites into p2 repo metadata, you can now install Beyond CVS/SVN without having to hunt down Subclipse or Subversive's update sites.

Here's the new site: http://beyondcvs.sourceforge.net/update/0.8.x. If you point Eclipse at it, you can install the features therein. If you point a browser at it, you can see instructions for how to do the install. And if you want an offline install, here's the zip: org.eclipse.externaltools-Update-0.8.9.v201003051612.zip (README).

2010-02-18

p2 Repository Association And The Fine Art Of Forceably Enabling Disabled Sites

This week I've spent some time exploring how to associate p2 repo sites with other sites. The reason for this is that we publish features which depend on other publishers' features, and so would like to be able to have our update site automatically enable sites on which we depend in the end user's Eclipse.

The first attempt to do this was by using p2.inf files. Based on examples on blogs and in newsgroups, I added this p2.inf file into our BIRT Integration feature:

instructions.configure=addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:0,name:Eclipse Galileo,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:1,name:Eclipse Galileo,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:0,name:Eclipse 3.5,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:1,name:Eclipse 3.5,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:0,name:BIRT 2.5,enabled:true); \
   addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:1,name:BIRT 2.5,enabled:true);
Unfortunately, this didn't work - I got this error while building:

 java.lang.IllegalArgumentException: No action found for: addRepository.

So, hoping that using a fully-qualified name for the action would help, I tried this:

instructions.configure=org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:0,name:Eclipse Galileo,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:1,name:Eclipse Galileo,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:0,name:Eclipse 3.5,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:1,name:Eclipse 3.5,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:0,name:BIRT 2.5,enabled:true); \
   org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:1,name:BIRT 2.5,enabled:true);

Success! The build no longer complained. Of course I wasn't getting any associate sites enabled when I added the new repo into Eclipse. Why, I wondered?

Thankfully, I was told that the now-deprecated Metadata Generator (org.eclipse.equinox.p2.metadata.generator.EclipseGenerator) doesn't support p2.inf files, so it was finally time to switch to Eclipse 3.5's Update Site Publisher (org.eclipse.equinox.p2.publisher.UpdateSitePublisher), which can be run like this as a drop-in replacement for the above application:

<echo>Run p2.publisher.UpdateSitePublisher using launcherjar = @{launcherjar}</echo>
<java jar="@{launcherjar}"
      fork="true" timeout="10800000"
      jvm="${java.home}/bin/java"
      failonerror="false" maxmemory="256m" taskname="p2">
 <classpath>
  <fileset dir="${builder.build.path}/plugins"
           includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
  />
  <fileset dir="${clean.eclipse.home}/plugins"
           includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
  />
  <pathelement location="${builder.build.path}/plugins" />
  <pathelement location="${clean.eclipse.home}/plugins" />
 </classpath>
 <arg line=" org.eclipse.equinox.launcher.Main -application org.eclipse.equinox.p2.publisher.UpdateSitePublisher" />
 <arg line=" -metadataRepository file:${updateSiteJarDir}/ -metadataRepositoryName "${update.site.product.name} ${update.site.description} Update Site"" />
 <arg line=" -artifactRepository file:${updateSiteJarDir}/ -artifactRepositoryName "${update.site.product.name} ${update.site.description} Artifacts"" />
 <arg line=" -source ${updateSiteJarDir}/" />
 <arg line=" -compress -publishArtifacts -reusePack200Files -configs *,*,*" />
</java>

Having switched out the deprecated app for its successor, I then discovered that using the instructions.configure touchpoint was way too late in the process to be of any use; this information is only used when you install the feature, not when you poll the repo for metadata. So to add the BIRT 2.5 update site I'd have to install the BIRT integration feature... which depends on the BIRT 2.5 site. I needed to inject this requirement in at an earlier point in the process.

So, the next step was to hit up #equinox-dev and ask for help, which I got from Pascal "LeNettoyeur" Rapicault, fellow IBM alumnus. He suggested that I append into the content.xml file in my repo the following information. Note that type 0 and 1 represent metadata repo and artifact repo, and options=1 or 0 is used to mark the site enabled or disabled.

  <references size='2'>
    <repository uri='http://download.eclipse.org/birt/update-site/2.5/'
url='http://download.eclipse.org/birt/update-site/2.5/' type='0' options='1'/>
    <repository uri='http://download.eclipse.org/birt/update-site/2.5/'
url='http://download.eclipse.org/birt/update-site/2.5/' type='1' options='1'/>
  </references>
</repository>

Success, I again thought, since the associated sites were finally being added and enabled into Eclipse when the repo was scanned for metadata. I even scripted this process so that it could be integrated into our build process, or simply reused, eg., to merge two update sites and add some associate site for optional feature installs.

But alas, every step forward brings another step backward, as I discovered that available but disabled sites are NOT enabled despite the instructions in the repo. In other words, I cannot force an existing, yet disabled site to become enabled because my repo requires it. I can only document the requirement and hope that users will RTFM without getting too annoyed.

So, what's the solution? Well, there currently isn't one. But if this scenario seems like something you'd like to see improved, please cast your votes & comments in one of these bugs:

2009-06-25

My love-hate with SVN, Part 4: Corrupt Metadata & Going Over Quota

Ever had one of those days where nothing seems to work? Most of June's been that way for me...

This week I decided to trust my OS and let Fedora update me automatically to the latest release, Fedora 11. I've never tried a distribution upgrade; in the past I've only ever done a clean install (be it Windows, Ubuntu, MEPIS, AntiX, or Fedora). But I figured if @dougschaefer could do it, so could I.

It was fairly smooth sailing, though the handy gui tool preupgrade only downloaded packages but didn't do the upgrade, so on reboot (still in F10) I had to run preupgrade-cli "Fedora 11 (Leonidas)". I suspect I must have fatfingered my hard drive password when I rebooted he first time because it worked like a charm the second time. Overall, way more successful than attempts so far to make a Mac Pro get Fedora'd, thatsfersure (grub, video, and network card issues, to name but a few).

Anyway, now I have updated versions of subversion and python, and as a result, my Subversive projects in Eclipse don't work. After much cursing and experimenting (and updating my CollabNet Subversion version to 1.6.3, the solution seems to be simply this:

Check out the projects anew within Eclipse, and if necessary, diff local changes from old project to new project.

But, if the project is too big (jbosstools trunk folder is over 1.1G) you may get a heap error. You can check the whole project out via commandline, but Eclipse (or Subversive? or Mylyn?) uses too much memory and the whole thing dies, despite my running Eclipse w/ a half-gig of heap:

/home/nboldt/eclipse/eclipse/eclipse -clean \
  -showLocation -data /home/nboldt/eclipse/workspace-jboss \
  -vmargs -Djava.library.path=/opt/CollabNet_Subversion/lib \
  -Xmx512M -XX:PermSize=512M

In this case, the solution is to check out the project without recursing into folders.

Commandline on in the Console view, that looks like this:

svn checkout "https://svn.jboss.org/repos/jbosstools/trunk@HEAD" -r \
  HEAD --depth empty  "/home/nboldt/eclipse/workspace-jboss/jbosstools-trunk"

You can then copy stuff you already checked out into the new target project, then refresh the project in Eclipse. Of course in my case Eclipse then thought all the files were new, so I had to Override and Update from the repo.

Another 24374 files or 1.1G to download. No wonder I went over my quota this month!

(Really, it was due to several different .iso torrent downloads for Fedora and CentOS, along with the movie Dead Alive, just in time for BLITEOTW day!)

So, unfortunately, I haven't been able to enjoy any of the 33 projects in this years' Eclipse Galileo release, unlike others on the Planet and the BirdsNest have. Hopefully next month will see calmer seas.

On the plus side now that I'm running Fedora 11, I can use Delta RPMs thanks to the yum-presto plugin... so next month's bill should be much, much smaller.