Much ado about scripting, Linux & Eclipse: card subject to change

Showing posts with label repo. Show all posts
Showing posts with label repo. Show all posts

2011-02-16

Simplifying the p2 Process, Part 4: Using p2.inf to add/remove update sites

In Part 1 of this series, I looked at use of composite repos to provide a way of combining update sites into a single URL for ease of use and a single point of entry from which to do updates.

In Part 2, I discussed why we switched from using a collection of SDKs against which to build - using the now-deprecated brute-force "just unzip into eclipse root folder or dropins" approach - to using a single target platform update site so as to simplify maintenance and provide a reusable artifact for both build and workspace provisioning.

In Part 3, I looked at the idea of associating your repo with its upstream requirement sites, so that end-users need only use a single URL, rather than a half-dozen.


Finally, let's look at how you can use a p2.inf file to remove sites you don't support and add sites you do.

In JBDS 4, we include only two update sites - one for core features, and one for certified third-party extras, so that users will only get official updates from us, rather than from Spring, Eclipse, or anywhere else. Sure, they can manually add other URLs themselves, but that's a bit like pulling off the 'do not remove this tag' tag on a mattress or removing the 'warranty void if removed' sticker on your laptop.

So, first, we remove all the update site and discovery site URLs from our upstream features' feature.xml files, so they don't trickle down into the product.

Next, we use a p2.inf file:

# To explicitly remove a site, use instructions.unconfigure
instructions.configure=\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:0,location:https${#58}//www.your.server.com/,name:Core Product Updates);\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:1,location:https${#58}//www.your.server.com/,name:Core Product Updates);\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:0,location:https${#58}//www.your.server.com/extras/,name:Extra Product Updates);\
org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:1,location:https${#58}//www.your.server.com/extras/,name:Extra Product Updates);\
 

Then, to generate a site using that p2.inf instruction, here's a bit of Ant code:

<echo>Run p2.publisher.UpdateSitePublisher using launcherjar = @{launcherjar}</echo>
<java jar="@{launcherjar}"
      fork="true" timeout="10800000" jvm="${java.home}/bin/java" failonerror="true"
      maxmemory="256m" taskname="p2"
>
        <classpath>
                <fileset dir="${builder.build.path}/plugins"
                         includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
                />
                <fileset dir="${clean.eclipse.home}/plugins"
                         includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar"
                />
                <pathelement location="${builder.build.path}/plugins" />
                <pathelement location="${clean.eclipse.home}/plugins" />
        </classpath>
        <arg line=" org.eclipse.equinox.launcher.Main -consolelog -application org.eclipse.equinox.p2.publisher.UpdateSitePublisher"
        />
        <arg line=" -metadataRepository file:${updateSiteJarDir}/ -metadataRepositoryName "${update.site.product.name} ${update.site.description} Update Site""
        />
        <arg line=" -artifactRepository file:${updateSiteJarDir}/ -artifactRepositoryName "${update.site.product.name} ${update.site.description} Artifacts""
        />
        <arg line=" -source ${updateSiteJarDir}/" />
        <arg line=" -compress -publishArtifacts -reusePack200Files -configs *,*,*" />
</java>

Or, put your p2.inf file in the same directory as your build.properties ...

product=${builderDirectory}/jbds-all.product
runPackager=true

p2.gathering=true
p2.category.site=file:${builderDirectory}/site.xml
# locations.  Don't need a baseLocation, the transformedRepoLocation will have what we need
buildDirectory=${product.build.directory}/jbds-all-package
transformedRepoLocation=${product.build.directory}/jbds-all-package/transformed
repoBaseLocation=${product.build.directory}/jbds-all-package/toTransform

# The prefix that will be used in the generated archive.
archivePrefix=studio

# The location underwhich all of the build output will be collected.
collectingFolder=${archivePrefix}

# The list of {os, ws, arch} configurations to build.
configs = linux,gtk,x86 & win32,win32,x86 & linux,gtk,x86_64 & macosx,cocoa,x86 & macosx,cocoa,x86_64

buildId=${product.name}-product-${versionTag}
buildLabel=${buildId}

skipBase=true
skipMaps=true
skipFetch=true

... and your .product file ...

<?xml version="1.0" encoding="UTF-8"?>
<?pde version="3.5"?>

<product name="JBoss Developer Studio for Web and SOA Development" uid="com.jboss.jbds.all" id="com.jboss.jbds.product.product" application="org.eclipse.ui.ide.workbench" version="4.0.0.qualifier" useFeatures="true" includeLaunchers="true">

   <configIni use="default">
   </configIni>

   <launcherArgs>
      <programArgs>--launcher.XXMaxPermSize
256m
--launcher.defaultAction
openFile</programArgs>
      <vmArgs>-Xms512m
-Xmx1024m
-Dosgi.bundles=reference:file:org.eclipse.equinox.simpleconfigurator_1.0.200.v20100503.jar@1:start
-Dosgi.instance.area.default=@user.home/workspace</vmArgs>
      <vmArgsMac>-XstartOnFirstThread -Dorg.eclipse.swt.internal.carbon.smallFonts
-Xdock:icon=../Resources/JBDevStudio.icns</vmArgsMac>
   </launcherArgs>

   <windowImages/>

   <splash
      location="com.jboss.jbds.product" />
   <launcher name="jbdevstudio">
      <solaris/>
      <win useIco="true">
         <ico path="jbds.ico"/>
         <bmp/>
      </win>
   </launcher>

   <vm>
   </vm>

   <plugins>
   </plugins>

   <features>
      <feature id="com.jboss.jbds.product.feature" version="4.0.0.qualifier"/>
   </features>


</product>

... and when generating a product using PDE, that file and its instructions should be read at the correct time.

Hope this series has been helpful! If you have any examples of what you've done with .product or p2.inf files, please feel free to send me a link to your post or the file in your cvs, svn, or git repo. I'd love to see what else you can do with p2 and product builds.

See also:

2011-02-13

Simplifying The p2 Process, Part 3: Associate Sites

In Part 1 of this series, I looked at use of composite repos to provide a way of combining update sites into a single URL for ease of use and a single point of entry from which to do updates.

In Part 2, I discussed why we switched from using a collection of SDKs against which to build - using the now-deprecated brute-force "just unzip into eclipse root folder or dropins" approach - to using a single target platform update site so as to simplify maintenance and provide a reusable artifact for both build and workspace provisioning.


Now, let's look at the no-brainer that says that "less is more" when it comes to telling p2 from where to get updates, and that less effort for your user when installing is always a win.

If this sounds familar, I did blog about this briefly back in August. Since then, we've also added a quick XSLT script to remove the "Uncategorized" category that Tycho automatically adds for features which are listed in your site.xml but are not associated with a category. While this isn't strictly related to associate sites, it is about ease of use; while I applaud the desire to have everything belong to a category bucket (perhaps because the model Tycho's using requires it), the reason we'd rather hide these is to declutter the install view and not confuse people by suggesting features that won't work on their OS (eg., for which there's no XulRunner port).

But I digress...

Associate Sites

In the old days of yore, you could situate an associateSites.xml next to your site.xml in your "classic" update site, and Eclipse Update Manager would happily read that file and add those extra sites to your list of available update sites.

Then came p2, and while the old way still worked, it was no longer ideal. So, the new approach was to insert these associate sites directly into the p2 metadata for the site, content.xml and artifacts.xml (or content.jar and artifacts.jar).

This could be accomplished via a somewhat hacky appraoch - unpacking the existing metadata (content.jar) and shoehorning in the information at the bottom of the content.xml file, using an ant script (see "add.associate.sites" target, below) and a list of sites to be added:

<target name="add.associate.sites" if="associate.sites">
        <if>
                <and>
                        <!-- Defined in aggregateSite.properties -->
                        <isset property="associate.sites" />
                        <not>
                                <equals arg1="${associate.sites}" arg2="" />
                        </not>
                </and>
                <then>
                        <if>
                                <available file="${update.site.source.dir}/content.jar" type="file" />
                                <then>
                                        <unzip src="${update.site.source.dir}/content.jar" dest="${update.site.source.dir}" />
                                        <delete file="${update.site.source.dir}/content.jar" />
                                </then>
                        </if>
                        <!-- counter variable -->
                        <var name="associate.sites.0" value="" />
                        <for param="associate.site" list="${associate.sites}" delimiter=", 
">
                                <sequential>
                                        <var name="associate.sites.0" value="${associate.sites.0}00" />
                                </sequential>
                        </for>
                        <length property="associate.sites.length" string="${associate.sites.0}" />

                        <loadfile srcfile="${update.site.source.dir}/content.xml" property="content.xml">
                                <filterchain>
                                        <tailfilter lines="-1" skip="1" />
                                </filterchain>
                        </loadfile>
                        <echo file="${update.site.source.dir}/content.xml" message="${content.xml}" />
                        <echo file="${update.site.source.dir}/content.xml" append="true">  <references size='${associate.sites.length}'>
</echo>
                        <for param="associate.site" list="${associate.sites}" delimiter=", 
">
                                <sequential>
                                        <!-- insert into content.xml -->
                                        <echo file="${update.site.source.dir}/content.xml" append="true">    <repository uri='@{associate.site}' url='@{associate.site}' type='0' options='1'/>
<repository uri='@{associate.site}' url='@{associate.site}' type='1' options='1'/>
</echo>
                                </sequential>
                        </for>
                        <echo file="${update.site.source.dir}/content.xml" append="true">  </references>
</repository>
</echo>
          <!--  
    workaround for Tycho bug: uncategorized features in site.xml are put into
    "Uncategorized" category, rather than just being uncategorized (hidden) 
   -->
                 <copy file="${update.site.source.dir}/content.xml" tofile="${update.site.source.dir}/content.old.xml" overwrite="true" />
                        <xslt style="remove-uncategorized.xsl" in="${update.site.source.dir}/content.old.xml" out="${update.site.source.dir}/content.xml" />
                        <zip destfile="${update.site.source.dir}/content.jar" basedir="${update.site.source.dir}" includes="content.xml" />
                        <delete file="${update.site.source.dir}/content.xml" />
                        <delete file="${update.site.source.dir}/content.old.xml" />
                </then>
        </if>
</target>

So, now, instead of telling people to add multiple update sites to resolve missing potentially dependencies when installing, we can cause those extra sites to be automatically added at the same time they add the single URL for JBoss Tools. Now the additional sites need only be listed for reference, but no additional effort is required by the user.

BONUS HACK: to force a site that may already be listed (but disabled) to be added again, and this time definitely be enabled, you can add an extra slash into URL. Thus http://download.eclipse.org/birt/update-site/2.6 becomes http://download.eclipse.org//birt/update-site/2.6/, and as p2 sees a new site, it adds the new site (instead of ignoring it because it's already present but disabled. Again, a win.

Alternatively, you could cast an arcane spell using a p2.inf file in your feature's root folder or plugin's META-INF/ folder to add these additional, required sites... or do whatever processing you might need. I'm not sure if Tycho supports this yet, or how fully PDE supports reading this information. Got sample code? Send it to me as a comment below or via twitter to @nickboldt. Thanks!


In part 4, I'll talk a little about how to prevent your product build from getting updates from unofficial sources, and preload your product with the official sites from which to get updates. Because it's important to balance ease of use with prevention of unsupported features. SPOILER ALERT: may contain p2.inf instructions.

2011-01-29

Simplifying The p2 Process, Part 1: p2 Composite Repos

With the release of JBoss Tools 3.2 and JBoss Developer Studio 4.0 just around the corner, you may be thinking to yourself, "Self, how many update sites and SDK zips and runtimes will I need to download THIS time?"

Or maybe you're thinking, "Self, why is this so damn complicated?"

Well, folks, we heard your kvetching and we did something about it.

Composite Repos

While this is not a new concept to many, we embraced the composite update site this past year and it's made life a lot easier for iterative, agile development cycles. Last year, JBoss Tools 3.1 was built as a single Hudson job, with a second one for JBoss Developer Studio. This meant that any change in any of the components would cause a build to be launched, and 4-6hrs later, we'd have fresh bits. Yeah, far from ideal.

This year, we split up the monolith (and added a few new components!) so that now we have 34 update sites to compose into a single one against which builds can then be built. This composite update site looks like this:

compositeArtifacts.xml

<?xml version='1.0' encoding='UTF-8'?>
<?compositeArtifactRepository version='1.0.0'?>
<repository name='JBoss Tools Staging Repository' 
  type='org.eclipse.equinox.internal.p2.artifact.repository.CompositeArtifactRepository' 
  version='1.0.0'>
<properties size='2'>
<property name='p2.compressed' value='true'/>
<!-- get new time w/ `date +%s000` -->
<property name='p2.timestamp' value='1294205433000'/>
</properties>
<children size='34'>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-3.2_trunk.component--archives/all/repo/'/>
...
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-3.2_trunk.component--ws/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-pi4soa-3.1_trunk/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-teiid-designer-7.1_trunk/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-drools-5.2_trunk/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-savara-1.1_trunk/tools/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/xulrunner-1.9.1.2/all/repo/'/>
</children>
</repository>

compositeContent.xml

<?xml version='1.0' encoding='UTF-8'?>
<?compositeMetadataRepository version='1.0.0'?>
<repository name='JBoss Tools Staging Repository' 
  type='org.eclipse.equinox.internal.p2.metadata.repository.CompositeMetadataRepository' 
  version='1.0.0'>
<properties size='2'>
<property name='p2.compressed' value='true'/>
<!-- get new time w/ `date +%s000` -->
<property name='p2.timestamp' value='1294205433000'/>
</properties>
<children size='34'>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-3.2_trunk.component--archives/all/repo/'/>
...
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-3.2_trunk.component--ws/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-pi4soa-3.1_trunk/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-teiid-designer-7.1_trunk/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-drools-5.2_trunk/all/repo/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-savara-1.1_trunk/tools/'/>
<child location='http://download.jboss.org/jbosstools/builds/staging/xulrunner-1.9.1.2/all/repo/'/>
</children>
</repository>

So, now that JBoss Tools is built in 34 pieces, the bits that haven't changed aren't rebuilt over and over and builds are faster. If that sounds insanely obvious to you, well, we used to have a lot of inter-component cyclic dependencies. We eliminated those early in the development cycle for JBoss Tools 3.2, and have been able to build smarter and faster ever since.

Added benefits to this composite site are:

  • Newly built and published bits are instantly available from the composite site - sure, the same was true under last year's PDE "uberbuild" regime, but that's because everything was built fresh every time, which was slow and near-impossible to get people to run at home.

  • Developers can use this site to install latest updates to components they're interested in testing - again, this was true before; but now using the same site and searching for updates, developers and beta testers can get incremental updates to the components that have actually changed, rather than having to pull down 160M every day to get a few K of changes.

  • Tycho can be pointed at this site (see below) in order to resolve binary p2 dependencies, so building a component deep in the dependency chain can be done w/o having to first build its upstream dependencies - this wasn't a concern before because everything was built from source every time, so by definition everything was already on disk. But now, if a developer only cares about a single component, like ModeShape or GWT, they need only have that source (and some bootstrapping code) on disk. Smaller, faster, more agile. And way more likely to be built locally before checking in code than before, making the painful "who broke what and when?" process much less painful. Fewer moving pieces and local dev builds at home mean - in theory - fewer incomplete or breaking commits.

When we first moved to Tycho, we needed to build a series of components locally in order to just get to a deep component. For example, the Struts component needs VPE, which needs JST and XulRunner. JST also needs the Common component, which in turn needs the Tests component.

So, to build Struts locally, 5 other components would have to be built locally first. This worked, but was still a fairly large barrier to entry for most developers (much less contributors!)

But with this new composite site, building Struts can be done without this lengthy bootstrapping; instead we just point Tycho at this composite site, and it pulls down the 5 upstream components' jars from this p2 repo - because the upstream deps are already built in Hudson.

Here's what we added to our parent pom.xml to have the builds find the binaries:

<repository>
        <id>jbosstools-nightly-staging-composite-trunk</id>
        <url>http://path.to.the.site/staging/_composite_/trunk/ </url>
        <layout>p2</layout>
        <snapshots>
                <enabled>true</enabled>
        </snapshots>
        <releases>
                <enabled>true</enabled>
        </releases>
</repository>

So, using this composite update site, we can use Maven 3 with Tycho 0.10 to generate a single update site (staged here, then ultimately published here).


In part 2, I'll look at why we switched from using a collection of SDKs (Eclipse, EMF, DTP, GEF, M2E, RSE, TPTP, UMl2, WTP, XSD and more) against which to build - using the now-deprecated brute-force "just unzip into eclipse root folder or dropins" approach - to using a single target platform update site. SPOILER ALERT: Easier to update and maintain.

In part 3, I'll look back at the success we've had using associate sites instead of asking people to manually add 3rd party URLs when installing JBoss Tools. SPOILER ALERT: one URL is easier for people to use than 6.

In part 4, I'll talk a little about how to prevent your product build from getting updates from unofficial sources, and preload your product with the official sites from which to get updates. Because it's important to balance ease of use with prevention of unsupported features. SPOILER ALERT: may contain p2.inf instructions.

By the way, JBoss Tools 3.2.0.CR1 and JBoss Developer Studio 4.0.0.CR1 are available. Get 'em while they're hot (and sourceforge is not).

2011-01-27

HOWTO: partially clone an SVN repo to Git, and work with branches

Skip to the code

I've blogged a few times now about Git (which I pronounce with a hard 'g' a la "get", as it's supposed to be named for Linus Torvalds, a self-described git, but which I've also heard called pronounced with a soft 'g' like "jet"). Either way, I'm finding it way more efficient and less painful than either CVS or SVN combined.

So, to continue this series ([1], [2], [3]), here is how (and why) to pull an SVN repo down as a Git repo, but with the omission of old (irrelevant) revisions and branches.

Using SVN for SVN repos

In days of yore when working with the JBoss Tools and JBoss Developer Studio SVN repos, I would keep a copy of everything in trunk on disk, plus the current active branch (most recent milestone or stable branch maintenance). With all the SVN metadata, this would eat up substantial amounts of disk space but still require network access to pull any old history of files. The two repos were about 2G of space on disk, for each branch. Sure, there's tooling to be able to diff and merge between branches w/o having both branches physically checked out, but nothing beats the ability to place two folders side by side OFFLINE for deep comparisons. So, at times, I would burn as much as 6-8G of disk simply to have a few branches of source for comparison and merging. With my painfullly slow IDE drive, this would grind my machine to a halt, especially when doing any SVN operation or counting files / disk usage.

Using Git for SVN repos naively

Recently, I started using git-svn to pull the whole JBDS repo into a local Git repo, but it was slow to create and still unwieldy. And the JBoss Tools repo was too large to even create as a Git repo - the operation would run out of memory while processing old revisions of code to play forward.

At this point, I was stuck having individual Git repos for each JBoss Tools component (major source folder) in SVN: archives, as, birt, bpel, build, etc. It worked, but replicating it when I needed to create a matching repo-collection for a branch was painful and time-consuming. As well, all the old revision information was eating even more disk than before:

  • jbosstools' trunk as multiple git-svn clones: 6.1G
  • devstudio's trunk as single git-svn clone: 1.3G

So, now, instead of a couple Gb per branch, I was at nearly 4x as much disk usage. But at least I could work offline and not deal w/ network-intense activity just to check history or commit a change. Still, far from ideal.

Cloning SVN with standard layout & partial history

This past week, I discovered two ways to make the git-svn experience at least an order of magnitude better:

  1. Standard layout (-s) - this allows your generated Git repo to contain the usual trunk, branches/* and tags/* layout that's present in the source SVN repo. This is a win because it means your repo will contain the branch information so you can easily switch between branches within the same repo on disk. No more remote network access needed!
  2. Revision filter (-r) - this allows your generated Git repo to start from a known revision number instead of starting at its birth. Now instead of taking hours to generate, you can get a repo in minutes by excluding irrelevant (ancient) revisions.

So, why is this cool? Because now, instead of having 2G of source+metadata to copy when I want to do a local comparison between branches, the size on disk is merely:

  • jbosstools' trunk as single git-svn clone w/ trunk and single branch: 1.3G
  • devstudio's trunk as single git-svn clone w/ trunk and single branch: 0.13G

So, not only is the footprint smaller, but the performance is better and I need never do a full clone (or svn checkout) again - instead, I can just copy the existing Git repo, and rebase it to a different branch. Instead of hours, this operation takes seconds (or minutes) and happens without the need for a network connection.


Okay, enough blather. Show me the code!

Check out the repo, including only the trunk & most recent branch

# Figure out the revision number based on when a branch was created, then 
# from r28571, returns -r28571:HEAD
rev=$(svn log --stop-on-copy \
  http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x \
  | egrep "r[0-9]+" | tail -1 | sed -e "s#\(r[0-9]\+\).\+#-\1:HEAD#")

# now, fetch repo starting from the branch's initial commit
git svn clone -s $rev http://svn.jboss.org/repos/jbosstools jbosstools_GIT

Now you have a repo which contains trunk & a single branch

git branch -a # list local (Git) and remote (SVN) branches

  * master
    remotes/jbosstools-3.2.x
    remotes/trunk

Switch to the branch

git checkout -b local/jbosstools-3.2.x jbosstools-3.2.x # connect a new local branch to remote one

  Checking out files: 100% (609/609), done.
  Switched to a new branch 'local/jbosstools-3.2.x'

git svn info # verify now working in branch

  URL: http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x
  Repository Root: http://svn.jboss.org/repos/jbosstools

Switch back to trunk

git checkout -b local/trunk trunk # connect a new local branch to remote trunk

  Switched to a new branch 'local/trunk'

git svn info # verify now working in branch

  URL: http://svn.jboss.org/repos/jbosstools/trunk
  Repository Root: http://svn.jboss.org/repos/jbosstools

Rewind your changes, pull updates from SVN repo, apply your changes; won't work if you have local uncommitted changes

git svn rebase

Fetch updates from SVN repo (ignoring local changes?)

git svn fetch

Create a new branch (remotely with SVN)

svn copy \
  http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x \
  http://svn.jboss.org/repos/jbosstools/branches/some-new-branch

2010-10-08

HOWTO: Find the feature that contains a plugin

Tycho is awesome.

However, like all build systems, it has its limitations.

One such limitation is that when you're building against a target platform, and something's missing, you get errors such as these:

[INFO] Cannot complete the request.  Generating details.
{org.osgi.framework.executionenvironment=OSGi/Minimum-1.0,OSGi/Minimum-1.1, osgi.ws=cocoa, osgi.arch=x86, osgi.os=macosx, org.eclipse.update.install.features=true, org.osgi.framework.system.packages=}
[Software being installed: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier, Missing requirement: org.eclipse.tptp.platform.instrumentation.ui 4.4.1.v201009092123 requires 'bundle org.eclipse.hyades.probekit [4.2.0,5.0.0)' but it could not be found, Cannot satisfy dependency: org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.1.v201009092123-797908s73533D4H6D56 depends on: org.eclipse.tptp.platform.instrumentation.ui [4.4.1.v201009092123], Cannot satisfy dependency: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier depends on: org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.0]
[ERROR] Internal error: java.lang.RuntimeException: org.eclipse.equinox.p2.core.ProvisionException: No solution found because the problem is unsatisfiable. -> [Help 1]
org.apache.maven.InternalErrorException: Internal error: java.lang.RuntimeException: org.eclipse.equinox.p2.core.ProvisionException: No solution found because the problem is unsatisfiable.

The important part of that error message is as follows:

org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier
   requirement: org.eclipse.tptp.platform.instrumentation.ui 4.4.1.v201009092123 
      requires 'bundle org.eclipse.hyades.probekit [4.2.0,5.0.0)' 
         but it could not be found

org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.1.v201009092123-797908s73533D4H6D56 
   depends on: org.eclipse.tptp.platform.instrumentation.ui [4.4.1.v201009092123]
     dependency: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier 
        depends on: org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.0]

So, how do you find which feature contains that plugin, so that you can add it to your target platform?

First, you need access to the repository. If you have direct server access to the repository from which the plugin comes (eg., the TPTP Helios update site), you can run this script in the root of that repository.

If you don't have server access (eg., you can't ssh to dev.eclipse.org and look in ~/downloads/tptp/updates/helios), then you can pull down a zip of the site (or use a p2.mirror script to fetch a copy of the site to your local machine)... and then run this script in the root of that repository.

Essentially the script finds matching feature jar files, unpacks them to extract the feature.xml files therein, and then greps those files for lines which suggest an included plugin matching the pattern for which you're searching:

$ findInFeature platform.probekit
./features/org.eclipse.tptp.platform.probekit_4.5.1.v201009092123-7H7BF8PAkF7B77ZARCNEK.jar
   <plugin
         id="org.eclipse.tptp.platform.probekit"

./features/org.eclipse.tptp.platform.trace_4.5.1.v201009092123-7L7O8bBgJ9E99jAfGWEM.jar
   <plugin
         id="org.eclipse.tptp.platform.probekit.launch"
From there, it's a trivial exercise to add another line item into your target platform file. First, paste in the feature jar:
./features/org.eclipse.tptp.platform.probekit_4.5.1.v201009092123-7H7BF8PAkF7B77ZARCNEK.jar

Then use vim to pattern-replace that string:

:%s/.\+\/\(org.\+\)_\(\d\+.\+\)\.jar/\t\t\t/g

And you'll end up with a new .feature.group added to the target:

<unit version="4.5.1.v201009092123-7H7BF8PAkF7B77ZARCNEK" id="org.eclipse.tptp.platform.probekit.feature.group"/>

2010-10-01

JBoss Tools: making it easier to build against a complex target platform

So you want to be a JBoss Tools developer? Awesome. Welcome to the family. SVN sources are here, JIRA's over here and there's cold beer in the fridge*.

But you say it's a pain in the tuchus to download over 25 zips or add a whole bunch of update sites and hope you get everything you need? Yeah, no argument there. If only there was an easier way to resolve all the dependencies you need to get building, much less to even RUN this stuff.

To make this process simpler, I've created a p2 repo (update site) from our target platform file, which has been recently updated to include Helios SR1 dependencies. You can track subsequent work in progress here: JIRA JBIDE-6982. You can also report any issues there too.


So, now, just add this single site** into your vanilla Eclipse 3.6.1 Classic (or a Helios SR1 bundle), uncheck the box for 'Group Items by Category' and you can install everything listed. For great justice.


Some handy links:


Some handy HOWTOs:


* - Due to beer2peer limitations, YMMV.

** - I'm aware that the update site throws a 403 if you open it in a browser. I can't be arsed to generate an index.html just yet, nor are there categorized features. Because really, you don't need either - this site is only meant to be used by p2.

2010-08-11

p2 Repository Association And The Fine Art Of Forceably Enabling Disabled Sites 2: Tycho Edition

Back in February, I blogged about how to add associate sites to an update site generated w/ the Buckminster or B3 aggregator.

Since we've moved our build infrastructure to use Tycho + Maven, I had to port the script over to work there for our update site aggregation.

Here are the moving pieces:

  • pom.xml that builds the site, listing input sites as <repository> entries, and a shout-out to build.xml, below, to do extra stuff during the install phase
  • build.xml ant script called by pom.xml, above, to do additional work after the creation of the site_assembly.zip update site zip
  • aggregateSite.jbosstools.properties properties file listing the aggregate sites to add

2010-06-30

Update Site Aggregation: B3 vs. Tycho

Last year, I used to aggregate update sites with the Buckminster Aggregator, but since that won't install into Eclipse 3.6 (Helios), I had to migrate to the B3 Aggregator. This new version of the Aggregator is greatly expanded and worked fine until recently, when it has begun to suffer from a rather nasty p2 problem: instead of the Director just installing the aggregator into an Eclipse instance prior to then running the aggregation, I get "The copies of profile SDKProfile are not in sync," and the whole process dies. http://www.blogger.com/img/blank.gif

So, in order to find a workaround, I went back to Tycho, and discovered you can merge update sites with little more than two simple files:

  • a site.xml, which lists the features to aggregate and how to categorize them, and
  • a pom.xml to list the source sites and drive the aggregation.

Unfortunately the Tycho solution doesn't include the ability to add associate sites to the metadata after generation, but that can simply be done as a downstream step (Tycho can call Ant using the maven-antrun-plugin).

2010-03-04

Athena: p2 repos w/ associate sites

Want to insert associate update sites into your user's Eclipse when they go to install your features from an update site/zip? With an Athena build this is now a trivial exercise. For example... say you're building a plugin which supports integration with Subclipse or Subversive. In a traditional install from an SDK or runtime zip, you'd unpack into eclipse/, and wonder why the new plugins didn't work. In the new p2 install world, you point Eclipse at the update site or zip, and are told that you cannot install the SVN integration because missing requirements cannot be found. So you hunt down the required sites, add them to Eclipse, and try again; this time you are allowed to install the new features. To improve the ease of use for the end user, Athena now allows you to define associate.sites in your build.properties. If this property is set, URLs will be added into your p2 repo's metadata and Eclipse will automatically add those sites when it scans your zip or site. No more manual copy+paste from documentation needed!

associate.sites=\
http://subclipse.tigris.org/update_1.6.x,\
http://download.eclipse.org/technology/subversive/0.7/update-site/,\
http://community.polarion.com/projects/subversive/download/eclipse/2.0/update-site/
And, of course, you can reuse these sites when resolving dependencies against which to build or test:
repositoryURLs=\
http://download.eclipse.org/releases/galileo/,\
http://download.eclipse.org/athena/repos/eclipse-Update-3.5.2-201002111343.zip,\
http://download.eclipse.org/eclipse/updates/3.5/,\
${associate.sites}

IUsToInstall=org.eclipse.sdk.feature.group+org.eclipse.sdk.ide+\
org.eclipse.team.svn+org.eclipse.team.svn.core+org.eclipse.team.svn.ui+\
org.tigris.subversion.subclipse.core+org.tigris.subversion.subclipse.ui

2009-11-27

Dash Athena: More Ant, More Tests, More Repos!

  • Infrastructure Changes
  • Cross-Platform / Ease of Use
  • bug 295670 Support running JUnit tests run from Ant script instead of Bash - tests can now be run on Linux and Windows (and to a lesser extent on Mac OSX) using new testLocal build step
  • Bug Fixes
  • bug 292486 Allow builds to fail if unit tests fail using failBuildIfTestFailuresOrErrors=true
  • bug 294678 Categories don't show up with IBM 1.6 JDK - implemented workaround so a different JDK can be used for p2 operations than for compilation of sources: use PACK200_JAVA_HOME=/path/to/different/JDK
  • bug 295773 Non-incubating projects no longer need to set "incubation=" in their build.properties
  • bug 292235 Included pre-compiled binary features/plugins are now included by default using PDE's runPackager property; can disable this behaviour with packageFeaturesIncludedBinaries=false
  • Documentation
  • Testing - Different ways to run or re-run tests, including as a secondary process after a build (so they can be run on a different platform or by a different user)
p2 does not natively support remote repo zips; to workaround this the zip is fetched and the URL is rewritten from http://path/to/foo.zip to jar:file:/tmp/path/foo.zip!/

Previous New & Noteworthy | More Athena Docs

2009-08-13

We Don't Need Another Repo

Re Wayne's blog about an Eclipse.org Maven Repo:

EMF has had an undocumented/unmarketed Maven2 repo for about 4 years now.

All you need is to take an update site zip, unpack it, rearrange the folder structure, and rename the jars. Then you create little XML files called .poms to describe the jars in the tree, and Maven-aware tools can read the tree. It's fairly trivial. http://download.eclipse.org/modeling/emf/emf/maven2/ is the URL, IIRC. About once every 2 years someone asks about our providing such a repo, and I give out the URL. Clearly not a huge demand for it.

One might argue that creation of such a folder structure is in the purview of Athena's publishing scripts, which today eases the process of copying your bits to download.eclipse.org, then unpacks your Update site so it can be scanned by p2 rather than downloaded as a single archive. It too is fairly trivial. I would not be adverse to converting my existing shell script for the EMF repo creation into a generic Ant script for use by Athena users.

Frankly though, I think it would be more valuable if the m2eclipse folks added support for reading/converting p2 repos. Publishing yet another file format would require another release-train-like workflow (we already have two: EPP and buckybuilder for Galileo) and more people maintain it. Even if every project published their own maven repo, we'd want for the sake of ease of use to aggregate them into a central place for easier navigation and discovery by maven tools. So, like with Ganymede, we'd have each project's bits copied to two places on disk for each build. (Galileo used composite repos to POINT at project repos rather than copying them saving tons of disk space and CPU cycles. AFAIK, Maven does not support this concept, but I could be wrong.)

There's also another benefit to having tooling to support converting from p2 repo to maven2 repo: the aggregate repo could be housed at apache.org and suck THEIR bandwidth and support resources instead. Thus just as Eclipse.org is upstream from Fedora's Eclipse project .rpms (which are upstream from Debian/Ubuntu's .debs), Eclipse.org p2 repos could be upstream from Apache's Maven repo(s). After all, Apache already collects maven artifacts for non-apache.org projects to facilitate the use and adoption of maven, so this is entirely in line with their standard operating procedures.

Thoughts?

Posted from Blackberry using Opera Mini, by the side of Shawnigan Lake, Vancouver Island, BC