MANIFEST.MF and feature.xml versioning rules

I'm forever forgetting what the rules are for dependency declarations in MANIFEST.MF and feature.xml for osgi plugins and features. And Googling often results in frustration rather than an answer. So, because today I actually found a concise list of the rules, I thought I'd repost them here, with some minor edits to help clarify.

OSGi Plugin Version Ranges

Dependencies on bundles and packages have an associated version range which is specified using an interval notation: a square bracket “[” or “]” denotes an inclusive end of the range and a round bracket “(” or “)” denotes an exclusive end of the range. Where one end of the range is to be included and the other excluded, it is permitted to pair a round bracket with a square bracket. The examples below make this clear.

If a single version number is used where a version range is required this does not indicate a single version, but the range starting from that version and including all higher versions.

There are four common cases:

• A “strict” version range, such as [1.2.3,1.2.3], which denotes that version and only that version.

• A “half-open” range, such as [1.2.3,2.0.0), which has an inclusive lower limit and an exclusive upper limit, denoting version 1.2.3 and any version after this, up to, but not including, version 2.0.0.

• An “unbounded” version range, such as 1.2.3, which denotes version 1.2.3 and all later versions.

• No version range, which denotes any version will be acceptable. NOT RECOMMENDED.

The complete text of the above snippet can be seen here (or here as PDF).

Example:

Require-Bundle: org.eclipse.core.runtime;bundle-version="[3.4.0,4.0.0)",
 org.eclipse.core.resources;bundle-version="[3.4.0,4.0.0)",
 org.eclipse.ui.ide;bundle-version="[3.4.0,4.0.0)",
 org.eclipse.ui.navigator;bundle-version="3.5.100",
 com.ibm.icu

See also:

• plugin manifest (plugin.xml)
• osgi bundle manifest (MANIFEST.MF)

---

In terms of feature manifest (feature.xml) rules, help.eclipse.org has pretty good documentation, but the most important thing to remember - and what I often have to look up - is how to state the matching rules for required upstream features & plugins. Experience says it's always better to state things explicitly so there's no downstream guesswork needed and anyone reading your manifest knows EXACTLY what version(s) are required for or compatible with your feature. Plus, while YOU might be using PDE UI to build, someone else might be using Tycho and Maven, and every tool can interpret missing metadata their own way.

When in doubt, spell it out.

Valid values and processing are as follows:

• if version attribute is not specified, the match attribute (if specified) is ignored.
• perfect - dependent plug-in version must match exactly the specified version. If "patch" is "true", "perfect" is assumed and other values cannot be set. [1.2.3,1.2.3]
• equivalent - dependent plug-in version must be at least at the version specified, or at a higher service level (major and minor version levels must equal the specified version). [1.2.3,1.3)
• compatible - dependent plug-in version must be at least at the version specified, or at a higher service level or minor level (major version level must equal the specified version). [1.2.3,2.0)
• greaterOrEqual - dependent plug-in version must be at least at the version specified, or at a higher service, minor or major level. 1.2.3

The complete text of the above snippet can be seen here.

Example:

<requires>
  <import feature="org.eclipse.m2e.feature" version="1.0.0" match="compatible"/>
  <import feature="org.maven.ide.eclipse.wtp.feature" version="0.13.0" match="greaterOrEqual"/>

  <plugin id="ch.qos.logback.classic" version="0.9.27.v20110224-1110" match="greaterOrEqual"/>
  <plugin id="ch.qos.logback.core" version="0.9.27.v20110224-1110" match="greaterOrEqual"/>
  <plugin id="ch.qos.logback.slf4j" version="0.9.27.v20110224-1110" match="greaterOrEqual"/>
  <plugin id="org.slf4j.api" version="1.6.1.v20100831-0715" match="compatible"/>
  <plugin id="com.ning.async-http-client" version="1.6.3.201106061504" match="equivalent"/>
  <plugin id="org.jboss.netty" version="3.2.4.Final-201106061504" match="perfect"/>
  <plugin id="org.hamcrest.core" version="1.1.0.v20090501071000" match="equivalent"/>
</requires>

Visualizing OSGi Dependencies

Yesterday I blogged about how to find dependencies in features on plugins or features using a shell script to rip through feature jars.

But maybe you're less commandline, and more visual? Well, it may be over three years old, but there's a way to visualize plugin interdependencies using Ian Bull's PDE Dependency View. Frankly, I'm amazed this isn't already a core feature in PDE (and correct me if it is).

To use this tool, simply install it from:

http://download.eclipse.org/eclipse/pde/incubator/visualization/site

After installing and restarting, hit CTRL-3 and type "Graph" to find the "Graph Plug-In Dependencies View". (It's also available from Window > Show View > Other... (ALT-SHIFT-Q,Q) under Plug-in Development, if you prefer to kick it old-school.)

Next, right-click in the view or hit the "Focus on" button in the view, and select the plugin on which you want to focus.

Now you can browse up or down through plugins to explore dependencies.

For example, to see what plugins depend on a given plugin, such as org.eclipse.tm.terminal, click the "Show Callers" button in the view.

Or, to see on which plugins org.jboss.ide.eclipe.as.rse.ui depends, click the "Show Callees" button in the view. You can shift-click on nodes to highlight them for emphasis, or click and drag them around.

HOWTO: Find osgi dependencies in features

Say you're trying to build something with Tycho & Maven 3 and while resolving dependencies before compilation, you're told:

[INFO] [Software being installed: 
  org.eclipse.tm.terminal.local.feature.group 0.1.0.v201006041240-10-7w312117152433, 
  Missing requirement: 
    org.eclipse.tm.terminal.local 0.1.0.v201006041322 
      requires 
        'bundle org.eclipse.cdt.core 5.2.0' 
          but it could not be found, 
  Cannot satisfy dependency: 
    org.eclipse.tm.terminal.local.feature.group 0.1.0.v201006041240-10-7w312117152433 
      depends on: 
        org.eclipse.tm.terminal.local [0.1.0.v201006041322]]

To quickly verify where this dependency is coming from, you can go look into the feature.xml for the org.eclipse.tm.terminal.local feature jar... but if you don't have it installed, this is somewhat more cumbersome; besides, you then have to unpack the jar before you can look inside it.

And maybe that feature contains a number of OTHER dependencies that you'll also need to resolve in your target platform when building. Sure, there are UI tools to do this within Eclipse, but when you're working on remote servers sometimes UI isn't available.

Workaround? Assuming you have a mirror of the update site(s) from which you're trying to resolve the dependency (eg., Helios) or can ssh to dev.eclipse.org, you can simply run a quick shell script to do the investigative work for you:

$ cd ~/downloads/releases/helios/201009240900/aggregate/; ~/bin/findDepInFeature "*tm*" cdt

./features/org.eclipse.tm.terminal.local_0.1.0.v201006041240-10-7w312117152433.jar
      <import feature="org.eclipse.cdt.platform" version="7.0.0" match="greaterOrEqual"/>
      <import plugin="org.eclipse.cdt.core" version="5.2.0" match="compatible"/>
      <import plugin="org.eclipse.core.runtime"/>

Where the script looks like this:

#!/bin/bash
# find plugins/feature deps by searching in some folder for feature jars, and searching through their feature.xml files for dependencies

# 1 - featurePattern - pattern of features to search (eg., "org.eclipse.tptp" or "\*" for all features)
# 2 - dependencyPattern  - pattern of plugins/feature deps for which to search (eg., "org.eclipse.tptp.platform.instrumentation.ui")
# 3 - location       - directory in which to search, if not "."

if [[ ! $1 ]]; then
        echo "Usage: $0 <featurePattern> <dependencyPattern> <location>"
        echo ""
        echo "Example: $0 tm.terminal cdt"
        exit 1
fi

# if no location, look in current dir (.)
if [[ $3 ]]; then location="$3"; else location="."; fi

# if no featurePattern, search all features for dependencyPattern
if [[ ! $2 ]]; then featurePattern="*"; dependencyPattern="$1"; else dependencyPattern="$2"; featurePattern="$1"; fi

rm -fr /tmp/findinfeature/; mkdir -p /tmp/findinfeature/features/
for f in $(find "$location" -type f -name "*${featurePattern}*" | egrep -v "pack.gz|source" | grep features | egrep "${featurePattern}"); do
        #echo "$f [$featurePattern, $dependencyPattern]"
        unzip -q $f -d /tmp/findinfeature/ feature.xml
        #       <import feature="org.eclipse.cdt.platform" version="7.0.0" match="greaterOrEqual"/>
        #       <import plugin="org.eclipse.cdt.core" version="5.2.0" match="compatible"/>
        if [[ ! $(cat /tmp/findinfeature/feature.xml | egrep "<import" -A3 | egrep "plugin=|feature=" -A1 -B1 | egrep "\".*${dependencyPattern}[^\"]*\"" -A1 -B1) ]]; then
                rm -fr /tmp/findinfeature/feature.xml
        else
                mv /tmp/findinfeature/feature.xml /tmp/findinfeature/${f}_feature.xml
                echo "${f}"
                cat /tmp/findinfeature/${f}_feature.xml | egrep "<import" -A3 | egrep "plugin=|feature=" -A1 -B1 | egrep "\".*${dependencyPattern}[^\"]*\"" -A1 -B1
                echo ""
        fi
        rm -fr /tmp/findinfeature/feature.xml
done

My First Maven Plugin

Having been working with Maven and Tycho for about the last 8 months, it was high time I got around to writing my first Maven plugin, because I need to break myself from my tendency to revert to ant and bash whenever I need to do something outside the normal Maven flow.

So, first I imported the git sources for Tycho 0.11.0-SNAPSHOT from https://github.com/sonatype/sonatype-tycho so I'd have something from which to learn. Sure, Hello World samples are nice, but a working example is always more fruitful.

git clone http://github.com/sonatype/sonatype-tycho.git

Next, I imported a few Tycho projects (tycho-p2-facade, tycho-p2-plugin, tycho-p2-publisher-plugin) from the git clone folder into Eclipse using m2eclipse 0.10.2. Because these are Maven projects and don't need Eclipse project clutter, there are no .project files and therefore they couldn't be imported as pre-existing projects.

So, I had to wobble through the New > Maven Project wizard a little until I figured out that I could check the 'Create a simple project (skip archetype selection)' box to create a new .project file in an existing project folder. Only wrinkle here is that if there's an existing pom.xml in a project, the New Maven Project wizard complains. The obvious workaround is to rename the existing pom.xml to pom.xml_, create the new Maven Project in w/ m2eclipse in the existing folder (as above), then replace the pom.xml that m2eclipse creates with the actual (renamed) one, pom.xml_ and refresh the project in Eclipse. Voila! I also updated the project settings to use JDK5 instead of the default JDK1.4. You need only do File > Import > Maven > Existing Maven Projects and browse for the pom files. So much easier than what I did above. Thanks to everyone for setting me straight!

So, with sample code to clone from it was time to create my own plugin project, then a second project on which that plugin depended in order to test the dependency chain thru Maven.

The first uses the "maven-archetype-plugin" archetype v1.1; the second, the "maven-archetype-quickstart" archetype v1.1. Here are the two projects:

1. sample-plugin
2. sample-project

The glue between these projects is in the sample-project's pom.xml:

<dependency>
 <groupId>org.jboss.maven.plugin</groupId>
 <artifactId>sample-plugin</artifactId>
 <version>0.0.1-SNAPSHOT</version>
</dependency>

And that's seemingly it. Next... adding actual useful functionality to a plugin, then using that functionality in our JBoss Tools build's parent pom.xml.

HOWTO: Find the feature that contains a plugin

Tycho is awesome.

However, like all build systems, it has its limitations.

One such limitation is that when you're building against a target platform, and something's missing, you get errors such as these:

[INFO] Cannot complete the request.  Generating details.
{org.osgi.framework.executionenvironment=OSGi/Minimum-1.0,OSGi/Minimum-1.1, osgi.ws=cocoa, osgi.arch=x86, osgi.os=macosx, org.eclipse.update.install.features=true, org.osgi.framework.system.packages=}
[Software being installed: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier, Missing requirement: org.eclipse.tptp.platform.instrumentation.ui 4.4.1.v201009092123 requires 'bundle org.eclipse.hyades.probekit [4.2.0,5.0.0)' but it could not be found, Cannot satisfy dependency: org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.1.v201009092123-797908s73533D4H6D56 depends on: org.eclipse.tptp.platform.instrumentation.ui [4.4.1.v201009092123], Cannot satisfy dependency: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier depends on: org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.0]
[ERROR] Internal error: java.lang.RuntimeException: org.eclipse.equinox.p2.core.ProvisionException: No solution found because the problem is unsatisfiable. -> [Help 1]
org.apache.maven.InternalErrorException: Internal error: java.lang.RuntimeException: org.eclipse.equinox.p2.core.ProvisionException: No solution found because the problem is unsatisfiable.

The important part of that error message is as follows:

org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier
   requirement: org.eclipse.tptp.platform.instrumentation.ui 4.4.1.v201009092123 
      requires 'bundle org.eclipse.hyades.probekit [4.2.0,5.0.0)' 
         but it could not be found

org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.1.v201009092123-797908s73533D4H6D56 
   depends on: org.eclipse.tptp.platform.instrumentation.ui [4.4.1.v201009092123]
     dependency: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier 
        depends on: org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.0]

So, how do you find which feature contains that plugin, so that you can add it to your target platform?

First, you need access to the repository. If you have direct server access to the repository from which the plugin comes (eg., the TPTP Helios update site), you can run this script in the root of that repository.

If you don't have server access (eg., you can't ssh to dev.eclipse.org and look in ~/downloads/tptp/updates/helios), then you can pull down a zip of the site (or use a p2.mirror script to fetch a copy of the site to your local machine)... and then run this script in the root of that repository.

Essentially the script finds matching feature jar files, unpacks them to extract the feature.xml files therein, and then greps those files for lines which suggest an included plugin matching the pattern for which you're searching:

$ findInFeature platform.probekit
./features/org.eclipse.tptp.platform.probekit_4.5.1.v201009092123-7H7BF8PAkF7B77ZARCNEK.jar
   <plugin
         id="org.eclipse.tptp.platform.probekit"

./features/org.eclipse.tptp.platform.trace_4.5.1.v201009092123-7L7O8bBgJ9E99jAfGWEM.jar
   <plugin
         id="org.eclipse.tptp.platform.probekit.launch"

From there, it's a trivial exercise to add another line item into your target platform file. First, paste in the feature jar:

./features/org.eclipse.tptp.platform.probekit_4.5.1.v201009092123-7H7BF8PAkF7B77ZARCNEK.jar

Then use vim to pattern-replace that string:

:%s/.\+\/\(org.\+\)_\(\d\+.\+\)\.jar/\t\t\t/g

And you'll end up with a new .feature.group added to the target:

<unit version="4.5.1.v201009092123-7H7BF8PAkF7B77ZARCNEK" id="org.eclipse.tptp.platform.probekit.feature.group"/>

Pattern Match Searches Within Multiple Files And Archives

Have you ever wondered what plugin is locking a particular file extension, and forcing it to be opened with a certain editor? Or wanted to scan some plugins to check their copyright information? Whatever the filetype, this script will allow you to search inside zips and jars for text files matching a given regular expression. Sure, there's probably a graphical way to do this, but nothing beats the purity of the almighty commanline.

Here's some sample output, looking for the plugins that lock *.mod files:

$ find.in.zip.sh ~/eclipse/eclipse-plugins-other "*.jar" "plugin.xml" "extensions.*.mod" -verbose

** [1] /tmp/plugins/org.eclipse.wst.dtd.core_1.1.200.v200710021541.jar!plugin.xml **
41-     <extension point="org.eclipse.core.runtime.contentTypes">
42-             <content-type
43:                     file-extensions="dtd,mod,ent"
44-                     priority="normal"
45-                     name="%DTD_Content_Type_Extension_Element.name"

** [2] /tmp/plugins/org.eclipse.wst.xml.ui_1.0.400.v200711071909.jar!plugin.xml **
16-     <extension point="org.eclipse.wst.xml.ui.catalogFileType">
17-             <fileType
18:                     extensions="dtd, ent, mod"
19-                     description="%_UI_PREF_DTD_FILES"
20-                     id="org.eclipse.wst.xml.core.ui.catalogFileType.dtd">

Looking for files that aren't in zips? Use this simpler script:

$ find.sh ~/eclipse/eclipse-plugins-enable34/ "feature.xml" "2007 by" -verbose

** [1] /tmp/features/org.eclipse.eclipsemonkey_1.0.0.200706060947/feature.xml **
11-
12-   <copyright>
13:      Portions copyright 2007 by Aptana Inc.
14:Other portions copyright 2006-2007 by Eclipse Foundation Inc.
15-   </copyright>
16-

Oh, and if you don't see red when you run these, make sure you have this in your ~/.bashrc, /etc/bashrc, or /etc/bash.bashrc file:

export GREP_OPTIONS='--color=auto'