DivByZero.com

Build Nomenclature Conventions: What's in a name?

2011-12-11T14:12:00.003-05:00

The following post is inspired by Mickael Istria's recent blog, Call a spade a spade, and a Nightly a Snapshot.

When I was doing builds for the Eclipse Modeling Project, I-builds were weekly published nightlies -- same level of stability as a SNAPSHOT (to use Maven parlance) or nightly, but published on a weekly schedule to bridge the gap between nightly/daily/SNAPSHOT/CI builds and the every-6-weeks milestone releases. The goal was to provide something stable enough for early adopters to grab once a week, but without the non-stop flux of nightlies. Regardless of the label on the build, the process was the same: tag CVS, then build using that tag.

The Final/GA/Release ("R") builds were done as simple renames of the last good milestone or release candidate build, so as to ensure binary-compatibility w/ the last-tested milestone/RC. The same was true for "M" and "S" builds -- they were just renamed "I" builds, and the letter was there simply to differentiate between a maintenance build (M), a stable milestone (S), or release (R).

Branching only happened when a release was done and it was time to produce the maintenance stream vs. the ongoing next-year-release. Sometimes branching would happen AFTER the x.y.1 maintenance because it saved duplication of commits in the x.y+1.0 and x.y.1 streams.

Now at JBoss, we publish "nightly" builds, which are keyed to SVN changes and therefore could be as often as hourly or as infrequent as weekly, depending on what's happening in the repo.

We also do milestone builds about once ever 6-8 weeks (similar to the Eclipse.org release train schedules), which is more carefully vetted, tested, and QE'd. It is produced using the same *process* as the nightlies, but are named differently and pulled from a freshly-created stable branch in the repo (so its degree of change/churn is less). (Branching happens right before every milestone or release candidate so that hardening/stabilization/documentation can happen in the branch while trunk stays open for new development.)

Bottom line -- I've only ever needed three types of builds, regardless of nomenclature or labelling differences. And of these 3, the last 2 are the same thing but renamed to underline the build quality/stability:

* nightly/CI/integration/weekly/SNAPSHOT build (unstable, for bleeding edge adopters)

* development milestone (probably a re-christened nightly; stable, early adopters)

* stable release / Final / GA (probably a re-christened milestone; release quality)

So... does it matter if it's called nightly, integration or SNAPSHOT? or Stable, Milestone, Maintenance, Final, GA or Release? As long as it's easily reproducible (yeah, Tycho!), what's in a name?

HOWTO: Make KDE remember dual-monitor randr settings

2011-11-09T16:24:00.006-05:00

Every time I boot up, KDE appears to forget that I want my monitors to be positioned left-to-right and instead defaults to mirrored config. But, after a lot of cursing and a little googling, I found an answer so it'll no so much keep your settings, but reset its broken config to your settings.

1. Hit ALT-F2, then enter "display" to run the Display Settings app.

2. Configure your settings as you'd like. Note that if the Apply button isn't active after your changes, you can change/revert something like a Position: button to make it active.

3. On restart, KDE may forget your dual-monitor settings. So, to prevent this, go look in your ~/.kde/share/config/krandrrc file:

[Display]
ApplyOnStartup=true
StartupCommands=xrandr --output "DVI-I-1" --pos 1920x0 --mode 1920x1200 --refresh 59.9502\nxrandr --output "HDMI-1" --pos 0x130 --mode 1920x1080 --refresh 60\nxrandr --noprimary

4. Copy the configuration into a new file, and replace \n with newlines. I like to put scripts like this in /etc/X11 because they relate to screen res and positioning.

# from ~/.kde/share/config/krandrrc
xrandr --output "DVI-I-1" --pos 1920x0 --mode 1920x1200 --refresh 59.9502 
xrandr --output "HDMI-1" --pos 0x130 --mode 1920x1080 --refresh 60 
xrandr --noprimary

5. Ensure the script is readable/executable for all users:

chmod 755 /etc/X11/1920x2.sh

6. Hit ALT-F2, then enter "autostart" to run the Autostart config tool.

7. Click Add script... and browse for the script you created above.

8. Reboot and watch the magic unfold.

HOWTO: See what happened in SVN between builds

2011-10-29T14:22:00.008-04:00

I was recently asked how to determine what changed between two builds. Jenkins provides nice interlinks into JIRA (issues), Fisheye (source changes), SVN (sources), but let's say you want to kick things a little more old school and investigate the old way... or the builds you want to compare are no longer shown in Jenkins because they expired and their metadata was automatically purged.

If you can't just look at the changelog in Jenkins to see what revision of source was used for the build, you can check the SVN log to find revision numbers based on the timestamp of the build.

So, if your build was generated on 2011-10-18, you can see that the log shows the last commit before that build was this:

$ svn log http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x/

...

r35735 | bfitzpat | 2011-10-17 15:35:23 -0400 (Mon, 17 Oct 2011) | 2 lines
Changed paths:
   A esb/plugins/.project
   M esb/plugins/org.jboss.tools.esb.project.core/src/org/jboss/tools/esb/core/runtime/ESBRuntimeResolver_410.java

JBDS-1889 - Now checking for juddi-client-3.1.2.jar as well as 3.1.0 and 3.1.1 when seeing if the runtime includes ESB 4.10

...

Want to see actual diffs between that build and the latest one?

$ svn diff http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x/@35735 http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x/

Or, if you want to collect just the section of log relevant to the change:

$ svn log -r35735:HEAD http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x/

Of course if you have all the sources locally, you don't need to log or diff via a URL - you can simply use local file paths. And if like me you use git-svn instead of pure svn, you can use that to diff or log too.

If you want to easily determine when a branch was created and get the SVN revision number for that branch point, use this:

# from r28571, returns -r28571:HEAD
rev=$(svn log --stop-on-copy \
  http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x \
  | egrep "r[0-9]+" | tail -1 | sed -e "s#\(r[0-9]\+\).\+#-\1:HEAD#")

If you'd like to view a specific svn revision in your browser, use !svn/bc/REVISION_NUMBER/ before the branch and path to file or folder:

http://svn.jboss.org/repos/jbosstools/!svn/bc/35735/branches/jbosstools-3.2.x/

HOWTO: Use Maven, Ant, and XSLT to scrub unwanted p2 metadata from an update site 2011-10-26T20:48:00.003-04:00 Some time ago, I wrote about Using p2.inf to add/remove update sites. Tonight I found a simpler way to remove references in p2 metadata to external 3rd party sites. For example, say you're repackaging some 3rd party features onto your own site, but don't want those features to provide references to the vendor's own update sites because you want to ensure that your product's site will only result in your sanctioned version being installed. When you generate an update site, p2 pulls the information in the included features and will result in a section of references in the site's metadata that looks like this: <references size="6"> <repository uri="http://download.eclipse.org/egit/updates" url="http://download.eclipse.org/egit/updates" type="0" options="0"/> <repository uri="http://subclipse.tigris.org/update_1.6.x" url="http://subclipse.tigris.org/update_1.6.x" type="1" options="0"/> <repository uri="http://download.eclipse.org/egit/updates" url="http://download.eclipse.org/egit/updates" type="1" options="0"/> <repository uri="http://subclipse.tigris.org/update_1.6.x" url="http://subclipse.tigris.org/update_1.6.x" type="0" options="0"/> <repository uri="http://eclipse.svnkit.com/1.3.x/" url="http://eclipse.svnkit.com/1.3.x/" type="0" options="0"/> <repository uri="http://eclipse.svnkit.com/1.3.x/" url="http://eclipse.svnkit.com/1.3.x/" type="1" options="0"/> </references> To remove that, you can play with p2.inf directives, or you can simply perform an XSL transformation on the generated content.xml (inside content.jar, if your metadata is compressed) to remove the <references/> node: <?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0"> <xsl:template match="/"> <xsl:apply-templates select="*"/> </xsl:template> <xsl:template match="*"> <xsl:copy > <xsl:for-each select="@*"> <xsl:copy /> </xsl:for-each> <xsl:apply-templates /> </xsl:copy> </xsl:template> <xsl:template match="references" /> </xsl:stylesheet> If you're generating your update site w/ Tycho, this transform can be called via a simple Ant script: <target name="remove.references">  <if> <available file="${update.site.source.dir}/content.jar" type="file" /> <then> <unzip src="${update.site.source.dir}/content.jar" dest="${update.site.source.dir}" /> <delete file="${update.site.source.dir}/content.jar" /> </then> </if> <copy file="${update.site.source.dir}/content.xml" tofile="${update.site.source.dir}/content.old.xml" overwrite="true" /> <xslt style="remove-references.xsl" in="${update.site.source.dir}/content.old.xml" out="${update.site.source.dir}/content.xml" /> <zip destfile="${update.site.source.dir}/content.jar" basedir="${update.site.source.dir}" includes="content.xml" /> <delete file="${update.site.source.dir}/content.xml" /> <delete file="${update.site.source.dir}/content.old.xml" /> </target> Then, in your site's pom.xml, to call the Ant script, do this: <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-antrun-plugin</artifactId>  <version>${maven.antrun.plugin.version}</version> <executions> <execution> <id>install</id> <phase>install</phase> <configuration> <quiet>true</quiet> <tasks>  <ant antfile="build.xml"> <property name="SOME_ANT_VARIABLE" value="${SOME_MAVEN_VARIABLE}" /> </ant> </tasks> </configuration> <goals> <goal>run</goal> </goals> </execution> </executions> <dependencies>  <dependency> <groupId>commons-net</groupId> <artifactId>commons-net</artifactId> <version>1.4.1</version> </dependency> <dependency> <groupId>org.apache.ant</groupId> <artifactId>ant</artifactId> <version>1.7.1</version> </dependency> <dependency> <groupId>org.apache.ant</groupId> <artifactId>ant-nodeps</artifactId> <version>1.7.1</version> </dependency> <dependency> <groupId>org.apache.ant</groupId> <artifactId>ant-trax</artifactId> <version>1.7.1</version> </dependency> <dependency> <groupId>org.apache.ant</groupId> <artifactId>ant-commons-net</artifactId> <version>1.7.1</version> </dependency> <dependency> <groupId>org.apache.ant</groupId> <artifactId>ant-apache-regexp</artifactId> <version>1.7.1</version> </dependency> <dependency> <groupId>ant-contrib</groupId> <artifactId>ant-contrib</artifactId> <version>1.0b3</version> </dependency> </dependencies> </plugin> </plugins> </build> I suppose there's probably a way to call a transform directly from Maven w/o the Ant wrapper, but this allows unpacking and repacking of the content.jar to get at the content.xml file. HOWTO: Move around between desktops & windows with keyboard or mouse 2011-09-01T02:03:00.011-04:00 Recently installed Fedora 15 KDE spin, partly because the XFCE spins wouldn't boot from CD but also because I've heard less-than-favourable things about Gnome3 and because I'm addicted to Konqueror as a graphical sftp/scp/ssh viewer, so figured might as well use kdm instead of xfwm4 or gdm. Still having some problems getting my 1600x1200 (or 1920x1200) monitor to do anything more than 1024x768 on the VGA port of the video card (works fine on the DisplayPort connector, either directly or via a DP-to-DVI cable, but not on the VGA connector, even with xorg.conf hackery). That said the options for display/monitor management under KDE are much better than under XFCE, and this is the first time I've been able to get two monitors working without HOURS of hacking away at xorg.conf scripts. So... big props for this release *almost* Just Working. Workaround I'm trying next is to install a second video card. Will update when/if that solves the problem once it arrives. But video resolution aside, I did recently figure out how to set keyboard bindings for moving windows between desktops (thanks to David Fisco). From the K-menu, select System Settings > Shortcuts and Gestures > Global Keyboard Shortcuts > KDE Component: KWin > "Window One Desktop To The Left/Right": ... and for switching between desktops ("Switch To Next/Previous Desktop") Also recently discovered some fun options for switching between windows (on all desktops). From the K-menu, select System Settings > Desktop Effects > Enable desktop effects > Effect for window switching: Present Windows (or any of the other options). There's also System Settings > Window Behavior > Task Switcher > Effect: Present Windows: You might want to set an animation for switching between desktops, though I find with multiple monitors this can be a bit dizzying. From the K-menu, select System Settings > Workspace Behavior > Virtual Desktops > Switching > Animation: Desktop Cube Animation. For something more subtle, try "Fade Desktop". Finally, you may want to set screen edge behaviours, such as making Present Windows appear when you cursor to the top-center of your screen. System Settings > Workspace Behavior > Screen Edges > right-click a target zone: Mounting Linux LVM drives 2011-08-16T15:15:00.005-04:00 Because my Thinkpad X200 has finally, after just under 3 years, decided to give up the ghost via a FAN ERROR and refusal to start (POST beeps & auto-shutdown), I'm now faced with the task of recovering all the data on the drive (about 120G) across multiple partitions. Here's the drive layout, as per cfdisk: cfdisk (util-linux-ng 2.17.2) Disk Drive: /dev/sdb Size: 160041885696 bytes, 160.0 GB Heads: 255 Sectors per Track: 63 Cylinders: 19457 Name Flags Part Type FS Type [Label] Size (MB) -------------------------------------------------------------------------------------------------- sdb1 Primary NTFS [^B] 26214.44 * sdb2 Boot Primary Linux ext3 209.72 * Logical Free Space 3.68 * sdb5 Logical Linux ext3 [HOME] 106043.70 * sdb6 NC Logical Linux LVM 20970.48 * Logical Free Space 1.09 * sdb4 Primary Compaq diagnostics 6595.71 * Unusable 0.49 * So, under a Fedora 13 LiveCD, the /boot (sdb2) and /home (sdb5) partitions automounted, along with the WinXP (sdb1) partition. But the root partition (/, part of sdb6) would not as it's part of a LVM. After a quick burst of googling, I found this solution, which digests down to simply this: yum install lvm2 -y; # install support for lvm2 pvscan # scan vol groups vgchange vg_x2lappy -a y # mark your vol group active lvscan # scan for logical volumes mkdir /media/sdb6 # create a mount point mount /dev/vg_x2lappy/lv_root /media/sdb6/ # mount the lv cd /media/sdb6/; ls -la # take off every zig! MANIFEST.MF and feature.xml versioning rules 2011-07-27T11:48:00.003-04:00 I'm forever forgetting what the rules are for dependency declarations in MANIFEST.MF and feature.xml for osgi plugins and features. And Googling often results in frustration rather than an answer. So, because today I actually found a concise list of the rules, I thought I'd repost them here, with some minor edits to help clarify. OSGi Plugin Version Ranges Dependencies on bundles and packages have an associated version range which is specified using an interval notation: a square bracket “[” or “]” denotes an inclusive end of the range and a round bracket “(” or “)” denotes an exclusive end of the range. Where one end of the range is to be included and the other excluded, it is permitted to pair a round bracket with a square bracket. The examples below make this clear. If a single version number is used where a version range is required this does not indicate a single version, but the range starting from that version and including all higher versions. There are four common cases: A “strict” version range, such as [1.2.3,1.2.3], which denotes that version and only that version. A “half-open” range, such as [1.2.3,2.0.0), which has an inclusive lower limit and an exclusive upper limit, denoting version 1.2.3 and any version after this, up to, but not including, version 2.0.0. An “unbounded” version range, such as 1.2.3, which denotes version 1.2.3 and all later versions. No version range, which denotes any version will be acceptable. NOT RECOMMENDED. The complete text of the above snippet can be seen here (or here as PDF). Example: Require-Bundle: org.eclipse.core.runtime;bundle-version="[3.4.0,4.0.0)", org.eclipse.core.resources;bundle-version="[3.4.0,4.0.0)", org.eclipse.ui.ide;bundle-version="[3.4.0,4.0.0)", org.eclipse.ui.navigator;bundle-version="3.5.100", com.ibm.icu See also: plugin manifest (plugin.xml) osgi bundle manifest (MANIFEST.MF) In terms of feature manifest (feature.xml) rules, help.eclipse.org has pretty good documentation, but the most important thing to remember - and what I often have to look up - is how to state the matching rules for required upstream features & plugins. Experience says it's always better to state things explicitly so there's no downstream guesswork needed and anyone reading your manifest knows EXACTLY what version(s) are required for or compatible with your feature. Plus, while YOU might be using PDE UI to build, someone else might be using Tycho and Maven, and every tool can interpret missing metadata their own way. When in doubt, spell it out. Valid values and processing are as follows: if version attribute is not specified, the match attribute (if specified) is ignored. perfect - dependent plug-in version must match exactly the specified version. If "patch" is "true", "perfect" is assumed and other values cannot be set. [1.2.3,1.2.3] equivalent - dependent plug-in version must be at least at the version specified, or at a higher service level (major and minor version levels must equal the specified version). [1.2.3,1.3) compatible - dependent plug-in version must be at least at the version specified, or at a higher service level or minor level (major version level must equal the specified version). [1.2.3,2.0) greaterOrEqual - dependent plug-in version must be at least at the version specified, or at a higher service, minor or major level. 1.2.3 The complete text of the above snippet can be seen here. Example: <requires> <import feature="org.eclipse.m2e.feature" version="1.0.0" match="compatible"/> <import feature="org.maven.ide.eclipse.wtp.feature" version="0.13.0" match="greaterOrEqual"/> <plugin id="ch.qos.logback.classic" version="0.9.27.v20110224-1110" match="greaterOrEqual"/> <plugin id="ch.qos.logback.core" version="0.9.27.v20110224-1110" match="greaterOrEqual"/> <plugin id="ch.qos.logback.slf4j" version="0.9.27.v20110224-1110" match="greaterOrEqual"/> <plugin id="org.slf4j.api" version="1.6.1.v20100831-0715" match="compatible"/> <plugin id="com.ning.async-http-client" version="1.6.3.201106061504" match="equivalent"/> <plugin id="org.jboss.netty" version="3.2.4.Final-201106061504" match="perfect"/> <plugin id="org.hamcrest.core" version="1.1.0.v20090501071000" match="equivalent"/> </requires> HOWTO: get xorg.conf to work w/ 1600x1200 res and an old Intel card 2011-06-06T17:20:00.003-04:00 Check your hardware spec, and determine how much memory your card has[1]: # lspci -vv | grep "Intel" -A7 | grep "VGA controller" -A7 | egrep "controller|Region" 00:02.0 VGA compatible controller: Intel Corporation 82852/855GM Integrated Graphics Device (rev 02) (prog-if 00 [VGA controller]) Region 0: Memory at e0000000 (32-bit, prefetchable) [size=128M] Region 1: Memory at d0000000 (32-bit, non-prefetchable) [size=512K] Use the above values to configure your /etc/X11/xorg.conf file - I suspect much of this is not needed, but here's what I have: Section "Device" Identifier "Intel" Option "AccelMethod" "UXA" VideoRam 130560 #Driver "intel" Driver "vesa" VendorName "Intel Corporation" BoardName "82852/855GM Integrated Graphics Device" BusID "PCI:0:2:0" EndSection Section "Monitor" Identifier "VGA" ModelName "Sceptre 24" HorizSync 31 - 80 VertRefresh 55 - 76 Option "DPMS" "true" EndSection Section "Screen" Identifier "Default Screen" Device "Intel" Monitor "VGA" DefaultDepth 16 SubSection "Display" Depth 16 Modes "1920x1440_60" "1920x1200_60" "1920x1080_60" "1680x1050_59.883" "1360x768_59.8" "1600x1200_60" "1280x1024_60" "1024x768_60" #Modes "1920x1440" "1600x1200" "1280x1024" "1280x768" EndSubSection EndSection Section "DRI" Mode 0666 EndSection Section "Extensions" Option "Composite" "Enable" EndSection Section "Module" Load "dri" EndSection In /boot/grub/menu.lst, add the correct vga mode for 1600x1200 (vga=8). If you enter what you think is the correct mode based on this table, you'll be told it's wrong and can manually correct it, boot up, then fix the file & reboot. default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz title Fedora (2.6.34.7-56.fc13.i686) root (hd0,0) kernel /vmlinuz-2.6.34.7-56.fc13.i686 ro root=/dev/mapper/vg_xlappy-lv_root rd_LVM_LV=vg_xlappy/lv_root rd_LVM_LV=vg_xlappy/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet vga=8 initrd /initramfs-2.6.34.7-56.fc13.i686.img title Fedora (2.6.34.7-56.fc13.i686) cmdline only, vga=8 = 1600x1200x16 root (hd0,0) kernel /vmlinuz-2.6.34.7-56.fc13.i686 ro root=/dev/mapper/vg_xlappy-lv_root rd_LVM_LV=vg_xlappy/lv_root rd_LVM_LV=vg_xlappy/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us init=3 init 3 vga=8 initrd /initramfs-2.6.34.7-56.fc13.i686.img See also: VGA Modes grub VGA Modes vim scripting: replacing timestamps 2011-04-19T21:57:00.005-04:00 After weeks of manually updating compositeArtifacts.xml and compositeContent.xml files' timestamps using `date +%s000`, I finally snapped and went looking for a better way. Using the Vim Tips wiki as inspiration, I cobbled together this keymapped string replacement function for use in Vim: " add this to your ~/.vimrc file, then type '\ts' to update timestamp to current fun! ReplaceTimestamp() let tstamp = strftime("%s000") exe ":%s#<property name='p2.timestamp' value='[0-9]\\+'/>#<property name='p2.timestamp' value='" . tstamp . "'/>#g" echo "New time: " . tstamp endfun nnoremap <Leader>ts :call ReplaceTimestamp()<CR> Git in colour 2011-03-03T17:14:00.005-05:00 I've been using Git for a while now, but only today realized I can have coloured output for diff, grep, branch, show-branch and status, without having to hook in any other external tools (like colordiff, for example). Here's my ~/.gitconfig file, which enables colour: [user] name = Nick Boldt email = nickboldt (at) gmail.com [giggle] main-window-maximized = false main-window-geometry = 1324x838+0+24 main-window-view = HistoryView [core] trustctime = false branch = auto diff = auto interactive = auto status = auto editor = vim [merge] tool = vimdiff [receive] denyCurrentBranch = warn [branch] autosetuprebase = local [color] ui = true diff = true grep = true branch = true showbranch = true status = true [color "diff"] plain = normal dim meta = yellow dim frag = blue bold old = magenta new = cyan whitespace = red reverse [color "status"] header = normal dim added = yellow untracked = magenta [color "branch"] current = yellow reverse local = yellow remote = red Simplifying the p2 Process, Part 4: Using p2.inf to add/remove update sites 2011-02-16T17:57:00.005-05:00 In Part 1 of this series, I looked at use of composite repos to provide a way of combining update sites into a single URL for ease of use and a single point of entry from which to do updates. In Part 2, I discussed why we switched from using a collection of SDKs against which to build - using the now-deprecated brute-force "just unzip into eclipse root folder or dropins" approach - to using a single target platform update site so as to simplify maintenance and provide a reusable artifact for both build and workspace provisioning. In Part 3, I looked at the idea of associating your repo with its upstream requirement sites, so that end-users need only use a single URL, rather than a half-dozen. Finally, let's look at how you can use a p2.inf file to remove sites you don't support and add sites you do. In JBDS 4, we include only two update sites - one for core features, and one for certified third-party extras, so that users will only get official updates from us, rather than from Spring, Eclipse, or anywhere else. Sure, they can manually add other URLs themselves, but that's a bit like pulling off the 'do not remove this tag' tag on a mattress or removing the 'warranty void if removed' sticker on your laptop. So, first, we remove all the update site and discovery site URLs from our upstream features' feature.xml files, so they don't trickle down into the product. Next, we use a p2.inf file: # To explicitly remove a site, use instructions.unconfigure instructions.configure=\ org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:0,location:https${#58}//www.your.server.com/,name:Core Product Updates);\ org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:1,location:https${#58}//www.your.server.com/,name:Core Product Updates);\ org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:0,location:https${#58}//www.your.server.com/extras/,name:Extra Product Updates);\ org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(type:1,location:https${#58}//www.your.server.com/extras/,name:Extra Product Updates);\ Then, to generate a site using that p2.inf instruction, here's a bit of Ant code: <echo>Run p2.publisher.UpdateSitePublisher using launcherjar = @{launcherjar}</echo> <java jar="@{launcherjar}" fork="true" timeout="10800000" jvm="${java.home}/bin/java" failonerror="true" maxmemory="256m" taskname="p2" > <classpath> <fileset dir="${builder.build.path}/plugins" includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar" /> <fileset dir="${clean.eclipse.home}/plugins" includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar" /> <pathelement location="${builder.build.path}/plugins" /> <pathelement location="${clean.eclipse.home}/plugins" /> </classpath> <arg line=" org.eclipse.equinox.launcher.Main -consolelog -application org.eclipse.equinox.p2.publisher.UpdateSitePublisher" /> <arg line=" -metadataRepository file:${updateSiteJarDir}/ -metadataRepositoryName "${update.site.product.name} ${update.site.description} Update Site"" /> <arg line=" -artifactRepository file:${updateSiteJarDir}/ -artifactRepositoryName "${update.site.product.name} ${update.site.description} Artifacts"" /> <arg line=" -source ${updateSiteJarDir}/" /> <arg line=" -compress -publishArtifacts -reusePack200Files -configs *,*,*" /> </java> Or, put your p2.inf file in the same directory as your build.properties ... product=${builderDirectory}/jbds-all.product runPackager=true p2.gathering=true p2.category.site=file:${builderDirectory}/site.xml # locations. Don't need a baseLocation, the transformedRepoLocation will have what we need buildDirectory=${product.build.directory}/jbds-all-package transformedRepoLocation=${product.build.directory}/jbds-all-package/transformed repoBaseLocation=${product.build.directory}/jbds-all-package/toTransform # The prefix that will be used in the generated archive. archivePrefix=studio # The location underwhich all of the build output will be collected. collectingFolder=${archivePrefix} # The list of {os, ws, arch} configurations to build. configs = linux,gtk,x86 & win32,win32,x86 & linux,gtk,x86_64 & macosx,cocoa,x86 & macosx,cocoa,x86_64 buildId=${product.name}-product-${versionTag} buildLabel=${buildId} skipBase=true skipMaps=true skipFetch=true ... and your .product file ... <?xml version="1.0" encoding="UTF-8"?> <?pde version="3.5"?> <product name="JBoss Developer Studio for Web and SOA Development" uid="com.jboss.jbds.all" id="com.jboss.jbds.product.product" application="org.eclipse.ui.ide.workbench" version="4.0.0.qualifier" useFeatures="true" includeLaunchers="true"> <configIni use="default"> </configIni> <launcherArgs> <programArgs>--launcher.XXMaxPermSize 256m --launcher.defaultAction openFile</programArgs> <vmArgs>-Xms512m -Xmx1024m -Dosgi.bundles=reference:file:org.eclipse.equinox.simpleconfigurator_1.0.200.v20100503.jar@1:start -Dosgi.instance.area.default=@user.home/workspace</vmArgs> <vmArgsMac>-XstartOnFirstThread -Dorg.eclipse.swt.internal.carbon.smallFonts -Xdock:icon=../Resources/JBDevStudio.icns</vmArgsMac> </launcherArgs> <windowImages/> <splash location="com.jboss.jbds.product" /> <launcher name="jbdevstudio"> <solaris/> <win useIco="true"> <ico path="jbds.ico"/> <bmp/> </win> </launcher> <vm> </vm> <plugins> </plugins> <features> <feature id="com.jboss.jbds.product.feature" version="4.0.0.qualifier"/> </features> </product> ... and when generating a product using PDE, that file and its instructions should be read at the correct time. Hope this series has been helpful! If you have any examples of what you've done with .product or p2.inf files, please feel free to send me a link to your post or the file in your cvs, svn, or git repo. I'd love to see what else you can do with p2 and product builds. See also: help.eclipse.org - Customizing p2 metadata p2.inf for building Eclipse SDK Simplifying The p2 Process, Part 3: Associate Sites 2011-02-13T09:00:00.000-05:00 In Part 1 of this series, I looked at use of composite repos to provide a way of combining update sites into a single URL for ease of use and a single point of entry from which to do updates. In Part 2, I discussed why we switched from using a collection of SDKs against which to build - using the now-deprecated brute-force "just unzip into eclipse root folder or dropins" approach - to using a single target platform update site so as to simplify maintenance and provide a reusable artifact for both build and workspace provisioning. Now, let's look at the no-brainer that says that "less is more" when it comes to telling p2 from where to get updates, and that less effort for your user when installing is always a win. If this sounds familar, I did blog about this briefly back in August. Since then, we've also added a quick XSLT script to remove the "Uncategorized" category that Tycho automatically adds for features which are listed in your site.xml but are not associated with a category. While this isn't strictly related to associate sites, it is about ease of use; while I applaud the desire to have everything belong to a category bucket (perhaps because the model Tycho's using requires it), the reason we'd rather hide these is to declutter the install view and not confuse people by suggesting features that won't work on their OS (eg., for which there's no XulRunner port). But I digress... Associate Sites In the old days of yore, you could situate an associateSites.xml next to your site.xml in your "classic" update site, and Eclipse Update Manager would happily read that file and add those extra sites to your list of available update sites. Then came p2, and while the old way still worked, it was no longer ideal. So, the new approach was to insert these associate sites directly into the p2 metadata for the site, content.xml and artifacts.xml (or content.jar and artifacts.jar). This could be accomplished via a somewhat hacky appraoch - unpacking the existing metadata (content.jar) and shoehorning in the information at the bottom of the content.xml file, using an ant script (see "add.associate.sites" target, below) and a list of sites to be added: <target name="add.associate.sites" if="associate.sites"> <if> <and>  <isset property="associate.sites" /> <not> <equals arg1="${associate.sites}" arg2="" /> </not> </and> <then> <if> <available file="${update.site.source.dir}/content.jar" type="file" /> <then> <unzip src="${update.site.source.dir}/content.jar" dest="${update.site.source.dir}" /> <delete file="${update.site.source.dir}/content.jar" /> </then> </if>  <var name="associate.sites.0" value="" /> <for param="associate.site" list="${associate.sites}" delimiter=", "> <sequential> <var name="associate.sites.0" value="${associate.sites.0}00" /> </sequential> </for> <length property="associate.sites.length" string="${associate.sites.0}" /> <loadfile srcfile="${update.site.source.dir}/content.xml" property="content.xml"> <filterchain> <tailfilter lines="-1" skip="1" /> </filterchain> </loadfile> <echo file="${update.site.source.dir}/content.xml" message="${content.xml}" /> <echo file="${update.site.source.dir}/content.xml" append="true"> <references size='${associate.sites.length}'> </echo> <for param="associate.site" list="${associate.sites}" delimiter=", "> <sequential>  <echo file="${update.site.source.dir}/content.xml" append="true"> <repository uri='@{associate.site}' url='@{associate.site}' type='0' options='1'/> <repository uri='@{associate.site}' url='@{associate.site}' type='1' options='1'/> </echo> </sequential> </for> <echo file="${update.site.source.dir}/content.xml" append="true"> </references> </repository> </echo>  <copy file="${update.site.source.dir}/content.xml" tofile="${update.site.source.dir}/content.old.xml" overwrite="true" /> <xslt style="remove-uncategorized.xsl" in="${update.site.source.dir}/content.old.xml" out="${update.site.source.dir}/content.xml" /> <zip destfile="${update.site.source.dir}/content.jar" basedir="${update.site.source.dir}" includes="content.xml" /> <delete file="${update.site.source.dir}/content.xml" /> <delete file="${update.site.source.dir}/content.old.xml" /> </then> </if> </target> So, now, instead of telling people to add multiple update sites to resolve missing potentially dependencies when installing, we can cause those extra sites to be automatically added at the same time they add the single URL for JBoss Tools. Now the additional sites need only be listed for reference, but no additional effort is required by the user. BONUS HACK: to force a site that may already be listed (but disabled) to be added again, and this time definitely be enabled, you can add an extra slash into URL. Thus http://download.eclipse.org/birt/update-site/2.6 becomes http://download.eclipse.org//birt/update-site/2.6/, and as p2 sees a new site, it adds the new site (instead of ignoring it because it's already present but disabled. Again, a win. Alternatively, you could cast an arcane spell using a p2.inf file in your feature's root folder or plugin's META-INF/ folder to add these additional, required sites... or do whatever processing you might need. I'm not sure if Tycho supports this yet, or how fully PDE supports reading this information. Got sample code? Send it to me as a comment below or via twitter to @nickboldt. Thanks! In part 4, I'll talk a little about how to prevent your product build from getting updates from unofficial sources, and preload your product with the official sites from which to get updates. Because it's important to balance ease of use with prevention of unsupported features. SPOILER ALERT: may contain p2.inf instructions. Simplifying The p2 Process, Part 2: Target Platform Repos 2011-02-09T18:25:00.001-05:00 In Part 1 of this series, I looked at use of composite repos to provide a way of combining update sites into a single URL for ease of use and a single point of entry from which to do updates. Defining a Target Now, I'd like to talk about how to escape the proliferation of zips needed to establish a target platform. For those unfamiliar with the term "target platform", it's either the installed base against which you're compiling your code, or it's the collection of things you have to install first before you can install something on top of that. For the JBoss Tools case, we have at least 8 prereqs for installation. Here's what you had to install prior to JBoss Tools 3.1.1: Now, admittedly, because there is also the Ganymede update site, you don't necessarily need to download and unpack all these zips in order to install JBoss Tools - instead, you need only enable the Ganymede site. (Same story for Helios and JBoss Tools 3.2.) However, to do a reproduceable PDE-based build, you still need to create this base install. Traditionally, PDE's approach was to download and unpack these zips into the root of the Eclipse install running the build. Athena attempted to improve on this situation by allowing you to define a list of update sites and IUs (features and/or plugins) which were needed to define the platform. But it was far from portable, and hardly reusable. Buckminster (later b3) also approached this problem by creating its own markup for defining what sites and what IUs to install, backed by an EMF model. But rather than dealing with a UI to create the model and populate it, I found it more useful to simply generate an instance of the aggregator model and then use the aggregator to fetch & install IUs. But as the aggregator is simply a wrapper for the underlying p2.mirror and p2.director tasks, you can use those directly too. But as they say... "Don't bore us, get to the chorus!" So, here's some sample code for the various solutions for build-time provisioning. Using the buckminster aggregator (properties file) - stopped working for us w/ Eclipse 3.6, so we switched to b3 Using the b3 aggregator (properties file) - stopped worked consistently due to network timeouts resolving deps & fetching IUs. Using p2.mirror - underlying p2 ant task for mirroring from one or more repos to local disk Using p2.director - underlying p2 ant task for installing IUs (from local or remote repo) into some target Eclipse So, with these tools, you could create a p2 repo from other repos - mirroring and installing IUs as needed - and even script an installation. But was there a better way? Target Platform Definition File Enter the target platform definition file (.target). This file contains a list of IUs and the p2 repos from which to provision them. So, it's like a b3 aggregator model, or an Athena build.properties file, but abstracted away from the concept of a build, because it can be used for building but ALSO for provisioning a user's installed Eclipse base. Unfortunately, the Target Platform Definition File editor in Eclipse 3.6 is less than optimal for large targets, or when your internet connection is suboptimal. So, after fighting with it for a while, filing bugs, and ultimately giving up, I went back to my handy-dandy XML editor (often just vim) to maintain it more simply. So rather than having Eclipse automatically install things based on a .target file, I revert to a workflow that actually works: installing by hand from an update site. While Buckminster does support .target files (or so I've read), I didn't want to be dependent on it any more, preferring a more "pure" solution. So, based on code from Peter Nehrer (@pnehrer), I then wrote an XSL transform to create a p2.mirror script from a .target file, wrapped with another Ant script (and optionally, a Maven pom.xml script). And why might you care? Well, this .target file can be used to: Provision a developer's Eclipse, using the Target Platform Definition Editor and a few clicks (when it doesn't time out) Provision a developer's Eclipse via script for offline or multiple users (getting the team up to speed) And yes, much (or all) of the above can be done w/ Buckminster and/or b3, if you like that approach. But I prefer to create the .target as input to a build process, rather than being explicitly tied to one. So, as I noted above, if you have a .target file, you can easily generate a p2 repo, and use that repo to run downstream builds. Now, instead of having a half-dozen zips to download and unpack with every build (using the deprecated and unsupported "dropins" method) you can use a fully-p2-friendly repo site which contains everything you need to do your builds - whether you're a Hudson server or a developer working at home or offline. Benefits Unlike "a collection of zips" this single-source-site can be versioned with each release. It only contains WHAT YOU ACTUALLY NEED rather than extraneous sources and doc and tangential plugins/features you don't. It's a bit like making muffins by first grinding your own flour, but at least you know there's nothing evil in that muffin mix, and you will be able to consistently reproduce the recipe every time, regardless of where you might be on teh interwebz. f you're a keener / beta tester who likes to build against the latest milestone (or even a weekly integration build) of Eclipse 3.next or 4.future, you can use the script above to self-update. So, while the TP itself is a contained snapshot listing the explicit versions of feature groups needed, it can also be run in "get the latest available" mode in order to keep your TP current against some HEAD or trunk development / releases. By splitting the TP out of the build, you can build it upstream. So, where in the past we had one "uberbuild" and an implied TP therein, now we have a TP build job, and it is then shared by the 34 downstream jobs which depend on it for their dependencies. Shut up and show me the code! # for the "foo.target" file, build a local target platform repo, fetching the latest versions and updating the .target file $ ant -f build.xml -DtargetFile=foo.target -DuseLatest=true # for the "bar.target" file, build a local target platform repo, but fetch only the stated versions of IUs $ ant -f build.xml -DtargetFile=bar.target -DuseLatest=false That's it. I also wrap the build.xml ant script w/ a pom which allows it to be called from an upstream Maven/Tycho process, but that's nothing more than just calling the script using the antrun plugin (and a few ant dependencies), like this: <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-antrun-plugin</artifactId> <version>1.6</version> <executions> <execution> <phase>validate</phase> <configuration> <tasks> <ant antfile="build.xml"> <property name="targetFile" value="multiple.target" />  </ant> </tasks> </configuration> <goals> <goal>run</goal> </goals> </execution> </executions> <dependencies> <dependency> <groupId>commons-net</groupId> <artifactId>commons-net</artifactId> <version>1.4.1</version> </dependency> <dependency> <groupId>org.apache.ant</groupId> <artifactId>ant-commons-net</artifactId> <version>1.7.1</version> </dependency> <dependency> <groupId>org.apache.ant</groupId> <artifactId>ant-trax</artifactId> <version>1.7.1</version> </dependency> </dependencies> </plugin> </plugins> </build> The rest of the code is here. In part 3, I'll look back at the success we've had using associate sites instead of asking people to manually add 3rd party URLs when installing JBoss Tools. SPOILER ALERT: one URL is easier for people to use than 6. In part 4, I'll talk a little about how to prevent your product build from getting updates from unofficial sources, and preload your product with the official sites from which to get updates. Because it's important to balance ease of use with prevention of unsupported features. SPOILER ALERT: may contain p2.inf instructions. Visualizing OSGi Dependencies 2011-02-02T16:48:00.005-05:00 Yesterday I blogged about how to find dependencies in features on plugins or features using a shell script to rip through feature jars. But maybe you're less commandline, and more visual? Well, it may be over three years old, but there's a way to visualize plugin interdependencies using Ian Bull's PDE Dependency View. Frankly, I'm amazed this isn't already a core feature in PDE (and correct me if it is). To use this tool, simply install it from: http://download.eclipse.org/eclipse/pde/incubator/visualization/site After installing and restarting, hit CTRL-3 and type "Graph" to find the "Graph Plug-In Dependencies View". (It's also available from Window > Show View > Other... (ALT-SHIFT-Q,Q) under Plug-in Development, if you prefer to kick it old-school.) Next, right-click in the view or hit the "Focus on" button in the view, and select the plugin on which you want to focus. Now you can browse up or down through plugins to explore dependencies. For example, to see what plugins depend on a given plugin, such as org.eclipse.tm.terminal, click the "Show Callers" button in the view. Or, to see on which plugins org.jboss.ide.eclipe.as.rse.ui depends, click the "Show Callees" button in the view. You can shift-click on nodes to highlight them for emphasis, or click and drag them around. Syntax highlighting for code snippets in blogs 2011-02-01T13:26:00.003-05:00 Where does he get those wonderful toys? - The Joker, Batman (1989) Recently I was asked how I make code snippets on this blog prettier. Here's how. In your blog's template, install Alex Gorbatchev's Syntax Highlighter. Then, in the body of the blog post (with all occurrences of "<" HTML-escaped as "&lt"): <pre class="brush:shell"> ... </pre> or <pre class="brush:java"> ... </pre> or <pre class="brush:xml"> ... </pre> There are other "brushes" available too with which you can "paint" your code. I generally only need java, xml, and shell. HOWTO: Find osgi dependencies in features 2011-02-01T01:17:00.011-05:00 Say you're trying to build something with Tycho & Maven 3 and while resolving dependencies before compilation, you're told: [INFO] [Software being installed: org.eclipse.tm.terminal.local.feature.group 0.1.0.v201006041240-10-7w312117152433, Missing requirement: org.eclipse.tm.terminal.local 0.1.0.v201006041322 requires 'bundle org.eclipse.cdt.core 5.2.0' but it could not be found, Cannot satisfy dependency: org.eclipse.tm.terminal.local.feature.group 0.1.0.v201006041240-10-7w312117152433 depends on: org.eclipse.tm.terminal.local [0.1.0.v201006041322]] To quickly verify where this dependency is coming from, you can go look into the feature.xml for the org.eclipse.tm.terminal.local feature jar... but if you don't have it installed, this is somewhat more cumbersome; besides, you then have to unpack the jar before you can look inside it. And maybe that feature contains a number of OTHER dependencies that you'll also need to resolve in your target platform when building. Sure, there are UI tools to do this within Eclipse, but when you're working on remote servers sometimes UI isn't available. Workaround? Assuming you have a mirror of the update site(s) from which you're trying to resolve the dependency (eg., Helios) or can ssh to dev.eclipse.org, you can simply run a quick shell script to do the investigative work for you: $ cd ~/downloads/releases/helios/201009240900/aggregate/; ~/bin/findDepInFeature "*tm*" cdt ./features/org.eclipse.tm.terminal.local_0.1.0.v201006041240-10-7w312117152433.jar <import feature="org.eclipse.cdt.platform" version="7.0.0" match="greaterOrEqual"/> <import plugin="org.eclipse.cdt.core" version="5.2.0" match="compatible"/> <import plugin="org.eclipse.core.runtime"/> Where the script looks like this: #!/bin/bash # find plugins/feature deps by searching in some folder for feature jars, and searching through their feature.xml files for dependencies # 1 - featurePattern - pattern of features to search (eg., "org.eclipse.tptp" or "\*" for all features) # 2 - dependencyPattern - pattern of plugins/feature deps for which to search (eg., "org.eclipse.tptp.platform.instrumentation.ui") # 3 - location - directory in which to search, if not "." if [[ ! $1 ]]; then echo "Usage: $0 <featurePattern> <dependencyPattern> <location>" echo "" echo "Example: $0 tm.terminal cdt" exit 1 fi # if no location, look in current dir (.) if [[ $3 ]]; then location="$3"; else location="."; fi # if no featurePattern, search all features for dependencyPattern if [[ ! $2 ]]; then featurePattern="*"; dependencyPattern="$1"; else dependencyPattern="$2"; featurePattern="$1"; fi rm -fr /tmp/findinfeature/; mkdir -p /tmp/findinfeature/features/ for f in $(find "$location" -type f -name "*${featurePattern}*" | egrep -v "pack.gz|source" | grep features | egrep "${featurePattern}"); do #echo "$f [$featurePattern, $dependencyPattern]" unzip -q $f -d /tmp/findinfeature/ feature.xml # <import feature="org.eclipse.cdt.platform" version="7.0.0" match="greaterOrEqual"/> # <import plugin="org.eclipse.cdt.core" version="5.2.0" match="compatible"/> if [[ ! $(cat /tmp/findinfeature/feature.xml | egrep "<import" -A3 | egrep "plugin=|feature=" -A1 -B1 | egrep "\".*${dependencyPattern}[^\"]*\"" -A1 -B1) ]]; then rm -fr /tmp/findinfeature/feature.xml else mv /tmp/findinfeature/feature.xml /tmp/findinfeature/${f}_feature.xml echo "${f}" cat /tmp/findinfeature/${f}_feature.xml | egrep "<import" -A3 | egrep "plugin=|feature=" -A1 -B1 | egrep "\".*${dependencyPattern}[^\"]*\"" -A1 -B1 echo "" fi rm -fr /tmp/findinfeature/feature.xml done Simplifying The p2 Process, Part 1: p2 Composite Repos 2011-01-29T16:40:00.008-05:00 With the release of JBoss Tools 3.2 and JBoss Developer Studio 4.0 just around the corner, you may be thinking to yourself, "Self, how many update sites and SDK zips and runtimes will I need to download THIS time?" Or maybe you're thinking, "Self, why is this so damn complicated?" Well, folks, we heard your kvetching and we did something about it. Composite Repos While this is not a new concept to many, we embraced the composite update site this past year and it's made life a lot easier for iterative, agile development cycles. Last year, JBoss Tools 3.1 was built as a single Hudson job, with a second one for JBoss Developer Studio. This meant that any change in any of the components would cause a build to be launched, and 4-6hrs later, we'd have fresh bits. Yeah, far from ideal. This year, we split up the monolith (and added a few new components!) so that now we have 34 update sites to compose into a single one against which builds can then be built. This composite update site looks like this: compositeArtifacts.xml <?xml version='1.0' encoding='UTF-8'?> <?compositeArtifactRepository version='1.0.0'?> <repository name='JBoss Tools Staging Repository' type='org.eclipse.equinox.internal.p2.artifact.repository.CompositeArtifactRepository' version='1.0.0'> <properties size='2'> <property name='p2.compressed' value='true'/>  <property name='p2.timestamp' value='1294205433000'/> </properties> <children size='34'> <child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-3.2_trunk.component--archives/all/repo/'/> ... <child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-3.2_trunk.component--ws/all/repo/'/> <child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-pi4soa-3.1_trunk/all/repo/'/> <child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-teiid-designer-7.1_trunk/all/repo/'/> <child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-drools-5.2_trunk/all/repo/'/> <child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-savara-1.1_trunk/tools/'/> <child location='http://download.jboss.org/jbosstools/builds/staging/xulrunner-1.9.1.2/all/repo/'/> </children> </repository> compositeContent.xml <?xml version='1.0' encoding='UTF-8'?> <?compositeMetadataRepository version='1.0.0'?> <repository name='JBoss Tools Staging Repository' type='org.eclipse.equinox.internal.p2.metadata.repository.CompositeMetadataRepository' version='1.0.0'> <properties size='2'> <property name='p2.compressed' value='true'/>  <property name='p2.timestamp' value='1294205433000'/> </properties> <children size='34'> <child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-3.2_trunk.component--archives/all/repo/'/> ... <child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-3.2_trunk.component--ws/all/repo/'/> <child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-pi4soa-3.1_trunk/all/repo/'/> <child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-teiid-designer-7.1_trunk/all/repo/'/> <child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-drools-5.2_trunk/all/repo/'/> <child location='http://download.jboss.org/jbosstools/builds/staging/jbosstools-savara-1.1_trunk/tools/'/> <child location='http://download.jboss.org/jbosstools/builds/staging/xulrunner-1.9.1.2/all/repo/'/> </children> </repository> So, now that JBoss Tools is built in 34 pieces, the bits that haven't changed aren't rebuilt over and over and builds are faster. If that sounds insanely obvious to you, well, we used to have a lot of inter-component cyclic dependencies. We eliminated those early in the development cycle for JBoss Tools 3.2, and have been able to build smarter and faster ever since. Added benefits to this composite site are: Newly built and published bits are instantly available from the composite site - sure, the same was true under last year's PDE "uberbuild" regime, but that's because everything was built fresh every time, which was slow and near-impossible to get people to run at home. Developers can use this site to install latest updates to components they're interested in testing - again, this was true before; but now using the same site and searching for updates, developers and beta testers can get incremental updates to the components that have actually changed, rather than having to pull down 160M every day to get a few K of changes. Tycho can be pointed at this site (see below) in order to resolve binary p2 dependencies, so building a component deep in the dependency chain can be done w/o having to first build its upstream dependencies - this wasn't a concern before because everything was built from source every time, so by definition everything was already on disk. But now, if a developer only cares about a single component, like ModeShape or GWT, they need only have that source (and some bootstrapping code) on disk. Smaller, faster, more agile. And way more likely to be built locally before checking in code than before, making the painful "who broke what and when?" process much less painful. Fewer moving pieces and local dev builds at home mean - in theory - fewer incomplete or breaking commits. When we first moved to Tycho, we needed to build a series of components locally in order to just get to a deep component. For example, the Struts component needs VPE, which needs JST and XulRunner. JST also needs the Common component, which in turn needs the Tests component. So, to build Struts locally, 5 other components would have to be built locally first. This worked, but was still a fairly large barrier to entry for most developers (much less contributors!) But with this new composite site, building Struts can be done without this lengthy bootstrapping; instead we just point Tycho at this composite site, and it pulls down the 5 upstream components' jars from this p2 repo - because the upstream deps are already built in Hudson. Here's what we added to our parent pom.xml to have the builds find the binaries: <repository> <id>jbosstools-nightly-staging-composite-trunk</id> <url>http://path.to.the.site/staging/_composite_/trunk/ </url> <layout>p2</layout> <snapshots> <enabled>true</enabled> </snapshots> <releases> <enabled>true</enabled> </releases> </repository> So, using this composite update site, we can use Maven 3 with Tycho 0.10 to generate a single update site (staged here, then ultimately published here). In part 2, I'll look at why we switched from using a collection of SDKs (Eclipse, EMF, DTP, GEF, M2E, RSE, TPTP, UMl2, WTP, XSD and more) against which to build - using the now-deprecated brute-force "just unzip into eclipse root folder or dropins" approach - to using a single target platform update site. SPOILER ALERT: Easier to update and maintain. In part 3, I'll look back at the success we've had using associate sites instead of asking people to manually add 3rd party URLs when installing JBoss Tools. SPOILER ALERT: one URL is easier for people to use than 6. In part 4, I'll talk a little about how to prevent your product build from getting updates from unofficial sources, and preload your product with the official sites from which to get updates. Because it's important to balance ease of use with prevention of unsupported features. SPOILER ALERT: may contain p2.inf instructions. By the way, JBoss Tools 3.2.0.CR1 and JBoss Developer Studio 4.0.0.CR1 are available. Get 'em while they're hot (and sourceforge is not). HOWTO: partially clone an SVN repo to Git, and work with branches 2011-01-27T19:39:00.015-05:00 Skip to the code I've blogged a few times now about Git (which I pronounce with a hard 'g' a la "get", as it's supposed to be named for Linus Torvalds, a self-described git, but which I've also heard called pronounced with a soft 'g' like "jet"). Either way, I'm finding it way more efficient and less painful than either CVS or SVN combined. So, to continue this series ([1], [2], [3]), here is how (and why) to pull an SVN repo down as a Git repo, but with the omission of old (irrelevant) revisions and branches. Using SVN for SVN repos In days of yore when working with the JBoss Tools and JBoss Developer Studio SVN repos, I would keep a copy of everything in trunk on disk, plus the current active branch (most recent milestone or stable branch maintenance). With all the SVN metadata, this would eat up substantial amounts of disk space but still require network access to pull any old history of files. The two repos were about 2G of space on disk, for each branch. Sure, there's tooling to be able to diff and merge between branches w/o having both branches physically checked out, but nothing beats the ability to place two folders side by side OFFLINE for deep comparisons. So, at times, I would burn as much as 6-8G of disk simply to have a few branches of source for comparison and merging. With my painfullly slow IDE drive, this would grind my machine to a halt, especially when doing any SVN operation or counting files / disk usage. Using Git for SVN repos naively Recently, I started using git-svn to pull the whole JBDS repo into a local Git repo, but it was slow to create and still unwieldy. And the JBoss Tools repo was too large to even create as a Git repo - the operation would run out of memory while processing old revisions of code to play forward. At this point, I was stuck having individual Git repos for each JBoss Tools component (major source folder) in SVN: archives, as, birt, bpel, build, etc. It worked, but replicating it when I needed to create a matching repo-collection for a branch was painful and time-consuming. As well, all the old revision information was eating even more disk than before: jbosstools' trunk as multiple git-svn clones: 6.1G devstudio's trunk as single git-svn clone: 1.3G So, now, instead of a couple Gb per branch, I was at nearly 4x as much disk usage. But at least I could work offline and not deal w/ network-intense activity just to check history or commit a change. Still, far from ideal. Cloning SVN with standard layout & partial history This past week, I discovered two ways to make the git-svn experience at least an order of magnitude better: Standard layout (-s) - this allows your generated Git repo to contain the usual trunk, branches/* and tags/* layout that's present in the source SVN repo. This is a win because it means your repo will contain the branch information so you can easily switch between branches within the same repo on disk. No more remote network access needed! Revision filter (-r) - this allows your generated Git repo to start from a known revision number instead of starting at its birth. Now instead of taking hours to generate, you can get a repo in minutes by excluding irrelevant (ancient) revisions. So, why is this cool? Because now, instead of having 2G of source+metadata to copy when I want to do a local comparison between branches, the size on disk is merely: jbosstools' trunk as single git-svn clone w/ trunk and single branch: 1.3G devstudio's trunk as single git-svn clone w/ trunk and single branch: 0.13G So, not only is the footprint smaller, but the performance is better and I need never do a full clone (or svn checkout) again - instead, I can just copy the existing Git repo, and rebase it to a different branch. Instead of hours, this operation takes seconds (or minutes) and happens without the need for a network connection. Okay, enough blather. Show me the code! Check out the repo, including only the trunk & most recent branch # Figure out the revision number based on when a branch was created, then # from r28571, returns -r28571:HEAD rev=$(svn log --stop-on-copy \ http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x \ | egrep "r[0-9]+" | tail -1 | sed -e "s#$r[0-9]\+$.\+#-\1:HEAD#") # now, fetch repo starting from the branch's initial commit git svn clone -s $rev http://svn.jboss.org/repos/jbosstools jbosstools_GIT Now you have a repo which contains trunk & a single branch git branch -a # list local (Git) and remote (SVN) branches * master remotes/jbosstools-3.2.x remotes/trunk Switch to the branch git checkout -b local/jbosstools-3.2.x jbosstools-3.2.x # connect a new local branch to remote one Checking out files: 100% (609/609), done. Switched to a new branch 'local/jbosstools-3.2.x' git svn info # verify now working in branch URL: http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x Repository Root: http://svn.jboss.org/repos/jbosstools Switch back to trunk git checkout -b local/trunk trunk # connect a new local branch to remote trunk Switched to a new branch 'local/trunk' git svn info # verify now working in branch URL: http://svn.jboss.org/repos/jbosstools/trunk Repository Root: http://svn.jboss.org/repos/jbosstools Rewind your changes, pull updates from SVN repo, apply your changes; won't work if you have local uncommitted changes git svn rebase Fetch updates from SVN repo (ignoring local changes?) git svn fetch Create a new branch (remotely with SVN) svn copy \ http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x \ http://svn.jboss.org/repos/jbosstools/branches/some-new-branch Using Git like CVS 2010-12-30T23:14:00.015-05:00 Tonight I moved the http://rainbowcinemas.ca sources from phpeclipse and CVS to PDT and Git. Below are some gotchas and tips for initial repo creation, how to keep the central remote copy up to date, and how to work around complaints about updating master directly from remote. I'm sure there's a better way to do this w/o the need for the workaround, but this is what I found worked. Initial setup To crawl a directory and create a git project for each subfolder: for d in $(find . -maxdepth 1 -type d -not -name "." | \ egrep -v ".ssh|logs|OLD|download|upload"); do cd $d; \ git init; git add .; git commit -m "initial commit" .; \ cd ..; \ done See also Setting up a shared repository. Create local clone via commandline git clone ssh://servername/path/to/.git folderToCreateLocally Create local clone w/ eGit Once your repo is created, you can clone a copy from the remote server onto your local box, and import it into Eclipse (with eGit installed) using File > Import > Git > Projects from Git > Clone... . Commit local changes via commandline As outlined before, you can git pull, git checkout, git commit, and finally git push your changes. If you encounter an error trying to commit changes back to the repo, see the section below, "Allow a ref update to the currently checked out branch of a non-bare repository". Commit local changes w/ eGit With eGit, you can pull, push, checkout, commit, merge, etc. using the context menu on a Git project or with the Synchronize view. I don't recommend using any of the change sets / models except the Git Change Set, since the others will tend to show more than is actually needed (like local changes which Git doesn't track). Allow a ref update to the currently checked out branch of a non-bare repository Update the ~/.gitconfig file on the remote server to look something like this: [user] name = Your Name email = your@email.address [color] branch = auto diff = auto interactive = auto status = auto [core] editor = vim [merge] tool = vimdiff [receive] denyCurrentBranch = warn Retrieve changes into remote repo Because I'm using the remote server to both host http-accessible files AND host the git repo, it's important that changes checked into the git repo be then checked out into the local filesystem so that the local workspace is in synch with the repo's metadata. To pull changes, I use git status (to review changes), git reset HEAD <filename> (to reset specified file, or omit filename to reset all changes) and finally git checkout to retrieve the changed file from the repo into the working directory. Access the server w/o a password prompt To skip being prompted for a password when you connect over ssh, scp, or fish, add your public SSH key to the ~/.ssh/authorized_keys file on the remote server. Access the server via an alias Instead of having to reference the server as username@server when connecting, you can add an entry to your ~/.ssh/config file that looks like this: Host shortName Hostname fully.qualified.domain.name.or.IP.address User yourUsername Port 22 HOWTO: Contributing to JBoss Tools using Git 2010-11-15T11:46:00.006-05:00 If you'd like to use Git instead of SVN as your SCM tool of choice, here's how you can connect to the JBoss Tools SVN repo, pull down all the sources, work on them locally, then either commit changes back into the SVN repo (or submit a patch, if you're not already a committer). The instructions below assume you have either Linux, Mac OSX, or Windows w/ cygwin. If you have none of those, YMMV. Fetch sources from SVN First, fetch sources from SVN using git-svn. If you don't want to check out all the components, use a subset of the components listed below. The complete list is here. # create a directory into which to check out the JBoss Tools projects mkdir ~/trunk; cd ~/trunk; # fetch projects - this will take quite some time # Committers, use http://svn.jboss.org/repos/jbosstools/trunk/ # Contributors, use http://anonsvn.jboss.org/repos/jbosstools/trunk/ for d in \ archives as birt bpel bpmn build cdi common \ deltacloud documentation download.jboss.org drools \ esb examples flow freemarker gwt hibernatetools \ jbpm jmx jsf jst maven modeshape portlet profiler \ requirements runtime seam site smooks struts \ tests tptp usage vpe ws xulrunner; do \ git svn clone http://anonsvn.jboss.org/repos/jbosstools/trunk/${d}; done Configure Eclipse Next, fire up Eclipse Helios 3.6 for Java EE Developers. Install the latest eGit from http://download.eclipse.org/egit/updates. Install the latest m2eclipse from http://m2eclipse.sonatype.org/sites/m2e/ and optionally, http://m2eclipse.sonatype.org/sites/m2e-extras/. Restart when prompted. Import Git projects into Eclipse Now, import Git projects into Eclipse using: File > Import Git > Projects from Git Click 'Add' then browse for ~/trunk/ Enable [x] Look for nested repositories Click 'Search', then click 'OK' when done Select a local repo from the list, click 'Next' (*) Import Existing Projects (*) Try to share automatically Click 'Next' Click 'Select All', then click 'Finish' Repeat for other components you want to import. You can add each component to a working set to keep your workspace sorted by component. Resolve missing dependencies While the Eclipse Helios 3.6 for Java EE Developers contains most of the dependencies against which JBoss Tools must be compiled, it does not contain everything. For that, you need to install extra dependencies. There are two places to go: JBoss Tools Target Platform p2 Repo (also available as an archived update site zip for offline use) - contains all the Eclipse.org, google.com, and sonatype.org features needed to compile / install all of JBoss Tools. You can install everything, or just the pieces you need. JBoss Tools Nightly Repo (Update Site) - if you don't have all the source projects in your workspace, you can resolve dependencies against this site and install them from here. Once again, you can install everything, or just the pieces you need. Build & run tests With m2eclipse installed, you can simply right-click a project and select 'Run As > Maven Build (ALT-SHIFT-X, M)', which will prompt you complete a run configuration dialog. Here are the simplest options you need to set: Goals: clean install [x] Resolve Workspace artifacts You can also run Maven to build your projects outside Eclipse, if you prefer. If running outside Eclipse, you can run tests which are still tied to the Eclipse debugger. Commit changes to master repo Because there's no support yet for 'git svn rebase' or 'git svn dcommmit' you're stuck pushing changes to the master repo using the commandline. However, you can shorten the amount of typing needed using an .alias file. See below. Use an .alias file To avoid having to type the same git commands over and over, I use these shortcuts in my ~/.alias file: # update local git-svn repo from master SVN repo alias gitup='for d in $(find . -maxdepth 1 -type d); do cd $d; echo $d; if [[ -d .git ]]; then git svn rebase; fi; cd -; done' # Push local changes to master SVN repo alias gp='git svn rebase; git svn dcommit' # commit local changes to local git-svn repo alias ci='git commit -m' # check status of local git-svn repo alias stat='git status' So, after committing changes (with eGit or via commandline) I can push those to the master SVN repo w/ a simple 'gp'. If your shell doesn't read the .alias file, make sure your .bashrc loads the file using one of these commands: source /home/yourUserName/.alias . /home/yourUserName/.alias Or, put them directly in your .bashrc file. Git vs. SVN - Basic Commandline Syntax Reference 2010-11-03T08:36:00.017-04:00 Learning the git workflow takes a bit of brain retraining, but since I've been using SVN almost entirely via commandline (because Subversive sucks and locks up my Eclipse when I try to use it for anything beyond synching/updating/committing a handful of files), adopting git's commandline syntax is reasonably similar. Consider these simple operations: Initial checkout from existing repo for a given branch git clone http://github.com/sonatype/sonatype-tycho.git; cd sonatype-tycho; git checkout origin/tycho-0.10.x svn checkout http://anonsvn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.0.Beta1/ Update locally checked out files from central repo git pull svn update List locally changes files/folders git status svn stat Diff locally changed file git diff somefile.txt svn diff somefile.txt Revert locally changed file* git checkout somefile.txt svn revert somefile.txt Revert ALL local changes (except untracked files)* git reset --hard HEAD svn revert . -R Add new file git add file.txt svn add file.txt Add new folder recursively git add folder svn add folder Delete file git rm file.txt svn rm file.txt Delete folder git rm -r folder (non-recursive by default; use -r to recurse) svn rm folder (recursive by default; use -N to not recurse) Commit changed file to central repo git commit -m "message" file.txt; git push svn ci -m "message" file.txt Ignore files/folders (in the current folder) echo "target *.class bin" > .gitignore; \ git ci -m "gitignore" .gitignore svn propset svn:ignore "target *.class bin" .; \ svn ci -N -m "svn:ignore" . Obviously you can do a lot more w/ Git than with SVN (like stashing local changes temporarily), but for the sake of simply moving from a VCS to a DVCS and being able to continue to work the same way you already do, the above table should provide a good introduction. My First Maven Plugin 2010-11-03T07:54:00.007-04:00 Having been working with Maven and Tycho for about the last 8 months, it was high time I got around to writing my first Maven plugin, because I need to break myself from my tendency to revert to ant and bash whenever I need to do something outside the normal Maven flow. So, first I imported the git sources for Tycho 0.11.0-SNAPSHOT from https://github.com/sonatype/sonatype-tycho so I'd have something from which to learn. Sure, Hello World samples are nice, but a working example is always more fruitful. git clone http://github.com/sonatype/sonatype-tycho.git Next, I imported a few Tycho projects (tycho-p2-facade, tycho-p2-plugin, tycho-p2-publisher-plugin) from the git clone folder into Eclipse using m2eclipse 0.10.2. Because these are Maven projects and don't need Eclipse project clutter, there are no .project files and therefore they couldn't be imported as pre-existing projects. So, I had to wobble through the New > Maven Project wizard a little until I figured out that I could check the 'Create a simple project (skip archetype selection)' box to create a new .project file in an existing project folder. Only wrinkle here is that if there's an existing pom.xml in a project, the New Maven Project wizard complains. The obvious workaround is to rename the existing pom.xml to pom.xml_, create the new Maven Project in w/ m2eclipse in the existing folder (as above), then replace the pom.xml that m2eclipse creates with the actual (renamed) one, pom.xml_ and refresh the project in Eclipse. Voila! I also updated the project settings to use JDK5 instead of the default JDK1.4. You need only do File > Import > Maven > Existing Maven Projects and browse for the pom files. So much easier than what I did above. Thanks to everyone for setting me straight! So, with sample code to clone from it was time to create my own plugin project, then a second project on which that plugin depended in order to test the dependency chain thru Maven. The first uses the "maven-archetype-plugin" archetype v1.1; the second, the "maven-archetype-quickstart" archetype v1.1. Here are the two projects: sample-plugin sample-project The glue between these projects is in the sample-project's pom.xml: <dependency> <groupId>org.jboss.maven.plugin</groupId> <artifactId>sample-plugin</artifactId> <version>0.0.1-SNAPSHOT</version> </dependency> And that's seemingly it. Next... adding actual useful functionality to a plugin, then using that functionality in our JBoss Tools build's parent pom.xml. In Which I Explain Once Again That Linux Is A Viable Alternative to Windows 2010-10-25T12:54:00.008-04:00 I was recently asked this question: Can linux be used for a normal computer, operating email programs, word processing, etc? I am quite frustrated with all the "improvements" that Windows keeps getting; each improvement making it slower and more prone to erratic behavior. I use a computer only for the above tasks, and would really like to get away from the problems. As I've been telling friends, colleagues, family, and everyone who'll listen for about the past 5 years... YES. Mail: Instead of Outlook, you can use Thunderbird. Calendar: use Sunbird or go online w/ Google Calendar Web: Instead of Internet Explorer, Firefox, Opera, Chrome, or Konqueror. None will "accidentally" install spyware for you. Office: Instead of the bloated and dinosauric (20 years old!) MS Office's Word, Excel & Powerpoint, use Open Office's Writer, Calc & Impress. (Open Office is now 10 years old - old enough to be feature rich, young enough to be standards-compliant.) Chat: instead of MSN, use pidgin (supports all IM protocols, including MSN, Yahoo, gtalk, IRC, Twitter, Facebook) and/or Skype (for audio/video chat) Audio/Video playback & streaming: instead of Windows Media Player, VLC player. Solitaire: PySol includes 200 solitaire games; Firefox "Cards" plugin contains dozens more. More advanced users: Graphics: instead of Photoshop, use GIMP MP3 ripping: grip or asunder I personally use linux flavours designed for older machines so they're lightweight, faster, and less bloated. Then, if I need a more "bloated" app (like something from the KDE school instead of the XFCE or GDE school), I simply install that into the operating system as an add-on. Of course if you *want* eye candy (like 3D desktops and transitions when you open/close applications) you can get that too. It's pure eye candy, but it's available if you need Vista or Win7-like "bling". If you want to try Linux before jumping in fully, I advise two options: Xubuntu, designed for old machines and to stay more-or-less the same over time. Download and install it into Windows without having to reformat your hard drive. Good for your grandmother's desktop machine. Fedora, designed for newer machines and to stay up to date with recent improvements in the Linux world. Can be installed onto a USB key so you can boot your system from that without having to touch your existing Windows install. Good for your parents or your machine, or for an office. There are of course lots of other Linux distros out there... HOWTO: Find the feature that contains a plugin 2010-10-08T00:05:00.003-04:00 Tycho is awesome. However, like all build systems, it has its limitations. One such limitation is that when you're building against a target platform, and something's missing, you get errors such as these: [INFO] Cannot complete the request. Generating details. {org.osgi.framework.executionenvironment=OSGi/Minimum-1.0,OSGi/Minimum-1.1, osgi.ws=cocoa, osgi.arch=x86, osgi.os=macosx, org.eclipse.update.install.features=true, org.osgi.framework.system.packages=} [Software being installed: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier, Missing requirement: org.eclipse.tptp.platform.instrumentation.ui 4.4.1.v201009092123 requires 'bundle org.eclipse.hyades.probekit [4.2.0,5.0.0)' but it could not be found, Cannot satisfy dependency: org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.1.v201009092123-797908s73533D4H6D56 depends on: org.eclipse.tptp.platform.instrumentation.ui [4.4.1.v201009092123], Cannot satisfy dependency: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier depends on: org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.0] [ERROR] Internal error: java.lang.RuntimeException: org.eclipse.equinox.p2.core.ProvisionException: No solution found because the problem is unsatisfiable. -> [Help 1] org.apache.maven.InternalErrorException: Internal error: java.lang.RuntimeException: org.eclipse.equinox.p2.core.ProvisionException: No solution found because the problem is unsatisfiable. The important part of that error message is as follows: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier requirement: org.eclipse.tptp.platform.instrumentation.ui 4.4.1.v201009092123 requires 'bundle org.eclipse.hyades.probekit [4.2.0,5.0.0)' but it could not be found org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.1.v201009092123-797908s73533D4H6D56 depends on: org.eclipse.tptp.platform.instrumentation.ui [4.4.1.v201009092123] dependency: org.jboss.tools.tptp.feature.feature.group 1.2.0.qualifier depends on: org.eclipse.tptp.platform.instrumentation.ui.feature.group 4.3.0] So, how do you find which feature contains that plugin, so that you can add it to your target platform? First, you need access to the repository. If you have direct server access to the repository from which the plugin comes (eg., the TPTP Helios update site), you can run this script in the root of that repository. If you don't have server access (eg., you can't ssh to dev.eclipse.org and look in ~/downloads/tptp/updates/helios), then you can pull down a zip of the site (or use a p2.mirror script to fetch a copy of the site to your local machine)... and then run this script in the root of that repository. Essentially the script finds matching feature jar files, unpacks them to extract the feature.xml files therein, and then greps those files for lines which suggest an included plugin matching the pattern for which you're searching: $ findInFeature platform.probekit ./features/org.eclipse.tptp.platform.probekit_4.5.1.v201009092123-7H7BF8PAkF7B77ZARCNEK.jar <plugin id="org.eclipse.tptp.platform.probekit" ./features/org.eclipse.tptp.platform.trace_4.5.1.v201009092123-7L7O8bBgJ9E99jAfGWEM.jar <plugin id="org.eclipse.tptp.platform.probekit.launch" From there, it's a trivial exercise to add another line item into your target platform file. First, paste in the feature jar: ./features/org.eclipse.tptp.platform.probekit_4.5.1.v201009092123-7H7BF8PAkF7B77ZARCNEK.jar Then use vim to pattern-replace that string: :%s/.\+\/$org.\+$_$\d\+.\+$\.jar/\t\t\t/g And you'll end up with a new .feature.group added to the target: <unit version="4.5.1.v201009092123-7H7BF8PAkF7B77ZARCNEK" id="org.eclipse.tptp.platform.probekit.feature.group"/> JBoss Tools: making it easier to build against a complex target platform 2010-10-01T22:43:00.008-04:00 So you want to be a JBoss Tools developer? Awesome. Welcome to the family. SVN sources are here, JIRA's over here and there's cold beer in the fridge*. But you say it's a pain in the tuchus to download over 25 zips or add a whole bunch of update sites and hope you get everything you need? Yeah, no argument there. If only there was an easier way to resolve all the dependencies you need to get building, much less to even RUN this stuff. To make this process simpler, I've created a p2 repo (update site) from our target platform file, which has been recently updated to include Helios SR1 dependencies. You can track subsequent work in progress here: JIRA JBIDE-6982. You can also report any issues there too. So, now, just add this single site** into your vanilla Eclipse 3.6.1 Classic (or a Helios SR1 bundle), uncheck the box for 'Group Items by Category' and you can install everything listed. For great justice. Some handy links: Latest Target Platform Repo Archived Target Platform Repo Target Platform Definition File JBoss Tools SVN Sources JBoss Tools JIRA Some handy HOWTOs: HOWTO build JBoss Tools (all or individual components) using Tycho HOWTO run JBoss Tools' tests (all or individual components) using Tycho HOWTO attach Eclipse's debugger to JUnit or SWTBot tests running w/ Tycho HOWTO convert from a .target into a repo HOWTO update a .target to include newer versions based on contents of a local repo HOWTO install from p2 repo(s) into Eclipse using p2.director script * - Due to beer2peer limitations, YMMV. ** - I'm aware that the update site throws a 403 if you open it in a browser. I can't be arsed to generate an index.html just yet, nor are there categorized features. Because really, you don't need either - this site is only meant to be used by p2. Linux modem tethering with berry4all, Rogers 3G network, and Blackberry Bold 9000 2010-08-16T00:56:00.012-04:00 Because I'll be out of town for the next couple days, part of my packing list included setting up tethering via my Blackberry. First I tried barry but was unsuccessful in trying to make a successful connection. Then I found berry4all. I had two problems, but thanks to the extensive FAQ, I managed to solve them: Ensure your device (in my case, Blackberry Bold 9000) has its APN correctly set in Setup > Options > Advanced > TCP Ensure other network connections are disabled using route -n; ifconfig -a before attempting to connect via tethered 3G modem. See also If your PPPD does not support replacedefaultroute Ensure /etc/resolv.conf is being correctly updated with the appropriate DNS entries (I have to do this by hand at the moment, but tweaked the default /opt/bbtether/berry4all.sh script to add this line so that I can automatically update the file immediately before connecting: #!/bin/sh cp -f /etc/resolv.conf.bbtether /etc/resolv.conf python bbgui.py & where /etc/resolv.conf.bbtether contains: domain rogers.com search rogers.com nameserver 64.71.255.198 #nameserver 64.71.255.253 Load up a bandwidth speed test such as bandwidthplace.com and see what network speed you can get. Admittedly, tethered speeds are nothing close to wifi speeds. With Rogers 3G network and Blackberry Bold 9000 attached via USB, I managed to get about 500-2800kbps down and 100-200 kpbs up. With Rogers "Exterme" cable internet access (wifi @ home) I get 9800kbps down, 960kbps up. But at least I can surf on my laptop using the 3G network while I'm away from the office. To disconnect wired & wifi, then tether automatically and stay connected until CTRL-C is pressed, try this script: #!/bin/bash if [[ $UID != 0 ]]; then echo "Must be run as root!" else /etc/init.d/network stop cp /etc/resolv.conf.bbtether /etc/resolv.conf route -n; ifconfig cd /opt/bbtether; python bbtether.py rogers -P **YOUR_DEVICE_PASSWORD_HERE** /etc/init.d/network start route -n; ifconfig fi p2 Repository Association And The Fine Art Of Forceably Enabling Disabled Sites 2: Tycho Edition 2010-08-11T11:39:00.004-04:00 Back in February, I blogged about how to add associate sites to an update site generated w/ the Buckminster or B3 aggregator. Since we've moved our build infrastructure to use Tycho + Maven, I had to port the script over to work there for our update site aggregation. Here are the moving pieces: pom.xml that builds the site, listing input sites as <repository> entries, and a shout-out to build.xml, below, to do extra stuff during the install phase build.xml ant script called by pom.xml, above, to do additional work after the creation of the site_assembly.zip update site zip aggregateSite.jbosstools.properties properties file listing the aggregate sites to add Troubleshooting Eclipse.org Mirrors / Using Profiles & Target Platform Definition Files With Tycho 2010-07-20T14:33:00.005-04:00 If you look in the mirror and you say his name 5 times, he'll appear behind you breathing down your neck. - Candyman If only troubleshooting mirrors was so simple. Have you ever been running a build or an install which stalls at the provisioning step, with a message like: [INFO] Fetching org.eclipse.birt.integration.wtp.ui_2.6.0.v20100617-1315.jar.pack.gz (4.3MB of 46.13MB at 171.56kB/s) from http://mirrors.xmission.com/eclipse/birt/update-site/2.6/plugins/org.eclipse.birt.integration.wtp.ui_2.6.0.v20100617-1315.jar.pack.gz The solution here is often simply to force p2 to pick a specific mirror rather than letting it choose any mirror it wants. How, you ask? Well, assuming you were polling this site looking for artifacts to install or update... http://download.eclipse.org/birt/update-site/2.6/ ... you would then change that URL to this, and look at the list of available mirrors: http://www.eclipse.org/downloads/download.php?file=/birt/update-site/2.6/ Now it's a trivial matter to select a mirror that's close to you and try that instead of the download.eclipse.org mirror, such as: ftp://mirror.csclub.uwaterloo.ca/eclipse/birt/update-site/2.6/ If you're running a Tycho build, this URL should be changed in your parent-pom.xml ... <url>http://download.eclipse.org/birt/update-site/2.6/</url> ... or your .target platform file, depending on which way you're building. <repository location="http://download.eclipse.org/birt/update-site/2.6/"/> If you rely on a parent-pom.xml, make sure you're activating the profile with the revised URL... mvn3 clean install -U -B -fae -Phelios ... or, if you're building against a .target platform file, make sure you update the URL in that file, and that your build points to the profile which will load the .target file. mvn3 clean install -U -B -fae -P!helios,helios-no-target UPDATE, 2010/08/11: Forgot to mention that there are a number of p2 update site zips available here to help with your offsite mirroring: http://download.eclipse.org/athena/repos/. Blog comments or spam? No idea. 2010-07-03T01:59:00.003-04:00 I don't speak Chinese. Or Korean. Or Japanese. I mention this because lately I've been getting a couple comments per post in a foreign character set, and I can't tell if it's spam or a legit comment. Please, if you have something to say, stick to English, French, German, or Spanish. Hell, even Latin, lolcode, or twitterspeak. Anything other than that I'll just delete as spam. kthxbai, Nick Update Site Aggregation: B3 vs. Tycho 2010-06-30T11:10:00.004-04:00 Last year, I used to aggregate update sites with the Buckminster Aggregator, but since that won't install into Eclipse 3.6 (Helios), I had to migrate to the B3 Aggregator. This new version of the Aggregator is greatly expanded and worked fine until recently, when it has begun to suffer from a rather nasty p2 problem: instead of the Director just installing the aggregator into an Eclipse instance prior to then running the aggregation, I get "The copies of profile SDKProfile are not in sync," and the whole process dies. http://www.blogger.com/img/blank.gif So, in order to find a workaround, I went back to Tycho, and discovered you can merge update sites with little more than two simple files: a site.xml, which lists the features to aggregate and how to categorize them, and a pom.xml to list the source sites and drive the aggregation. Unfortunately the Tycho solution doesn't include the ability to add associate sites to the metadata after generation, but that can simply be done as a downstream step (Tycho can call Ant using the maven-antrun-plugin). pogoplug network attached storage (NAS) for linux, mac, windows 2010-06-21T17:22:00.010-04:00 A couple months ago I bought a PogoPlug. Because it's an ungodly pink colour and because I've been hella busy with other stuff, I finally unboxed it today, and discovered that while: it's a snap to configure through the web UI it works with ext3 and fat32 drives it's SSH accessible, and there are other installable linux apps for it ... setting it up for Linux was not entirely trivial. While the above article helped, a couple points were not immediately obvious: If you mount the drive as root, it's not accessible as anyone else; if you mount as the user, not even root can read it. Use of a /etc/pogoplugfs.conf file is highly recommended, so that you can log in w/o having to key in your username and password every time. Create this file as root, and set it chmod 644 if not already: svcuser=you@email-provider.com svcpassword=web-login-password logseverity=10 To automount the drive on startup (eg., on my old xubuntu 6 system), as root, try creating a new file, /etc/init.d/pogoplug: #! /bin/sh ### BEGIN INIT INFO # Provides: # Required-Start: $remote_fs $syslog # Required-Stop: $remote_fs $syslog # Should-Start: $named # Default-Start: 2 3 4 5 # Default-Stop: 1 # Short-Description: pogoplug fs mount as myusername # Description: pogoplug fs mount as myusername ### END INIT INFO sudo -u myusername /usr/bin/pogoplugfs --mountpoint /media/pogoplug exit 0 Then, to activate this file, as root: chmod +x /etc/init.d/pogoplug; \ for x in 2 3 4 5; do \ cd /etc/rc${x}.d; ln -s /etc/init.d/pogoplug S99pogoplug; \ done With Fedora 12, I could simply create a startup task using System < Preferences < Startup Applications from the gnome menu. #FailWhale v2.0 2010-06-18T12:42:00.005-04:00 #Fail Bot? -- Someone's been watching too much made-for-tv sci-fi, I think My love-hate with SVN, Part 8: Installation Ease Of Use (UPDATED) 2010-05-25T16:19:00.004-04:00 Back in July 2009, I blogged about My love-hate with SVN, Part 6: Installation Ease Of Use. With Helios just around the corner, I wanted to produce an updated repo for use with the latest & greatest Eclipse 3.6. Now the <p2.mirror/> script will fetch and use Ant-Contrib automatically. Here's the updated 15M update site zip, which includes the following: Subversive 0.7.9 SVN Connector 2.2.2 SVNKit 1.3.3 JNA 3.2.3 ECF 3.1.0 Any problems, please report them in bug 284077. So long and thanks for all the faece 2010-05-04T01:11:00.003-04:00 I'm done with Faecebook. Here's why. And here's how to quit. I encourage you to do the same. Instead, find me on Twitter and LinkedIn. HOWTO: Build a XulRunner 1.9.1.2 Update Site with Tycho 0.8 + Maven 3 2010-04-16T03:11:00.008-04:00 0. Install Maven 3 by downloading then unpacking the tar.gz: cd /tmp; \ wget http://mirror.csclub.uwaterloo.ca/apache/maven/binaries/apache-maven-3.0-alpha-7-bin.tar.gz; \ tar xvzf apache-maven-3.0-alpha-7-bin.tar.gz; \ chmod 755 apache-maven-3.0-alpha-7/bin/mvn; \ alias mvn3='/tmp/apache-maven-3.0-alpha-7/bin/mvn' 1. Check out sources: cd /tmp; \ svn co http://anonsvn.jboss.org/repos/jbosstools/branches/modular_build/xulrunner/; \ wget http://anonsvn.jboss.org/repos/jbosstools/branches/modular_build/parent-pom.xml 2. Run build: cd xulrunner; mvn3 -fae clean install 3. You will get a p2 repo / update site in the target folder of the site project, from which you can install XulRunner or XPCOM into your Eclipse. cd site/org.mozilla.xulrunner.site/target/site Note that the parent-pom.xml used above can in fact be much simpler. You only need the following: <?xml version="1.0" encoding="UTF-8"?> <project> <modelVersion>4.0.0</modelVersion> <groupId>org.jboss.tools</groupId> <artifactId>org.jboss.tools.parent.pom</artifactId> <version>0.0.1-SNAPSHOT</version> <name>JBoss Tools Parent</name> <packaging>pom</packaging> <properties> <tychoVersion>0.8.0</tychoVersion> </properties> <build> <plugins> <plugin> <groupId>org.sonatype.tycho</groupId> <artifactId>tycho-maven-plugin</artifactId> <version>${tychoVersion}</version> <extensions>true</extensions> </plugin> <plugin> <groupId>org.sonatype.tycho</groupId> <artifactId>target-platform-configuration</artifactId> <version>${tychoVersion}</version> <configuration> <resolver>p2</resolver> <environments> <environment> <os>macosx</os> <ws>cocoa</ws> <arch>x86</arch> </environment> <environment> <os>macosx</os> <ws>carbon</ws> <arch>x86</arch> </environment> <environment> <os>win32</os> <ws>win32</ws> <arch>x86</arch> </environment> <environment> <os>linux</os> <ws>gtk</ws> <arch>x86</arch> </environment> <environment> <os>linux</os> <ws>gtk</ws> <arch>x86_64</arch> </environment> </environments> </configuration> </plugin> </plugins> </build> </project> Better Bootstrapping For Athena Projects 2010-04-15T16:51:00.003-04:00 Inspired by Tycho and Maven, I've simplified the self-bootstrapping process for Athena builds. Previously, you needed a start.sh script to fetch org.eclipse.releng.basebuilder and org.eclipse.dash.common.releng before you could run a build; if on build.eclipse.org this is done for you automatically. Or, if building locally (or on Windows), you had to fetch these yourself. Now, you can simply tell your build.xml to bootstrap itself, and it will fetch those projects for you. You can also tell it which version of basebuilder or common.releng to use, rather than being tied to the defaults. For details, see Getting Started - Bootstrapping. Cleaner Ant Build Logs: Get Rid of "Trying to override old definition of task" 2010-04-04T12:57:00.005-04:00 If you use macros in Ant, you have probably seen clutter in your log such as Trying to override old definition of task ... Here's how to purge this garbage from your log and keep your build output cleaner. move your macros into a target such as <target name="init"> have your main target(s) depend on the target that defines the macros call your macros You can make your log even quieter with -q when running ant, eg., ant -f build.xml -q Here's a complete example that will first self-bootstrap itself by downloading Ant-Contrib, define two ant macros, and produce very quiet output (unless debug=true): <project default="run"> <property name="COMMON_TOOLS" value="${java.io.tmpdir}" /> <property name="debug" value="false"/> <target name="get.ant-contrib" unless="ant-contrib.jar.exists"> <property name="ANTCONTRIB_MIRROR" value="http://downloads.sourceforge.net/ant-contrib/" /> <get usetimestamp="true" dest="${COMMON_TOOLS}/ant-contrib-1.0b2-bin.zip" src="${ANTCONTRIB_MIRROR}/ant-contrib-1.0b2-bin.zip" /> <touch file="${COMMON_TOOLS}/ant-contrib-1.0b2-bin.zip" /> <mkdir dir="${java.io.tmpdir}/ant-contrib-1.0b2-bin.zip_" /> <unzip src="${COMMON_TOOLS}/ant-contrib-1.0b2-bin.zip" dest="${java.io.tmpdir}/ant-contrib-1.0b2-bin.zip_" overwrite="true" /> <copy file="${java.io.tmpdir}/ant-contrib-1.0b2-bin.zip_/ant-contrib/lib/ant-contrib.jar" tofile="${COMMON_TOOLS}/ant-contrib.jar" failonerror="true" /> <delete dir="${java.io.tmpdir}/ant-contrib-1.0b2-bin.zip_" includeemptydirs="true" quiet="true" /> </target> <target name="init"> <available file="${COMMON_TOOLS}/ant-contrib.jar" type="file" property="ant-contrib.jar.exists" /> <antcall target="get.ant-contrib" /> <taskdef resource="net/sf/antcontrib/antlib.xml"> <classpath> <pathelement location="${COMMON_TOOLS}/ant-contrib.jar" /> </classpath> </taskdef> <macrodef name="debug"> <text name="echo" /> <sequential> <if> <and> <isset property="debug" /> <istrue value="${debug}" /> </and> <then> <echo message="@{echo}" /> </then> </if> </sequential> </macrodef> <macrodef name="list.count"> <attribute name="list" default="" /> <sequential> <var name="count" value="" /> <for param="listitem" list="@{list}" delimiter=", "> <sequential> <var name="count" value="${count}0" /> </sequential> </for> <length property="list.count.return" string="${count}" /> </sequential> </macrodef> </target> <target name="run" depends="init"> <property name="list" value="foo bar baz"/> <list.count list="${list}" /> <debug>For list [${list}], size = </debug> <echo>${list.count.return}</echo> </target> </project> HOWTO: Build Plugin & Feature Projects, Then Run Their Unit Tests w/ Tycho :: GEF Example 2010-03-31T16:55:00.006-04:00 Note that the instructions below are for Linux (or MacOSX). On Windows, your YMMV, but the process is the same. 1. Check out the entire source tree of your project from CVS, SVN, or Git into ~/build. 2. If needed, move plugins, features & test plugins into: ~/build/plugins/ ~/build/features/ ~/build/tests/ (Test features should go into features/ folder too.) 3. Install scala from http://www.scala-lang.org/downloads 4. Fetch genpom.scala and parent-pom.xml; save in ~/build or equivalent. 5. Tweak parent-pom.xml to suit your needs or use as-is. 6. Run this to generate pom.xml files for plugins, features, and tests: cd ~/build/plugins/; scala ../genpom.scala cd ~/build/features/; scala ../genpom.scala cd ~/build/tests/; scala ../genpom.scala 7. Download Maven 3 from http://www.apache.org/dyn/closer.cgi?path=/maven/binaries/apache-maven-3.0-alpha-7-bin.tar.gz 8. Install Maven 3 sudo su; cd /opt; tar xvzf apache-maven-3.0-alpha-7-bin.tar.gz ln -s apache-maven-3.0-alpha-7 maven3 9. For convenience, alias mvn3 to the new maven: alias mvn3='/opt/maven3/bin/mvn 2>&1 clean install | tee buildlog.latest.txt' 10. Build the plugins, features, and finally tests: cd ~/build/plugins/; mvn3 cd ~/build/features/; mvn3 cd ~/build/tests/; mvn3 Look in ~/build/plugins/org.eclipse.*/target/ for generated jars. Look in ~/.m2/repository/org/eclipse/* for published jars. To automate steps 6 and 10, you can run this script in the ~/build/ folder: #!/bin/bash for f in plugins/ features/ tests/; do \ cd $f; scala ../genpom.scala; \ /opt/maven3/bin/mvn 2>&1 clean install | tee buildlog.latest.txt; \ cd ..; \ done (The above blog is also posted here.) Dash Athena: Post-EclipseCon Wrap-Up 2010-03-28T13:07:00.008-04:00 Updates to Athena since Feb 2010 Ease of use New documentation available for migrating from Bash to Ant build.xml is now simpler. Sample project template updated. You can now use the same build.xml script to run a build in Eclipse or in Hudson from an Ant-based job. Note that the page Build In Hudson/Ant Script is no longer current. See also bug 304800. You can now run buildExtra.xml#extraPackaging via build.steps=buildExtra rather than requiring build.steps=buildZips. Publishing support will improve thanks to bug 306854... anyone want to contribute to writing a Hudson job, please feel free! Packaging Support bug 306300 Athena removes .jar files and only contains pack 200 files in update site - new default setting (keep both artifact types) is removeUnpackedJars=false, but can revert to old behaviour (and smaller update site) with removeUnpackedJars=true bug 307016 SDK and Runtime zips produced with buildZips target are missing {notice,epl-v10}.html files; if root files exist they will now be copied into the SDK, Runtime, Examples zips; ${allZip} will also include this so it can be copied from there too (custom packaging in buildExtra.xml#extraPackaging task) Testing Support bug 296352 Can't connect to vnc server - fixed using Xnvc option in Hudson job and improvements to testLocal task Publishing Support bug 302170 Work around Hudson's missing lastS*Build folders - promote.xml will now recurse into Hudson job tree looking for correct build to publish Publishing support will improve thanks to bug 306854... anyone want to contribute to writing a Hudson job, please feel free! Bug Fixes bug 304800 Temporary regression caused by adopting new build.xml script with too-aggressive cleanup default bug 307016 SDK and Runtime zips produced with buildZips target are missing {notice,epl-v10}.html files; if root files exist they will now be copied into the SDK, Runtime, Examples zips; ${allZip} will also include this so it can be copied from there too (custom packaging in buildExtra.xml#extraPackaging task) Documentation & Branding New documentation available for migrating from Bash to Ant EclipseCon 2010 Presentation, "Dash Athena Exposed" is available here. Common Build Infrastructure/Getting Started and Common Build Infrastructure/Getting Started/Build In Eclipse updated. Sample project template & generic build.xml updated. bug 272723 Logo design contest for Athena under way: vote early, vote often! I'm in love with Tycho 0.8 and Maven 3 2010-03-24T14:43:00.009-04:00 Monday: Signed up for GitHub, downloaded apache-maven-3.0-alpha-7-bin.tar.gz, and built Tycho from source. Took a bit of path wrangling (namely figuring out that maven runs based on current directory), but it built and most of the tests passed. Not bad for my first day ever using Maven. # First download Maven3 and unpack into /opt/ export MAVEN_OPTS="-Xmx512m" export TYCHO_TARGET_PLATFORM=/home/nboldt/eclipse/35clean/eclipse mvn=/opt/maven/bin/mvn $mvn clean install -e -V -Pbootstrap-1 $mvn clean install -e -V -Pbootstrap-2 -Dtycho.targetPlatform=$TYCHO_TARGET_PLATFORM $mvn clean test -e -V -Pits -Dtycho.targetPlatform=$TYCHO_TARGET_PLATFORM The really confusing part was "what now? what did I just build? And how do I use it?" Answer: You can now run maven3 from .../sonatype-tycho/tycho-its/target/apache-maven-3.0-alpha-7/bin/mvn Tuesday: Attended the Sonatype-sponsored Tycho Build Workshop, and amazed Pascal by saying good things about Tycho and Maven3. Apparently I'm known to complain about all things p2, despite my numerous blogs and wiki contributions touting its success and usefulness. Also installed m2eclipse to see how effective the pom editor is. During the workshop, I got the following JBoss Tools projects to build: jmx, archives, as, flow, jbpm3/4, drools, bpel, smooks, common. That's nearly half our projects! Thanks to Jason and Igor for their invaluable assistance. The reason it was so easy was that Max had already created a scala script to generate all the pom.xml files we need to build JBoss Tools, and a parent-pom.xml on which those tiny pom.xml files depend. (For me as a Maven noob, not having to think at all about creating poms made this transition a no-brainer. Speaking of no-brainers, my slides from Monday's talk about Dash Athena and Zombies are here (PDF). As an added bonus, Tycho, being more strict than PDE, exposes missing information in MANIFEST.MF files such as dependencies on plugins. PDE-based builds are apparently more forgiving because they generally include more crap than is actually needed in the target platform against which the build runs; Tycho builds against what you tell it you need. So, the mavenification of JBoss Tools will actually cause us to have better described plugins, and therefore make them easier to build anywhere. I'm very impressed so far. Unlike Dash Athena which needs to be told all the IUs (features/plugins) against which your project needs to build, Tycho determines this information from your manifests, so the information need not be duplicated. I still need to look at how to build update sites from the output of a Tycho build, and how to hook in SWTBot/JUnit4 tests. I have a suspicion this may be handled using Buckminster tools such as their Aggregator, or my own custom Ant hackery using the SWTBot wiki docs. Time will tell. Wed This morning on a whim I decided to see if I could port Max's scala script (aside: I've never seen scala before yesterday) to generate pom.xml files for EMF or GEF. Success! With some minor tweaks, I've managed to build GEF 3.6 into my local maven repo (~/.m2). Obviously there's more to be done here... but damn, this is easy. Beyond CVS 0.8.9 Released! 2010-03-05T17:07:00.004-05:00 It's true that the shoemaker's children often go shoeless; so to my little Sourceforge project, Beyond CVS/SVN - maintained tirelessly over the years by Chris Callendar - has had to wait *years* to finally get its own update site. Sad, I know. But thanks to a recent breakthough in being able to insert associate sites into p2 repo metadata, you can now install Beyond CVS/SVN without having to hunt down Subclipse or Subversive's update sites. Here's the new site: http://beyondcvs.sourceforge.net/update/0.8.x. If you point Eclipse at it, you can install the features therein. If you point a browser at it, you can see instructions for how to do the install. And if you want an offline install, here's the zip: org.eclipse.externaltools-Update-0.8.9.v201003051612.zip (README). Athena: p2 repos w/ associate sites 2010-03-04T20:23:00.005-05:00 Want to insert associate update sites into your user's Eclipse when they go to install your features from an update site/zip? With an Athena build this is now a trivial exercise. For example... say you're building a plugin which supports integration with Subclipse or Subversive. In a traditional install from an SDK or runtime zip, you'd unpack into eclipse/, and wonder why the new plugins didn't work. In the new p2 install world, you point Eclipse at the update site or zip, and are told that you cannot install the SVN integration because missing requirements cannot be found. So you hunt down the required sites, add them to Eclipse, and try again; this time you are allowed to install the new features. To improve the ease of use for the end user, Athena now allows you to define associate.sites in your build.properties. If this property is set, URLs will be added into your p2 repo's metadata and Eclipse will automatically add those sites when it scans your zip or site. No more manual copy+paste from documentation needed! associate.sites=\ http://subclipse.tigris.org/update_1.6.x,\ http://download.eclipse.org/technology/subversive/0.7/update-site/,\ http://community.polarion.com/projects/subversive/download/eclipse/2.0/update-site/ And, of course, you can reuse these sites when resolving dependencies against which to build or test: repositoryURLs=\ http://download.eclipse.org/releases/galileo/,\ http://download.eclipse.org/athena/repos/eclipse-Update-3.5.2-201002111343.zip,\ http://download.eclipse.org/eclipse/updates/3.5/,\ ${associate.sites} IUsToInstall=org.eclipse.sdk.feature.group+org.eclipse.sdk.ide+\ org.eclipse.team.svn+org.eclipse.team.svn.core+org.eclipse.team.svn.ui+\ org.tigris.subversion.subclipse.core+org.tigris.subversion.subclipse.ui p2 Repository Association And The Fine Art Of Forceably Enabling Disabled Sites 2010-02-18T21:36:00.005-05:00 This week I've spent some time exploring how to associate p2 repo sites with other sites. The reason for this is that we publish features which depend on other publishers' features, and so would like to be able to have our update site automatically enable sites on which we depend in the end user's Eclipse. The first attempt to do this was by using p2.inf files. Based on examples on blogs and in newsgroups, I added this p2.inf file into our BIRT Integration feature: instructions.configure=addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:0,name:Eclipse Galileo,enabled:true); \ addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:1,name:Eclipse Galileo,enabled:true); \ addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:0,name:Eclipse 3.5,enabled:true); \ addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:1,name:Eclipse 3.5,enabled:true); \ addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:0,name:BIRT 2.5,enabled:true); \ addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:1,name:BIRT 2.5,enabled:true); Unfortunately, this didn't work - I got this error while building: java.lang.IllegalArgumentException: No action found for: addRepository. So, hoping that using a fully-qualified name for the action would help, I tried this: instructions.configure=org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:0,name:Eclipse Galileo,enabled:true); \ org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/releases/galileo/,type:1,name:Eclipse Galileo,enabled:true); \ org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:0,name:Eclipse 3.5,enabled:true); \ org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/eclipse/updates/3.5/,type:1,name:Eclipse 3.5,enabled:true); \ org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:0,name:BIRT 2.5,enabled:true); \ org.eclipse.equinox.p2.touchpoint.eclipse.addRepository(location:http${#58}//download.eclipse.org/birt/update-site/2.5/,type:1,name:BIRT 2.5,enabled:true); Success! The build no longer complained. Of course I wasn't getting any associate sites enabled when I added the new repo into Eclipse. Why, I wondered? Thankfully, I was told that the now-deprecated Metadata Generator (org.eclipse.equinox.p2.metadata.generator.EclipseGenerator) doesn't support p2.inf files, so it was finally time to switch to Eclipse 3.5's Update Site Publisher (org.eclipse.equinox.p2.publisher.UpdateSitePublisher), which can be run like this as a drop-in replacement for the above application: <echo>Run p2.publisher.UpdateSitePublisher using launcherjar = @{launcherjar}</echo> <java jar="@{launcherjar}" fork="true" timeout="10800000" jvm="${java.home}/bin/java" failonerror="false" maxmemory="256m" taskname="p2"> <classpath> <fileset dir="${builder.build.path}/plugins" includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar" /> <fileset dir="${clean.eclipse.home}/plugins" includes="org.eclipse.equinox.launcher_*.jar, org.eclipse.equinox.p2.publisher_*.jar, org.eclipse.equinox.p2.updatesite_*.jar" /> <pathelement location="${builder.build.path}/plugins" /> <pathelement location="${clean.eclipse.home}/plugins" /> </classpath> <arg line=" org.eclipse.equinox.launcher.Main -application org.eclipse.equinox.p2.publisher.UpdateSitePublisher" /> <arg line=" -metadataRepository file:${updateSiteJarDir}/ -metadataRepositoryName "${update.site.product.name} ${update.site.description} Update Site"" /> <arg line=" -artifactRepository file:${updateSiteJarDir}/ -artifactRepositoryName "${update.site.product.name} ${update.site.description} Artifacts"" /> <arg line=" -source ${updateSiteJarDir}/" /> <arg line=" -compress -publishArtifacts -reusePack200Files -configs *,*,*" /> </java> Having switched out the deprecated app for its successor, I then discovered that using the instructions.configure touchpoint was way too late in the process to be of any use; this information is only used when you install the feature, not when you poll the repo for metadata. So to add the BIRT 2.5 update site I'd have to install the BIRT integration feature... which depends on the BIRT 2.5 site. I needed to inject this requirement in at an earlier point in the process. So, the next step was to hit up #equinox-dev and ask for help, which I got from Pascal "LeNettoyeur" Rapicault, fellow IBM alumnus. He suggested that I append into the content.xml file in my repo the following information. Note that type 0 and 1 represent metadata repo and artifact repo, and options=1 or 0 is used to mark the site enabled or disabled. <references size='2'> <repository uri='http://download.eclipse.org/birt/update-site/2.5/' url='http://download.eclipse.org/birt/update-site/2.5/' type='0' options='1'/> <repository uri='http://download.eclipse.org/birt/update-site/2.5/' url='http://download.eclipse.org/birt/update-site/2.5/' type='1' options='1'/> </references> </repository> Success, I again thought, since the associated sites were finally being added and enabled into Eclipse when the repo was scanned for metadata. I even scripted this process so that it could be integrated into our build process, or simply reused, eg., to merge two update sites and add some associate site for optional feature installs. But alas, every step forward brings another step backward, as I discovered that available but disabled sites are NOT enabled despite the instructions in the repo. In other words, I cannot force an existing, yet disabled site to become enabled because my repo requires it. I can only document the requirement and hope that users will RTFM without getting too annoyed. So, what's the solution? Well, there currently isn't one. But if this scenario seems like something you'd like to see improved, please cast your votes & comments in one of these bugs: http://bugs.eclipse.org/303258 http://bugs.eclipse.org/241307 http://bugs.eclipse.org/229183 Ontario's capital now in the U.S.A. 2010-02-16T11:44:00.005-05:00 Air Canada does not permit flight searches which have both an origin and destination in the United States. -- When did Toronto (YYZ) move to the US? I suspect this is why Air Canada keeps needing federal bail out money: geography lessons. Monitor fail 2010-01-18T09:54:00.002-05:00 Well, that's a week of my life I'll never have again. Spent most of last week playing with WinXP, LinuxMint 8, and Fedora 11 & 12 (both 32- and 64-bit) trying to find a solution to my external display issues. Turns out that it's a software issue: the latest xorg intel driver has a bug. Beyond Compare 3 for 64-bit ubuntu linux 2010-01-12T11:56:00.005-05:00 Last week, I switched my xubuntu 8.04 (Hardy Heron) box to Linux Mint 8 (Helena), to see how things had evolved in the past 18 months. Unfortunately, I also decided to switch from 32- to 64-bit, and of course not everything works the way it used to. Luckily, the good folks at Scooter Software pointed me to this hack to allow their 32-bit Linux flavour of Beyond Compare 3 to find the 32-bit libraries it needs to run. So, here's an updated version of that hack for ubuntu 9.10 Karmic Koala or Linux Mint 8 Helena: Download the i386 libqt3-mt .deb package into /tmp Download the latest Beyond Compare 3 .deb package into /tmp $ sudo su apt-get install ia32-libs cd /tmp; dpkg-deb --extract /tmp/libqt3-mt*.deb libqt3-mt cp -R libqt3-mt/usr/lib/* /usr/lib32; rm -fr /tmp/libqt3-mt dpkg -i --force-architecture /tmp/BCompareLinux*.deb PDE Best Practices : A Noob's Guide To Plugin & Build Health 2010-01-05T12:19:00.002-05:00 I've drafted a collection of tips for keeping your plugins healthy and your builds green. Feel free to add to, change, or comment on the list here: http://wiki.eclipse.org/Common_Build_Infrastructure/Getting_Started/FAQ/Plugin_Health_And_Best_Practices Switching monitors with xrandr 2010-01-05T01:11:00.000-05:00 Sure, there are applets and other GUI tools to do this, but nothing beats the purity of going commando, er, commandline. :) Thanks to this article for the inspiration. # reset xrandr --auto # turn off laptop screen xrandr --output LVDS --off # 24" 16x12 on xrandr --output VGA --mode 1600x1200 # reset xrandr --auto # turn off external screen xrandr --output VGA --off # laptop screen on xrandr --output LVDS --mode 1280x800 UPDATE, 2010/01/05: To enable both screens and force the larger one to include the taskbar: # reset xrandr --auto # laptop screen on xrandr --output LVDS1 --mode 1280x800 --noprimary --pos 1600x0 # 24" 16x12 on xrandr --output VGA1 --mode 1600x1200 --primary --pos 0x0 # echo current status xrandr --current Discount Fail 2009-12-31T02:40:00.006-05:00 I'm having a problem w/ a purchase. I added a Blackberry Bold compatible 2400mAh battery to my cart, on sale for 20% off ($24.07). On page two, the price was still 20% discounted: On page three of the cart, the price jumped back up to the full $30.09 price. So... is this item on sale, or not? Feeding The Right Sources To Your Builds 2009-12-04T11:32:00.007-05:00 I was recently asked the best approach for how to control the input to one's builds, and while there is no single solution to this that fits all projects, here are two ways you might want to go. Freeze Then Branch The current approach for JBoss Tools development involves continuous nightly builds from trunk until such time as the project is deemed frozen in prep for an upcoming milestone or release candidate. At that time, the trunk is copied to branches/JBossTools-3.1.0.M4, and very little is done to that branch - only urgent tweaks and fixes. A new Hudson job is cloned from the current one, and adjusted to point at the branched sources instead of trunk. The maps are also updated to point at the new source location in SVN. This allows more nightly builds towards an upcoming stable development milestone, while new development can continue in parallel in trunk. When the milestone is released, the unneeded job is disabled or removed. The only issue with this approach is that all plugins, built from the head of a branch (or trunk) are re-versioned w/ the latest timestamp every time they're compiled. Thus upgrading from one 80M update to the next requires another 80M download. To mitigate this, milestones are released only once every month or two. Tag And Release The current approach for numerous Eclipse.org projects, such as GEF, is to develop continuously in HEAD and when a number of changes are ready to be released, a tool such as the org.eclipse.releng.tools plugin can be used to, in a single operation, both tag the sources and release those tags into the map file. This permits a granular approach to plugin versioning wherein only the plugins which have actually changed are renumbered, and thus incremental updates between releases is possible, and if only a single plugin changes from week to week, only that plugin will be upgraded. This approach also allows your adopters to get more stable, weekly updates rather than hourly or ad hoc nightlies which may include incomplete changes. Sure, you can only watch for good builds in Hudson, but a more predictable schedule makes inter-build communication easier. The only issue with this approach is that it introduces extra overhead, unless the tag&release process can be automated and run on a (weekly?) schedule. For CVS sources, there is the org.eclipse.releng.tools.tagandrelease crontab script available; for SVN, no such script (yet) exists. Want to help? See bug 264713. Hybrid Approach With the Athena Common Build, you can set up a Hudson job to run from tagged maps, but also override those tags on demand to force a build from head (or trunk) instead of from the specific repo tags. To do so, pass in the following flags to EXTRAFLAGS via your build's job configuration / parameters. To build from the workspace's sources instead of fetching anything new from the repo (bug 252774): -localSourceCheckoutDir /opt/users/hudsonbuild/.hudson/jobs/${JOB_NAME}/workspace/ To fetch from the repo, but ignore tags (bug 251926): -forceContextQualifier -fetchTag HEAD Or, like many projects on build.eclipse.org have done, set up two jobs: one for nightly from trunk, and one for weekly integration from maps. Then instead of doing CI builds when you remember to, they will run automatically when the repo changes so you'll have immediate feedback when something has broken. Add email, IRC, or Twitter notification to the job, and everyone else will know too. Dash Athena: More Ant, More Tests, More Repos! 2009-11-27T00:57:00.005-05:00 Infrastructure Changes bug 295394 - Headless UI tests are broken on build.eclipse.org! See workaround here. Cross-Platform / Ease of Use bug 282987 Can now define Eclipse platform as a p2 repo + required features/plugins. Can also use platform, SDK, or RCP zip, as before. See also Defining Binary Dependencies. bug 292589 Support building from Ant in Hudson (instead of via Bash script) bug 295670 Support running JUnit tests run from Ant script instead of Bash - tests can now be run on Linux and Windows (and to a lesser extent on Mac OSX) using new testLocal build step Bug Fixes bug 292486 Allow builds to fail if unit tests fail using failBuildIfTestFailuresOrErrors=true bug 294678 Categories don't show up with IBM 1.6 JDK - implemented workaround so a different JDK can be used for p2 operations than for compilation of sources: use PACK200_JAVA_HOME=/path/to/different/JDK bug 295773 Non-incubating projects no longer need to set "incubation=" in their build.properties bug 292235 Included pre-compiled binary features/plugins are now included by default using PDE's runPackager property; can disable this behaviour with packageFeaturesIncludedBinaries=false Documentation Testing - Different ways to run or re-run tests, including as a secondary process after a build (so they can be run on a different platform or by a different user) Defining Binary Dependencies - All four types of repos are now currently supported: remote zip, remote site, local zip, local site. p2 does not natively support remote repo zips; to workaround this the zip is fetched and the URL is rewritten from http://path/to/foo.zip to jar:file:/tmp/path/foo.zip!/ Build In Hudson - Ant Job, Ant Script - new, simpler way to run builds in Hudson using pure Ant Build In Hudson - Bash Job - updated & refactored Previous New & Noteworthy | More Athena Docs Ontario [GNU] Linux Fest 2009 2009-10-19T16:58:00.003-04:00 Annoyed that your boss didn't approve that expense to fly over to Germany for Eclipse Summit Europe? Well, fret no more! Have we got a deal for you! For only $40 you can attend Ontario [GNU] Linux Fest 2009 - a whole day of open source geekery from the following speakers! And, while there, stop by the Eclipse and Fedora tables for your chance to win some schwag! I'd like to say we'll be giving away Slap-Chops or Sham-Wows too, but Vince will be rappin' & reppin' elsewhere. :) HOWTO: Join Architecture Council, Break Hudson, Break Athena 2009-10-19T00:28:00.006-04:00 About a week ago, Wayne Beaton approached me with an unexpected question: would I like to be drafted into the rank and file of the Eclipse Architecture Council? I was honoured. Flabbergasted. And a bit apprehensive: could I measure up? Well, it seems my first act as an AC member was, Thursday night, to cripple our shared Hudson build server by dumb-assedly performing a seemingly innocuous update. Heartfelt thanks to the Webmasters for once again extracting my cheeks from the fire and getting everyone's builds back up and running. My second act, Friday afternoon, was to commit some "fixes" to the Athena Common Build code which effectively prevented it from reading your project's build.properties. Thankfully, the ever-vigilant Dave Carver spotted the snafu, filed a bug, and I had it my error fixed shortly after my return from Zombieland last night (aside: best zom-rom-com since Shaun Of The Dead!) Thanks to Matt, Dave C, David W, Denis, Gunnar, and everyone else for reporting &/or fixing these issues, and to the AC for welcoming me into their fold. It's going to be interesting! Dash Athena News, Oct 2009 2009-10-13T20:00:00.001-04:00 Have you heard the latest? Here's what's been going on in the Dash Athena project lately. Infrastructure Changes There are now 43 Athena jobs on build.eclipse.org! Of those, 30 are green, 1 is yellow, and 6 have not yet been enabled. These jobs represent 29 different projects' builds! 6 of them use SVN sources instead of CVS. bug 257074 comment 12 build.eclipse.org now has SVN 1.6.5; if your map files no longer work (your build complains it can't resolve plugin or features sources) then read this. New Features bug 291446 Provide hook for extra actions after fetching code from repo and before compiling it (e.g. code generation, parser generators, etc.) bug 275529 Athena is now a full Project rather than a Component! Now if we could just get someone to design a logo... Do I need to offer up prizes? I will, if you comment on bug 272723 with some design ideas (or prize ideas). Better Documentation Tips for Building on Windows - community contributed! Thanks to Nicolas Bros! New category created for User-contributed build stories: Category:Athena Common Build Users Athena bugs moved from product=Dash&component=Common+Builder to product=Dash+Athena. Here's what's changed this week. See also Older New & Noteworthy. HOWTO: Enable Zimbra filtered message folders on Blackberry w/ BES service 2009-09-10T12:04:00.002-04:00 Thanks to the folks at my IT helpdesk for this tip on how to enable your BB to get mail from within Zimbra folders: Go to your Messages folder, hit menu key and select Options. Go to Email Settings. Hit menu button, select Folder Redirection. You will see your mailbox which can be expanded. Expand your inbox and select the folders you want to be sync'd (on my device they're blue instead of grey). Hit menu button, select Save. Any NEW messages routed to those folders will now be located in the folders on the device as well. As new messages, they'll appear in your Messages folder along with SMSs, MMSs, BBIMs, and other email. Once read, they'll disappear into the appropriate folder(s). Re-Return to Athena 2009-09-07T23:46:00.006-04:00 Thanks to Miles for taking the time to slap Athena into submission on his local system - frequent and regular stress testing is what's going to make this facade over PDE bridge the gap between today and the far-flung future (namely B3). Because he took the time to itemize his concerns / problems / gotchas, I thought I'd take the time to explain why these happen... and which are bugs vs. features. Speaking of bugs and features, Athena has a handy New & Noteworthy wiki page which I update about once a month. If you've never seen the N&N, it's here. # skipPack=true is useful if you want to test locally. I found that the update manager does not work with pak'd jars when running locally. Perhaps P2 is relying on something on the web server side..? If p2 isn't properly unpacking pack'd jars, the problem is either: you're using a JDK with pack200 bugs (like Sun 1.6), or you have jars which should never be packed. How can you exclude jars from packing? See JarProcessor Options. # The build scripts appear to be simply searching for the occurrence of strings to determine wether a given map entry type is being used. They do not appear to respect the comment lines. I guess I should file a bug report on this one -- it cost a bit of time because I have a dodgy SVN setup on my home laptop. I'm actually still not sure what is going on with that (and does the mysterious Java 134 error msg have anything to do with it) but I'm trying to learn not to fix things for which I have a work-around, i.e. if ti works on a different machine, just use that instead! The purpose of the collectMapTypes task is to simply know what types of maps are being used so that warnings can be echoed to the user if appropriate. If you use SVN on linux, there's a good chance you'll get this message: [echo] If you get a SIGSEGV or Java Result: 134 when fetching from SVN, remove [echo] subversion-javahl and svnkit to force PDE to use commandline svn client. Why? Because it's a pain in the butt to debug, and at least having a message in the build log gives you some tea leaves to read. I favour documentation over performance in this case. # There is a somewhat lengthy description at http://wiki.eclipse.org/Common_Build_Infrastructure/Getting_Started of how to setup your local build to use a local cache CVS site but I'm not sure what scenario that would really be helpful in. I just use COPY and it works fine. I suppose that using the CVS approach might excercie some issues that you wouldn't run into with a plain COPY. And I'm not sure wether the local build copies over the binary files or not. If it does, you could get different build results if say your local environment happens to place some artifacts in the copied directories that aren't cleaned out by the PDE build. Not going to worry about it! A more useful scenario is to take a dump of a CVS, SVN, or Git tree, and point your build at that folder using localSourceCheckoutDir=/path/to/full/source/tree/root in your build.properties file. Then it'll build against that local source cache instead of having to fetch from the repo. Once you have the dump of the repo, from Eclipse do File > Import > Existing projects, and you can have those same projects compilable/editable in your workspace. You can also use Eclipse (eg., with Subversive, Subclipse, eGit) to update the cache before doing a build. # A useful hack when building locally is to pack up your difficult dependency(s) into a zip file and refer to them in your dependencyURL. It doesn't matter what you use for the URL, because it will use the cached file instead. For example, GEF3D doesn't yet have a build or update site. (We've been collaborating a bit on this so hopefully the build I've got working will help them get one up soon.) So instead of solving the the SVN stuff right away, I put that off until everything else worked. So I created a zip and added an entry for http://download.eclipse.org/tools/gef/fake/gef3d.zip. You have to be careful to have your zip file packaged up correctly or it won't work. That's easy to get wrong..your file structure needs to conform to the standard structure, i.e. eclipse/plugins/some.plugin. On the other hand, you have to be careful to remove these fake dependencies when you begin testing the real fetched versions! The fetch part then fails silently; your build succeeds but using the previously downloaded files. I generally add comments (TODO, FIXME) in my build.properties to mark temporary hacks, or use a new file, like build.properties.local.hack so I'm not breaking the server-friendly version. (Just make sure you tell build.xml which build.properties file to use, if not the default one -- and DON'T CHECK IN the changed file.) # But in any case getting the SVN map file right has been a major headache. Again, the only reason I want this is to grab the GEF3D sources, but I can't seem to let this one go as it seems that it should be so easy. The Athena build seems to mangle things in the fetch script; or perhaps this is a PDE Build issue. So I tried the sourceforge style entry and after a couple of hours of fiddling (see below) I found a magic incantation that seems to work -- plugin@org.eclipse.draw3d=SVN,trunk,http://dev.eclipse.org/svnroot/technology/org.eclipse.gef3d,,org.eclipse.draw3d@HEAD @HEAD is optional; you only need to specify a tag if it's not the HEAD of the branch or trunk. Here's a sample SVN map. # It would be super nice if there were a built-in dependency analyzer so that one could find the root plugin issues easily. As it is, if you have a bad dependency you get the whole tree and you have to walk the tree by hand to find out the root cause. Overall I think this is another demonstration of the ultimate weakness of the standard build systems out there. They are by nature linear (batch) approaches and so usually you can't do post hoc or inline analysis of issues. There is no semantic information in logs! You end up having to parse them back if you want to do any useful analysis. In a parallel lifetime, I would love to work on a simple semantic logging system that would allow structured analysis of this sort. I believe the Buckminster guys are doing work in this area, so this may make its way into B3. # Here is something else that I would love to do given time. The map files are pretty well structured - it would be nice to whip up a quick XText editor that would at least check syntax and more ambitiously provide a mechanism for hooking in semantic checks -- wouldn't it be nice if you could check that that SVN url is correct while editing the map file itself?! Add that to the wishlist for B3! # I'm still not sure how Athena determines what goes in the sdk vs. runtime vs. example zips so for now I'm just building the examples manually. Anyone come across documentation for this? This arbitrary allocation of files is done in common.releng/tools/scripts/buildAllHelper.xml, lines 1063-1097, and is only done if the build.steps entry 'buildZips' is enabled. This behaviour is as far as I'm concerned deprecated, but in place simply to ease transition for existing builds using the old Common Modeling Build to the Athena Common Build. IMHO, no one needs these zips - you only need an update site zip. If you don't like what's in the zips, you can omit the 'buildZips' step and package your own zips by hand, eg: <zip destfile="${buildDirectory}/${buildLabel}/${SDKZip}" update="true"> <zipfileset src="${buildDirectory}/${buildLabel}/${allZip}" dirmode="775" filemode="664" excludes="**/*.pack.gz, **/pack.properties, **/features/*.jar, **/${domainNamespace}.*.all*, **/${domainNamespace}.*.all*/**" /> </zip> (For those unfamiliar with the story of the Re-Return, you need to watch HIMYM, S1E15.) Opera 10 as Hudson Helper 2009-09-06T17:38:00.006-04:00 As there's still no Hudson Helper for Blackberry (David: hint, hint!), I've been forced to do my own monitoring in a browser (when not monitoring by email, that is). Now, with Opera 10 (and its new tab layout options), it's even easier to monitor multiple builds via a single view: HOWTO: AVI to DVD Conversion, Part 2: Merging Multiple AVIs 2009-08-27T01:24:00.007-04:00 Converting a single AVI to DVD format is easy. However, if you want to merge multiple files into a single DVD image, you must: Convert the AVI files to MPEG: for f in $(ls /path/to/season1/{GI_Joe.S1E0*.avi,GI_Joe.S1E1*.avi,GI_Joe.S1E2{0,1,2,3}*.avi}); do \ g=${f/.avi/.mpg}; g=${g/season1/season1_mpg}; \ transcode -i $f -y ffmpeg --export_prof dvd-ntsc --export_asr 3 -o movie \ -D0 -s2 -m movie.ac3 -J modfps=clonetype=3 --export_fps 29.97; \ mplex -f 8 -o $g movie.m2v movie.ac3; \ rm -fr movie.m2v movie.ac3; \ done; \ cd /path/to/season1_mpg Next, using a more complex dvdauthor.xml file... <dvdauthor dest="DVD_Season1_Ep01-05"> <vmgm /> <titleset> <titles> <pgc> <vob file="GI_Joe.S1E01.The_Further_Adventures_of_G.I.Joe.mpg" chapters="0,8:00,16:00,21:00"/> <vob file="GI_Joe.S1E02.Rendezvous_in_the_City_of_the_Dead.mpg" chapters="0,8:00,16:00,21:00"/> <vob file="GI_Joe.S1E03.Three_Cubes_to_Darkness.mpg" chapters="0,8:00,16:00,21:00"/> <vob file="GI_Joe.S1E04.Chaos_in_the_Sea_of_Lost_Souls.mpg" chapters="0,8:00,16:00,21:00"/> <vob file="GI_Joe.S1E05.Knotting_Cobras_Coils.mpg" chapters="0,8:00,16:00,21:00"/> </pgc> </titles> </titleset> </dvdauthor> ...merge the MPEG files into a single disc image: dvdauthor -x dvdauthor_s1e01-05.xml Verify the video and audio will play. xine dvd:/full/path/to/DVD_Season1_Ep01-05 Burn the DVD. growisofs -Z /dev/dvd1 -dvd-video DVD_Season1_Ep01-05/ Yo, Joe! We Don't Need Another Repo 2009-08-13T16:46:00.002-04:00 Re Wayne's blog about an Eclipse.org Maven Repo: EMF has had an undocumented/unmarketed Maven2 repo for about 4 years now. All you need is to take an update site zip, unpack it, rearrange the folder structure, and rename the jars. Then you create little XML files called .poms to describe the jars in the tree, and Maven-aware tools can read the tree. It's fairly trivial. http://download.eclipse.org/modeling/emf/emf/maven2/ is the URL, IIRC. About once every 2 years someone asks about our providing such a repo, and I give out the URL. Clearly not a huge demand for it. One might argue that creation of such a folder structure is in the purview of Athena's publishing scripts, which today eases the process of copying your bits to download.eclipse.org, then unpacks your Update site so it can be scanned by p2 rather than downloaded as a single archive. It too is fairly trivial. I would not be adverse to converting my existing shell script for the EMF repo creation into a generic Ant script for use by Athena users. Frankly though, I think it would be more valuable if the m2eclipse folks added support for reading/converting p2 repos. Publishing yet another file format would require another release-train-like workflow (we already have two: EPP and buckybuilder for Galileo) and more people maintain it. Even if every project published their own maven repo, we'd want for the sake of ease of use to aggregate them into a central place for easier navigation and discovery by maven tools. So, like with Ganymede, we'd have each project's bits copied to two places on disk for each build. (Galileo used composite repos to POINT at project repos rather than copying them saving tons of disk space and CPU cycles. AFAIK, Maven does not support this concept, but I could be wrong.) There's also another benefit to having tooling to support converting from p2 repo to maven2 repo: the aggregate repo could be housed at apache.org and suck THEIR bandwidth and support resources instead. Thus just as Eclipse.org is upstream from Fedora's Eclipse project .rpms (which are upstream from Debian/Ubuntu's .debs), Eclipse.org p2 repos could be upstream from Apache's Maven repo(s). After all, Apache already collects maven artifacts for non-apache.org projects to facilitate the use and adoption of maven, so this is entirely in line with their standard operating procedures. Thoughts? Posted from Blackberry using Opera Mini, by the side of Shawnigan Lake, Vancouver Island, BC My love-hate with SVN, Part 8: Unprotected & Unhidden Metadata 2009-08-07T13:25:00.004-04:00 SVN metadata appears when searching Why would anyone ever want to see .svn folders and their children in Eclipse? If they're hidden from the Package Explorer, why can't I hide them from a Search? And if I did a naive find and replace, wouldn't this corrupt the metadata and prevent me from committing the update to the repo? Hudson & Virtual Box, Part 2: Creating a common share with vboxsf 2009-08-06T22:31:00.007-04:00 My experiment last week with sshfs came to a grinding halt when I realized that while the share works, and can be automatically started, the permissions do not work and Hudson can't actually use the sshfs share; also, I couldn't get sshfs to compile for OpenSolaris, so it's really a non-starter. Next up was mount_nfs, but after fighting with that, /etc/nfs.conf and /etc/exports for a couple days off and on, I realized there's a much simpler solution using Virtual Box itself: vboxsf. Note that it's sf (shared folder) not fs (file system). Thanks to David Herron for the inspiration. Here's how to set up a vboxfs share: Ignore the Virtual Box wiki which suggests that you can't just use a Shared Folder: you can. Guest: Launch your guest, then install the Virtual Box Guest Additions onto it. Host: Create a user and set its uid and gid to some value which can be created on the guest / slave. For example, on Mac OSX 10.5 Server, launch System Preferences > System > Accounts, and create a "hudsonbuild" user with User ID and Group ID set to 500. Note that uid 500 and gid 500 are the defaults for the first user on a Fedora system; on Ubuntu it's 1000; on other systems YMMV. Host: Create a folder in the root of your host called /shared. Put some files there. Give it full write permissions, and set its ownership to the user who'll be sharing the files: find /shared -type f -exec chmod 666 \; find /shared -type d -exec chmod 777 \; chown -R 500:500 /shared Host: Add a Shared Folder. NOTE: If your guest is running, shut it down or see the next item. From the Virtual Box GUI (on the host), select Settings > Shared Folders > New (+). Add a shared folder called "shared" mapped to /shared (or call it something else and map it to a different path on your host). Guest: Alternatively, you can add a Shared Folder to a running guest from its Devices menu. Add a shared folder called "shared" mapped to /shared. Guest: As root, create a folder on the guest called /shared. Leave it empty, but make sure it's owned by your user. Guest: As root, add this to the guest's /etc/fstab: shared /shared vboxsf auto,exec,rw,uid=500,gid=500 0 0 Guest: As root, mount the share one of three ways: # mount everything in /etc/fstab mount -a # mount using mount (and optional -o options) mount -t vboxfs -o auto,exec,rw,uid=500,gid=500 shared /shared # mount using mount.vboxsf (and optional -w and -o options) /sbin/mount.vboxsf -w -o rw,uid=500,gid=500 shared /shared Guest: If this mount doesn't start automatically when the VirtualBox guest is started, you can add one of the above commands to the script you use to start the Hudson slave, eg: !/bin/bash # # Init file for Hudson server daemon # # chkconfig: 2345 65 15 # description: Hudson server # HOWTO: # 1. rename this file on target server (Hudson slave node) to /etc/init.d/hudson # 2. to enable this script to run on startup, as root type the following: # Fedora: # chkconfig --add hudson; chkconfig --list hudson # OpenSolaris: # for n in 0 1 6; do cd /etc/rc${n}.d; ln -s /etc/init.d/hudson K15hudson; done # for n in 2 3 4 5; do cd /etc/rc${n}.d; ln -s /etc/init.d/hudson S65hudson; done # svccfg add hudson; svcs | grep hudson RETVAL=0 #run Hudson as a slave node java -jar /opt/hudson/slave.jar -jnlpUrl \ http://vboxsvr:8080/computer/${slaveNodeName}/slave-agent.jnlp & /sbin/mount.vboxsf -w shared /shared & exit $RETVAL Hudson & Virtual Box: Creating a common share with SSHFS 2009-07-28T01:19:00.006-04:00 I've been working a recently with my own Hudson server, setting up Fedora and OpenSolaris images as slave nodes to prototype new ways to build. Tonight I found a very handy trick for mounting shared drives using SSHFS. Kudos to the man page writers -- all the tips I found via Google were outdated or just plain wrong. man sshfs to the rescue! Step 1: create a folder on the Hudson master / Virtual Box host with files you want to share, eg., /shared Step 2: create an empty folder on each Hudson slave / Virtual Box guest as a mount point for the shared folder, eg., /shared Step 3: on the guest, install sshfs, eg., sudo yum install fuse-sshfs. Step 4: mount the share: sshfs user@hudson-master-host.your-domain.com:/shared /shared Of course you'll probably want to set up ssh keys and add this to your startup scripts so that the mount happens automatically when the guest is launched, but I'll leave that for next time. Athena Common Builder: Thanks to the early adopters! 2009-07-24T02:12:00.006-04:00 Here's a quick list of some of the projects @ Eclipse.org currently using Athena. If you haven't tried Athena, maybe it's time! Linux Tools Visual Editor (VE) Voice Tools (VTP) PDE Declarative Services Modeling Incubator (pde.ds.modeling.incubator) Nebula Widgets Gallery Faceted Project Framework (fproj) EMF Query (EMF MQ) EMF Validation (EMF VF) EMF Transaction (EMF MT) Ajax Tools Framework (ATF) And @ JBoss.org, all available in JBoss Tools 3.1.0.M2 for Eclipse 3.5 (Galileo) ... JMX Console BPEL Editor* jBPM 3 and 4 FreeMarker ...to name but a few. * - will be available in M3 or you can get a nightly build here. What are people saying about Athena? "[I] really like the new build. It is much less confusing then the old [Common Modeling] one." "[C]ongrats for this builder. It is quite good and I'm eager to rely fully on it." "We are playing around with Athena and finding it really useful. It is already deployed to one of our customer developments. Thanks a lot for the hard work! We will definitely try to contribute back anything useful we will have in the process." So, how do *you* get started? Here's a FAQ. Here's our New and Noteworthy. And here's all the rest of the knowledge base articles. Oh, and I'd be remiss if I didn't mention that there's a big list of requested features still waiting for contributions. If you use Athena, you're already half-way to contributing back. Want to help? Drop me a line any time and we can discuss what holes need filling that match your skills and most directly improve your use of PDE, p2 and Athena. Using Hudson for parallel Athena builds 2009-07-22T18:48:00.006-04:00 This week I set up a multi-configuration Hudson job to try doing EMF's Query, Validation, and Transaction projects as a single job (instead of the traditional three linked jobs), then Bernd took over to do most of the work required to migrate to the new, simpler build system... and unearthed a few sticky bugs along the way. For example, Athena now supports running from jarred test plugins, as well as test plugin folders. (Yes, I'm amazed no one ever tested that use case before now either. Anyway, it works now.) Doing these builds in parallel (rather than serially) may be a bad idea, but for now, it lets the Q,V,T developers build all three linked components via a single build button, and if the sources for any of the three changes, all three will be rebuilt and tests will be run. Since Transaction depends on Query and Validation, this is certainly useful; however, Query and Validation do not depend on each other, so there may be some extraneous build churn with this set up. Still, it's nice to see a the status of all three builds in one place: What's next? Well, hooking up FindBugs, eliminating all the old SDK/runtime zips (because we have the new, shiny, p2 update site zip instead, and you can convert from repo to "runnable" SDK easily using this script), and figuring out how to make the builds use each others' newer binary output instead of building from a known entity (Query 1.4.0.N.ten.minutes.ago vs. 1.3.0.R.last.month). Luckily, Athena supports building against p2 repos and Hudson has APIs for fetching the latest good output, so this should be as simple as configuring the Transaction build to fetch from the latest successful Query zip using properties like these: dependencyURLs=https://build.eclipse.org/hudson/.../path/to/Update.zip Of course if you want to build against the latest PUBLISHED update, you could do this: repositoryURLs=http://download.eclipse.org/modeling/emf/query/updates/nightly/ featureIDsToInstall=org.eclipse.emf.query.sdk My love-hate with SVN, Part 7: Setting svn:ignore properties 2009-07-21T21:07:00.001-04:00 If you've inherited code where someone has accidentally committed *.class files, here's how you can find those bin/ folders, wipe them out, and hopefully never see them in the Synchronize view again. Sure, you could use Team > Add to svn:ignore... on a file or Team > Set Property... on a folder in Eclipse using Subversive, but this is faster and less clickity-click. for d in $(find -maxdepth 1 -mindepth 1 -type d ~/workspace/ -name "org.eclipse.*"); do pushd $d svn up svn propset svn:ignore "bin bin/*" . svn propget svn:ignore . svn propset svn:ignore "*.class */*.class **/*.class" bin svn propget svn:ignore bin cd bin rm -fr * svn delete org svn delete model # find other deleteables with `svn status` cd .. svn ci -m "svn:ignore" popd done HOWTO: generate .m3u playlist from .mp3 directory 2009-07-21T01:36:00.003-04:00 Been trying to find a solution to this one for ages. Turns out it's stupidly simple - just dump the results of a find into a file. #!/bin/bash dir="$1" echo "Create playlist for $1 ..." if [[ $2 ]]; then list="$2"; else list="$1"; fi pushd "$dir" 2>&1 >/dev/null find . -type f -name "*.mp3" > "$list.m3u" echo "Found these files:" cat "$list.m3u" popd 2>&1 >/dev/null With or without the "./" prefix on each found file, the resulting .m3u files work on my Blackberry (and are found automatically), including nested folders and paths with spaces. To run the above, just do this: $ ./m3u.sh "artist folder/disc folder" "playlist name" My love-hate with SVN, Part 6: Installation Ease Of Use 2009-07-20T23:45:00.007-04:00 For months I've been annoyed by the fact that installation of Subversive (or Subclipse) requires fetching features and plugins from 3 or more update sites. No more! Today, as an exercise to learn how to use the <p2.mirror/> task and provide a reproduceable, offline way to get Subversive into a virtual machine, I've created an update site zip, complete with site.xml and p2 metadata, which can be used to install Subversive from a single source. Here's the Ant script if you'd like to try this at home. Because let's be real: you can only complain so long before it's time to roll up your sleeves and pitch in, right? That's how open source survives - thanks to people who care enough to complain AND care enough to help. Here's the 13M update site zip, which includes the following: Subversive 0.7.8 SVN Connector 2.2.1 SVNKit 1.2.3 JNA 3.0.9 ECF 3.0.0 Any problems, please report them in bug 284077. HOWTO: Burn ISO image to DVD w/ linux commandline 2009-07-18T12:46:00.005-04:00 When burning an ISO image to disc, I simply use this: $ growisofs -dvd-compat -speed=1 -Z /dev/dvd=disc.iso Using -speed=1 takes a little longer than the default "as fast as possible" mode, but guarantees the disc can be properly read in the pickiest of drives (eg., a Wii DVD drive). DVD-R (not DVD+R) is also recommended. HOWTO: Be full of C.R.A.P. 2009-07-16T21:17:00.003-04:00 In part 1 I rambled on at length about what I think needs to be done to prove yourself to a project team in order to become a committer on tht project. So, what the crap's up with being full of C.R.A.P.? I'm not referring to the four principles of design (Contrast, Repetition, Alignment, Proximity), though there are some similarities here. For me, being full of C.R.A.P is about the transition from one state to another: Contributing Responsive Able Peer to Committed Responsible Active Participant How do you move from one state to another? Simple, right? 'Till next time... HOWTO: Becoming an open source project committer 2009-07-16T18:20:00.003-04:00 The Tweetosphere/blogosphere has been buzzing with discussions about what one needs to do to be a committer @ Eclipse.org. I got my rights by working for IBM and being handed the keys to the Porsche when I started working at the Toronto Lab as a member of the EMF team, oh so many lunar eclipses ago. No longer with IBM, I'll retain my committerships until I manually ask to be removed, or they claw 'em from my cold dead hands. After all, what's a revised patch but a 2nd Amendment? (Aside: seriously, people, it's 2009. You don't need a gun. There's no Imperial Army coming to steal your land. LET IT GO.) For most committers, however, you can't just be appointed to the job; you have to earn it. So, here are my tips for getting on *my* project, the Athena Common Build. Easiest way to get on the project: be invited by someone already on the team by personal recommendation (see criteria below). Others can +1/-1 the suggestion based on the criteria below, but in my experience with other projects, no one ever vetoes a nomination. (I've seen it once, and it only delayed that person's committership by about a month.) So cozy up to the existing committers, and you're in. Why is this? Because it's OPEN source, and how can you be open if you exclude people who want to contribute? The nominee must use the project at least weekly, if not daily. For Athena, this means you have to be actively writing Ant scripts, doing builds, or at least be active in PDE or p2 development. Why is this important? a) I don't want "dump and run" code which I'll then have to maintain, and b) if you're not a user, you can't intelligently decide what pains exist and which are important to solve I'd like to see two accepted patches to prove you've got the technical skill, and that you're willing to thrown down and help with existing known issues - see 2 (b) above. If you're not technical (or not *yet* technical), then you need demonstrated skills or commitment, or have worked in a related field with someone mutually known who can vouch for you. So, what constitutes "commitment?" Lots of things... show up to meetings comment on or write bugs, blogs, wiki, articles, recipes, HOWTOs, newsgroup, mailing list, IRC submits patches or test cases help triage bugs mentor students (GSoc or other) runs contests, does viral marketing, etc. Now, of course, these items are not all mesaurable, but if people know you're involved, and you'd like to be a committer, you'll likely be voted in. (Many people trying out Athena may have noticed I've offered them committer rights in exchange for code or doc contributions. So far, no takers, but the offer stands.) Frankly, I'd rather have more people as committers who do little to the code base but who have the power to do so when needed. For example, (if the data is accurate) Kim's only committed 48 LOC in the past 9 months, compared to my 80,000 LOC (seriously, that can't be right) - but what she, Andrew and Andrew have done has been invaluable. And, often much more valuable, they've all helped out with with advice in bugs. Thanks! Good planning trumps code any day. Continued in part 2 Tracking Build Status With Hudson Data APIs 2009-07-16T00:46:00.008-04:00 A number of people have been twittering recently about Hudson Helper, and the fact that it can't (yet) support http access to Hudson servers. (There's just no pleasing some people, eh David?) UPDATE: David reports that Hudson Helper has worked with both http and https since day one. He invites direct feedback if you're having problems. To help fill this gap, I'd like to detail some of the handy API features of Hudson I've discovered since I first started using it back in October, which cane be fetched via http (or https) in a browser or via a script. Datum Example Latest Successful build number buildNumber Latest Successful zip (published artifact) GEF-Update-*.zip All checked out Project Set Files (Hudson workspace) *.psf XML Digest of Latest Stable Build lastStableBuild/api/xml SVN revision used for Latest Stable Build //changeSet/revision/revision For more on the APIs available to the Latest Successful, Stable, Failed, or in fact simply the Latest Build, see: /lastSuccessfulBuild/api/ /lastStableBuild/api/ /lastFailedBuild/api/ /lastBuild/api/ Of course, should you want details on a specific build rather than the latest, you can replace the "last*Build" token with an actual build number. Finally, because no post about APIs should be complete with out some script showing how to exploit that interface, here's a quick example of how to fetch the latest successful, and as yet unreleased Drools 5.1 runtime library zip for use in our JBoss Tools 3.1.0.M2 builds. In this example, we fetch the build number for the last successful build and compare it to a cached version. We also fetch and cache the latest SVN revision number (in a build.properties file) so that we can later fetch Drools sources from the same point in time as the precompiled Drools binaries in the zip. This guarantees we're building from trunk, but only a good build in trunk, skipping over any failed builds or intermediate states (partial commits). #!/bin/bash droolsSNAPSHOTnum=drools-SNAPSHOT-num.txt droolsSNAPSHOTrev=drools-SNAPSHOT-rev.txt droolsSNAPSHOTzip=drools-SNAPSHOT-bin.zip droolsSNAPSHOTurl=http://jboss-hudson-server/hudson/job/drools/lastSuccessfulBuild/artifact/trunk/target/drools-5.1.0.SNAPSHOT-bin.zip buildNumOld=0; if [[ -f $droolsSNAPSHOTnum ]]; then buildNumOld=$(cat $droolsSNAPSHOTnum); fi buildNumNew=$(wget -q --no-clobber -O - http://jboss-hudson-server/hudson/job/drools/lastSuccessfulBuild/buildNumber) buildRevOld=0; if [[ -f $droolsSNAPSHOTrev ]]; then buildRevOld=$(cat $droolsSNAPSHOTrev); fi buildRevNew=$(wget -q --no-clobber -O - http://jboss-hudson-server/hudson/job/drools/lastSuccessfulBuild/api/xml?xpath=//changeSet/revision/revision) if [[ $buildNumNew -gt $buildNumOld ]]; then # get: 27013; must change to 27013 echo $buildRevNew > $droolsSNAPSHOTrev; sed -i "s#\|##g" $droolsSNAPSHOTrev buildRevNew="$(cat $droolsSNAPSHOTrev)"; #echo "."$buildRevNew"." # replace "defaultTag=trunk:\d+" with defaultTag=trunk:${buildRevNew} in build.properties # defaultSvnUrl=http://anonsvn.jboss.org/repos/labs/labs/jbossrules # defaultTag=trunk:27013 sed -i "s#defaultTag=trunk:\d\+#defaultTag=trunk:$buildRevNew#g" build.properties; # grep "defaultTag=trunk:" build.properties echo $buildNumNew > $droolsSNAPSHOTnum; echo "Download $droolsSNAPSHOTurl ..." wget -q --no-clobber -O $droolsSNAPSHOTzip $droolsSNAPSHOTurl # ... fi Oh, and BTW, if you're ever looking for the latest hudson.war executable, it's always here. Workin' For The Wiikend 2009-07-13T00:48:00.007-04:00 After acquiring my first DriveKey-powered "try before you buy" Wii game via torrent (and having a little fun fighting the Joker's minions off while occasionally blowing Robin into his component bricks with a well-placed BatBomb), I decided tonight to do a little more hacking. Thanks, CanadaMods.ca! So, with the wife out watching some chick-flick w/ a friend, I got to spend a few hours playing with the HomeBrew Channel on my Wii. Very cool stuff available, from game emulators & ports, to new games, media players, and utilities. Complete list here. To set up the HomeBrew Channel, follow these steps, including installation of the DVDx application so your Wii can play video DVDs. Then, install the HomeBrew Browser, and grab some more software. After numerous tests, crashes, and reboots, I found that the best three options for playing video are these, all available through the HomeBrew Browser or via manual download from wiibrew.org. Here's what I tested: App 2G SD card w/ .mp3 Bus-powered 2.5" 500G USB drive w/ .avi DVD-R w/ .avi DVD-R Video DVD (burned w/ growisofs from dl'd .avi torrent) [1] Video DVD (original, possibly DVD-DL? or DVD+R) GeeXboX (embedded linux) Y Y N N N MPlayer CE Y Y Y Y N MPlayer TT Y Y N So, while I have scripted the process for easily converting .avi to DVD, I now no longer need to do so -- I can just plug my USB drive directly into the Wii and watch it on the big screen w/o having to waste hours in format conversion. Wii! E-Fail 2009-07-09T11:59:00.005-04:00 It's taken a while, but I've managed to get some metrics for how much mail I actually process. Here's my inbox 3 weeks ago before I went on vacation for a week, then went without VPN access for a few days. The xkcd strip is particularly appropos. Here's that same inbox today, sporting a newer version of Thunderbird. Note the pileup of over 1,000 emails in three weeks, in just ONE of the mailing list filter/folders I monitor. So, other than filtering by sender & subject, automatically marking my own mailing list replies read, colourizing emails to make the more important ones stand out, and using "Show Unread Threads" view filtering ... what else can one do to manage the deluge? Does anyone have any good, realistic strategies for dealing with 1000s of emails a month? Simplified Win XP Pro EULA 2009-07-09T11:07:00.003-04:00 http://www.microsoft.com/windowsxp/eula/pro.mspx -- Reminds me of the WTFPL license... Mac OS X - VPN vs. LAN: DNS Royal Rumble 2009-07-08T16:58:00.008-04:00 I've been "sharing the Mac experience" for the past day trying to get access to my local LAN and VPN concurrently. So far, it's only one or the other, but never both at the same time. I've tried the Cisco client, the Shimo client, vpnc (compiled from scratch with and without openssl support), vpnc 0.5.3 from DarwinPorts, and even this custom bit of script I wrote based on some tips about using scutil. #!/bin/bash # goal here is to collect the DNS entries from the active services and merge them into the Global list tmpfile=$(mktemp) # get IPs from services using scutil function getIPs () { return_IPs="" keys=$(echo "list State:/Network/"$1 | scutil | awk '{print $4}') for f in $keys; do echo "> show $f" printf "get "$f"\nshow "$f | scutil | grep "\." echo "show $f" | scutil 2>&1 | grep "\." 2>&1 | \ awk '{print $3}' 2>&1 >> $tmpfile done #cat $tmpfile IPlist=$(cat $tmpfile | sort -r 2>&1 | uniq 2>&1) for i in $IPlist; do return_IPs=$return_IPs" "$i done #echo $return_IPs rm -fr $tmpfile } function setIPs () { label="$1" IPs="$2"; # echo $IPs printf "get State:/Network/$label\nd.add ServerAddresses *$IPs\nset State:/Network/$label" | scutil echo "> show State:/Network/"$label printf "get State:/Network/"$label"\nshow State:/Network/"$label | \ scutil | grep "\." } echo "--- BEFORE ---" getIPs "Service/.+/DNS" IPs=$return_IPs echo ""; echo "--- AFTER ---" setIPs "Service/com.cisco.VPN/DNS" "$IPs" setIPs "Global/DNS" "$IPs" mv /etc/resolv.conf /etc/resolv.conf.bak for i in $IPs; do echo "nameserver $i" >> /etc/resolv.conf; done # ./dnsfix.sh --- BEFORE --- > show State:/Network/Service/F1C45B82-45A1-4F44-89AC-82102F187F0B/DNS 0 : 192.168.x.y > show State:/Network/Service/com.cisco.VPN/DNS 0 : a.b.c.d 1 : e.f.g.h --- AFTER --- > show State:/Network/Service/com.cisco.VPN/DNS 0 : 192.168.x.y 1 : a.b.c.d 2 : e.f.g.h > show State:/Network/Global/DNS 0 : 192.168.x.y 1 : a.b.c.d 2 : e.f.g.h Obviously, since it's a Mac, there's got to be a dead-simple way for this to work. Anyone know how? Learning to Love the Mac, Part 2: Mouse Tips & Desktop Management 2009-07-07T22:05:00.007-04:00 I have an 8-button Logitech MX500 optical mouse, and this week is the first time I've ever successfully mapped functionality to all the buttons. Windows did a reasonable job with a few of the buttons; Linux doesn't support anything beyond the first three; Mac OS X Server just gets it done. Out of the box, my third button (scroll wheel) is mapped to the seemingly pointless Dashboard, which is a huge pain when you're used to middle-clicking to open a link in a new tab or to copy/paste text in a console. To get that functionality back, go to Applications > System Preferences > Exposé & Spaces then remove Mouse Button 3 from the Dashboard's "Hide and Show" feature. Next, I set Mouse buttons 5, 6, and 8 to All windows, Application windows, and Show Desktop. But even cooler than these is Spaces, though as yet I can't find a way to replicate Gnome or XFCE's ability to move windows from from Space to Space which lets you drag open app windows from Space to Space which in the Spaces view (F8). Still, having up to 16 virtual desktops is very handy, particularly when you need to virtualize Windows and Linux. If you want to be able to have console windows on all Spaces rather than having them all collected on a single Space, uncheck the "When switching to an application, switch to a space with open windows for the application". My love-hate with SVN, Part 5: Fedora 11 + Eclipse 3.5 + Subversion 1.6 2009-07-07T21:05:00.000-04:00 Finally figured out how to make Eclipse 3.5 play nicely on Fedora 11 w/ Subversion, and I owe this bit of knowledge to our new MacPro. *sigh* I also owe a great deal of gratitude to Cloudsmith for providing their Cloudsmith Galileo+ repository, which includes these features: I still wish the version numbers would better align, in that I have to install the SVN Team Provider v0.7.8 with the SVN Connector v2.2.0 and the SVNKit 1.3.0 implementation v2.2.0 to make all this work with Subversion 1.6. Oof. Learning to Love the Mac: 13 Tips 2009-07-06T16:49:00.011-04:00 A month ago a very large package arrived in the mail: my first MacPro server. I at once fell in love with the case design - clean, simple, and dead-easy to take apart in order to add more drives and RAM. However, that's where the love boat ran aground. To say it's been a gradual learning curve would be an understatement. Here are a few things I've learned over the past month of dealing with Mac hardware and OS, as well as retraining my fingers to use Mac keyboard bindings (META = Apple Key or Windows Key, depending on your keyboard). Use META-TAB instead of ALT-TAB to cycle applications Use META-LEFT/RIGHT instead of HOME/END to jump to start/end of a line Use ALT-LEFT/RIGHT instead of CTRL-LEFT/RIGHT to jump to prev/next word on a line META-A, META-X, META-C, META-V replace CTRL-A, CTRL-X, CTRL-C, CTRL-V for select all, cut, copy, & paste. META-L, META-T, META-N replace CTRL-L, CTRL-T, CTRL-N (jump to location bar, new tab, new window). But CTRL-TAB still switches tabs. However, if you have multiple Firefox windows open, there is no way to toggle between them with the keyboard. Same problem with multiple Terminal windows. META-TAB only switches between groups of applications, but not windows within an application. Sometimes ESC works to dispose a dialog; sometimes only clicking the red X works. Q replaces qemu, but doesn't seem to work very well for my existing vmware or Virtual Box images Virtual Box rocks on Windows, Linux and Mac XCode provides gcc, make, etc. Fink and DarwinPorts replace Debian/Ubuntu's apt-get and Gentoo's emerge, respectively. Once XCode and DarwinPorts are installed, you can port install vpnc (to fetch deps and compile on the fly) or apt-get install curl (to fetch deps and install). rEFIt replaces grub, and more or less works as I'd expect. /efi/refit/refit.conf approximately replaces /boot/grub/menu.lst at least as far as picking what partition to default-boot and how long to wait Java is in /System/Library/Frameworks/JavaVM.framework/Home instead of /opt/ or /usr/lib/jvm/java Subversion was easier to set up on Mac (using Fink) than on Fedora 10 (using yum), especially since there's now the Galileo+ Update Site from Cloudsmith so you don't have to download from multiple update sites to get it installed. However, the version of Subversion available via Fink doesn't work with projects checked out using Eclipse - seems that the commandline client (Subversion 1.4.4) and Subversive with SVNKit (SVN 1.6.1 w/ SVNKit 1.3.0.beta.r5741) are not compatible: svn: This client is too old to work with working copy '.'; please get a newer Subversion client. Using DarwinPorts to update the subversive client to 1.6.3 fixed this issue, but installed it into a different path (/opt/local/bin instead of /sw/bin or /usr/bin). Eclipse looks better on Mac than on Linux; however, I recently stumbled across a great tip for making Eclipse waste less screen space under gtk on Linux. Highly recommended bit of gtk hackery - one file makes a world of difference! Do you have any other tips for Linux or Windows people, surviving the transition to Mac OSX? Is there any way to tell OSX to use Windows or Linux keyboard defaults so I don't have to retrain myself? My love-hate with SVN, Part 4: Corrupt Metadata & Going Over Quota 2009-06-25T18:34:00.010-04:00 Ever had one of those days where nothing seems to work? Most of June's been that way for me... This week I decided to trust my OS and let Fedora update me automatically to the latest release, Fedora 11. I've never tried a distribution upgrade; in the past I've only ever done a clean install (be it Windows, Ubuntu, MEPIS, AntiX, or Fedora). But I figured if @dougschaefer could do it, so could I. It was fairly smooth sailing, though the handy gui tool preupgrade only downloaded packages but didn't do the upgrade, so on reboot (still in F10) I had to run preupgrade-cli "Fedora 11 (Leonidas)". I suspect I must have fatfingered my hard drive password when I rebooted he first time because it worked like a charm the second time. Overall, way more successful than attempts so far to make a Mac Pro get Fedora'd, thatsfersure (grub, video, and network card issues, to name but a few). Anyway, now I have updated versions of subversion and python, and as a result, my Subversive projects in Eclipse don't work. After much cursing and experimenting (and updating my CollabNet Subversion version to 1.6.3, the solution seems to be simply this: Check out the projects anew within Eclipse, and if necessary, diff local changes from old project to new project. But, if the project is too big (jbosstools trunk folder is over 1.1G) you may get a heap error. You can check the whole project out via commandline, but Eclipse (or Subversive? or Mylyn?) uses too much memory and the whole thing dies, despite my running Eclipse w/ a half-gig of heap: /home/nboldt/eclipse/eclipse/eclipse -clean \ -showLocation -data /home/nboldt/eclipse/workspace-jboss \ -vmargs -Djava.library.path=/opt/CollabNet_Subversion/lib \ -Xmx512M -XX:PermSize=512M In this case, the solution is to check out the project without recursing into folders. Commandline on in the Console view, that looks like this: svn checkout "https://svn.jboss.org/repos/jbosstools/trunk@HEAD" -r \ HEAD --depth empty "/home/nboldt/eclipse/workspace-jboss/jbosstools-trunk" You can then copy stuff you already checked out into the new target project, then refresh the project in Eclipse. Of course in my case Eclipse then thought all the files were new, so I had to Override and Update from the repo. Another 24374 files or 1.1G to download. No wonder I went over my quota this month! (Really, it was due to several different .iso torrent downloads for Fedora and CentOS, along with the movie Dead Alive, just in time for BLITEOTW day!) So, unfortunately, I haven't been able to enjoy any of the 33 projects in this years' Eclipse Galileo release, unlike others on the Planet and the BirdsNest have. Hopefully next month will see calmer seas. On the plus side now that I'm running Fedora 11, I can use Delta RPMs thanks to the yum-presto plugin... so next month's bill should be much, much smaller. I know it's a beta, but c'mon... 2009-06-24T14:37:00.002-04:00 In addition to crashing a few times a day, Firefox 3.5b4 has this cool overlapping button feature on its "fail whale" page: HOWTO: Enable Firefox 3.0 extensions in Firefox 3.5 for great justice 2009-06-24T01:14:00.005-04:00 Just updated to Fedora 11, and with it Firefox 3.5b4. Sadly, that meant most of my extensions (including mouse gestures!) no longer worked... until I found this: In A.D. 2009, Firefox 3.5 was beginning. Captain: What happen ? Mechanic: Fedora 11 repo set up us the beta. Operator: We get signal. Captain: What ! Operator: Main browser turn on. Captain: It's you !! FF35: How are you gentlemen !! FF35: All your extensions are belong to 3.0. FF35: You are on the way a vanilla Firefox. Captain: What you say !! FF35: You have no chance to survive wait for GA. FF35: Ha ha ha ha.... Operator: Captain !! * Captain: To put back every 'extension'!! Captain: You know what you doing. Captain: Install this. Captain: For great justice. Captain's Log - Additional: Here's the same extension for Thunderbird 3.0b2, though I had to disable the quicksearch toolbar box as it wouldn't close properly in TB3; also found some handy toolbar buttons here for filtering a mailbox for all/unread. Workspace #fail 2009-06-23T17:59:00.006-04:00 Another obscure and unhelpful error message that now pops up about once every five minutes while I'm working in Eclipse. Upgraded to Eclipse 3.5 a day before the official release (thanks to my Friends of Eclipse membership), but to no avail. Evidently my workspace is pooched somehow. Why can't more error messages tell you *HOW TO SOLVE THE PROBLEM*, rather than just reporting that something went wrong? Surely as software devs we should be able to do better... if for no better reason that to avoid having to listen to end users like myself complain? :P Eclipse 'Vote For Pedro' Plugin? 2009-06-15T12:23:00.003-04:00 Got this today. No idea why / how. Anyone ever seen this? WTP Cruise Control #fail #eclipse35 2009-06-15T11:38:00.002-04:00 Nine days without a green build this close to GA? Seriouslywtfbbq, people! P2: The Publisher 2009-06-03T16:11:00.003-04:00 The publishment has begun [1]. Read all about this enhanced replacement for the Metadata Generator. We404 2009-05-28T12:50:00.003-04:00 Eclipse Community Survey: 4 More Insights 2009-05-28T11:45:00.013-04:00 Ian blogged 6 insights from this year's Eclipse Community Survey; here are a few more to get us to a full Top Ten list. What is your primary operating system? Linux is certainly a strong player in both development (26%) and deployment (40%), beating Mac (7% and 3%) but losing to Windows (64% and 38%). More interesting to me is the fragmentation of Linux, showing that Ubuntu beats RHEL/Fedora by 10% in the desktop space (development), but loses in the server space (deployment). Where do you typically go to find Eclipse-related information? About 2/3rds said Google and/or the Eclipse home page, which suggests that the homepage has certainly improved - but a lot of people would rather just search. However, the survey didn't mention our finely crafted wiki.eclipse.org, or help.eclipse.org. Survey #FAIL. Are you or the organization you work for a member of the Eclipse Foundation? Five out of six respondents (83%) said No. So either we've done a terrible job of converting users into members, or people would rather give back in the form of testing, documentation, filing bugs, and writing articles. I suspect it's a little of both, but mostly the former. Kudos to the contributors, and shame on the corporate drones for not convincing their queen to send a little honey back to Eclipse. In the last year, how have you participated in the Eclipse community? While nearly a quarter of respondents (24%) said "I entered at least one bug into Bugzilla", more than 2/3rds said they "used Eclipse but didn't actively participate in the community." To me that's a clear sign we have more users than contributors. Is that because most Eclipse users are Windows folks who don't grok that Open Source works best when everyone sees themselves as part of the process, rather than just a consumer? I've been reading More Joel On Software recently, thanks to winning a prize for bringing a bag purchased in Alaska to EclipseCon this past March. One article stands out from there in this context, Building Communities with Software, from March 2003. Here's an excerpt: The social scientist Ray Oldenburg talks about how humans need a third place, besides work and home, to meet with friends, have a beer, discuss the events of the day, and enjoy some human interaction. Coffee shops, bars, hair salons, beer gardens, pool halls, clubs, and other hangouts are as vital as factories, schools and apartments ["The Great Good Place", 1989]. But capitalist society has been eroding those third places, and society is left impoverished. ... So it's no surprise that so many programmers, desperate for a little human contact, flock to online communities - chat rooms, discussion forums, open source projects, and Ultima Online. In creating community software, we are, to some extent, trying to create a third place. If you feel your third place is lacking, please consider contributing more to Eclipse, to Fedora or CentOS, to JBoss Tools, or whatever tickles your fancy. Just give something back. Your community will thank you, since, after all, "A rising tide lifts all boats." UPDATE, 2009/05/30: Mike's right, calling our users "freeloaders" isn't fair. I just wish there was a more obvious way to convert users into contributors. Dash Athena: Eclipse Common Build System / Running Tests On Your System 2009-05-26T12:23:00.003-04:00 Bjorn recently kvetched that Eclipse projects met two or three of those goals, but fell down on the "common build system" and "tests run on your system" [1]. While it's true I've seen a number of projects who don't have, don't run, or don't publish their tests, I'm a little disappointed to see Bjorn's no longer committed to the common build solution we've been working on since September 2006 (in earnest since June 2008). We do have a project to solve both those concerns, but like all things at Eclipse, it's powered by YOU. You want it to happen, you have to help. I'm looking for a few good contributors and committers for the Dash Athena project to supplement the great people we already have. Or, if you don't have time to contribute code, you can help by using the system, testing it, opening bugs, enhancing documentation, and blogging about it. So, what is Dash Athena? Well, it's a common build system using Hudson and PDE which can also be run commandline on Linux, Windows or Mac, or in Eclipse. It can produce zips of plugins, features, examples, tests, then run those tests. It can also produce update sites with p2 metadata, which can then be published to eclipse.org (or sourceforge.net, for that matter) so everyone can get your bits via Update. Tests will currently only run on Linux - if you'd like to help us port to Mac OS X and Windows, please step up. The system works with CVS, SVN, and probably Git/Bzr/Hg too, since it supports building from locally checked-out sources and will copy your features/plugins so they're in the format that PDE requires. It supports source input via map files (soon Project Set Files (*.psf), too!) and binary inputs via zips and p2 repos / update sites. If you aren't sure how to get started w/ an Athena build, please don't hesitate to ask. If you feel the docs are insufficient, incomplete, or inaccurate, let me know - or better - fix them! Want your own Hudson job to run your build? Just open a bug and we'll set you up. Oh, and incidentally, the irony is not lost on me that I'm using American iconography above even though 5 of the 6 committers on the project are Canucks. :) They're Coming To Make Me Write ASP! 2009-05-25T15:49:00.002-04:00 watch video Remember when you ran away Big Blue got on their knees And begged you not to leave PDE'd go berserk You left 'em anyhow And then the days got worse and worse And now I see you've gone (Completely out of your mind) And they're coming to take you away ha-haaa They're coming to take you away ho ho hee hee ha haaa To the Redmond farm Where life is beautiful all the time And you'll be happy to see those nice young men In their - see? Sharp coats And they're coming to take you away ha haaa We thought it was a joke And so we laughed We laughed when you had said That you could leave the FLOSS and work for Bill Right? You know we laughed You heard us laugh. We laughed We laughed and laughed but still you left But now we know you're utterly mad And they're coming to take you away ha haaa They're coming to take you away ho ho hee hee ha haaa To the happy home with bugs and Vista and viruses Security "fixes" which patch and patch and open new hacks and holes And they're coming to take you away ha haaa We've read your blogs And used your code And this is how you pay us back For all our kind unselfish, loving deeds? Ha! Well you just wait They'll find you yet and when they do They'll make you write with ASP.net You well-dressed geek And they're coming to take you away ha haaa They're coming to take you away ha haaa ho ho hee hee To Camp Microserf where life is beautiful all the time And you'll be happy to drink that nice Kool-Aid In their clean white cups And they're coming to take you away Neuroticfish - They're Coming To Take Me Away Use Your Metadata, Vol. 2 [Update] 2009-05-22T13:12:00.006-04:00 Wednesday I went off on a bit of a G'n'R-fueled rant about metadata, documentation, and the shotgun blues. Today, I'd like to focus on something more positive. As Pascal blogged the other day, the new p2 is almost done and is ready for tire-kicking. Some new features I personally like include: a new p2.director app / task, which includes support for installing multiple IUs (feature.groups) in the same step and finally has commandline help a new p2.repo2runnable ant task, used to convert an update site zip to the old-school unpacked "runnable" features/plugins format so that one day we will be able to throw away all those extra zips. UPDATE, 2009-06-02: repo2runnable now works as a commandline application too, thanks to Andrew's fix. Wiki updated. Composite Repo, Mirroring and Slicing Tasks - haven't tried these yet, but they look like they'll be very handy for one day replacing the hack that is buildUpdateSite.sh for our Modeling Project composite repos with something more robust and easily maintainable. I'm also impressed that there is new, current documentation regarding the above tasks, as well on the new Publisher which replaces the Metadata Generator. Will this release be p2's salvation? click to zoom Well, I'm split on the new default behaviour in the update UI, such that when you add a new update site p2 won't by default search ALL your other listed sites. This is a great performance gain if you're installing a new self-contained feature, but a pain if you're installing something like VE which depends on EMF and GEF, and you don't already have those deps installed. Simple workaround is to just pick the "all sites" entry in the dropdown. I'm also waiting to see if there will be something better done about recovery from slow/incomplete mirrors. But other than these minor concerns, I'd say YES. With lots more commandline and ant toys available, p2 is certainly maturing. And with more people adopting its use and spinning p2 repo zips, more testing is being done, and more use cases are being covered. So... get in the ring, and go a few rounds with p2. It's worth the battle. :) Use Your Metadata, Vol. 1 2009-05-20T15:07:00.006-04:00 It's been a bad week for update sites and Galileo contribution from Modeling... and I confess I'm partly to blame. That and the fact that despite documenting processes, workarounds, tips, tricks, and advice... no one Reads The Fine Mediawiki (Category:Releng or Modeling Project Releng). Highlights: The mysterious appearance of a new version of org.eclipse.osgi_*.jar in releng.basebuilder's R35_M5 tag, which caused an ant <copy/> used to rename a file to fail because copy can't merge two jars into one file. Still no idea why an old basebuilder tag would magically grow a new jars, but I've worked around the now-faulty assumption w/ smarter Ant code. A change to the way our sites are created, in an attempt to workaround what I believe (but can't yet prove) is a flaw in the way content.xml is produced - namely, if the xml file is > 21M, it gets truncated or corrupted. We used to cache 2 or 3 releases of a given project (eg., M5 and M6) on the same site, in order to give people a way to "back up" to the previous release; now, you only get the latest (bug 271486). I confess I screwed up here and instead of replacing a folder with new contents, I was copying INTO that folder - `cp -r one two` resulted in folder one/two/ instead of two/. I fixed that by changing to a move instead of a copy, but a downstream process failed because of the assumption that both one/ and two/ would exist. The lessons here are: a) shotgun debugging sucks, and b) don't change the way stuff is created after M7. People publishing two updates to a site at the same time, resulting in the appearance of two </site> tags in a site.xml file, causing p2 metadata generation to be incomplete or fail entirely; unfortunately, no error is logged when this happens so it's rather difficult to decipher the tea leaves. This may be the real source of the metadata corruption, if not the "file is too big" issue above. Observations about obsolete jars corrupting metadata, but no one taking it upon themselves to clean up the site or do some troubleshooting People inconsistently naming their milestones (it's 2.0.0M7, not just M7!) and corrupting our Release Notes database. This one amazes me the most since it takes seconds to see what was done last time (check any of the following: RSS feeds, release notes, downloads pages, update sites) and follow suit. And, of course, the conventions are documented, along with the rationale (consistent patterns == simpler code). Or, to put it another way... Sick of this life Not that you'd care I'm not the only one with whom these feelings I share Nobody understands Quite why we're here We're searchin' for answers That never appear But maybe if I looked real hard I'd I'd see your tryin' too To understand this life, That we're all goin' through Sometimes I feel like I'm beatin' a dead horse And I don't know why you'd be bringin' me down I'd like to think that your love's Worth a tad more It may sound funny but you'd think by now I'd be smilin' I guess some things never change Never change So, please, can we stop opening bugs (277172, 277105, 277034, 276928, 276641) and just use the tools and docs already available?

Initial checkout from existing repo for a given branch	git clone http://github.com/sonatype/sonatype-tycho.git; cd sonatype-tycho; git checkout origin/tycho-0.10.x	svn checkout http://anonsvn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.0.Beta1/
Update locally checked out files from central repo	git pull	svn update
List locally changes files/folders	git status	svn stat
Diff locally changed file	git diff somefile.txt	svn diff somefile.txt
Revert locally changed file*	git checkout somefile.txt	svn revert somefile.txt
Revert ALL local changes (except untracked files)*	git reset --hard HEAD	svn revert . -R
Add new file	git add file.txt	svn add file.txt
Add new folder recursively	git add folder	svn add folder
Delete file	git rm file.txt	svn rm file.txt
Delete folder	git rm -r folder (non-recursive by default; use -r to recurse)	svn rm folder (recursive by default; use -N to not recurse)
Commit changed file to central repo	git commit -m "message" file.txt; git push	svn ci -m "message" file.txt
Ignore files/folders (in the current folder)	echo "target *.class bin" > .gitignore; \ git ci -m "gitignore" .gitignore	svn propset svn:ignore "target *.class bin" .; \ svn ci -N -m "svn:ignore" .

Datum	Example
Latest Successful build number	buildNumber
Latest Successful zip (published artifact)	GEF-Update-*.zip
All checked out Project Set Files (Hudson workspace)	*.psf
XML Digest of Latest Stable Build	lastStableBuild/api/xml
SVN revision used for Latest Stable Build	//changeSet/revision/revision