Using Hudson for parallel Athena builds

This week I set up a multi-configuration Hudson job to try doing EMF's Query, Validation, and Transaction projects as a single job (instead of the traditional three linked jobs), then Bernd took over to do most of the work required to migrate to the new, simpler build system... and unearthed a few sticky bugs along the way. For example, Athena now supports running from jarred test plugins, as well as test plugin folders. (Yes, I'm amazed no one ever tested that use case before now either. Anyway, it works now.)

Doing these builds in parallel (rather than serially) may be a bad idea, but for now, it lets the Q,V,T developers build all three linked components via a single build button, and if the sources for any of the three changes, all three will be rebuilt and tests will be run.

Since Transaction depends on Query and Validation, this is certainly useful; however, Query and Validation do not depend on each other, so there may be some extraneous build churn with this set up.

Still, it's nice to see a the status of all three builds in one place:

What's next? Well, hooking up FindBugs, eliminating all the old SDK/runtime zips (because we have the new, shiny, p2 update site zip instead, and you can convert from repo to "runnable" SDK easily using this script), and figuring out how to make the builds use each others' newer binary output instead of building from a known entity (Query 1.4.0.N.ten.minutes.ago vs. 1.3.0.R.last.month). Luckily, Athena supports building against p2 repos and Hudson has APIs for fetching the latest good output, so this should be as simple as configuring the Transaction build to fetch from the latest successful Query zip using properties like these:

dependencyURLs=https://build.eclipse.org/hudson/.../path/to/Update.zip

Of course if you want to build against the latest PUBLISHED update, you could do this:

repositoryURLs=http://download.eclipse.org/modeling/emf/query/updates/nightly/
featureIDsToInstall=org.eclipse.emf.query.sdk

Vacation Planning / Process Modeling

This summer I'm attending the wedding of two friends in Victoria, BC. So, since we're travelling across the country anyway, we're going to take some time to explore Vancouver Island a little.

And because I'm a total Eclipse fanboy (despite the fact that certain people tell me Eclipse is dead), I thought it was time to play with the BPMN Modeler again. Version 1.0.2 (for Eclipse 3.4 / Ganymede) just came out today (update site).

So here's me planning the trip:

Aside: Wondering what's in Telegraph Cove? Check this out.

It may not have the same graphical wow factor as, say, Google Maps' "My Maps" feature...

... but it does clearly show two things:

a) the BPMN Modeler is stable and works great for modeling my workflows, regardless of whether I intend to do anything programmatic w/ the resulting diagram, and

b) air fare and car rentals cost too much. :)

I also tried out the in-progress BPMN Modeler 1.1M6, and while it adds some cool features to help align items in your swimlanes, it's also not quite ready for prime-time use. Looking forward to 1.1M7 next month.

Who says modeling can't be fun?

Modeling: Home Star Loader

Last week I reported that there are now 70 committers / contributors in Modeling.

Today, Denis announced on the Phoenix mailing list that EMF, EMFT, MDT and GMF are in the top 30 requested home pages for Feb 2009.

The top three popular project pages were, in order: PDT, EMFT, and WTP.

However, if you combine the numbers for the four Modeling subprojects, you get 162K hits for the month, placing Modeling in the #4 slot behind only the Eclipse.org home page, Downloads page, and Categories page, and making us the #1 (aggregate) project page.

Modeling: From Zero to Seventy

Remember when no one had heard of EMF? When it was just Ed Merks, Frank Budinsky, and Dave Steinberg? Me neither.

These days modeling is everywhere, and with good reason. There are now a total of 70 people contributing to and committing for the various Modeling projects, the most recent additions being from MDT (Papyrus, BPMN2), EMF (CDO), and TMF (Xtext).

So, is it any surprise that there are 29 Modeling Track talks this year at EclipseCon, from the introductory Mega Modeling Mania BoF to the double-black-diamond GMF and QVT: roads (finally) crossed?

And that's not even including a couple of e4 talks about The Modeled UI in Eclipse e4 and The e4 Programming Model Across Web and Desktop or one about Designing an Android Domain Specific Modeling Language using EMF, GEF and GMF!

I guess some people are just unbearably stupid, eh?

Download Stats & Missing Data

According to Google Analytics' new "snot graph" tool, the top three downloads in Modeling these days are, in order: EMF, UML2, UML2 Tools.

OK, no big surprise there. But what's really surprising is that people are still downloading zips. Isn't it easier to get your bits via Update? Isn't that why we provide you, the community, with over 20 Modeling update sites (three per project), so you can pick and choose what features to install, rather than downloading & unpacking zips? (Not to mention the countless other sites provided by DSDP, WTP, BIRT, DLTK, DTP, Mylyn, PDT, Equinox, Eclipse, GEF...)

Hell, with the advent of Eclipse 3.4, you can now even update the platform itself, so the Eclipse experience need only be one zip + a ton of multi-threaded updates. Say what you will about p2 (and I've said my share) but isn't it faster and easier than having to download zips?

I can understand being stuck on a platform version from a while ago, but with Eclipse 3.5 only 4 months away (yikes!), and e4 due next year, should anyone still be using Eclipse 3.0 or 3.1 and EMF 2.0 or 2.1? You're missing out on over three years of new development!

What's even more bizarre to me is that when maintenance releases are available, people still download the point-zero releases. Are we wasting our time doing maintenance? Should we be focusing on the next release instead? (In related news, Ganymede SR2 is just around the corner. Is everyone ready for what will be the final chapter for the Eclipse 3.4 stack?)

Of course the one glaring hole in all this data are the numbers around how many update jars are downloaded. If you'd like to see this resolved, have a look at bugs 239668 and 187968 and vote accordingly. It's really depressing to think that since September 15 we've only had 150,000 downloads (100,000 unique) when in 2006 we broke 1,000,000 downloads for 2006 on December 1 of that year, averaging just under 100,000 downloads a month.

Either the popularity of EMF - and of Modeling - has waned by two thirds, or we need a better way to capture stats.

Blackberry Linux Mail, Calendar, Tasks & Contact Sync, Part 2

Things have improved since my last post, despite the fact that I managed to kill Thunderbird along the way and lost a good chunk of today figuring out to restore it. Luckily I can read French, or I may never have found the solution to the elusive -203 problem.

Aside:

To recover from a corrupt Thunderbird profile, create a new profile using thunderbird -ProfileManager, then transfer your data to the new profile. Easy-peasy.

Next, reinstall all your extensions. I use these 15 extensions.

I've also managed to get barry to work for backing up my Blackberry on Fedora. Turns out the problem is that it simply needs root privileges.

Here's how the data flows, drawn with JBoss Tools' jBPM Graphical Process Designer:

click to enlarge

I don't yet have a solution for Zimbra contacts & calendar being accessible on the Blackberry, short of manually duplicating entries from one repo to the other. Still, 5 out of 7 data sources synched is pretty decent.

Clarifying Our All-In-Ones

In order to make it clearer what's in the various flavours of all-in-one zips provided in Modeling, GEF, and PDT, I've concocted some new icons to better differentiate them.

This is actually moving in the opposite direction from what I'd proposed in bug 240223 regarding Simplified EMF Downloads as we're now providing more, not less... but IMHO that's okay as long as the community is well served. If you have an opinion on update site vs. update zips vs. SDK zips vs. EPP bundles, please feel free to weigh in on bug 240223.

And if you hate the new icons, please let me know. Better still, design me a replacement. Thanks to iconeden.com for the source pixels.

Request For Comment: The End Of SDO?

Four months ago, I started a discussion about simplifying the EMF download experience. Consensus appears to be that we should keep the EMF-SDO-XSD "All-in-one" SDK zip, perhaps more.

Now, a new proposal is in the works: removing SDO from the EMF build, and from this year's coordinated release, Galileo. The source would still be in CVS, but we'd stop providing SDO binaries. This would also mean the much-loved^* EMF-SDO-XSD "All-in-one" SDK zip would be renamed to EMF-XSD "All-in-one" SDK, but would otherwise continue as is.

For more on this topic see:

EMF 2.5 - Simplified Downloads (bug 240223)
Remove SDO from Galileo (bug 251402)
Consider improvements to the termination process (bug 249408)

To voice your opinion, please post your feedback in any or all of the three bugs linked above.

Thanks!

50,000 Zips

Since turning on Google Analytics for tracking Modeling project zips on September 15, we've gotten over 50,000 pageviews (33,000 unique) of the download tracker. While this doesn't necessarily mean every one of those hits resulted in a download, it certainly shows a lot of interest in downloading EMF, UML2, UML2 Tools, GMF, Query, OCL, and XSD, to name but a few of the more than two dozen Modeling Project components.

Unfortunately, this only scratches the surface -- most of our downloads come from EPP bundles and update sites. But thanks to bug 239668, we will one day be able to track downloads done via p2. If download stats interest you, please vote for bug 239668.

And Don't It Make Your Blue Eyes Red

Today I resigned from IBM to join Red Hat Canada as their new Productization Lead for JBoss Tools and JBoss Developer Studio, working with Max R Andersen, Rodney Russ, and Andrew Overholt. Yet more release engineering? Oh, hells yeah.

So, because it's been nine great years, and because I'm a mathie from way back, I'd like to present...

IBM By The Numbers

• Number of Jobs Within IBM: 3
• Number of Thinkpads Used: 4 (770, T23, T40, T60p)
• Versions of Desktop Windows Used: 3 (NT4, 2K, XP)
• Versions of Desktop Windows Anti-Virus & Firewall Software Used: 6
• Versions of Desktop Linux Used: 3 (MEPIS 3.x, MEPIS 6, AntiX 7)
• Versions of Desktop Linux Anti-Virus & Firewall Software Used: 0 (suck it, Windows!)
  
  
• Memorable Website Launches While at IBM Global Services: 7+
• Memorable Applications Built While at IBM.com: 7+
• Memorable Open Source Software Releases While at IBM Rational: 30 in 5 years (EMF 2.0-2.4 = 5, UML2 1.0-2.1 = 4, OCL & EMF-QTV 1.0-1.2 = 12, JET 0.7-0.9 = 3, EODM 0.7-0.9 = 3, GEF 3.3.2-3.4 = 2, GMF 2.1 = 1)
  
  
• Supported Modeling Builds in 1Q2004: 1 (EMF)
• Supported Modeling Builds in 4Q2008: 22 (EMF x 7, EMFT x 5, MDT x 3, M2T x 3, M2M x 2, GMF x 1, TMF x 1)
• Other Builds in 4Q2008: 3 (GEF, PDT)
  
  
• Number of Eclipse Project Committerships in 1Q2004: 3 (EMF, SDO, XSD)
• Number of Eclipse Project Committerships in 4Q2008: 25 (EMF x 8, EMFT x 6, MDT x 5, M2T, M2M, GMF, GEF, SOC, Dash)
  (Note: some projects define releng committership separately from project committership.)

And now, to get the other side of the brain fired up, I leave you with my version of...

Fall Out Boy - Thnks Fr Th Mmrs

Another build will bend and break
(But it's an easy fix, these things)
Automate, yeah let the crontab roll
In case ya don't wanna show
(Let the crontab roll, let the crontab roll)

And I want these words to make things right
Avoid any wrongs, and keep Modeling alive
Empower EMF
So after a brief break,
Gonna put my fingers back to the keys

Nine years, one helluva time
Thanks for the memories
Thick and thin, they've been so great
Workin' for the Big Blue leader

Nine years, yeah and one more time
Thanks for the memories
Thanks for the memories
With two woof'rs who needs a tweeter? Ohhh!

Been looking forward to the future
But my eyesight is going bad
Must be the late nights
It's always cloudy except for (except for)
When you look into the past (look into the past)
Nine whole years. (Nine whole years, now!)

Three jobs within Big Blue
Thanks for the memories
Websites, apps and FOSS
Wonder what became of Peter?

Nine years, yeah one more time
Thanks for the memories
Thanks for the memories
O'en source always tastes a bit sweeter

(They say) I only think in the form of crunching numbers
Up late at night, exploring six-gig log files
Goin' out of my mind
Runnin' out of disk space
I'm a line away
From getting it all to work! Woo!

(In) AD 2101
Captain say 'what happen ?'
Operator: 'we get signal'
Mech: 'Someone set us up the bomb.'

Main screen turn on. It's you !!
How are you gentlemen !!
No chance, yeah make your time
You, are on the way to destruction. Ohhh!

Nine years, it's been a slice (had a blast, one more time)
Thanks for the memories
Addicted to Eclipse now (can't get enough Eclipse now)
But what still sucks? That'd be features!

Nine years, yeah and one more time (been great years, one more time)
Thanks for the memories (for the memories)
Thanks for the memories (for the memories)
Time on Sprockets to dance vith Dieter! Ohhh!

And the plan said...

With everyone's standardized project plans due next week, I've been tasked to draft a consistent workflow for the Modeling projects. And because we're such Model Citizens, I'd like to share it with others, in case this approach works for your project too. See details here.

On that note...

And the plan said usability, motherhood, apple pie
So I shined up my Chrome
Logged into the Bugz and I went in to run a query
It came back with a list of items, too many to possibly do
So I flagged a few, set milestones, and voom!
A plan came into view

woah!

(Chorus)
Plan, plan everywhere a plan
Charting out development, done for The Man
Do this, can't do that, did you read the plan?

And the plan said a lot of these
Have been deferred outta sight
So I looked at those bugs and thought to myself,
Hey! I'll submit a patch tonight
To contribute to this project or to get some new feature in
If God was here, he'd tell you to your face,
Thanks, man you're some kinda winner

Chorus

Now, hey you Mister! Can't you read?
You ain't got to be a committer to get a seat
You can write a patch, or build a new feature,
Ain't supposed to be hard
Plan says anyone can contribute,
Just look for the 'helpwanted' keyword!

And the plan said everybody welcome
Come in, site down and code
But when they passed around the hat
At the end of it all,
I didn't have a buck to unload
So I got me an account with Blogger
And I made up my own little site
I said thank you Eclipse for thinking about me,
I'm alive and you're alright

Chorus x 2

Five Man Electrical Band - Signs

Incidentally, there are over 2500 open 'helpwanted' bugs, with over 1100 closed. Thanks to everyone who's contributed over the years!

Where in the world is Modeling Sandiego?

A couple days ago, I pointed out some stats about Popular Eclipse Projects.

Today, let's look at some more detailed stats about the Modeling Project itself, thanks to Google Analytics, which has been installed at on www.eclipse.org since July 19, 2008.

In just under two months since we turned on stat tracking, we've gotten over 90,000 Visits & 300,000 Pageviews from over 40,000 Visitors hailing from over 150 Countries!

View larger images

Not included in the screenshots above are some interesting stats about user agents -- what people are using to visit the Modeling Project's site. As you'd probably expect, it's mostly Firefox, IE, and Opera. But would you believe five Playstation hits? Would you believe a Nokia N80, two SonyEricsson phones, but no shoe phones? How about over 1000 hits from Google Chrome users? (I'm so jealous there's no Chrome for Linux yet!) Here's the complete list:

Rank   User-Agent   Hits   (%)
1.  Firefox  56,037  (59.17%)
2.  Internet Explr 30,649  (32.36%)
3.  Opera  2,745  (2.90%)  
4.  Safari  2,297  (2.43%)  
5.  Mozilla  1,373  (1.45%)  
6.  Chrome  1,174 (1.24%)  Already in the #6 spot!
7.  Konqueror 319  (0.34%)  
8.  SeaMonkey 29  (0.03%)  
9.  (not set) 18  (0.02%)  
10.  Mozilla Compatible Agent 12  (0.01%)  
11.  Netscape 12  (0.01%)
12.  Camino  8  (0.01%)
13.  Galeon  8  (0.01%)
14.  Playstation 3 4
15.  Googlebot 2
16.  MyBrowser; MyWindows 2
17.  NetFront 2
18.  SmallProxy 3.4.1 Free 2
19.  Axis2 1
20.  DoCoMo 1
21.  Jakarta Commons-HttpClient 1
22.  MSIE 7.0 1
23.  NETSCAPE FOREVER 1
24.  NokiaN80 1
25.  OpenWave 1
26.  Playstation Portable 1
27.  SonyEricssonW910i 1
28.  X-Browser 1
29.  aaa 1
30.  iexplorer 7.0 WebWasher 3.4 1
31.  openwave 1
32.  preview 1
33.  sonyericssonk750i 1

---

Wondering how you can track your eclipse.org site's stats using your own Google Analytics account? Here's how:

  # see http://wiki.eclipse.org/Using_Phoenix#Google_Analytics
  $App->SetGoogleAnalyticsTrackingCode("UA-2345678-9"); # use your UA account

Request For Comment: Will Anyone Miss All Those EMF Zips?

With the advent of p2 and the smaller EMF features introduced in EMF 2.3, the EMF team is evaluating if the All-In-One, Runtime (3), Sourcedoc (3) & Examples Zips are still meaningful and useful to consuming teams, projects, and products. We'd like to reduce those 8 zips down to a single archived update site zip.

For more on this topic see:

EMF 2.5 - Simplified Downloads

To voice your opinion (be it 'I need that/those zip(s)!' or 'I'm good with using Update'), please post your feedback in bug 240223.

Thanks!

When I'm Sixty-Four

There are now over 5 dozen committers on the Modeling Project!

Used to be modeling people were rare
Just a few of them
Now it seems that everyone talks 'bout modelin'
Over a beer or bottle of wine
If you're a modeler, but remain anon
Codin' with Ecore
Time to contribute, time to commit it
Cuz we're sixty-four!

Send me a postcard, drop me a line,
Stating point of view
Indicate precisely what you mean to say
Yours sincerely, modelin' away
Give me your answer, fill in a form
Set for evermore
Time to contribute, so I can commit it
++sixty-four.

The Beatles - When I'm Sixty-Four

Alice in Change

Still more changes & news to report:

• With the explosion of new projects and components in Modeling, do you find installing all your bits to be increasingly a bit of a chore, even with Update Manager? Then come check out the Modeling Amalgamation Project proposal review tomorrow @ 1500 UTC!
• Are you textually active? The drop by the Textual Modeling Framework proposal review, also tomorrow @ 1500 UTC!
• Ever wondered why the EMF project uses Bugzilla's ASSIGNED state to mean "fix in CVS", and RESOLVED:FIXED to mean "fix available in a build?" Wonder no more. We've switched to a more logical workflow, using RESOLVED:FIXED and VERIFIED:FIXED and silently moved all the old bugs to VERIFIED to help with cleaner queries. For details, see bug 206558.

50 Ways To Be A Modeler

The Eclipse Modeling Project stands poised on the precipice, ready to break the 50 committer mark, just in time for M4 the holidays.

Are you that 50th person? Send me your headshot and details, and I'll add you into the fold.

If you're already listed but your information is out of date (because you are now a committer for more than one project?) reopen bug 182613 with your updated info. If you've so far neglected to send me a photo, now's your chance!

And now... some musical encouragement:

The model is all inside your head
Ed said to me
The answer is easy if you
Take it logically
I’d like to help you in your struggle
For notoriety
There must be fifty ways
To be a Modeler

He said it’s really not my habit
To intrude
Furthermore, I hope my meaning
Won’t be lost or misconstrued
But I’ll repeat myself
At the risk of flak from Euge
There must be fifty ways
To be a Modeler
Fifty ways to be a Modeler

Paul Simon - 50 Ways To Leave Your Lover

Beer-to-Peer Networking, Part 1

The following is an experiment in B2P networking, and an attempt to implement a cross-province UUBP transfer. (Aside: I had considered a cross-country experiment too, eg., Toronto to Porland, but I'm not sure how to implement such an idea, especially given the ever-tighter security policies at the US-Canada border. Any suggestions?)

I'd also like to say upfront that this experiment should not be construed as an irresponsible attempt to encourage others to mix beer (or any form of alcohol, really) w/ software development. Drinking and coding can lead to serious problems. Please, B2P responsibly.

While I realize I'm deviating from the spec, I've decided to implement using a more MDD-based approach, in that the message I plan to send via UUBP will not be an actual beer, but rather a model of a beer, which can then be extrapolated and ingested (and later digested!) at the receiving end.

Raw materials can be acquired from your nearest Beer Store and your office's mail room.

Required Materials

Next, the message is encapsulated for security using the "gift envelope" technique, which I presume won't cause licensing headaches or adoption problems as this technique is already widely used in the public domain.

Assembling the Inner Packet

Completed Inner Packet

Now, a second layer of encapsulation is required to ensure the packet can be properly delivered to its intended audience. While there is no real protection of malicious packet sniffing, this at least ensures that the casual observer won't want to intercept the packet. The procedure is very similar to the above, and so will be lest as an exercise for the reader.

With the completed packet, it was time to attempt delivery. The UUBP spec says nothing about transport methods, so I'll first try a web-based approach:

First Delivery Attempt

Unfortunately, even with a 20" monitor, the packet is too large to be sent via the web:

Delivery Attempt Failure

After this initial setback, it was decided to try a more conventional delivery mechanism. This Canada Post node will be employed to transfer the completed packet to its destination.

Second Delivery Attempt: Success!

The UUBP packet is now in transit via the B2P network. With any luck, its recipient will receive it in a couple of days and take some shots, er, photos to complete Part 2.

UPDATE: See Part 2.

Eclipse Modloling

As it's Friday and I'm not feeling creative, here's a lolcat (thanks to ICHC!) you can fill in yourself. I'll even give you a topic.

The Tigerstripe project proposal is now posted at Eclipse, ready for community feedback.

Is this a good thing? A bad thing? A corporate version of the existing Modeling Project? Should it be debranded and contributed to Modeling? Should it stay branded and exist on its own? Can the Eclipse Foundation really support a project coming with its own branding, if that branding isn't owned by Eclipse? (Consider the rebranding issues with Mylar/Mylyn & Maya/Maynstall.)

And as with Subversive/Subclipse and PDT/phpeclipse, is it really the best approach to have two projects competing for resources when they could instead be sharing them?

Frankly, I don't know enough about Tigerstripe to form an opinion yet. Read it yourself. Post your comments or lolz below.

Woof, meow, and get out of the way.

Modeler Bees

And now a little ditty to cheer up my fellow modelers out there on this miserably wet & grey day. If you're not already a regular in #eclipse-modeling, please drop in and say hi! Thanks to Dave Steinberg for his invaluable assistance in writing this one, and to Ed for the bee pic.

March on, modeler bees!
Know your MDD!

We take our orders given by the spec
Our PMC leaders are Ed and Rich
If you resist us you will feel our sting
Don't code by hand when you can use codegen

Control input with importers!
Control output with genmodel!
Follow the schema totally!
We can reuse our models

March on, modeler bees!
Use your MDT!

A Ph.D. once coming from the west,
And in a flash! OMG, made a nest
Supply of models flowing bottomless
Input any form, you're not powerless

Bidi reference handshaking!
Reflection: code generically!
Conform to schema totally!
We save time with our models

And we'll merge... along, with our genmodel
And we'll fly... our JETs, with our templates drawn
Can the suits... in blue, believe what we've done?
We can regen our models

And we'll merge... along, with our codegen on
And define... constraints, validators drawn
Can even... record, changes that we've done
When we modify data

Can we persist our models?
Yes, go see E-M-F-T!

Billy Talent - Worker Bees

Lies, damned lies, and a side order of acronym soup

By my calculations, EMF has scored over 2 million download requests since 2006-01-01. That's a huge number, but it's also an inaccurate number because our stats gen was broken for a while so there's a big hole in the Update Manager download stats. As such it's probably much higher, given the traffic we've had since the stats tool was fixed.

Looking at the overall numbers for all the tracked Modeling projects, we've had 5 million download requests since 2006-01-01 for EMF + M2T + MDT ... and that's just the Update Manager requests!

It's also fun to see who's downloading our bits. As expected, the US tops the chart, with Canada a respectable 8th. To get a sense for where EMF is used consider where the following projects are developed:

1. US: GMF, M2T-JET, WTP, TPTP, VE, ...
3. China: MDT-EODM, TPTP
5. France: EMFT, MDDi, M2M, GMT
8. Canada: EMF & EMF-QTV, MDT (UML2 & OCL), M2T-JET, GEF, GMF, WTP, TPTP, ...
12. The Netherlands: EMFT-Teneo
17. Czech Republic: GMF, MDT-UML2 Tools

To paraphrase Ed and the Zulu people, "umuntu ngumuntu MDD!" ;-)