Much ado about scripting, Linux & Eclipse: card subject to change

2009-05-11

Automatic Eclipse mirror selection / Better download pages?

Have you ever wanted to fetch a whole stack of Eclipse project runtimes so you can build against them? For example, say you want all the Galileo M7 builds from Platform, TPTP, BIRT, DTP, WTP, and dependencies (EMF, GEF, XSD). You can find the URLs for each zip you want on the projects' pages, and download them one-by-one from the closest mirror, but that's time- and bandwidth-consuming, esp. if you want these on a remote server, not your local box.

Enter the "&r=1" option on http://www.eclipse.org/downloads/download.php, which will fetch from the closest mirror automatically.

So, now, you can script the M7 stack fetch like this:

for u in $*; do
  if [[ ! ${u##*file=*} ]]; then # add the r=1 suffix
    u=${u}"&r=1"
  fi
  echo "wget $u ..."
  wget --no-clobber "$u"
done

Then run it like this:

./fetch.sh \
"http://www.eclipse.org/downloads/download.php?file=/tools/gef/downloads/drops/3.5.0/S200905011522/GEF-runtime-3.5.0M7.zip" \
"http://www.eclipse.org/downloads/download.php?file=/birt/downloads/drops/M-R1-2.5M7-200905061338/birt-report-framework-2.5M7.zip" \
"http://www.eclipse.org/downloads/download.php?file=/birt/downloads/drops/M-R1-2.5M7-200905061338/birt-wtp-integration-sdk-2.5M7.zip" \
"http://www.eclipse.org/downloads/download.php?file=/datatools/downloads/drops/N_DTP_1.7/dtp-1.7.0M7-200905052200.zip" \
"http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/S-3.5M7-200904302300/eclipse-SDK-3.5M7-win32.zip" \
"http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/S-3.5M7-200904302300/eclipse-SDK-3.5M7-linux-gtk.tar.gz" \
"http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/S-3.5M7-200904302300/eclipse-SDK-3.5M7-linux-gtk-x86_64.tar.gz" \
"http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/S-3.5M7-200904302300/eclipse-SDK-3.5M7-macosx-carbon.tar.gz" \
"http://www.eclipse.org/downloads/download.php?file=/modeling/emf/emf/downloads/drops/2.5.0/S200905041408/emf-runtime-2.5.0M7.zip" \
"http://www.eclipse.org/downloads/download.php?file=/tptp/4.6.0/TPTP-4.6.0M7-200904260100/tptp.runtime-TPTP-4.6.0M7.zip" \
"http://www.eclipse.org/downloads/download.php?file=/webtools/downloads/drops/R3.1/S-3.1M7-20090505073946/wtp-S-3.1M7-20090505073946.zip" \
"http://www.eclipse.org/downloads/download.php?file=/webtools/downloads/drops/R3.1/S-3.1M7-20090505073946/wtp-jpt-S-3.1M7-20090505073946.zip" \
"http://www.eclipse.org/downloads/download.php?file=/modeling/emf/emf/downloads/drops/2.5.0/S200905041408/xsd-runtime-2.5.0M7.zip" 

So, now the only problem is that every project structures & styles their downloads pages differently...

  1. DTP -> choose file(s)
  2. GEF -> choose file(s)
  3. EMF/XSD -> choose file(s)
  4. WTP -> choose build -> choose file(s)
  5. Platform -> choose build -> choose file(s) & platform(s) -> click through warnings
  6. TPTP -> choose branch tab -> choose build -> choose file(s)
  7. BIRT -> More Downloads -> Recent Builds -> choose build -> choose file(s)

Am I the only person that finds this inconsistency annoying? Is it time for a more consistent UI? I'm exploring what to do for Athena-based builds, and welcome suggestions in bug 275682. What pages do you like best? Worst? Which are easiest to use? Hardest? Do you prefer the old blue-and-white pages? The purple Phoenix pages? The grey Nova pages? Any UI designers want to contribute?

Or, really, are downloads obsolete? If we could collect statistics on p2 jar downloads, I'm sure we'd see that most people prefer that approach, and I for one would certainly prefer to just build against a p2 repo (or 7) than a pile of zips. I suppose the hybrid solution for now is to provide zipped p2 repos for download, many projects do today (Modeling, GEF, PDT, VE ...).

3 comments:

Ben Vitale said...

Thanks for posting this.

UI consistency is important. I care more about the fact that there is not an easy way for me to "just build against a p2 repo (or 7)". That is badly needed.

oisin said...

I find this inconsistency annoying!

I wonder if there already exists a script which implements:

"Get me the latest R-build of this project X on development stream Y, unless there is an S-build that is newer. Actually if there's a fresh I-build I'll have that instead, now that I think of it."

And which will produce a list of input to wget, or indeed to cp if you are running on b.e.o.

nickb said...

I'm working on it... see bug 273518.

If you input three sites (R, S, I) it should crawl through them and find the latest; if you only want R or S, feed it those sites and it'll install from there. Or just use the Galileo/Ganymede site.

I just need to start testing M7 to see if the new p2.director supports installing multiple IUs from multiple sites in a single operation, as it apparently does.