Git in colour

I've been using Git for a while now, but only today realized I can have coloured output for diff, grep, branch, show-branch and status, without having to hook in any other external tools (like colordiff, for example).

Here's my ~/.gitconfig file, which enables colour:

[user]
        name = Nick Boldt
        email = nickboldt (at) gmail.com

[giggle]
        main-window-maximized = false
        main-window-geometry = 1324x838+0+24
        main-window-view = HistoryView

[core]
        trustctime = false
        branch = auto
        diff = auto
        interactive = auto
        status = auto
        editor = vim

[merge]
        tool = vimdiff

[receive]
        denyCurrentBranch = warn

[branch]
        autosetuprebase = local

[color]
        ui = true
        diff = true
        grep = true
        branch = true
        showbranch = true
        status = true

[color "diff"]
        plain = normal dim
        meta = yellow dim
        frag = blue bold
        old = magenta
        new = cyan
        whitespace = red reverse

[color "status"]
        header = normal dim
        added = yellow
        untracked = magenta

[color "branch"]
        current = yellow reverse
        local = yellow
        remote = red

HOWTO: partially clone an SVN repo to Git, and work with branches

Skip to the code

I've blogged a few times now about Git (which I pronounce with a hard 'g' a la "get", as it's supposed to be named for Linus Torvalds, a self-described git, but which I've also heard called pronounced with a soft 'g' like "jet"). Either way, I'm finding it way more efficient and less painful than either CVS or SVN combined.

So, to continue this series ([1], [2], [3]), here is how (and why) to pull an SVN repo down as a Git repo, but with the omission of old (irrelevant) revisions and branches.

Using SVN for SVN repos

In days of yore when working with the JBoss Tools and JBoss Developer Studio SVN repos, I would keep a copy of everything in trunk on disk, plus the current active branch (most recent milestone or stable branch maintenance). With all the SVN metadata, this would eat up substantial amounts of disk space but still require network access to pull any old history of files. The two repos were about 2G of space on disk, for each branch. Sure, there's tooling to be able to diff and merge between branches w/o having both branches physically checked out, but nothing beats the ability to place two folders side by side OFFLINE for deep comparisons. So, at times, I would burn as much as 6-8G of disk simply to have a few branches of source for comparison and merging. With my painfullly slow IDE drive, this would grind my machine to a halt, especially when doing any SVN operation or counting files / disk usage.

Using Git for SVN repos naively

Recently, I started using git-svn to pull the whole JBDS repo into a local Git repo, but it was slow to create and still unwieldy. And the JBoss Tools repo was too large to even create as a Git repo - the operation would run out of memory while processing old revisions of code to play forward.

At this point, I was stuck having individual Git repos for each JBoss Tools component (major source folder) in SVN: archives, as, birt, bpel, build, etc. It worked, but replicating it when I needed to create a matching repo-collection for a branch was painful and time-consuming. As well, all the old revision information was eating even more disk than before:

• jbosstools' trunk as multiple git-svn clones: 6.1G
• devstudio's trunk as single git-svn clone: 1.3G

So, now, instead of a couple Gb per branch, I was at nearly 4x as much disk usage. But at least I could work offline and not deal w/ network-intense activity just to check history or commit a change. Still, far from ideal.

Cloning SVN with standard layout & partial history

This past week, I discovered two ways to make the git-svn experience at least an order of magnitude better:

1. Standard layout (-s) - this allows your generated Git repo to contain the usual trunk, branches/* and tags/* layout that's present in the source SVN repo. This is a win because it means your repo will contain the branch information so you can easily switch between branches within the same repo on disk. No more remote network access needed!
2. Revision filter (-r) - this allows your generated Git repo to start from a known revision number instead of starting at its birth. Now instead of taking hours to generate, you can get a repo in minutes by excluding irrelevant (ancient) revisions.

So, why is this cool? Because now, instead of having 2G of source+metadata to copy when I want to do a local comparison between branches, the size on disk is merely:

• jbosstools' trunk as single git-svn clone w/ trunk and single branch: 1.3G
• devstudio's trunk as single git-svn clone w/ trunk and single branch: 0.13G

So, not only is the footprint smaller, but the performance is better and I need never do a full clone (or svn checkout) again - instead, I can just copy the existing Git repo, and rebase it to a different branch. Instead of hours, this operation takes seconds (or minutes) and happens without the need for a network connection.

---

Okay, enough blather. Show me the code!

Check out the repo, including only the trunk & most recent branch

# Figure out the revision number based on when a branch was created, then 
# from r28571, returns -r28571:HEAD
rev=$(svn log --stop-on-copy \
  http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x \
  | egrep "r[0-9]+" | tail -1 | sed -e "s#\(r[0-9]\+\).\+#-\1:HEAD#")

# now, fetch repo starting from the branch's initial commit
git svn clone -s $rev http://svn.jboss.org/repos/jbosstools jbosstools_GIT

Now you have a repo which contains trunk & a single branch

git branch -a # list local (Git) and remote (SVN) branches

  * master
    remotes/jbosstools-3.2.x
    remotes/trunk

Switch to the branch

git checkout -b local/jbosstools-3.2.x jbosstools-3.2.x # connect a new local branch to remote one

  Checking out files: 100% (609/609), done.
  Switched to a new branch 'local/jbosstools-3.2.x'

git svn info # verify now working in branch

  URL: http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x
  Repository Root: http://svn.jboss.org/repos/jbosstools

Switch back to trunk

git checkout -b local/trunk trunk # connect a new local branch to remote trunk

  Switched to a new branch 'local/trunk'

git svn info # verify now working in branch

  URL: http://svn.jboss.org/repos/jbosstools/trunk
  Repository Root: http://svn.jboss.org/repos/jbosstools

Rewind your changes, pull updates from SVN repo, apply your changes; won't work if you have local uncommitted changes

git svn rebase

Fetch updates from SVN repo (ignoring local changes?)

git svn fetch

Create a new branch (remotely with SVN)

svn copy \
  http://svn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.x \
  http://svn.jboss.org/repos/jbosstools/branches/some-new-branch

Using Git like CVS

Tonight I moved the http://rainbowcinemas.ca sources from phpeclipse and CVS to PDT and Git.

Below are some gotchas and tips for initial repo creation, how to keep the central remote copy up to date, and how to work around complaints about updating master directly from remote. I'm sure there's a better way to do this w/o the need for the workaround, but this is what I found worked.

Initial setup To crawl a directory and create a git project for each subfolder: for d in $(find . -maxdepth 1 -type d -not -name "." | \ egrep -v ".ssh|logs|OLD|download|upload"); do cd $d; \ git init; git add .; git commit -m "initial commit" .; \ cd ..; \ done See also Setting up a shared repository.
Create local clone via commandline git clone ssh://servername/path/to/.git folderToCreateLocally	Create local clone w/ eGit Once your repo is created, you can clone a copy from the remote server onto your local box, and import it into Eclipse (with eGit installed) using File > Import > Git > Projects from Git > Clone... .
Commit local changes via commandline As outlined before, you can git pull, git checkout, git commit, and finally git push your changes. If you encounter an error trying to commit changes back to the repo, see the section below, "Allow a ref update to the currently checked out branch of a non-bare repository".	Commit local changes w/ eGit With eGit, you can pull, push, checkout, commit, merge, etc. using the context menu on a Git project or with the Synchronize view. I don't recommend using any of the change sets / models except the Git Change Set, since the others will tend to show more than is actually needed (like local changes which Git doesn't track).
Allow a ref update to the currently checked out branch of a non-bare repository Update the ~/.gitconfig file on the remote server to look something like this: [user] name = Your Name email = your@email.address [color] branch = auto diff = auto interactive = auto status = auto [core] editor = vim [merge] tool = vimdiff [receive] denyCurrentBranch = warn Retrieve changes into remote repo Because I'm using the remote server to both host http-accessible files AND host the git repo, it's important that changes checked into the git repo be then checked out into the local filesystem so that the local workspace is in synch with the repo's metadata. To pull changes, I use git status (to review changes), git reset HEAD <filename> (to reset specified file, or omit filename to reset all changes) and finally git checkout to retrieve the changed file from the repo into the working directory. Access the server w/o a password prompt To skip being prompted for a password when you connect over ssh, scp, or fish, add your public SSH key to the ~/.ssh/authorized_keys file on the remote server. Access the server via an alias Instead of having to reference the server as username@server when connecting, you can add an entry to your ~/.ssh/config file that looks like this: Host shortName Hostname fully.qualified.domain.name.or.IP.address User yourUsername Port 22

HOWTO: Contributing to JBoss Tools using Git

If you'd like to use Git instead of SVN as your SCM tool of choice, here's how you can connect to the JBoss Tools SVN repo, pull down all the sources, work on them locally, then either commit changes back into the SVN repo (or submit a patch, if you're not already a committer).

The instructions below assume you have either Linux, Mac OSX, or Windows w/ cygwin. If you have none of those, YMMV.

Fetch sources from SVN

First, fetch sources from SVN using git-svn. If you don't want to check out all the components, use a subset of the components listed below. The complete list is here.

# create a directory into which to check out the JBoss Tools projects
mkdir ~/trunk; cd ~/trunk;

# fetch projects - this will take quite some time
# Committers, use http://svn.jboss.org/repos/jbosstools/trunk/
# Contributors, use http://anonsvn.jboss.org/repos/jbosstools/trunk/
for d in \
  archives as birt bpel bpmn build cdi common \
  deltacloud documentation download.jboss.org drools \
  esb examples flow freemarker gwt hibernatetools \
  jbpm jmx jsf jst maven modeshape portlet profiler \
  requirements runtime seam site smooks struts \
  tests tptp usage vpe ws xulrunner; do \
    git svn clone http://anonsvn.jboss.org/repos/jbosstools/trunk/${d};
done

Configure Eclipse

Next, fire up Eclipse Helios 3.6 for Java EE Developers.

Install the latest eGit from http://download.eclipse.org/egit/updates.

Install the latest m2eclipse from http://m2eclipse.sonatype.org/sites/m2e/ and optionally, http://m2eclipse.sonatype.org/sites/m2e-extras/.

Restart when prompted.

Import Git projects into Eclipse

Now, import Git projects into Eclipse using:

File > Import 
    Git > Projects from Git
        Click 'Add' then browse for ~/trunk/ 
        Enable [x] Look for nested repositories 
        Click 'Search', then click 'OK' when done

        Select a local repo from the list, click 'Next'
        (*) Import Existing Projects
        (*) Try to share automatically
        Click 'Next'
        Click 'Select All', then click 'Finish'

Repeat for other components you want to import. You can add each component to a working set to keep your workspace sorted by component.

Resolve missing dependencies

While the Eclipse Helios 3.6 for Java EE Developers contains most of the dependencies against which JBoss Tools must be compiled, it does not contain everything. For that, you need to install extra dependencies. There are two places to go:

1. JBoss Tools Target Platform p2 Repo (also available as an archived update site zip for offline use) - contains all the Eclipse.org, google.com, and sonatype.org features needed to compile / install all of JBoss Tools. You can install everything, or just the pieces you need.
2. JBoss Tools Nightly Repo (Update Site) - if you don't have all the source projects in your workspace, you can resolve dependencies against this site and install them from here. Once again, you can install everything, or just the pieces you need.

Build & run tests

With m2eclipse installed, you can simply right-click a project and select 'Run As > Maven Build (ALT-SHIFT-X, M)', which will prompt you complete a run configuration dialog. Here are the simplest options you need to set:

   Goals: clean install
   [x] Resolve Workspace artifacts

You can also run Maven to build your projects outside Eclipse, if you prefer.

If running outside Eclipse, you can run tests which are still tied to the Eclipse debugger.

Commit changes to master repo

Because there's no support yet for 'git svn rebase' or 'git svn dcommmit' you're stuck pushing changes to the master repo using the commandline. However, you can shorten the amount of typing needed using an .alias file. See below.

Use an .alias file

To avoid having to type the same git commands over and over, I use these shortcuts in my ~/.alias file:

# update local git-svn repo from master SVN repo
alias   gitup='for d in $(find . -maxdepth 1 -type d); do cd $d; echo $d; if [[ -d .git ]]; then git svn rebase; fi; cd -; done'

# Push local changes to master SVN repo
alias   gp='git svn rebase; git svn dcommit'

# commit local changes to local git-svn repo
alias   ci='git commit -m'

# check status of local git-svn repo
alias   stat='git status'

So, after committing changes (with eGit or via commandline) I can push those to the master SVN repo w/ a simple 'gp'. If your shell doesn't read the .alias file, make sure your .bashrc loads the file using one of these commands:

source /home/yourUserName/.alias
. /home/yourUserName/.alias

Or, put them directly in your .bashrc file.

Git vs. SVN - Basic Commandline Syntax Reference

Learning the git workflow takes a bit of brain retraining, but since I've been using SVN almost entirely via commandline (because Subversive sucks and locks up my Eclipse when I try to use it for anything beyond synching/updating/committing a handful of files), adopting git's commandline syntax is reasonably similar. Consider these simple operations:

Initial checkout from existing repo for a given branch	git clone http://github.com/sonatype/sonatype-tycho.git; cd sonatype-tycho; git checkout origin/tycho-0.10.x	svn checkout http://anonsvn.jboss.org/repos/jbosstools/branches/jbosstools-3.2.0.Beta1/
Update locally checked out files from central repo	git pull	svn update
List locally changes files/folders	git status	svn stat
Diff locally changed file	git diff somefile.txt	svn diff somefile.txt
Revert locally changed file*	git checkout somefile.txt	svn revert somefile.txt
Revert ALL local changes (except untracked files)*	git reset --hard HEAD	svn revert . -R
Add new file	git add file.txt	svn add file.txt
Add new folder recursively	git add folder	svn add folder
Delete file	git rm file.txt	svn rm file.txt
Delete folder	git rm -r folder (non-recursive by default; use -r to recurse)	svn rm folder (recursive by default; use -N to not recurse)
Commit changed file to central repo	git commit -m "message" file.txt; git push	svn ci -m "message" file.txt
Ignore files/folders (in the current folder)	echo "target *.class bin" > .gitignore; \ git ci -m "gitignore" .gitignore	svn propset svn:ignore "target *.class bin" .; \ svn ci -N -m "svn:ignore" .

Obviously you can do a lot more w/ Git than with SVN (like stashing local changes temporarily), but for the sake of simply moving from a VCS to a DVCS and being able to continue to work the same way you already do, the above table should provide a good introduction.

Git 'Er Done

The discussions in bug 257706: Host a git repository on Eclipse Foundation servers, support git as the repository of Eclipse projects rages on, wind blowing in both directions.

Let's look at the objections to Git @ Eclipse:

Implementing a common build infrastructure would also be complicated by additional code repositories as well.

Not true; the Athena system already supports CVS and SVN, plus a "build from local sources" mode which works w/ a cvs/svn tree dump, a workspace w/ checked out projects, or (TBD, we haven't tested this yet) with a git repo. And we have an open bug to make repo tree structure irrelevant to the local checkout mode. Party on. There is even a Git plugin for Hudson so you can use Hudson to watch your repo for changes, like it does with CVS and SVN.

Can't use unapproved or non-EPL code at Eclipse.org

Not true; from discussions w/ legal@eclipse.org, I've been told at least twice that as long as you're not SHIPPING code that falls under a non-EPL or non-approved-CQ you're entirely fine to USE that code as server-based infrastructure. Rock on.

Cannot include tooling in a release train or host its project at Eclipse.org

Not true; since eGit is EPL and jGit is BSD, I don't see a problem with distributing the tooling that would connect to a Git repo hosted at Eclipse.org. We worked around the license woes for SVN tooling support. We can do it again. (Of course IANAL, TINLA.)

Conclusion:

No legal concerns with use of Git as hosted server infrastructure. Dash Athena (Common Builder) will support Git. Tooling is safe for inclusion in release trains (either fully like CVS is or partially like SVN is).

Only remaining issue is therefore to get it installed and allocate resources to support/manage it. With all the erosion going on lately, this need should not be trivialized.

Before the thaw this spring, this tree was on top of the bluff. With nothing to support it, it was dropped like an unchampioned feature request.

However, in the spirit of open source, several people on the above bug have offered to help w/ setup, testing, support, etc. So the burden here will be shared, like many things at Eclipse (eg., Babel, Hudson). Erosion continues, but we can all help to shore up the loss.

As most people prolly already know, Sourceforge supports the whole spectrum of VCS and DVCS options. If we don't want people to host projects there, Eclipse has to at least offer something from the DVCS world to encourage participation here. Keep the barrier to entry high, and people will go elsewhere. Lower the barrier, and people will come here to party instead.

With everyone feeling the economic- and time-pinch these days, can we really afford to discourage contributions at Eclipse simply because, as the silverbacks say, "why, back in my day, we only had CVS, vi, and notepad, and dangit, that was good enough!" ?

After all, the new world is inevitable.

Dash Athena: Epic Fail

Damn airport scanners.

On Sunday while I was flying home from EclipseCon, I decided to try out Git, and migrated the whole Dash CVS tree into a local Git repo. Then, I deleted the CVS stuff in dev.eclipse.org because Git is so much better and DVCS is the Way of the Future.

Unfortunately, somewhere between Calgary and Toronto, my hard drive died, taking with it the whole repo.

Since we now have to start over completely from scratch, Andrew and I have decided to take this opportunity to re-architect a few things:

1. Instead of Ant scripts which wrap & simplify PDE, we're going to use perl scripts which wrap ant4eclipse. No one really cares about OSGi anymore anyway.

2. Instead of being able to run a build in Eclipse, we're going to support running on MIDP devices only (smartphones) and older hardware (bug 260000). This will make builds much smaller and more portable for everyone.

3. We're dropping the built-in support for running JUnit tests as part of an automated build, because the Agile way is to do Build Breakage Driven Development (BBDD) rather than TDD.

4. Because the Hudson instance on build.eclipse.org is only accessible to an elite few (bug 270633), we're dropping that too, in favour of Cruise Control, much lauded by the WTP and Orbit dev teams for its friendly user interface and many extensible configuration options.

Should you have any concerns with this new plan, don't hesitate to post your comments here. We value the community's input, and will take your thoughts under advisement. Then ignore them and do what we want.