Monday, July 20, 2009

Migration to Mercurial

Liquid Metal, VeniceImage by ms sdb via Flickr

I've taken the plunge and decided to integrate all of the Bunjalloo code onto the Google Code site. This meant I migrated from the original Github-hosted Git repository to make use of the new Mercurial support that GC added not so long ago.

The main reason I migrated was to get all the issue tracking, wiki and code changes on to one site. I really had 2 choices: migrate the main site to Github and use the issues and wiki on there, or migrate the code to Google Code and use Mercurial. I quite like Github and I think it is amazing that there are so many brilliant free hosting services, but I really prefer the Google Code interface from a user's point of view. It is generally less cluttered.

So what does this change get us? First off, we lose github's famous "social coding" features. I had one fork in 2 years, with 0 additional commits. No great loss there then. You can still email patches to the discussion list of course, and of course with either DVCS the author information is retained. I gain the fact that I can now reference changes a bit easier in bug reports and close tickets straight from commit messages. Assuming I even write any more code or fix any bugs ;-)

I had a couple of pet peeves with Mercurial, mostly that I really missed gitk's features. Luckily I discovered what I assume everyone else must use: Tortoise HG. This provides "hgtk log" on Linux that works a lot better than the "hg view" history viewer, which is a port of a really ancient version of gitk. The most important feature that hgtk has is a refresh button so I don't have to keep killing and restarting the application each time I make some changes.

Some other areas that I saw as weaknesses were to do with Mercurial's history fiddling, or lack thereof. However I decided that I'm probably better off not messing with history too much anyway. Lately I've tended to avoid doing that in Git and I get more done, even if the revision log is not as clean as it could be. I finally understood the way to do this using Mercurial Queues anyway, even if it is a bit more fiddly.

My final missing feature was the "commit -v" feature, which shows you the patch of the commit that you are making without having to open up a separate console. This hasn't been fixed, but I've worked around it by writing a Vim script to do something similar. Pressing "K" shows the diff of the current tree in a new buffer. This actually works out pretty well as I can see a patch and write a comment at the same time, rather than having to jump back to the top as I did with the "commit -v" thing.

To do the actual Git to Mercurial conversion I used the "hg convert" extension that is included with core Mercurial. That worked flawlessly and made switching really easy. The conversion guide on the Google Code support site has detailed steps on what to do when converting from Subversion, but I'll describe a few gotchas that I found with the Git to Hg transition.

The recently released Mercurial 1.3 included a tiny patch that I wrote to generate a slightly nicer log for git conversions, so I used that version. You see Git tracks both the author of a patch and the person who made the actual commit, but Mercurial only tracks the "user". The user is equivalent to Git's "committer" by default, while author information is assumed to be the same dude. When you ran hg convert, it added a line like "committer: A.Committer " to the Mercurial log message for every commit, even if the author and committer were one and the same in the original git repository. It looked a bit silly when 99.9% of the commits were my own. So my patch made sure that the "committer" line only got added if the author of the original change was not the same as the commiter of the change.

Another interesting niggle: Git has 2 different types of tag; lightweight and annotated. Annotated tags can also be gpg signed, but that wasn't the case in my repository. The difference between lightweight and annotated tags is pretty subtle. As far as I understand it, a lightweight tag is simply a reference to a commit ID, while an annotated tag also has its own blob in the Git database.

By default "git tag mytag HEAD" will create a tag of the lightweight variety. This is apparently the Wrong Thing To Do and one of the few places that Git's default behaviour is not the best option. You really should pass the "-a" option to create an annotated tag. Suffice to say that I used the default lightweight type of tags for quite a while until I discovered my mistake. The "hg convert" extension doesn't convert lightweight tags at all, it only converts the chunky annotated kind. This is possibly by design (maybe by misunderstanding?) as it would be easy to fix the convert extension to convert either type of tag.

The easiest workaround for me was to just convert my git lightweight tags to annotated tags in the source git repository using the "--force" option to overwrite the old ones. The convert process picked these up and converted them over correctly. Interestingly enough Justin Williams had posted about a similar problem and his timing was perfect to ask it over on

Now that I've used DVCS a bit more and the novelty of branching has worn off, I decided that I wanted the minimal number of heads in my new Mercurial repository. I also wanted to maintain as much of the released history as possible. Luckily the history was mostly linear. I did create a couple of branches for maintenance releases of Bunjalloo early on, but after about version 0.4 I just made releases from the trunk.

Originally the repository was in Subversion and I pulled in the tag branches with git-svn too. This lead to a few branch stubs with a single commit ("creating tag blah") with a corresponding git tag that I must have created at some point later on. I used the Mercurial Queues extension to trim these out of the history where applicable so that the final repo has just 2 heads - the main trunk and an old, closed maintenance branch from the 0.3 days.

Oh, when you install Mercurial from source on Ubuntu (possibly on any Debian derivative?) it rather inexplicably creates an /etc/mercurial/hgrc file that enables all of the extensions. This lead me to (re)discover a bug with the inotify extension when used in conjunction with Mercurial Queues. My solution was to simply disable the inotify extension (in fact just removing the /etc/mercurial directory and enabling what you need in $HOME/.hgrc is a better idea overall).

Anyway, feel free to check out the code and send me your patches to fix all of those open issues! :-)

Wednesday, July 08, 2009

Bunjalloo 0.7.5

hand_and_bugcolorImage by beneneuman via Flickr

I've just put up yet another new version of Bunjalloo. This one fixes a load of bugs that caused lots of top pages to be broken. In particular you can log in to GMail again. Yay!

Changes in 0.7.5:
  • Improvements to caching - logging in to GMail works again
  • Clicking preference icon goes straight to preferences
  • Fix encoding problems that caused crashes
  • Fixed lots of non-ascii character keyboard bugs
  • Fix configuration changes that use escapable % characters
You may have to manually fix the download path in your configuration settings. This is because the download path could have become messed up and show all path separators as %2F instead of /.

In the next release I want to fix cookies so that you don't have to enter your name and password into all the sites that you log in to. I changed the password on my google account from something like "password" to a good strong one and it's a pain typing it in all the time on the DS ;-)

Monday, July 06, 2009

Creative Zen Mozaic on Ubuntu 9.04 Jaunty

Where Birthday Happens!Image by ♪ Sleeping Sun ♪ via Flickr
I've just been given a Creative Zen Mozaic portable audio player for my birthday (it's a bit early, but I'm not complaining!). I've had a Zen Micro for a few years now and it's a pretty decent gizmo. Small enough to fit in my pocket, has stood up to several falls, has a UI that gets out of the way and it works with Gnomad2.

Update for Ubuntu 9.10: The Mozaic now runs out of the box, no changes needed. I'll leave this here for posterity.

After plugging the new Mozaic in to my laptop, which runs the latest release of Ubuntu (9.04 Jaunty Jackalope at the time of writing) I discovered that horror! it wasn't recognised by Gnomad2. It seems like it's a common problem. The solution is sort of linked to on that bug report, but the help is in French and the file you have to change has moved since that was written. You couldn't make up a better "open source documentation sucks" anecdote even if you tried.

So.. using your favourite editor open up the file /lib/udev/rules.d/45-libmtp8.rules and add the magic lines:
# Creative ZEN Mozaic
ATTR{idVendor}=="041e", ATTR{idProduct}=="4161", SYMLINK+="libmtp-%k", MODE="660", GROUP="audio"
You'll have to do that with super powers, so use something like this to open the file with the correct permissions using gedit:

gksu gedit /lib/udev/rules.d/45-libmtp8.rules

Add the lines right before the first "#Creative ZEN" line that is already in there, just to be on the safe side. This can be derived from first priciples actually, because if you run lsusb it'll say something like:

Bus 001 Device 004: ID 041e:4161 Creative Technology, Ltd

Which ties in pretty nicely to the line you have to add to the 45-libmtp8.rules file.

Anyway, after adding that unplug the Mozaic and plug it back in again. That forces udev to reread the settings you have just changed. Start Gnomad2 and hopefully it will recognise your player and list all the tracks on it. Yipee!

Thursday, July 02, 2009

Like buses

What a disaster! You wait ages for a release of Bunjalloo, then 2 arrive in 2 days. Let me tell you how this happened.

I use version control for all of the main code. All my own code, that is. I also have a big bunch of downloadable 3rd party libraries that Bunjalloo needs. The build system only deals with my own code, it doesn't handle the 3rd party libraries that I install to $DEVKITPRO/libnds and assume are pretty much fixed. So stuff like libpng, jpeg, zlib, the matrix ssl library, all of these have to be downloaded, compiled and installed separately. I have a few patches that I've applied on top of these libraries too. All of these steps are handled by a script that downloads the source files, patches them if needed, compiles and installs. These patches are changes that would either never be accepted in the core library - removing printfs for example, or changing the Makefiles for the DS - or in the case of matrix, the upstream authors don't seem to have any real "community" or way to send them patches.

These libraries are all relatively stable, and when I upgrade devkitPro I can just run my script to install the latest versions. I've also put a tarball of the compiled code on the Google Code site to make your life easier if you want to compile Bunjalloo from code.

There are some libraries here that I've not yet mentioned and that I mostly take "as is". These are the core devkitPro libraries for the NDS - libnds, libfat, and friends. But! I have also been patching dswifi since about Bunjalloo v0.7 to fix an issue with non blocking sockets that just can't be worked around.

Thanks to an oversight I didn't apply the patch when I installed the latest dswifi 0.3.9, which meant that the sockets are not dealt with properly. I assume a socket has connected before it really has, and start shoving data in before it's ready. This has the result that after a few page loads sockets just stop connecting, presumably because I've filled up some buffers with junk. Pages no longer load and the whole thing grinds to a halt.

The solution? Short term, I've patched dswifi again and uploaded v0.7.4. Longer term, I've added a bug report to devkitpro on sourceforge. Hopefully my patch can get integrated and I don't have to apply the changes manually each time I upgrade the DS toolchain. I should have done this a while ago really.

Wednesday, July 01, 2009

Bunjalloo 0.7.3 released

I've released a new version of Bunjalloo on the Google Code site. This is a fairly minor release that mostly updates the code for devkitARM r26 and the various library updates. Full details are in the ChangeLog over on the wiki.

So it's a minor release, but Bunjalloo itself has gone through quite a lot since 0.7.2 came out in December. Development of new features went on hiatus while I tried to incorporate the Woopsi GUI library. After that failed to really work - mostly down to design decisions on both parts; Woopsi uses the DS hardware for really fast scrolling whereas I want flicker free double buffering - I started to go down this crazy architecture astronomy path that went nowhere either.

Now I want to get back to basics. I want to start releasing versions a bit more regularly. I'm going to try getting a version out every month from now on, even if it is crap! Let's see how long I can last. You may notice the new banner logo that can be seen when using moonshell and other menu programs. That is thanks to Sam and I'd like to get some more community contributions. The only way for that to happen is if Bunjalloo is alive and kicking, and I think regular releases is the best way for that.