Sunday, April 26, 2009

Mercurial

Google Code is going to add Mercurial support soon, in fact it's already available for some pilot projects. I'm more familiar with Git though, so I needed a good translation guide and I found the Git HG Rosetta Stone. That isn't bad, but there are still some gaps, so I thought I'd share my experience fiddling around with Mercurial. To try and pad out some of those gaps. This is a big long dull technical entry. Sorry.

The first thing I tried was to convert a small Git repository to Mercurial. The way to do this is to use the convert extension.

Now a bit of a rant about extensions here. This goes for Mercurial, but I think it applies to Bazaar too. Anyway, the issue I have is that in order to activate extensions in Mercurial, you have to add a line to a file in your home directory called .hgrc. This seems a bit anti intuitive to me. If these extensions are available in the core, and they are installed and everything, then why do I have to activate them by changing a preferences file? It's probably to keep the core set of commands simple or something, but it adds needless complexity to the whole process.

The documentation on the conversion page mentions converting from CVS, Subversion and Darcs, but there is no mention of converting from Git. I wonder if this is because nobody would really want to convert from Git to something else? ;-) Anyway, the conversion was easy for a trivial repository, just run "hg convert /path/to/repo".

This creates a new directory repo-hg that contains your shiny new Mercurial-converted repository. The directory is initially empty, except for the .hg directory. That's equivalent to the .git directory in a git repository. The odd thing here is that running "hg status" on this empty directory reports no changes - compare this to a "git status" in the same situation, where it would say to you "hey, you've deleted all of your files!", and it seemed a little strange. You have to run "hg checkout" to get all your files back in the directory, which is pretty much the same as the "git reset --hard" idiom that you'd use in the same situation.

Converting my Bunjalloo repository wasn't as easy. I've used a few branches, and some of the history came from subversion originally and there are strange tag/branches there. Mercurial doesn't seem to like this at all and just creates a load of branch-less "things". There's probably a good way to do this, but it would need some investigation.

After compiling and running "hg status" I had a load of unknown files, marked with things like "? build/foo.o". I tried the simplest solution: "mv .gitignore .hgignore". This almost worked, but it game me the following strange error:

could not start inotify server: /tmp/repo-hg/.hgignore: invalid pattern (relre): *.[oa]
abort: /tmp/repo-hg/.hgignore: invalid pattern (relre): *.[oa]

By default Mercurial uses Python's regular expression syntax, not globs. That probably makes some sense, it means it was less effort to code I bet. The error message about inotify is a strange red herring though. The fix was to add a "syntax: glob" line to the file.

After this I made a change in my code to see how Mercurial handles the normal workflow. Git's diff command shows differences in colour. Well, that's not quite true - by default it doesn't, you have to set a preference by running something like "git config color auto". Mercurial does colour too, you have to edit the .hgrc file adding "hgext.color=".

All these properties and preferences can either go in your global $HOME/.hgrc file or in the repository-local .hg/hgrc file. There's no core "hg config" command, but there is an extension to do it. It is a second class citizen that isn't shipped with the core Mercurial and requires an extra not-core-Python library (I suspect this is why it isn't in the default Mercurial). In git you can run the core command "git config --global" to add a config option to your global file, or by default it changes the repository local configuration in the .git directory. It makes things a bit more newbie friendly than editing strange dot files. I can't imagine an ex-Subversion user being comfortable editing a file below .hg, for example. In subversion you just don't go futzing about in the .svn directories.

Once all that was done, I decided to commit the change I had made. Here hg doesn't use git's staging area. That's fine. It is something that works well, but a lot of people don't get it. Mercurial uses the classic approach to this - pass what you want to commit on the command line - which works too. What I do miss here from git is the ability to run "commit -v". With the -v option, you get shown the patch that you are committing as well as the file names. This saves running "$VCS diff" in a separate shell, which is what I always ended up doing with Subversion. A definite regression from git here.

After committing a change or two, I sometimes run gitk to see what is going on overall. Mercurial ships "hgk" which is a really ancient version of gitk tuned for Mercurial commands instead of Git ones. Unfortunately it isn't turned on by default and, you guessed it, you have to edit the .hgrc file. Not only that, but the examples given in the Mercurial wiki are not entirely correct, at least when you install from source. They all state silly paths for hgk.py - the real location is $PREFIX/lib/python$VERSION/site-packages/hgext/hgk.py, where PREFIX is usually /usr/local and VERSION is the Python version, probably 2.5. But even if you set that in your .hgrc, "hg view" doesn't work. You need to copy the file "hgk" from the source contrib directory into your path. This is quite awkward. If something ships with the core, it should be installed and Just Work (TM).

Sadly hgk is not very as good as the current gitk. It's missing a lot of the search options, the "you are here" yellow ball on the current
checkout, status bars for slower updates, and probably some other features that I've forgotten for the moment. Running help says "About gitk" and shows that it is version 1.2 (C) 2005 - the current gitk must be about version 1.80 now and is in active development. The hgk equivalent is about 4 years behind and appears to be more or less abandoned.

The other graphical interface that git has is the "git gui" command. This is really needed for managing the index well, adding chunks of patches and so on. The index is handled automatically in Mercurial, so that's a big chunk of usage that you wouldn't need a gui for. There is an extension - hg record - that emulates the "git add -i" behaviour for adding partial patches. A hg gui would be handy for using that recording extension, if nothing else, but there's nothing included by default. The only likely candidate is hgct, which has not been updated for a couple of years.

Another command I use a lot is "git rebase -i". This lets you squash up commits, change the order of unrelated commits, drop patches, break
up monolithic changesets into smaller ones, and so on. It is really good and easy to use. You run "git rebase -i someid" and get back a
list of commits to do stuff with, something like this:

pick 84d3267 Add the key release waits back in
squash 63656f4 This seems to work on desmume at least
pick d34b4f3 WIP, damn sprites do not show up...
edit c9b6718 Mostly works

# Rebase 7c1db7e..c9b6718 onto 7c1db7e
#
# Commands:
# p, pick = use commit
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

Mercurial doesn't have rebase, but on the other hand it does have the incomprehensible queues system. I've read the documentation, but I just don't get it at all. You need to remember a whole bunch of new Q commands, and you need to initialise the repository to tell it that you want to use queues. Then you can add, push and pop patches. I dunno. It just feels too heavyweight and requires too much planning up front.

The rebase command was added to the git core late in the game, but it integrates seamlessly with everything else (except submodules, but
git submodules have other problems anyway). You don't need to plan to use git-rebase, you just do it. The visualisation is done in your editor - you don't have to push and pop patches to get them in the right order. You just move them about as lines of text. If it all goes wrong, you can run "rebase --abort" and you don't get screwed over by having all these strange extra patch commits, which is what MQs tends to do.

Now to push things to a new bitbucket account, since Google Code doesn't let everyone use Mercurial yet. Creating the empty repository on Bit Bucket was dead simple. Pushing was equally simple - hg push http://bitbucket.org/quirky/repo. You enter a username and password, which is far easier than the git equivalent, where you have to generate a ssh key, perhaps having to fiddle with ~/.ssh/config to tell it that SSH should use the user "git" when it connects to the github server, and so on. I can see how this would be simpler for someone used to Subversion over HTTP. The Mercurial documentation on the push command seems easier to follow than the git equivalent, too.

Running that line will just push changes to the remote repository - it doesn't set up the remote branch, the equivalent of a "git remote add origin git@github.com:user/repo.git". To do that you have to edit .hg/hgrc and add a "[paths]" section pointing to the remote URL. The convention here is to set the URL to a value "default". i.e.

[paths]
default = http://example.com/repo

To see which changes have not been pushed to the remote repo with git, you can use gitk or git log --decorate. Both of these show where the remote head is and where the local one is. By inspection you can tell which commits are missing from the remote branch. The equivalent in Mercurial is to use "hg outgoing", which tells you which changes have yet to be pushed. Ah, another difference for the gitk/hgk list here: hgk doesn't show the remote branches at all. Anyway, Mercurial's outgoing command is handy. You could mostly emulate it in git using "git log origin/master..HEAD", but you need to know the name of the remote branch, which is surprisingly non-trivial in some cases.

Another feature of Git that I tend to use is for creating patches to send to other people. I mostly use this together with git-svn. That way I can clone a Subversion repository, make my local commits and then send patches of those commits off in bug reports or to mailing lists or whatever. This is "git-format-patch" and "git-send-email". Mercurial has a single command for that, "hg email". It's pretty much the same, but also sends email by default (like git-send-email). Sometimes you need to attach patches to HTML forms rather than emailing them, and so generating files is my preferred approach. By default git generates one file per commit - in fact I think that is the only way it works- Mercurial on the other hand only ever generates one file, with multiple patches inside. A nice option in Mercurial is "-o", which automatically generates the patches that are not upstream on the remote branch. Anyway, the command lines are subtly different, and the output is similar-but-not-the-same. Git has its single-patch approach, where you get out what you put in, plus it is less irritating in the number of questions it asks you (none). Mercurial creates a patch-bomb mbox and asks you for a To: email address, even if you don't want to email the file to anyone. It is probably a personal preference thing, but I like the way Git does it better here. They are both usable and scale better than "svn diff > ~/whatever.patch".

Another gotcha that caught me out was that "hg add" without arguments adds all untracked files for commit. This is like "git add .", but normally commands without arguments don't do naughty things. I didn't like the default behaviour here, it was too surprising. Especially if your hgignore file is not set up right. To fix it, you have to run "hg revert -a", which forgets about the added files.

Summary

Here's a big ol' list with the subtle, and not-so-subtle differences that bugged me. I really wanted to use a table, but blogger doesn't do tables without adding a bunch of blank lines...

Git gitignore
Hg .hgignore, syntax: glob to get the same behaviour as git

Git .git/config, ~/.gitconfig, use git-config to modify the values
Hg .hg/hgrc, ~/.hgrc, hg config is a non-core extension

Git git commit -v
Hg hg diff | less; hg commit

Git gitk
Hg hg view - there is some set up involving .hgrc and placing hgk in the path.

Git git rebase
Hg Mercurial Queues, kind of.

Git git push URL ; git remote add origin URL
Hg hg push URL; $EDITOR .hg/hgrc ; [paths] default = URL

Git gitk, git log origin/master..HEAD
Hg hg outgoing

Git git format-patch RANGE
Hg hg email -m filename -o

Git git gui
Hg Nothing equivalent. There's "hg record" for "git add -i", but it is less user friendly.

Git git add . ; Note the dot
Hg hg add ; No dot needed. Take care with that! You'll have to run "hg revert" to fix the mess

The main arguments for using Mercurial over Git are better Windows support and ease of use. I can't comment on how well either work on Windows, but for Linux Mercurial worked more or less as expected. The second argument about hg being much easier than git... I don't know. Maybe. I'm tainted now that I more or less grok DVCS. But given the difficulty in setting up most of the core plugins and the lack of good core GUIs (gitk/git-gui), I disagree. The basic work flow is more or less the same, but Git has this batteries included feel while Mercurial is more limited. Mostly by default, by enabling optional extensions there isn't that much real difference. Besides, I think that for someone used to Subversion who has never used DVCS, either tool would be overwhelming at first.

What I do know is that the GUI tools that come with Git are a huge help for getting up to speed with what is going on "under the hood". Seeing all the commits and their hierarchy in gitk is far easier than reading the log or viewing the gitweb page. Especially if you start doing rebases and history changing stuff. Maybe Mercurial's one-branch approach and lack of history-munging commands makes all these extras unnecessary. Who knows. Time will probably tell.

Either way it's time to get stuck in to learning Mercurial. I'll definitely start using Mercurial when Google Code offers it to everyone, because the current github/GC integration works but it's a bit ad-hoc. It's missing automatic links from issues to commits, the source tab looks a bit wrong, the Updates feed doesn't update, little things like that. I like the Github UI for code - you get the last change right there on the front page - but for non coders, I think it could be a bit daunting. Google Code's interface is cleaner and easier to navigate. Also subversion is missing features like real tags (no the /tags directory is not the same) and a couple of projects I've got on GC still use the default Subversion just because it's easier to set up and everything. Mercurial will be a nice bonus feature.
Reblog this post [with Zemanta]

Tuesday, April 07, 2009

Elite DS version 0.6

I've just cooked up a new release of Elite DS. This adds the music back in that I had to take out when my own GBA XM library proved too tricky/time consuming to port to the DS. I've updated the code so that it compiles with devkitARM r25 and the sound now uses devkitpro's default ARM7 core.

From a technical POV the changes aren't very interesting, but I'll waffle on about them anyway!

Prior to this release, I was using the DS's PSG sound system. This is the bleep-bleep-bleep square/triangle wave sound system that Nintendo has used in its handhelds since the dawn of time. I think it is pretty much unchanged since the original Gameboy ;-) The good thing about this "format" is that it is small - you can say "use this pitch", "play it!", "hold it!" in just a few bytes and the sound lasts ages. You just change the pitch after a given length of time to add new exciting effects. The bad part is that it sounds terrible! Also, the sound format is completely arbitrary/homegrown as you have to just poke registers with values to make the DS go BEEP.

Additional amusement creeps in as you have to poke the sound registers on the ARM7, but the sound events trigger on the ARM9. Like when you get shot or fire a laser. I used the inter-processor FIFO to pass the "bleep now dammit!" messages from the '9 to the '7. Most of the sounds were taken from running Elite under emulation in pocket beeb, capturing the sound values that were written to the BBC sound registers, converting them into the equivalent values for the DS registers and then using these values in Elite DS. They sound more or less like the original BBC sounds this way. This is not necessarily a good thing :-)

The new sound effects were easier to create. I used this simple (to use) sound generator. I spent a pleasant afternoon creating random sounds, some of which may be slightly above the "terrible" level. I forget where I stole the Elite music files from, I think they are nabbed from the NES version. Of course on the DS these all use the fantastic maxmod library. I wish this had been available years ago, it is very easy to use and works exceptionally well.

Ah, there was one random bug that had me scratching my head for a while - I used to use a bit of assembler to clear the screen. This sets all the ARM registers to zero and creates an unrolled loop to blit these zeroes to the video RAM. For some reason it caused crashes on hardware when I started using maxmod. The only thing I can think of is that somehow registers were getting clobbered that maxmod uses in an interrrupt, or vice versa. My fix was to replace that clear screen routine with a DMA fill. Speed is not an issue on the DS compared to the GBA :-) It's a bit cargo-culty but I was desperate and with that register-clobbering explanation I sort of convinced myself. Anyway, I've been playing 0.6 all afternoon and it hasn't shown any more crashes (of the game; there have been colisions between myself and a few Cobra Mk IIIs). Enjoy!

Sunday, April 05, 2009

DSi thoughts

I'm fairly weak and couldn't resist buying a new DSi on Friday when it was released here in Spain (and indeed, the rest of Europe). Here are my impressions of the new console. Keep in mind that I have never owned a DS lite, if that makes a difference.

First of all the bad news: my EZ Flash V doesn't work with the DSi. I knew this would be the case, but had to check anyway. The card is recognised on the menu, but when selected an error message immediately appears. Oh well, I still have the phat DS for homebrew until someone hacks the DSi.

The console itself has a most peculiar finish to it. It's as if the plastic used is "fuzzy". I don't really know how to describe the feeling that you get when you touch the outer case, but it sets my teeth on edge somewhat. Rather like touching polystyrene or chalk.

Once you boot up, the main menu is a new version of the old firmware. I'd completely forgotten about the new-world paranoia too, all those messages about health and safety that loopy's firmware hacks got rid of so many years ago are of course back on the 'i. And there doesn't seem to be a way to get the console to boot directly to the inserted game. You always have to go through the menu.

This menu also has options for the camera, the sound thing, connect to the DSi shop, as well as the old DS download play and pictochat that have been there since day 1 (and nobody actually uses). In addition, any downloaded games from the store appear on this menu in their own icon.

A new feature that was long overdue is in the way the wifi-settings are changed. Rather than have every game that needs to connect to the internet reimplement the settings screen, the firmware itself has a way to update them. This is a good move, although all the existing DS games will still use a different method to the new built in one, even if the settings themselves are stored the same way in the internal memory.

The first thing I did was to download the new Web Browser application for free from the DSi shop. The browser is based on Opera 9.50 and is pretty nifty. More on that in a minute though. First off, it seems that the firmware itself is updateable. The first time you connect to the "DSi Ware" shop, it updates the system. This means that any future firmware-hacks to get homebrew running will be an inevitable arms war with Nintendo. A bit sad really. If the shop were done correctly (i.e. like the Apple app store or the Android equivalent) then the DSi could have been an amazing platform for homebrew. But Nintendo are real control freaks, they make Apple look like some bunch of free for all hippies. For example, before you connect to the DSi Ware store, you have to agree to some sort of Excentric Unenforceable Licence Agreement (or "EULA" as they are known). One of the clauses states that if you sell your DSi you must erase any software you have downloaded from the shop first, as the right to use the software is not transferable. It's quite clear that Nintendo do not want to sell you games, but to sell you a licence to play a game on a given console.

I must say that when you download a game or program from the DSi Ware store, the "progress bar" is great!

I haven't really tried out the built in sound program/toy thing, so no comments on that.

The cameras that are built in are low res - I've seen ambiguous reports on the interwebs that one of them is 0.3 megapixels and the other is 3.0 megapixels. I can confirm that they are both lame-oh 0.3 megapixels. Never mind, they work quite well for the amusing games and stuff that are available at the moment.

You can store your photos onto either internal storage, of which there is about 32MB, or onto SD cards that fit into the side of the console. The internal storage is shared with applications that you download from the store. Once again, Nintendo have screwed up by making it impossible to save these games to the SD card! They made this mistake on the Wii and have repeated it here with the DSi. Update: I have discovered the option hidden away in the settings menu to copy the applications over to the SD card. By default they are stored internally, but it seems that lessons have been learnt since the Wii.

Speaking of cock ups.... There is no way, via either the camera tool or via the Web Browser, to upload your photos to any sort of web site or anything. For example, in gmail the "attach file" control is greyed out - the browser doesn't implement the input type="file" form element.

The camera software lets you trade images locally, i.e. with other DSi users in the same room, but that is all. Hilariously there is another paranoid legalese message when you try to do this: "you acknowledge that you are entitled to distribute these photos and frames and agree that they may be modified, distributed and/or posted by the recipient and/or third parties." Given that you can only exchange the photos with someone stood next to you, this seems a bit overkill to me. A console that has a camera and is wifi enabled, but has no way to send those photos via the intertubes seems like a huge missed oportunity to me. I look forward to some homebrew program that will allow us to do this. Oh. No. Homebrew doesn't run on the DSi. Gah!

You get 1000 "free" points for the store when you first connect to it. Free as in "you just payed 170 euros for the console, the least we can do is give you some cheapo games for it". Apart from a couple of rehashes (Pyoro, Paper Plane) and some puzzle games (I'm more than fed up with brain draining), there is a Wario game that makes use of the camera, "Warioware: Snapped!" it is called. I wish I had read this review (rating: 4/10) before downloading it though. It is amusing for a few minutes, but has no replay value and is not even very practical to play. I get the feeling that this is going to be the pinnacle of camera based games on the DSi, however. It plays similar to the eye-toy games on the Playstation, with the added irritation of the screen being tiny. On the plus side, the replay mode at the end of each round is amusingly done.

The first DS game I plugged in to the 'i was Etrian Odyssey II. It ran fine, so at least the much touted region lock out is not applied retro-actively. I'm on to the 4th stratus, which (if you forgive the minor spoiler) has some quite colourful cherry blossoms and trees. On the DSi they really are extra colourful. The new screen is a lot clearer than the phat one. Even on brightness 1, the lowest, it is brighter than the phat DS screen. The size is slightly larger too, but not so much that it makes any difference. Even side by side it is difficult to notice the difference. I suppose it couldn't be made too big at the same resolution or games would start looking pixelated.

So all in all what do I think of the DSi? Well, as an upgraded DS it is OK if you don't have a DS lite, which I don't. The lack of uploadable photos from the camera is a missed opportunity, what with everyone having Facebook accounts and whatnot nowadays. The browser is cool, even if it does run out of memory when reading pages on blogger.com. In column mode it feels a lot like Bunjalloo ;-) of course these Opera guys have a lot more resources than I do to get things right. They also have Nintendo breathing down their necks, so get things (like no file uploads) wrong too. I wonder if their will be updates to the browser? Given that it is stored in RAM, rather than ROM, it is possible that new versions will appear. Given how tightly Nintendo control everything, together with their restrictive software quality control process - which would make even minor upgrades an expensive effort for Opera -, I won't hold my breath.

Thursday, April 02, 2009

Avoid hard coded variant directories by using a SCons Repository

One of the things about SCons that I didn't like was that by default it built your object files in the same directory as your source code. This means you end up with this:
src/foo.c
src/foo.o
Your ${VCS} status output would be full of this sort of thing:
Untracked files:
? src/foo.o
Tab completion would stop at vi src/foo., grep would add spurious "Binary file foo.o matches". Yuck, what a mess.

Then you discover VariantDir in the SCons documentation and this looks great. You add variant_dir="build", quickly followed by duplicate=False, and all is well. However, this hard coding of "build" is a bit lame. Think about autotools - you could build out-of-source to any directory you liked...
mkdir /tmp/build
cd /tmp/build
/path/to/source/configure && make
Wouldn't it be great if SCons could do this?

It can!

This is a little known feature that I recently stumbled upon completely by chance. Assuming the code is in ~/projects/code, the magic incantation is as follows:
scons -Y ~/projects/code -f ~/projects/code/SConstruct
This uses a SCons secret weapon - Repositories. As a bonus, this particular feature is not even mentioned in the documentation. It means you can compile into any directory using the original source code. The explanation is that the options do the following:
-Y REPOSITORY Search REPOSITORY for source and target files.
-f FILE Read FILE as the top-level SConstruct file.
By saying use this SConstruct to build the code and if you don't find a file, look here, you emulate the autotools/CMake out-of-source behaviour almost perfectly. If the command line looks long, just add --interactive. That way hitting "b" and enter will rebuild.

Hang on, because there are 2 caveats.

First, you have to be careful that when you specify files that are not handled natively by SCons that you use the right names, i.e. in your own Builders. Also internal include directories should use # so that they mean "from the top of the source tree" rather than from the top of the build tree. That means env.Append(CPPPATH=['#mylib']) instead of just "mylib".

Secondly, and more annoyingly, the build directory can't be below the source directory. SCons gets all confused and starts putting another build directory in your build directory (so you can build while you build). But then crashes out saying it can't find the source code. Boo!

Apart from that, you're good to go.