Sunday, November 27, 2011

Non-ARM Android Apps

I've had 2 requests now for my Lords of Midnight remake on x86-based Android devices. A while ago I updated the scons-android build system I use to support multiple architectures, but I didn't actually upload an x86 version to the Market. I thought nobody actually owned one of these mythical devices. Turns out that times are changing.


Back in July, Google added multiple APK support to the Market. Sadly, the multiple APK idea is a bit limited in what you can do. The APKs have to be different in one of 3 key features only. These features are Open GL texture compression type, screen size and API level. Differences in other features - such as whether the device has a touch screen or not, has a hardware keyboard or not, the type of processor it has, and so on, are all ignored. If the only differences between the APKs are in these "minor" features then they are considered identical by the Market software and will not be allowed to coexist.

The explanation I've read as to why this is the case is that the Market client application on devices doesn't provide enough information to the Market server for it to make use of all the feature filters. This makes some sense, as there have only really been multiple ARM ABIs available since June 2010 with release 4b of the NDK. But the Market does already know how to hide non-compatible APKs from you, so I get the feeling this is more a case of "we can't be bothered to do this" rather than "we can't do this".

As it stands, in order to do an x86 version of an app that uses native code while still maintaining compatibility with the majority ARM devices, you need to provide a so-called "fat binary". This is an APK that contains a copy of the ARM version (versions) of your library and the x86 version too. I think it's a bit of a shame that users with one type of processor have to pay the download-tax due to the inclusion of its "evil twin" opposing processor library. Fortunately the Android installation tools are smart enough not to copy useless libraries onto the device for use at run-time, so it is only the increased download size that is a problem.

With this in mind, I've uploaded a new version of War of the Solstice and Chaos that support x86 too. Since I don't own anything that I can actually test these on it is a stab in the dark - it may work, but it will likely just crash, let me know with your 1 star reviews! ;-)

Screenshot of the strange Android Market upload warnings
One thing that worried me slightly is that when you upload an ARM/x86 dual APK to the Market it says "this apk requests 2 native platforms that will be used for filtering". That makes it sounds like you need a strange ARM and x86 device to make the game work. I hope that this is just a typo, and that the final APK works on just about everything.

Tuesday, November 01, 2011

3D Space Game Bug Fixes

Back in July I wrote that I was working on getting rid of the bugs in 3D Space Game. It has certainly taken me a while to get most of them squashed, but I've just uploaded a new version to the Android Market [edit - no longer available] that fixes many crashes (hopefully!). You can also view the source code on the googlecode site [edit - no longer available].

One of the nasty side effects of the old version was it's battery hogging behaviour. This was caused by my dreadfully inefficient Game Boy Advance-like emulation layer that was baked into the game. Over the last few months I've also improved my android-ndk-profiler so it works a lot better on older devices, and I've used it here to try and get better performance. Even with that I still did most of my tuning and debugging on the SDL and Linux port - Android is still missing something like valgrind for native code and GDB doesn't work prior to 2.3.

For the umpteenth time I learned the following lesson: if an (Android) application crashes, chances are it is because I am writing past the end of a buffer into something important! Let's see if the lesson sticks this time :-)

Tuesday, August 30, 2011

Android Upload Fail

Nobody spotted the *ahem* "deliberate" mistake I made with the last release of War of the Solstice - I updated the textual list of changes on the Market, but completely failed to update the APK file to 1.3! In fact, I'd uploaded it to the developer's console, but didn't press the right combination of buttons to get it to actually go live.

face palm

This is the danger of testing via side-loading - I never check that the actual version on the market is what I expect. It's all too much like hard work. Fixed it now though.

Monday, July 25, 2011

Another War of the Solstice update

Despite my prediction, it actually took 2 comments to get the infamous "this game crashes my HTC Desire". The good news is that I've uploaded a new version that may fix the problem.

You can read about the fixes on the Market, the biggy was a couple of buffer overflows that caused random crashes. Once again, valgrind and some carefully crafted unit tests to the rescue. The trick to embedded coding is to get the code running on a real PC :-)

There's a new feature that's inspired by a pretty novel feature in the original Lords of Midnight - a screen shot record! On the original LOM, you could print out the current screen. I bet nobody actually did that though, as you needed a Speccy printer which were rarer than hens' teeth. Anyway, on this Android version pressing volume up or down at any time captures the current screen and saves it to the SD card. Then you can view the screens in your "Gallery" application.

I've also been practicing and finally beat the game with a carefully planned "Ultimate Victory", defeating Doomdark by taking Ushgarak and destroying the Ice Crown in the same game! To celebrate I added in the text for this, which was never contemplated in the original game. It's probably a bit of a let down after an epic 90-day war of attrition, so don't get too excited.

Next up, summer hols permitting, I'm working on "3D Space Game" to fix the crashes there too. I've got it crashing on my phone now, which is half the battle. I get the feeling I'm making some nasty assumptions about video ram because the crashes are inexplicable and "trace-less".

Wednesday, July 06, 2011

Updates to The War of the Solstice

I've uploaded a bug fix version of The War of the Solstice (my [edit - no-longer-available] remake of "The Lords of Midnight") to the Android Market. This adds a way to exit the game, both returning to the title screen and from there actually quitting the game completely so it no longer runs. There are also a bunch of bug fixes.
  • Fix minor Lord-under-control bug when Luxor is killed and the Moonring lost. Previously if Luxor was killed at night while you controlled another lord, you would still be able to control that lord the next day despite the loss of the moonring. This bug may actually have been in the Speccy original too, seems that way from looking at the z80 code.
  • Added fading transition between some screens - this was always in the GBA version, but I never got round to implementing it on Android
  • There was no way to just "think" before on the Android version, so I've changed touch-text-on-look-view to "think". Think is really just when you want to get a more detailed view of what is at a location without using up the location's seekable object/event. So if you're at a town, you don't rest if you just hit the text, whereas before you would use up the town's ability to let you rest if you first used "seek".
  • Fix sometimes corrupt gfx on the left side of the screen when panning left/right. This was a hangover from the GBA port, I miscalculated the offset for the screen scrolling by one block.
  • Better screen repaint for less CPU use. Now it does less work if you stay in the same place, only using CPU when it has to. It still isn't the lowest it can go, but it is much better now.
  • Fixed textual representation of all numeric values. This was a real mess up. Let's just say I wasn't careful enough with my use of C preprocessor macros and leave it at that. This meant that before instead of 10 saying "ten" it said "no". As in "Luxur has no warriors". All other numbers were a factor of 100 out. The actual values were correct though, it was just the representation that was messed up.
  • Fixed white rectangle shield bug after pause/resume. This is an Android platform thing and had been the bane of my Androidy coding existence. I'm not sure I've really fixed it 100%, but it does seem to work on my phone (so you can all come round and use that... yeah right)
  • Added Exit Game option, Back on title screen quits. Now you don't have to use an app-killer (which are pointless anyway, but whatever). Also if you get completely stuck, or are going for a Morkin win and he gets himself killed it's much easier to restart.
There's also hardware keyboard support in this version. It mimics the Spectrum setup 100%, so now all those letters and numbers on the menus make sense. I've only tested this on an emulator though, so expect it to break horribly on the real deal. For the record, as well as the aforementioned menu shortcuts here are the keyboard controls:
Q - move forwards
R - think
T - choose
E - look
U - night
S - save
D - load
1-8  - change to face direction 1:N,2:NE,3:E,4:SE,5:S,6:SW,7:W,8:NW 
That is all, hope you like it. I've been playing this one quite a bit and despite knowing all there is to know about the game, I'm still crap at it! :-)

Monday, June 27, 2011

NDK Profiler Improvements

I've uploaded a new version of my Android NDK profiler library. The major change in this version is much better timing accuracy through the use of ptrace. This requires Android 2.3 aka Gingerbread to work properly, so if like me you're sporting Froyo it's time to upgrade (if you can...)

If you do use Froyo or earlier then you will get timing information, but it will not be precise. In particular it will give too much time to functions called from other functions, time that is not attributed to the calling function. The actual call counts are still accurate, and it can still give you an idea of bottlenecks.

I've also added a new minimal example that is included in the source release. It's a silly example with 3 buttons and is pretty fragile, but should give you an idea of what you need to do to add profiling. There's also a new snippet of Make code that you can include to avoid copy-pasting all those TARGET_thumb_release_CFLAGS lines.

Sunday, June 26, 2011

The Lords of Midnight remake for Android

I've released another Android game, this time The Lords of Midnight gets the remake treatment. [edit - in the end, this has been removed at the request of Mike Singleton] Once more it's a port from a Nintendo platform, this time the GBA as I never got round to updating it for the DS. It's so close to the Spectrum original that you can almost smell the rubber keyboard. Check out that colour clash.


I've called the remake "The War of the Solstice", as it looks like Mike Singleton together with Chris Wild may be making an official game for iPod and possibly Android at some point. Rumours have been going on for several months now with nothing actually released, so it may come to nothing. They are concentrating on the iPlatforms first too, so hopefully my remake will not have any impact on their sales. I can't blame them for doing iPhone first; if you were to charge people for games on Android it'd be a support nightmare. There are too many devices with too many random reasons to crash or otherwise go wrong.

That's why this is yet another freemake. Hurrah! No adverts either (I'd edit /etc/hosts to block the advert networks myself if I actually used my Android phone for anything). I bet the first comment on the Market for this is going to be "this crashes on my HTC Desire".

Anyway, this remake fixes at least 1 major bug in the AI that was in my original remake (but not in the original game, oh man!), as well as adding touch controls throughout. I've learned my lesson from Elite: people want touch controls. The only exceptions here are back (for obvious back-like things) and the menu button, which is need to save the game. Everything else is fat finger friendly.

Monday, June 13, 2011

3D Space Game Source Code Release

I've received a few emails recently asking about the source code for the ahem Elite port that's in the Android Market [edit - no longer on the Android Market]. The TL;DR summary: the source is now available [edit - no longer available due to copyright concerns] and it includes updates to the DS and GBA versions with all core bug fixes that have gone into the Android version too.

The game began life on the Game Boy Advance and the source was always available in tar-ball form from my now-defunct Geocities remakes site (later moved to a new home). From there I ported the same code to the DS, though I didn't really make the most of the platform. There was no touch input and no use of the 2nd screen at all. The source was in a svn repository now though, on Google Code.

When I ported to the DS I did it the wrong way really: I ripped out the GBA parts and added in the DS bits instead. This meant that the Android port had to at least rip out the DS bits. I knew I didn't want to abandon the DS and GBA just yet, so as well as taking out the platform-specific code in the core part of the game, I added in hooks that each platform would have to implement in its own way. At first I only did the Android, Linux/SDL and DS parts of these hooks, but in the end I've re-done the GBA part too.

There's not a lot of documentation on building the source yet, I'll update that later this week if I get time. If you're interested in compiling all this, you'll need the Android development kits as well as devkitARM. The devkitARM dependency is for grit, so if you get that some other way and don't want to build the DS/GBA versions then you can do that too. Run "scons --help-variables" from the top level directory to see what things you can alter about the build. Or you can just browse the source code and marvel at the hideousness of it all.

As I've pretty much moved on from DS and GBA coding these versions are probably not as well tested as they ought to be. Let me know of any problems you encounter and I'll see what I can do. Patches are welcome :-) Also: saves are, sadly, not backwards compatible with previous releases. This is due to the addition of a new flag that stores the type of game - "Classic" or "Elite-A" mode. Elite-A is also know as "difficultly turned up to 11 mode".

Sunday, May 22, 2011

Bug Fixes in Android NDK Profiler

One of the biggest outstanding bugs in my Android NDK profiler library was that it would produce the gmon.out file but when you ran it through gprof you would see "no accumulate time" and the timing information was nowhere to be found. Now you can thank Justin Wick for debugging the root cause of the problem and providing a fix.

The library size I was using was too small to trigger the problem. It turned out that on larger libraries the calculation that inserted the addresses into the hystogram buckets had an overflow bug that caused the information to be lost. This was a left over from using the profiling code from Visual Boy Advance, since on the GBA the address space tends to be much smaller especially as it doesn't use random address layouts.

I tested this out using the Android port of FreeCiv, which prior to the change didn't show timing information, and with the change it now shows the times for each call. Once again, thanks to Justin for debugging the problem and figuring this out.

Using armeabi-v7a

The other problem you may have seen mentioned in the comments was with armeabi-v7a. In theory you can combine armeabi and armeabi-v7a together and things just work. Sadly I've had reports that doing this causes code compiled as armeabi-v7a to crash.

I finally got round to looking at what gcc spits out when you use the APP_ABI variable and set it to armeabi-v7a. There was a slight surprise in store for me that would explain the crashes. In a nutshell, when I said in my original post that I had to save all the registers in the profiling glue code, I had made an assumption that the link register (lr) did not have to be saved. With Thumb2 it looks like I do have to save that one as well, since gcc forgets that calling __gnu_mcount_nc (the profiling glue) is a function call that wrecks lr.

Look at this bit of C code:

int foo(int x)
{
        return x * 3;
}

When I compile it with -Os -mthumb --save-temps it gives the following:

foo:
        lsl     r3, r0, #1
        add     r0, r3, r0
        bx      lr

Now if I add in -pg to get the profiling calls I get this:

foo:
        push    {r4, lr}
        push    {lr}
        bl      __gnu_mcount_nc
        lsl     r3, r0, #1
        add     r0, r3, r0
        pop     {r4}
        pop     {r1}
        bx      r1

You can see it pushes lr twice - once for the use in the actual function, and a second time for __gnu_mcount_nc to do what it likes with it. Now if I compile for thumb2 using -march=armv7-a -mthumb I get this:

foo:
        @ link register save eliminated.
        push    {lr}
        bl      __gnu_mcount_nc
        movs    r3, #3
        mul     r0, r3, r0
        bx      lr

That comment is quite interesting. If you squint a bit it almost looks like a subtle compiler bug, since when you add profiling it should consider the call to __gnu_mcount_nc just like another function call and save lr. But as I haven't been able to test this on a real device yet, I'm not 100% sure that it's the only problem with using armeabi-v7a in combination with the profiling library. Plus once I saw what was going on I've been able to work around it anyway. If you get the chance to try this out, and are using armeabi-v7a let me know how the latest release works out for you.

Friday, May 20, 2011

Black screen of death on HTC Desire

It seems that my version of Elite [edit - no longer available] on Android is not behaving very well on the HTC Desire. The comments speak for themselves:
Black screen on start up, freezes phone, have to remove battery on HTC desire. I loved this game back in the day too.
by starbuck (May 20, 2011)
Don't download this app, this app has crashed my f*cking htc desire
by Marco (May 19, 2011)
I am giving it a 4 , but it has crashed a few times and clearer instructions would help. Desire hd
by Thomas (May 7, 2011)
Intro music played but black screen and phone locked. Had to remove battery. Desire
by Robert (May 6, 2011)
Black screen lock-up on starting. Had to remove battery to recover. Not good. Desire.
by Grampian (April 26, 2011)

If there are any HTC Desire owners who want to help fix this problem then please get in touch. I'm obviously doing something wrong, but the platform shouldn't crash to such an extent that you have to take the battery out! Especially when the game does run on some HTC Desires as well as on many other types of phone.

I don't know which is more frustrating: the crashes themselves or the attitude of the people leaving these drive-by 1 star reviews. Writing such a comment may make them feel better (hey! my free stuff doesn't work!) but in the long term it doesn't help anybody.

So if you have a Desire and are experiencing crashes, please, please try and send me a "logcat" of the crash so I can work out what's going on and fix it. There's an application in the Market called aLogcat that lets you capture the log and email it out. Check that no personal information is in there (there shouldn't be any) and send me the info if you can. Otherwise there is nothing I can do, short of blacklisting Desires.

On a happier note: A new version of Chaos is out that fixes all the bugs that were reported, except one. The Nintendo DS and GBA versions are based on this same code. Fixes on all platforms are:
  • Make sound fx volume more consistent
  • Fix dismount wizard bug
  • Speed up magic bolt
  • Add a message when Meditate's side effect kicks in (it freezes the wizard)
The one bug that I can't reproduce is (surprise) Android only, and seems like another "Android hates me" oddity:
  • black screen after pause/resume
I've made a change that may help, but who can tell. Interestingly, a lot of commercial games seem to just kill the activity when you move away from it instead of dealing with this issue.

Saturday, April 30, 2011

Android NDK Profiler Example

I've added an example to the android-ndk-profiler project based on the native-activity from Google's Android SDK. As Google Code doesn't let you mix'n'match licences, and importing a bunch of random code into my SVN seemed a bit crap, I've not included the original source in the repository.

What I have done is supply a patch and a script to copy the code from your NDK installation, patch it up to use profiling, update the build scripts and then build the example. You'll need to have the "patch" program installed to get this working, or apply the changes manually.

Since this is Android 2.3 only, I've only been able to test it on the emulator. Also the emulator doesn't like armeabi-v7a, so that remains completely untested - I'd stick to using APP_ABI := armeabi for profiling for now.

Hopefully this'll clear up any misunderstandings like those mentioned in the comments on the last post I made on the subject. Full details are on this wiki page about getting the example working.

Saturday, April 09, 2011

Why are my textures not showing up on Android?

What a disaster the initial week of "3D Space Game" has been! I recieved lots of 1 star reviews with vague hints about what the problem was - "doesn't work", "goofed up". When I realised there was a problem and added a note on the market asking people to email their problems before "down voting" I got a couple of very helpful bug reports.

It turns out that on the Galaxy S and other larger-screened devices all of the textures I created from PNG images were not showing up. This meant that the control buttons and menu icons just show as white rectangles, completely unusable.

The usual reason for the white rectangle effect is that the size of the texture you're trying to use is not a power of 2. Textures have to be 32x32, 64x32, 128x512, etc - you can't have a 257x123 sized texture (unless it's on the Android emulator, which doesn't care about texture size). I even mentioned this in my "lessons learned" post in January. The code I use creates a texture straight from the PNG like this:

Bitmap bmp = BitmapFactory.decodeResource(getResources(), R.drawable.my_image);
GLUtils.texImage2D(GL10.GL_TEXTURE_2D, 0, bmp, 0);

The size of the texture is taken from the PNG, which I had carefully made sure was a power of 2. On my HTC Magic the textures show up fine, so I thought everything was working correctly.

Well, it turns out that the BitmapFactory uses a default set of options that includes a "resize" flag. The flag is by default set to true, or "resize the image to screw up Open GL textures", as it should be known. The fix for this was to create my own set of Options and set the flag to false like this:

BitmapFactory.Options opts = new BitmapFactory.Options();
opts.inScaled = false;
Bitmap bmp = BitmapFactory.decodeResource(getResources(),
                R.drawable.my_image, opts); /* pass in opts */
GLUtils.texImage2D(GL10.GL_TEXTURE_2D, 0, bmp, 0);

Problem solved. Let's hope for a reversal of the 1-star trend, eh readers?

Saturday, April 02, 2011

3D Space Game for Android

This is pretty much "Elite for Android" [edit - no longer available due to copyright concerns]. It's a port of a port of a remake of a fan-tweaked version of Elite. I first wrote this as a remake of Angus Duggans's Elite A for the GBA (Elite A being a tweaked version of the original Elite on the BBC Micro), ported that to the DS and now here it is on Android. It's not been without problems due to a severe lack of buttons on a phone. This version adds in "classic mode" to play the game without the Elite A modifications.



I started the Android port back in October 2010 and it was more or less complete by December 2010. I've spent the months since then squashing bugs and refining the controls. Hopefully it will work on a lot of devices - I've been testing on 2009 vintage hardware - but there's no way to know for sure. Try it and let me know. In particular Samsung really threw a spanner in the works by not supplying a d-pad or trackball on the Galaxy S.

There are still some known problems in this first release:
  • Yellow line under title is not always shown
  • Sprite text still shown on panels
  • Long press should show route to planet (bit flakey, ditto Galactic Hyperspace)
  • Text occasionally does not update
Most of these are due to the GBA/DS heritage, but none are show stoppers. I'll be updating with fixes as people complain :-) Also, the Nintendo DS and GBA "backports" of this code are not ready, so this is Android-only for now. There are still some outstanding problems that I need to fix on the DS, and the GBA code is further off, but I'm starting to get project fatigue here and need a change.

This video shows the gameplay, it's a little choppy - if anyone knows a good way to video off an Android device I'd love to know. I used droid@screen for this, which seems to be the best option out there, but it doesn't record video. I had to record the X11 window separately using xvidcap and this made things slightly worse. Anyway, on with the show:

Sunday, March 27, 2011

Profiling Android NDK Code

Profiling code on embedded systems is tricky. The Google tools for profiling Android applications only cover Java code, not native C code. So I've created a new project on Google Code for profiling Android NDK shared libraries. I had the idea when reading this post on the GBAdev.org forum - there coder "sverx" asked how to profile code, and I remembered Visual Boy Advance did it pretty well on the GBA, but there was no way to do so on the DS - nor on Android. Well, the Android situation just improved :-)

I read the VBA source, lifted the relevant chunks of gprof-inspired code, hacked in some timing information, and added a touch of assembly to tie together what gcc -pg does and what the profile-creating code does. If you look at a chunk of the output from gcc with -pg, it looks like this:
 
0001184c <do_raisedead_cast>:
   1184c:   b5f8        push    {r3, r4, r5, r6, r7, lr}
   1184e:   b500        push    {lr}
   11850:   f004 f830   bl  158b4 <__gnu_mcount_nc>
   ...
   1189c:   3a01        subs    r2, #1
   1189e:   701a        strb    r2, [r3, #0]
   118a0:   bdf8        pop {r3, r4, r5, r6, r7, pc}

So the compiler pushes the link register (lr) and the __gnu_mcount_nc function has to pop it off the stack. This isn't a normal calling convention - usually each function uses the stack symmetrically, pushing and popping the same number of times. Also I learned that the mcount function must preserve all the registers. You can't smush a single one or Bad Things happen (I've seen a lot of weird crashes lately, I can tell you). This little conundrum took me quite some time to figure out. It can probably be done better, but the implementation I came up with was this:
 
        push {r0-r3}
        push {lr}
        ldr r0, [sp, #20]  @ r0 = lr pushed by calling routine
        mov r1, lr    @ address of calling routine
        bl profCount
        pop {r0}
        mov lr, r0
        pop {r0-r3}
        @ pop previous address
        add sp, #4
        bx lr

Since registers r4+ must be preserved by the called function, I can ignore those when the code calls out to profCount. Keeping r0-r3 clean was the problem, as usually these are "scratch" registers and you're free to mess them up. The function profCount takes the C function-just-called and its caller addresses as arguments for use in generating the profile. Anyway, the results are pretty good for the code I've tested it on - you get the usual call counts and timing information that gprof provides. Here's a bit of the information from Chaos...
 
Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total           
 time   seconds   seconds    calls  ms/call  ms/call  name    
 65.31      0.32     0.32      288     1.11     1.11  android_render_screen
 18.37      0.41     0.09      450     0.20     0.20  clear_square
  4.08      0.43     0.02        3     6.67     6.67  clear_bg
  2.04      0.44     0.01       25     0.40     0.40  print_text16
  2.04      0.45     0.01        3     3.33     3.33  draw_decor_border_internal
  2.04      0.46     0.01        2     5.00    33.00  splash_touch
  2.04      0.47     0.01        1    10.00    52.00  options_back
  2.04      0.48     0.01        1    10.00    54.80  show_options
  2.04      0.49     0.01                             chaos_start
  0.00      0.49     0.00   442368     0.00     0.00  draw_8x8_tile
  0.00      0.49     0.00    28800     0.00     0.00  platform_set_tile_entry
  0.00      0.49     0.00     3646     0.00     0.00  platform_set_text_map_entry
  0.00      0.49     0.00     2671     0.00     0.00  platform_key_pressed
  0.00      0.49     0.00     1186     0.00     0.00  platform_set_palette_entry

index % time    self  children    called     name
                0.32    0.00     288/288         chaos_gl_render [2]
[1]     65.3    0.32    0.00     288         android_render_screen [1]
                0.00    0.00  442368/442368      draw_8x8_tile [22]
                0.00    0.00     286/286         InterruptProcess [30]

Functions with no "calls", "self" or "total" are because they got called via JNI. The "draw_8x8_tile" call count there is accurate - it is called for each 8x8 tile on the screen in each frame. There are 32 x 24 tiles, and 2 layers of tiles. 32 * 24 * 2 * 288 = 442368, as expected. And the android_render_screen count is correct, since at 30 fps that is around 9 seconds of use and I only ran the game for a few seconds to check.

More details on the android-ndk-profiler page, including a guide on how to use the library.

Sunday, February 13, 2011

How to use Bitbucket's free private Mercurial repositories with Git

Update: Since I wrote this post, bitbucket has added native Git support. Are you spotting a pattern here? Anyway, I'll leave the rest of the post here for the historians.

When Atlassian bought bitbucket in September 2010 one of the first things they did was to allow single users an unlimited amount of private repositories for free. If you want to create an backup of your own code without making the source available this seems like a difficult offer to beat. I've been using it for some months to do exactly this.
 
The only problem is that bitbucket uses Mercurial, what if you prefer git as your day to day version control? No worries as the Atlassians might say, you can bridge from Git to Mercurial using hg-git and a bit of fiddling. Here's what I did to set this all up, in case I forget later.


I use Ubuntu 10.04 and although hg-git is in the repositories it is an older version. Instead I installed python-dulwich via apt-get install python-dulwich, cloned the repository at https://bitbucket.org/durin42/hg-git and then added a suitable entry to my ~/.hgrc file:
hggit = /path/to/hg-git/hggit
The first gotcha is that hg-git can't clone a local file:// repository. When you try it, you get an error like this:
/tmp$ hg clone file:///home/rich/projects/test destination directory: test importing Hg objects into Git fatal: 'file:///home/rich/projects/test' does not appear to be a git repository abort: the remote end hung up unexpectedly
The workaround is to serve the repositories using the git-daemon and then clone them using the default git network protocol. I have the code I want to back-up in a directory ~/projects, so I start the git daemon to export all of them:
git daemon --verbose --export-all --base-path=~/projects/ --export-all
That is not particularly secure, but it isn't available outside my home network and means I can clone each repository using hg with the hg-git plugin enabled:
hg clone git://localhost/name name-hg
Once I have a local Mercurial clone, it is easy to set up the push to bitbucket. That is just vanilla Mercurial configuration. In name-hg/.hg/hgrc I added the bitbucket path so the full configuration looks like this:
[paths]
default = git://localhost/name
bb = http://user:password@bitbucket.org/user/project
Now name and project depend on the local name and the remote bitbucket project name, but hopefully you get the idea. I push this out to the server with hg push bb. Next time I want to copy all the changes I've made in the git repository since the last backup, I fire up the git-daemon, sync up the mercurial repository then push out the changes. I have a script that does all this, which boils down to the following:
#!/bin/sh # backup git repos to bitbucket private repos... cd ~/projects git daemon --verbose  --export-all --base-path=~/projects/ --export-all & for name in project1 project2 ; do     # echo "did this once... not needed again"     # hg clone git://localhost/${name} ${name}-hg     cd ~/projects/${name}-hg || exit 1     hg pull || exit 1     hg update || exit 1     hg push bb || exit 1 done
This way I can use git for day to day hacking, and have a secure backup on a server somewhere for if the worst happens.

Saturday, February 05, 2011

Chaos 1.7 and Android Market on the web

I added Chaos 1.7 to the Android Market the other day. It's now passed 3000 downloads and I've had several emails and 5 star reviews from Chaos fans, so I'm pretty pleased that it's gone down that well. The only downer is that it really is for fans only, I just don't have any idea how to teach anyone brought up on modern games how to fight their way through an 8-bit world. Patience is needed, I guess.

This release was the first time I'd linked to the brand new web-based Android Market. Until now I've always pointed to Appbrain. I think Appbrain has been struggling to keep up with the new additions that Google have made - like bigger icons, more screenshots and promo graphics. Appbrain has also had some down time and synchronisation issues, with comments being out of date or not appearing. I'm not sure what will happen to the 'brain. They have done a great service for Android users and it would be sad to see them disappear, but their business model was the classic snatching nickels from the path of an oncoming steamroller. I do hope they stick around, if only to keep Google on their toes.

Anyway, back to Chaos. The changes in this release are:
  • Remember preferences
  • Several performance improvements for better battery life
  • More consistent frame rate on newer Android devices
  • Attack sound no longer plays when sound is disabled in the options
Saving the preferences was suggested in the comments on the Market. I hadn't added it before because I didn't know how to save stuff on Android. Turns out it is really easy and doesn't even need extra permissions. Actually, I'd already learned how to save for the next game I'm doing so a nice bit of "copy paste" later and it was done. Ah, good old copy and paste. The best form of code re-use.

The performance improvements included not redrawing every single square every single frame, which for some reason I was doing before. If you spot any odd graphical artefacts because of this, do let me know.

The frame rate fixes were due to my recent discovery that newer Android devices are locked to a maximum of 30 frames-per-second. Hopefully Chaos now runs at the same speed across the board. If anyone wants to buy me a 2nd generation Android phone to check this on, please get in touch ;-)

The attack sound fix was a silly non-interesting bug. I fixed it ages ago but also got a prod by email to actually release a version with the fix in place.

I've updated the Nintendo DS and GBA versions too. These don't save the preferences, and I haven't tested them on hardware so let me know if they do not work at all. Apart from that, the DS and GBA are compiled with optimizations now (somehow this got lost in the last GBA/DS release) and should go faster. The GBA version was particularly sluggish last time round.

Sunday, January 16, 2011

Lessons Learned with Android

Around the summer of 2010 I decided to switch from hacking around on the Nintendo DS to hacking around on Android.

Android piqued my interest for the reaons I outlined at the time, but also because it has been changing at a breakneck pace especially if you compare it to the DS. I bought a HTC Magic off of ebay to test out my code. At the time the Magic had been out for a year or so, yet it was already considered out of date! To everyone's surprise Google updated the version of Android running on the phone to the very recent Froyo 2.2.1 at the end of December 2010. Meanwhile, Nintendo only updated the firmware on the DS to block flash cartridges that allow homebrew. From a fun-filled hacking point of view Android is a more entertaining prospect.

The distribution method - upload to the Market, and users are automatically notified of updates - is really great. As a user it's always nice to see applications that have been updated and to read the "recent changes" section to see what's new. As a developer, there's no need to advise people that you've released a bug fix. You just upload a new apk file, update the recent changes section, and you're done.

Here are notes on some of the things I've learned while coding for Android.

The emulator is not great for games


When I wrote about my switch to Android I said "coding for Android feels like cheating. The emulator is highly accurate". I must have been viewing the new (to me) platform through rose tinted glasses there. The reality is that the emulator became useless as soon as I started using OpenGL to render Chaos. On hardware I was getting close to 60 frames per second (FPS), but on the emulator I was lucky to get above 10 FPS. This made testing on anything but hardware impossibly slow.

Fortunately I had my experience with emulators from my DS days and I knew that the best way to test embedded code is to get that code running natively on desktop Linux. The majority of the time I compile and test the Linux build, only occasionally switching to Android to check things still work. All the real gameplay testing I do is on hardware though, as the feel of using a touch screen is nothing like the feel of clicking things with a mouse.

There is an inaccuracy in the OpenGL emulation that could catch you out if you never tested on hardware. On the emulator you don't need to call glTexParameterf and set GL_TEXTURE_MIN_FILTER and GL_TEXTURE_MAG_FILTER to GL_LINEAR or similar, which sets up the texture "mip maps". If you fail to do this on a real device, your textures will only show up as white polygons.

I think Android should use an approach more akin to the iPhone or Windows Phone, where you compile to a native simulation of the application. Instead of emulating a full Android phone and running your app, you have the Android APIs call into native code directly. It's probably too late for Google to change this now and I have my own workarounds. Hey, at least if your game runs fast on the emulator, it will run very well on the real deal :-)

Each Android device is a precious snowflake


A game can run well on the phone you have tested with, but this doesn't guarantee anything. Soon after you put the game on the Market, you can be sure someone will post a 1-star review stating that the game doesn't run on their particular Android device. The Market dashboard for developers has a tab for "Error reports" that never shows anything, but it would be nice if it somehow gave a logcat dump of crashes and force closes.

If you want people to get in touch with you instead of posting a 1-star review, you really need to explicitly state that on your game's description. Otherwise you will never get to the bottom of errors. I think the instant gratification of giving a free game 1 star trumps the hassle of getting in touch with the author. It's perfectly understandable really and something other people have noticed with free app users vs paying customers - people using free things actually have higher expectations!

The SoundPool class, which is used to play back sampled sounds and seems ideal for games, will randomly crash on some hardware platforms and some versions of Android if you use MP3. MP3 is one of the supported media formats, but there are some weird issues on some phones. For example Chaos crashed on an HTC Hero, which has almost identical hardware to a Magic, running similar versions of Android (2.2.1 "official" vs 2.2.1 cyanogen mod). Random internet forum posts show that OGG files work better, but that WAV files can crash if looped. When I switched from using MP3 to OGG the crash reports I was seeing dried up.

When managing the life-cycle of a native application, strange things can happen with your OpenGL context. According to the documentation all you need to do is call the "onPause()" and "onResume()" methods of your GLSurfaceView when the user switches away and then returns to the game. However, on Android 1.6 at least you can get in a state were the "onSurfaceChanged()" method is not called again, so you don't have a hook to setting up the correct GL state. This seems to have gone away with Android 2.2, or at least the game's already set-up context is not ruined in the same way. On 1.6 I'd seen Chaos have a wrecked graphical display after the phone went to sleep, but since the 2.2 update this hasn't happened to me.

You should use 30 FPS as a baseline


A few weeks ago I wrote about getting a good frame rate on Android using the NDK. On my HTC Magic I was getting nearly 60 FPS with that approach. Well, it turns out that new devices are limited to 30 fps. So if I assume a 60 fps rate (as seen on the DS), I would make everything run half as fast. The solution is to limit the frame rate to 30 fps even on "faster" (older!) hardware.

An advantage to running at a lower frame rate is that it saves the battery, but it still seems a bit backwards to me.



More ratings after an update


Almost every time I've posted a new version of Chaos it has received a few new ratings on the Market. I imagine that this is because people download the game then go off and play it, but forget to go back to the Market and rate it. When a new version appears, they return to the Market and now they know it's amazing (!) so rate it 5 stars and install the update.

Don't abuse this phenomenon though. If you upload lots of minor versions with no user-visible changes, people will quickly tire and start rating your game worse. I saw this effect in a few comments for the game "My Paper Plane 2", which had dozens of minor updates with no new features. One reviewer said "Minus 2 stars for frequency of updates and no explanation."

Eclipse is not great


I don't know if the issues I've had are related to the ADT plugin, communication with the emulator or Eclipse itself, but the integration of the whole Android SDK + Eclipse IDE is flaky. Often the text editing GUI hangs for 30+ seconds when you type a "." and auto-complete kicks in, with the CPU grinding away at 100%.

Other times Eclipse fails to connect to the emulator to debug or upload new apk files, and you have to restart the emulator. The emulator takes minutes to restart, it is slower to boot than even the hardware phone. Then there's the built in logcat view that will miss messages if they pass by too quickly. The command line adb logcat equivalent will show you what you missed.
I haven't tried the GUI builder nor XML editor, so cannot comment. At least in the last ADT version opening the XML layout files no longer causes a NullPointerException.

As a result of all of this, I continue to do all my development using Vim. For auto-complete I use CTags and Vim's omni completion. This is enough for C code. When editing Java I just write out all my imports the old fashioned way, like we did back before IDEs hid this crufty aspect from us all. I can always say it is helping me learn the API...

I have a Vim command :AdbReinstall that loads the APK file onto the emulator or device. I can build using :make. It's great that the Android SDK is flexible enough to allow me to do this, there are even instructions on how to set parts of it up in the official documentation. I doubt it is as easy to do for iPhone or Windows development.

Filtering based on the hardware used


One interesting characteristic of the Android Market that has matured a lot in the last few months is the way applications are filtered based on the hardware features they use.

As an example, Chaos uses the touch screen and is compiled for arm-eabi. The touch controls are actually optional - if you want you can just about get by with the trackball or d-pad and the buttons - but originally I didn't state that in the manifest. This meant that Chaos would not be seen on x86 Android devices (are there any?) nor on devices without a touch screen (examples?). This information was all auto-detected by the Market server - I didn't have to mention anything anywhere.

Once I decided that Chaos was usable without a touch screen, I added a "uses-feature" section to the manifest stating that the touch screen was optional.

There may be hardware fragmentation, but this part at least is handled in a quite developer-friendly way.

The next game I'm doing will use the compass and accelerometer for tilt controls. As far as I can tell, I won't have to state anything in the manifest to avoid disappointing folk with phones without these features, the Market will add the filters automatically (we'll see!).

Touch screen is bad for controls


The lack of physical buttons on Android means that several genres of games are not really possible, or you need to add a virtual d-pad and buttons on screen that react to touch. Unfortunately there seems to be a horrible bug in the way Android processes events when the user leaves their finger on the touch screen. When you hold your finger on the screen it leads to the touch input completely swamping the CPU to the detriment of the game's frame rate. The onTouchEvent method is called every 10 milliseconds with what amount to non-events, such as subtle changes in pressure detected by the screen's sensor. Even if all you do in the event handler is store the touch position and type, to later process in the main thread, the application will lose out on framerate as the touch event thread is given priority.

Now I may be doing something wrong here, but several similar-looking questions on stackoverflow.com seem to agree with my experience. There's a bug report on the Android project that has been fixed in Gingerbread. Sadly the lower-end devices that really need this fix will never see Android 2.3, at least not officially. Meanwhile, all the "sleep" workarounds proposed don't seem to make things any better.



There we are then. Android has some issues when compared to a closed hardware platform like the DS. You can't be 100% sure that what runs on your device will run on everyone else's. This is scary, but then there's a good chance your game will work OK. And it will probably work fine with different Android versions, screen resolutions, types of processor and hardware features. I think for free games, or if you have a free demo for people to try before they buy, then this is an acceptable compromise.

After my albeit limited experience with this, I understand now why a limited hardware platform is going to be more popular for games. If you compare the amount and variety of games released on the Apple hardware (iP*d/iPhone) to Android, the difference is huge. Giving stuff away for free is obviously not too cost effective (having embedded adverts can mitigate this, but adverts are only viable on huge hits) and creating a free demo takes time away from creating the core game. So given the choice, developers would rather have people pay for the game, right? If you expect people to pay for your game, then you'd like to think it works on their device. When you write for a platform with a limited range of hardware devices, you only have to test on those devices. With the DS, the DSi and DSiXL are identical platforms - Nintendo is very careful about backwards compatibility. You don't need need to test GBA games on a DS to check they run correctly, for example. These things just work.

Now if we consider Android, there's no way any regular developer could test all possible devices. The best you could hope for is to test on maybe some representative devices. If you support old versions of Android, a first generation HTC phone. Then maybe something like a Nexus One, then a Sony phone as they seem to have quite different Open GL hardware, maybe a Motorola and a Samsung due to their popularity. Factor the tablet-sized Android devices and we're talking quite an investment.

This doesn't include the various Android versions that may run on these devices - you probably have to target a lower API level and hope not to slip up and include newer Java API calls. Those would result in run time errors on older versions than your target API. When you write a game primarily in C it's harder to slip up like this, but still possible. For example, getting the name of the SD card directory to write to changed a lot in Froyo. Given all of this, the openness of Android is definitely off-putting if you're only in it for the money. As a tinkerer into this stuff, it certainly keeps me on my toes. Maybe the new Gingerbread SDK with its "look ma! no Java!" C-only Activities and touch screen bug fixes will lead to more Android 2.3-only (paid) games. Especially if the Sony Play-not-station phone runs this version and has any kind of popularity.

Who am I kidding? It's more likely that the iP*s will continue their rise in popularity with gamers, while Android becomes the new N-Gage. Hey I wonder if the 3DS will be hackable? :-)