Retro Coding Corner: Loading ZX Spectrum Snapshots off Microdrives - Part 1
Thursday, 26th July 2012, 16:57

The older, less American amongst you, will probably be at the very least familiar with the ZX Spectrum microcomputer. During it's long life, before Amstrad swallowed it up and redesigned the hardware to make it cheaper and give it a real keyboard, original makers Sinclair Research created their own fast storage medium.

You can skip the next bit if you know what they are...

When I say fast storage, things are of course relative. With a capacity ranging between about 70k-120k, depending on how much the tape had stretched over time (yes really), it could load a full 48k worth of memory in about 5 seconds. That might sound a lot these days, but the alternative back then was cassette tape, and a few minutes worth of it.

Far from perfect as a medium, the stretching of the tape would render the data unreadable over time, and inside it was actually a long tape loop which meant to read a bit of info that had ran past the head required the full go-around to get there again. Random access it was not.

But this legendary doomed 8-track inspired system has a special place in my heart, not because I could afford it during the original short lived release cycle, but because it's just so unique as solutions to problems go. It went on to appear in the even more ill-fated computer system, the Sinclair QL, something worth reading about if you like tech disaster stories.

Saving Games to Microdrives

This was my goal, save first 48k and then 128k games so they can be loaded off a microdrive cartridge, complete with a loading menu to allow multiple ones per cartridge, room permitting.

Not as easy as it sounds, firstly games have copy protection in the form of fancy loaders, and secondly when it comes to 128k games, if a typical cartridge can store say 90k, how do you save 128k worth of it? The answer to the latter is of course compress it, but that in itself leads to interesting challenges.

A solution to the copy protection issue is to use emulator snapshots as the starting point. These are a saved state of the computer, with every byte of memory and CPU register all recorded at a moment in time. It's very easy to launch an emulator, start loading a game and then pause it at the point just before the loading screen disappears, and save a snapshot.

Doing things this way also has another added advantage, you can write a quick command line program to read in the snapshot, compress it, then write it out with a loader.

Now if only it all turned out to be this easy. :)

Compressing the Data

Because my C was a tiny bit rusty, I caved straight away and installed Visual C++ Studio Express. Even though I don't like Microsoft stuff, it had a built in debugger that would speed up finding silly mistakes I would no doubt make due to being out of practise with all things C.

I did however decide to intentionally keep things as portable as possible, and limit myself to non-Windows specific libraries, which also meant command line only. Still, whilst UIs are nice and all, anyone who is taking snapshots in emulators and trying to convert them to run on the original machines is probably already more than capable of coping with a command line tool.

So with fopen and friends, details of the Z80 file format from various places to cross reference, reading in the limited amount of data you need to support standard retail 48K and 128K Speccies is pretty straight forward. Even the Z80 compression side of things is trivial. However the reason it is trivial, is because it has a terrible ratio, so we need to use something else.

Options for compression when it comes to Z80 is actually quite varied, with some great ratios. However selecting one requires a few things to think about, the first being the decompressor must be small, ideally under 150 bytes. The second is there must be a compressor option written in C.

This lead me ultimately to MegaLZ, written by Russian coder mayHem. It not only includes source code for a C compressor and decompressor, but the Z80 code is pretty fast, fully relocatable and just 110 bytes in length. With that whacked in, I now had a method for compressing the memory data, all that I needed to add was the bits at the other end.

The BASIC Loader

After a hard reset, a ZX Spectrum can launch a file on a microdrive cartridge by doing nothing more than typing the RUN command. The moment you issue any other command though, and this no longer works, it defaults to running the program you've typed or loaded in already, which is obviously not there. But it's a great shortcut, and after doing some simple tests I soon realised that when it came to 48K based games, more than one was going to fit on a cartridge.

So the best thing to do would be write a simple BASIC loader, since that is the only thing that RUN will launch automatically anyway. This loader would catalogue the microdrive and present a list if files in a simple menu. When you select the game to load, it would load that into memory and then call the decompression code.

Easy enough in theory, in practise I came across two annoying issues. The first was, BASIC has only one way to catalogue the files on a microdrive, the CAT command, and that directs output to the screen which means there is no easy way to grab it into an array for presenting as a list to load.

However, wandering through an old book called Make the Most of Your ZX Microdrive, I found a machine code routine with a matching type in that redirected output of the CAT command to a variable. From here you just have to work through the variable and build an array of files, excluding of course the one called "run" and anything beginning with an underscore, for reasons I'll explain later.

Making the loader as small as possible, whilst still leaving room for variables it used, resulted in the ability to lower the top of BASIC RAM (RAMTOP) to 26999. That gives us a maximum total of 37.6K for our compressed data, obviously not enough for a 128K snapshot but we'll deal with the workaround for that later.

The 48K Loader

As previously stated, with roughly 37K to play with, and that includes the 250 odd bytes we'd need to store our restoration code, if MegaLZ cannot save us 11K (a compression ratio of 78.3%) then whatever game fails to be reduced that much just won't work. So far though, most games do a lot better than that, the worst I've found is Sabre Wulf which reduces to 30K. Small games like Cookie are a mere 10K, and Manic Miner is only 15K.

Now, I like things pretty, so I decided to do restore the screen area first, then the remaining memory so you had something to look at whilst it went off on one. This did mean compressing the screen area (16384-23295) separately, and then the remaining block (23296-65535) after, but there is method to my madness. The ultimate plan here would be 128K games, and that is a lot more loading time to await so having a loading screen to view is more aesthetically pleasing.

Restoring a ZX Spectrum to an identical state as possible to that of the machine in a different point of time snapshot requires a number of things.

Set All Memory Contents to Match the Snapshot

Setting all the memory to match a snapshot is, firstly, impossible on a real machine without special external hardware. This is because the program that restores everything has to sit in memory, and clearly whatever area it sits in overwrites data in the snapshot.

So we stick it in screen memory, because whilst it will corrupt that a bit and make it look a tiny bit messy, all this will effect is us looking at it, 99.99999% of programs won't care one jot, especially straight off a loading screen. Screen memory is treated very much as an output device, not a place to store important data or code.

Set Any Important Ports to Their Correct State

Port wise, for 48K snapshots the only one we care about really is 0xFE, because it sets the colour of the border and we want that to be correct. However, because 128K users could be loading in 48K snapshots, we will also write to 0x77FD and disable RAM paging. A lot of early games don't run in 128K mode for the simple reason that they unintentionally output to ports which switch RAM banks, and nothing screws a program flow up more than half of its code being swapped out for something completely different without warning.

Setup All the CPU Registers to Match Those in the Snapshot

Speccies have a number of registers, and then a duplicate set of most of them that you can swap between. The general ones BC, DE, HL, IX, IY, AF', BC', DE', HL' are trivial to reset to their old values. The interrupt register I isn't too bad either, but PC, SP, AF and R require more thought.

PC, being the program counter, points to the current instruction. The last thing our loader code will do is jump to wherever PC pointed to in the snapshot. SP is the stack pointer, and we'll need to position it somewhere safe for the decompression code to use, not to mention later on when we need to load more things from microdrive for 128k snapshots and are calling ROM routines.

But just before we set PC, we'll set SP (stack pointer) to wherever it used to be, so again quite simple. AF is more of an issue, we'll set it just before we set the SP, because whilst A (the accumulator) is only affected directly, F is the flags register and all sorts of commands can cause the Carry and Zero flags to get set/unset. Changing PC and setting the SP shouldn't affect F though so again providing we do it at the end, we are all good.

Now the R register, a mystical beast, this is the refresh register and is connected to memory. Trying to understand how this 7 bit wonder behaves is, er, fun. So why bother? Well some games actually use the R register to detect attempts to circumvent copy protection, and in the interests of completeness, if we can set it why the hell not?

A reason that makes it difficult is... it's the refresh register! If we set it to the same as the snapshot, then the moment we execute a single other instruction it changes, so by the time we set PC it will be wrong. So what we do is calculate backwards from what it should be after JP call, to where we set it, so by the time PC is set correctly, R is what it was in the snapshot.

Relocation, Relocation, Relocation

All 251 bytes of our 48K loader code is prefixed before the compressed screen, which in turn is followed by the remainder of compressed memory. Our BASIC loader will load this in at 27000, then execute it. The code itself will decompress the screen, and then copy the absolute minimal remaining code to the start of the screen display and execute that. Because MegaLZ can potentially overwite the compressed code with the decompressed code before it is finished, we also need to move that as high as possible in memory first.

So the source code for all of this becomes the following...

   ORG 27000
MAIN
   DI
   IM 1

So we disable interrupts and set IM1, although if IM2 is active our C program will change this code as appropriate. It's very important to disable interrupts as we'll be changing not only the interrupt register but also the memory it may point to, which would result in a crash until the full snapshot is restored.

SET48K
   LD A,0x30
   LD BC,0x77FD
   OUT (C),A

Here we make sure the 48K ROM is paged in, which does nothing on a 48K Spectrum, just a 128K version. And we also disable further paging on 128K models, this is what essentially happens if you select the 48K BASIC option in the 128K menu.

LOADREG
   LD SP,REGDATASTACK1
   POP BC ; BC'
   POP DE ; DE'
   POP HL ; HL'
   EXX
   POP IX ; IX
   POP IY ; IY
   POP AF ; IR
   LD I,A

Since we aren't actually going to use anything like all of the Z80 registers during our decompression routine, we can safely load their original values early. We abuse the stack pointer for this purpose, to save a few bytes.

DECOMPSCREEN
   LD HL,SCREENDATA
   LD DE,16384
   CALL DEC40

Next we call the MegaLZ decompression routine to show us that nice pretty loading screen we compressed.

SHIFTDATA
   LD HL,SCREENDATA + 0x1010
   LD DE,65535
   LD BC,0x1010 ; Set to data length
   LDDR

Now we copy the remaining compressed data to the top of memory with a backwards copy so as not to overwrite itself. Note that BC and HL are set to the correct values by our C program, since otherwise this generic loader could not possibly know the correct values.

SHIFTDEPACK
   LD HL,NEWDEPACK
   LD DE,16384
   LD BC,END_DEC40 - NEWDEPACK
   LDIR
   LD SP,16384 + (REGDATASTACK2 - NEWDEPACK)
CALLDEPACK
   LD HL,65536 - 0x1010 ; Set to new start of data
   LD DE,23296 ; Destination
JUMPTONEWDEPACK
   JP 16384

This section is initially a bit complicated without an explanation. Essentially what we do is move all of the remaining loader code to the start of the screen memory, then we fix the stack pointer so it also sits in screen memory. Then we load up HL and DE for the source/destination for the MegaLZ decompressor (which we've also relocated), but then what we do is call the start of screen memory where our remaining code now resides.

Doing it this way minimises the screen memory required, and corruption we cause.

NEWDEPACK
   CALL 16384 + (DEC40 - NEWDEPACK)

This part and all code following now resides at 16384, and not 27000, so we can safely decompress all the data after screen memory to the top of RAM.

   LD HL,16384 + (LASTBYTES - NEWDEPACK)
   LD DE,65536 - 10
   LD BC,10
   LDIR

Now, due to the threat of overlaps with MegaLZ, it is entirely possible that when it reaches close to the end of memory it begins to overwrite the compressed data with the decompressed data. To help alleviate this, what we actually do with our C program is compress 23296-65525, and then copy the final 10 bytes by hand using the section above. Not perfect, but it gives us a little leeway which we didn't have before.

In testing, I did find some snapshots that ended up with the last few bytes corrupted, and that can effect a number of things not least custom interrupt routines. Doing the above really helped solve this, but is by no means a guaranteed flawless solution for every single snapshot.

SETLASTREGS
   POP DE ; DE
   POP HL ; HL
   POP AF ; AF'
   EX AF,AF' ;'
   POP BC ; R and Border
   LD A,C
   LD C,0xFE
   OUT (C),A ; set Border
   LD A,B
   LD R,A ; set R
   POP BC ; BC
   POP AF ; AF
   LD SP,0xFFFF ; SP
   EI ; NOP out if DI
RUNPROG
   JP 0xFFFF ; PC

Now everything is decompressed, we set up the border colour and remaining registers, re-enable interrupts (or if they shouldn't be the C program overwrites the command with a NOP instead), and then jump to the required value of PC.

   DEFW 0x0000 ; for depacker usage
   DEFW 0x0000 ; for DE call
REGDATASTACK2
   DEFW 0x0606 ; DE
   DEFW 0x0707 ; HL
   DEFW 0x0808 ; AF'
   DEFB 0x09 ; R
   DEFB 0x01 ; Border Colour
   DEFW 0x1010 ; BC
   DEFW 0x1111 ; AF
LASTBYTES
   DEFB "0123456789"

Following the main chunk of our code is a chunk of data, this is used to set the final registers, the last 10 bytes of RAM and provide us with a tiny bit of space for stack usage.

DEC40
   LD A,0x80
   blah blah blah blah blah blah
   JR M0
END_DEC40

This next bit is a heavily summarised version of the MegaLZ decompression code. Suffice to say it is a wonderous thing, 110 bytes long and available as part of the MegaLZ package.

REGDATASTACK1
   DEFW 0x0101 ; BC'
   DEFW 0x0202 ; DE'
   DEFW 0x0303 ; HL'
   DEFW 0x0404 ; IX
   DEFW 0x0505 ; IY
   DEFW 0xFFFF ; IR
SCREENDATA

Lastly, we have a bit more data that we use to set the registers at the very beginning of our program, and then SCREENDATA marks the start of our compressed code.

The 128K Loader

The above is very simple stuff, alas things get a lot more complicated with the 128K Spectrum. Firstly, if we can only load in data as 37K at a time, there is no way we are going to be able to load a 128K compressed file in one go. We need to split it up into multiple files, and three is the minimum.

It is probably possible to make a funky custom microdrive loader that lets you read it all from one large file, but that is a very difficult proposition and also makes transfering data from your PC to the real thing harder. At the moment the output of the C loader is a TZX tape file, so you'd need a custom loader for that as well.

So now we need to do a lot more than just load in some code from our BASIC program and execute it, we need to then have that code load more code, twice. To complicate matters even further, the 128K and +2 models can only page in the extra RAM into two banks, and only one of those can have all RAM banks paged into it.

The solution to this, is a juggling act and an awful lot of copying. A story for another time.