Retro Coding Corner: Loading ZX Spectrum Snapshots off Microdrives - Part 2
Tuesday, 31st July 2012, 18:20

So now for the harder parts. Using a BASIC loader with one unified chunk of decompression code and data, that is easy stuff. So how do you load 128K of data? Well actually it won't be 128K because we are using Microdrive cartridges that Sinclair only ever advertised as containing 85K, and after some use get stretched and offer usually 90-100K. But I digress...

Anyhow, check out the ZX Spectrum+ 128K memory map what I made, it shows the maximum 64k of address space that the Z80 CPU can address, split into four pages, along with the 16K banks of memory that can be mapped into various ranges.

The first thing to note about this diagram is how the bottom 16K is always mapped to a ROM chip, with the Interface 1 expansion connected, a further 8K ROM can be paged into the bottom half of that bottom 16K but this only happens under specific circumstances which are explained in the source code later.

The start of RAM at 0x4000 (or 16384 in old money) is where the screen display begins, ending at 0x5AFF. We can safely ruin as much of this memory with our code as possible and assume it won't affect the restored snapshot execution, however we still want to destroy as little as possible.

Putting aside the technical issues of file formats, say a compressed snapshot fits in about 85K, where do we load it? There is only 48K of contiguous space to squeeze it into, some of that we'd need for our loading routine and some of it is used by the Interface 1 routines.

Now perhaps we could rewrite the ROM Microdrive routines that deal with reading data from the cartridge, but this is a stupid idea for two reasons. Firstly, it is highly likely that we'd need to make a custom format (including utilities for saving in it) which means learning far more about how the whole thing works than I particularly want to, for no real benefit. And secondly, it would push the size of the loader code by a few K, something we are trying to make as small as possible.

So, we will just split our data up into multiple files, the minimum of which in a perfect world would be 128K / 48K = 2.66. Since we clearly can't have 0.66 of a file, that means three files in total, and we have a fair few K spare for our loader which we'll certainly need.

To make things pretty, we'll carry on with splitting the screen memory from the rest of the data, and restore it first, much like we did with our 48K snapshot code. However what we do next will be different, since we'll be calling ROM routines they expect the system variables area that sits just after the screen to be in tact.

The generic plan becomes as follows:

• BASIC loader reads in and executes the loader code, compressed screen and compressed first third of our memory banks
• Loader code decompresses the screen first, so we have something nice to look at
• Loader code then relocates to 0x61A8 (or 25000 in decimal)
• Relocated code will load in the remaining two blocks of compressed data

Alas things are not even that simple, since the address 25000 sits inside the second page of memory, where the screen also lies, we can load code that spans two and a bit banks. But the only page we can map in any bank we like is the top one at 0xC000!

On a train ride from London to Liverpool Lime Street, I bashed away at my laptop trying to figure the most efficient way to do everything, and this is what I came up with...

Follow This if You Can

The first file contains the loader code, compressed screen data, and a combined compressed block of Bank 1 and Bank 3. We relocate our loader to 25000, decompress the screen code to the screen area of RAM, then decompress what was originally Banks 1 + 3 to Banks 2 + 0. Then we page in Bank 1 to 0xC000 and copy the data from Bank 2 to it. We then page Bank 0 back into 0xC000, copy that data to Bank 2, page in Bank 3 at the top, then copy Bank 2 to Bank 3.

So Banks 1 + 3 now have their correct data from the original snapshot, via this shuffling process. And we page Bank 0 back in and load the next of our three files, this time containing Banks 7, 4 and 6. But only 4 and 6 are compressed contiguously, 7 is not. The reason being, we can page in Bank 7 and then extract the data straight to this area saving us a step.

After this, we do almost exactly the same thing we did before, extracting Bank 4 + 6 data to Banks 2 + 0, then paging in Bank 4, copying Bank 2 to 4, paging back in Bank 0, copying Bank 0 to Bank 2, and finally paging in Bank 6 and copying Bank 2 to Bank 6.

This leaves us with one last block of code to load in, which contains the remaining data from Bank 5 (don't forget we've already restored the first part of this Bank because it is the screen area), along with the actual data for Banks 2 + 0. At this stage, we are almost in a position where we can do things in an identical manner to our 48K loader.

We no longer have to worry about the system variables area, so bar an extra few steps to ensure that the correct banks are swapping into the correct positions, where we go from here is pre-trodden turf.

The Source

Before we start with the source, just a quick note that for the MegaLZ decompression code you need to visit mayHem's website. And also the microdrive code came in part from old books but mainly from Jim P. on the World of Spectrum forums.

ERR_SP EQU $5C3D ; address of item on machine stack to be used as err return
L_STR1 EQU $5CD9 ; device type "m","n","t" or "b"
N_STR1 EQU $5CDA ; length of filename.
T_STR1 EQU $5CDC ; address of filename.

HD_00 EQU $5CE6 ; file type
HD_0D EQU $5CE9 ; start of data
HD_11 EQU $5CED ; line number

PROG EQU $5C53
VARS EQU $5C4B
ELINE EQU $5C59

NEWSTART EQU 25000

Some boring defines before we get going with the code proper. Only included so people don't go, OMG WTF is that?

    ORG 27000
INIT
    DI

So our initial code will get loaded at 27000, first job of the day is disable interrupts. We don't want any pesky kids stopping us from getting away with things, do we?

CLEARBASIC
    LD HL,(PROG)
    LD (VARS),HL
    LD (HL),$80
    INC HL
    LD (ELINE),HL
    CALL $16B0 ; SET-MIN

First order of the day is to relocate our code so we have more space to load in compressed data. We are going to move from 27000 to 25000, but there is a snag. When calling the Microdrive ROM routines after relocating, memory is too limited and things don't work properly. After I narrowed the problem down to this, JP passed on a way to effectively do a little mini-clear of the BASIC environment that destroys any chances of returning to the BASIC loader, but frees up enough memory for the system variables needed during the forthcoming loading routines.

It apparently originates from Andrew Pennell's * MOVE utility, he knows his stuff, he wrote a book about them in the 80s.

SETNEWSTACK
    LD SP,NEWSTART
    LD (ERR_SP),SP
RELOCATE
    LD HL,MAIN
    LD DE,NEWSTART
    LD BC,SCREENDATA - MAIN
    LDIR
    LD SP,NEWSTART
CALLNEWMAIN
    JP NEWSTART

This bit is pretty simple, we reposition the stack pointer so the ROM routines can push and pop away without damaging any important data, then we actually relocate our remaining code to 25000. We also set a system var that I'm not sure is necessary, but I've been too scared to remove it.

    ORG NEWSTART
MAIN
    LD HL,NEWSTART + (ERROR - MAIN)
    PUSH HL
    LD HL,NEWSTART + (LOADALLDATA - MAIN)
    LD (HD_11),HL
    RST $08
    DEFB $32 ; hook code to call any shadow rom address

    DI
    IM 2
    JP NEWSTART + (FINALDECOMP - MAIN)

This is the first part of our newly relocated code, and we set an error routine onto the stack which should in theory get called if there is a problem with any of our ROM calls. Normally, before you call the Microdrive routines you need to initialise the system variables area for it, but since we've already had this code loaded from a Microdrive we can safely presume it has been done.

The middle section of this code is initially confusing, but related to how the IF1 8K ROM is paged in and out. Unlike the main ROMs, you can't just OUTput to a port to swap it in and out, you have to execute a sneaky RST $08 instruction which is followed by a single byte that indicates what function you are trying to execute. It is a very clever way to achieve the mapping of the ROM, whilst being backwardly compatible with the original 48K system.

Various values of the following byte result in the call of various functions inside the ROM, but 0x32 allows us to call a specific address of our choosing. And the one we choose is not in the ROM at all, it is our own custom loader code routine. This allows us to then call a bunch of IF1 ROM routines without having to page it in and out, safe in the knowledge that when we return from our called routine, the IF1 will page itself out correctly.

So what the above effectively does, is page in the IF1, call our loader, and then page out the IF1 ROM, then disable interrupts, set interrupt mode 2, and jump to our final decompression routine.

ERROR
    LD A,$02
    LD C,0xFE
    OUT (C),A
    JR ERROR
NOERROR
    LD A,$04
    LD C,0xFE
    OUT (C),A
    JR NOERROR

These two little routines we use to call if an error reading the file has occurred. The second is just there for testing purposes, the first actually gets called by the ROM if any of our calls fail. All they do is turn the border red or green, and run an infinite loop.

FNLEN
    DEFW 0x0008
FILENAME
    DEFB "RoboSnap "

This is a bit of data that contains our filename, we'll replace the first character with an underscore for the first bit of code our loader reads, and the second character as well for the second. So our three filenames for a 128K snapshot entitled Robocop become:

• Robocop
• _obocop
• __bocop

DOLOAD
    LD A,"M"
    LD (L_STR1),A
    BIT 4,(IY+124) ; signify load operation
    LD A,3
    LD (HD_00),A ; signify a code file
    LD HL,NEWSTART + (FILENAME - MAIN)
    LD (T_STR1),HL ; address of filename
    LD HL,(NEWSTART + (FNLEN - MAIN))
    LD (N_STR1),HL ; length of filename

DOLOAD is our generic loading routine, which we'll call twice in all. We actually call it later in our code.

The only other thing to note here really, other than this code loads a file from the Microdrive in the first slot, is it presumes IY was correctly set by BASIC before our routine got called, so resetting it is not needed. I could explain the rest but I decided to leave the comments in this one and let them do the talking.

Besides, I've kind of forgotten how it works!

LOADFILE
    CALL NEWSTART + (OPTEMPM - MAIN) ; create temporary channel
    ; after the above call IX becomes the temp M channel area address
    PUSH IX
    POP HL
    ADD HL,DE
    LD DE,HD_00
    LD BC,$0009
    LDIR

    LD L,(IX+$55) ; retrieve code start
    LD H,(IX+$56)
    LD (HD_0D),HL
    LD E,(IX+$53) ; retrieve code length
    LD D,(IX+$54)
    INC DE
    ADD HL,DE

    CALL NEWSTART + (LVMCH - MAIN) ; dive into microdrive load/verify block
    CALL NEWSTART + (CLOSEM2 - MAIN) ; close file

    RET

This is the rest of the load file routine, again with comments, because Microdrive file reading is a bit of a dark art and way beyond the score of this article here. Besides, there are whole books on the subject so if you are really keen, check those out!

PAGEBANKZERO
    LD A,0x10
PAGEBANK
    LD BC,0x77FD
    OUT (C),A
    RET
BANKMIDTOHIGH
    LD HL,32768
    LD DE,49152
    LD BC,16384
    LDIR
    RET
BANKHIGHTOMID
    LD HL,49152
    LD DE,32768
    LD BC,16384
    LDIR
    RET
UPSHIFT
    LD HL,65535 - 10;
    LD DE,65535
    LD BC,32768
    LDDR
    RET

Now we have a few important routines to save a bit of code space. Generally when it comes to Z80 coding, you either code for speed or you code for memory space. Everything will run perfectly fast enough for our purposes, so saving memory space is our priority, hence why we use routines here rather than inline code, and the slower memory copying commands like LDIR and LDDR.

PAGEBANK does what it sounds like, it pages in whatever bank specified in the A register. PAGEBANKZERO extends the function a tiny bit to page in Bank 0 rather than a specified one.

BANKMIDTOHIGH and BANKHIGHTOMID copy data between the top two pages, and we will be doing a lot of shifting of data around with them.

UPSHIFT just copies the top two pages up by 10 bytes. If you remember from our 48K routine, there is a danger when extracting the compressed data that it can overwrite the compressed data with the uncompressed, so we actually extract everything 10 bytes lower in memory, then shift it up.

ROMTABS
ROM1TAB
    JP $1b29 ; OPTEMPM
    JP $15ac ; LVMCH + 3
    JP $12a9 ; CLOSEM2
ROM2TAB
    JP $1b05 ; OPTEMPM
    JP $199d ; LVMCH + 3
    JP $138e ; CLOSEM2
JUMPTAB 
OPTEMPM DEFS $03
LVMCH DEFS $03
CLOSEM2 DEFS $03

This is a little jump table, a solution to the problem that there are two different versions of the Interface 1, a result of bugs in the initial release. And as we call the ROM routines directly, we need to take into account of the fact that the addresses are different.

LOADALLDATA
INITTABLE
    LD A,($00ED)
    LD B,$01
    CP $19 ; issue 1 $1981
    JR Z,CPYTAB
    INC B
    CP $1E ; issue 2 $1e71
    JR Z,CPYTAB
    RET
CPYTAB
    LD HL,NEWSTART + (ROMTABS-(ROM2TAB-ROM1TAB) - MAIN)
    LD DE,ROM2TAB-ROM1TAB
TABINC
    ADD HL,DE
    DJNZ TABINC
    LD DE,NEWSTART + (JUMPTAB - MAIN)
    LD BC,ROM2TAB-ROM1TAB ; how many jumps to copy
    LDIR

This is the start of our routine proper, and is called by the IF1 RST $08 code, after the ROM has been paged in. It detects whether this is an issue 1 or 2 IF1, and then sets up the jump table we mentioned earlier.

DECOMPSCREEN
    LD HL,27000 + (SCREENDATA - INIT) ; data start
    LD DE,16384
    CALL NEWSTART + (DEC40 - MAIN)

First task, decompress the screen so we have something nice to look at whilst the remainder of the snapshot loads. This is pretty much what we do with the 48K snapshots too, so nothing new here.

    ; copy data to top of bank 0->2
    LD HL,27000 + (SCREENDATA - INIT) + 3794 + 18115 - 1 ; Set to end of data
    LD DE,65535
    LD BC,18115 ; Set to data length
    LDDR
    ; decompress data(1+3) to bank 2+0
    LD HL,65536 - 18115 ; Set to new start of data
    LD DE,32768 - 10 ; Destination
    CALL NEWSTART + (DEC40 - MAIN)
    ; shift decompressed data to its proper position
    CALL NEWSTART + (UPSHIFT - MAIN)
    ; page in bank 1
    LD A,0x01
    CALL NEWSTART + (PAGEBANK - MAIN)
    ; copy bank 2(1) to bank 1
    CALL NEWSTART + (BANKMIDTOHIGH - MAIN)
    ; page in bank 0
    CALL NEWSTART + (PAGEBANKZERO - MAIN)
    ; copy bank 0(3) to bank 2
    CALL NEWSTART + (BANKHIGHTOMID - MAIN)
    ; page in bank 3
    LD A,0x03
    CALL NEWSTART + (PAGEBANK - MAIN)
    ; copy bank 2(3) to bank 3
    CALL NEWSTART + (BANKMIDTOHIGH - MAIN)
    ; page in bank 0
    CALL NEWSTART + (PAGEBANKZERO - MAIN)

Here we start the actual work of decompressing Banks 1 + 3, and copying them to their correct places.

LOAD2
    LD HL,NEWSTART + (FILENAME - MAIN)
    LD (HL), "_"
    PUSH HL
    CALL NEWSTART + (DOLOAD - MAIN)

Now we replace the first character in the filename with an underscore. And we save the position of the register we used to do this on the stack, to easily get it back later for the second load. Then we load the next files worth of data.

    ; page in 7
    LD A,0x07
    CALL NEWSTART + (PAGEBANK - MAIN)
    ; decompress data(7) to bank 7
    LD HL,27000 ; Set to new start of data
    LD DE,49152 ; Destination
    CALL NEWSTART + (DEC40 - MAIN)
    ; page in bank 0
    CALL NEWSTART + (PAGEBANKZERO - MAIN)
    ; copy data(4+6) to top of bank 0->2
    LD HL,27000 + 10098 + 19232 - 1 ; Set to end of data
    LD DE,65535
    LD BC,19232 ; Set to data length
    LDDR
    ; decompress data(4+6) to bank 2+0
    LD HL,65536 - 19232 ; Set to new start of data
    LD DE,32768 - 10; Destination
    CALL NEWSTART + (DEC40 - MAIN)
    ; shift decompressed data to its proper position
    CALL NEWSTART + (UPSHIFT - MAIN)
    ; page in bank 4
    LD A,0x04
    CALL NEWSTART + (PAGEBANK - MAIN)
    ; copy bank 2(4) to bank 4
    CALL NEWSTART + (BANKMIDTOHIGH - MAIN)
    ; page in bank 0
    CALL NEWSTART + (PAGEBANKZERO - MAIN)
    ; copy bank 0(6) to bank 2
    CALL NEWSTART + (BANKHIGHTOMID - MAIN)
    ; page in bank 6
    LD A,0x06
    CALL NEWSTART + (PAGEBANK - MAIN)
    ; copy bank 2(6) to bank 6
    CALL NEWSTART + (BANKMIDTOHIGH - MAIN)
    ; page in bank 0
    CALL NEWSTART + (PAGEBANKZERO - MAIN)

Now we extract Bank 7 directly to its correct place, and then repeat with Banks 4 + 6 what we did earlier with Banks 1 + 3 to get them where they belong.

LOAD3
    POP HL
    INC HL
    LD (HL), "_"
    CALL NEWSTART + (DOLOAD - MAIN)

A quick retrieval of our filename pointer into HL, which we increase and make the second character an underscore, then call the load routine for the final time.

    LD HL,27000 + 30263 - 1 ; Set to end of data
    LD DE,65535
    LD BC,30263 ; Set to data length
    LDDR
    
    RET

What we've loaded in is basically all the standard Bank data after the screen, from 23296 to 65535, in the default banks, much as they would be if the machine was in a 48K mode. We shift this to the top of memory, and then we return from our main loader routine which causes the IF1 to page itself out and our starting code to be returned to. That code calls FINALDECOMP which comes next.

FINALDECOMP
LOADREG
    LD SP,NEWSTART + (REGDATASTACK1 - MAIN)
    POP BC ; BC'
    POP DE ; DE'
    POP HL ; HL'
    EXX
    POP IX ; IX
    POP IY ; IY
    POP AF ; IR
    LD I,A
    ; relocate to screen 
SHIFTDEPACK
    LD HL,NEWSTART + (NEWDEPACK - MAIN)
    LD DE,16384
    LD BC,END_DEC40 - NEWDEPACK
    LDIR
    LD SP,16384 + (REGDATASTACK2 - NEWDEPACK)
CALLDEPACK
    LD HL,65536 - 30263 ; Set to new start of data
    LD DE,16384 + 6912 ; Destination
JUMPTONEWDEPACK
    JP 16384
    ; decomp pages 5+2+0 to banks 5+2+0
NEWDEPACK
    CALL 16384 + (DEC40 - NEWDEPACK)
    ; fill last 10 bytes
    LD HL,16384 + (LASTBYTES - NEWDEPACK)
    LD DE,65536 - 10
    LD BC,10
    LDIR
    ; set last regs
SETLASTREGS
    POP DE ; DE
    POP HL ; HL
    POP AF ; AF'
    EX AF,AF' ; '
    LD A,0x00 ; Set ram/rom pages
    LD BC,0x77FD
    OUT (C),A
    POP BC ; R and Border
    LD A,C
    LD C,0xFE
    OUT (C),A ; set Border
    LD A,B
    LD R,A ; set R
    POP BC ; BC
    POP AF ; AF
    LD SP,0xAA45 ; SP
    EI ; NOP out if DI
RUNPROG
    JP 0x9BA3 ; PC
    DEFW 0x0000 ; for depacker stack usage
    DEFW 0x0000 ; for DE call
REGDATASTACK2
    DEFW 0x4000 ; DE
    DEFW 0x5C08 ; HL
    DEFW 0x5F4D ; AF'
    DEFB 0x02 ; Border Colour
    DEFB 0x1B ; R (R minus 5)
    DEFW 0xC05F ; BC
    DEFW 0x2108 ; AF
LASTBYTES
    DEFB "0123456789"
DEC40
        LD A,0x80
        blah blah blah MegaLZ code!
        JR M0
END_DEC40
REGDATASTACK1
    DEFW 0x0000 ; BC'
    DEFW 0x0001 ; DE'
    DEFW 0x2758 ; HL'
    DEFW 0xCA98 ; IX
    DEFW 0xE258 ; IY
    DEFW 0x8310 ; IR
SCREENDATA

The only difference between this code and the 48K loader code now, is the need to write out to 0x77FD which sets the correct Banks expected at the various pages (including the ROMs) as they existed in the original snapshot. This is the only time we corrupt the screen display.

Future Plans

I've been thinking of moving on to supporting Spectrum +3 discs with this system, and have already made a BASIC loader for it. I haven't quite got the code to read in the next block working yet, which is holding me back a bit. Every now and again I sit down and try to get it working, fail and decide to come back another time. I went through this with the microdrive code, but once I got it going the rest happened quickly.

I'd also like to have a version which detects the presence of the Multiface hardware, using the 8K of RAM that has which overlays the top half of the ROM area. This would increase the size of the code a bit but only by a tiny amount I suspect.

One other idea I had which is quite difficult, requires rewriting the code so it overlays just the bottom 8 lines of the screen, but that requires some thought because of the structure the screen is written in. It might be possible, after all the code I wrote for the 48K reconstruction was 251 bytes and there are 256 bytes available to use in the bottom 8 lines of the screen. But the distance actual memory locations are for consecutive lines rules out relatives jumps, since they can only be a maximum of 128 bytes, and the next line is a minimum of 224 bytes away from the end of the previous one.

(Interface 1 and Microdrive picture courtesy of Wikipedia)