Infocom Z5 games: Improving my "InterL" project?

[Steve Nickolas]

So far, the Z5 version of InterL is able to create usable disks only for
interpreter A. Unfortunately, this interpreter is full of bugs and
doesn't work very well with the majority of games - not only failing hard
with Inform games, but even with some actual Infocom games.

Sherlock uses the F interpreter; H2G2 uses the E interpreter. (There is a
cracked version of Leather Goddesses of Phobos using the H interpreter;
since it can't tell when the wrong disk is inserted, I don't want to use
cracked H even though it would save me having to encode 18-sector tracks
for the second disk.)

The method by which I inject games now is basically the reverse of the
method by which tools for extracting them work. I'm obviously not doing
something right, since the newer interpreters don't like my disks.

(This is http://3.buric.co/interlz5-001.zip from about a year ago.)

The specific need for this is related to 5.25" disks; as Z5 files can be
up to 256K in size the need to span them across 2 disks is still somewhat
necessary. For larger media, well, qkumba's been working on that. ;)

-uso.

[qkumba]

As we noted elsewhere, the Sherlock interpreter runs all Infocom Z5 files, and many modern ones, too.
The Beyond Zork interpreter runs only a subset of Infocom Z5 files, but succeeds on a collection of modern ones where Sherlock fails.

From reading your source code, one thing stands out: the 100864 bytes limit is specific to certain titles. The limit was removed in the later interpreters, allowing them to go up to track 35 for the static data (though I haven't found any that do).

[qkumba]

For the titles where the limit was removed, bytes 4 and 5 in the .z5 are the size in big-endian format.
Divide it by 64 and you'll have your block limit.
Then multiply it by 256 and you'll have the byte limit.
For Sherlock, it's $78EA -> $01E3 -> $1E300 -> 123648 bytes.

[Steve Nickolas]

Post by qkumba
For the titles where the limit was removed, bytes 4 and 5 in the .z5 are the size in big-endian format.
Divide it by 64 and you'll have your block limit.
Then multiply it by 256 and you'll have the byte limit.
For Sherlock, it's $78EA -> $01E3 -> $1E300 -> 123648 bytes.

Hm.

So I'm doing this...

l=fgetc(file);
if (l!=5)
{
fclose(file);
fprintf(stderr, "%s is not a Z5 file\n", argv[2]);
return -1;
}
/* ignore these bytes */
l=fgetc(file);
l=fgetc(file);
l=fgetc(file);

max1=fgetc(file);
max1<<=8;
max1|=fgetc(file);
max1<<=2;

(i.e., I'm reading a big-endian word from offset 4-5, then multiplying it
by 4 (i.e., 256/64) and treating this as the byte size of disk 1... it
looks like I've misunderstood something.)

-uso.

[qkumba]

Post by Steve Nickolas
(i.e., I'm reading a big-endian word from offset 4-5, then multiplying it
by 4 (i.e., 256/64)

That's not exactly what I intended.
If you don't divide by 64 and then multiply by 256, then you'll need to round the result down to a multiple of 256.

Post by Steve Nickolas
and treating this as the byte size of disk 1... it
looks like I've misunderstood something.)

Count from zero not one. You're picking up bytes 3 and 4, not 4 and 5.

[Steve Nickolas]

Post by qkumba

Post by Steve Nickolas
(i.e., I'm reading a big-endian word from offset 4-5, then multiplying it
by 4 (i.e., 256/64)

That's not exactly what I intended.
If you don't divide by 64 and then multiply by 256, then you'll need to round the result down to a multiple of 256.

Post by Steve Nickolas
and treating this as the byte size of disk 1... it
looks like I've misunderstood something.)

Count from zero not one. You're picking up bytes 3 and 4, not 4 and 5.

There's 4 "fgetc"s, followed by the 2 that become max1.

In its current form I get...something that works for Sherlock, but not for
(say) Zork 1...

l=fgetc(file);
if (l!=5)
{
fclose(file);
fprintf(stderr, "%s is not a Z5 file\n", argv[2]);
return -1;
}
/* ignore these bytes */
l=fgetc(file);
l=fgetc(file);
l=fgetc(file);

max1=fgetc(file);
max1<<=8;
max1|=fgetc(file);
printf ("%04X\n", max1);
max1>>=6;
max1<<=8;

printf ("%lu\n", max1);

will display the hex 78EA and the decimal 123648, which seems to be right
- and the diskset appears to work (boots correctly, $VERIFY works).

I try the same method with Zork 1, and I get 2BC0 and 44800, and a broken
diskset.

-uso.

[qkumba]

Post by Steve Nickolas
There's 4 "fgetc"s, followed by the 2 that become max1.

Excuse me. I missed that first one.

Post by Steve Nickolas
In its current form I get...something that works for Sherlock, but not for
(say) Zork 1...

I've never seen the z5 version of Zork I. If you can share it, then I'd be happy to investigate further.

[Steve Nickolas]

Post by qkumba

Post by Steve Nickolas
There's 4 "fgetc"s, followed by the 2 that become max1.

Excuse me. I missed that first one.

Understandable. (Though it was separated off by the check for the magic
number "5".)

Post by qkumba

Post by Steve Nickolas
In its current form I get...something that works for Sherlock, but not for
(say) Zork 1...

I've never seen the z5 version of Zork I. If you can share it, then I'd
be happy to investigate further.

http://3.buric.co/zork1.z5

-uso.

[I am Rob]

Post by Steve Nickolas
So far, the Z5 version of InterL is able to create usable disks only for
interpreter A. Unfortunately, this interpreter is full of bugs and
doesn't work very well with the majority of games - not only failing hard
with Inform games, but even with some actual Infocom games.
Sherlock uses the F interpreter; H2G2 uses the E interpreter. (There is a
cracked version of Leather Goddesses of Phobos using the H interpreter;
since it can't tell when the wrong disk is inserted, I don't want to use
cracked H even though it would save me having to encode 18-sector tracks
for the second disk.)
The method by which I inject games now is basically the reverse of the
method by which tools for extracting them work. I'm obviously not doing
something right, since the newer interpreters don't like my disks.
(This is http://3.buric.co/interlz5-001.zip from about a year ago.)
The specific need for this is related to 5.25" disks; as Z5 files can be
up to 256K in size the need to span them across 2 disks is still somewhat
necessary. For larger media, well, qkumba's been working on that. ;)
-uso.

Got the F-interpreter moved to a Prodos disk, now to match the data on the sectors with the data in memory.

[qkumba]

Okay, the Sherlock interpreter carries an assumption about a minimum size for the side A data, which Zork I isn't achieving.
However, if I change the interpreter code T00 S04 A$62 and A$73 from #$6F to #$71 then it loads (though it prompts twice to turn over the disk in Zork's case because of the hard-coded prompt in the second case).
The #$6F is a zpage location that holds the low number of blocks left on side A ($70 holding the high part).
It was being zeroed by the prompt, causing a failure to resume reading after turning over the disk. Changing it to #$71 (a zpage location that is initialised later, thus safe to use at that point) avoids that problem.

So, Steve - your conversion process works!

[qkumba]

The ProDOS version didn't need this change because it never prompts, thus the count remains always valid.

[I am Rob]

Post by qkumba
The ProDOS version didn't need this change because it never prompts, thus the count remains always valid.

Has the Sherlock Interpreter already been converted to Prodos?

And have there been any z5 games converted to a file yet to be used with the Prodos version?

[Steve Nickolas]

Post by I am Rob

Post by qkumba
The ProDOS version didn't need this change because it never prompts, thus the count remains always valid.

Has the Sherlock Interpreter already been converted to Prodos?

From what I understand, it has (by qkumba), but it hasn't been released
yet (except to 4am).

Post by I am Rob
And have there been any z5 games converted to a file yet to be used with
the Prodos version?

See above - there's some screenshots on 4am's Twitter.

-uso.

[I am Rob]

Post by Steve Nickolas

Post by I am Rob

Post by qkumba
The ProDOS version didn't need this change because it never prompts, thus the count remains always valid.

Has the Sherlock Interpreter already been converted to Prodos?

From what I understand, it has (by qkumba), but it hasn't been released
yet (except to 4am).

Post by I am Rob
And have there been any z5 games converted to a file yet to be used with
the Prodos version?

See above - there's some screenshots on 4am's Twitter.
-uso.

I was hoping for something a little more simplified like, the interpreter having a loader running under Prodos that can then load any z5 game in and start running it.

I will keep plugging away at it. I am just about there.

[Steve Nickolas]

Post by I am Rob

Post by Steve Nickolas

Post by I am Rob

Post by qkumba
The ProDOS version didn't need this change because it never prompts, thus the count remains always valid.

Has the Sherlock Interpreter already been converted to Prodos?

From what I understand, it has (by qkumba), but it hasn't been released
yet (except to 4am).

Post by I am Rob
And have there been any z5 games converted to a file yet to be used with
the Prodos version?

See above - there's some screenshots on 4am's Twitter.
-uso.

I was hoping for something a little more simplified like, the interpreter having a loader running under Prodos that can then load any z5 game in and start running it.
I will keep plugging away at it. I am just about there.

I understand it uses ProDOS startup protocol, so such a loader could be
made (the videos I've seen show Bitsy Bye being used as the menu).

-uso.

[I am Rob]

Post by Steve Nickolas

Post by I am Rob

Post by Steve Nickolas

Post by I am Rob

Post by qkumba
The ProDOS version didn't need this change because it never prompts, thus the count remains always valid.

Has the Sherlock Interpreter already been converted to Prodos?

From what I understand, it has (by qkumba), but it hasn't been released
yet (except to 4am).

Post by I am Rob
And have there been any z5 games converted to a file yet to be used with
the Prodos version?

See above - there's some screenshots on 4am's Twitter.
-uso.

I was hoping for something a little more simplified like, the interpreter having a loader running under Prodos that can then load any z5 game in and start running it.
I will keep plugging away at it. I am just about there.

I understand it uses ProDOS startup protocol, so such a loader could be
made (the videos I've seen show Bitsy Bye being used as the menu).
-uso.

I guess I am making assumptions here. The interpreter that came on the Dos3.3 disk resides in the language card where Prodos resides.

On 4AM's twitter he mentioned Qkumba's support program for loading from Prodos hard drives. I am assuming then that the interpreter is still loaded into the language card over top of Prodos.

If this is so, I have a better way keeping Prodos alive.

[qkumba]

What we have created is a file-based interpreter that can be launched via Basic.system or BitsyBye, etc. It can run arbitrary z3/z4/z5 files (e.g. straight from ifarchive.org) directly from a ProDOS-based hard disk. It uses ProDOS only as far as loading the interpreter file (in the case of Z4 and Z5 because the games use 127.25kb for themselves, leaving me with only 0.75kb for both read and write support). The Z3 interpreter retains ProDOS in memory, and can even exit back to ProDOS.

The aim is to complete the project in time for KFest 2018.

Rob, what is your better way of keeping ProDOS alive? Moving it to aux isn't an option because it will be overwritten by game data. You'd need more than 128kb of RAM to keep it resident in memory.

[I am Rob]

Post by qkumba
What we have created is a file-based interpreter that can be launched via Basic.system or BitsyBye, etc. It can run arbitrary z3/z4/z5 files (e.g. straight from ifarchive.org) directly from a ProDOS-based hard disk. It uses ProDOS only as far as loading the interpreter file (in the case of Z4 and Z5 because the games use 127.25kb for themselves, leaving me with only 0.75kb for both read and write support). The Z3 interpreter retains ProDOS in memory, and can even exit back to ProDOS.
The aim is to complete the project in time for KFest 2018.
Rob, what is your better way of keeping ProDOS alive? Moving it to aux isn't an option because it will be overwritten by game data. You'd need more than 128kb of RAM to keep it resident in memory.

That is exactly what I had in mind. The AuxLC is being used basically just as a ramdisk for data that gets moved to the lower Main or Aux. There is only one call to that move routine that moves data in and out of the language card.

It probably was a better solution due to the slowness of a floppy drive. But with the use of a hard drive, it is no longer necessary to load every inch of Ram with data.

The data that is stored in AuxLC can be stored on disk until needed, and when it is loaded, there won't be a huge wait during load. 4 kb is the same size as a dbl hi-res graphics file, so maybe 1-1/2 to 2 seconds to load.

Basically then, Prodos will still be in MainLC, the interpreter in AuxLC, and data in lower mem.

I am not seeing the huge loss of memory that you are encountering. The pages of memory that are required for the conversion of disk nybbles to full bytes are no longer needed, saving 3 pages. The floppy disk routines could be all removed as they are already a part of Prodos. The calls for loading and saving the position points as well as the data, could easily be done in a smaller memory area than required for the floppy driver.

There has to be a trigger byte that tells when to load chunks of new data in, as even 127.5 kb of data cannot fit into memory all at once along with the interpreter and other reserved areas.

I found the 5 tables that decipher the data for the interpreter, and I am confident that the data can also be broken down to so called "levels". I am quite surprised how small the interpreter really is, and I see even a lot more space savings once I start re-arranging the source code.

I believe the interpreter can quite easily be reduced to work on a 64 kb machine with an 80-col card (and no need for a 128 kb card). It would still need to have Basic.system removed from memory, and my estimates for memory use would look something like this.

$800.37FF - interpreter and Load/Save routines, game flags
$3800.B7FF - $8000 bytes for data is more than enough to load each level in chunks. 4x 32kb chunks is a 128 kb game and even larger games can be played.

spare room $B800.BAFF

$BB00.BEFF - Prodos disk buffer
$BF00.BFFF - Prodos global page

I would like to see this fully working as well, with the ability to load z5 games. If I can be of some assistance, don't hesitate to ask.

[I am Rob]

Post by I am Rob
The data that is stored in AuxLC can be stored on disk until needed, and when it is loaded, there won't be a huge wait during load. 4 kb is the same size as a dbl hi-res graphics file, so maybe 1-1/2 to 2 seconds to load.

I was thinking of 4x 16-page chunks but meant 16 kb in total for the language card

[Steve Nickolas]

Post by I am Rob
I believe the interpreter can quite easily be reduced to work on a 64 kb
machine with an 80-col card (and no need for a 128 kb card). It would
still need to have Basic.system removed from memory, and my estimates
for memory use would look something like this.

There is a Z5 interpreter for the Commodore 64, so it might be possible.
IIRC, the program that converts Z5 games into disk sets for the 64 does
have to make tweaks to the game code so that the games are able to fit in
64K.

-uso.

[qkumba]

This post might be inappropriate. Click to display it.

[I am Rob]

This post might be inappropriate. Click to display it.

[I am Rob]

Post by Steve Nickolas

Post by I am Rob

Post by qkumba
The ProDOS version didn't need this change because it never prompts, thus the count remains always valid.

Has the Sherlock Interpreter already been converted to Prodos?

From what I understand, it has (by qkumba), but it hasn't been released
yet (except to 4am).

Post by I am Rob
And have there been any z5 games converted to a file yet to be used with
the Prodos version?

See above - there's some screenshots on 4am's Twitter.
-uso.

There are way too many interpreters.

Z3 - has up to an M interpreter
Z5 - up to version H
Z6 - unknown version, but handles dbl hi-res graphics and is Prodos compliant.

Although someone stated that a Z5-vH interpreter will play most if not all Z3 files, this Z6 interpreter look like the holy grail.

[Steve Nickolas]

Post by I am Rob
There are way too many interpreters.
Z3 - has up to an M interpreter
Z5 - up to version H
Z6 - unknown version, but handles dbl hi-res graphics and is Prodos compliant.

Z4 goes up to H too, and there's one or two versions of the Z3 interpreter
that predate A.

Post by I am Rob
Although someone stated that a Z5-vH interpreter will play most if not
all Z3 files, this Z6 interpreter look like the holy grail.

I don't think so? Far as I know, none of these are backward-compatible.

However, a few (at least Zork 1, H2G2, Leather Goddesses) of the Z3 games
were *upgraded* to Z5.

-uso.

[qkumba]

The interpreters are tied to the formats that they support. i.e. Z3 only does Z3, Z4 does Z4, Z5 does Z5, ...
However, within Z5 at least, there is apparently no universal interpreter.
For Z3 and Z4, we found one of each that handles all games in that format. For Z5, we found games that wouldn't run properly in one version but did in another, and vice versa, and none that play certain games properly at all.

[I am Rob]

Ouch! Would that also mean then that there are that many encoders (parsers?) to create the z-code? Or can one parser create the code that many different ways?

Looks like under Asimov IIGS adventures someone went to a lot of trouble to make a single compatible interpreter for all the Lost Treasures of Infocom.


Of the Z6 interpreters I see there are versions 9, 14, 16 and 17. That should mean there are at least 17 games for the Z6-interpreter, so far only 4 have been uploaded.

Another note is that the Z6 interpreters say they are Apple IIGS interpreters, but they have no IIGS or even 65c02 instructions in them and it uses Dbl Hi-res graphics. I hope this means it can be used and run on a 128kb IIe or IIc/IIc+.

So much to play with here. Curse you Steve for bringing this to my attention.

[Steve Nickolas]

Post by I am Rob
Ouch! Would that also mean then that there are that many encoders
(parsers?) to create the z-code? Or can one parser create the code that
many different ways?

They probably had separate compilers for each version.

More recent compilers like Inform can generate multiple versions of
Z-code.

Post by I am Rob
Looks like under Asimov IIGS adventures someone went to a lot of trouble
to make a single compatible interpreter for all the Lost Treasures of
Infocom.
Of the Z6 interpreters I see there are versions 9, 14, 16 and 17. That
should mean there are at least 17 games for the Z6-interpreter, so far
only 4 have been uploaded.

I think there's only 4, but there may have been multiple versions of each.

Post by I am Rob
Another note is that the Z6 interpreters say they are Apple IIGS
interpreters, but they have no IIGS or even 65c02 instructions in them
and it uses Dbl Hi-res graphics. I hope this means it can be used and
run on a 128kb IIe or IIc/IIc+.
So much to play with here. Curse you Steve for bringing this to my attention.

XD

-uso.

[Delfs]

So it sounds like the Eamon adventures, every one was customized for each story. That sort of makes sense.
If the one IIgs interpreter handles more than one then it most likely is the one to study and compare back to the original interpreters for those stories to find the common code.
Maybe you can find the core code and then the extra routines that are unique and just add those in - like MAME adds in support for new games. :)
Or not.

[Steve Nickolas]

Post by Delfs
So it sounds like the Eamon adventures, every one was customized for
each story. That sort of makes sense.

Especially true of version 6; not so much so of 3 (and the same
interpreter is used for multiple games, and multiple interpreters for the
same game, with the version 3 stuff).

Post by Delfs
If the one IIgs interpreter handles more than one then it most likely is
the one to study and compare back to the original interpreters for those
stories to find the common code.

Well, there's the rewritten ones (like jzip and frotz) too.

-uso.