Post by Greg SmithI apologize for not responding sooner. I saw your original message
after I returned from a vacation and was thinking I replied to it but
guess I just thought about it and decided that too much time had passed
or something. Lets call it summertime fever :)
The amount I pay you guys for support, and this is what I get... :-)
Post by Greg SmithAn unexpected power-off can be devastating to a cckd file. I'm not a
hardware expert but I understand that during power-off while a write is
in progress to a host disk that garbage can be written to the disk
because the disk could be reading bits/bytes from the bus but the CPU is
down.
That is/was actually architected that way on S/360 and 370, and I'm
sure there's room for similar trouble on any kind of drive, but I
would guess that pretty much any power supply will keep drive and CPU
going for a few milliseconds. But certainly it's not a highly
predictable state of affairs.
Post by Greg SmithI've implemented a number of solutions to try to mitigate the problem.
When I first started debugging this problem I kept thinking "this CCKD
and shadow file stuff is really neat, but it obviously wasn't designed
by someone with a background in reliable computing". Then as I
discovered each of the mitigating schemes, I got more and more
impressed, and eventually I completely took back what I had thought at
the beginning.
Post by Greg SmithOne is lazy-write. All outstanding writes are `flushed' at each
`garbage collection' interval which, by default, is every 10 seconds.
I'm still not sure if FSYNC must be specified to make this happen.
Post by Greg SmithFifth, there is a fifth meta-data structure and that is free space.
This, I think, didn't make it into the html doc. The code talks about
"new format" free space.
Post by Greg Smithcckdcdsk -4 is interesting because it tries to recover track images
without any help from meta-data at all.
I started to look hard at -4 because it became clear that my metadata
was unreliable. In particular, several pointers were to valid looking
track data, but for the wrong track. Then I realized that even -4
wasn't recovering those valid-looking track images, and I assumed they
were corrupt as a result of the power failure (as you suggested
above). So I modified the standalone cckddiag program to allow me to
specify an offset and length in the file to try to decompress and
display. (Actually this was trivial, because there is already an
offset/length option which will display the raw data, so it was just a
matter of adding an option to try to uncompress, and format the
results.) In every case I looked at, cckddiag uncompressed and
displayed good looking data. As you know, cckddiag is modeled on
cckdcdsk, so I had to try to figure out why one worked and the other
failed on the same data, and nearly identical calls. A mess of
debugging statements later, it was clear that the compressed data was
identical, and the lengths being passed to uncompress() were
identical, so how could there possible be any difference? (Well,
actually, as a colleague pointed out, the length in cckddiag is
probably too big by CKDDASD_TRKHDR_SIZE, but cckddiag is the one that
was working!) It also turns out that ZLIB's uncompress() is pretty
forgiving about its input length; it generally uncompresses as best it
can, and doesn't complain if the input length is too high, and even if
it's too low, it still returns what it can, and complains with a
Z_DATA_ERROR rather than the Z_BUF_ERROR I was getting. I read some of
the ZLIB code, and there are some more subtle reasons for a
Z_BUF_ERROR that are not as simple as "your output buffer is too
small", but since the input data was identical, I couldn't see any of
those being the problem. So really cckddiag was the clue; it uses a
size_t for its output buffer length, vs the int that cckdcdsk uses.
That resolved the problem, and only later I realized that cckddasd.c
uses an unsigned long, which is probably correct.
I don't know anything about how library parameters and arguments are
resolved in a mixed mode environment like Linux on I-64. I suppose the
call by value ones pretty much look after themselves, and probably it
was just that (void *) that allowed this one to sneak by. The
uncompress() routine in ZLIB itself declares the parameter as "uLong",
which presumably is the same as "unsigned long".
Sigh - my c programming days are mostly far behind me...
Post by Greg SmithBut you know all this already, and I am super impressed ;-)
I just read the code; you designed and wrote it.
Post by Greg SmithI am also impressed by you going the extra mile.
I really wanted that data back, and more important, to know *why* I
couldn't have it.
Post by Greg SmithIf you can give me a `diff -u cckdutil.c cckdutil.c.fixed' I will try to
make the necessary code changes to properly get rid of all the warnings
etc and will give you full credit for the fix if you wish.
It's a one-word change from "int" to, I guess, "unsigned long".
Probably the cast to void * should be removed to let the compiler warn
about similar problems in future. All the debugging prints I added are
just clutter, but I think it would be very nice to have a summary of
what was recovered and how, and what was tried and failed. Controlled
by yet another option, of course. :-) Maybe I will try to add that.
I will clean up and send you the changes to cckddiag; I think those
can be quite handy when working at this level of diagnosis. That
module is flagged "James M. Morrison 2003", which is a name I
remember, but can't quite place.
Post by Greg SmithThe `subtle' comment above bothers me a bit. If you can critique the
code a bit to make it less subtle, it would be appreciated. Don't worry
about hurting my feelings, I write so much code that code I wrote a year
or two ago is as foreign to me as code written by someone else.
I meant subtle in the best possible way! The thing that stuck me the
longest was where it scans forward for the *next* potential track
header, and then uses the difference between the current one and that
to try to decompress. Only after trying each possible difference does
it go back for one last-ditch try with every length starting at 13 and
working up to the track size. Since the uncompress() was failing, it
always fell through into trying all possible lengths, and that threw
me. I did add a few block comments at the front of this area as I
worked through it.
I believe this is a 64-bit problem only, and even that may depend on
the architecture and OS/library. But I'm also not sure that it won't
occur with any level of recovery, in other words it is not necessary
to specify -4 to cause failure to recover data that is there. Why it
ever works in this environment is a bit of a mystery. I think it
depends on the content of the word following the int on the stack, and
I don't want to bother thinking through all the little-endian evilness
that might allow that to sometimes work.
Tony H.