SuperCard Pro Flux Image format (.scp)
#11
OK OK
I think I understand why it is confusing! This is probably coming from the terminology that I use. When I first described the protection I had to come with a name but it would have been better if I had called it High Frequency Input Signal – No Output Pulse Data Signal.

All explanation below are speculative based on my understanding of the protection but I might be wrong and would be very interested to hear from you when you receive Turrican…

Some basic reminders: in a floppy drive we have a read/write head that is used to perform “saturated recording” (in contrast to a tape recorder doing a “linear recording”). Recording is simple each time you send a signal on the write data channel the flux on the head is reversed. So if you turn on the write gate but do not send a signal you are actually erasing the track (unformatted area). Now on the read channel you read “transitions” (fluxes reversal) from the head with a preamplifier (with Automatic Gain Control) then this analog signal is converted to pulse with the pulse detector (usually LP filter + differentiator + zero crossing detector …). All this is very well explained in AN917 page 3-4 et 14 and AN413
       
What is of interest to us is that in the read channel electronic the output data pulse are shifted from the analog input and when signal are getting close the output pulse is shifted more (see AN413 page 11). If the input transition are close enough the data detector will end up by “shifting the data pulse outside of normal window” and this will result in NO OUTPUT pulse.

This is quite different from an unformatted area which we could call No Input Flux Transition – Random Output Pulse Data Signal. In that case we do not have any flux reversal under the head and therefore the input circuit works more like a tape recorder. The head pick-up some noise and the input circuitry pushes the ACG to its maximum. This result in the white noise coming from the input head to be turned into random output pulse.
Below is what I think is happening
   
So what would be interesting when you receive the Turrican game would be the following:
You place a probe either directly after the head or better after the input amplifier but before the integrator and you place a second probe after the pulse detector. The trigger signal should be the index pulse detector. Now you read track 7 of Turrican. I said track 7 because on this track the HFFA-NOFA area is just above the index (a bit before and a bit after) and see if the theory is right!
Reply
#12
I fully understood what you were talking about. My point is that deliberately writing super fast flux transitions makes no difference on the read data in my testing. Regardless of how fast the write pulse is, the read pulse does in fact complete the return to zero and does not float. However, this could be very well drive dependent. The ability to write weakbits is definitely drive dependent and some drives are not compatible for this function. It could be that the main drive I use either always reads the super fast pulses (which appear as weakbits that add up in total time correctly), or this drive can't write pulses that fast.

I will know a lot more when Turrican arrives.

Have you tried reading or writing in high density mode? This changes the characteristics of the drive.
Reply
#13
Questions / Feedback on scp format.
The feedback is based on experiment with the provided sample.

FILE HEADER
1) version/revision = currently 0.8?
Disk Type = only informative: on the sample I received I read 14???

2) TrackStart – TrackEnd info: are NOT used to read the following Track headers. In other word the region 0x010-0x02A7 is ALWAYS present independent of number of track sampled.
To save space and to allow more that 162 track its would have been possible to just have Track Header entries from TrackStart till TrackEnd?

3) Width in BITS of bit cell time? Why define in bits? If not 8 / 16 / 24 /32 … would require the data to be bit packed witch is a pain. (but it is not important at this stage as not really used)

4) Checksum: If I understand correctly we just need to add the bytes with a 32 bits integer ignoring overflow and the sum = the checksum provided? Question the specified range is 0x010-EOF: is this correct? With the described format it is possible to have “unused sections of bytes” in the file as all parts are given by pointers. In that case are the unused part also added in the checksum or is the checksum only for the used bytes (i.e. the bytes you read or write during decoding/encoding)?

5) track header pointer section: always 1have 62 pointers.

TRACK HEADER
6) Track number:??? Different format if single or double sided FD??? How do you know which one to use? Also I suppose this track number should be equal to the number of the track we are pointing at for verification purpose. So if in the File header you are pointing to the track 10 header you should find 10 in this field? In the experiment I have done it seems that this field is always 0 but I may be wrong?

7) Length of track: If I understand correctly this is the number of flux transitions recorded for a specific revolutions (data bytes = 2 * length)

8) Not mentioned in the doc but probably you should have a line that says:
BYTES 0x0C-0x0F contains the offset to get to the first flux transition data. The offset is from the Track Header?

9) If I understand correctly we ALWAYS have 5 times “index time”, “length”, “offset” (revolution information). Therefore even if number of revolution in FILE HEADER is one we still have 5 entries. Here again this is a waste of space but more important is that you will never be able to do more than 5 revolutions (probably not that important). Like what I said for the track header pointers the number could have been variable based on the number of revolutions?

10) flux data: I think you should FORCE (instead of recommend) that the data for the different revolutions must be adjacent. This is a MUST to be able to decode correctly DATA/ID segment bytes of a sector that span over the index pulse.

11) time stamp: little strange to place it here. It is hard to access because floating in middle of nowhere
Currently I read the file as follow:

Code:
Read file header
For 162 times read Track header
   If not null
      For 5 times read revolution information
         If not null
            For each revolution read data
Therefore unless I store the maximum upper byte read in all data segments I do not have idea where the time stamp is located!

More flexible and more optimized way for track / revolution would be:
Code:
For trackStart till trackEnd read Track Header
   For number of revolutions recorded read revolution information
      For each revolution read data
Read time stamp (would be located just after the array of track header)
Reply
#14
Quote:1) version/revision = currently 0.8?

The version/revision byte is the version/revision of SuperCard Pro software that created the disk image.


Quote:Disk Type = only informative: on the sample I received I read 14???

If you look at the latest image file structure info, you will see the Disk Type has changed. 0x14 is manufacturer of man_Atari (0x10) and disk type of disk_AtariSTSS (0x04). So, OR'd together that is 0x14. Actually, it should be 0x15 since both sides are being dumped, so I will change that.


Quote:3) Width in BITS of bit cell time? Why define in bits? If not 8 / 16 / 24 /32 … would require the data to be bit packed witch is a pain. (but it is not important at this stage as not really used)

Yes, it is a pain, but I left this option open for future use.


Quote:4) Checksum: If I understand correctly we just need to add the bytes with a 32 bits integer ignoring overflow and the sum = the checksum provided? Question the specified range is 0x010-EOF: is this correct? With the described format it is possible to have “unused sections of bytes” in the file as all parts are given by pointers. In that case are the unused part also added in the checksum or is the checksum only for the used bytes (i.e. the bytes you read or write during decoding/encoding)?

The entire file, starting from 0x10 through the very last byte of the file are added together, byte by byte. If the overflow occurs (past 16777216), then the checksum value wraps around to 0 and continues.

Here is the VB code used to calculate the checksum:

Code:
ChecksumTest()
    Z = 0                                           ' start with checksum = 0
    For X = IFF_THOFFSET To FileLength - 1
    Z = Z + FluxImageBuffer(X)                      ' get byte from flux image buffer
    Next X

    X = FluxImageBuffer(IFF_CHECKSUM + 3) * 65536 * 256&
    X = X + FluxImageBuffer(IFF_CHECKSUM + 2) * 65536
    X = X + FluxImageBuffer(IFF_CHECKSUM + 1) * 256&
    X = X + FluxImageBuffer(IFF_CHECKSUM + 0)
    
' compare checksum and quit with error message if they don't match!

    If Z <> X Then
        X = MsgBox("Flux image file checksum failure! Continue with viewing?", vbYesNo + vbDefaultButton2, "Checksum Failure!")
        If X = vbNo Then Exit Sub
    End If


Quote:5) track header pointer section: always 1have 62 pointers.

There are 165 pointers, not 162. 0x0010-0x2A4.


Quote:6) Track number:??? Different format if single or double sided FD??? How do you know which one to use? Also I suppose this track number should be equal to the number of the track we are pointing at for verification purpose. So if in the File header you are pointing to the track 10 header you should find 10 in this field? In the experiment I have done it seems that this field is always 0 but I may be wrong?

Track number should be the same as the pointer.


Quote:7) Length of track: If I understand correctly this is the number of flux transitions recorded for a specific revolutions (data bytes = 2 * length)

Correct! The length is the number of flux transitions * bit cell size. Default is 16 bits. So, if the track length was 0x1000 transitions and the bit cell size was 16 bits, the total length of data would be 0x1000 * 2 = 0x2000 bytes long.


Quote:8) Not mentioned in the doc but probably you should have a line that says:
BYTES 0x0C-0x0F contains the offset to get to the first flux transition data. The offset is from the Track Header?

Yes, that is correct. Please look at the latest specification (first post in this thread). I think it is more clear in the revised information.


Quote:9) If I understand correctly we ALWAYS have 5 times “index time”, “length”, “offset” (revolution information). Therefore even if number of revolution in FILE HEADER is one we still have 5 entries. Here again this is a waste of space but more important is that you will never be able to do more than 5 revolutions (probably not that important). Like what I said for the track header pointers the number could have been variable based on the number of revolutions?

Yes, there 5 sets of entries in the table. There is no waste of space because you could fill these in at a later date if the image is modified by writing new track data. The image format was designed for emulations, where you could change the data at any given time. That is why everything is offset based. We don't technically need any more than 2 revolutions in order to duplicate any type of disk. The extra space available is for the people that want "preservation" quality dumps.


Quote:10) flux data: I think you should FORCE (instead of recommend) that the data for the different revolutions must be adjacent. This is a MUST to be able to decode correctly DATA/ID segment bytes of a sector that span over the index pulse.

It doesn't matter where the data is placed in the file. The offsets are there to join everything. With this file structure, I can add new revolutions to the end of the file and just poke in the new track length, index-index time, and offset. Right now, the imager places all revolutions consecutively and the pointers point to the start of each revolution. But the data could be located anywhere so you need to be using the offsets when fetching the data from the image file.


Quote:11) time stamp: little strange to place it here. It is hard to access because floating in middle of nowhere
Currently I read the file as follow:

You are making it too difficult! The last 20 bytes of the file are always the timestamp. Just do a Seek(LOF-20).. that is always the location of the timestamp! If you are loading the entire file into memory, just use: ImageStart+ImageLength-20. If the image file changes, the timestamp will be erased, and the new data will started where the time stamp was located, and a new time stamp will be added to the end of the new data. This way, the last 20 bytes will always be the timestamp.
Reply
#15
(12-22-2013, 08:55 AM)admin Wrote:
Quote:10) flux data: I think you should FORCE (instead of recommend) that the data for the different revolutions must be adjacent. This is a MUST to be able to decode correctly DATA/ID segment bytes of a sector that span over the index pulse.

It doesn't matter where the data is placed in the file. The offsets are there to join everything. With this file structure, I can add new revolutions to the end of the file and just poke in the new track length, index-index time, and offset. Right now, the imager places all revolutions consecutively and the pointers point to the start of each revolution. But the data could be located anywhere so you need to be using the offsets when fetching the data from the image file.

Thanks for the replies. It's all clear.

The only thing that I do not like is this capability of having flux transitions from one revolution separated from the next one. But this might be a special requirement from the way I decode the flux. Currently when I am in a middle of decoding a sector I do not have to worry to know if the transitions belong to one revolution or the next as I have them all adjacent. But I can copy the data from your file into a large buffer. So this is OK.

One last thing were I am not clear is how to find out the position of the indew relative to the transitions. As i mentioned this is usually not a problem unless a huge transition is placed over the index like in Turrican sector 8?
Reply
#16
Technically, you could check bit 3 (TYPE) of byte 0x08 (FLAGS). If the bit is clear, the image is a read-only image and you would be guaranteed that the revolutions are stored consecutive in the image and will never change. I will make this point in the documentation. Yes, copying the flux data to another buffer, placing the revolutions end to end would also work.

The index to index time is given as part of each of the 5 entries (along with total number of bit cells for the revolution, and the offset from the TDH). The bit cell times added together will equal the index to index time. Until I see Turrican, I can not believe yet that this will ever not be the case. Like I mentioned before, this could be drive dependent. It could be that some types of 3.5" floppy drives do not return read data during this 'no flux areas', while others do. As soon as Turrican arrives, I will know for sure and I will test a variety of drives. What other games use the same protection as Turrican?

Do you know of the program Crown Of Creation 3D?
Reply
#17
On the latest doc

number of tracks:
Code:
BYTES 0x06 and 0x07 are the start track and end track bytes.  Tracks are numbered 0-163.
This is 164 tracks or 82 double sided
But SCP program goes up to 83x2=166 ???

And in SCP IMAGE FORMAT
Code:
; 0006              START TRACK (0-165)
; 0007              END TRACK (0-165)
and later
Code:
IFF_START = 0x06                   ; start track (0-165)
IFF_END = 0x07                     ; end track (0-165)

What is the truth?

in text description
Code:
BYTES 0x08-0x0B contain the length of the track in bit cells (flux transitions).  If a
16 bit width is used (selected above), then this value *2 will be the total number of
bytes of data that is used for that track (16 bit = 2 bytes per word).

There are 5 entries (starting at BYTES 0x08-0x0B), one for each revolution (up to
5 revolutions) contained in a table that follows the track number.  If an entry contains
all zeros (0x00000000) then no more data is available for that track.

You are missing
BYTES 0x0C-0x0F Data Offset

described later in
Code:
; 000C              DATA OFFSET (1st REVOLUTION)    - 4 bytes (offset is from start of Track Data Header - should always be 0x00000044 for this entry)
Reply
#18
Thanks for pointing this out... I will make the corrections.

There are 164 entries in the table, starting at 0x0010 with the last entry at 0x02A0. 0x2A4 is reserved and 0x02A8 is the start of the first TDH (Track Data Header). So, it should be 0-163.
Reply
#19
OK, the specs file has been updated. Please download the new version. There were several corrections made.

Please also note that there is a new definition for a read/write type image. If a disk is image is read/write capable, then the flux space allocated for the revolution (only 1) is defined in long word at 0x40 in the TDH. 0x40 was defined as a checksum, but that was not actually be used. So, it has been re-purposed for this. The flux space will typically be 125,000 to represent 125,000 bit cell entries available. The actual flux data is contained in this space (starting at the beginning). The unused space will be 0's. So, if the number of bit cells for a track was 72,000, then the remaining space afterwards (53000 entries) would all be 0's for each bit cell entry.

I also made it clear that all revolutions are stored sequentially together and can not be broken up. This eliminates your concerns about data having to be copied in chunks located at various offsets.
Reply
#20
So does the scp format identify where the index pulse is in relation to the data for a non-index read image? I couldn't see a way to determine that from the spec. The only thing it shows is time from index to index, but that's unrelated to data location.

But, if you're storing 2 revolutions of data into the image file, wouldn't it just make sense to always start at the index anyways? You'll still get all the data, and the index location is implied as the start of the track data in the image. The challenge would then be to determine the actual gap location in the image, look at it's location relative to the index (which is now known as the start of track data in image) and calculate the time between them. Then when writing, wait for index, then wait X nanosec before writing out the gap-to-gap portion of the track data.
Reply




Users browsing this thread: 4 Guest(s)