DokuWiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== HiMDDiskFormat ======

HiMD-Media are read and written in Mass-Storage mode. So one only needs to know the on-disk format to be able to read/write from Linux. The disks are formatted with a standard FAT16-filesystem but with a 2kb-sectorsize. Once the disk is attached to a PC it can be used as a normal mass-storage device (format is superfloppy, i.e. no partitions). When SonicStage (or HiMD Music Transfer on the Mac) is started, the minidisc is unmounted by SonicStage and thus inaccessible by the operating system. This is necessary because SonicStage uses special SCSI read/write-commands to communicate with the MD-Walkman. This is probably necessary for a secured data-exchange between Walkman and the host PC.


===== Disk-Images of various MiniDiscs for testing =====
  * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_with_one_mp3.img.gz|md_with_one_mp3.img.gz]] - Image of a 305MB HiMD containing one MP3 - Disk-ID: 0202000002000102A36E0000CAEF24F1 - created with Sony MZ-RH10 and HiMD Music Transfer for Mac
  * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_empty.img.gz|md_empty.img.gz]] - Image of a freshly formatted 305MB HiMD - Disk-ID: 0202000002000102A36E000040E633F2 - created with Sony MZ-RH10 (built-in format functionality)
  * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/himd_with_dozens_of_mp3s.img.gz|himd_with_dozens_of_mp3s.img.gz]] - Image of 1GB HiMD containing dozens of MP3s - Disk-ID: 02000000000006121202000000718002 - created with Sony MZ-RH10 and SonicStage 4.3/4 Japanese
  * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/mdw60.img.gz|mdw60.img.gz]] - Image of 220MB MD (MD-W60) containing one PCM-recording - Disk ID: 020200000200010228310000779D3C0A - Recorded with a Sony MZ-NH600 via Toslink, recording is untitled
  * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_with_one_pcm.img.gz|md_with_one_pcm.img.gz]] - Image of MD containing one PCM-recording - Disk ID: 0202000002000121B40E0000F9EFA991 - Recorded with a Sony MZ-RH1 from **input** (thus with suspected less encryption), recording is untitled
  * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_with_atrac3.img.gz|md_with_atrac3.img.gz]] - Image of MD containing three ATRAC3-recordings with 105 kbit/s recorded with MZ-RH1 - Disc ID: 0202000002000121B40E0000B3AFA5B0
  * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/mixedmode_md.img.gz|mixedmode_md.img.gz]] - Image of MD containing 4 Tracks, PCM/ATRAC3/MP3 recorded with MZ-RH1 - Disc ID: 0202000002000121B40E0000F9EFA991
  * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_with_one_pcm_nh700.img.gz|md_with_one_pcm_nh700.img.gz]] - one PCM-recording transfered with SonicStage onto a MZ-NH700
  * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_with_one_pcm_rh1.img.gz|md_with_one_pcm_rh1.img.gz]] - one PCM-recording transfered with SonicStage onto a MZ-RH1 (same PCM as with NH700)

===== The DiskID =====

The disk-id is the unique identifier of each HiMD and is read using special SCSI commands. A small program-code to fetch the disk-id can be found here:

  * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/minikey.c]] - Read the disc-ID of a MiniDisc in Linux
  * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/mp3key.c]] - Generate MP3-encryption key from DiscID

The disk-ID is used to encrypt the (audio-)data on the HiMD. The latest versions, however, can be found in the git-repository.

===== Filesystem Issues with some Models on Linux =====

It has been observed, that some older HiMD-Walkman (MZ-NH600 and MZ-NH900) cause some trouble in Linux (but not in Windows) when listing the contents of "HMDHIFI"-directory:

<code>
root@z6:/mnt/hmdhifi> ls
ls: cannot access n/n╒ì╬∩_._▐]: No such file or directory
ls: cannot access ▌▀eu╒╟╜«.eΘ«: Input/output error
ls: cannot access ╡m ┘╟_╜.■√╓: Input/output error
ls: cannot access ·jl_φ╠ _.▐₧▐: Input/output error
ls: cannot access  ▀m▀╒╩²«.oφ¡: Input/output error
ls: cannot access ■_▀öª▀_┐.≈√╒: Input/output error
ls: cannot access ■\▀┐▐»^.²²ƒ: Input/output error
ls: cannot access j/no¡╬δ_.■▀φ: No such file or directory
(...)
?             ²_▀¥∩δ ┐.÷√▐  ª╜²⌡o▌≈▀.┐nw  ª▌╜ {▌⌡▀.▌mg  jont▌╬╧_.▐₧ì  n»j]φ∞δu.▀▀∞  ^on▌═nδ_.▀₧φ  ■\▀▀▐»z?.²φ▀  √ƒ╒▌╧▌ⁿ∩.εº▌  ⁿ ▀╣≈█_┐.≈█▀  ▐\ █▀»z .²∙╫
?             «∙╜²}▌µ┘.▀εe  ª╗╜²{▌≈▀.┐εe  ▀▀e▀_╧▀º.nφ╝  k/l]¡─√_.▐ƒ▌  n/lu¡╧√_.▀╒═  ▀▐ou╓ñ ».nΘ¼  z/nu═╠δ_.▐╛▌  εoj]═fδ].▀ƒ▄  ≈\ █▐½^{.² ƒ  ⌡▀ ╣≈δ]┐.▀√▌
00010012.hma  ²ε▀╜╓δ_».■[₧  ª╗╜²{▀σ².¥┼g  ▌▀eu╒╟╜«.eΘ«  «¥╜⌡k▌÷▌.┐φe  n-n╒═╬█^.▐ƒ╪  ▀▐ouuº º.~¡«  ⌡\┐▀▀»z?.φδ▀  εonu▌╬√?.▀▀╛   ▀╡▌╧╒²∩.∞╖┼   ┐▌∙╧]╝δ.Ωº▌
_0010012.hma  «█∙²?¥╖▐.╜∩   ª╗∙²{▌σ².ƒεu  ▀▀e]╫╟═».∩Θ╜  √█╜∙╧lⁿ⌐.l÷═  n»n▌∞╬√_.▐₧▌  ╜█o]u╧ ª.oφ«  ▐\▐▀▀»z .u²ü  εo ▌╧╦_º.µ√▀   ▀euu╟²«.jφ¼  ■_ ¥τk_╖.µ√▐
■\┐▌▐½z?.²√ù  ²▀ ╣▀╦█».■√₧  ª╜╜²}▌≈ .▀∩g  ⁿ_▀¥g╦_┐.■√▀  l/l┼ì╬δ].▀ƒ\  n/n╒ì╬∩_._▐]  ⁿ]▀▌»▀_┐.t√▐  ■\┐▀▌»z .⌡ ƒ  ε╬▀▌τ╦[».■√▀   ▀m▀╒╩²«.oφ¡  ∞[ ▌τδ ».⌠√▀
■\▀┐▀½z?. ▀┐  ■² ╣∩δ ┐.⌡√▀  ª╗▌¡k╜╖▌.▀εg  ·jl_φ╠ _.▐₧▐  l»ne¡e█u.▐₧▄  n/n╟¡l√_.▀ƒ]  ■t▐█▀∩^?.²²ƒ  ⌡∩▀▌╟δ_┐.≈∙▐  ╝_▀▌τ╦^┐.ε·╓  ■_ ö∩╦]┐.≈√▌  ⁿ▐ ▌τ╦_┐.σ√╓
■\▀▀▐½z .⌡ ƒ  ▐\■▀▐»^². φƒ  ª√∙╜o¥╡▌.ƒ∩w  j/n─¡╧∩_.▐▀▌  ~»lw═╠ u.▀ƒ╩  n/nu═n█_.▀ƒ╧  trkidx01.hma  δnnu¡dδ_.\┐■  ∩_▀¥τ ]┐.÷√╫  ⁿ▀ ö╟╦]╡.÷[▐   ┐▌∙╧▌ⁿ».Ωτ²
ⁿ²▀▌∩╦_╖.╒√▀  ²_ ╣ ╦_┐. √▀  ª█╣⌡o¥≈█.┐■τ  j»n┼═╠╧].╘ƒ▌  mclist01.hma  ▀█n]t╧²«.εφ╜  trkidx02.hma  ÷_▀¥╧δ_º.÷{▐  &╜╜φk¥⌡▌.ƒ∩g  ⁿ╬ ╜╟╦_º.µ{▐  ⌡] ¥▀√_┐. █▀
■\▀┐▐»^?.²²ƒ  √█⌡┘╧▌╜».·ª▌  atdata02.hma  j∩n▌¡╠√_.▐ƒ▌  mclist02.hma  ^/n╫═╠δ]._▀²  ÷\■▀▀»┌?.⌡▀ù  ⁿ]█┘╞δ_╜.⌠y▐  φ∩n╫╜╬▀_.▐┐▌  «¥╜ o▌º▀.²φg    ╜┘å╒■Θ.ε╡▌
█▀╜²╧e²φ.∞º═  ª╣²²m▌÷▌.▌d⌡  ª█⌡┐}▌σ▀.²∩σ  j/nm╠─Ω_.\▐î  _mdhifi.hma   ∩»n]¡ε√_.▀ü▌  ■\₧▀▐∩z}.}²ƒ  δφn╫¡╞√_.▐┐▌  √▀¡∙═╘∞».Ω╖═  ■ ▀╜ùδ_».╓[▐  ■_ ▌ δ▀╜.≈√▀
²o ╜≈√_».⌡[▀  ª╣²∩m▌ñ².▀σe  ª┐╜╜?▌≈▀.▌∩τ  j/no¡╬δ_.■▀φ  ╡m ┘╟_?╜.■√╓  ■_▀öª▀_┐.≈√╒  ÷\┐█▀»z².²√ƒ  ■}▀¥╟╦_».ε{▐  Ω»n▄═╬δ].▐ƒm  ■| ╧▐»z?.⌡√ƒ  ■\  ▀»┌?.≈ ╖
²o ▌τ█_».ε{₧  ª┐┘²o▌÷².█∩e  ª√╜»?▌τ².╜∩σ  j/n▄═╠φ].▐₧┌  ╬-n┼╠╠█_.▐▐═  ▀▀o]■╟┐ª.oφ╣  ÷\┐█▀»z?.⌡²ù  ε/n╓═╬δ].▐₧▄  ▐\ƒ╗▀»^?. √ƒ  ÷\ ▀▀∩z?.} ┐
root@z6:/mnt/hmdhifi>
</code>

However the important files can still be accessed properly. The kernel suggests that something is wrong with the FAT:

<code>
[1213211.340236] FAT: Filesystem panic (dev loop0)
[1213211.340244]     invalid access to FAT (entry 0x0000ffe5)
[1213211.340248]     File system has been set read-only
[1213211.340579] FAT: Filesystem panic (dev loop0)
[1213211.340585]     invalid access to FAT (entry 0x0000ffe7)
[1213211.340810] FAT: Filesystem panic (dev loop0)
[1213211.340810]     invalid access to FAT (entry 0x00005fde)
[1213211.340810] FAT: Filesystem panic (dev loop0)
[1213211.340810]     invalid access to FAT (entry 0x0000fbef)
[1213211.340810] FAT: Filesystem panic (dev loop0)
</code>

and

<code>
root@z6:~> fsck.vfat /amd/z6/0/mdw60.img 
dosfsck 3.0.1, 23 Nov 2008, FAT32, LFN
/HMDHIFI
  Has a large number of bad entries. (245/268)
Drop directory ? (y/n) 
</code>

It has not yet been confirmed what causes these errors. All we know is that they apparently only occur with older models and that the filesystems list fine in Windows. Probably some undocument FAT-features ?!?

===== Files =====

The following files can be found on a freshly formatted HiMD (MiniDisc mounted to /mnt) in the directory "HMDHIFI". The root-directory of an HiMD contains this folder only and a zero-byte file called "HI-MD.IND" which presumably is only meant to be an identifier for a HiMD-disc.

<code>
root@znote2:/mnt/hmdhifi> md5sum /mnt3/hmdhifi/*
64c228b7c4c1b02b6c2f1c11bc4148e6  /mnt3/hmdhifi/00010012.hma <---- EKB (key container) for standard HiMD keys
5363c96604a0db55e0fafc787385e36b  /mnt3/hmdhifi/_0010012.hma <---- Backup of EKB
d41d8cd98f00b204e9800998ecf8427e  /mnt3/hmdhifi/atdata01.hma <---- Containerfile containing all audio data
bb7df04e1b0a2570657527a7e108ae23  /mnt3/hmdhifi/_clist01.hma <---- Backup of Keyfile for Digital Rights Management
28147b60982b017008e75594ab2bdf00  /mnt3/hmdhifi/mclist01.hma <---- Keyfile for Digital Rights Management (unconfirmed)
bb7df04e1b0a2570657527a7e108ae23  /mnt3/hmdhifi/_mdhifi.hma  <---- Backup of the directory information of this directory
1aca77e2188f52a62674fe8a873bdaba  /mnt3/hmdhifi/_rkidx01.hma <---- Backup of last Track Index File
09b5367f524ecdbfcae6af4b3173d3db  /mnt3/hmdhifi/trkidx01.hma <---- Track Index File
</code>

The following files are found on an HiMD containing several dozen MP3-tracks:

<code>root@burns:/mnt/hmdhifi> md5sum *
64c228b7c4c1b02b6c2f1c11bc4148e6  00010012.hma
5363c96604a0db55e0fafc787385e36b  _0010012.hma
d6c8df08ff525dd3be7a90978acb39c7  atdata02.hma
bb7df04e1b0a2570657527a7e108ae23  mclist01.hma
1fcd128c53d2bd58fd50c666b2fc6475  mclist02.hma
fafb8cbcdc3cfc70972558b6e458fd30  _mdhifi.hma
4b0f6c1dd7e893026295d4b9c3c5a8be  _rkidx03.hma
6b3e2b2c54b25b752fa831ba96a54615  text_g01.hma
cdf55e77362fb9a510207f8bb73fb826  thumbg01.hma
370af574809d15d7eaea86030200b020  trkidx02.hma
root@burns:/mnt/hmdhifi> hexedit mclist01.hma 
</code>


<code>
glaubitz@z6:..sdf/HMDHIFI> ls -l
total 551904
-rw------- 1 glaubitz ag-wolf       172 2004-01-01 00:00 00010012.HMA
-rw------- 1 glaubitz ag-wolf       172 2004-01-01 00:00 _0010012.HMA
-rw------- 1 glaubitz ag-wolf 562167808 2008-09-20 18:31 ATDATA02.HMA
-rw------- 1 glaubitz ag-wolf     32768 2004-01-01 00:00 MCLIST01.HMA
-rw------- 1 glaubitz ag-wolf     32768 2004-01-01 00:00 MCLIST02.HMA
-rw------- 1 glaubitz ag-wolf     32768 2004-01-01 00:00 _MDHIFI.HMA
-rw------- 1 glaubitz ag-wolf    327680 2008-09-20 18:32 _RKIDX03.HMA
-rw------- 1 glaubitz ag-wolf     65536 2008-09-16 18:33 TEXT_G01.HMA
-rw------- 1 glaubitz ag-wolf   2097152 2008-09-20 18:12 THUMBG01.HMA
-rw------- 1 glaubitz ag-wolf    327680 2008-09-20 18:32 TRKIDX02.HMA
glaubitz@z6:..sdf/HMDHIFI> 
</code>

The md5sums for all files are always the same after a fresh format except for the file "mclist01.hma" whose md5sum changes on every disk-write and is always different, even for the same minidisc used.

The file atdata01.hma is the containerfile which keeps all audio-data (this file grows once new tracks are loaded onto the disk). trkidx01.hma appears to be an index-file for the walkman.

The numeric appendices are increased from time to time. Obviously, simply adding one mp3-track with the transfer-software for Mac does not increase the numbers. It might be only increased when adding ATRAC-tracks or when adding more than one file. The files with the highest index are the only to contain valid data. When several trkidx0X.hma can be found for example, only the file with the highest index contains non-zero content (the other are zero-filled, but not zero-size). 

==== The file mclist0X.hma ====

The filename "mclist01.hma" obviuosly derives from "MAClist" which is the name of one of the modules in OpenMG. MAC here stands for "Message authentication code" (see [[http://en.wikipedia.org/wiki/Message_authentication_code]]). In this case, the DRM license info of each track is considered a message, and the authenticity of these messages is confirmed by adding a 8-byte field to the track information. The MAC list contains all the MAC values of the different tracks and a master MAC that ties these together into one bundle.
Furthermore, the MAC list contains a header that ties the MAC list to a specific "generation" of a specific medium. The generation is increased every time the DRM info changes.
The final authentication is done by combining checksums of the medium-specific and the track-specific parts into a master "integrity check value" (ICV) which is stored outside of the FAT file system and only accessible using authenticated and encrypted SCSI commands.

The contents of "mclist01.hma" are mostly zero until DRM protected tracks are uploaded, the non-zero part looks like this:

<code>
0000000: 4d4c 5354 0000 0124 0000 0000 0000 0000  MLST...$........
0000010: 32af 72b6 0a4f 1612 e490 ce90 90b2 07ec  2.r..O..........
0000020: 0000 0001 0000 0000 0000 0000 0000 0000  ................
0000030: 0000 0000 0000 0000 0001 0012 0000 0000  ................
0000040: 0202 0000 0200 0102 a36e 0000 caef 24f1  .........n....$.
0000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000060: fdd5 d64b 046f 53cf 4853 312d 8091 82ab  ...K.oS.HS1-....
</code>

The MAC list file is divided into three parts. The first 16 bytes are the file header, the next 80 bytes are for disc authentication and the remaining part is for track authentication. Both authentication parts start with an encrypted 3DES key used for authentication, followed by the data to authenticate. To decrypt the authentication key, a master key needs to be used. In the Sony DRM system, these master keys are stored in EKBs, in encrypted form. The disc authentication header contains an indication which EKB (that also means which master key) to use.

<code>
0000 BYTES  magic signature "MLST"
0004 DWORD  unknown purpose
0008 BYTES  8 unknown bytes, always zero, maybe padding.

0010 BYTES  16 bytes encrypted 3DES key for authenticating the first MAC list part
0020 DWORD  Generation number of the DRM info
0024 BYTES  20 unknown bytes, always zero, maybe padding
0038 DWORD  ID of the EKB used for decrypting the authentication keys
003C DWORD  4 unknown bytes, always zero, maybe padding
0040 BYTES  16 bytes disc ID. This is a copy of the real disc ID stored outside of the file system
0050 BYTES  16 unknown bytes, always zero, maybe padding

0060 BYTES  16 bytes encrypted 3DES key for authenticating the second MAC list part
0070 BYTES  4000 * 8 bytes MAC values of tracks. Intersting count, as Hi-MD only has 2048 tracks.
</code>


==== The file 00010012.hma ====

This file contains the "EKB", the enabling key block #00010012. This is the standard key block for HiMD audio. It's the string "EKB" followed by 29 null bytes (making a 32 byte header), and then the contents of the EKB that is also provided in OpenMG as 00010012-umd.EKB. It should never change, but if encrypted payware audio with other EKBs are uploaded, further all-numeric .hma files might appear.

==== The Audio Data File ====

The audio data file contains the raw audio data.

It is divided into 16k blocks of the following layout

<code>
0000 DWORD  Block Type ("LPCM" for PCM,"A3D " for ATRAC3, "ATX " for ATRAC3+, "SMPA" for MPEG)
0004 WORD   Number of frames (MP3 blocks only, PCM/Atrac blocks have fixed size)
0006 WORD   "MCode"
0008 WORD   Data size in bytes (MP3 blocks only)
000A WORD   Reserved
000C DWORD  Serial number of block in stream
0010 BYTES  PCM/Atrac: Encrypted DES key for the audio data
0018 BYTES  PCM/Atrac: DES CBC initialization vector for the audio data

0020 BYTES  up to 3FC0 of encrypted audio data (ATRAC never uses all 3FC0 bytes)

3FE0 BYTES  Backup of encrypted key (offset 0010)
3FE8 BYTES  8 bytes reserved
3FF0 DWORD  backup of Block Type
3FF4 WORD   reserved
3FF6 WORD   backup of "MCode"
3FF8 DWORD  low order 32 bits of the Content ID
3FFC DWORD  backup of serial number
</code>

In case of MP3 audio, the data is XOR encrypted. The key for encryption is created from the DiscID which read/written using special SCSI commands. In case of ATRAC/PCM audio, the data is DES CBC encrypted. See below for more info on encryption.
Each audio block contains only complete frames. The space in the audio block after the last complete frame is unused - which is especially wasteful for 352kbit/s ATRAC3+, as the frame size is exactly 2K, so 1984 bytes per frame are ignored, i.e. 12% of the block size.

==== The _MDHIFI.HMA File ====

This file is obviously a backup-file for the directory content:

<code>
root@znote2:..disk/HMDHIFI> strings _MDHIFI.HMA 
.          
!0!0
..         
!0!0
TRKIDX01HMA 
!0!0
ATDATA01HMA 
!0!0
MCLIST01HMA 
!0!0
00010012HMA 
!0!0
_MDHIFI HMA 
!0!0
_RKIDX01HMA 
!0!0
_CLIST01HMA 
!0!0
_0010012HMA 
!0!0
root@znote2:..disk/HMDHIFI> 
</code>

==== The Track-Index File ====

The Track Index File (or TIF for short, trkidxXX.hma) contains a TOC like data structure telling about differnt groups/tracks/parts of the music, which is contained in the audio data file (atdataXX.hma). The size of the index file is fixed 320K; it has the following structure (derived from symbol infos in the MAC software)

<code>
00000..00100 Header (256 Bytes)
00100..01100 Play order table (4K)
01100..02100 Programmed play order table (4K)
02100..02900 Group info table (2K)
02900..08000 Padding
08000..30000 Track info table (160K)
30000..40000 Part info table (64K)
40000..50000 Name table (64K)
</code>

=== The Header ===
<code>
0000 DWORD  Magic number to identify the Track index file "TIF " (54 49 46 20)
0004 WORD   Version number (seen as 01 00, i.e. 1 in little endian)
0006 WORD   "MCode" ??? (seen as 01 24)
0008 DWORD  reserved
000C DWORD  revision (seen as 00 00 00 01, i.e. 1 in big endian)
0010 DWORD  "GrpThmRevision"
0014 DWORD  "TrkThmRevision"
0018 DWORD  "GrpTxtRevision"
001C DWORD  "TrkTxtRevision"
0020..0100  reserved
</code>

All quoted field names have been directly taken from RTTI in the MacOS software. All revision fields were observed as zero.

As long as no endianess is specified, big endian is used.

=== The Play Order Table ===
Is an array of 2048 16 bit words. The first word is the number of tracks on the MD, the next words is the order in which the tracks should be played. A one track MD has the first and second word 1.

=== The Programmed Play Order Table ===
Is also an array of 2048 16 bit words. It is completely empty on a simple one-track MD.

=== The Group Info Table ===
Contains 256 group descriptors of the following layout
<code>
0000 WORD From
0002 WORD To
0004 WORD Name (string index)
0006 WORD Fringe
</code>

The group table on the one-track MD is completly empty (everything zero)

=== The Track Info Table ===
Contains 2048 track descriptors of the following layout
<code>
0000 DWORD   - date of recording (FAT format, 16 bit date, 16 bit time of day)
0004 DWORD   ! "EkbVersion" (Mac MP3: 0, Mac WAV: 10012; upload requirement: == 10012)
0008 WORD    - Title (string number)
000A WORD    - Artist (string number)
000C WORD    - Album (string number)
000E BYTE    - Track number (within Album, not on MD)
000F BYTE    - "Mode"
0010 8 BYTES ! MgrCK (upload requirement: completely zero)
0018 8 BYTES   CMac
0020 BYTE    - CodecId (see below)
0021 3 BYTES - Codec specific info
0024 WORD    - Part Number (index into Parts Info Table)
0026 WORD    - Track Number
0028 WORD    + Total time (units of seconds)
002A BYTE    + "Lt" (Mac MP3: 10, Mac WAV: 1; upload requirement: == 1)
002B BYTE    + "Dest" (upload requirement: == 1)
002C WORD    + More codec specific info
002E WORD    + reserved
0030 20 BYTE + Content ID (for Mac Transferred Data: 02 03 00 00 + 16 random bytes)
0044 DWORD   + Start of playback license validity (FAT format, or 0 for no restriction)
0048 DWORD   + End of playback license validity (FAT format, or 0 for no restriction)
004C BYTE    + "Xcc" (Mac MP3/WAV: 01; upload requirement: == 03 || == 07)
004D BYTE    + "Ct" 
004E BYTE    + "Cc" (Mac MP3: 40, Mac WAV: 44; upload requirement: == 08 || == 48)
004F BYTE    + "Cn" (Mac MP3: 00, Mac WAV: 3)
</code>

The +/-/! means: "+" this field is included in MAC calculation. "-" this field does not influence the MAC. "!" this field controls the MAC calculation

"upload requirement" means that the HiMD Transfer Tool for Mac checks the given condition before allowing an upload of a track to the PC. These checks are independent from the track format. "Mac MP3"/"Mac WAV" means that these values are set by the HiMD transfer tool if downloading that type of music. Probably the fields annotated that way are involved in copy controlling. The first entry in the track info table is a dummy entry that contains the number of the first free track in its Track number field. All free entries are chained using that field.

Codec IDs (Hex):
  * 00: ATRAC3
  * 01: ATRAC3+ or MPEG
  * 80: LPCM

== Codec specific info for ATRAC3 ==
  * 0021 Bit 1: Set for joint stereo (LP4)
  * 0022 Bit 7-5: Sample rate
    * 00: 32kHz
    * 20: 44.1kHz
    * 40: 48kHz
    * 60: 88.2kHz
    * 80: 96kHz
  * 0023 Bit 5-0: Frame size in 8 bytes units
    * 18: (Frame size 192 bytes) 66 kbit/s (LP4)
    * 26: (Frame size 304 bytes) 105 kbit/s
    * 30: (Frame size 384 bytes) 132 kbit/s (LP2)

== Codec specific info for ATRAC3+ ==
  * 0022 Bit 7-5: Sample rate (like ATRAC3)
    * 00: 32kHz
    * 20: 44.1kHz
    * 40: 48kHz
    * 60: 88.2kHz
    * 80: 96kHz
  * 0022 Bit 4-2: Number of Channels
    * 04: Mono
    * 08: Stereo
  * 0023 Frame size in 8 bytes units (-1)
    * 17: (Frame size 192 bytes) 32 kbit/s
    * 22: (Frame size 280 bytes) 48 kbit/s
    * 2E: (Frame size 376 bytes) 64 kbit/s (Hi-LP)
    * 8B: (Frame size 1120 bytes) 192 kbit/s
    * B9: (Frame size 1488 bytes) 256 kbit/s (Hi-SP)
    * FF: (Frame size 2048 bytes) 352 kbit/s

== Codec specific info for MPEG ==
  * 0021 constant 03, used to identify MPEG audio
  * 0022 unused, zero
  * 0023 bit field
    * 80: Always set
    * 40: Variable MPEG version (tracks with this bit set cause "cannot play")
    * 20: Variable MPEG Layer (tracks with this bit set cause "cannot play")
    * 10: Variable bitrate
    * 08: Variable sample rate
    * 04: Variable channel mode (Joint stereo/split stereo)
    * 02: Variable preemphasis
    * 01: Always cleared
  * 002C: Bit field:
    * C0: MPEG version (C0 = MPEG1, 80 = MPEG2, 00 = MPEG2.5, as in MPEG spec)
    * 30: MPEG layer (encoded as in MPEG stream)
    * 0F: Bitrate number (see MPEG spec)
  * 002D: Bit field:
    * C0: Sample rate from header (see MPEG spec)
    * 30: Channel mode (00 = split stereo; 10 = joint stereo; 20 = 2*mono; 30 = mono)
    * 0C: Preemphasis (see MPEG spec)
    * 02, 01: always cleared.

=== The Part Info Table ===
Is an array of 4096 part descriptors:
<code>
0000 8 BYTES "PartKey"
0008 WORD    first block number in ADF for this part
000A WORD    last block number in ADF for this part
000C BYTE    number of first frame in first block for this part
000D BYTE    number of last frame in last block of this part
000E WORD    link to next part info for this part
</code>

Like in the track info table, the first entry contains the number of the first free entry in its link field. Each free entry contains the number of the next free entry in its link field (or zero at the end of the chain)

ADF blocks are 16KB. Thus the maximum adressable size is 1GB (16 bit block numbers)

=== The String Table ===
Contains 4096 string slots. Each string slot contains 14 bytes string data and a 2 byte link field. Free entries have the number of the next free slot in the link field; used entries have the number of the continuation in the low 12 bits and type info in the high four bits (8 = title; 9 = artist, A = album, C = disk or group name, 1 = continuation)

The first byte of the concatenated string of all the slots indicates the encoding (05 = Latin1, 84 = UTF16, 90 = ShiftJIS)

===== Differences between models - Sony MZ-NH700 and MZ-RH1 =====

The last model released by Sony, the MZ-RH1, is suspected to use less encryption on recorded PCM-material. It has been observed that the key stored in the track entry (at offset 10, MgrCK) is all zeroes for the RH1-model whereas it is non-zero for older models (RH10 etc). The images [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_with_one_pcm_nh700.img.gz|md_with_one_pcm_nh700.img.gz]] and [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_with_one_pcm_rh1.img.gz|md_with_one_pcm_rh1.img.gz]] contain the same PCM-recording, but were transfered onto an MZ-NH700 and MZ-RH1 respectively. A check with md5sums shows which files are different, obviously the audio-data is *not* different, so the encryption must be the same. This means, that audio-data transfered with SonicStage onto the RH1 uses the strong encryption like the older models as well. The Sony-website for the Mac-Software, which we suspect to be able to upload less encrypted audio only, states: //(1) An audio data transferred using SonicStage or MD Simple Burner, and an audio data downloaded from this software cannot be imported//, which agrees with our observations.

MZ-NH700:

<code>
root@znote2:~> md5sum /mnt2/hmdhifi/*
64c228b7c4c1b02b6c2f1c11bc4148e6  /mnt2/hmdhifi/00010012.hma
5363c96604a0db55e0fafc787385e36b  /mnt2/hmdhifi/_0010012.hma
8c30a8e02a9f0cf0d3dbb43c570848d9  /mnt2/hmdhifi/atdata02.hma
925b99d01399fef20783fa1c9b666871  /mnt2/hmdhifi/mclist01.hma
924346a0e6b878561a92610be83f82ed  /mnt2/hmdhifi/mclist02.hma
4a90f43a7f8ede61f2e28ac844479468  /mnt2/hmdhifi/_mdhifi.hma
555584d9452203ad201b033c88ba6a6f  /mnt2/hmdhifi/trkidx01.hma
7e0c3645525530f05b06222e25fba164  /mnt2/hmdhifi/trkidx02.hma
</code>

MZ-RH1:

<code>
root@znote2:~> md5sum /mnt3/hmdhifi/*
64c228b7c4c1b02b6c2f1c11bc4148e6  /mnt3/hmdhifi/00010012.hma
5363c96604a0db55e0fafc787385e36b  /mnt3/hmdhifi/_0010012.hma
8c30a8e02a9f0cf0d3dbb43c570848d9  /mnt3/hmdhifi/atdata02.hma
114bc6cceff19adf8475753e576a53ac  /mnt3/hmdhifi/mclist01.hma
6700901c313c9bbfe2751ff3a26b600b  /mnt3/hmdhifi/mclist02.hma
b4a6398f6f908fa83031fc6ff66688aa  /mnt3/hmdhifi/_mdhifi.hma
fd7ae076d3a5116e6c478642d9007c81  /mnt3/hmdhifi/trkidx01.hma
f67cab3966a6f314099b37f9c8375494  /mnt3/hmdhifi/trkidx02.hma
root@znote2:~> 
</code>

===== The following information courtesy Rene Rebe =====

<code>
File sizes on an empty disc:

-rwxr-xr-x  1 root root    172 Jul 23 11:07 himd-empty/hmdhifi/00010012.hma
-rwxr-xr-x  1 root root    172 Jul 23 11:07 himd-empty/hmdhifi/_0010012.hma
-rwxr-xr-x  1 root root  32768 Jul 23 11:07 himd-empty/hmdhifi/_clist01.hma
-rwxr-xr-x  1 root root  32768 Jul 23 11:07 himd-empty/hmdhifi/_mdhifi.hma
-rwxr-xr-x  1 root root 327680 Jul 23 11:07 himd-empty/hmdhifi/_rkidx01.hma
-rwxr-xr-x  1 root root      0 Jul 23 11:07 himd-empty/hmdhifi/atdata01.hma
-rwxr-xr-x  1 root root  32768 Jul 23 11:07 himd-empty/hmdhifi/mclist01.hma
-rwxr-xr-x  1 root root 327680 Jul 23 11:07 himd-empty/hmdhifi/trkidx01.hma

6x tracks:

-rwxr-xr-x  1 root root       172 Jul  2 00:08 00010012.hma
-rwxr-xr-x  1 root root       172 Jul  2 00:08 _0010012.hma
-rwxr-xr-x  1 root root     32768 Jul  2 00:08 _mdhifi.hma
-rwxr-xr-x  1 root root    327680 Jul 23  2006 _rkidx05.hma
-rwxr-xr-x  1 root root 360841216 Jul 23 15:37 atdata05.hma
-rwxr-xr-x  1 root root     32768 Jul 22 15:17 mclist04.hma
-rwxr-xr-x  1 root root     32768 Jul 22 15:22 mclist05.hma
-rwxr-xr-x  1 root root     65536 Jul  2 20:52 text_g01.hma
-rwxr-xr-x  1 root root    327680 Jul 23  2006 trkidx05.hm

Hexediting the Track meta data as well as abbritary positions
in the audio data does work (so they are not checksummed). That
is the player shows the updated meta data and still plays the audio.

The audio data is zero padded to 64k boundaries, that is a new
track added starts at:

002C FFF0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........
002D 0000: 53 4D 50 41 00 27 00 03  3F 87 00 00 00 00 00 00  SMPA.'.. ?.......

Other data beside SMPA: PCM and ATX (seen on Atrac3+ encoded data on MZ-RH1).

SMPA appears to be 3des (tripple des) encrypted using one of the schemes:

plain, des ecb with session key, des cbc with session key

 Media Type name: audio
 Media subtype name: ATRACFAMILY
 profile: ATRAC, ATRAC3, ATRAC-X
 frameSampleSize: 512, 1024, 2048
 sampleRate: 32000, 44100, 48000, 88100, 96000
 frameLength: [license required]
 channels: 1, 2, 3, 4, 5, 6, 7, 8
 File extensions: at1, atr, at3, atx

Date and time encoding:

2006-07-24 : 09:14
0000 8050: 34 F8 49 C3 00 01 00 12  00 00 00 00 00 00 00 00  4.I..... ........
0000 8060: 00 00 00 00 00 00 00 00  49 74 E8 31 F5 C0 D0 AC  ........ It.1....
0000 8070: 80 06 28 07 00 01 00 01  00 0A 01 01 00 00 00 00  ..(..... ........
 0000 8080: 02 03 00 00 02 00 01 16  AF 49 00 00 03 8C 7F D9  ........ .I......
0000 8090: 26 4F D1 43 00 00 00 00  00 00 00 00 07 00 08 03  &O.C.... ........

2006-07-24 : 09:14
0000 80A0: 34 F8 49 DA 00 01 00 12  00 00 00 00 00 00 00 00  4.I..... ........
0000 80B0: 00 00 00 00 00 00 00 00  37 0F 59 2B DB 9E 50 E7  ........ 7.Y+..P.
0000 80C0: 01 00 28 2E 00 02 00 02  00 08 01 01 00 00 00 00  ..(..... ........
 0000 80D0: 02 03 00 00 02 00 01 16  AF 49 00 00 96 7D 33 3A  ........ .I...}3:
0000 80E0: 81 4F 34 59 00 00 00 00  00 00 00 00 07 00 08 03  .O4Y.... ........

2006-07-24 : 09:15
0000 80F0: 34 F8 49 FD 00 01 00 12  00 00 00 00 00 00 00 00  4.I..... ........
0000 8100: 00 00 00 00 00 00 00 00  A0 02 C7 B3 18 04 7E 51  ........ ......~Q
0000 8110: 01 00 28 B9 00 03 00 03  00 06 01 01 00 00 00 00  ..(..... ........
 0000 8120: 02 03 00 00 02 00 01 16  AF 49 00 00 D8 3D 73 50  ........ .I...=sP
0000 8130: BE D0 CE 7F 00 00 00 00  00 00 00 00 07 00 08 03  ........ ........


When a file is added:

Not touched:
0010012.hma
_0010012.hma
mclist01.hma
mclist02.hma
_rkidx02.hma

Removed:
_rkidx03.hma

Modified:

_hmhifi.hma:

  A lot of noise.

dc278fcb39eab5e97f5d1c5de1aadfc3  himd-current/hmdhifi/trkidx02.hma

8bca8235b1774769d437227a6be78dba  mnt/himd/hmdhifi/trkidx02.hma

atdata02.hma:

  Of course the whole data content. Just appended. Starting with "SMPA"

trkidx02.hma:

0000 0100: 00 01 00 01 00 00 00 00
0000 0100: 00 02 00 01 00 02 00 00

0000 8020: 00 00 00 00 00 00 00 02
0000 8020: 00 00 00 00 00 00 00 03

0000 80A0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........
0000 80B0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........
0000 80C0: 00 00 00 00 00 00 00 03  00 00 00 00 00 00 00 00  ........ ........
0000 80D0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........
0000 80E0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........

0000 80A0: 2E 21 00 00 00 00 00 00  00 04 00 08 00 0A 01 00  .!...... ........
0000 80B0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........
0000 80C0: 01 03 00 84 00 02 00 02  01 22 10 01 D9 10 00 00  ........ ."......
0000 80D0: 01 0F 50 00 00 04 00 00  00 14 0E BD 96 C3 33 72  ..P..... ......3r
0000 80E0: B2 53 2B 68 00 00 00 00  00 00 00 00 01 00 40 00  .S+h.... ......@.

0003 0000: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 02  ........ ........
0003 0010: 00 00 00 00 00 00 00 00  00 00 00 B1 00 1F 00 00  ........ ........
0003 0020: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 03  ........ ........

0003 0000: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 03  ........ ........
0003 0010: 00 00 00 00 00 00 00 00  00 00 00 B1 00 1F 00 00  ........ ........
0003 0020: 00 00 00 00 00 00 00 00  00 B4 01 D0 00 26 00 00  ........ .....&..

0004 0000: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 04  ........ ........

0004 0000: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 0C  ........ ........

And the meta info we already know how to decode:

0004 0040: 84 00 47 00 65 00 74 00  20 00 52 00 65 00 80 05  ..G.e.t.  .R.e...
0004 0050: 61 00 64 00 79 00 20 00  46 00 6F 00 72 00 10 06  a.d.y. . F.o.r...
0004 0060: 20 00 54 00 68 00 65 00  20 00 46 00 75 00 10 07   .T.h.e.  .F.u...
0004 0070: 74 00 75 00 72 00 65 00  00 00 00 00 00 00 10 00  t.u.r.e. ........
0004 0080: 84 00 54 00 65 00 6E 00  20 00 57 00 69 00 90 09  ..T.e.n.  .W.i...
0004 0090: 74 00 68 00 6F 00 75 00  74 00 20 00 50 00 10 00  t.h.o.u. t. .P...
0004 00A0: 84 00 28 00 44 00 65 00  6D 00 6F 00 20 00 A0 0B  ..(.D.e. m.o. ...
0004 00B0: 41 00 6C 00 62 00 75 00  6D 00 29 00 00 00 10 00  A.l.b.u. m.).....
</code>

See also [[http://svn.exactcode.de/minidisc/trunk/himd.txt]].

===== Fragmentation =====

In order to implement download, we have to understand what happens when tracks are removed inbetween, e.g. have 3 tracks and delete #2. This creates a hole in the data-container.

The following screenshot illustrates a test with SonicStage and himdtest:

{{:mp3test.jpg?1000}}

  - download 3 MP3s onto HiMD, show tracks with himdtest
  - delete MP3 #2, show tracks with himdtest
  - add another, larger MP3, show tracks with himdtest
  - download 3 MP3s onto HiMD, show tracks with himdtest
  - delete MP3 #2, show tracks with himdtest
  - add another, smaller MP3, show tracks with himdtest

==== Fragmentation of the FAT-Filesystem ====

After deleting track Nr. 2:

{{:md_after_deleted_track_2.jpg?750}}

After downloading a new track after deletion of track 2:

{{:md_after_added_track.jpg?750}}

During defrag:

{{:md_while_defragmentation.jpg?750}}

===== Encryption =====

==== General picture ====

{{:himd_encryption.png|Encryption scheme of ATRAC and PCM tracks}}

Red boxes indicate data stored on HiMD, Black boxes indicate "black boxes" - Operations we don't know how they work. For so called "weakly encrypted tracks" the Track key is 0000000000000000, the EKB ID is 00010012 and the fragment keys are 0000000000000000. This information fixes all input parameters to the Key encryption Key. This value is nearly known from analyzing the Mac software - it is F2266C6464C0D65C. As it is used as an DES key, the low bits of each bytes are unknown.

==== Fragment keys ====

Joining two tracks created on an RH10 (so not using zero-key-encryption) with an RH1 walkman (using zero-key-encryption) creates a result of two fragments employing fragment keys. Here are himdtest dumps before/after joining (track 47 is the first part and track 46 is the second part, yes, these numbers are backward, this has nothing to do with this experiment but with disorder of the free list from previous experiments); note the new content ID too:

=== Before join ===
<code>
  46: 0:07 AT3+ Unknown artist:Unknown title (Unknown album 0)
       0@06012 ..   6@06026 (0000000000000000)
     Content ID: 0203000002000102a36e0000a80d0fe3c9aeae71
     Key: 167960f39f024906 (EKB 00010012); MAC: 06f503a38982762b
  47: 0:07 AT3+ Unknown artist:Unknown title (Unknown album 0)
       0@05996 ..   0@06010 (0000000000000000)
     Content ID: 0203000002000102a36e0000d1ed2108778cad89
     Key: 5966e351a690c9f4 (EKB 00010012); MAC: f3c0f999f13d48e0
</code>

=== After join ===
<code>
  47: 0:13 AT3+ Unknown artist:Unknown title (Unknown album 0)
       0@05996 ..   0@06010 (437c871b58fe0a7e)
       0@06012 ..   6@06026 (b27b46d967e14fec)
     Content ID: 02030000020001171d9300005eab87d393f35e44
     Key: 0000000000000000 (EKB 00010012); MAC: 448b34515a6df04d
</code>

===== Links =====

  * [[http://www.marcnetsystem.co.uk/]] - HiMD Lister, software that lists tracks on an HiMD
  * [[http://svn.exactcode.de/minidisc/trunk/main.cc]] - Program-code to parse HiMD meta data
  * [[http://forums.minidisc.org/index.php?showtopic=7944&hl=]] - Simple Hi-MD Music Structure Reader + Sources
  * [[http://forums.minidisc.org/lofiversion/index.php/t10877.html]] - the additional file "text_gXX.hma" from SonicStage 3.1
  * [[http://minidisc.org/hi-md_faq.html]] - HiMD FAQ from minidisc.org