====== HiMDDiskFormat ====== HiMD-Media are read and written in Mass-Storage mode. So one only needs to know the on-disk format to be able to read/write from Linux. The disks are formatted with a standard FAT16-filesystem but with a 2kb-sectorsize. Once the disk is attached to a PC it can be used as a normal mass-storage device (format is superfloppy, i.e. no partitions). When SonicStage (or HiMD Music Transfer on the Mac) is started, the minidisc is unmounted by SonicStage and thus inaccessible by the operating system. This is necessary because SonicStage uses special SCSI read/write-commands to communicate with the MD-Walkman. This is probably necessary for a secured data-exchange between Walkman and the host PC. ===== Disk-Images of various MiniDiscs for testing ===== * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_with_one_mp3.img.gz|md_with_one_mp3.img.gz]] - Image of a 305MB HiMD containing one MP3 - Disk-ID: 0202000002000102A36E0000CAEF24F1 - created with Sony MZ-RH10 and HiMD Music Transfer for Mac * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_empty.img.gz|md_empty.img.gz]] - Image of a freshly formatted 305MB HiMD - Disk-ID: 0202000002000102A36E000040E633F2 - created with Sony MZ-RH10 (built-in format functionality) * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/himd_with_dozens_of_mp3s.img.gz|himd_with_dozens_of_mp3s.img.gz]] - Image of 1GB HiMD containing dozens of MP3s - Disk-ID: 02000000000006121202000000718002 - created with Sony MZ-RH10 and SonicStage 4.3/4 Japanese * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/mdw60.img.gz|mdw60.img.gz]] - Image of 220MB MD (MD-W60) containing one PCM-recording - Disk ID: 020200000200010228310000779D3C0A - Recorded with a Sony MZ-NH600 via Toslink, recording is untitled * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_with_one_pcm.img.gz|md_with_one_pcm.img.gz]] - Image of MD containing one PCM-recording - Disk ID: 0202000002000121B40E0000F9EFA991 - Recorded with a Sony MZ-RH1 from **input** (thus with suspected less encryption), recording is untitled * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_with_atrac3.img.gz|md_with_atrac3.img.gz]] - Image of MD containing three ATRAC3-recordings with 105 kbit/s recorded with MZ-RH1 - Disc ID: 0202000002000121B40E0000B3AFA5B0 * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/mixedmode_md.img.gz|mixedmode_md.img.gz]] - Image of MD containing 4 Tracks, PCM/ATRAC3/MP3 recorded with MZ-RH1 - Disc ID: 0202000002000121B40E0000F9EFA991 * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_with_one_pcm_nh700.img.gz|md_with_one_pcm_nh700.img.gz]] - one PCM-recording transfered with SonicStage onto a MZ-NH700 * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_with_one_pcm_rh1.img.gz|md_with_one_pcm_rh1.img.gz]] - one PCM-recording transfered with SonicStage onto a MZ-RH1 (same PCM as with NH700) ===== The DiskID ===== The disk-id is the unique identifier of each HiMD and is read using special SCSI commands. A small program-code to fetch the disk-id can be found here: * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/minikey.c]] - Read the disc-ID of a MiniDisc in Linux * [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/mp3key.c]] - Generate MP3-encryption key from DiscID The disk-ID is used to encrypt the (audio-)data on the HiMD. The latest versions, however, can be found in the git-repository. ===== Filesystem Issues with some Models on Linux ===== It has been observed, that some older HiMD-Walkman (MZ-NH600 and MZ-NH900) cause some trouble in Linux (but not in Windows) when listing the contents of "HMDHIFI"-directory: root@z6:/mnt/hmdhifi> ls ls: cannot access n/n╒ì╬∩_._▐]: No such file or directory ls: cannot access ▌▀eu╒╟╜«.eΘ«: Input/output error ls: cannot access ╡m ┘╟_╜.■√╓: Input/output error ls: cannot access ·jl_φ╠ _.▐₧▐: Input/output error ls: cannot access  ▀m▀╒╩²«.oφ¡: Input/output error ls: cannot access ■_▀öª▀_┐.≈√╒: Input/output error ls: cannot access ■\▀┐▐»^.²²ƒ: Input/output error ls: cannot access j/no¡╬δ_.■▀φ: No such file or directory (...) ? ²_▀¥∩δ ┐.÷√▐ ª╜²⌡o▌≈▀.┐nw ª▌╜ {▌⌡▀.▌mg jont▌╬╧_.▐₧ì n»j]φ∞δu.▀▀∞ ^on▌═nδ_.▀₧φ ■\▀▀▐»z?.²φ▀ √ƒ╒▌╧▌ⁿ∩.εº▌ ⁿ ▀╣≈█_┐.≈█▀ ▐\ █▀»z .²∙╫ ? «∙╜²}▌µ┘.▀εe ª╗╜²{▌≈▀.┐εe ▀▀e▀_╧▀º.nφ╝ k/l]¡─√_.▐ƒ▌ n/lu¡╧√_.▀╒═ ▀▐ou╓ñ ».nΘ¼ z/nu═╠δ_.▐╛▌ εoj]═fδ].▀ƒ▄ ≈\ █▐½^{.² ƒ ⌡▀ ╣≈δ]┐.▀√▌ 00010012.hma ²ε▀╜╓δ_».■[₧ ª╗╜²{▀σ².¥┼g ▌▀eu╒╟╜«.eΘ« «¥╜⌡k▌÷▌.┐φe n-n╒═╬█^.▐ƒ╪ ▀▐ouuº º.~¡« ⌡\┐▀▀»z?.φδ▀ εonu▌╬√?.▀▀╛  ▀╡▌╧╒²∩.∞╖┼  ┐▌∙╧]╝δ.Ωº▌ _0010012.hma «█∙²?¥╖▐.╜∩  ª╗∙²{▌σ².ƒεu ▀▀e]╫╟═».∩Θ╜ √█╜∙╧lⁿ⌐.l÷═ n»n▌∞╬√_.▐₧▌ ╜█o]u╧ ª.oφ« ▐\▐▀▀»z .u²ü εo ▌╧╦_º.µ√▀  ▀euu╟²«.jφ¼ ■_ ¥τk_╖.µ√▐ ■\┐▌▐½z?.²√ù ²▀ ╣▀╦█».■√₧ ª╜╜²}▌≈ .▀∩g ⁿ_▀¥g╦_┐.■√▀ l/l┼ì╬δ].▀ƒ\ n/n╒ì╬∩_._▐] ⁿ]▀▌»▀_┐.t√▐ ■\┐▀▌»z .⌡ ƒ ε╬▀▌τ╦[».■√▀  ▀m▀╒╩²«.oφ¡ ∞[ ▌τδ ».⌠√▀ ■\▀┐▀½z?. ▀┐ ■² ╣∩δ ┐.⌡√▀ ª╗▌¡k╜╖▌.▀εg ·jl_φ╠ _.▐₧▐ l»ne¡e█u.▐₧▄ n/n╟¡l√_.▀ƒ] ■t▐█▀∩^?.²²ƒ ⌡∩▀▌╟δ_┐.≈∙▐ ╝_▀▌τ╦^┐.ε·╓ ■_ ö∩╦]┐.≈√▌ ⁿ▐ ▌τ╦_┐.σ√╓ ■\▀▀▐½z .⌡ ƒ ▐\■▀▐»^². φƒ ª√∙╜o¥╡▌.ƒ∩w j/n─¡╧∩_.▐▀▌ ~»lw═╠ u.▀ƒ╩ n/nu═n█_.▀ƒ╧ trkidx01.hma δnnu¡dδ_.\┐■ ∩_▀¥τ ]┐.÷√╫ ⁿ▀ ö╟╦]╡.÷[▐  ┐▌∙╧▌ⁿ».Ωτ² ⁿ²▀▌∩╦_╖.╒√▀ ²_ ╣ ╦_┐. √▀ ª█╣⌡o¥≈█.┐■τ j»n┼═╠╧].╘ƒ▌ mclist01.hma ▀█n]t╧²«.εφ╜ trkidx02.hma ÷_▀¥╧δ_º.÷{▐ &╜╜φk¥⌡▌.ƒ∩g ⁿ╬ ╜╟╦_º.µ{▐ ⌡] ¥▀√_┐. █▀ ■\▀┐▐»^?.²²ƒ √█⌡┘╧▌╜».·ª▌ atdata02.hma j∩n▌¡╠√_.▐ƒ▌ mclist02.hma ^/n╫═╠δ]._▀² ÷\■▀▀»┌?.⌡▀ù ⁿ]█┘╞δ_╜.⌠y▐ φ∩n╫╜╬▀_.▐┐▌ «¥╜ o▌º▀.²φg   ╜┘å╒■Θ.ε╡▌ █▀╜²╧e²φ.∞º═ ª╣²²m▌÷▌.▌d⌡ ª█⌡┐}▌σ▀.²∩σ j/nm╠─Ω_.\▐î _mdhifi.hma ∩»n]¡ε√_.▀ü▌ ■\₧▀▐∩z}.}²ƒ δφn╫¡╞√_.▐┐▌ √▀¡∙═╘∞».Ω╖═ ■ ▀╜ùδ_».╓[▐ ■_ ▌ δ▀╜.≈√▀ ²o ╜≈√_».⌡[▀ ª╣²∩m▌ñ².▀σe ª┐╜╜?▌≈▀.▌∩τ j/no¡╬δ_.■▀φ ╡m ┘╟_?╜.■√╓ ■_▀öª▀_┐.≈√╒ ÷\┐█▀»z².²√ƒ ■}▀¥╟╦_».ε{▐ Ω»n▄═╬δ].▐ƒm ■| ╧▐»z?.⌡√ƒ ■\  ▀»┌?.≈ ╖ ²o ▌τ█_».ε{₧ ª┐┘²o▌÷².█∩e ª√╜»?▌τ².╜∩σ j/n▄═╠φ].▐₧┌ ╬-n┼╠╠█_.▐▐═ ▀▀o]■╟┐ª.oφ╣ ÷\┐█▀»z?.⌡²ù ε/n╓═╬δ].▐₧▄ ▐\ƒ╗▀»^?. √ƒ ÷\ ▀▀∩z?.} ┐ root@z6:/mnt/hmdhifi> However the important files can still be accessed properly. The kernel suggests that something is wrong with the FAT: [1213211.340236] FAT: Filesystem panic (dev loop0) [1213211.340244] invalid access to FAT (entry 0x0000ffe5) [1213211.340248] File system has been set read-only [1213211.340579] FAT: Filesystem panic (dev loop0) [1213211.340585] invalid access to FAT (entry 0x0000ffe7) [1213211.340810] FAT: Filesystem panic (dev loop0) [1213211.340810] invalid access to FAT (entry 0x00005fde) [1213211.340810] FAT: Filesystem panic (dev loop0) [1213211.340810] invalid access to FAT (entry 0x0000fbef) [1213211.340810] FAT: Filesystem panic (dev loop0) and root@z6:~> fsck.vfat /amd/z6/0/mdw60.img dosfsck 3.0.1, 23 Nov 2008, FAT32, LFN /HMDHIFI Has a large number of bad entries. (245/268) Drop directory ? (y/n) It has not yet been confirmed what causes these errors. All we know is that they apparently only occur with older models and that the filesystems list fine in Windows. Probably some undocument FAT-features ?!? ===== Files ===== The following files can be found on a freshly formatted HiMD (MiniDisc mounted to /mnt) in the directory "HMDHIFI". The root-directory of an HiMD contains this folder only and a zero-byte file called "HI-MD.IND" which presumably is only meant to be an identifier for a HiMD-disc. root@znote2:/mnt/hmdhifi> md5sum /mnt3/hmdhifi/* 64c228b7c4c1b02b6c2f1c11bc4148e6 /mnt3/hmdhifi/00010012.hma <---- EKB (key container) for standard HiMD keys 5363c96604a0db55e0fafc787385e36b /mnt3/hmdhifi/_0010012.hma <---- Backup of EKB d41d8cd98f00b204e9800998ecf8427e /mnt3/hmdhifi/atdata01.hma <---- Containerfile containing all audio data bb7df04e1b0a2570657527a7e108ae23 /mnt3/hmdhifi/_clist01.hma <---- Backup of Keyfile for Digital Rights Management 28147b60982b017008e75594ab2bdf00 /mnt3/hmdhifi/mclist01.hma <---- Keyfile for Digital Rights Management (unconfirmed) bb7df04e1b0a2570657527a7e108ae23 /mnt3/hmdhifi/_mdhifi.hma <---- Backup of the directory information of this directory 1aca77e2188f52a62674fe8a873bdaba /mnt3/hmdhifi/_rkidx01.hma <---- Backup of last Track Index File 09b5367f524ecdbfcae6af4b3173d3db /mnt3/hmdhifi/trkidx01.hma <---- Track Index File The following files are found on an HiMD containing several dozen MP3-tracks: root@burns:/mnt/hmdhifi> md5sum * 64c228b7c4c1b02b6c2f1c11bc4148e6 00010012.hma 5363c96604a0db55e0fafc787385e36b _0010012.hma d6c8df08ff525dd3be7a90978acb39c7 atdata02.hma bb7df04e1b0a2570657527a7e108ae23 mclist01.hma 1fcd128c53d2bd58fd50c666b2fc6475 mclist02.hma fafb8cbcdc3cfc70972558b6e458fd30 _mdhifi.hma 4b0f6c1dd7e893026295d4b9c3c5a8be _rkidx03.hma 6b3e2b2c54b25b752fa831ba96a54615 text_g01.hma cdf55e77362fb9a510207f8bb73fb826 thumbg01.hma 370af574809d15d7eaea86030200b020 trkidx02.hma root@burns:/mnt/hmdhifi> hexedit mclist01.hma glaubitz@z6:..sdf/HMDHIFI> ls -l total 551904 -rw------- 1 glaubitz ag-wolf 172 2004-01-01 00:00 00010012.HMA -rw------- 1 glaubitz ag-wolf 172 2004-01-01 00:00 _0010012.HMA -rw------- 1 glaubitz ag-wolf 562167808 2008-09-20 18:31 ATDATA02.HMA -rw------- 1 glaubitz ag-wolf 32768 2004-01-01 00:00 MCLIST01.HMA -rw------- 1 glaubitz ag-wolf 32768 2004-01-01 00:00 MCLIST02.HMA -rw------- 1 glaubitz ag-wolf 32768 2004-01-01 00:00 _MDHIFI.HMA -rw------- 1 glaubitz ag-wolf 327680 2008-09-20 18:32 _RKIDX03.HMA -rw------- 1 glaubitz ag-wolf 65536 2008-09-16 18:33 TEXT_G01.HMA -rw------- 1 glaubitz ag-wolf 2097152 2008-09-20 18:12 THUMBG01.HMA -rw------- 1 glaubitz ag-wolf 327680 2008-09-20 18:32 TRKIDX02.HMA glaubitz@z6:..sdf/HMDHIFI> The md5sums for all files are always the same after a fresh format except for the file "mclist01.hma" whose md5sum changes on every disk-write and is always different, even for the same minidisc used. The file atdata01.hma is the containerfile which keeps all audio-data (this file grows once new tracks are loaded onto the disk). trkidx01.hma appears to be an index-file for the walkman. The numeric appendices are increased from time to time. Obviously, simply adding one mp3-track with the transfer-software for Mac does not increase the numbers. It might be only increased when adding ATRAC-tracks or when adding more than one file. The files with the highest index are the only to contain valid data. When several trkidx0X.hma can be found for example, only the file with the highest index contains non-zero content (the other are zero-filled, but not zero-size). ==== The file mclist0X.hma ==== The filename "mclist01.hma" obviuosly derives from "MAClist" which is the name of one of the modules in OpenMG. MAC here stands for "Message authentication code" (see [[http://en.wikipedia.org/wiki/Message_authentication_code]]). In this case, the DRM license info of each track is considered a message, and the authenticity of these messages is confirmed by adding a 8-byte field to the track information. The MAC list contains all the MAC values of the different tracks and a master MAC that ties these together into one bundle. Furthermore, the MAC list contains a header that ties the MAC list to a specific "generation" of a specific medium. The generation is increased every time the DRM info changes. The final authentication is done by combining checksums of the medium-specific and the track-specific parts into a master "integrity check value" (ICV) which is stored outside of the FAT file system and only accessible using authenticated and encrypted SCSI commands. The contents of "mclist01.hma" are mostly zero until DRM protected tracks are uploaded, the non-zero part looks like this: 0000000: 4d4c 5354 0000 0124 0000 0000 0000 0000 MLST...$........ 0000010: 32af 72b6 0a4f 1612 e490 ce90 90b2 07ec 2.r..O.......... 0000020: 0000 0001 0000 0000 0000 0000 0000 0000 ................ 0000030: 0000 0000 0000 0000 0001 0012 0000 0000 ................ 0000040: 0202 0000 0200 0102 a36e 0000 caef 24f1 .........n....$. 0000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000060: fdd5 d64b 046f 53cf 4853 312d 8091 82ab ...K.oS.HS1-.... The MAC list file is divided into three parts. The first 16 bytes are the file header, the next 80 bytes are for disc authentication and the remaining part is for track authentication. Both authentication parts start with an encrypted 3DES key used for authentication, followed by the data to authenticate. To decrypt the authentication key, a master key needs to be used. In the Sony DRM system, these master keys are stored in EKBs, in encrypted form. The disc authentication header contains an indication which EKB (that also means which master key) to use. 0000 BYTES magic signature "MLST" 0004 DWORD unknown purpose 0008 BYTES 8 unknown bytes, always zero, maybe padding. 0010 BYTES 16 bytes encrypted 3DES key for authenticating the first MAC list part 0020 DWORD Generation number of the DRM info 0024 BYTES 20 unknown bytes, always zero, maybe padding 0038 DWORD ID of the EKB used for decrypting the authentication keys 003C DWORD 4 unknown bytes, always zero, maybe padding 0040 BYTES 16 bytes disc ID. This is a copy of the real disc ID stored outside of the file system 0050 BYTES 16 unknown bytes, always zero, maybe padding 0060 BYTES 16 bytes encrypted 3DES key for authenticating the second MAC list part 0070 BYTES 4000 * 8 bytes MAC values of tracks. Intersting count, as Hi-MD only has 2048 tracks. ==== The file 00010012.hma ==== This file contains the "EKB", the enabling key block #00010012. This is the standard key block for HiMD audio. It's the string "EKB" followed by 29 null bytes (making a 32 byte header), and then the contents of the EKB that is also provided in OpenMG as 00010012-umd.EKB. It should never change, but if encrypted payware audio with other EKBs are uploaded, further all-numeric .hma files might appear. ==== The Audio Data File ==== The audio data file contains the raw audio data. It is divided into 16k blocks of the following layout 0000 DWORD Block Type ("LPCM" for PCM,"A3D " for ATRAC3, "ATX " for ATRAC3+, "SMPA" for MPEG) 0004 WORD Number of frames (MP3 blocks only, PCM/Atrac blocks have fixed size) 0006 WORD "MCode" 0008 WORD Data size in bytes (MP3 blocks only) 000A WORD Reserved 000C DWORD Serial number of block in stream 0010 BYTES PCM/Atrac: Encrypted DES key for the audio data 0018 BYTES PCM/Atrac: DES CBC initialization vector for the audio data 0020 BYTES up to 3FC0 of encrypted audio data (ATRAC never uses all 3FC0 bytes) 3FE0 BYTES Backup of encrypted key (offset 0010) 3FE8 BYTES 8 bytes reserved 3FF0 DWORD backup of Block Type 3FF4 WORD reserved 3FF6 WORD backup of "MCode" 3FF8 DWORD low order 32 bits of the Content ID 3FFC DWORD backup of serial number In case of MP3 audio, the data is XOR encrypted. The key for encryption is created from the DiscID which read/written using special SCSI commands. In case of ATRAC/PCM audio, the data is DES CBC encrypted. See below for more info on encryption. Each audio block contains only complete frames. The space in the audio block after the last complete frame is unused - which is especially wasteful for 352kbit/s ATRAC3+, as the frame size is exactly 2K, so 1984 bytes per frame are ignored, i.e. 12% of the block size. ==== The _MDHIFI.HMA File ==== This file is obviously a backup-file for the directory content: root@znote2:..disk/HMDHIFI> strings _MDHIFI.HMA . !0!0 .. !0!0 TRKIDX01HMA !0!0 ATDATA01HMA !0!0 MCLIST01HMA !0!0 00010012HMA !0!0 _MDHIFI HMA !0!0 _RKIDX01HMA !0!0 _CLIST01HMA !0!0 _0010012HMA !0!0 root@znote2:..disk/HMDHIFI> ==== The Track-Index File ==== The Track Index File (or TIF for short, trkidxXX.hma) contains a TOC like data structure telling about differnt groups/tracks/parts of the music, which is contained in the audio data file (atdataXX.hma). The size of the index file is fixed 320K; it has the following structure (derived from symbol infos in the MAC software) 00000..00100 Header (256 Bytes) 00100..01100 Play order table (4K) 01100..02100 Programmed play order table (4K) 02100..02900 Group info table (2K) 02900..08000 Padding 08000..30000 Track info table (160K) 30000..40000 Part info table (64K) 40000..50000 Name table (64K) === The Header === 0000 DWORD Magic number to identify the Track index file "TIF " (54 49 46 20) 0004 WORD Version number (seen as 01 00, i.e. 1 in little endian) 0006 WORD "MCode" ??? (seen as 01 24) 0008 DWORD reserved 000C DWORD revision (seen as 00 00 00 01, i.e. 1 in big endian) 0010 DWORD "GrpThmRevision" 0014 DWORD "TrkThmRevision" 0018 DWORD "GrpTxtRevision" 001C DWORD "TrkTxtRevision" 0020..0100 reserved All quoted field names have been directly taken from RTTI in the MacOS software. All revision fields were observed as zero. As long as no endianess is specified, big endian is used. === The Play Order Table === Is an array of 2048 16 bit words. The first word is the number of tracks on the MD, the next words is the order in which the tracks should be played. A one track MD has the first and second word 1. === The Programmed Play Order Table === Is also an array of 2048 16 bit words. It is completely empty on a simple one-track MD. === The Group Info Table === Contains 256 group descriptors of the following layout 0000 WORD From 0002 WORD To 0004 WORD Name (string index) 0006 WORD Fringe The group table on the one-track MD is completly empty (everything zero) === The Track Info Table === Contains 2048 track descriptors of the following layout 0000 DWORD - date of recording (FAT format, 16 bit date, 16 bit time of day) 0004 DWORD ! "EkbVersion" (Mac MP3: 0, Mac WAV: 10012; upload requirement: == 10012) 0008 WORD - Title (string number) 000A WORD - Artist (string number) 000C WORD - Album (string number) 000E BYTE - Track number (within Album, not on MD) 000F BYTE - "Mode" 0010 8 BYTES ! MgrCK (upload requirement: completely zero) 0018 8 BYTES CMac 0020 BYTE - CodecId (see below) 0021 3 BYTES - Codec specific info 0024 WORD - Part Number (index into Parts Info Table) 0026 WORD - Track Number 0028 WORD + Total time (units of seconds) 002A BYTE + "Lt" (Mac MP3: 10, Mac WAV: 1; upload requirement: == 1) 002B BYTE + "Dest" (upload requirement: == 1) 002C WORD + More codec specific info 002E WORD + reserved 0030 20 BYTE + Content ID (for Mac Transferred Data: 02 03 00 00 + 16 random bytes) 0044 DWORD + Start of playback license validity (FAT format, or 0 for no restriction) 0048 DWORD + End of playback license validity (FAT format, or 0 for no restriction) 004C BYTE + "Xcc" (Mac MP3/WAV: 01; upload requirement: == 03 || == 07) 004D BYTE + "Ct" 004E BYTE + "Cc" (Mac MP3: 40, Mac WAV: 44; upload requirement: == 08 || == 48) 004F BYTE + "Cn" (Mac MP3: 00, Mac WAV: 3) The +/-/! means: "+" this field is included in MAC calculation. "-" this field does not influence the MAC. "!" this field controls the MAC calculation "upload requirement" means that the HiMD Transfer Tool for Mac checks the given condition before allowing an upload of a track to the PC. These checks are independent from the track format. "Mac MP3"/"Mac WAV" means that these values are set by the HiMD transfer tool if downloading that type of music. Probably the fields annotated that way are involved in copy controlling. The first entry in the track info table is a dummy entry that contains the number of the first free track in its Track number field. All free entries are chained using that field. Codec IDs (Hex): * 00: ATRAC3 * 01: ATRAC3+ or MPEG * 80: LPCM == Codec specific info for ATRAC3 == * 0021 Bit 1: Set for joint stereo (LP4) * 0022 Bit 7-5: Sample rate * 00: 32kHz * 20: 44.1kHz * 40: 48kHz * 60: 88.2kHz * 80: 96kHz * 0023 Bit 5-0: Frame size in 8 bytes units * 18: (Frame size 192 bytes) 66 kbit/s (LP4) * 26: (Frame size 304 bytes) 105 kbit/s * 30: (Frame size 384 bytes) 132 kbit/s (LP2) == Codec specific info for ATRAC3+ == * 0022 Bit 7-5: Sample rate (like ATRAC3) * 00: 32kHz * 20: 44.1kHz * 40: 48kHz * 60: 88.2kHz * 80: 96kHz * 0022 Bit 4-2: Number of Channels * 04: Mono * 08: Stereo * 0023 Frame size in 8 bytes units (-1) * 17: (Frame size 192 bytes) 32 kbit/s * 22: (Frame size 280 bytes) 48 kbit/s * 2E: (Frame size 376 bytes) 64 kbit/s (Hi-LP) * 8B: (Frame size 1120 bytes) 192 kbit/s * B9: (Frame size 1488 bytes) 256 kbit/s (Hi-SP) * FF: (Frame size 2048 bytes) 352 kbit/s == Codec specific info for MPEG == * 0021 constant 03, used to identify MPEG audio * 0022 unused, zero * 0023 bit field * 80: Always set * 40: Variable MPEG version (tracks with this bit set cause "cannot play") * 20: Variable MPEG Layer (tracks with this bit set cause "cannot play") * 10: Variable bitrate * 08: Variable sample rate * 04: Variable channel mode (Joint stereo/split stereo) * 02: Variable preemphasis * 01: Always cleared * 002C: Bit field: * C0: MPEG version (C0 = MPEG1, 80 = MPEG2, 00 = MPEG2.5, as in MPEG spec) * 30: MPEG layer (encoded as in MPEG stream) * 0F: Bitrate number (see MPEG spec) * 002D: Bit field: * C0: Sample rate from header (see MPEG spec) * 30: Channel mode (00 = split stereo; 10 = joint stereo; 20 = 2*mono; 30 = mono) * 0C: Preemphasis (see MPEG spec) * 02, 01: always cleared. === The Part Info Table === Is an array of 4096 part descriptors: 0000 8 BYTES "PartKey" 0008 WORD first block number in ADF for this part 000A WORD last block number in ADF for this part 000C BYTE number of first frame in first block for this part 000D BYTE number of last frame in last block of this part 000E WORD link to next part info for this part Like in the track info table, the first entry contains the number of the first free entry in its link field. Each free entry contains the number of the next free entry in its link field (or zero at the end of the chain) ADF blocks are 16KB. Thus the maximum adressable size is 1GB (16 bit block numbers) === The String Table === Contains 4096 string slots. Each string slot contains 14 bytes string data and a 2 byte link field. Free entries have the number of the next free slot in the link field; used entries have the number of the continuation in the low 12 bits and type info in the high four bits (8 = title; 9 = artist, A = album, C = disk or group name, 1 = continuation) The first byte of the concatenated string of all the slots indicates the encoding (05 = Latin1, 84 = UTF16, 90 = ShiftJIS) ===== Differences between models - Sony MZ-NH700 and MZ-RH1 ===== The last model released by Sony, the MZ-RH1, is suspected to use less encryption on recorded PCM-material. It has been observed that the key stored in the track entry (at offset 10, MgrCK) is all zeroes for the RH1-model whereas it is non-zero for older models (RH10 etc). The images [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_with_one_pcm_nh700.img.gz|md_with_one_pcm_nh700.img.gz]] and [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/md_with_one_pcm_rh1.img.gz|md_with_one_pcm_rh1.img.gz]] contain the same PCM-recording, but were transfered onto an MZ-NH700 and MZ-RH1 respectively. A check with md5sums shows which files are different, obviously the audio-data is *not* different, so the encryption must be the same. This means, that audio-data transfered with SonicStage onto the RH1 uses the strong encryption like the older models as well. The Sony-website for the Mac-Software, which we suspect to be able to upload less encrypted audio only, states: //(1) An audio data transferred using SonicStage or MD Simple Burner, and an audio data downloaded from this software cannot be imported//, which agrees with our observations. MZ-NH700: root@znote2:~> md5sum /mnt2/hmdhifi/* 64c228b7c4c1b02b6c2f1c11bc4148e6 /mnt2/hmdhifi/00010012.hma 5363c96604a0db55e0fafc787385e36b /mnt2/hmdhifi/_0010012.hma 8c30a8e02a9f0cf0d3dbb43c570848d9 /mnt2/hmdhifi/atdata02.hma 925b99d01399fef20783fa1c9b666871 /mnt2/hmdhifi/mclist01.hma 924346a0e6b878561a92610be83f82ed /mnt2/hmdhifi/mclist02.hma 4a90f43a7f8ede61f2e28ac844479468 /mnt2/hmdhifi/_mdhifi.hma 555584d9452203ad201b033c88ba6a6f /mnt2/hmdhifi/trkidx01.hma 7e0c3645525530f05b06222e25fba164 /mnt2/hmdhifi/trkidx02.hma MZ-RH1: root@znote2:~> md5sum /mnt3/hmdhifi/* 64c228b7c4c1b02b6c2f1c11bc4148e6 /mnt3/hmdhifi/00010012.hma 5363c96604a0db55e0fafc787385e36b /mnt3/hmdhifi/_0010012.hma 8c30a8e02a9f0cf0d3dbb43c570848d9 /mnt3/hmdhifi/atdata02.hma 114bc6cceff19adf8475753e576a53ac /mnt3/hmdhifi/mclist01.hma 6700901c313c9bbfe2751ff3a26b600b /mnt3/hmdhifi/mclist02.hma b4a6398f6f908fa83031fc6ff66688aa /mnt3/hmdhifi/_mdhifi.hma fd7ae076d3a5116e6c478642d9007c81 /mnt3/hmdhifi/trkidx01.hma f67cab3966a6f314099b37f9c8375494 /mnt3/hmdhifi/trkidx02.hma root@znote2:~> ===== The following information courtesy Rene Rebe ===== File sizes on an empty disc: -rwxr-xr-x 1 root root 172 Jul 23 11:07 himd-empty/hmdhifi/00010012.hma -rwxr-xr-x 1 root root 172 Jul 23 11:07 himd-empty/hmdhifi/_0010012.hma -rwxr-xr-x 1 root root 32768 Jul 23 11:07 himd-empty/hmdhifi/_clist01.hma -rwxr-xr-x 1 root root 32768 Jul 23 11:07 himd-empty/hmdhifi/_mdhifi.hma -rwxr-xr-x 1 root root 327680 Jul 23 11:07 himd-empty/hmdhifi/_rkidx01.hma -rwxr-xr-x 1 root root 0 Jul 23 11:07 himd-empty/hmdhifi/atdata01.hma -rwxr-xr-x 1 root root 32768 Jul 23 11:07 himd-empty/hmdhifi/mclist01.hma -rwxr-xr-x 1 root root 327680 Jul 23 11:07 himd-empty/hmdhifi/trkidx01.hma 6x tracks: -rwxr-xr-x 1 root root 172 Jul 2 00:08 00010012.hma -rwxr-xr-x 1 root root 172 Jul 2 00:08 _0010012.hma -rwxr-xr-x 1 root root 32768 Jul 2 00:08 _mdhifi.hma -rwxr-xr-x 1 root root 327680 Jul 23 2006 _rkidx05.hma -rwxr-xr-x 1 root root 360841216 Jul 23 15:37 atdata05.hma -rwxr-xr-x 1 root root 32768 Jul 22 15:17 mclist04.hma -rwxr-xr-x 1 root root 32768 Jul 22 15:22 mclist05.hma -rwxr-xr-x 1 root root 65536 Jul 2 20:52 text_g01.hma -rwxr-xr-x 1 root root 327680 Jul 23 2006 trkidx05.hm Hexediting the Track meta data as well as abbritary positions in the audio data does work (so they are not checksummed). That is the player shows the updated meta data and still plays the audio. The audio data is zero padded to 64k boundaries, that is a new track added starts at: 002C FFF0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........ 002D 0000: 53 4D 50 41 00 27 00 03 3F 87 00 00 00 00 00 00 SMPA.'.. ?....... Other data beside SMPA: PCM and ATX (seen on Atrac3+ encoded data on MZ-RH1). SMPA appears to be 3des (tripple des) encrypted using one of the schemes: plain, des ecb with session key, des cbc with session key Media Type name: audio Media subtype name: ATRACFAMILY profile: ATRAC, ATRAC3, ATRAC-X frameSampleSize: 512, 1024, 2048 sampleRate: 32000, 44100, 48000, 88100, 96000 frameLength: [license required] channels: 1, 2, 3, 4, 5, 6, 7, 8 File extensions: at1, atr, at3, atx Date and time encoding: 2006-07-24 : 09:14 0000 8050: 34 F8 49 C3 00 01 00 12 00 00 00 00 00 00 00 00 4.I..... ........ 0000 8060: 00 00 00 00 00 00 00 00 49 74 E8 31 F5 C0 D0 AC ........ It.1.... 0000 8070: 80 06 28 07 00 01 00 01 00 0A 01 01 00 00 00 00 ..(..... ........ 0000 8080: 02 03 00 00 02 00 01 16 AF 49 00 00 03 8C 7F D9 ........ .I...... 0000 8090: 26 4F D1 43 00 00 00 00 00 00 00 00 07 00 08 03 &O.C.... ........ 2006-07-24 : 09:14 0000 80A0: 34 F8 49 DA 00 01 00 12 00 00 00 00 00 00 00 00 4.I..... ........ 0000 80B0: 00 00 00 00 00 00 00 00 37 0F 59 2B DB 9E 50 E7 ........ 7.Y+..P. 0000 80C0: 01 00 28 2E 00 02 00 02 00 08 01 01 00 00 00 00 ..(..... ........ 0000 80D0: 02 03 00 00 02 00 01 16 AF 49 00 00 96 7D 33 3A ........ .I...}3: 0000 80E0: 81 4F 34 59 00 00 00 00 00 00 00 00 07 00 08 03 .O4Y.... ........ 2006-07-24 : 09:15 0000 80F0: 34 F8 49 FD 00 01 00 12 00 00 00 00 00 00 00 00 4.I..... ........ 0000 8100: 00 00 00 00 00 00 00 00 A0 02 C7 B3 18 04 7E 51 ........ ......~Q 0000 8110: 01 00 28 B9 00 03 00 03 00 06 01 01 00 00 00 00 ..(..... ........ 0000 8120: 02 03 00 00 02 00 01 16 AF 49 00 00 D8 3D 73 50 ........ .I...=sP 0000 8130: BE D0 CE 7F 00 00 00 00 00 00 00 00 07 00 08 03 ........ ........ When a file is added: Not touched: 0010012.hma _0010012.hma mclist01.hma mclist02.hma _rkidx02.hma Removed: _rkidx03.hma Modified: _hmhifi.hma: A lot of noise. dc278fcb39eab5e97f5d1c5de1aadfc3 himd-current/hmdhifi/trkidx02.hma 8bca8235b1774769d437227a6be78dba mnt/himd/hmdhifi/trkidx02.hma atdata02.hma: Of course the whole data content. Just appended. Starting with "SMPA" trkidx02.hma: 0000 0100: 00 01 00 01 00 00 00 00 0000 0100: 00 02 00 01 00 02 00 00 0000 8020: 00 00 00 00 00 00 00 02 0000 8020: 00 00 00 00 00 00 00 03 0000 80A0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........ 0000 80B0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........ 0000 80C0: 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 00 ........ ........ 0000 80D0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........ 0000 80E0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........ 0000 80A0: 2E 21 00 00 00 00 00 00 00 04 00 08 00 0A 01 00 .!...... ........ 0000 80B0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........ 0000 80C0: 01 03 00 84 00 02 00 02 01 22 10 01 D9 10 00 00 ........ ."...... 0000 80D0: 01 0F 50 00 00 04 00 00 00 14 0E BD 96 C3 33 72 ..P..... ......3r 0000 80E0: B2 53 2B 68 00 00 00 00 00 00 00 00 01 00 40 00 .S+h.... ......@. 0003 0000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02 ........ ........ 0003 0010: 00 00 00 00 00 00 00 00 00 00 00 B1 00 1F 00 00 ........ ........ 0003 0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 ........ ........ 0003 0000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 ........ ........ 0003 0010: 00 00 00 00 00 00 00 00 00 00 00 B1 00 1F 00 00 ........ ........ 0003 0020: 00 00 00 00 00 00 00 00 00 B4 01 D0 00 26 00 00 ........ .....&.. 0004 0000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04 ........ ........ 0004 0000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0C ........ ........ And the meta info we already know how to decode: 0004 0040: 84 00 47 00 65 00 74 00 20 00 52 00 65 00 80 05 ..G.e.t. .R.e... 0004 0050: 61 00 64 00 79 00 20 00 46 00 6F 00 72 00 10 06 a.d.y. . F.o.r... 0004 0060: 20 00 54 00 68 00 65 00 20 00 46 00 75 00 10 07 .T.h.e. .F.u... 0004 0070: 74 00 75 00 72 00 65 00 00 00 00 00 00 00 10 00 t.u.r.e. ........ 0004 0080: 84 00 54 00 65 00 6E 00 20 00 57 00 69 00 90 09 ..T.e.n. .W.i... 0004 0090: 74 00 68 00 6F 00 75 00 74 00 20 00 50 00 10 00 t.h.o.u. t. .P... 0004 00A0: 84 00 28 00 44 00 65 00 6D 00 6F 00 20 00 A0 0B ..(.D.e. m.o. ... 0004 00B0: 41 00 6C 00 62 00 75 00 6D 00 29 00 00 00 10 00 A.l.b.u. m.)..... See also [[http://svn.exactcode.de/minidisc/trunk/himd.txt]]. ===== Fragmentation ===== In order to implement download, we have to understand what happens when tracks are removed inbetween, e.g. have 3 tracks and delete #2. This creates a hole in the data-container. The following screenshot illustrates a test with SonicStage and himdtest: {{:mp3test.jpg?1000}} - download 3 MP3s onto HiMD, show tracks with himdtest - delete MP3 #2, show tracks with himdtest - add another, larger MP3, show tracks with himdtest - download 3 MP3s onto HiMD, show tracks with himdtest - delete MP3 #2, show tracks with himdtest - add another, smaller MP3, show tracks with himdtest ==== Fragmentation of the FAT-Filesystem ==== After deleting track Nr. 2: {{:md_after_deleted_track_2.jpg?750}} After downloading a new track after deletion of track 2: {{:md_after_added_track.jpg?750}} During defrag: {{:md_while_defragmentation.jpg?750}} ===== Encryption ===== ==== General picture ==== {{:himd_encryption.png|Encryption scheme of ATRAC and PCM tracks}} Red boxes indicate data stored on HiMD, Black boxes indicate "black boxes" - Operations we don't know how they work. For so called "weakly encrypted tracks" the Track key is 0000000000000000, the EKB ID is 00010012 and the fragment keys are 0000000000000000. This information fixes all input parameters to the Key encryption Key. This value is nearly known from analyzing the Mac software - it is F2266C6464C0D65C. As it is used as an DES key, the low bits of each bytes are unknown. ==== Fragment keys ==== Joining two tracks created on an RH10 (so not using zero-key-encryption) with an RH1 walkman (using zero-key-encryption) creates a result of two fragments employing fragment keys. Here are himdtest dumps before/after joining (track 47 is the first part and track 46 is the second part, yes, these numbers are backward, this has nothing to do with this experiment but with disorder of the free list from previous experiments); note the new content ID too: === Before join === 46: 0:07 AT3+ Unknown artist:Unknown title (Unknown album 0) 0@06012 .. 6@06026 (0000000000000000) Content ID: 0203000002000102a36e0000a80d0fe3c9aeae71 Key: 167960f39f024906 (EKB 00010012); MAC: 06f503a38982762b 47: 0:07 AT3+ Unknown artist:Unknown title (Unknown album 0) 0@05996 .. 0@06010 (0000000000000000) Content ID: 0203000002000102a36e0000d1ed2108778cad89 Key: 5966e351a690c9f4 (EKB 00010012); MAC: f3c0f999f13d48e0 === After join === 47: 0:13 AT3+ Unknown artist:Unknown title (Unknown album 0) 0@05996 .. 0@06010 (437c871b58fe0a7e) 0@06012 .. 6@06026 (b27b46d967e14fec) Content ID: 02030000020001171d9300005eab87d393f35e44 Key: 0000000000000000 (EKB 00010012); MAC: 448b34515a6df04d ===== Links ===== * [[http://www.marcnetsystem.co.uk/]] - HiMD Lister, software that lists tracks on an HiMD * [[http://svn.exactcode.de/minidisc/trunk/main.cc]] - Program-code to parse HiMD meta data * [[http://forums.minidisc.org/index.php?showtopic=7944&hl=]] - Simple Hi-MD Music Structure Reader + Sources * [[http://forums.minidisc.org/lofiversion/index.php/t10877.html]] - the additional file "text_gXX.hma" from SonicStage 3.1 * [[http://minidisc.org/hi-md_faq.html]] - HiMD FAQ from minidisc.org