Differences

This shows you the differences between two versions of the page.

--- himddiskformat [2009/04/28 22:07] – megadiscman
+++ himddiskformat [2012/01/05 22:55] (current) – Mention that tracks with variable MPEG version and variable MPEG layer can't be played back megadiscman
@@ Line 138: / Line 138: @@
 ==== The file mclist0X.hma ====
-The filename "mclist01.hma" obviuosly derives from "MAClist" which is the name of one of the modules in OpenMG. MAC here stands for "Message authentication code" (see [[http://en.wikipedia.org/wiki/Message_authentication_code]]).
+The filename "mclist01.hma" obviuosly derives from "MAClist" which is the name of one of the modules in OpenMG. MAC here stands for "Message authentication code" (see [[http://en.wikipedia.org/wiki/Message_authentication_code]]). In this case, the DRM license info of each track is considered a message, and the authenticity of these messages is confirmed by adding a 8-byte field to the track information. The MAC list contains all the MAC values of the different tracks and a master MAC that ties these together into one bundle.
+Furthermore, the MAC list contains a header that ties the MAC list to a specific "generation" of a specific medium. The generation is increased every time the DRM info changes.
+The final authentication is done by combining checksums of the medium-specific and the track-specific parts into a master "integrity check value" (ICV) which is stored outside of the FAT file system and only accessible using authenticated and encrypted SCSI commands.
-The contents of "mclist01.hma" are mostly zero, the non-zero part looks like this:
+The contents of "mclist01.hma" are mostly zero until DRM protected tracks are uploaded, the non-zero part looks like this:
 <code>
@@ Line 152: / Line 154: @@
 </code>
-It contains information for the DRM system Sony employs (mostly for the ATRAC encoded tracks). Most notably, it contains a copy of the disc ID (a master key to derive further encryption keys) which is stored at offset 40h in this file. Probably this field is used to tie the file system image to a specific medium. As the disc ID even changes if the medium formatted, a file system image only works until reformatting the medium and only on this medium.
+The MAC list file is divided into three parts. The first 16 bytes are the file header, the next 80 bytes are for disc authentication and the remaining part is for track authentication. Both authentication parts start with an encrypted 3DES key used for authentication, followed by the data to authenticate. To decrypt the authentication key, a master key needs to be used. In the Sony DRM system, these master keys are stored in EKBs, in encrypted form. The disc authentication header contains an indication which EKB (that also means which master key) to use.
+<code>
+BYTES  magic signature "MLST"
+DWORD  unknown purpose
+BYTES  8 unknown bytes, always zero, maybe padding.
+BYTES  16 bytes encrypted 3DES key for authenticating the first MAC list part
+DWORD  Generation number of the DRM info
+BYTES  20 unknown bytes, always zero, maybe padding
+DWORD  ID of the EKB used for decrypting the authentication keys
+C DWORD  4 unknown bytes, always zero, maybe padding
+BYTES  16 bytes disc ID. This is a copy of the real disc ID stored outside of the file system
+BYTES  16 unknown bytes, always zero, maybe padding
+BYTES  16 bytes encrypted 3DES key for authenticating the second MAC list part
+BYTES  4000 * 8 bytes MAC values of tracks. Intersting count, as Hi-MD only has 2048 tracks.
+</code>
-The fields at offset 10h and 60h are presumably encryption keys for 3DES. It's not yet confirmed, whether those are the keys in cleartext or those are the keys in an encrypted form themselves.
 ==== The file 00010012.hma ====
-This file has never been observed to change. It's md5sum always remains the same, being 64c228b7c4c1b02b6c2f1c11bc4148e6. It's presumably a reserved file for future application *or* it might get only touched when transferring ATRAC-audio-data.
+This file contains the "EKB", the enabling key block #00010012. This is the standard key block for HiMD audio. It's the string "EKB" followed by 29 null bytes (making a 32 byte header), and then the contents of the EKB that is also provided in OpenMG as 00010012-umd.EKB. It should never change, but if encrypted payware audio with other EKBs are uploaded, further all-numeric .hma files might appear.
-==== The Audio-Data File ====
+==== The Audio Data File ====
+The audio data file contains the raw audio data.
+It is divided into 16k blocks of the following layout
+<code>
+DWORD  Block Type ("LPCM" for PCM,"A3D " for ATRAC3, "ATX " for ATRAC3+, "SMPA" for MPEG)
+WORD   Number of frames (MP3 blocks only, PCM/Atrac blocks have fixed size)
+WORD   "MCode"
+WORD   Data size in bytes (MP3 blocks only)
+A WORD   Reserved
+C DWORD  Serial number of block in stream
+BYTES  PCM/Atrac: Encrypted DES key for the audio data
+BYTES  PCM/Atrac: DES CBC initialization vector for the audio data
+BYTES  up to 3FC0 of encrypted audio data (ATRAC never uses all 3FC0 bytes)
+FE0 BYTES  Backup of encrypted key (offset 0010)
+FE8 BYTES  8 bytes reserved
+FF0 DWORD  backup of Block Type
+FF4 WORD   reserved
+FF6 WORD   backup of "MCode"
+FF8 DWORD  low order 32 bits of the Content ID
+FFC DWORD  backup of serial number
+</code>
-The audio-data file contains the raw audio-data, which is XOR-encrypted in case of MP3-audio. The key for encryption is created from the DiscID which read/written using special SCSI-commands.
+In case of MP3 audio, the data is XOR encrypted. The key for encryption is created from the DiscID which read/written using special SCSI commands. In case of ATRAC/PCM audio, the data is DES CBC encrypted. See below for more info on encryption.
+Each audio block contains only complete frames. The space in the audio block after the last complete frame is unused - which is especially wasteful for 352kbit/s ATRAC3+, as the frame size is exactly 2K, so 1984 bytes per frame are ignored, i.e. 12% of the block size.
 ==== The _MDHIFI.HMA File ====
@@ Line 246: / Line 290: @@
 Contains 2048 track descriptors of the following layout
 <code>
-DWORD   date of recording (FAT format, 16 bit date, 16 bit time of day)
+DWORD   - date of recording (FAT format, 16 bit date, 16 bit time of day)
-DWORD   "EkbVersion" (Mac MP3: 0, Mac WAV: 10012; upload requirement: == 10012)
+DWORD   ! "EkbVersion" (Mac MP3: 0, Mac WAV: 10012; upload requirement: == 10012)
-WORD    Title (string number)
+WORD    - Title (string number)
-A WORD    Artist (string number)
+A WORD    - Artist (string number)
-C WORD    Album (string number)
+C WORD    - Album (string number)
-E BYTE    Track number (within Album, not on MD)
+E BYTE    - Track number (within Album, not on MD)
-F BYTE    "Mode"
+F BYTE    - "Mode"
-8 BYTES MgrCK (upload requirement: completely zero)
+8 BYTES ! MgrCK (upload requirement: completely zero)
-8 BYTES CMac
+8 BYTES   CMac
-BYTE    CodecId (see below)
+BYTE    - CodecId (see below)
-3 BYTES Codec specific info
+3 BYTES - Codec specific info
-WORD    Part Number (index into Parts Info Table)
+WORD    - Part Number (index into Parts Info Table)
-WORD    Track Number
+WORD    - Track Number
-WORD    Total time (units of seconds)
+WORD    + Total time (units of seconds)
-A BYTE    "Lt" (Mac MP3: 10, Mac WAV: 1; upload requirement: == 1)
+A BYTE    + "Lt" (Mac MP3: 10, Mac WAV: 1; upload requirement: == 1)
-B BYTE    "Dest" (upload requirement: == 1)
+B BYTE    + "Dest" (upload requirement: == 1)
-C WORD    More codec specific info
+C WORD    + More codec specific info
-E WORD    reserved
+E WORD    + reserved
-20 BYTE Content ID (for Mac Transferred Data: 02 03 00 00 + 16 random bytes)
+20 BYTE + Content ID (for Mac Transferred Data: 02 03 00 00 + 16 random bytes)
-DWORD   Start time (FAT format)
+DWORD   + Start of playback license validity (FAT format, or 0 for no restriction)
-DWORD   End time (FAT format)
+DWORD   + End of playback license validity (FAT format, or 0 for no restriction)
-C BYTE    "Xcc" (Mac MP3/WAV: 01; upload requirement: == 03 || == 07)
+C BYTE    + "Xcc" (Mac MP3/WAV: 01; upload requirement: == 03 || == 07)
-D BYTE    "Ct"
+D BYTE    + "Ct"
-E BYTE    "Cc" (Mac MP3: 40, Mac WAV: 44; upload requirement: == 08 || == 48)
+E BYTE    + "Cc" (Mac MP3: 40, Mac WAV: 44; upload requirement: == 08 || == 48)
-F BYTE    "Cn" (Mac MP3: 00, Mac WAV: 3)
+F BYTE    + "Cn" (Mac MP3: 00, Mac WAV: 3)
 </code>
+The +/-/! means: "+" this field is included in MAC calculation. "-" this field does not influence the MAC. "!" this field controls the MAC calculation
 "upload requirement" means that the HiMD Transfer Tool for Mac checks the given condition before allowing an upload of a track to the PC. These checks are independent from the track format. "Mac MP3"/"Mac WAV" means that these values are set by the HiMD transfer tool if downloading that type of music. Probably the fields annotated that way are involved in copy controlling. The first entry in the track info table is a dummy entry that contains the number of the first free track in its Track number field. All free entries are chained using that field.
@@ Line 281: / Line 327: @@
 == Codec specific info for ATRAC3 ==
-  * 0023 Bit rate in bits 0..5
+  * 0021 Bit 1: Set for joint stereo (LP4)
-  ** 18: LP4 (66 kbit/s)
+  * 0022 Bit 7-5: Sample rate
-  ** 26: 105 kbit/s
+    * 00: 32kHz
-  ** 30: LP2 (132 kbit/s)
+    * 20: 44.1kHz
+    * 40: 48kHz
+    * 60: 88.2kHz
+    * 80: 96kHz
+  * 0023 Bit 5-0: Frame size in 8 bytes units
+    * 18: (Frame size 192 bytes) 66 kbit/s (LP4)
+    * 26: (Frame size 304 bytes) 105 kbit/s
+    * 30: (Frame size 384 bytes) 132 kbit/s (LP2)
 == Codec specific info for ATRAC3+ ==
-  * 0023 Bit rate
+  * 0022 Bit 7-5: Sample rate (like ATRAC3)
-  ** 17: 32 kbit/s
+    * 00: 32kHz
-  ** 22: 48 kbit/s
+    * 20: 44.1kHz
-  ** 2E: Hi-LP (64 kbit/s)
+    * 40: 48kHz
-  ** 8B: 192 kbit/s
+    * 60: 88.2kHz
-  ** B9: Hi-SP (256 kbit/s)
+    * 80: 96kHz
-  ** FF: 352 kbit/s
+  * 0022 Bit 4-2: Number of Channels
+    * 04: Mono
+    * 08: Stereo
+  * 0023 Frame size in 8 bytes units (-1)
+    * 17: (Frame size 192 bytes) 32 kbit/s
+    * 22: (Frame size 280 bytes) 48 kbit/s
+    * 2E: (Frame size 376 bytes) 64 kbit/s (Hi-LP)
+    * 8B: (Frame size 1120 bytes) 192 kbit/s
+    * B9: (Frame size 1488 bytes) 256 kbit/s (Hi-SP)
+    * FF: (Frame size 2048 bytes) 352 kbit/s
 == Codec specific info for MPEG ==
@@ Line 299: / Line 361: @@
   * 0022 unused, zero
   * 0023 bit field
-  ** 80: Always set
+    * 80: Always set
-  ** 40: Variable MPEG version
+    * 40: Variable MPEG version (tracks with this bit set cause "cannot play")
-  ** 20: Variable MPEG Layer
+    * 20: Variable MPEG Layer (tracks with this bit set cause "cannot play")
-  ** 10: Variable bitrate
+    * 10: Variable bitrate
-  ** 08: Variable sample rate
+    * 08: Variable sample rate
-  ** 04: Variable channel mode (Joint stereo/split stereo)
+    * 04: Variable channel mode (Joint stereo/split stereo)
-  ** 02: Variable preemphasis
+    * 02: Variable preemphasis
-  ** 01: Always cleared
+    * 01: Always cleared
   * 002C: Bit field:
-  ** C0: MPEG version (C0 = MPEG1, 80 = MPEG2, 00 = MPEG2.5, as in MPEG spec)
+    * C0: MPEG version (C0 = MPEG1, 80 = MPEG2, 00 = MPEG2.5, as in MPEG spec)
-  ** 30: MPEG layer (encoded as in MPEG stream)
+    * 30: MPEG layer (encoded as in MPEG stream)
-  ** 0F: Bitrate number (see MPEG spec)
+    * 0F: Bitrate number (see MPEG spec)
   * 002D: Bit field:
-  ** C0: Sample rate from header (see MPEG spec)
+    * C0: Sample rate from header (see MPEG spec)
-  ** 30: Channel mode (00 = split stereo; 10 = joint stereo; 20 = 2*mono; 30 = mono)
+    * 30: Channel mode (00 = split stereo; 10 = joint stereo; 20 = 2*mono; 30 = mono)
-  ** 0C: Preemphasis (see MPEG spec)
+    * 0C: Preemphasis (see MPEG spec)
-  ** 02, 01: always cleared.
+    * 02, 01: always cleared.
 === The Part Info Table ===
@@ Line 516: / Line 578: @@
 See also [[http://svn.exactcode.de/minidisc/trunk/himd.txt]].
+===== Fragmentation =====
+In order to implement download, we have to understand what happens when tracks are removed inbetween, e.g. have 3 tracks and delete #2. This creates a hole in the data-container.
+The following screenshot illustrates a test with SonicStage and himdtest:
+{{:mp3test.jpg?1000}}
+  - download 3 MP3s onto HiMD, show tracks with himdtest
+  - delete MP3 #2, show tracks with himdtest
+  - add another, larger MP3, show tracks with himdtest
+  - download 3 MP3s onto HiMD, show tracks with himdtest
+  - delete MP3 #2, show tracks with himdtest
+  - add another, smaller MP3, show tracks with himdtest
+==== Fragmentation of the FAT-Filesystem ====
+After deleting track Nr. 2:
+{{:md_after_deleted_track_2.jpg?750}}
+After downloading a new track after deletion of track 2:
+{{:md_after_added_track.jpg?750}}
+During defrag:
+{{:md_while_defragmentation.jpg?750}}
+===== Encryption =====
+==== General picture ====
+{{:himd_encryption.png|Encryption scheme of ATRAC and PCM tracks}}
+Red boxes indicate data stored on HiMD, Black boxes indicate "black boxes" - Operations we don't know how they work. For so called "weakly encrypted tracks" the Track key is 0000000000000000, the EKB ID is 00010012 and the fragment keys are 0000000000000000. This information fixes all input parameters to the Key encryption Key. This value is nearly known from analyzing the Mac software - it is F2266C6464C0D65C. As it is used as an DES key, the low bits of each bytes are unknown.
+==== Fragment keys ====
+Joining two tracks created on an RH10 (so not using zero-key-encryption) with an RH1 walkman (using zero-key-encryption) creates a result of two fragments employing fragment keys. Here are himdtest dumps before/after joining (track 47 is the first part and track 46 is the second part, yes, these numbers are backward, this has nothing to do with this experiment but with disorder of the free list from previous experiments); note the new content ID too:
+=== Before join ===
+<code>
+: 0:07 AT3+ Unknown artist:Unknown title (Unknown album 0)
+@06012 ..   6@06026 (0000000000000000)
+     Content ID: 0203000002000102a36e0000a80d0fe3c9aeae71
+     Key: 167960f39f024906 (EKB 00010012); MAC: 06f503a38982762b
+: 0:07 AT3+ Unknown artist:Unknown title (Unknown album 0)
+@05996 ..   0@06010 (0000000000000000)
+     Content ID: 0203000002000102a36e0000d1ed2108778cad89
+     Key: 5966e351a690c9f4 (EKB 00010012); MAC: f3c0f999f13d48e0
+</code>
+=== After join ===
+<code>
+: 0:13 AT3+ Unknown artist:Unknown title (Unknown album 0)
+@05996 ..   0@06010 (437c871b58fe0a7e)
+@06012 ..   6@06026 (b27b46d967e14fec)
+     Content ID: 02030000020001171d9300005eab87d393f35e44
+     Key: 0000000000000000 (EKB 00010012); MAC: 448b34515a6df04d
+</code>
 ===== Links =====
@@ Line 523: / Line 647: @@
   * [[http://forums.minidisc.org/index.php?showtopic=7944&hl=]] - Simple Hi-MD Music Structure Reader + Sources
   * [[http://forums.minidisc.org/lofiversion/index.php/t10877.html]] - the additional file "text_gXX.hma" from SonicStage 3.1
+  * [[http://minidisc.org/hi-md_faq.html]] - HiMD FAQ from minidisc.org