This section covers the porting of the NetMD Python code to C and integrating it into qhimdtransfer.

UPDATE: As of September, 03 2011, the porting of NetMD to C has been accomplished in the form of an updated libnetmd.

Python NetMD

• command line interface only
• difficult to integrate with QHiMDTransfer (graphical transfer application for HiMD)
• requires Python (version dependency)
• requires some basic knowledge on setting up Python
• not intuitively usable
• code is in some parts highly experimental

The current Python NetMD implementation allows both downloads (all NetMD models) as well digital uploads (MZ-RH1/MZ-RH200 models only) as well as titling, listing and editing of tracks as well as remote controlling NetMD units. Analog capture for upload is supported for models other than MZ-RH1/MZ-RH200. Download code contains a valid but unencumbered root key to allow necessary encryption during transfers. The download code itself is highly experimental, however.

There is already a recent attempt to port the Python code to C. The resulting tarball can be downloaded here: libnetmd. However, it is highly recommend to use the old, much more complete libnetmd (see: http://libnetmd.sourceforge.net/) as a basis and extend it with transfer capabilities.

The current NetMD implementation in Python is divided into several scripts

Utilities:

• lsmd.py - list the contents of any NetMD
• upload.py - transfer tracks from NetMD to the computer (works with MZ-RH1 Walkman only)
• downloadhack.py - allows transfer of tracks from the computer to NetMD (using unencumbered keys)
• dump_md.py - allows analogue transfers of tracks from NetMD to the computer (any NetMD Walkman)
• lsusb.py - sample implementation of lsusb command using usb1.py
• mdctl.py - pdb-based command line to test libnetmd.py methods

Libraries:

• libnetmd.py - implementation of a NetMD library in Python (core module)
• libusb1.py - Ctypes-based python wrapper around libusb1
• usb1.py - object definitions for libusb1.py functions

Since there is already a C library for NetMD available (libnetmd), it would be good idea to use this as a basis for a new implementation instead reinventing the wheel. This old implementation already allows renaming, moving, deleting tracks and so on (see: http://libnetmd.sourceforge.net/) but lacks the capabilities to transfer tracks to the device (in Sony terminology this is referred to as download) and to transfer tracks from the device (Sony calls this upload; upload is supported by the MZ-RH1/200 devices only).

Considering the old libnetmd is used as a basis, the first step would be to import the source code of this library into the git repository of linux-minidisc. Then, since the library has already been without maintenance since 2004, the second step would be compiling the library on a current Linux installation and fix any compiler issues that might arise. After the library has been verified to work in a current environment, it should be tested to work with Sony's MZ-RH1 MiniDisc Walkman (the latest and also last MiniDisc device available) and any problems that might arise should be patched.

After the previous rejuvenation steps, libnetmd is now ready to be extended with download and upload capabilities (note the terms mentioned above), with the help of the PythonNetMD code which serves as sample code for an implementation of both download and upload capabilities (see above summary of the Python scripts available). Since there is with the MZ-RH1 (the MZ-RH200 is a RH1 with different accessoires shipped) essentially only one MiniDisc model available which supports uploads, it's preferred to work on the implementation of the download code first.

While the upload code works unencrypted and thus without the need of any encryption keys, downloads are encrypted and require the data to be encrypted with a key accepted by the NetMD device. Since the keys are proprietary secrets of Sony Corp., we cannot use the original keys for the encryption. Luckily, the keys accepted by the NetMD hardware for an encrypted transfer can be calculated dynamically. In fact, the root key used by the downloadhack.py was encrypted with the help of a secret Sony master key. However, it is not necessary to know the secret Sony key, but it is enough to use the key that is already provided by downloadhack.py. Since we created this key ourselves, we do not have to fear any issues regarding copyright infringement of proprietary code and/or information from Sony Corp.

To summarize the proposed steps for the port:

1. import libnetmd into linux-minidisc, subfolder libnetmd (analogous to libhimd)
2. get libnetmd compile and work on a current release of any Linux distribution (might already work without patching)
3. get libnetmd to work properly with the Sony MZ-RH1 Walkman (see http://libnetmd.sourceforge.net/ for required functionality)
4. with the help of the PythonNetMD code (downloadhack.py), extend libnetmd with download capabilities
5. optional: implement upload capabilities for MZ-RH1 Walkman with the help of upload.py

For documentation, please refer to the the local wiki page netmdlinux with many extremely helpful links for further reading regarding NetMD (includes specifications and patents) and ask any questions on the list https://lists.fu-berlin.de/listinfo/linux-minidisc and/or the IRC channel (#linux-mindisc on FreeNode).

The following section describes the milestones planned for Google Summer of Code 2012 (GSoC 2012). It directly continues where the previous GSoC work left off, albeit there have been some minor bug fixes and improvements on the code which have been committed to the git master repository.

Currently, libnetmd has been integrated in the git master repository and has been updated to support NetMD downloads and USB uploads (supported on the MZ-RH1 only). Downloads are currently supported for LP2 audio files only. The planned changes therefore include the extension of the audio file format support.

As mentioned before, the current version of netmdctl provided as frontend for libnetmd allows for downloads of LP2 files only. The first task will therefore be to extend the code such that also PCM, SP and LP4 audio files are properly recognized. If the input files are provided in a different audio file format, ffmpeg and related utilities can be used to transcode these files prior to transfer. For creating LP2 and LP4 tracks, ATRAC3 encoding is needed at the computer. There is no free ATRAC3 encoder available, so this is only possible with material that is already ATRAC3 encoded. On the other hand, for creating SP tracks, raw PCM material is sent to the NetMD unit, an a transfer utility might decode other media formats on-the-fly.

After libnetmd has gained full functionality, the next step will be integration of the library into the GUI application qhimdtransfer. For this, qhimdtransfer needs to be extended such it will nicely work both with HiMD and NetMD devices. Currently, HiMD devices are supported only via simple file I/O (HiMD act like normal USB mass storage devices).

To summarize the proposed steps:

• redesign the transfer in libnetmd so that it is possible to stream the transfer (currently, the whole file needs to be in (virtual) memory before starting the transfer)
• support other input formats than LP2 encoded WAV in netmdctl
• integrate the upload function of libnetmd into qhimdtransfer
• add download infrastructure to qhimdtransfer
• add support for downloads to NetMD

(a bit apart from the previous list)

• port the analog NetMD copy script to libnetmd

Finally, the four proposed project tasks are to be explained in more detail here.

The netmd download code currently requires the whole data to download in memory at once as soon as the download command is submitted to the NetMD unit. This is not desireable, especially not when downloading raw uncompressed PCM data to record a ATRAC1 (SP) track. You do need to know the total number of frames and the total number of "packets" you download before starting the download, though. As PCM and ATRAC are both constant bit rates (constant frame size), knowing the number of packets is a simple task given the input file. The concept of packets is interesting mainly for DRM stuff, as each "packet" may be encrypted with a different data encrypt key, but it could make sense to re-use this concept for the transfer chunks. Implementing streaming means the interface has to be changed in one of two ways: Either, there is no single "download track" call, but a start download, send partial data, and end download call is exposed separately, or the download function gets passed a pointer to a callback function to read a packet worth of data. The callback function could return arbitrary sized packets, as long as after transferring the initially announced number of packets also the initally announced number of frames is transferred.

Currently, all NetMD functionality can be accessed by means of the command line utility netmdctl only. While such a command line utility might the interface of choice for many power users, most users are used to the comfort of a graphical user interface and usually avoid the command line. We have received many support requests from users asking why their NetMD Walkman when hooked up to the computer doesn't show up in qhimdtransfer. Hence, there is clearly a need to integrate the NetMD functionality into the GUI.

Since qhimdtransfer was originally designed to access libhimd (i.e., the library for HiMD access), some changes will have to be introduced to the Qt code of the UI app to be able to integrate with libnetmd which provides a different interface model as compared to libhimd, however there are also many similarities. For example, both libhimd and libnetmd provide a means to retrieve the tracklist which is used in qhimdtransfer to fill the listview containing the tracks. For libhimd, a Qt model class is used to abstract the contents of a HiMD, the same should be used for libnetmd. The model is then subsequently used to translate access to the listview of the UI to libnetmd (download, upload etc).

Since libnetmd principally provides a means to select the active NetMD device in its open routine, qhimdtransfer should be equipped with a dialogue to allow the user to choose the NetMD device they want to use. We suggest an additional function (in terms of Qt, an "action") in the UI called "Connect to NetMD" and rename the existing "Connect" to "Connect to HiMD". Clicking "Connect to NetMD" will initiate libnetmd to iterate over the connected NetMD devices, present the user a list with the connected devices and then let them choose the device they'd like to use. After that, qhimdtransfer should request the tracklist from libnetmd and build up the model and hence the list view with the track information retrieved from libnetmd.

An appropriate "Disconnect" function should be added as well which will allow disconnecting both HiMD and NetMD devices (depending on what kind of devices is currently actively connected to).

Naturally, digital upload functionality should be enabled only if the NetMD hardware supports it (which is the case for the MZ-RH1 only).

The analog NetMD copy script provides a means to allow uploads for NetMD devices which do not support digital uploads. The code can be found in netmd/dump_md.py in the git repository. The code was written by Vincent Pelletier and he should be contacted on list regarding any questions. In principal, it will already be sufficient if the functionality has been ported to libnetmd and can be used through netmdctl.

This task is optional and is recommended to be tackled after the previous tasks have been finished (order of priority). However, it's naturally up to the student what they choose to work on.