====== PortingNetMD ====== This section covers the porting of the [[netmdpython|NetMD Python ]] code to C and integrating it into [[qhimdtransfer]]. **UPDATE**: As of September, 03 2011, the porting of NetMD to C has been accomplished in the form of an updated //libnetmd//. ===== Background and motivation ===== Python NetMD * command line interface only * difficult to integrate with QHiMDTransfer (graphical transfer application for HiMD) * requires Python (version dependency) * requires some basic knowledge on setting up Python * not intuitively usable * code is in some parts highly experimental ===== Current status ===== The current Python NetMD implementation allows both downloads (all NetMD models) as well digital uploads (MZ-RH1/MZ-RH200 models only) as well as titling, listing and editing of tracks as well as remote controlling NetMD units. Analog capture for upload is supported for models other than MZ-RH1/MZ-RH200. Download code contains a valid but unencumbered root key to allow necessary encryption during transfers. The download code itself is highly experimental, however. There is already a recent attempt to port the Python code to C. The resulting tarball can be downloaded here: [[http://users.physik.fu-berlin.de/~glaubitz/linux-minidisc/mdlib.tar.gz|libnetmd]]. However, it is highly recommend to use the old, much more complete //libnetmd// (see: [[http://libnetmd.sourceforge.net/]]) as a basis and extend it with transfer capabilities. ==== Overview Python NetMD ==== The current NetMD implementation in Python is divided into several scripts Utilities: * //lsmd.py// - list the contents of any NetMD * //upload.py// - transfer tracks from NetMD to the computer (works with MZ-RH1 Walkman only) * //downloadhack.py// - allows transfer of tracks from the computer to NetMD (using unencumbered keys) * //dump_md.py// - allows **analogue** transfers of tracks from NetMD to the computer (any NetMD Walkman) * //lsusb.py// - sample implementation of lsusb command using usb1.py * //mdctl.py// - pdb-based command line to test libnetmd.py methods Libraries: * //libnetmd.py// - implementation of a NetMD library in Python (core module) * //libusb1.py// - Ctypes-based python wrapper around libusb1 * //usb1.py// - object definitions for libusb1.py functions ===== Proposed project plan ===== Since there is already a C library for NetMD available ([[http://sourceforge.net/projects/libnetmd/|libnetmd]]), it would be good idea to use this as a basis for a new implementation instead reinventing the wheel. This old implementation already allows renaming, moving, deleting tracks and so on (see: [[http://libnetmd.sourceforge.net/]]) but lacks the capabilities to transfer tracks //to the device// (in Sony terminology this is referred to as **download**) and to transfer tracks //from the device// (Sony calls this **upload**; upload is supported by the MZ-RH1/200 devices **only**). Considering the old //libnetmd// is used as a basis, the first step would be to import the source code of this library into the git repository of //linux-minidisc//. Then, since the library has already been without maintenance since 2004, the second step would be compiling the library on a current Linux installation and fix any compiler issues that might arise. After the library has been verified to work in a current environment, it should be tested to work with Sony's MZ-RH1 MiniDisc Walkman (the latest and also last MiniDisc device available) and any problems that might arise should be patched. After the previous rejuvenation steps, //libnetmd// is now ready to be extended with download and upload capabilities (note the terms mentioned above), with the help of the [[netmdpython|PythonNetMD code]] which serves as sample code for an implementation of both download and upload capabilities (see above summary of the Python scripts available). Since there is with the MZ-RH1 (the MZ-RH200 is a RH1 with different accessoires shipped) essentially only one MiniDisc model available which supports uploads, it's preferred to work on the implementation of the download code first. While the upload code works unencrypted and thus without the need of any encryption keys, downloads are encrypted and require the data to be encrypted with a key accepted by the NetMD device. Since the keys are proprietary secrets of Sony Corp., we cannot use the original keys for the encryption. Luckily, the keys accepted by the NetMD hardware for an encrypted transfer can be calculated dynamically. In fact, the root key used by the //downloadhack.py// was encrypted with the help of a secret Sony master key. However, it is not necessary to know the secret Sony key, but it is enough to use the key that is already provided by //downloadhack.py//. Since we created this key ourselves, we do not have to fear any issues regarding copyright infringement of proprietary code and/or information from Sony Corp. To summarize the proposed steps for the port: - import [[http://sourceforge.net/projects/libnetmd/|libnetmd]] into //linux-minidisc//, subfolder //libnetmd// (analogous to //libhimd//) - get //libnetmd// compile and work on a current release of any Linux distribution (might already work without patching) - get //libnetmd// to work properly with the Sony MZ-RH1 Walkman (see [[http://libnetmd.sourceforge.net/]] for required functionality) - with the help of the [[netmdpython|PythonNetMD code]] (//downloadhack.py//), extend //libnetmd// with download capabilities - **optional**: implement upload capabilities for MZ-RH1 Walkman with the help of //upload.py// For documentation, please refer to the the local wiki page [[netmdlinux]] with many extremely **helpful** links for further reading regarding NetMD (includes specifications and patents) and ask any questions on the list [[https://lists.fu-berlin.de/listinfo/linux-minidisc]] and/or the IRC channel (#linux-mindisc on FreeNode). ===== Project plan for GSoC 2012 ===== The following section describes the milestones planned for //Google Summer of Code 2012 (GSoC 2012)//. It directly continues where the previous GSoC work left off, albeit there have been some minor bug fixes and improvements on the code which have been committed to the git master repository. ===== Current status ===== Currently, //libnetmd// has been integrated in the git master repository and has been updated to support NetMD downloads and USB uploads (supported on the MZ-RH1 only). Downloads are currently supported for LP2 audio files only. The planned changes therefore include the extension of the audio file format support. ===== Proposed project plan ===== As mentioned before, the current version of //netmdctl// provided as frontend for //libnetmd// allows for downloads of LP2 files only. The first task will therefore be to extend the code such that also PCM, SP and LP4 audio files are properly recognized. If the input files are provided in a different audio file format, ffmpeg and related utilities can be used to transcode these files prior to transfer. For creating LP2 and LP4 tracks, ATRAC3 encoding is needed at the computer. There is no free ATRAC3 encoder available, so this is only possible with material that is already ATRAC3 encoded. On the other hand, for creating SP tracks, raw PCM material is sent to the NetMD unit, an a transfer utility might decode other media formats on-the-fly. After //libnetmd// has gained full functionality, the next step will be integration of the library into the GUI application //qhimdtransfer//. For this, //qhimdtransfer// needs to be extended such it will nicely work both with HiMD and NetMD devices. Currently, HiMD devices are supported only via simple file I/O (HiMD act like normal USB mass storage devices). To summarize the proposed steps: * redesign the transfer in //libnetmd// so that it is possible to stream the transfer (currently, the whole file needs to be in (virtual) memory before starting the transfer) * support other input formats than LP2 encoded WAV in //netmdctl// * integrate the upload function of //libnetmd// into //qhimdtransfer// * add download infrastructure to //qhimdtransfer// * add support for downloads to NetMD (a bit apart from the previous list) * port the analog NetMD copy script to libnetmd ===== Detailed description of the tasks ===== Finally, the four proposed project tasks are to be explained in more detail here. ==== Redesigning the transfer code for streaming ==== The netmd download code currently requires the whole data to download in memory at once as soon as the download command is submitted to the NetMD unit. This is not desireable, especially not when downloading raw uncompressed PCM data to record a ATRAC1 (SP) track. You do need to know the total number of frames and the total number of "packets" you download before starting the download, though. As PCM and ATRAC are both constant bit rates (constant frame size), knowing the number of packets is a simple task given the input file. The concept of packets is interesting mainly for DRM stuff, as each "packet" may be encrypted with a different data encrypt key, but it could make sense to re-use this concept for the transfer chunks. Implementing streaming means the interface has to be changed in one of two ways: Either, there is no single "download track" call, but a start download, send partial data, and end download call is exposed separately, or the download function gets passed a pointer to a callback function to read a packet worth of data. The callback function could return arbitrary sized packets, as long as after transferring the initially announced number of packets also the initally announced number of frames is transferred. ==== Supporting other formats than LP2 encoded WAV files ==== ==== Integration into qhimdtransfer ==== Currently, all NetMD functionality can be accessed by means of the command line utility //netmdctl// only. While such a command line utility might the interface of choice for many power users, most users are used to the comfort of a graphical user interface and usually avoid the command line. We have received many support requests from users asking why their NetMD Walkman when hooked up to the computer doesn't show up in //qhimdtransfer//. Hence, there is clearly a need to integrate the NetMD functionality into the GUI. Since //qhimdtransfer// was originally designed to access //libhimd// (i.e., the library for HiMD access), some changes will have to be introduced to the Qt code of the UI app to be able to integrate with //libnetmd// which provides a different interface model as compared to //libhimd//, however there are also many similarities. For example, both //libhimd// and //libnetmd// provide a means to retrieve the tracklist which is used in //qhimdtransfer// to fill the listview containing the tracks. For //libhimd//, a Qt model class is used to abstract the contents of a HiMD, the same should be used for //libnetmd//. The model is then subsequently used to translate access to the listview of the UI to //libnetmd// (download, upload etc). Since //libnetmd// principally provides a means to select the active NetMD device in its //open// routine, //qhimdtransfer// should be equipped with a dialogue to allow the user to choose the NetMD device they want to use. We suggest an additional function (in terms of Qt, an "action") in the UI called "Connect to NetMD" and rename the existing "Connect" to "Connect to HiMD". Clicking "Connect to NetMD" will initiate //libnetmd// to iterate over the connected NetMD devices, present the user a list with the connected devices and then let them choose the device they'd like to use. After that, //qhimdtransfer// should request the tracklist from //libnetmd// and build up the model and hence the list view with the track information retrieved from //libnetmd//. An appropriate "Disconnect" function should be added as well which will allow disconnecting both HiMD and NetMD devices (depending on what kind of devices is currently actively connected to). Naturally, digital upload functionality should be enabled only if the NetMD hardware supports it (which is the case for the MZ-RH1 only). ==== Port the analog NetMD copy script ==== The analog NetMD copy script provides a means to allow uploads for NetMD devices which do not support digital uploads. The code can be found in **netmd/dump_md.py** in the git repository. The code was written by Vincent Pelletier and he should be contacted on list regarding any questions. In principal, it will already be sufficient if the functionality has been ported to //libnetmd// and can be used through //netmdctl//. This task is **optional** and is recommended to be tackled after the previous tasks have been finished (order of priority). However, it's naturally up to the student what they choose to work on.