Integrating MusicIP with SqueezeCenter

Scope

This Wiki entry only addresses SC integration with MiP headless.

Integrating MusicIP with SqueezeCenter

Background

This Wiki entry attempts to serve the following purposes:

• provide a very basic overview of what MusicIP (MiP) does and therefore what you stand to gain by integrating it with SqueezeCenter (SC).
• Explain the underlying logic of MusicIP integration (insofar as I've been able to decipher it). This should help you to help yourself and get the best out of the integration with SC
• Provide a step by step HOWTO explaining what to do to get the integration working

Caveat: I run Linux as my desktop OS and I also have SC/MiP running on a Linux based NAS, so I may not be as clear on some Windows aspects as users more familiar with Windows may be.

What is MusicIP?

In a nutshell, MiP is an acoustic analysis and playlisting engine that allows you to dynamically generate playlists using the acoustic properties of individual tracks/ albums or a combination of tracks/ albums as the seed. MiP uses the acoustic properties of the seed to dynamically generate a playlist of tracks from your music library. Various aspects of the seed can be fine-tuned to inform the mix. Further information can be obtained at MusicIP.com

MiP can operate as a standalone tool/ playlist generator (pushing playlists to SC) via MiP GUI, or it can be integrated into SC via MiP headless. The standalone tool is generally referred to as MusicIP Mixer (MiP GUI) and the version that can be integrated with SC is known as MusicIP Server or MusicIP headless. Headless refers to it running in the background as a server with its UI accessible via a web interface. By default the web interface for MiP headless is accessed via http://localhost:10002 (assuming of course it is in fact installed and running on the local PC).

Using MusicIP

To use MiP it is first necessary to have it analyse your music library. The analysis process is CPU intensive and requires Internet access to expedite the process. It is theoretically possible to do the analysis without Internet access, however, this will very significantly increase analysis time. The reason for this is that MiP calculates a unique hash based on the song's acoustic signature and is usually able to use this hash to identify the song in its vast online MusicDNS database, and retrieve the acoustic attributes relating to that song, negating the need for further analysis on your PC.

Once the analysis is done, MiP stores the results of the analysis in a .m3lib database file and uses the information in this database to generate the playlists. Unfortunately, MiP GUI is not entirely stable (albeit it is improving with time) and is known to occasionally corrupt the m3lib database. This is particularly the case if your music library is large, which results in a correspondingly large MiP database. For this reason, I recommend that you use MiP's capability of writing the analysis information to the tags of your audio files. This way, if the MiP database is ever corrupted there is no need to reperform the analysis, ever – you'd simply have MiP rescan your library and it will rebuild its database using the analysis information saved to the audio file's tag.

There have been confirmed reports of MiP corrupting files during the archiving process, however, this related to much earlier versions of MiP. Additionally, it was only the tag header that was corrupted – the audio stream was unaffected. At the time of writing MiP 1.8 was current.

Doing the analysis

Analysis is the voodoo required for MusicIP to work its mixing magic. Analysis of your audio files can be done using MiP GUI for Linux/ Windows or Genpuid for Linux/ Windows. Genpuid is a command line based tool that has the added flexibility of allowing you to invoke multiple concurrent analyses on different portions of your audio library.

Some MiP analysis lessons I've learned:

• if you're using an older/ slower PC to do the analysis work, do your scanning overnight or run it as a low priority process in the background and accept that it may take days/ perhaps weeks depending on the size of your library - the end result is worth the wait
• if you're doing analysis across a network and a song comes back unanalyzable, try copying the song to your local drive and reanalyzing. This is particularly the case if you're running a Windows box (don't ask me why, but I suspect it has something to do with the CIFS protocol)
• if you have a dual/quad core PC you can pretty much do the analysis at high priority without impacting the PC's overall performance.
• if you have a dual/quad core PC you can run multiple instances of Genpuid, with each pointed at a different portion of your library = get the job done a hell of a lot faster.

To facilitate this you may need to temporarily move your library structure around to be able to point Genpuid to discrete parts of the library. For example, my library is stored on my NAS as follows:

• AudioLIB/0-E/
• AudioLIB/F-J/
• AudioLIB/K-O/
• AudioLIB/P-R/
• AudioLIB/S-T/
• AudioLIB/U-Z/
• AudioLIB/playlists/

with a separate sub-folder for each album. This means I can launch two or perhaps three instances of Genpuid to run in parallel, with each instance pointing at a different branch of the abovementioned tree. So long as I have the CPUs available, each instance of Genpuid will use a different CPU and the analysis will be completed that much faster.

If you choose to use Genpuid, you will need to register a profile with musicip.com and obtain a DNSkey which must be included at the command line when invoking Genpuid.

• there is nothing stopping you using SC whilst analysis is being performed (unless you have it set to archive tags on the fly. Even then, it is easy to see which part of your library is being worked on and thus avoid playing it) so keep enjoying your music and let MiP do its thing in the background.
• archive the analysis results to tags. A large MiP database is prone to corruption by MiP GUI as it doesn't handle very large databases well (my MiP analysis database is around 75MB, and MiP GUI crashes frequently if you try to navigate the database). This is especially the case on MiP GUI on Linux (whilst it's the poor cousin of the Windows version and contains only a subset of the Windows GUI's functionality, the GUI is not used in integration with SC).

Option 1: Using Genpuid

StuartH: I had started a page describing automating rescans with GenPuid here: Automating_MusicIP_with_GenPuid - it might be helpful to merge the two.

Genpuid syntax

genpuid dns-key [options] [file1] [file2] [...]

Options are as follows:

dns-key

Get a MusicDNS key from http://www.musicip.com/dns

-archive

Archive tags when songs are identified

-ext=[opt]

Return extended metadata (XML format only)

-logex

Write analysis results to a log file

m3lib=[cache.m3lib]

Save cache in [cache.m3lib] after processing

-r

Process files recursively

-rmd=#

Return metadata (XML format only, #=1 or 2)

-v

Print version and exit

-xml

[XML format]

Personally, I use Genpuid as follows:

genpuid [dnskey] -r -logex -m3lib=cache.m3lib -archive .

• Basically it means run genpuid on the current folder, traversing all child folders, write activities to a log file, generate a MiP database called cache.m3lib containing the analysis results and archive the analysis results to the file tags.

• Note the period at the end of the command line – it implies the current directory is the one to be traversed, which is in general my approach – navigate to that part of the tree you want to work on and have Genpuid analyse the contents of that folder and all its children.
• By having Genpuid create an analysis database I can load it in MiP GUI and inspect the results of the analysis if I choose to. For all intents and purposes however, these analysis databases of a subset of a library are redundant and can be deleted [or not generated in the first place].

Option 2 - Using MiP GUI to do the analysis (Linux or Windows)

If you prefer staying away from the command line then you can do the analysis and archiving via MiP GUI. All that is required is that you execute MiP GUI, point it to your library folder and tell it to get going with the analysis. If you've a large library, I'd definitely recommend you do the analysis on only a subset at a time otherwise you risk the database being corrupted and having to reperform the analysis.

There are two ways to write the analysis to the audio tags:

• The analysis data can be written to tag manually by selecting the songs whose analysis you wish to archive and selecting “Archive Analysis”
• tags can be automatically updated by configuring MiP GUI to write analysis data to tags as part of the analysis process. This option can be found under Preferences.

Setting things up

MiP can be a real pain in the proverbial ass to work with, especially as analysis databases are not transportable across different network and/or physical file paths unless you've set up a "Device" in MiP. Even then, it remains a pain in the ass to work with unless you take some steps to simplify it. Through lots of experimenting and questioning of the developers I've come to the following means of using it, which has consistently reproducible results.

Quick tips

• do the analysis as mentioned above and archive the analysis to tag
• download Spicefly's replacement headless HTML UI - it exposes additional functionality and is simple to use. It is available from http://www.spicefly.com/
• build a MiP database for SC integration via the headless API once your analysis is complete and archived.
• before having SC run the MiP integration scan, it pays to flush the MiP cache like so: http://localhost:10002/api/flush
• configure SC to use MiP and remove any references in SC's setup to the library path - SC will take its cue from MiP's database and it will traverse the paths to your audio files as referenced in the MiP DB and find your artwork etc. SC will still take Artist, album and Track data from the audio files themselves. This approach also potentially eliminates SC doing a triple scan - 1^st the MiP database, then the folders/ files it points to and then the path you've pointed to in SC's library path. As far as I can ascertain, the last scan would be redundant and just adds to your scanning time which we all know and love as it is :)
• make sure MiP headless loads BEFORE SC – if this doesn't happen there is nothing you can do to get SC to see/ integrate with MiP. For Linux users detailed instructions are set out below. If you're a Windows user, refer to Spicefly's guides at http://www.spicefly.com/ - it contains everything you need to know.

Installation of MiP headless

You need to install MiP headless on the same machine you're running SC on. In general terms MiP headless is installed by default when you install MiP, whether it be Linux or Windows. Remember, for integration to work it is essential that MiP headless be loaded PRIOR to SC on startup.

General instructions on how to configure and start MiP headless for Linux, MAC and Windows platforms are located at http://www.musicip.com/mixer/server.jsp

MiP headless under Linux

Download MusicIP for Linux from MusicIP.com and save the file to your desktop or a temp folder. Open a terminal window and change to the temp folder, extract the archive and optionally remove it:

cd ~/egd/Desktop
gzip -dc MusicMixer_x86_1.8.tgz | tar xf -
rm MusicMixer_x86_1.8.tgz

The above can also be accomplished by right clicking on the archive and selecting "Extract here".

The extraction process will create a folder called MusicIP, containing a subfolder called MusicMagicMixer . It's the MusicMagicMixer folder and its contents you're interested in. Rename it to MusicIP (or leave it as it is and modify guidance that follows as required to adjust for the difference in folder names) and using drag and drop operation to move the folder to /home/<username>/. One of the files that is extracted from the archive is called "mmserver". This is the startup script for MiP headless. It needs to be customised to reflect your configuration and then added to your Linux boot process.

The default file looks as follows:

#! /bin/sh
# NON-PRIVIELEGED USER TO RUN MUSICMAGICSERVER.
USER=
# PATH TO THE MUSICMAGICMIXERSERVER 
export MUSICHOME=
case $1 in
    start)
        su - $USER -c $MUSICHOME"MusicMagicServer start  & > /dev/null" 
        echo "Running MusicMagicServer"
        exit
        ;;
    stop)
        su - $USER -c $MUSICHOME"MusicMagicServer stop  & > /dev/null" 
        echo "Stopped MusicMagicServer"
        exit
        ;;
    *)
        echo "Usage: /etc/rc.d/init.d/mmserver { start | stop }"
        exit
        ;;
esac

There are two variables that need be customised:

• USER
• MUSICHOME

"USER=" should be set to your Ubuntu/ Debian login name e.g. "USER=egd".

"MUSICHOME=" should be edited to reflect the location of your MusicIP installation (if you're following this step by step it would become: MUSICHOME=/home/<username>/MusicIP/

You can then copy mmserver to /etc/init.d/ as follows:

sudo cp mmserver /etc/init.d/

Now you need to tell Ubuntu/ Debian to start MiP BEFORE SqueezeCenter. Install Boot-up-manager (BUM) and then run it (it will also appear as an entry in your Ubuntu System/Administration menu):

sudo apt-get install bum
sudo bum

• Click on BUM's Advanced option and navigate to its Services tab.
• Find SqueezeCenter, right click on it and select "Change Start/Stop priority". Note the numeric priority assigned to it. Hit Cancel.
• Find mmserver, right click on it and select "Change Start/Stop priority". Set the numeric priority assigned to it to a higher absolute umber than that assigned to SqueezeCenter. Hit OK.
• Make sure there is a tick in the Activate field for mmserver (it's not enabled by default).
• Restart your Ubuntu machine (if you know what you're doing you can stop SC, start MiP and then restart SC, thus avoiding the need for a reboot, however, a reboot is probably warranted to ensure it all works when you start up your machine).

MiP headless under Windows

Download MusicIP for Windows from MusicIP.com and install MusicIP. Thereafter, point your browser to Spicefly's guides at http://www.spicefly.com/ - it contains everything you need to know to get MiP and SC talking under Windows and Vista. When you're done there, the rest of this Wiki should be reviewed also.

Building the MiP database

Once you've configured MiP, you need to build a MiP analysis database and configure SC to use MiP. To build the MiP database, access MiP's HTTP API via (http://localhost:10002/), enter the path to your music library in the text field next to the 'Add Music' button, then click the button to have MiP traverse your library and create the analysis database. This is generally a pretty quick process as all that's happening is all audio files in the path you've specified are opened and the analysis tags read and written to the MiP database. Note, the MiP database will be stored and named as specified in mmm.ini's cache= setting. MiP will look for mmm.ini in /home/<user>/.MusicMagic - if you put it anywhere else it will be ignored. In my case, as MiP and SC are running on my NAS, it is set to cache=/raid/data/module/MusicIP/default.m3lib.

Configuring SC to use MiP

This is done by:

• enabling MiP in the Plugins tab and then selecting "Use MusicIP" under MiP settings.
• under basic settings, remove the path entry for Music Folder (SC will get its info from the MiP database).
• When you're first building the SC database using MiP integration I recommend disabling Plugins/MusicIP settings/ MusicIP Reload. This is done by setting MusicIP Reload Interval to 0. The reason for doing this is because a large library may result in SC initiating a scan of the MiP database while it is still being built. SC then detects changes and in turn tries to do an update scan and so on...you get the picture. Once this initial scan is done you can reset the parameter to a suitable timeframe. Personally, I prefer to explicitly initiate an update scan at a time of my choosing. That way I know I can listen to music without the possibility of SC kicking off a rescan whilst I'm trying to enjoy my music/ mixes.
• Finally, do a clear and rescan from within SC. When it's done your albums and tracks should have an “m” indicator next to them, selecting the “m” will generate a mix based on the album/ track.
• On the IR remote, press and hold play to generate a playlist based on the current album/track and press play again to start the playlist.
• On the Controller you can do the same but you'll need SC >= 7.1.

What happens when I add new tracks to my library? Do I have to explicitly select them for analysis, or will MiP headless detect new tracks in the library?

You will need to explicitly analyse and add new tracks to the MiP database. Analyzing the tracks be done independently via MiP GUI, Genpuid or via MiP headless (assuming the machine running MiP headless is Internet facing) and archiving analysis to tags. Getting the analysis done and MiP headless to recognise newly added tracks is in my experience best done by way of the methods outlined below.

SC can be set to autodetect changes in the MiP database and there is thus no need to explicitly have SC rescan, albeit there is nothing stopping you from doing this. SC's behaviour in this regard is controlled through the Plugins/MusicIP settings/ MusicIP Reload Interval parameter.

Adding/ deleting/ moving tracks in your audio library will necessitate updating your MiP database. You have two choices here:

• rebuild the MiP database: Rebuilding the MiP database is done as described above, but is not the preferred/ recommended method to deal with changes to your music library because it will mean that SqueezeCenter will be forced to rescan your entire library when in all likelihood you're itching to listen to your new album/ tracks.
• update the MiP database: This is the preferred course of action as it allows SC to scan only the changes/ additions rather than rebuild its entire library. To do this on a headless server WITHOUT using genpuid, go to the web UI (localhost:1002) and do the following steps: 1. Add Music (yes, add your library's root path again, it will recognize your known tracks). 2. Refresh Songs. 3. Start Validation. It should now validate your added tracks.

If you have genpuid installed, proceed as follows:

Maintaining the MiP database

Updating the MiP database is most easily accomplished as follows:

• rip, tag and copy/ move your new albums/ tracks to wherever it is they belong in your audio library folder structure
• run genpuid in the root folder of your music library, having it traverse all subdirectories and be sure to instruct it to use the existing cache file, i.e. that located at /home/<username>/.MusicMagic/ or wherever you've located your cache file. Don't worry - it will skip over files that have already been analyzed and will append the cache file/ database with changes and/ or additions.
• once this is completed, carry out a "Reload Cache" via the MiP HTTP API
• finally, have SC do a "Look for new and changed music" scan. Your additions/ changes/ deletions will be incorporated.

The genpuid syntax to accomplish the above (under Linux) is as follows:

genpuid <DNSKEY> -r -m3lib=/home/<username>/.MusicMagic/default.m3lib -archive .

Note: the inclusion of -archive will cause analysis to be written to tags. If you don't favour this approach (and therefore by implication run the risk of possibly having to reperform analysis at some future date) omit it.

For Windows, the syntax would be the same other than you would need to point to your cache file having regard to its Windows location.

I have deleted and moved some folders in my music library. How can I get the MusicIP database to recognise these changes without having to analyse and validate the whole library again ?

So long as you've archived analysis to tags, no further analysis is necessary and all that needs to be done is to update the MiP analysis database through the MiP HTTP API as described above in Maintaining the MiP database.

Conclusion

This Wiki entry attempts to capture my understanding of how to get MiP and SC integrated. It has been based on tips and assistance I've gleaned from the Slimdevices and MusicIP forums as well as assistance I've received from members of both forums in response to specific questions. Thanks to all involved. I hope this is of some use to others wanting to get MusicIP and SqueezeCenter talking.

Happy listening :)