Backing up Your Music
From SqueezeboxWiki
These step by step instructions were done on a VortexBox NAS. These steps may also work on any Linux based music server.
We all love our valuable, lossless, and bit-perfect audio data. And we have invested so much time in doing all this ripping, naming, MusicBrainz and tagging stuff.
So what about backup?
If you use a VortexBox (v0.5 or newer) and need to backup your music library not to a local external disc, but across the network to another machine, this "How To" is for you!
In the following, will describe a means of doing an automatic, fast, bit-perfect, incremental backup to and from your VortexBox, which will work across your network to and from Linux and/or Windows computers.
Part 1: (Linux) The basic rsync command
Linux users will now probably smile, since they of course know about the great rsync tool already, and might think of something like Code:
rsync -v -rlt vortexbox::files/music/ ~/backup/vortexbox/music/
And yes, this will actually work. (Thanks, Andrew, for including a default rsync daemon!) So you could type it manually, include it in a script, put it in your crontab … whatever.
End of story for Linuxers just wanting to backup their Vortexbox.
Part 2: (Linux, Windows) Make your VortexBox writable
But what if you have a new VortexBox and need to transfer your audio files there? Or you want to keep your other Linux/Windows machine’s Music Library in sync with your VortexBox?
Sure, you could map your \\vortexbox\files\music folder in Windows and use the supplied Samba shared folders to copy your music from a Windows machine. Or use any other method available under Linux. Or use one of those fancy-looking, slow and buggy "file synchronization" programs for Windows.
Forget it. They will all be slower, probably not copy all your files, or just look at some "file date & time stamp" instead of actually verifying what’s in the file. Most Windows programs will also not know how to transfer just changed parts of a file. Say, you just have changed a tag in a big FLAC file, so theoretically only the changed part had to be transferred. Most programs (especially Windows) will instead copy the whole multi-megabyte file! Imagine a 50,000 file library, and you’ve just changed some tags in 18,000 of them … It will probably take days …
No good for real music lovers. No good for impatient people like me.
So we will set up our VortexBox to allow writing via rsync, which can do bit-perfect, partial file transfers and always looks at the file contents, not only some date stamps.
Log in to your VortexBox using an attached monitor/keyboard, or a remote ssh session. From Windows, I recommend using a program called PuTTY for this. (Remember to set Window->Translation to UTF-8 so you will see correct characters displayed for filenames with non-ASCII characters.)
Use nano or an editor of your choice to modify the file /var/vortexbox_load/rsyncd.conf as follows: Code:
max connections = 3
log file = /var/log/rsync.log
timeout = 300
[files]
comment = VortexBox files
path = /storage
read only = no
list = yes
uid = root
gid = root
Do the same changes for the file /etc/rsyncd.conf.
Reboot the VortexBox (by typing reboot at the command prompt) to see that your changes are active and working, even after a reboot.
Your VortexBox can now be written to from another machine, effectively synchronizing the Music Library on your "other" machine to the VortexBox. Useful if you keep your "main" library on another machine, or just want to keep two machines in sync.
Take care: The music library on your "other" machine will overwrite what’s on your VortexBox, so be very careful when doing this! In the worst case, you could destroy your whole VortexBox Music Library!
Also, your library should follow VortexBox’s standard for the /storage/music folder, i.e., be separated like this: music/flac music/mp3 music/playlist
Linuxers can now synchronize VortexBox’ library with their "other" library by using an rsync command like this: Code:
rsync -v -rlt ~/music/ vortexbox::files/music/
End of story for Linuxers.
Part 3: (Windows) Make Windows use rsync in the most comfortable way
Now this is the most complicated part, but be assured that your efforts will pays off — both in backup time saved and you being able to sleep again, knowing that you have a bit-perfect backup!
Unfortunately, Windows doesn’t know anything about UNIX or Linux commands, and doesn’t understand their protocols. So we have to install some little "Linux emulation" (cygwin). Again unfortunate: Cygwin doesn’t know how to translate the UTF-8 filenames we use on Linux into the UTF-16 Unicode used by modern Windows (NT, XP, Vista, Windows 7, …).
Even more unfortunate: Windows users like to use nice windowed GUIs, not cryptical command lines.
So how do we solve all these problems?
During the last months, I must have tried out about a zillion different "file syncing" programs for Windows. To put it bluntly: Most of them are crap, even those you have pay for.
We want the best, most usable, most reliable, fastest, and cheapest solution around. Okay, so here we go:
Part 3a: Install DeltaCopy
Go to DeltaCopy’s site and download "DeltaCopy with Installer" (top right of screen). It is a ZIP file, so you need some unzipping program. Extract it to some temporary location and run the setup.exe that’s included. Do not "Launch the application" at the end of the install!
You might also want to download the DeltaCopy Documentation (PDF file).
DeltaCopy will usually be installed in a folder called C:\Program Files\Synametrics Technologies\DeltaCopy. The installation folder might differ if you changed it at install time, if you run a non-English Windows version, or if you run Windows Vista.
Part 3b: Install cygwin patch for correct file name translation
The cygwin used by DeltaCopy doesn’t know how to translate non-ASCII filenames to Windows Unicode. Windows uses UTF-16 while your VortexBox uses UTF-8. If not using this patch, filenames containing foreign characters will be unusable on either your Windows or your VortexBox system, depending on copy direction.
Fortunately, a Japanese programmer has developed a modified cygwin that can correctly create correct filenames for both Windows Unicode and Linux UTF-8. Go to his website and download the compiled Windows binary cygwin1-dll-20-11-18.tar.bz2. This again is a compressed tar.bz2 file, so you’ll need a compatible unpacker to extract this to a temporary location on your disc. I suggest using 7-zip.
The archive contains a DLL file called cygwin1.dll. Copy this file over the cygwin1.dll file that is included in your DeltaCopy installation directory.
Hint: Cygwin uses shared memory, so if you use a standalone cygwin installation or other software that uses cygwin1.dll, you’ll need to replace cygwin1.dll for all other software that uses it! If that is the case, it might be wise to reboot your Windows system after the change in order to make sure that only the new cygwin1.dll is used.
Part 3c: Preparation and terminology used
DeltaCopy uses a "client/server" terminology. For our purposes, your VortexBox will always be the Server, your Windows machine will be the Client. A "Backup" means copying from the Client (Windows) to the Server (VortexBox). A "Restore" means copying back from the Server (VortexBox) to the Client (Windows).
Caution! Be aware that a perfect backup is always a one-way street. So if you accidently erase all your music files on the Windows machine, then do a "Backup" to your VortexBox, you will end up with a perfect copy — your VortexBox will also have lost all your valuable music files! So be very careful when trying all this out! The safest method might be to first try it out with some other folder you don’t really use, and put a few test files in that!
If you want to make an exact copy of music from your Windows machine to your VortexBox, you need to have the same folder structure on your Windows machine as your VortexBox requires for operation, i.e. (at minimum): M:\music\flac M:\music\mp3 M:\music\playlist
VortexBox will take the files it plays through SqueezeCenter from music/flac (which is actually /storage/music/flac as seen from the VortexBox). SqueezeCenter playlists are stored in music/playlist and music/mp3 just holds MP3 copies of your lossless FLAC files.
Part 3d: Setting up DeltaCopy for work
Assuming you already have some music on your VortexBox and your Windows backup folders are still empty (or already contain an exact copy of what’s on your VortexBox), we will now set up a "Profile" for DeltaCopy.
Remember that in DeltaCopy terminology, you will do a "Backup" if copying from Windows to VortexBox, a "Restore" if you copy from VortexBox to Windows.
Start DeltaCopy using the START ->*All Programs -> DeltaCopy -> DeltaCopy Client.
Double-click on "<Add New Profile>" and enter the following in the "New Backup Profile" window: Profile Name: Backup Music to VortexBox Server IP/Host Name: vortexbox (or VortexBox’s IP number if this doesn’t work out) Virtual Directory Name: files
A new profile will be created. Click on its name in the left-hand pane, then click on "Add Folder" at the right side to add the folder in Windows that shall be the backup/base folder of your music library on the Windows machine (corresponds to /storage/music on your VortexBox), i.e. M:\music.
Caution: DeltaCopy will suggest the rightmost subfolder’s name as the target name for your VortexBox, so if your music folder was called, say, "M:\Backup\Musik", DeltaCopy would copy everything to /storage/Musik on your VortexBox!
So be sure to right-click on the newly added folder in the right pane once, select "Modify Target Path..." from the context menu and verify that it says music/. Change if needed, then click "OK".
In the right pane, select the "Options" tab and select these options: [x] Recursive and preserve time (recommended) [_] Compress files during transfer [x] Remove deleted files from server [_] Skip files that are newer on the server [_] Connect via SSH Additional Parameters: (leave empty)
Hint: Under "Copying Schedule" below, you can specify automatic, unattended backups using the Windows Scheduler later on. Simply click on "Modify Schedule" to specify when and how often the backup will be done. I suggest NOT using this at this time but instead use manual backups/restores, since this function can only do "Backups", i.e. mirror from your Windows machine to your VortexBox. You might usually want to do it the other way round, i.e. do a "Restore" instead (copy from your VortexBox to Windows).
For this case, it makes more sense to use the DeltaCopy Server on Windows, and set up a regular rsync in your VortexBox’s crontab. But I’ll leave this as an exercise for the user ;-)
Part 3e: Copying all music from your VortexBox to Windows for the first time!
We assume that you already have music on your VortexBox, and enough space somewhere on your Windows box to make a perfect mirror. You might have connected an external USB disc to your machine, for example. (I use this setup, and my music is in M:\music.)
Before we have a go at this, you might want to verify if the "Restore" command is correctly set up. (Remember: "Restore" = Mirror the Server (VortexBox) to your Client (Windows machine).)
Right-click the profile name "Backup Music to VortexBox" and select "Display Restore Command" from the context menu. It should show something like this: Code:
Executing: rsync.exe -v -rlt "vortexbox::files/music/" "/cygdrive/M/music/" Profile 'Backup Music to Vortexbox' executed in 0 milliseconds. It ran successfully.
If all this is okay, we now click "Close" in this window and prepare for the first REAL "Restore" of all music from the VortexBox!
Right-click the profile name "Backup Music to VortexBox" and select "Restore...".
This will build a list of files to check/transfer, initiate the copying process, and probably take a very long time, since on the first copy of course everything must be copied! You might want to run this overnight and check the results next day.
The copying speed depends very much on the disc(s) speed and the network available.
Hint: Gigabit network cards and switches are cheap nowadays, you should definitely get some if doing this more often. (I do an incremental backup every night.)
Hint 2: For German users of AVM Fritz!Box 7050 routers: Do not use the LAN1 and LAN2 ports to copy files between computers! It can be extremely slow, especially when using VoIP telephony at the same time! It pays investing 10-15 Euros for a switch and connecting that to LAN1 only.
Hint 3: Due to the unbelievable amount of character set issues we experience in the IT world, do not be alarmed if the status window shows "funny filenames" — if using Windows XP or newer, the filenames with foreign characters will show messed up in the status window but still be correctly converted between UTF-8 (Linux) and UTF-16 (Windows). You can verify this by opening Windows Explorer and checking the copied files.
Part 3f: Doing it all again after changing/adding files on the VortexBox
The first time through, everything took a long time. Now, if you have added more files to your VortexBox, or made some changes, it’s time to do it all over again!
Proceed as in step (3e) — you can leave out the "Display Restore Command" this time — and see what happens: DeltaCopy will build a list of files, then check through all of them (which might take a few seconds per file), then add all your new files and delete all those that you have deleted on the VortexBox in the meantime. Presto! — Perfect mirror!
You might wonder why DeltaCopy uses a few seconds per file where other programs just skip them. Other software often uses only the file’s date/time to "check" if they have changed — but DeltaCopy (or actually the underlying rsync) really checks each file’s content, builds blocks and checksums them, and — if needed — intelligently copies only those parts of a file that have changed, maybe only the tags! — Well worth a few seconds for perfect safety, don’t you think?
Well, now I’ve told you the real story, and all that’s left to say is: Enjoy!
Part 4: The other way round? I want to mirror my music TO the VortexBox!
Okay. I promised to tell you the other way round: Already have an organized music library on Windows and wanting to mirror that onto your VortexBox — Maybe you just got a new VortexBox, after all?
Safety first: You might want to create a "Test" profile that goes to another folder (like "test") to try all this out! This is also a good experiment if you later wish to backup other data from your client machines and use the VortexBox as central file storage.
In our "client/server world", what we want to do is a "Backup" from the Windows machine to the VortexBox. By the way, you can of course also "mis-use" your VortexBox as a big file server to store real backups from your other machines in a central location, maybe your important documents on /storage/documents in your VortexBox … Possibilities are endless, once you have understood and set up what can be done using rsync!
I cannot say it often enough: Be careful and understand what you do! Remember, if you have an empty set of Windows folders and "backup" these to your VortexBox, you will get a perfect mirror — nothing!
Also please remember that you need a "VortexBox-compatible" file structure before copying things to the VortexBox, i.e. (at minimum): M:\music\flac M:\music\mp3 M:\music\playlist
That said, fire up DeltaCopy, select the profile "Backup Music to VortexBox", right-click it and select "Display Run Command" this time. It should show something like this: Code:
Executing: rsync.exe -v -rlt --delete "/cygdrive/M/music/" "vortexbox::files/music/" Profile 'Backup Music to Vortexbox' executed in 0 milliseconds. It ran successfully.
If all this is okay, we now click "Close" in this window and prepare for the "Backup" of all music to the VortexBox!
Right-click the profile name "Backup Music to VortexBox" and select "Run Now...".
This will build a list of files to check/transfer, initiate the copying process, and probably take a very long time if it is the first copy in this direction. Everything I said about copying speed and "funny characters" in step (3e) also applies here.
You should end up with a perfect mirror of your Windows folders on the VortexBox. All file names with foreign characters of course translated properly, so they could easily be copied back, or used from both Windows and Linux. (Our setup will translate file names into what each system needs, but will not change the file’s contents — ever.
Have fun and let us all know your findings! I hope you’ll enjoy using this setup as much as I enjoyed finding all this out and writing it up for you.
See also
Contributors: Moonbase
