Custom Scan plugin
From SqueezeboxWiki
Contents |
Overview
The Custom Scan plugin makes it possible to get more information about artist, albums, tracks than available from the standard SqueezeCenter(formerly known as SlimServer) scan. The standard SqueezeCenter scans a predefined list of tags from the music files and there is no way to connect the scanned information with information stored in other tags or in other places. The Custom Scan plugin tries to solve this problem by providing a framework for scanning modules that retrieves additional information from various places. This makes it possible to retrieve additional information from other tags in your music files but also from totally different places. Custom Scan provides the following scanning modules by default but new scanning modules can be implemented as separate plugins and will then be accessible through the Custom Scan interface.
- CustomTag - A scanning module that reads additional tags from the music files that are not normally stored in the SqueezeCenter database.
- MixedTag - A scanning module that retrieves standard information about artist, albums, genres and years from the SqueezeCenter database tables and also collects custom tags from the Custom Tag scanning module.
- LastFM - A scanning module that reads different information from the LastFM database related to artists.
- Amazon - A scanning module that read a number of different information from the amazon.com database related to albums.
- RatingTag - A scanning module that reads rating tags from the music files and stores them in the SqueezeCenter database.
See the separate scanning module section for more information about supported scanning modules.
Note! The LastFM and Amazon scanning modules will make SqueezeCenter work slowly during rescan due to licensing rules on LastFM and Amazon.com that don't allow many requests in a short time.
Installation
- Remove any previous version of Custom Scan from the SqueezeCenter Plugins directory if you have manually installed it earlier (this is not needed if you have installed it through Extension Downloader)
- Goto SqueezeCenter Settings/Plugins and select to install Custom Scan. You might need to add Erland's repository to see it, see here for more information regarding this: SqueezeCenter_Repositories. If you don't want to install it through Extension Downloader, you can also download it from the download page and manually unzip the new version in the SqueezeCenter Plugins directory
- Please note that earlier versions of this plugin will require you to also install the License Manager plugin.
- Please note that earlier versions of this plugin will require you to purchase a license.
- Optionally Install other plugins that extends Custom Skip with more functionality
- Custom Browse plugin (Almost a requirement)
- When you have installed everything you might also want to look at
Note!
- More information about how to purchase and activate the commercial license can be found through the License Manager plugin
If you are using SlimServer 6.5 or earlier, you may want to use the old guide instead.
Bugs and new features
The current list of known bugs and wishes for new features can be found here:
If you want to encourage future development of this plugin you should also consider making a donation or purchasing a license
Enable or disable scanning modules
Most of the scanning modules are set to disabled by default. The meaning with this is that they will not be included in automatic rescans due to a rescan in standard SqueezeCenter. You can still click on the Scan link in the Custom Scan section in SqueezeCenter Settings in the web interface for a disable module to perform a scan with it. Scanning modules such as LastFM and Amazon is typically best left disable, the reason is that they will pretty much make SqueezeCenter unusable during the scanning process so you will typically have the control of when the scanning shall happen.
If you enable a scanning module in the Custom Scan section in the SqueezeCenter Settings section of the web interface it will be automatically included in rescans after this.
Database structure
The scanned data are by default stored in the following three tables in the database
- customscan_contributor_attributes
- customscan_album_attributes
- customscan_track_attributes
Besides this each scanning module can choose to store data also in other places, as an example the RatingTag and Amazon scanning module can write rating information to the standard SqueezeCenter database tables.
customscan_contributor_attributes
The customscan_contributor_attributes contains attributes connected to an artist/contributor. In the built in scanning modules this means that this is where the LastFM scanning module store its information. Also note that even though custom tags scanned with the Custom Tag scanning module can be related to contributors, this information is not stored in this table.
The table has a number of different columns:
- module - Indicates which module the data belongs to
- attr - Indicates the attribute name of the data
- value - The value of the attribute that can be displayed to the user
- valuesort - The value of the attribute in sorting order syntax (Not currently used)
- extravalue - The value of the attribute that can be used for searching (faster than using the value column)
- valuetype - Indicates the type of the value. (Not currently used)
The relation columns in the table that connects the information to the standard SqueezeCenter contributors table are:
- contributor - The unique number identifier that connects the information to contributors.id. This is the column that shall be used when joining the information together with the contributors table.
Additional relation columns in the table that connects the information to the standard SqueezeCenter contributors table are:
- name - The name of the contributor, same value as in contributors.name. This value is here only to make it possible to synchronize the contributor column after a SqueezeCenter rescan.
- musicbrainz_id - The musicbrainz id of the contributor if it has been stored in the tags, this is the same value as in contributors.musicbrainz_id. This value is here only to make it possible to synchronize the contributor column after a SqueezeCenter rescan.
LastFM
The LastFM scanning module stores it information as:
- module - cslastfm
- attr - One of:
- picture - The url to the LastFm picture for the artist
- artisttag - The LastFM tag for the artist, several rows with different tags can exist for the same artist
- similarartist - Another artist that exist in your library and is considered as similar according to LastFM, several rows with different similar artists can exist for the same artist
- retrieved - Just an indication that we have tried to retrieve the artist from LastFM once (this is used to avoid further calls to LastFM in next rescan)
- value - The tag value that shall be shown to the user
- valuesort - Not used
- extravalue - Only used for artisttag and similarartist attributes, it then contains the similarity/relevance percentage as a value between 0 to 100
- valuetype - Not used
customscan_album_attributes
The customscan_album_attributes contains attributes connected to an album. In the built in scanning modules this means that this is where the Amazon scanning module store its information. Also note that even though custom tags scanned with the Custom Tag scanning module can be related to albums, this information is not stored in this table.
The table has a number of different columns:
- module - Indicates which module the data belongs to
- attr - Indicates the attribute name of the data
- value - The value of the attribute that can be displayed to the user
- valuesort - The value of the attribute in sorting order syntax (Not currently used)
- extravalue - The value of the attribute that can be used for searching (faster than using the value column) (Not currently used)
- valuetype - Indicates the type of the value. (Not currently used)
The relation columns in the table that connects the information to the standard SqueezeCenter albums table are:
- album - The unique number identifier that connects the information to albums.id. This is the column that shall be used when joining the information together with the albums table.
Additional relation columns in the table that connects the information to the standard SqueezeCenter albums table are:
- title - The title of the album, same value as in albums.title. This value is here only to make it possible to synchronize the album column after a SqueezeCenter rescan.
- musicbrainz_id - The musicbrainz id of the album if it has been stored in the tags, this is the same value as in albums.musicbrainz_id. This value is here only to make it possible to synchronize the album column after a SqueezeCenter rescan.
Amazon
The Amazon scanning module stores it information as:
- module - csamazon
- attr - One of:
- asin - The Amazon ASIN number that represents the album
- avgrating - The Amazon average customer rating for the album
- subject - The Amazon category the album belongs to, several rows with attr equal to subject can exist for the same album
- retrieved - Just an indication that we have tried to retrieve the album from Amazon once (this is used to avoid further calls to Amazon in next rescan)
- value - The tag value that shall be shown to the user
- valuesort - Not used
- extravalue - Not used
- valuetype - Not used
customscan_track_attributes
The customscan_track_attributes contains attributes connected to a track. In the built in scanning modules this means that this is where the Custom Tag, Mixed Tag and Rating Tag scanning modules store their information. Also note that even though custom tags scanned with the Custom Tag scanning module can be related to albums and contributors, this information is stored in this table.
The table has a number of different columns:
- module - Indicates which module the data belongs to
- attr - Indicates the attribute name of the data
- value - The value of the attribute that can be displayed to the user
- valuesort - The value of the attribute in sorting order syntax
- extravalue - The value of the attribute that can be used for searching (faster than using the value column)
- valuetype - Indicates the type of the value
The relation columns in the table that connects the information to the standard SqueezeCenter tracks table are:
- track - The unique number identifier that connects the information to tracks.id. This is the column that shall be used when joining the information together with the tracks table.
Additional relation columns in the table that connects the information to the standard SqueezeCenter tracks table are:
- url - The url of the track, same value as in tracks.url. This value is here only to make it possible to synchronize the track column after a SqueezeCenter rescan.
- musicbrainz_id - The musicbrainz id of the track if it has been stored in the tags, this is the same value as in tracks.musicbrainz_id. This value is here only to make it possible to synchronize the track column after a SqueezeCenter rescan.
Custom Tag
The Custom Tag scanning module stores it information as:
- module - customtag
- attr - The tag name with capital letters
- value - The tag value that shall be shown to the user
- valuesort - The value of the corresponding sorting tag, if a "Sort tag mapping" has been configured. If no sorting tag has been configured, this will contain the same value as the value column but in capital letters.
- extravalue - Not used
- valuetype - Not used
Mixed Tag
The Mixed Tag scanning module stores it information as:
- module - mixedtag
- attr - The tag name with capital letters, this can be both SqueezeCenter information with the configured attribute names or the tag name from a tag scanned with the Custom Tag scanning module.
- value - The tag value that shall be shown to the user
- valuesort - The value that shall be used when sorting
- extravalue - For SqueezeCenter information the unique identifier will be stored here, that is the same value as in contributors.id, albums.id, genres.id, years.id. For custom tags this column will contain the same value as the value column.
- valuetype - Only used for SqueezeCenter information, not for custom tags. It will contain the type of object the extravalue columns refers to. Available values are: year, album, artist, genre
Rating Tag
The Rating Tag scanning module stores it information as:
- module - ratingtag
- attr - RATING
- value - The rating value as a number between 0 and 100
- valuesort - Not used
- extravalue - Not used
- valuetype - Not used
Scanning modules
| CustomTag | A scanning module that reads additional tags from the music files that are not normally stored in the SqueezeCenter database. The tags read can be configured in the Custom Scan section of SqueezeCenter Settings in the web interface. By default read tags will be splitted in same way as genres in standard SqueezeCenter, if you don't want the tag to be splitted you can also add it to the "Single value tags" settings in the Custom Scan web interface. The mapping between MP3 ID3 standard frame names to tag names can be found here. |
| MixedTag | A scanning module that retrieves standard information about artist, albums, genres and years from the SqueezeCenter database tables and also collects custom tags from the Custom Tag scanning module. The result is that you can combine standard tags handled by SqueezeCenter and custom tags in the same menu by using the Custom Browse plugin. When you have used the Mixed Tag scanning module a new Tags menu will automatically appear in the Custom Browse menu, this menu makes it possible to browse the information in a very flexible and dynamic way. |
| LastFM | A scanning module that reads a number of different information from the LastFM database. Please note that the information read is only free to use for non commercial usage, see the licence for more information. The module currently reads the following additional information for all artists:
If you specify a picture directory in the settings for the LastFM scanning module it will retrieve artist images during scanning and store these in the specified directory. These images can later be used together with Custom Browse plugin to show an artist image when you have selected an artist in the web interface. |
| Amazon | A scanning module that read a number of different information from the amazon.com database related to albums. Please note that the information read is free but the web service usage is restricted according to Amazon Web Services Licensing Agreement. The module currently read the following additional information for all albums:
The Amazon module can also optionally set the ratings in SqueezeCenter and TrackStat plugin, this functionallity is disabled by default but you can enable it in the Custom Scan web interface. Genres/subjects imported are only imported if they are shorter than the setting which can be configured in the Custom Scan web interface. |
| RatingTag | A scanning module that reads rating tags from the music files and stores them in the SqueezeCenter database. This is for example used by MediaMonkey and optionally also by Windows Media Player if you have choosed to store the rating information in the music files. The name of the rating tag read can be configured in the Custom Scan section of SqueezeCenter Settings in the web interface. By default read ratings will be stored in the standard tables in the SqueezeCenter database, but this can be turned off in the Custom Scan web interface and then the ratings will just be stored in the customscan_track_attributes table. If configured to store ratings in the standard tables, it will write rating information both to TrackStat plugin (if installed) and to the standard SqueezeCenter tables. |
NOTE!!! The Amazon and LastFM modules are quite slow, the reason for this is that Amazon and LastFM restricts the number of calls per second towards their services in the licenses. Please respect these licensing rules. This also results in that SqueezeCenter will perform quite bad during scanning when these scanning modules are active. The information will only be scanned once for each artist/album, so the next time it will only scan new artists/albums and will be a lot fast due to this. Approximately scanning time for these modules are:
- LastFM: 1-2 seconds per artist in your library
- Amazon: 1-2 seconds per album in your library
The information read by the above modules is just stored in a separate table in the database and cannot be viewed in standard SqueezeCenter.
- If you install the SQL Playlist plugin you can use the read information to create smart playlists.
- If you install the Custom Browse plugin you can use the read information to create browse menus.
Tag mapping
The Custom Tag scanning module have an advanced feature where you can combine and extract information in tags and store them in a new virtual tag, this is configured in the Tag mapping parameter of the Custom Tag scanning module settings page.
There are three ways to create a virtual tag:
- oneof - Selects information from one of several specified tags, the information will be taken from the first tag that exists
- combine - Concatenates information from several tags into a single tag
- as - Takes information from one tag, this can be used to rename tags or extract parts of a tag into a virtual tag
For each tag in the above described ways, you can also specify how the information from a tags should be used:
- exp - Extract information from the tag using a regular expression with a catch group
- text - Use the specified text instead of the tag value if the tag exists
This is probably best descried using some samples, let's say we have the following real tags in the file:
- ARTIST = Lisa Stansfield
- ALBUMARTIST = Whitney Houston
- DATE = 1992-01-01
- COMMENT = Songs-DB_Custom1: Soundtrack
- GENRE = Pop
Some sample tag mappings:
- To get all artists:
- ALLARTISTS=combine ALBUMARTIST|ARTIST(text= and )|ARTIST|TRACKARTIST(text= and )|TRACKARTIST
- The result will be: ALLARTISTS="Whitney Houston and Lisa Stansfield"
- To get the decade:
- DECADE=combine DATE(exp=^(\d\d\d))|DATE(text=0)
- The result will be: DECADE="1990"
- To extract the custom tag information from MediaMonkey from the COMMENT tag:
- CUSTOM1=as COMMENT(exp=^Songs-DB_Custom1:\s*(.+)$)
- The result will be: CUSTOM1="Soundtrack"
- To extract the YEAR from the DATE tag:
- YEAR=as DATE(exp=^(\d\d\d\d))
- The result will be: YEAR="1992"
- To extract composer from the COMPOSER tag if it exists else from the ARTIST tag
- COMPOSER=oneof COMPOSER|ARTIST
- The result will be: COMPOSER="Lisa Stansfield"
Title formats
In the Custom Scan section of the "SqueezeCenter Settings" in the web interface it is possible to choose which custom information that should be available as title formats. It is only possible to make custom track information available as title formats. You use a custom title format as follows:
- Goto "SqueezeCenter Settings"/"Plugins" and goto the "Title Formats" parameter in the settings for the "Custom Scan" plugin.
- Select the title format you like to use, note the exact text
- Goto "SqueezeCenter Settings"/"Formatting"/"Title Format"
- Add the text from step 2 in one of the existing title formats or at the last line as a completely new title format. You can combine several title format in a long string, for example if you would like a track to be displayed as:
1. A nice track (Erland)
where Erland is the OWNER tag scanned with Custom Scan you would enter:
TRACKNUM. TITLE (CUSTOMSCAN_CUSTOMTAG_OWNER) - To actually start using the the title format you:
- For Web Interface listings: Goto "SqueezeCenter Settings"/"Formatting"/"Title Format" and select the title format you want to use by selecting the radio button.
- For SqueezeBox Display: Goto "SqueezeCenter Settings/Player Settings/"xxx"/"Title Format" and select the title format you want to use bt selecting the title format and selecting the radio button between it
CLI interface
Scan status command
The status query command is used for checking scanning status, the syntax is:
customscan status <module>
The module parameter is optional, if specified it will only return the status of the specified module, else it will return status for all modules.
A status query command could look something like:
customscan status Plugins::CustomScan::Modules::RatingTag
The response to the status query command has the following syntax:
customscan status <module> count:<noOfResults> id:<module> name:<modulename> status:<status>
The response for a status query command could look something like:
customscan status Plugins::CustomScan::Modules::RatingTag count:1 id:Plugins::CustomScan::Modules::RatingTag name:Rating Tag status:0
The parameters in the result is:
- count - Number of items in the result
- id - The identifier of a module
- name - The name of the module that should be displayed to the user
- status - The current status of the module, valid values are:
- 1 - Scanning,
- 0 - Not scanning
- -1 - Failed, but still scanning
- -2 - Failed, finished scanning
Scan command
The scan command is used for initiate scanning:
customscan scan <module>
The module parameter is optional, if specified it will only start scanning of the specified module, else it will start scanning on all modules which has been configured to be part of automatic and Scan All scannings.
A scan command could look something like:
customscan scan Plugins::CustomScan::Modules::RatingTag
Clear command
The clear command is used for deleting scanning data:
customscan clear <module>
The module parameter is optional, if specified it will only delete scanning data of the specified module, else it will delete scanning data of all modules.
A scan command could look something like:
customscan clear Plugins::CustomScan::Modules::RatingTag
Abort command
The abort command is used for aborting an ongoing scanning:
customscan abort
An abort command could look something like:
customscan abort
Changed status notification
The changed status notification is sent as soon as a scanning module changes its scanning state
customscan changedstatus <module> <status>
The module result item indicates which scanning module that has changed state. The status result item indicates the new state, available states are:
- 1 - Scanning
- 0 - Not Scanning
- -1 - Failed, but still scanning
- -2 - Failed, finished scanning
An changed status notification could look something like:
customscan changedstatus Plugins::CustomScan::Modules::RatingTag 1
Scanning modules as separate plugins
A scanning module is implemented in a separate plugin by implemeting a getCustomScanFunctions function. You can see the included LastFM and CustomTag scanning modules for more detailed samples about the implementation of this function. Basically it shall return an array where each entry is a map with the following keys:
| id | A unique identifier of the scanning module, will be used as module when storing the information in the database |
| name | A user friendly name of the scanning module that shall be shown to the user |
| description | A user friendly description of the scanning module that shall be shown to the user |
| scanArtist | A pointer to the function that shall be called when scanning an artist, if not specified artists will not be scanned by this module. This function will get a contributor object as in-parameter when called, it will be called once for each artist in the SqueezeCenter database. |
| scanAlbum | A pointer to the function that shall be called when scanning an album, if not specified albums will not be scanned by this module. This function will get an album object as in-parameter when called, it will be called once for each album in the SqueezeCenter database. |
| scanTrack | A pointer to the function that shall be called when scanning a track, if not specified tracks will not be scanned by this module. This function will get an track object as in-parameter when called, it will be called once for each track in the SqueezeCenter database. |
| alwaysRescanArtist | If set to 1, old artist data will always be deleted before scanning. If not specified only artists with no previous data will be scanned. |
| alwaysRescanAlbum | If set to 1, old album data will always be deleted before scanning. If not specified only albums with no previous data will be scanned. |
| alwaysRescanTrack | If set to 1, old track data will always be deleted before scanning. If not specified only tracks with no previous data will be scanned. |
| scanInit | A pointer to the function that shall be called when scanning begins, this is optional and is only required if you need to do something in the scanning module before scanning begins. |
| initScanTrack | A pointer to the function that shall be called when scanning of tracks begins, this is optional and is only required if you need to do something in the scanning module before scanning of tracks begins. It is also useful if your scanning doesn't need to know about existing tracks and you like to do the whole scanning in a single operation. The function will get a hash as parameter, this hash can be used to save values between different calls to the function. The function can be called several times, if the function returns undef it will be called again, if it returns something else than undef the scanning will continue with the next step. |
| initScanAlbum | A pointer to the function that shall be called when scanning of albums begins, this is optional and is only required if you need to do something in the scanning module before scanning of album begins. It is also useful if your scanning doesn't need to know about existing albums and you like to do the whole scanning in a single operation. The function will get a hash as parameter, this hash can be used to save values between different calls to the function. The function can be called several times, if the function returns undef it will be called again, if it returns something else than undef the scanning will continue with the next step. |
| initScanArtist | A pointer to the function that shall be called when scanning of artists begins, this is optional and is only required if you need to do something in the scanning module before scanning of artists begins. It is also useful if your scanning doesn't need to know about existing artists and you like to do the whole scanning in a single operation. The function will get a hash as parameter, this hash can be used to save values between different calls to the function. The function can be called several times, if the function returns undef it will be called again, if it returns something else than undef the scanning will continue with the next step. |
| scanExit | A pointer to the function that shall be called when scanning ends, this is optional and is only required if you need to do something in the scanning module after scanning has finished. |
| exitScanTrack | A pointer to the function that shall be called when scanning of tracks ends, this is optional and is only required if you need to do something in the scanning module after scanning of tracks has finished. It is also useful if your scanning doesn't need to know about existing tracks and you like to do the whole scanning in a single operation. The function will get a hash as parameter, this hash can be used to save values between different calls to the function. The function can be called several times, if the function returns undef it will be called again, if it returns something else than undef the scanning will continue with the next step. |
| exitScanAlbum | A pointer to the function that shall be called when scanning of albums ends, this is optional and is only required if you need to do something in the scanning module after scanning of album has finished. It is also useful if your scanning doesn't need to know about existing albums and you like to do the whole scanning in a single operation. The function will get a hash as parameter, this hash can be used to save values between different calls to the function. The function can be called several times, if the function returns undef it will be called again, if it returns something else than undef the scanning will continue with the next step. |
| exitScanArtist | A pointer to the function that shall be called when scanning of artists ends, this is optional and is only required if you need to do something in the scanning module after scanning of artists has finished. It is also useful if your scanning doesn't need to know about existing artists and you like to do the whole scanning in a single operation. The function will get a hash as parameter, this hash can be used to save values between different calls to the function. The function can be called several times, if the function returns undef it will be called again, if it returns something else than undef the scanning will continue with the next step. |
| properties | An array of properties, where each property hash contains the following:
|
If you want to have some simple settings for your scanning module, you can either implement a web interface in your plugin for setting it or you can choose to let the user specify it as a property in the Custom Scan web interface. You use the Plugins::CustomScan::Plugin::getCustomScanProperty function to retreive a Custom Scan property from your scanning module.
See the included scanning modules in the Modules directory for sample implementations of the different functions.
