AudioTests

Automated Audio Testing

This page describes the automated audio tests that are required for both the ip3k and arm based players. Wallace is building an improved set of audio tests based on the autotest system used for SqueezeCenter testing.

The tests are located in: https://svn.web.slimdevices.com/repos/slim/7.3/trunk/tests/audio_tests

Several different audio tests are used in the system (in order of importance):

Audio Conformance Tests 

This set of tests verifies the correctness of the audio decoders and audio playback.

Ramp Test 

The ramp test is used to detect audio glitches, for example underruns and overruns over an extended period of time.

Playback Control 

This tests randomly sends playback commands (play, pause, stop, fwd, rew, etc), in an attempt to crash or deadlock the system.

Internet Streaming Tests 

Used to verify Internet Radio and Music Service Partner streams.

System Requirements

The test system uses Windows XP with a sound card for audio capture. It would be possible to run the tests on other operating systems by modifying the record_player.pl script. Obviously you'll also need a SqueezeCenter running, and a ip3k (Squeezebox, Transporter, Boom) or arm (jive, fab4) player.

TODO: the system needs upgrading to a digital sound card for bit perfect capture and testing lossless formats (eg flac).

Audio Conformance Tests

The audio test system is used to automatically verify the audio playback of the Squeezebox product family. It performs the following steps:

1. A control script parses a configuration file that specifies the playlists for the tests, and the audio quality tests to run.
2. It calls a script that plays a playlist on the player, and records the results in a wav file.
3. A reference audio file is used for the audio quality tests:
   1. For the audio conformance tests (mp3, wma) this is specified in the configuration file.
   2. A script can be used to download and decompress the audio from SC for reference.
4. The audio quality tests use the recorded wav file and the conformance wav file to verify the output. The tests include:
   1. Length test, to check the two wav files are the same length (within limits)
   2. Gapless test, to check that no silence gaps are in the file, due to underruns or failure of flac/mp3/ogg gapless playback.
   3. Lossless tests, to check that flac and wma are played bit perfect.
   4. MP3 compliance tests.
   5. WMA compliance tests.

Audio Formats and Test Files

QA already have a set of sine wave tests to test flac, mp3 ogg and wav in various bitrates, sample sizes and channels.

TODO: The test files need extending to include:

1. conformance tests for flac, mp3 and wma
2. all audio formats, including SC transcoded formats
3. different encoding quality parameters within each format
4. flac and mp3 gapless (ogg?) tests
5. playback regression tests from the bug database
6. left/right channel tests

The test files will be large. To make it easy to recreate the tests, where possible, scripts should be used to recreate the files (eg the sine wave tests).

Scripts

The following scripts are used in the tests.

audio_test.pl

This script controls the audio tests.

The script can be run using the following command line arguments:

  --squeezecenter=<squeezecenter_ip_address>
  --playerid=<player_mac_address>
  --config=<test configuration>

The configuration file includes the playlist(s) to be tested, any special requirements (eg player volume) and the tests to use.

[TODO define this in more detail]

record_player.pl

This script loads a playlist, and plays in on the DUT while recording the audio.

The script can be run using the following command line arguments:

  --squeezecenter=<squeezecenter_ip_address>
  --playerid=<player_mac_address>
  --volume=<volume, optional defaults to 100%>
  --playlist=<playlist to play for the tests>
  --wavfile=<output wav filename>

So, this can be called on the command line using:

  record_player.pl --squeezecenter=127.0.0.1 --playerid=00:04:20:16:72:35 --playlist="test.m3u" --wavfile="test.wav"

This script will then:

1. connect to SC over the CLI
2. turn off repeat and shuffle for the player
3. set the player volume, this defaults to 100%
4. load the given playlist
5. start recording
6. wait 5 seconds (recording silence)
7. tell the player to start
8. wait for the playlist to complete
9. wait 5 seconds (recording silence)
10. stop recording, save the wav in <wavfile>

If any errors occur the script should return with a non zero exit status, or exit 0 on success.

This script makes it easy to use any player and SC, and does not require any hardcoded paths to music files (it's all done using the search command). Also it means that the SC and test script do not have to be running on the same PC.

record_stream.pl

This script downloads the files in a playlist, and decodes them with a reference decoder.

The script can be run using the following command line arguments:

  --squeezecenter=<squeezecenter_ip_address>
  --playlist=<playlist to play for the tests>
  --reffile=<reference wav filename>

So, this can be called on the command line using:

  record_stream.pl --squeezecenter=127.0.0.1 --playlist="test.m3u" --reffile="ref.wav"

This script will then:

1. connect to SC over the CLI
2. get a track list from the playlist
3. download the tracks
4. decode them using a reference decoder
5. save the decoded data as a wav in <reffile>

If any errors occur the script should return with a non zero exit status, or exit 0 on success.

[TODO define this in more detail]

length_test.pl

This script compares the length of two wav files. Any silence at the start and end of the file is ignored. The file length must match to within the specified limit.

The script can be run using the following command line arguments:

 --wavfile=<input wav filename>
 --reffile=<reference wav filename>
 --maxdiff=<maximum milliseconds difference in length>

So, this can be called on the command line using:

 length_test.pl --wavfile="test.wav" --reffile="ref.wav" --maxdiff=50

mp3_conformance_test.pl

The mp3 conformance tests are described at http://www.underbit.com/resources/mpeg/audio/compliance/.

The mp3 conformance bitstreams are available at ftp://ftp.tnt.uni-hannover.de/pub/MPEG/audio/mpeg1/compliance/.

wma_conformance_test.pl

[TODO] we have a wma conformance test suite with the ip3k firmware. We need to look at how this works again.

ra_conformance_test.pl

Needed after the Real Audio codecs are integrated. See https://datatype.helixcommunity.org/2004/realcodecs/tck.

Ramp Test

The ramp test is described in Audio Anecdotes Volume I page 295. This tests for sample accurate fidelity, which is simple the guarantee that a system can reliably output an audio signal without corrupting it by dropping samples, adding samples or playing the wrong samples. The important distinction about the ramp test from the other audio tests is that it can run for an indefinite period of time.

Two programs are required for the ramp test:

1. A generator for the ramp test, this can output a wav file that is played using 'repeat song' on the DUT. The output is a simple repeating saw tooth (counting from 0 .. 2^24).
2. An analyser that verifies the signal is correct. The algorithm for the analyser is described in the Audio Anecdotes book.

To ensure we have left/right channel lock then two saw tooths should be used, with the left channel counting up and the right channel counting down.

When this test is being run the DUT should be stressed with an additional CPU load.

Playback Control

TODO

1. must cover all supported formats
2. should be used to test both SC and music service partners

Internet Streaming Tests

TODO

1. listen to each stream for 10 mins?
2. don't expect each test to succeed always as the radio station/internet may have problems
3. have web page showing last time when each radio station was listened too
4. if stream fails schedule it for listening again soon, eg following day