SqueezeOS Smoke Tests

An important part of our open development process are the nightly builds that make the latest beta and bug fix Controller firmware available for people to test. The Controller nightly firmware must pass an automated smoke test before being released, this is to ensure that the basic functionality works correctly. An especially important test is to verify that firmware upgrades work correctly in the latest build.

What are the Smoke Tests

The smoke tests are developed as SqueezePlay Macros. The following tests are run on every nightly build:

Forced Software Upgrades 

This test verifies that a forced software upgrade works.

User Software Upgrades 

This test verifies that a user requested software upgrade works.

Home Menu 

This test verifies that the Controller home menu is displayed correctly.

Playmode 

This test verifies that the Play/Pause/Stop functions on the Controller work.

Factory Reset 

This test verifies that the Controller can be factory reset.

WPA Setup 

This test verifies that network setup with a WPA network works.

The following tests will be added soon:

SD Card Software Upgrade 

This test verifies that an SD Card upgrade works.

Running the Smoke Tests

To run the smoke tests you will need SqueezeCenter, a Squeezebox Player, a Controller and an SD Card. The tests assume that the Controller is associated with a Player.

1. From SVN copy the files from squeezeplay/src/squeezebox_jive/test/smoke/ onto the SD Card in a squeezeplay/ folder. (note: copy the files only and do not create a smoke/ subdirectory in squeezeplay/)
2. Put the SD Card into your Controller and turn it on.
3. The smoke tests will run automatically, the results of the tests are displayed on the Controller, and also written into the system log.

[todo screenshot]

The Macros.lua file controls which tests are automatically run. This file is also updated with the results of the tests. To automatically run the tests again you will need to copy to the SD Card the Macros.lua file from SVN. Some macros allow parameters to be set for your local system (for example ssid and wpa keys), this is also done in the Macros.lua file. For example:

   ...
   setupWPA={
     desc="Setup using WPA network. This macro only works after a Factory Reset.",
     file="SetupWPAMacro.lua",
     name="Setup WPA",
     param={
       region="Europe",
       ssid="linksys",
       wpa="99999999",
       player="Transporter"
     }
   }
   ...

The tests can be run manually from the Extras > Test macros menu.

Smoke Test Automation

For an automated smoke test you will need a host system (these instructions are use a Linux host), SqueezeCenter, a Squeezebox Player, a Controller and an SD Card.

First generate an SSH key pair on the host system. This key will allow you to ssh and scp onto the Controller without a password.

richard@jive-arm:$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ubuntu/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/ubuntu/.ssh/id_rsa.
Your public key has been saved in /home/ubuntu/.ssh/id_rsa.pub.
The key fingerprint is:
67:20:ee:aa:00:9a:5e:2c:c5:85:53:c2:9b:84:0d:97 richard@jive-arm

Copy the SSH public key and a squeezeos-post-reset.sh script to the SD Card. This script will be automatically run after a factory reset and will enable the host system to automatically access the Controller.

richard@jive-arm:$ echo "sed s/#ssh/ssh/ -i /etc/inetd.conf" > /media/usb/squeezeos-post-reset.sh
richard@jive-arm:$ echo "mkdir /root/.ssh" >> /media/usb/squeezeos-post-reset.sh
richard@jive-arm:$ echo "cp /mnt/mmc/authorized_keys /root/.ssh" >> /media/usb/squeezeos-post-reset.sh
richard@jive-arm:$ echo "chmod 200 /root/.ssh" >> /media/usb/squeezeos-post-reset.sh
richard@jive-arm:$ echo "chmod 200 /root/.ssh/authorized_keys" >> /media/usb/squeezeos-post-reset.sh
richard@jive-arm:$ chmod +x /media/usb/squeezeos-post-reset.sh

Put the SD Card into the Controller, and dock the Controller so it is always powered. You will need to take a note of the Controller's IP address.

Edit the Macros.lua script as required, then copy the smoke test macros onto the Controller, and then reboot. The tests will start automatically.

richard@jive-arm:$ scp * root@10.1.1.227:/mnt/mmc/squeezeplay
Advanced.bmp                                                    100%  225KB 225.1KB/s   00:01    
CopyingUpdate.bmp                                               100%  225KB 225.1KB/s   00:01    
FactoryReset1.bmp                                               100%  225KB 225.1KB/s   00:00    
...
richard@jive-arm:$ ssh -l root 10.1.1.227 reboot

This last step can be repeated as required to re-run the tests. To access the test results the Macros.lua can be copied from the Controller using scp.

Creating new reference screenshots

Occasionally the format of a given screen might change slightly (e.g., with the addition of a new menu item), and in these cases it is sometimes necessary to create a new reference screenshot for the Smoke Tests to test against.

Creating a screenshot

1. Place an SD card in a Squeezebox Controller and boot it with the applicable version of firmware (that is, one that has screens in the format you want to test against).
2. Go to the screen(s) you want reference shots of, and hold pause and rew together to take a screenshot
3. Repeat as needed. Screenshots will be saved on the SD card as jive*.bmp files

Editing the screenshot

Often, parts of a given reference screenshot should not be analyzed against the test image. For example, the software update screen lists the version to upgrade to at the bottom of the screen, which will change with each successive version of firmware. The Macro applet allows for sections of a reference screen to be "pinked out". These sections of the screen are removed from the test analysis.

1. open your preferred image manipulation program. I use GIMP, which is free and runs on both OS X and Linux.
2. in this program, open an existing reference screenshot that contains at least one "pinked out" area
3. using the eyedropper tool, click on the pink area to save it as the new fg color
4. open the screenshot file intended as the new reference screen
5. using the box select tool, select an area to be pinked out
6. using the paint fill tool, with an option to "use fg color", click on the box-select from the previous step to fill it with pink
7. save the new image to the correct file name on the SD card, and restart the Smoke Tests
8. if the new image is to be used for all future smoke tests against that version of firmware, check it into the svn repository