Quality assurance

Release and Testing Plan for SlimServer

The SlimServer project's approach to quality assurance is, and will continue to be, based on one of the basic tenets of open source development - "Release Early. Release Often." Our community of users and developers download nightly builds, identifying, reporting and fixing bugs early in the development process. The following plan is meant to supplement (and not replace) this approach.

The success of a project like SlimServer depends on its ability to quickly and easily incorporate changes and additions from the community. The goal of release and test planning is to introduce quality and predictability into the software process. These should not be contradictory goals. The release process should not slow down the pace of innovation (at least at this time...this goal may change as the project matures and stabilizes). At the same time, developers checking in changes should not sacrifice stability and quality for the sake of speed. The point of the steps suggested below is not to introduce process just for the sake of process, but to create a balance between competing tensions.

Release Planning

• Each official release will be driven by a Release Manager. This role may be traded off between multiple people.

• The Release Manager(s) determines the goals and requirements for a release with input from the community. With that in mind, all bugs and features should have a Bugzilla entry. Members of the community can nominate bug fixes or features for inclusion in the next release by sending mail to the developer or discuss mailing lists. The Release Manager(s) can choose to include the bug fix or feature in the next release by setting the Target Milestone field of the corresponding Bugzilla entry. The list of approved bug fixes/features represent the minimum requirement for completing the release - a TODO list - not the exclusive list of changes to be included.
• The Release Manager(s) uses the testing process to maintain a consistently high level of quality. If the quality of the trunk drops below an acceptable level, the Release Manager(s) can close the tree to all but approved checkins till quality improves.
• As the project gets closer to a release, the Release Manager(s) can move the project into Yellow Flag mode. In this mode, he can use any of the following options to reach the milestone:
  • Move the release to a branch so as not to slow down trunk development.
  • Ask developers to delay checkin of potentially destabilizing changes till after the imminent release.
  • Ask for all changes to be reviewed by a code buddy.
• The bottom line for developers is "Use common sense". If you have a small, contained fix or enhancement, you don't necessarily have to go through an approval process. If you have a large and potentially destabilizing change, talk to the Release Manager(s) early and often. Except when the project is in Yellow Flag mode, don't let the process slow you down.

Testing

The following test suites will be built and maintained over time:

• A set of manual SmokeTests that should be run at least daily (if not with every checkin). The SmokeTests represent a base level of functionality that should ''always'' work. If any of the SmokeTests fail, the source tree closes to all but approved changes until they work again.
• An exhaustive set of manual AcceptanceTests across all components and platforms to be run to validate a release. This is a script that will grow as new features, components, and platforms are added.
• AutomatedTests at both the component and application level. These will be built over time. Ideally, they will be run regularly - per checkin, per build cycle (e.g. using Tinderbox), or daily.

Bug Process

There's a process for dealing with new bugs: detailed here: BugProcess.
There's also an old page on submitting bugs here: FilingBugs. N.B.: These pages should probably be merged.