Given that I work for a company that develops an automated build management server, I get to hear about some of the different build and release processes that are in use. Generally, they all follow the same structure, but there are some variations. In this month's column, I would like to present that general structure and talk a little about some of those variations.

Multi-stage Process of Release Builds

Release builds, unlike other types of builds, go through a multi-stage process. This is because when performing a release build, it is not enough to produce the build artifacts; those artifacts have to go through the QA stage before being released into production. The typical sequence of stages is as follows:

• Integration
• QA
• Production

The process begins with the production of a candidate build, which we'll examine in detail later on. For now lets take a closer look at how the candidate build moves through the different stages.

At the time that the candidate build is being produced, there are a number of tests to which it is automatically submitted. Even if automated unit tests are not included in the build process, there still is the very important compile test that the build must pass. If the sources included in the build do not compile, then this test fails and the candidate build never makes it out of the Integration stage (as illustrated by attempt 1 in the figure below). Ideally, the build should include a large number of automated unit tests that must all pass before the build is successful and promoted to the QA stage.

Once the candidate build passes the first level of tests (the compile test as well as automated unit tests) it is said to have passed the Integration stage and is promoted to the QA stage for testing. In the QA stage, QA team obtains the build artifacts and runs them through the test plan. If the QA team decides that the candidate build does not pass the test plan, then the candidate build is not promoted to the Production stage (as illustrated by attempt 2 in the figure below). If, on the other hand, the QA team determines that the build passes the entire test plan and is deemed ready for production, the QA team marks the candidate build as having passed the QA stage and the build is promoted to the Production stage (attempt 3 in the figure below).

Production of the Candidate Build

Most of the variability in release builds is in the way that the candidate build is created. Although the steps used to produce the candiate build are the same, the details within the steps may vary. So lets first take a look at the steps required to produce the candidate build and then we'll take a look at the ways in which they may vary. Ideally, the whole release build process would be automated. The following are the steps that the build would go throught, whether automated or not.

Clean Build Environment

A release build should always be performed in a clean and controlled environment. This is as opposed to a development environment, which is considered to be not clean because there is a high likelyhood that it has been tampered with during development. The clean environment presents the same starting point that can be used for all builds. Having a starting point that can be relied on to be the same goes a long way towards allowing builds to be reproduced in the future. The need for a clean environment is often fulfilled by a physical build server. This separate machine is not used for development and its environment is explicitly controlled during the build.

Getting the Sources

Once we have a clean environment in which to perform the build, we then need to obtain the sources. This is one issue that provides for some variability, which we'll return to later. For now, lets just say that we need to obtain the sources for the build from the version control system.

Producing the Build Artifacts

After the sources for the project are available in the clean environment, all the processes required to produce the build artifacts (sometimes also called derived objects) are invoked. I'm talking about compiling the sources, linking them if necessary, running unit tests, producing documentation from the sources, producing source metrics, packaging, and any number of other processes that you may choose to run during the build. Here, it is almost mandatory to have some script or set of scripts that will invoke all these processes automatically. Opting for a manual process here would not only be time consuming but also error prone and not reproducible (especially when you may need to reproduce a build performed a year or two back).

Storing the Build Artifacts

Once the build artifacts have been created, they need to be stored in a location suitable for distribution and possibly reuse. This is another area where there is a lot of variability. This variability is dependent mostly on the type of tools (specifically the version control system) being used. At the very least, the build artifacts need to be placed in a location that can be accessed by the QA team during the QA stage and by the production team during the production stage.

Applying the Release Label

At some point during the build, typically after the build artifacts have been produced, a release label has to be applied to the version control system marking the exact revisions of all the sources used to produce the build. There is a lot of variability in the format of the label but the essential requirement is that it be able to uniquely identify the build. A build number that is incremented on every build would suffice as a label although it would not be a very human firendly label. Most people opt for a label that includes some designation of a release build, the version number, the build number, perhaps a promotion level, a date stamp, and other information.

Variations on the Candidate Build Process

Two variations that come up often are deal with the way in which the sources for the build are obtained and the location used to store build artifacts. Lets take a closer look at these two below.

Push vs Pull

The first variation that we typically see is how the sources for the build are obtained from the version control system. There are two basic approaches here which in my mind come down to a push vs pull distinction. In the push approach, the build is based on a label that is applied prior to the build. In the pull approach, the build is based on the latest sources from the branch. The motivation for the push approach is that there is a seemingly greater control over the sources used for the build if they are labeled ahead of time. Proponents of the push approach envision a scenario where developers are committing changes at a regular pace. In such a scenario performing a pull based build may provide very little control over the contents of the build, since a developer might have been able to commit a series of changes just prior to the build unbeknown to anyone. But the same problem can happen with the push approach since the same developer could just as easily have committed those changes just prior to the label being applied. The assumption of developers committing changes at a regular pace during the time of a candidate release build is probably flawed as the number of issues and new features being developed tends to go towards zero as we get closer to a release. Most of the developers are busy working on new features for the next release on another branch, which is isolated from the branch used for the release.

Artifact Store

The location that stores the artifacts of a candidate can also vary widely. This variation is mostly due to the tools used though as some version control systems handle binary files very well where as others do not. Projects using tools that handle binary artifacts well typically store the build products in their version control system. Another variation here is whether the artifacts are meant to be reused by other projects. If reuse is intended, then the build artifacts should be stored in a central location that is easily accessible during the build of other projects. A very simple (and not neccessarily recomended) version of this is a common directory holding different intermediate artifacts such as jar files. A more advanced example is a Maven repostiory.