home • products • solutions • support • resources • company • blog

home > resources > build-pain-relief > Reducing the Pain of Build Failures

Reducing the Pain of Build Failures

The occasional build breakage is not be a big deal. However, many software teams live in fear of build breakages because their build is either always broken, used to always be broken, or fixing breakages is extremely painful. This guide examines why build breakages are painful, and discusses the strategies and practices that help reduce the pain points. Hopefully, the result of this guide will be that your team will have more confidence in the build, and use build breakage notifications as useful information rather than a notification of disaster.

Pillars of Pain Free Automated Building

The practices that keep build failures cheap typically support one or more pillar(s) of pain-free building:

1. Keep the rate of failure relatively low and the length of breakages short.
2. Maintain short builds.
3. Limit the number of people impacted by a failed build.
4. Provide impacted team members with effective workarounds.

Why are Build Breakages Painful?

To understand how best practices can help alleviate the pain of building, it's helpful to first study the ways that build breakages hurt teams. Build breakages can leave developers without a stable codebase to develop from, and hinder productivity. To address this, many teams scramble to fix a broken build, dropping everything else they are working on. This can make the situation even worse. At the same time, other groups who depend on a steady stream of newly produced builds, such as QA, may be stymied by prolonged build breakages. Learn more.

Practice 1: Small Teams

Small development teams have fewer integration errors, and the errors they do encounter impact few people. Practices like Componentized Projects and Team-based Development Streams allow a large team to work in small, efficient groups. Learn more.

Practice 2: Componentized Projects

Component-based development has been a key practice that supports code reuse. With good dependency management in place, it plays an important role in reducing the likelihood and impact of a failed build. Learn more.

Practice 3: Team-based Development Streams

Stream-based source control systems such as AccuRev and ClearCase allow teams within a project to work on their own tasks, and integrate another team's changes only when the other team has tested and promoted their code. Like Component-based Development, this practice supports breaking a large application team into smaller groups that build faster, break the build less often, and are hurt less by a breakage. Learn more.

Practice 4: Good Communication

Quickly fixing builds without pulling unnecessary people from their work is critical to the success of many teams. Excellent (automated) communication from the build system is critical to ensuring this happens. Learn more.

Practice 5: Promote and Deploy Built Binaries

A build breakage should not interfere with deploying earlier, successful builds to testing teams (or production). Likewise, special builds for testing should not take up build resources nor delay other builds. Learn more.

Practice 6: Establish a Build Cop

If failed builds are ignored by the team, assign responsibility to a single person to ensure that breakages are fixed in a timely manner. The Build Cop may fix the problem herself or make sure that the developer responsible for the breakage addresses the problem in a timely manner. Learn more.

Practice 7: Easy Checkout of Earlier Code

For teams that face extended or frequent outages, it's critical that developers can update their workspaces to the latest working code. "The build is broken" shouldn't be an excuse not to get work done. Instead, developers should be able to easily identify the latest working code and update their workspaces to it. Learn more.

Practice 8: Reduce Build Fragility

Builds that are easy to break and require very specific or complex configuration will break more often than simpler builds. Strive for simpler, more robust builds that break less frequently. Learn more.

Practice 9: Speed Up the Build

If techniques like incremental builds and componentization of an application fail, build acceleration may help teams speed up their builds. Learn more.

Practice 10: Developers Should Build Before Committing

Most build breakages can be avoided by developers building (and running fast tests) on their own machines prior to committing their work to the authoritative build system. Developer builds should be as similar to authoritative builds as possible. Learn more.

Practice 11: Preflight Builds in the Authoritative Build System

For teams transitioning to accurate developer builds, or those building against specialized hardware, preflight builds can be helpful. These take the developer's code and build it on the authoritative build environment before the developer commits to source control. Learn more.