In this second installment, we will try to answer the following question:
How and why has continuous integration impacted build management?

We took a pretty good look at continuous integration in last month's column. We pointed out that the main idea behind continuous integration is to merge changes into a common code line as often as possible. Why? To lessen the pain of merging two or more code lines. Continuous integration makes the assumption that the difficulty of merging two or more code lines is proportional to the length of time they have been diverging. Seems like a pretty reasonable assumption, and as with all good assumptions we are willing to accept it. Once, we do accept it though, it follows that to decrease the pain of integration, we have to decrease the length of time that code lines spend diverging. So, rather than merging changes on a weekly or bi-weekly cycle, continuous integration suggests that we should merge changes several times per day, or as soon as they are made.

Now, if you're reading this with some apprehension with regard to the resulting code quality, then I'm with you. Having a team of 10, 30, or 100 developers committing their changes multiple times a day or even on a daily basis has the potential of creating chaos. This is a big point, and one that I would like to explore in a little more detail, even though it is rather tangential to our discussion on build management.

Taming Chaos

Just because we are dealing with an activity that is inherently chaotic does not mean that we should throw our hands up in the air and give up on the goal of managing it or making any sense out of it. On the contrary, once we acknowledge that we are dealing with a chaotic process, we can then begin to arm ourselves with tools that are able to cope with such a beast. And these tools are very different than those suggested by the "traditional" methodologies. Rather than being proscriptive (which does not really work with chaos), the approach to managing a chaotic process is grounded in industrial process theory. The basic premise is that instead of trying to impose order (a futile goal), the appropriate course of action is to measure and respond. Success depends on the ability to respond quickly to the present state of the process. Thus, these new methodologies are called "Agile." Agility is key.

Chaotic Continuous Integration

So what does all of this have to do with continuous integration and build management anyhow? From the first paragraph above, it seems that continuous integration has nothing to do with build management. And, that's true; the idea of merging changes frequently in order to minimize the pain of merging, on its own, has very little in common with build management. But, as we noted, this frequent merging can bring about a chaotic process. And since the key to managing a chaotic process is measurement, we need some way to measure it and take appropriate course of action based on those measurements.

In the context of continuous integration, if we want to manage the process, we have to measure the quality of the code line. Not for the purpose of gauging progress on the project or daily productivity, or anything like that; we are only interested in measuring the quality of the code line to ensure that it does not decrease as changes are being merged in. If, via these measurements we find that the code quality did decrease compared with the previous measurement, then we have to respond. And just like in the case of a manufacturing line where you would remove any parts that failed to meet the quality test, the appropriate response to a decrease in code quality is to remove the offending part -- the last merged change. Once the offending change has been removed and the quality of the code line has been restored to it's previous high, production can resume.

Establishing a system to measure the quality of the code line between merged changes is crucial to our ability to manage the continuous integration process. In last month's column I called this measurement the verification step.

Continuous Integration and Build Management

The best way we know of to measure the quality of code is to compile it and run it through a suite of tests. Traditionally that has been the job of the QA team -- at least the running of the test suite part; and it still is. But using a QA team to measure the code quality in the context of continuous integration is like sending every mass produced widget from a plant in Kansas to China for a laborious quality control cycle that lasts several days. With production running at thousands of widgets per day, that system is just not going to work. If the plant started producing faulty widgets, it could be days before anyone is made aware of the problem. Our plant would do a lot better with an automated QC system that inspected each widget as it rolled off the line. That way, as soon as a faulty widget is detected, the entire line can be temporarily halted until the problem is identified and fixed. In this analogy, each widget represents a change that is integrated. Only by having an automated QC system that can inspect each change as it is integrated are we going to be able to manage the continuous integration process.

It is this exact repercussion of implementing continuous integration (an automated verification step) that brings us to build management. Automation of the verification step requires what essentially amounts to a pretty sophisticated build management system. After every integration, a build of the entire project has to be performed, unit tests (and perhaps more involved tests such as smoke tests or functional tests) have to be run to ensure that a project that was moving along quite happily before the last integration did not all of the sudden get sick.

The practices required to make this automation happen are independent of Continuous Integration and have been around long before continuous integration, but many of them have definitely popularized by CI. Below I compiled a list of practices that, in my opinion, benefited most from the rejuvenation brought to them by CI. We'll take a look at each of these practices in detail.

• build mechanism needs to be decoupled from the Development Environment
• builds are performed on a clean and controlled environment
• definition of a project as being more than only the source code

Build Mechanism Needs to be decoupled from the Development Environment

Most developers, especially when starting out, rely on their IDE to produce builds of their software. While this is very convenient, it can pose a serious threat to the implementation of a good build management process and system. A breakthrough on this issue came in the Java space a few years ago when ANTT, an Open Source build tool from the Apache Software Foundation, gained such market momentum that IDE vendors started supporting it. Now, most of the Java IDEs make it just as easy for developers to build a project by calling an external ANT script right from within the IDE, as by using the IDE specific build functionality.

The reason why it is so important to have a build mechanism that is decoupled from the IDE is to allow builds to be completed without having to fire up the IDE. Once the build mechanism is separated from the IDE, builds can be automated, scheduled, and scripted. Also, IDEs are typically very heavy products by nature and most like to set up the environment to their particular liking. Having an independent build mechanism implies that it will be lighter weight, and it will allow more explicit control over the environment. Some IDEs allow operation in a "headless" mode so that the IDE specific build mechanism can be automated, scheduled and scripted. While that is definitely a move in the right direction, the sheer weight of the IDE and it's control over the environment are still issues.

Continuous integration requires that the project be capable of being built outside of the development environment. Without this requirement, the verification step would not be possible. But requiring a build mechanism that is independent of the development environment is not new. Virtually all mature development organizations have been using independent build mechanisms such as make or some sort of scripting for a long time. Other types of builds, such as development, QA, and release builds also benefit from having a build mechanism that is separate from the development environment.

Build on a Clean and Controlled Environment

This practice is critical to a well-designed build management process and system. All too often, builds (including release builds) are performed by developers from within their development environments. There are many reasons why having anything but development builds performed in the development environment is a bad idea. Perhaps the most basic reason is the fact that, when the build is performed in the development environment, there is no assurance that sources included in the build are indeed committed to the source control repository. While true that this can be looked at as a procedural issue, is it really worth the risk? You can spend a lot of effort writing and enforcing procedures, or you can simply perform release builds in a clean environment and pull the sources for the build from the source code repository, thus ensuring that the build can be traced back to the sources.

Another reason for having builds performed in a clean environment is the old "it works on my machine" explanation. While this does touch on another topic, that of including tests (whether unit or smoke tests) as part of the build, the fact remains that only by having non-development builds be performed in a clean environment can we control and make explicit the environmental dependencies.

Needless to say, all non-development builds (that is CI, QA and release builds), must be performed in a clean and controlled environment. Decisions as to what is part of the environment as opposed to a dependency (which we'll look at below) are very interesting and to a large part have to be made in the context of the project. Only very rough guidelines, which follow from the forces involved, can be given, as we'll see below.

A Project is more than the Source Code

There are many environmental dependencies that an IDE takes care of for us automatically. Once we separate the build mechanism from the IDE, the source code alone becomes insufficient to build our project; thus we need to reproduce all of those environmental dependencies. While there are many taxonomies we can come up with to classify dependencies (i.e. infrastructure vs. project dependencies, etc.), at the end of the day there are two basic approaches that we can use to reproduce all the dependencies our projects require:

• we can refer to a dependency in a location outside of the project, or
• we can include the dependency as part of the project

Which approach is the right one to take depends on a variety of factors such as the size of the dependency, the ease of relocation, whether the dependency can be expected to exist in an original state on the build environment, etc. The larger (in terms of bytes) the dependency is, the more impetus there is to place it outside the project, in a well-known location, and refer to it there. Also, the more difficult the dependency is to relocate, the less likely we are to include it as part of the project. A dependency can be difficult to relocate perhaps because it requires a special installation routine to be run.

In the Java world, projects typically include all the libraries that are required to compile the project source code. These libraries are included as jar files, typically located in a "lib" directory from the project root. Other dependencies such as the Java compiler are not included as part of the project, but are referenced in a well-known or standard location.

The point is that there is a lot more that is required to produce a build than just the source code. And a project should contain all the information required to build it. In the case of dependencies the project can either contain the dependencies directly or it can contain information on how to access the dependencies. This is perhaps the practice that has gained the most visibility in recent years. A lot of the interest in this practice can be attributed to other factors, but continuous integration has also had a big impact here. This is another crucial practice for continuous integration because as CI requires that the code line be built, all of the dependencies have to be identified and made available. Making the dependencies explicitly available allows them to be controlled, and that has always been the goal in build management.

Conclusion

Perhaps the biggest impact of continuous integration on build management has been the need to automate the entire build process. Procedures that have to be implemented by humans do not provide for the same level of control as automated (scripted) implementations of those same procedures. By control I mean assurance specificity as opposed to flexibility. Although automation of the build process is also not new, as many development teams have been using fully automated builds long before continuous integration, it has definitely been popularized lately.

There are many practices I did not list here that are also required by CI as well as QA and release builds. One example of such a practice is obtaining the sources for a build from the version control system. I did not list such practices here because I believe that their popularity or acceptance did not benefit greatly from the rise in popularity of CI. This is a very subjective decision and was based on my experiences working with and observing various development organizations.

Add a comment

Posted by Maciej Zawadzki on November 1, 2003 3:00:00 PM EST #