(Today we offer a guest blog entry from Paul Julius.)

I loved the whitepaper that Eric and Jeffrey wrote about the maturity model for Enterprise Continuous Integration. The corresponding "handout" has proved invaluable for me as a consultant focusing on helping clients adopt continuous integration (CI). I frequently bring the handout to executive boardroom meetings to use as a discussion framework. I can highlight where they are today and where I think they should be headed. If we all agree, then we can work out a plan that helps them get there. Aiding in the delivery of that plan, seeing all the pieces start to fall into place really gets me excited.

As I started to use this model more and more, I realized that there was an element missing that I wanted to highlight for my clients — culture. There are plenty of organizations that have paid big bucks for a CI tool. Some of those have setup a working continuous integration system. They may be somewhere in the middle of the adoption curve, say "Intermediate" on most of the aspects of the maturity model. Yet, they are failing to see many of the benefits promised by continuous integration. Promises like green builds almost all the time and high developer productivity can prove elusive, no matter how elaborate the CI system. I wondered to myself, "What are we missing?"

I realized Continuous Integration is a human endeavor! James Shore touched on something really important back in his 2006 blog article "Continuous Integration on a Dollar a Day". Being really successful at CI starts with the human side. When Kent Beck wrote about CI in his 1999 Extreme Programming Explained book, he was describing a human process. He was describing a behavior that the team adopted which greatly increased their ability to create working software. People like me with the CruiseControl team and the good folks at Urbancode with Anthill Pro wrote software to automate the process. Had we forgotten about the human side? Had we eliminated the repetitive task with automation, while losing the essential people component that makes it a success? Nope.

It doesn't have to be that way. I've seen teams be very successful with automated CI tools. But those teams tend to use a lot of human interaction in conjunction with their CI tooling. Broken build games, big visible CI monitors, Homer Simpson sound bites, flashing red lights and lava lamps all appeal to the human side of the equation. Teams that remember that side of the equation seem to succeed with CI, and generally succeed with their software delivery. I wanted to add that aspect to the Maturity Model. I wanted to codify it and get it in front of my clients from the beginning of the process. So here we go, my "Culture" addendum to the Enterprise CI Maturity Model.

At the Introductory level, teams will start by getting developers to check in their changes at least once every day. In order to continuously integrate changes, those changes must be available. If developers don't commit their code, then nothing gets integrated. Frequent check-ins are the hallmark of a well functioning CI team.

The second indicator that you have achieved the Introductory level is that the Build Engineer always knows the status of the build. In a manual CI world there might not be a dedicated build engineer. The last person to run the build on the integration machine will know the status. They will alert the team if necessary. In an automated CI world, someone usually gets to own the CI server. The person keeping the CI machine ticking along will want to keep a close eye on the build statuses. If anything breaks, they should help get it fixed right away. Otherwise, their build machine devolves into a "broken build machine".

As your team matures, they will realize that adding more change to a broken build only complicate matters. Novice level teams will introduce a policy that only the person (or pair) fixing the build are allowed to check-in. Some teams will even use an integration token, like a small stuffed animal or a gaudy ceramic figurine. If you have the integration token, you are cleared to commit changes. If you don't, then go looking for it. Someone else might be fixing the build, or trying to get their changes committed. I don't recommend teams use an integration token. I prefer that everyone just "knows" when it's OK to commit, but that's the next level up. So, for know, continue using your plastic statuette.

An essential part of keeping the build working and successfully integrating your changes with everyone else is the ability to run a build locally to find most problems. Novice teams will have a single button (or single command) build that developers run locally prior to any commit. Sometimes called the "pre-commit" build, it will find many problems before they are thrust upon the rest of the team. Everybody on the team runs this build by convention. The first thing a build master will ask when they get to your desk is, "Did you run the pre-commit build?"

Remember earlier when I mentioned that team members should be omniscient, i.e. they should just know when its OK to commit? They don't have to be all-seeing beings, they should have something highly visible making them aware of the build status at all times. The aforementioned lava lamps are a popular option for Intermediate teams. If any build is broken, then the red lamp glows. If the build is working then the green lamp bubbles happily. Whenever it's green make sure you are up to date with the main line, run your pre-commit build and send those changes off.

To keep this all working, most Intermediate teams will drop everything to fix a broken build. Every team member knows that it's in their best interest to keep that build lamp green. In practice this might be as simple as a quick holler around the room to figure out who owns the broken build. If everyone agrees that "PJ is going to fix the build", then everybody else can go back to work. However, if we can't agree on who is going to fix the build, or worse yet, how to fix the build, then we are in a real jam. We might have to get everyone together to look at the build output. We might have to white board a solution to the problem. We might need to agree that rolling back the last changeset is appropriate.

Usually during this stage of development, Intermediate teams will begin to realize the importance of build speed. Of course, their build includes testing, right? As they run more and more builds and write more and more tests, things might start to bog down. A Novice team will naively push through the pain and wait for the build anyway. An Intermediate team will begin to realize that practice just won't scale. They will begin to change the way they test. They will start to look at how to truly unit test code. Too often, developers are writing heavyweight integration tests, using unit testing tools like JUnit. They get included in the same part of the build as regular unit tests, but cause the build to grind on for minutes, or even tens of minutes! A focused effort will migrate some tests to a separate integration testing, or "slow", build. Other tests will be modified to use mocks, stubs, or in-memory DBs. The team will take to heart the lessons outlined in a Michael Feathers blog article on Unit Testing Rules. The end result: lots of fast running tests that protect them from themselves, or the mistakes they might make in the future.

As teams progress into the Advanced stage, developers aren't happy with checking-in only once a day. They have their fast running tests. They always know when everything is working. An Advanced team will start to get into informal "races" against the lava lamp. They try hard to fix any build breakage before the lava starts to boil. That's a pretty small window, say 5 or 10 minutes for most lava lamps. Only teams with really fast development turnaround can beat the lava. That means fast builds are essential. An Advanced team will setup their build so that when it takes too long, it will break in spite of itself. A really forward thinking team will start this spiteful build breakage timer at the beginning of the project. Set it for 1 minute to start. Once the team hits the one minute mark, it's time to gather round the table for a discussion. "Do we really want our builds to take more than a minute?" The answer usually ends up being, yes. Bump it up to 2 minutes. It's a promise to have another conversation. Should you go to 3 minutes? Fight it. Advanced teams will know the right limit for them.

Either way, the Advanced team will have a flurry of commits all day long. Developers will be optimistically committing changes left and right. They are backed by good tests, a fast build and continuous feedback. Why not? Their velocity shoots through the roof. They are almost at that CI nirvana. Are they willing to get really radical? Some teams will never feel compelled to go any further.

Other teams, the truly Insane teams, will push it to the limit in every direction. They take their builds so seriously that they won't even let changes pollute their codebase unless they pass a rigorous set of tests. If your change doesn't stand up to such scrutiny, it gets the boot. Your changeset comes staggering back to you battered and bruised.

Automatically rolling back any changes that break the integration build might seem radical, but think about those teams doing manual CI. That's pretty much what they were doing. The schlub that was sitting there running their code on the clean "integration" machine would probably get harassed if it was taking too long. Eventually they would be peer pressured into a rollback so that someone else could have a go at it. The Insane team won't see much difference. Whatever the problem might be, it shouldn't be slowing everyone down. Keep that assembly line pumping!

I have worked with teams throughout this spectrum. These characteristics and their impact on the team are truly exciting. Advanced and Insane team members have a twinkle in their eye when they get to work in the morning, because they know that nothing stands in their way. They are free to create, in code, a software masterpiece.

Responses[3]

Posted by Jeffery Fredrick on September 11, 2009 4:02:00 PM EDT#