Our latest webcast - Enterprise Continuous Integration Maturity Model - presented tools for deciding how mature your organization is with ECI and automation.

The webcast was recorded, and is available here.
Its companion white-paper: Enterprise Continuous Integration Maturity Model

We had a lot of great questions were weren't able to answer live. We're taking the time to answer them on the blog for anyone who's interested.

You mention testing after deployment? Shouldn't it come first?

Testing should ideally happen before deployment to production. But many forms of testing can only happen after a deployment to a test environment.

I agree that 100% code coverage is insane, but what should be the right expectation in terms of coverage?

This is a very difficult question. Generally, more tests are better but when the value derived from more tests is less than the cost of adding them, stop. Those in favor of 100% coverage argue that teams get lazy and underestimate the value of tests in order to have an excuse to test less. Further if a benchmark like 80% coverage is set, many teams and individuals will test the easiest 80% of their application rather than the 80% where tests would be the most helpful. Dealing with these problems is why the target of 100% has started to become popular in leading edge agile shops.

Define "100%" testing for the Insane testing. How do you test software 100%?

Right... 100% coverage is mostly a bogus concept. Usually it's picking a basic metric like lines of code covered or 100% coverage of all apparent logical branches in the software. This things are measurable. It's generally impossible to test the system on every imaginable platform with every imaginable input.

Is 40% about the % of customers you see that have not adopted any CI concepts, let alone CI tools?

About 40% of the attendees of the webcast identified themselves as not using any CI tooling. Practices like frequent integrations to a common development code line may be a bit more common than that, but we weren't shocked at the response.

When you talk about automated deployments to test/prod environments, are you only referring to software, or are you also including other related automation like database changes, restarting instances of Weblogic or Websphere, etc?

We answered this in the webcast, but it was such a good question we'll touch on it here as well. We're referring to both moving the files to some location and installing them. For J2EE web applications, this could certainly involve restarting application servers, using their proprietary install tools, and/or making automated/scripted database updates.

Where do you start to automate deployment into a test environment?

If you have a fully scripted deployment, we often see this script just executed as part of a CI or nightly build process. If the team just has helper scripts or deployment steps, those need to be included in a tool that provides some deployment automation support - like AnthillPro.  As far as the maturity model, we see this behavior being integrated in at the Novice to Intermediate levels.

How do code reviews plug in to ECI?

Very interesting question. Ideally, if code reviews are recorded somewhere, they'd be integrated into ECI in parallel with source change information. That would allow the team to see which code was reviewed and which was not, perhaps allowing manual testers an additional hint about where to go digging.

At what point would CI be overkill for a development process?

We tend to use CI even for single developer projects. Simply getting feedback that the build works on a machine other than your own is powerful. Much of ECI though is focused on streamlining interactions between larger groups of people. Our rule of thumb is that if you do it twice, automate it.
Not everyone will want to reach the same levels of maturity though. As you look at your own needs, evaluate the payoff you'll see from increasing maturity against the effort.

To what degree should an organization take commonality of source/deploy/test environments and process to effectively be able to implement and maintain an effective ECI system?

Heterogeneous environments are reality. I would look to make certain classes of project homogeneous so developers can move between them with ease. Aside from that, heterogeneous environments should be able to be addressed well in ECI, however the work involved will be somewhat higher.

What is the typical timeframe for a company to achieve target CI?

This varies quite a bit. We see engineers subversively setting up a CI server that emails their work group in an afternoon and corporate efforts that take years to roll-out across the enterprise. In general, I'd expect an ECI effort to start yielding positive results after a month or two. With time, the scope and maturity of the effort can be expanded to get more and more out of the system.