View as printable PDF
Build & Release Automation Maturity Survey Results
Thank you for participating in the Build & Release Automation Maturity Survey sponsored by Urbancode. Hopefully, the survey results below will give you a snapshot as to where you stand relative to other organizations. In combination with Urbancode's recent webcast -- Trends in Continuous Integration, Automated Testing and Enterprise Build Automation -- the survey results may be helpful when developing the next "easy win" to expand automation and increase efficiency.
While a large number of respondents automate the build process, a majority of respondents utilize either a manual or mostly manual process for deployments to the test environment and for releases into production. Considering that nearly half of the respondents run automated functional tests (and a substantial number also run automated performance tests), it is surprising that only about 1 in 10 respondents have a fully automated deployment to the test environment. The data suggests that test automation has become an integral part of the application lifecycle.
Overall, the survey points to the increasing importance of automation. The data suggests that respondents have introduced automation within environmental silos, which may correlate to organizational structures and/or the availability of automation tools. The data also suggests that some organizations have yet to maximize automation for their deployment and release processes.
For teams trailing the norm, the survey shows that automation has become mainstream, and that the gap between those with the least amount of automation and those with the greatest can be substantial. Bridging this gap is not always easy, especially for those without existing lifecycle automation initiatives. Commercial tools are often used as a focal point for introducing automation into the application lifecycle. However, many automation tools only provide automation within a well-defined silo, and do not provide automation for and among the various lifecycle stages as AnthillPro does.
Basics and Build
While SCMs are used by most organizations, not all organizations have everything under source control -- which can reduce traceability and reproducibility. Items not in source control can make it difficult to track and/or reproduce when and where a bug was introduced. Often, automation efforts can be hampered if everything is not under source control because manual procedures may be necessary.
Often (not always), being able to build on the command line is a prerequisite for automation. If the automation is going to build through command-line tools, it's helpful if developers use the same tools in order to make their pre-commit testing better. Build tools, such as AnthillPro, can be used as a standard platform that allows command-line builds.
Not every project will be able to attain extremely rapid feedback; however, every project should provide feedback as rapidly as possible. The sooner a developer learns that her changes caused the build to break, the easier and quicker it is to address the problem. Notifying the team of a build breakage gives all the developers a better understanding of the latest code. Developers with this information are more likely to update regularly, and end up with faster and easier integrations. Because quick feedback is a major component of Continuous Integration, a Continuous Integration server like AnthillPro will provide commit triggers, automated testing, and flexible notifications.
Either answers 1 or 2 are preferred, so long as the options that are available are easily understood and logged. Teams without this ability tend to have a document that explains how a build should take place, or have a small number of build engineers who "just know the process". When knowledge is locked within a few keyed individuals, an organization may not be able to perform builds when a team member is out of the office. Automation tools, such as AnthillPro, can be used to solve the problem -- particularly for routine and boring tasks -- by enhancing knowledge sharing and process automation.
A centralized location that everyone can download the latest build provides openness and makes it easier to run tests. Among the available build and automation tools, AnthillPro provides a single platform that any team member can go to for information.
Be it a developer, quality assurance engineer, manager, or sales personnel, quick and easy downloads of the current production build just make sense. Also, for some companies, maintaining a repository of software that has gone to production is a regulatory requirement (ITIL refers to this repository as a Definitive Software Library). Many current build and automation servers do not provide the equivalent of AnthillPro's artifact repository (Codestation), which provides a centralized store for every build artifact.
Tests
Automated Unit testing is one of the first indications that automation is being pushed down to application lifecycle. As more Unit tests are run, confidence with every build increases -- and this spills over into other testing. The more automated tests run, the more quality assurance engineers are freed up to do high value testing, boundary checking, and creating more automated tests. More automated testing has been shown to increase not only productivity, but just as important product quality. Considering the rise in automated testing, lifecycle automation tools must be able to automatically run test suites and provide results to the appropriate team members. Unfortunately, AnthillPro is among the minority in providing this type of functionality.
Many teams have rigid lifecycle stages (pipelines), and if a bad build clogs the pipeline, maintaining the rest of the system is difficult. Automation solutions such as AnthillPro provide a way for teams to continue work on a previous build, or one that has not reached a particular stage in the pipeline, even while diagnosing the broken build. Being able to run tests against multiple builds in the various stages of the lifecycle provides the flexibility necessary to avoid catastrophic breakages.
Deployments and Releases
Manual processes tend to take up a lot of time (especially for development, quality assurance, or operations team members) that could really be spent doing something more valuable. In addition, manual processes are quite brittle and error prone. Automated deployments to the test environment can be used to tie the build artifacts directly to the changes that went into the build. Many build tools utilize a variation of AnthillPro's Living Build concept, which provides traceability and reproducibility for every build, deployment, and automated test suite.
Interestingly, many of the issues that effect deployments also hamper releases to production, but automated releases are less common among respondents. What's going on here? One reason could be that production deployment should be the one that is least vulnerable to manual errors; however, the costs of bad releases to customers or to production environments can be huge. This is the stage that should be most controlled and automated, but it appears to be less automated. In a similar fashion to deployments, automated solutions for releases must ensure that the build reaching production is the correct build, and that everything that went into the build is completely traceable.
Auditability is an important part of the software delivery lifecycle. For example, comparing the number of automated functional tests run, with the ability to automatically make determinations regarding the tests, reveals that the decision making process is generally manual. Above, the importance of production releases was established, and only half of the respondents can determine who pushed a particular build to production release. This leads to the obvious question: How do you decide to go to production if you don't know how the tests executed? In this scenario, the organization is dependent on manual approval, and careful communication about exactly what is approved and what is not. That's error prone. In regulated environments, this could be an audit finding. Any good automaton solution (such as AnthillPro) will allow for complete auditability back to the originating source code.
To find out how AnthillPro can help improve your software delivery process, explore the AnthillPro website or download a free, 30-day evaluation.
