Table of Contents
A build is generally seen as the process of converting source code into artifacts and outputting reports. When a build is requested, the source code is then transformed into executable code. This process, as a single event in time, is thought as complete; that the compiling and packaging that went into that build produced a distinct set of output unavailable for additional processes. But when a build is seen as part of a living process, it becomes possible to perform additional processes on that build.
AnthillPro’s concept of a Living Build treats builds not as distinct events in time but as part of a Build Life. The Build Life represents all the transformations a build has gone through, and traces that build back to the originating source. In effect, the Build Life (or a Living Build) is a map of what occurred during the build. A Build Life can then be thought of as a way to exactly trace a build back through its life to reveal all that went into it, as well as a way to revisit a build should you need to perform additional tests, etc.
Traditionally, each build is confined to a single event in time. To deal with this problem, different build types have been developed. For example, a Continuous Integration build would get the source from the SCM, build it, and then run unit tests.
A nightly build would get the source from the SCM, build it, and then run functional tests on it.
And a release build would get the source from the SCM, build it, and then deploy it to a target environment.
In each build type, there are actually two different processes going on. The first retrieves the source code from the repository and builds it (a pure build process). The second runs the unit tests, functional tests, or deployment.
Pure builds take source code from the SCM as input and transforms it to produce artifacts. The type of transformation depends on the type of source code. If the source code is in a compilable language, then the transformation will compile the source code, possibly linking and packaging it. If the source code is in a non-compilable language, then the transformation will probably just package it. Pure builds do not include secondary processes such as running unit or functional tests, or deploying the artifacts. Pure builds simply take the source as input and transform it into artifacts. In AnthillPro, pure build processes are implemented as originating workflows.
Secondary processes act on the artifact produced by the pure build. A secondary process can run unit tests or functional tests on the artifact, deploy the artifacts to a specified environment, or perform any other automated task using the artifacts. In AnthillPro, secondary processes are defined as non-originating workflows.
Living Builds, implemented as Build Lives in AnthillPro, provide traceability of artifacts from the pure build processes, or originating workflows, producing the artifacts to the secondary processes, or non-originating workflows, acting on the artifacts (see Workflows). Every Living Build must start with a pure build process that produces artifacts. Those artifacts are then forever associated with that specific Living Build instance. As AnthillPro runs secondary processes on the originating workflow, it implicitly uses artifacts associated with the Living Build, thus providing traceability back to the source.
A Build Life must start out with a single pure build process, the originating workflow in AnthillPro. After the originating workflow (pure build), an unlimited number of secondary processes, the non-originating workflows, is possible. The one exception to this rule is archiving. When a Living Build is archived, all the artifacts are deleted to save space. Later, if you need to retrieve it from the archive, the pure build process will execute again to recreate the artifacts (see Archive and Unarchive Build Life).
Agents allow resource intensive tasks to be offloaded from the central server. To make better use of available resources and allow for higher build throughput, simultaneous builds can be performed on different agents. Agents can participate in multiple environments.
AnthillPro agents are light-weight Java processes that run on the agent machine. The agent contacts the server, via a TCP/IP socket, whenever the agent process is started. Since the agent can communicate with the central server via a WAN link, it need not be on the same network as the central server. However, the agent must be able to open a socket connection to the server. By default, all communication between the central server and the agent is not secure. Communication can be secured using SSL. See Configure Server-Agent SSL.
Before an agent can start accepting work from the central server, it must first be configured (see Manage Agents). During installation, you need to provide the agent’s location to the central server. Once the agent is configured, it can then start to accept work from the server. If for some reason you don’t want an agent to perform work, AnthillPro allows you to ignore that agent.
When configuring the central server to use an agent, you can assign a throughput metric. The throughput metric is an arbitrary integer indicating the relative throughput of one agent relative to another. For example, if one agent is running on a four CPU machine and a second is running on a single CPU machine, then the throughput metric of the first agent may be four and the throughput metric of the second agent may be one. Based on current job loads and throughput, jobs will be delegated to agents (with the use of agent filters).
The agent must be assigned to one or more environment(s). An agent not part of an environment will have few, if any, jobs assigned to it. AnthillPro allows you to also assign an agent to multiple environments, thus allowing for greater distribution when the server is handing out work.
Agent filters are used to select one or more agent(s) at run time, and are associated with a job. Agents need to be selected because all jobs within AnthillPro are executed on agents. AnthillPro uses agent filters to figure out on which agent a job should be executed. It is important to point out that agents are allocated to environments and agent filters are scoped by environments. When an agent filter is asked to select one or more agents, it selects them from the agents available in the environment within which it is executing. See Configure and Edit Agent Filters.
There are three different types of agent filters in AnthillPro:
Any Agent Filter returns all online agents in the environment. The agents returned are ordered by a combination of their current load and throughput metric, and then the server selects the agent(s) to use based on their loads.
Fixed Agent Filter returns a specified agent. Use of this agent filter is discouraged because the job will fail if the agent is not available in the selected environment.
Scripted Agent Filter returns any agent that matches an arbitrary script. Usually this is used to check variables on agents. For example, to find an agent with a specific tool installed or one that is used for a special purpose. The script-driven filter will also allow you to specify the number of agents to use. For simple jobs, the default of one is appropriate. For load testing, where a set number would be useful, putting in a number like five would work better. You can also instruct AnthillPro to run the job on every matching machine in an environment. This is pretty typical for doing things like deploying static HTML to a load balanced set of front-end servers. The script knows how to select the front-end servers, and the number setting directs it to deploy to all of them.
Agent variables allow the user to configure a build or other process using parameters with values dependent on the agent on which the build (or process) is actually executed. For example, an agent variable can be used to specify that an agent’s local settings be used during a build, etc. One thing too keep in mind is that agent variables are resolved at run time (i.e., when the build is actually run), and on the Agent on which the build is running. See Manage Agents.
Implicit variables have a designation indicating where they come from. All environment variables as well as all Java system variables on an agent are automatically loaded and made available as agent variables.
Agent variables are made possible by a distributed variable service. You can add new agent variables through the user interface on the central server. These variables are immediately usable, just like the implicit environment and system variables.
You’re configuring a build and need to use a specific version of Ant. You have a Linux machine and a Windows machine, both with AnthillPro agents, that could end up running the build. Further, Ant is installed in different locations on the Linux and Windows machines. You need to use an agent variable to configure the build.
Using agent variables, you can define a variable named
user/ANT_HOME_1_6_5 on both agents. On the Windows agent,
the value of the variable would be user/ANT_HOME_1_6_5 =
C:\apache\apache-ant-1.6.5. On the Linux agent, the value of
the variable would be user/ANT_HOME_1_6_5 =
/opt/apache/apache-ant-1.6.5.
The ‘user’ designation before the variable is optional, and may be useful for consistency.
By configuring the AnthillPro agent variable as above, either agent would be able to perform the job.
User access to an agent is managed on the Security tab. Administrators can define what roles have access to read, write, or determine security for agents. You need administrative permissions to set agent security. See Manage Security.
A schedule defines the intervals at which some actions should be executed by AnthillPro, and is used by triggers, for cleanup, by the Derby database backup, etc. A schedule may either be a simple interval that fires after a certain number of minutes has elapsed, or it might specify or restrict the exact time of day the actions should occur on (either interval or Cron). See Triggers and Scheduled Builds.
It is up to various items, such as workflow triggers, to register themselves with the schedule. The schedule fires at the designated time, and the event listeners (of triggers, etc.), when the schedule fires, are what actually kick off the desired action.
A list of existing schedules is found under the Schedules link of the System tab, where you can create, delete, and edit schedules. When setting a schedule, you have the option of using either a simple interval schedule or a Cron scheduler.
Interval Schedule is the more basic form of a schedule, with the start time and number of minutes between firings specified. The start time is used to calculate offsets. If the schedule is an hourly schedule, a start time of 00:30 will cause the schedule to trigger at half past the hour every hour, while a start time of 00:00 will cause it to trigger on the hour.
If a schedule can’t be satisfied by the interval scheduler, the Cron scheduler should meet the need. Common uses for it are to shut down schedules during times set aside for routine server maintenance, or to run the schedule only on weekdays.
An environment is a partition grid of agents that is specific for different stages of a project Life-Cycle (QA, PROD, etc.). Each environment may also be configured to a specific technology (Java, .NET, etc.).
Workflows within AnthillPro execute on a specified environment. This allows the definition of a single deployment workflow that can then be used to deploy the application to separate QA and production environments, etc. The same deployment workflow executed on the QA environment deploys the application to the QA environment. When executed on the production environment, it deploys the application to the production environment.
Agents can participate in multiple environments. This allows the use of an agent that communicates with a network deployment manager capable of deploying applications to multiple environments.
An environment group determines the set of environments that project workflows may be executed on. Each environment group must contain at least one environment (typically models a development, QA, production, or other environment).
In AnthillPro, the build farm is an implied environment and can’t be deleted. It provides a default environment containing all agents used for pure-build processes. Because there are no restrictions on how the Build-Farm is used, it may be used for deployments as well. Additionally, AnthillPro has no restriction to prevent you from using other environment groups for pure build processes.
Consider an organization that develops software using two different technologies. The first is J2EE. Our hypothetical organization has a development, a QA, and a production J2EE environment, with each containing application servers configured to the corporate standard. The second technology used is C++, with development, QA, and production environments for all C++ projects. The C++ environments use different physical servers than the corresponding J2EE environments. In this scenario, J2EE projects and C++ projects are configured into two environment groups: one called ‘J2EE Env Group’, and the other called ‘C++ Env Group’. The ‘J2EE Env Group’ contains the J2EE development, QA, and production environment groups, and the ‘C++ Env Group’ contains the C++ environment groups. With these environment groups, the organization derailed any confusion about what set of servers a J2EE or C++ project should be deployed to.
User access to an environment is managed on the Security tab. Administrators can define what roles have access to read, write, or determine security for environments. You need administrative permissions to set environment security. See Manage Security.
Life-Cycle Models unite the concepts and strategies of Statuses, Stamps, Artifact Sets, and Cleanup, into a single collection. To create or edit Life-Cycle Models, you must have Administrative permission. Once a project’s Life-Cycle Model is created, it may be reused for other projects with similar Life-Cycles without having to reconfigure a list of Statuses, Stamps, and Cleanup. See Using Life-Cycle Models.
In AnthillPro, there are three project types: Life-Cycle Based, used to run builds, deployments, etc.; Operational (Non-Life-Cycle Based), used for administrative, operational, and system maintenance; and Codestation Projects, used to manage dependencies of third-party tool kits and software libraries.
Life-Cycle Based Projects provide a wealth of information and visibility into the build and release cycle. See Life-Cycle Based Projects.
Operational (Non-Life-Cycle Based) Projects provide a simplified interface that allows AnthillPro users to execute ancillary tasks not easily run during a build, deployment, etc., workflow. See Operational (Non-Life-Cycle Based) Projects.
Codestation Projects model products not built by your organization as an AnthillPro project. This allows the use of third-party tools as dependencies, testing them as new releases come out to track their approval status within the Application. See Codestation Projects.
A Life-Cycle Based Project creates a Build Life when a workflow is run, and is used for builds, deployments, etc. See Simple Continuous Integration and Run a Simple Deployment.
A Life-Cycle Based Project is defined by its:
Use Operational Projects to define workflows by specifying the individual steps involved. Steps can be run in parallel, on different agents, execute arbitrary commands, integrate with third-party tools, execute scripts, or perform any task or series of tasks that a normal project can perform. See Operational Projects.
An Operational Project is defined by its:
Because of their flexibility, Operational Projects can be used to automate most tasks associated with the traditional build and release cycles. For example, an Operational Project can be used to verify the integrity of deployed artifacts; start and recycle services on restricted machines; and automate other process that do not fit neatly into a traditional build.
In addition to making stored artifacts available to the build process, Codestation Projects also make the configured dependency graph, and the artifacts themselves, available to developers. Through either an IDE plug-in, a command-line utility, or Ant tasks, a Codestation resolve operation on the developer’s machine fetches the appropriate artifacts. See Using Codestation Projects and Codestation (Developers).
A Life-Cycle Based Project utilizes a Life-Cycle Model to unite existing AnthillPro concepts into a single library. Once a Life-Cycle Model is created, it may be used for other projects with similar Life-Cycles. See Using Life-Cycle Models.
A Life-Cycle Model consists of:
Cleanup Policy specifies when to delete information about old Build Lives and other tasks associated with the project.
Stamp Style Group creates common names for types of stamps.
Artifact Set Group is a grouping of related build products, such as the scripts used to populate a database, artwork for a video game, a CD ISO, or code library. Because most projects generate similar types of artifacts, the artifact set group holds the names of artifact sets for a type of project.
Status Group is a set of common names for statuses. ‘Failed’ and ‘Successful’ are default status names, but common other names might be ‘Deployed to Test’, ‘Deployed to Production’, or ‘Retired’.
Essentially, a project’s environment groups is the collection of environments it uses. If a project will never build on or deploy to certain environments, an environment group can be used to help segregate the project.
Workflows, which can be thought of as processes, manage the order and parallelization of jobs. When manually starting a build or scheduling work, a workflow is being executed. Workflows assemble many of the other configurable items into something usable:
A job is a series of steps detailing how to get something done. It may specify the steps in a build, deployment, or other business need. The job configuration also dictates what types and number of machines the job will be run on (a deployment job may target every machine in a cluster).
A trigger is an automated mechanism for kicking off a workflow.
Source Configuration identifies exactly which source code should be retrieved from a repository. How these are configured will vary between source code management tools.
A workflow is a series of jobs that should be run, how they should be run, and where they should be run. Workflows assemble jobs into processes, and are the unit of automation. Workflows may be intimidating at first, because much of what has been configured elsewhere is assembled in a workflow; however, understanding workflows is not as daunting as it may first seem. This section will give you a brief introduction to how AnthillPro uses workflows.
To start, AnthillPro has two basic types: originating and non-origination. Originating workflows are used to start Build Lives, utilizing Life-Cycle Models, source configuration, and dependency information. Non-origination workflows specify secondary processes that can be run on an existing Build Life (e.g., a deployment). Underneath, the difference is that originating workflows have corresponding BuildProfile objects.
Workflows can either be created for individual projects or be used by multiple projects. To use the same Workflow for multiple projects, a Workflow Library (Workflow Definition) is created. Once configured, Workflow Definitions are added to existing projects as part of the project-workflow creation process (on the project-workflow Definition tab). Workflow Definitions do not necessarily have to be placed in Workflow Library folder. Any Workflows (on the Administration page) not in a folder or as part of a project subdirectory are Workflow Definitions. See Use the Workflow Library.
To see how the components fit together for originating workflows, see Simple Continuous Integration. The Run a Simple Deployment section highlights non-originating workflows.
Workflow Definition. Defines which job should be run, specifies the order of jobs, as well as the elements to be run in parallel.
Environment. For builds, this will almost always be the Build-Farm. For workflows that handle deployments, these might be deployment targets like QA Servers or production.
An originating workflow may only select one environment, while a non-originating can select any number of target environments.
Notification Scheme. Determines who to notify of build status, what conditions to notify them on, and what mechanism to notify them with.
Lockable Resource. Specifies a resource that might be used by multiple workflows, gives it a name, and forces arbitrary workflows to run one at a time or in small groups.
Properties. Identifies various properties that must be specified when the workflow is executed. See Workflow Property.
Triggers. Allows workflow to be kicked off automatically.
Source Configuration (Originating Only). Specifies the source that should be used for the workflow. Often this means selecting a branch of your project to run a workflow against. By creating a workflow for each branch, the stamping strategy used for each can be unique, while activities and jobs only need to be configured once.
Stamp Mapping (Originating Only). For every stamp style in the project’s stamp style group, one must specify what stamping strategy will be used. A typical stamp style group might specify the styles of development and production. In the workflow, you map the concept of a style used for development builds to a version number and strategy for incrementing that number.
Dependencies (Originating Only). Specifies the specific artifacts, including version, from other projects that should be retrieved in order to support the workflow. Also specifies the artifacts from other projects and the directory in the checked-out source that those artifacts should be placed in.
Artifacts (Originating Only). For any artifact associated with this project, the files produced by a workflow published as the artifact.
User access to a workflow is managed on the Security tab. Administrators can define what roles have access to read, write, or determine security for workflows. You need administrative permissions to set environment security. See Manage Security.
Job configuration comprises a major component of a project. Each job step may be laid out explicitly or generated implicitly by a special job definition. Typical jobs are "build", "deploy", or "run functional tests".
AnthillPro provides a Job Wizard to assist in the creation of build jobs. The build job creates the series of steps behind the scenes (the available steps depend on the source control system being used). The Wizard takes you through the steps to configure the builder and the publisher, as well as selecting the agent filter required by the job. See Simple Continuous Integration.
Quiet periods are configured on the project (Administration tab), and play an integral part in ensuring that the source code AnthillPro obtains from the SCM contains a consistent set of changes. The quiet period is a measure of time that must pass without anyone committing changes to the source. When the specified time has passed without commits, AnthillPro runs a build.
The quiet period also allows AnthillPro to perform intelligent merging of build requests. When two requests of related projects are within the same quiet period, AnthillPro checks the dependency graph to see if the can be combined into a single build.
To understand the need for a quiet period, consider this: A developer, let’s call him Henry, is working on source code changes. In order to resolve the issue, Henry needs to edit five source code files: two are in one directory and three in another. Once Henry completed the source code changes and tested them on his development environment, he starts committing the changes to the SCM. First, Henry commits the two files in one directory, lets call this Commit One. Then, in a separate commit (Commit Two), Henry commits the three files in the other directory. We also need to point out that the files included in Commit One will not compile without the changes in Commit Two. It is possible that a trigger might kick off a build of Henry’s project between Commit One and Commit Two. If that happened, without the use of a quiet period, the sources obtained for the build would contain Commit One but not Commit Two, and thus the build would fail.
If a trigger fired between Commit One and Commit Two, the quiet period would notice that Commit One took place just 10 seconds ago, which is shorter than the quiet period duration, and would thus wait for the quiet period duration to expire. The idea is that the quiet period duration is long enough so that it would allow enough time for Henry to perform both Commit One and Commit Two.
A trigger is an automated mechanism for starting a workflow, and may act on a schedule to build routinely or listen for notifications from an external source. By default, a trigger requesting an originating workflow will make a soft request. The build will be canceled if there were no changes to source control. If a trigger uses a schedule that fires every 10 minutes, there will not be a new build every 10 minutes unless new changes are committed during that time. As an alternative, the trigger can be configured to always execute the workflow regardless of changes. This may make more sense for nightly builds.
A common pairing would be to use a rapid scheduled trigger or repository trigger to run builds throughout the day and a nightly schedule to deploy to the testing environment (for the front line QA team to test the next day).
Trigger configuration falls into three general types:
Scheduled. Runs on a schedule. This may be every five minutes, hourly, nightly, or a more complicated schedule detailed by a Cron expression.
Repository. Runs when a specific AnthillPro URL is hit. This URL should be hit by a hook script in the SCM system when files are committed. These triggers could also be used as an alternative to the SOAP interface as an easy way for a third-party application to trigger builds.
Event. Allows for a custom filter that taps into the AnthillPro event service and triggers actions when certain events occur. For example, an event trigger would listen for build completed events, check those events against the project’s dependencies, and force a build of the project if a dependency builds successfully.
AnthillPro properties use the syntax
${property:[property_name]} to pass property values to
commands being executed by AnthillPro. Often, properties are used to
manage variables passed into commands, agent filters, and custom stamping
algorithm templates. Properties are resolved from the smallest scope to
the most general scope.
Property Resolution Order:
Job Iteration Property. Configured in the settings of an iterating job, they set the parameters for a single job that is run many times. See Job Iteration Properties.
Workflow Property. Used to specify a property for a given execution of a workflow. Workflow properties are also used to send a build to a particular platform when writing native code for multiple platforms, etc. See Workflow Properties.
Project Environment Property. Used to customize deployments based on the environment they are deployed to (see Environments and Environment Groups). Project Environment properties are automatically placed as environment variables for all commands run in the target environment. See Project Environment Properties.
Project Property. Used for all workflows regardless of the target environment. See Project Properties.
Server Property. Used to set default values for a particular property for all workflows and projects server wide. See Server Properties.
In addition, AnthillPro has two other property types (Builder and Agent), which are resolved separately from those listed above. See Setting Properties.
Dependencies between projects are very common. Often a project will produce a library used by another project. When few project dependencies exist, manually adding the products of one project into another is straightforward. However, when dependencies start to get complicated, you need an automated solution.
In AnthillPro, dependencies between projects are described at the workflow level. Workflows in dependencies specify the location (in the working directory) of the artifacts used by dependent projects. A workflow may be both a dependency and dependent, but cycles are not allowed. When a workflow uses dependency artifacts, AnthillPro maintains a record of which build life the artifacts came from. This provides instant traceability from the dependent project back to its own source files, as well as those of each of its dependencies. See Configuring Dependencies.
Dependency relationships are often used to configure build scheduling. Typically, there are three main approaches: use of a scheduling method independent of the dependency graph; a pulling model; and a pushing model:
Independent Scheduling. Related projects do not need to know about each other for independent scheduling. A dependent project starts each build by gathering the most recent artifacts from its dependencies.
Push Scheduling. Also called the bottom-up model. When a dependency builds or is promoted, the dependent workflow automatically builds. Because changes to the dependency result in changes to the dependent, rapid feedback as to whether the dependent was broken by the changes is available.
Pull Scheduling. The dependent is not automatically built whenever the dependencies are built. Rather, it is built according to its own schedules. When it builds, however, it starts the process by checking to see if there have been any changes to the dependencies and building them first if required. This is helpful to ensure that the latest check-ins are tested together.
AnthillPro provides an internal artifact store to track generated artifacts, tie them to build lives, and provide them to the dependent workflows. This store also makes it easy for a workflow to specify that it needs an artifact from a dependency that comes from a specific build life. For instance, this could be a requirement that the shared library has passed testing. Or an artifact could be locked down to a specific version. This approach is common as a project nears release because it helps prevent the team from making any deleterious changes. See Using Codestation Projects.
Codestation. The name for AnthillPro’s artifact repository. In addition to making stored artifacts available to the build process, Codestation also makes the configured dependency graph, and the artifacts themselves, available to developers. Through either an IDE plug-in, a command-line utility, or Ant tasks, a Codestation resolve operation on the developer’s machine fetches the appropriate artifacts.
Third-party Artifacts. Codestation provides a utility for uploading approved versions of third-party libraries. It provides these libraries to builds (similar to the way in-house dependencies are managed).
