AoAD2 Practice: Continuous Integration

Book cover for “The Art of Agile Development, Second Edition.”

Second Edition cover

This is a pre-release excerpt of The Art of Agile Development, Second Edition, to be published by O’Reilly in 2021. Visit the Second Edition home page for information about the open development process, additional excerpts, and more.

Your feedback is appreciated! To share your thoughts, join the AoAD2 open review mailing list.

This excerpt is copyright 2007, 2020, 2021 by James Shore and Shane Warden. Although you are welcome to share this link, do not distribute or republish the content without James Shore’s express written permission.

Revised: July 17, 2021

Continuous Integration

Audience
Programmers, Operations

We keep our latest code ready to release.

Most software development has a hidden delay between the team saying “we’re done” and when they’re actually ready to release. Sometimes that delay can stretch on for months. It’s the little things: getting everybody’s code to work together, writing a deploy script, pre-populating the database, and so forth.

When on-site customers are ready to release, you push a button and release.

Continuous integration is a better approach. Teams using continuous integration keep everyone’s code working together and ready to release. The ultimate goal of continuous integration is to make releasing a business decision, not a technical decision. When on-site customers are ready to release, you push a button and release. No fuss, no muss.

Allies
Collective Code Ownership
Refactoring

Continuous integration is also essential for collective code ownership and refactoring. If everybody is making changes to the same code, they need a way to share their work. Continuous integration is the best way to do so.

Continuous Integration is a Practice, Not a Tool

One of the early adopters of continuous integration was ThoughtWorks, a software development outsourcing firm. They built a tool called “CruiseControl” to automatically run their continuous integration scripts. They called it a continuous integration (CI) server, also known as a CI/CD server or build server.

Since then, the popularity of these tools has exploded. They’re so popular, the tools have taken over from the actual practice. Today, many people think “continuous integration” means using a CI server.

Continuous integration is about much more than running a build.

It’s not true. CI servers only handle one small part of continuous integration: they build and merge code on cue. But continuous integration is about much more than running a build. Fundamentally, it’s about being able to release your team’s latest work at will. No tool can do that for you. It requires three things:

Integrate many times per day

Integration means merging together all the code the team has written. Typically, it involves merging everyone’s code into a common branch of your source code repository. That branch goes by a variety of names: “main,” “master,” and “trunk” are common. I use “integration,” because I like clear names, and that’s what the branch is for. But you can use whatever name you like.

Teams practicing continuous integration integrate as often as possible. This is the “continuous” part of continuous integration. People integrate every time they complete a task, before and after every major refactoring, and any time they’re about to switch gears. The elapsed time can be anywhere from a few minutes to a few hours, depending on the work. The more often, the better. Some teams even integrate with every commit.

If you’ve ever experienced a painful multiday merge, integrating so often probably seems foolish. Why go through that pain?

The secret of continuous integration is that it actually reduces the risk of a bad merge. The more often you integrate, the less painful it is. More frequent integrations mean smaller merges, and smaller merges mean less chance of merge conflicts. Teams using continuous integration still have occasional merge conflicts, but they’re rare, and easily resolved.

Never break the integration build
The integration branch must always build and pass its tests.

When was the last time you spent hours chasing down a bug in your code, only to find that it wasn’t your code at all, but an out-of-date configuration, or somebody else’s code? Conversely, when was the last time you spent hours blaming a problem on your configuration or somebody else’s code, only to find that it was your code all along? To prevent these problems, the integration branch needs to be known-good. It must always build and pass its tests.

Allies
Zero Friction
Test-Driven Development
Fast Reliable Tests

This is actually easier than you might think. You’ll need an automated build with a good suite of tests, but once you have that, guaranteeing a known-good integration branch is just a matter of validating the merged code before promoting it to the integration branch. That way, if the build fails, the integration branch remains in its previous, known-good state.

However, the build must be fast, finishing in less than ten minutes. If it isn’t, it’s too hard to share code between team members. You can work around a slow build with multi-stage integration, as I discuss in “Multistage Integration Builds” on page XX.

Keep the integration branch ready to release

Every integration should get as close to a real release as possible. The goal is to make preparing for release such an ordinary occurrence that, when you actually do release, it’s a non-event. One team I worked with got to the point that they were releasing multiple times per week. They wrote a small mobile app with a big red button. When they were ready to release, they’d go to the local pub, order a round, and push the button.

Allies
Done Done
Build for Operation
Feature Flags

This means that every story includes tasks to update the build and deployment scripts, when needed. Code changes are accompanied by tests. Code quality problems are addressed. Data migrations are scripted. Important but invisible stories such as logging and auditing are prioritized alongside their features. Incomplete work is hidden behind feature flags or keystones.

“Getting as close as possible to a real release” includes running the deployment scripts and seeing them actually work. You don’t need to deploy to production—that’s continuous deployment, a more advanced practice—but you should deploy to a test environment. The same goes for software that isn’t online. If you’re building embedded software, install it to test hardware or a simulator. If you’re building a mobile app, create a submission package. If you’re building a desktop app, build an install package.

Don’t save the grunt work for the end. (See “Key Idea: Minimize Work in Progress” on page XX.) Take care of it continuously, throughout development. From the very first day, focus on creating a walking skeleton that could be released, if it only had a bit more meat on its bones, and steadily add to it with every story and task.

The Many Flavors of Continuous Integration

Continuous integration is so popular, and so misunderstood, people keep coming up with new terms for different aspects of the underlying idea:

  • CI Server. A tool that automatically runs build scripts. Not continuous integration at all.

  • Trunk-based development. Emphasizes the “integration” part of continuous integration. [Hammant 2020]

  • Continuous delivery. Emphasizes the “deploy” part of continuous integration. [Humble and Farley 2010] Commonly thought of as “continuous integration + deploy to test environment.”

  • Continuous deployment. A genuinely new practice. It deploys to production with every integration. Commonly thought of as “continuous delivery + deploy to production.”

Although continuous delivery is often seen as a separate practice from continuous integration, Kent Beck described it as part of continuous integration way back in 2004:

Integrate and build a complete product. If the goal is to burn a CD, burn a CD. If the goal is to deploy a web site, deploy a web site, even if it is to a test environment. Continuous integration should be complete enough that the eventual first deployment of the system is no big deal. [Beck 2004] (p. 50)

The Continuous Integration Dance

When you use continuous integration, every day follows a little choreographed dance:

  1. Sit down at a development workstation and reset it to a known-good state.

  2. Do work.

  3. Integrate (and possibly deploy) at every good opportunity.

  4. When you’re finished, clean up.

Ally
Zero Friction

These steps should all be automated as part of your zero-friction development environment.

For step 1, I make a script called reset_repo, or something similar. With git, the commands look like this (before error handling):

git clean -fdx                       # erase all local changes
git fetch -p origin      # get latest code from repo, removing outdated branches
git checkout integration             # switch to integration branch
git reset --hard origin/integration  # reset integration branch to match repo
git checkout -b $PRIVATE_BRANCH      # create a private branch for your work
$BUILD_COMMAND_HERE                  # verify that you’re in a known-good state

During step 2, you'll work normally, including committing and rebasing however your team prefers.

Step 3 is to integrate. You can do so any time the tests are passing. Try to integrate at least every few hours. When you’re ready to integrate, you’ll merge the latest integration branch changes into your code, make sure everything works together, then tell your CI server to test your code and merge it back into the integration branch.

Your integrate script will automate these steps for you. With git, it looks like this (before error handling):

git status --porcelain         # check for uncommitted changes (fail if any)
git pull origin integration    # merge integration branch into local code
$BUILD_COMMAND_HERE            # build, run tests (to check for merge errors)
$CI_COMMAND_HERE               # tell CI server to test and merge code

# The following steps help git resolve merge conflicts
$WAIT_COMMAND_HERE             # wait for CI server to finish
git checkout integration       # check out integration branch
git pull origin integration    # update integration branch from repo
git checkout $PRIVATE_BRANCH   # check out private branch
git merge integration          # merge repo's integration branch changes

The CI command varies according to your CI server, but will typically involve pushing your code to the repository. Be sure to set up your CI server to build and test your code before merging back to the integration branch, not after. That way your integration branch is always in a known-good state. If you don’t have a CI server that can do that, you can use the script in the next section instead.

Repeat steps 2 and 3 until you’re done for the day. After you integrate the final time, clean up:

git clean -fdx                       # erase all local changes
git checkout integration             # switch to integration branch
git branch -d $PRIVATE_BRANCH        # delete private branch
git fetch -p origin      # get latest code from repo, removing outdated branches
git reset --hard origin/integration  # reset integration branch to match repo

These scripts are only suggestions, of course. Feel free to customize them to match your team’s preferences.

Continuous Integration Without a CI Server

It’s surprisingly easy to perform continuous integration without a CI server. In some environments, this may be your best option, as cloud-based CI servers can be woefully underpowered. All you need is an integration machine—a spare development workstation or VM—and a small script.

To start with, program the integrate script you run on your development workstation to push your changes to a private branch. The git command is git push origin HEAD:$PRIVATE_BRANCH.

After the code has been pushed, manually log in to the integration machine and run a second integration script. It should check out the private branch, double-check that nobody else has integrated since you pushed it, run the build and tests, then merge the changes back into the integration branch.

Running the build and tests on a separate integration machine is essential for ensuring a known-good integration branch. It prevents “it worked on my machine” errors. With git, the commands look like this (before error handling):

# Get private branch
git clean -fdx                     # erase all local changes
git fetch origin                   # get latest code from repo
git checkout $PRIVATE_BRANCH       # check out private branch
git reset --hard origin/$PRIVATE_BRANCH   # reset private branch to match repo

# Check private branch
git merge integration --ff-only    # ensure integration branch has been merged
$BUILD_COMMAND_HERE                # build, run tests

# Merge private branch to integration branch using merge commit
git checkout integration           # check out integration branch
git merge $PRIVATE_BRANCH --no-ff --log=500 -m "INTEGRATE: $MESSAGE"  # merge
git push                           # push changes to repo

# Delete private branch
git branch -d $PRIVATE_BRANCH      # delete private branch locally
git push origin :$PRIVATE_BRANCH   # delete private branch from repo

If the script fails, fix it on your development machine and then integrate again. With this script, failed integrations don’t affect anyone else.

Note that only one person can integrate at a time, so you’ll need some way to control access. If you have a physical integration machine, whoever is sitting at the integration machine wins. If your integration machine is remote, you can configure it to only allow one login at a time.

This script is meant for synchronous integration, which means you have to wait for the integration to complete before doing other work. (I’ll explain more in a moment.) If you need asynchronous integration, you’re better off using a CI server. Multi-stage builds can use this script for the synchronous portion, for speed, then hand off to a CI server for the secondary build or deployment.

Synchronous vs. Asynchronous Integration

Allies
Zero Friction
Fast Reliable Tests

Continuous integration works best when you wait for the integration to complete. This is called synchronous integration, and it requires your build and tests to be fast—preferably completing in less than five minutes, or ten minutes at most. Achieving this speed is usually a matter of creating fast, reliable tests.

If the build takes too long, you’ll have to use asynchronous integration instead. In asynchronous integration, which requires a CI server, you start the integration process, then go do other work while the CI server runs the build. When the build is done, the CI server notifies you of the result.

Asynchronous integration sounds efficient, but it turns out to be problematic in practice. You check in the code, start working on something else, and then half an hour (or more) later, you get a notification that the build failed. Now you have to interrupt your work and go fix the problem. In theory, anyway. More often, it gets set aside until later. You end up with a chunk of work that’s hours or even days out of date, with much more likelihood of merge conflicts.

It’s a particular problem with poorly-configured CI servers. Although your CI server should only merge code to the integration branch after the build succeeds, so the integration branch is known-good, some CI servers default to merging the code first, then running the build afterwards. If the code breaks the build, then everybody who pulls from the integration branch is blocked.

Combine that with asynchronous integration, and you end up with a situation where people unwittingly check in broken code and then don’t fix it because they assume somebody else broke the build. The situation compounds, with error building on error. I’ve seen teams whose builds remained broken for days on end.

It’s better to make it structurally impossible to not break the build by testing the build first. It’s better still to use synchronous integration. When you integrate, wait for the integration to succeed. If it doesn’t, fix the problem immediately.

Multistage Integration Builds

Some teams have sophisticated tests, measuring qualities such as performance, load, or stability, that simply cannot finish in under ten minutes. For these teams, multistage integration is a good idea.

A multistage integration consists of two separate builds. The normal build, or commit build, contains all the items necessary to demonstrate that the software works: compiling, linting, unit tests, narrow integration tests, and a handful of smoke tests. This build runs synchronously, as usual.

When the commit build succeeds, the integration is considered to be successful, and the code is merged to the integration branch. Then a slower secondary build runs asynchronously. It contains the additional tests that don’t run in a normal build: performance tests, load tests, stability tests, and so forth. It can also include deploying the code to staging or production environments.

If the secondary build fails, everyone stops what they’re doing to fix the problem.

If the secondary build fails, the team is notified, and everyone stops what they’re doing to fix the problem. This ensures the team gets back to a known-good build quickly. However, failures in the secondary build should be rare. If they’re not, the commit build should be enhanced to detect those types of problems, so they can be fixed synchronously.

Although a multistage build can be a good idea for a mature codebase with sophisticated testing, most teams I encounter use multistage integration as a workaround for a slow test suite. In the long term, it’s better to improve the test suite instead.

In the short term, introducing a multistage integration can help you transition from asynchronous to synchronous integration. Put your fast tests in the commit build and your slow tests in the secondary build. But don’t stop there. Keep improving your tests, with the goal of eliminating the secondary build and running your integration synchronously.

Pull Requests and Code Reviews

Pull requests are too slow for continuous integration.

Pull requests aren’t a good fit for continuous integration. They’re too slow. Continuous integration works best when the time between integrations is very short—less than a few hours—and pull requests tend to take a day or two to approve. This makes merge conflicts much more likely, especially for teams using evolutionary design (which I’ll discuss in chapter “Design”).

Allies
Pair Programming
Mob Programming

Instead, use pairing or mobbing to eliminate the need for code review. Alternatively, if you want to keep code reviews, you can conduct code reviews after integrating, rather than as a pre-integration gate.

Although pull requests don’t work well on teams using continuous integration, they can still work as a coordination mechanism between teams that don’t share ownership.

Questions

You said we should clean up at the end of the day, but what if I have unfinished work and can’t integrate?

Allies
Feature Flags
Test-Driven Development

If you’re using feature flags and practicing test-driven development, you can integrate any time your tests are passing, which should be every few minutes. You shouldn’t ever be in a position where you can’t integrate.

If you’ve gotten stuck, it might be a good idea to delete the unfinished code. If you’ve been integrating frequently, there won’t be much. You’ll do a better job with a fresh start in the morning.

Isn’t synchronous integration a waste of time?

No, not if your build is as fast as it should be. It’s a good opportunity to take a break, clear your head, and think about design, refactoring opportunities, or next steps. In practice, the problems caused by asynchronous integration take more time.

We always seem to run into merge conflicts when we integrate. What are we doing wrong?

One cause of merge conflicts is infrequent integration. The less often you integrate, the more changes you have to merge. Try integrating more often.

Another possibility is that your changes are overlapping with other team members’ work. Try talking more about what you’re working on and coordinating more closely with the people that are working on related code. See “Making Collective Ownership Work” on page XX for details.

The CI server (or integration machine) constantly fails the build. How can we integrate more reliably?

Ally
Fast Reliable Tests

First, make sure you have reliable tests. Intermittent test failures are the most common reason I see for failed builds. If that isn’t the problem, you might need to merge and test your code locally before integrating. Alternatively, if you have frequent problems with incorrect dependencies, you might need to put more work into reproducible builds, as described in “Reproducible Builds” on page XX.

Prerequisites

Allies
Zero Friction
Pair Programming
Mob Programming
Test-Driven Development
Fast Reliable Tests

Continuous integration works best with synchronous integration, which requires a zero-friction build that takes less than ten minutes to complete. Otherwise, you’ll have to use asynchronous integration or multistage integration.

Asynchronous and multistage integration require the use of a CI server, and that server should be configured so that it validates the build before it merges changes to the integration branch. Otherwise, you’re likely to end up with compounding build errors.

Pull requests don’t work well with continuous integration, so another approach to code review is needed. Pairing or mobbing work best.

Continuous integration relies on a build and test suite that thoroughly tests your code, preferably with fast, reliable tests. Test-driven development using narrow, sociable tests is the best way to achieve this.

Indicators

When you integrate continuously:

  • Deploying and releasing is painless.

  • Your team experiences few integration conflicts and confusing integration bugs.

  • Team members can easily synchronize their work.

  • Your team can release with the push of a button, whenever your on-site customers are ready.

Alternatives and Experiments

Allies
Collective Code Ownership
Reflective Design
Refactoring
Feature Flags
Continuous Deployment

Continuous integration is essential for teams using collective code ownership and evolutionary design. Without it, significant refactoring becomes impractical, because it causes too many merge conflicts. That prevents the team from continuously improving the design, which is necessary for long-term success.

The most common alternative to continuous integration is feature branches, which merge from the integration branch on a regular basis, but only integrate to the integration branch when each feature is done. Although feature branches allow you to keep the integration branch ready to release, they usually don’t work well with collective code ownership and evolutionary design, because merges to the integration branch are too infrequent. Feature flags are a better way to keep the integration branch ready to release.

The experiments I’ve seen around continuous integration involve taking it to further extremes. Some teams integrate on every commit—every few minutes—or even every time the tests pass. The most popular experiment is continuous deployment, which has entered the mainstream, and is discussed later in this book.

Further Reading

Martin Fowler’s article, “Patterns for Managing Source Code Branches,” [Fowler 2020b] is an excellent resource for people interested in digging into the differences between feature branches, continuous integration, and other branching strategies.

XXX Continuous Delivery? [Humble and Farley 2010]

XXX Continuous Integration? [Duvall 2006] (BV prefers the MF article.)

Share your feedback about this excerpt on the AoAD2 mailing list! Sign up here.

For more excerpts from the book, or to get a copy of the Early Release, see the Second Edition home page.

If you liked this entry, check out my best writing and presentations, and consider subscribing to updates by email or RSS.