AoAD2 Practice: Continuous Integration

Book cover for “The Art of Agile Development, Second Edition.”

Second Edition cover

This is a pre-release excerpt of The Art of Agile Development, Second Edition, to be published by O’Reilly in 2021. Visit the Second Edition home page for information about the open development process, additional excerpts, and more.

Your feedback is appreciated! To share your thoughts, join the AoAD2 open review mailing list.

This excerpt is copyright 2007, 2020, 2021 by James Shore and Shane Warden. Although you are welcome to share this link, do not distribute or republish the content without James Shore’s express written permission.

Continuous Integration

Audience
Programmers, Operations

We keep our latest code ready to release.

Most software development efforts have a hidden delay between when the team says “we’re done” and when the software is actually ready to release. Sometimes that delay can stretch on for months. It’s the little things: getting everybody’s code to work together, writing a deploy script, pre-populating the database, and so forth.

When your customers are ready to release, you push a button and release.

Continuous integration is a better approach. Teams using continuous integration keep everyone’s code working together and ready to release. The ultimate goal of continuous integration is to make releasing a business decision, not a technical decision. When your on-site customers say it’s time to release, you push a button and release. No fuss, no muss.

Allies
Collective Code Ownership
Refactoring

Continuous integration is also essential for collective code ownership and refactoring. If everybody is making changes to the same code, they need a way to keep their changes in sync. Continuous integration is the best way to do so.

Continuous Integration is a Practice, Not a Tool

One of the early adopters of continuous integration was ThoughtWorks, a software development outsourcing firm. They built a tool called “CruiseControl” to automatically run their continuous integration scripts. They called it a continuous integration (CI) server, also known as a CI/CD server or build server.

Since then, the popularity of these tools has exploded. They’re so popular, the tools have taken over from the actual practice. Today, many people think “continuous integration” means using a CI server.

Continuous integration is about much more than running a build.

It’s not true. CI servers only handle one small part of continuous integration: they build and merge code on cue. But continuous integration is about much more than running a build. Fundamentally, it’s about being able to release your team’s latest work whenever your on-site customers want. No tool can do that for you.

Achieving this goal requires three things:

1. Integrate many times per day

Integration means merging together all the code the team has written. Typically, that means merging everyone’s code into a common branch of your source code repository. That branch goes by a variety of names: “main,” “master,” and “trunk” are common. I use “integration,” because I like clear names, and that’s what the branch is for. But you can use whatever name you like.

Ally
Task Planning

Teams practicing continuous integration integrate as often as possible. This is the “continuous” part of continuous integration. People integrate every time they complete a task card, before and after every major refactoring, and any time they’re about to switch gears. The elapsed time can be anywhere from a few minutes to a few hours, depending on the work. The more often, the better. Some teams even integrate with every commit.

If you’ve ever experienced a painful multiday merge, integrating so often probably seems foolish. Why go through that pain?

The more often you integrate, the less painful it is.

The secret of continuous integration is that it actually reduces the risk of a bad merge. The more often you integrate, the less painful it is. More frequent integrations mean smaller merges, and smaller merges mean less chance of merge conflicts. Teams using continuous integration still have occasional merge conflicts, but they’re rare, and easily resolved.

2. Never break the integration branch

When was the last time you spent hours chasing down a bug in your code, only to find that it wasn’t your code at all, but an out-of-date configuration, or somebody else’s code? Conversely, when was the last time you spent hours blaming a problem on your configuration or somebody else’s code, only to find that it was your code all along?

The integration branch must always build and pass its tests.

To prevent these problems, the integration branch needs to be known-good. Without exception, it must always build and pass its tests.

Ally
Zero Friction
Test-Driven Development

This is actually easier than you might think. You’ll need an automated build with a good suite of tests, but once you have that, guaranteeing a known-good integration branch is just a matter of validating the merged code before promoting it to the integration branch. That way, if the build fails, the integration branch remains in its previous, known-good state.

3. Keep the integration branch ready to release

Every integration should get as close to a real release as possible. The goal is to make preparing for release such an ordinary occurrence that, when you actually do release, it’s a non-event. One team I worked with got to the point that they were releasing multiple times per week. They wrote a small mobile app with a big red button. When they were ready to release, they’d go to the local pub, order a round, and push the button.

Allies
Done Done
Build for Operation
Feature Toggles
Continuous Deployment

This means that every story includes tasks to update the build and deployment scripts, when needed. Code changes are accompanied by tests. Code quality problems are addressed. Data migrations are scripted. Important but invisible stories such as logging and auditing are prioritized alongside their features. Incomplete work is hidden from users.

“Getting as close as possible to a real release” includes running the deployment scripts and seeing them actually work. You don’t need to deploy to production—that’s continuous deployment, a more advanced practice—but you should deploy to a test environment. The same goes for software that isn’t online. If you’re building embedded software, install to test hardware or a simulator. If you’re building a mobile app, create a submission package. If you’re building a desktop app, build an install package.

Don’t save the grunt work for the end. (See “Key Idea: Minimize Work in Progress” on p.XX.) Take care of it continuously, throughout development. From the very first day, focus on creating a walking skeleton that could be released, if it only had a bit more meat on its bones, and steadily add to it with every story and task.

The Many Flavors of Continuous Integration

Continuous integration is so popular, and so misunderstood, people keep coming up with new terms for different aspects of the underlying idea:

  • CI Server. A tool that automatically runs build scripts. Not continuous integration at all.

  • Trunk-based development. Emphasizes the “integration” part of continuous integration. [Hammant 2020]

  • Continuous delivery. Emphasizes the “deploy” part of continuous integration. [Humble and Farley 2010] Commonly thought of as “continuous integration + deploy to test environment.”

  • Continuous deployment. A genuinely new practice. It deploys to production with every integration. Commonly thought of as “continuous delivery + deploy to production.”

Although continuous delivery is often seen as a separate practice from continuous integration, Kent Beck described it as part of continuous integration way back in 2004:

Integrate and build a complete product. If the goal is to burn a CD, burn a CD. If the goal is to deploy a web site, deploy a web site, even if it is to a test environment. Continuous integration should be complete enough that the eventual first deployment of the system is no big deal. [Beck 2004] (p. 50)

Extreme Programming Explained, 2nd ed.

The Continuous Integration Dance

When you use continuous integration, every day follows a little choreographed dance:

  1. Sit down at a development workstation and reset it to a known-good state.

  2. Do work.

  3. Integrate (and possibly deploy) at every good opportunity.

  4. When you’re finished, clean up.

Ally
Zero Friction

These steps should all be automated as part of your zero-friction development environment.

For step 1, I make a script called reset_repo, or something similar. With git, the commands look like this (before error handling):

git clean -fdx                       # erase all local changes
git fetch origin                     # get latest code from repo
git reset --hard origin/integration  # reset to integration branch
git checkout -b $PRIVATE_BRANCH      # create a private branch for your work
$BUILD_COMMAND_HERE                  # verify that you’re in a known-good state

During step 2, you'll work normally, including committing and rebasing however your team prefers.

Step 3 is to integrate. You can do so any time the tests are passing. Try to integrate at least every few hours. When you’re ready to integrate, you’ll merge the latest integration branch changes into your code, make sure everything works together, then tell your CI server to test your code and merge it back into the integration branch.

Your integrate script will automate these steps for you. With git, it looks like this (before error handling):

git status --porcelain         # check for uncommitted changes (fail if any)
git pull origin integration    # merge integration branch into local code
$BUILD_COMMAND_HERE            # build, run tests (to check for merge errors)
$CI_COMMAND_HERE               # tell CI server to test and merge code

The integration command varies according to your CI server, but will typically involve pushing your code to the repository. Be sure to set up your CI server to build and test your code before merging back to the integration branch, not after. That way your integration branch is always in a known-good state. If you don’t have a CI server that can do that, you can use the script in “Continuous Integration Without a CI Server” on p.XX instead.

Repeat steps 2 and 3 until you’re done for the day. After you integrate the final time, clean up. With git, that means erasing the private branch:

git branch -d $PRIVATE_BRANCH

These scripts are only suggestions, of course. Feel free to customize them to match your team’s preferences.

Synchronous vs. Asynchronous Integration

Allies
Zero Friction

Continuous integration works best when you wait for the integration to complete. This is called synchronous integration, and it requires your build and tests to be fast—preferably completing in less than five minutes, or ten minutes at most. Achieving this speed is usually a matter of improving the team’s test suite. “Fast and Reliable Tests” on p.XX describes how.

If the build takes too long, you’ll have to use asynchronous integration instead. In asynchronous integration, which requires a CI server, you start the integration process, then go do other work while the CI server runs the build. When the build is done, the CI server notifies you of the result.

Asynchronous integration sounds efficient, but it turns out to be problematic in practice. You check in the code, start working on something else, and then half an hour (or more) later, you get a notification that the build failed. Now you have to interrupt your work and go fix the problem. In theory, anyway. More often, it gets set aside until later. You end up with a chunk of work that’s hours or even days out of date, with much more likelihood of merge conflicts.

It’s a particular problem with poorly-configured CI servers. Although your CI server should only promote code to the integration branch after the build succeeds, so the integration branch is known-good, some CI servers default to merging the code first, then running the build afterwards. If the code breaks the build, then everybody who pulls from the integration branch is blocked.

Combine that with asynchronous integration, and you end up with a situation where people unwittingly check in broken code and then don’t fix it because they assume somebody else broke the build. The situation compounds, with error building on error. I’ve seen teams whose builds remained broken for days on end.

It’s better to make it structurally impossible to not break the build by testing the build first. It’s better still to use synchronous integration. When you integrate, wait for the integration to succeed. If it doesn’t, fix the problem immediately.

Multistage Integration Builds

Some teams have sophisticated tests, measuring qualities such as performance, load, or stability, that simply cannot finish in under ten minutes. For these teams, multistage integration is a good idea.

A multistage integration consists of two separate builds. The normal build, or commit build, contains all the items necessary to demonstrate that the software works: compiling, linting, unit tests, narrow integration tests, and a handful of smoke tests. (See “Fast and Reliable Tests” on p.XX for details.) This build runs synchronously, as usual.

When the commit build succeeds, the integration is considered to be successful, and the code is promoted to the integration branch. Then a slower secondary build runs asynchronously. It contains the additional tests that don’t run in a normal build: performance tests, load tests, stability tests, and so forth. It can also include deploying the code to staging or production environments.

If the secondary build fails, everyone stops what they’re doing to fix the problem.

If the secondary build fails, the team is notified, and everyone stops what they’re doing to fix the problem. This ensures the team gets back to a known-good build quickly. However, failures in the secondary build should be rare. If they’re not, the commit build should be enhanced to detect those types of problems, so they can be fixed synchronously.

Although a multistage build can be a good idea for a mature codebase with sophisticated testing, most teams I encounter use multistage integration as a workaround for a slow test suite. In the long term, it’s better to improve the test suite instead.

In the short term, introducing a multistage integration can help you transition from asynchronous to synchronous integration. Put your fast tests in the commit build and your slow tests in the secondary build. But don’t stop there. Keep improving your tests, with the goal of eliminating the secondary build and running your integration synchronously.

Pull Requests and Code Reviews

Pull requests are too slow for continuous integration.

Pull requests aren’t a good fit for continuous integration. They’re too slow. Continuous integration works best when the time between integrations is very short—less than a few hours—and pull requests tend to take a day or two to approve. This makes merge conflicts much more likely, especially for teams using evolutionary design.

Allies
Pair Programming
Mob Programming

Instead, use pairing or mobbing to eliminate the need for code review. Alternatively, if you want to keep code reviews, you can conduct code reviews after integrating, rather than as a pre-integration gate.

Although pull requests don’t work well on teams using continuous integration, they can still work as a coordination mechanism between teams that don’t share ownership.

Questions

You said we should clean up at the end of the day, but what if I have unfinished work and can’t integrate?

Ally
Test-Driven Development

If you’re practicing test-driven development, you can integrate any time your tests are passing, which should be every few minutes. You shouldn’t ever be in a position where you can’t integrate. If you are, you’ve probably gotten stuck. It might be a good idea to delete the unfinished code and start fresh in the morning.

You don’t have to delete unfinished work, but if you’ve been integrating frequently, the loss of code will be minimal, and you’re likely to do a better job in the morning.

If we use synchronous integration, what should we do while waiting for the integration to complete?

Take a break. Get a cup of tea. Perform ergonomic stretches. Talk with your pair or mob about design, refactoring opportunities, or next steps. If your build is under ten minutes, you should have time to clear your head and consider the big picture without feeling like you’re wasting time.

We always seem to run into merge conflicts when we integrate. What are we doing wrong?

One cause of merge conflicts is infrequent integration. The less often you integrate, the more changes you have to merge. Try integrating more often.

Another possibility is that your changes are overlapping with other team members’ work. Try talking more about what you’re working on and coordinating more closely with the people that are working on related code. See “Making Collective Ownership Work” on p.XX for details.

The CI server (or integration machine) constantly fails the build. How can integrate more reliably?

You might have a problem with flaky tests (tests that fail intermittently). See “Fast and Reliable Tests” on p.XX for help.

If your tests are fine, you probably need to run more tests locally. Run a full build and test before merging the integration branch with your code. That will make sure your changes didn’t break anything. Then, after you merge, run another full build and test. That will make sure the merge didn’t break anything. At this point, the CI build should proceed without issue. If it doesn’t, it means your local configuration is different from the CI server configuration.

If you have frequent problems with mismatched configuration, you might need to put more work into reproducible builds. See “Reproducible Builds” on p.XX for details.

Prerequisites

Allies
Zero Friction
Pair Programming
Mob Programming
Test-Driven Development

Continuous integration works best with synchronous integration, which requires a zero-friction build that takes less than ten minutes to complete. Otherwise, you’ll have to use asynchronous integration or multistage integration.

Asynchronous and multistage integration require the use of a CI server, and that server has to be configured so that it validates the build before it promotes the changes to the integration branch. Otherwise, you’re likely to end up with compounding build errors.

Pull requests don’t work well with continuous integration, so another approach to code review is needed. Pairing or mobbing work best.

Continuous integration relies on a build and test suite that thoroughly tests your code, preferably with fast and reliable tests. Test-driven development using narrow, sociable tests is the best way to achieve this.

Indicators

When you integrate continuously:

  • Deploying and releasing is painless.

  • Your team experiences few integration conflicts and confusing integration bugs.

  • Team members can easily synchronize their work.

  • Your team can release whenever your on-site customers are ready.

  • You team can release with the push of a button.

Alternatives and Experiments

Ally
Collective Code Ownership
Reflective Design
Refactoring

Continuous integration is essential for teams using collective code ownership and evolutionary design. Without it, significant refactoring becomes impractical, because it causes too many merge conflicts. That prevents the team from using reflective design, which is necessary for long-term success.

The most common alternative to continuous integration is feature branches, which merge from the integration branch on a regular basis, but only integrate to the integration branch when each feature is done. Feature branches can do a good job of keeping the integration branch ready to release, but they’re still prone to merge conflicts. They don’t work well for teams using refactoring and reflective design.

Ally
Continuous Deployment

The experiments I’ve seen around continuous integration involve taking it to further extremes. Some teams integrate on every commit—every few minutes—or even every time the tests pass. The most popular experiment is continuous deployment, which has entered the mainstream, and is discussed later in this book.

Further Reading

Martin Fowler’s article, “Patterns for Managing Source Code Branches,” [Fowler 2020b] is an excellent resource for people interested in digging into the differences between feature branches, continuous integration, and other branching strategies.

XXX Continuous Delivery? [Humble and Farley 2010]

XXX Trunk-Based Development? [Hammant 2020]

Share your feedback about this excerpt on the AoAD2 mailing list! Sign up here.

For more excerpts from the book, or to get a copy of the Early Release, see the Second Edition home page.

If you liked this entry, check out my best writing and presentations, and consider subscribing to updates by email or RSS.