AoAD2 Chapter 1: What is Agile?

Book cover for “The Art of Agile Development, Second Edition.”

Second Edition cover

This is a pre-release excerpt of The Art of Agile Development, Second Edition, to be published by O’Reilly in 2021. Visit the Second Edition home page for information about the open development process, additional excerpts, and more.

Your feedback is appreciated! To share your thoughts, join the AoAD2 open review mailing list.

This excerpt is copyright 2007, 2020, 2021 by James Shore and Shane Warden. Although you are welcome to share this link, do not distribute or republish the content without James Shore’s express written permission.

What Is Agile?

Agile is everywhere. And paradoxically, nowhere.

In the 20 years after the Agile freight train roared into software developers’ consciousness, the number of companies calling themselves “Agile” increased by orders of magnitude. The number of teams actually taking an agile approach to their work? Not so much. “Agile,” the easily-repeated name, is enormously successful. The ideas behind Agile—well, most of them are ignored.

Let’s fix that.

Agile’s Genesis

In the 1990s, software development was believed to be in crisis. They actually called it that: “The Software Crisis.” Software projects were over-budget, late, didn’t meet requirements, and—according to the oft-quoted and ominously named “CHAOS Report”—nearly a third of them were cancelled outright. [Standish 1994]

Agile wasn’t a response to this crisis. Far from it. Agile was a response to the response.

To bring software development under control, big organizations had created highly detailed processes that defined exactly how software was to be created. Everything was tightly controlled so that no mistakes could be made. (In theory, anyway.)

First, business analysts would interview stakeholders and document the system requirements. Next, software architects would read the requirements documents and create detailed design documents specifying every component of the system and how they related to each other. Then programmers would convert the design documents to code. In some organizations, this was considered low-skill work—just a mechanical translation exercise.1 Meanwhile, test leads would use the same documents to generate test plans, and when coding was done, armies of QA personnel would manually follow those test plans and report variances as defects. After each phase, everything would be carefully documented, reviewed, and signed off.

1In fact, programming was considered such a mechanical activity that, for a while, there was a movement called CASE—Computer Aided Software Engineering—to automatically convert architectural diagrams into code.

These phase-based approaches came to be called “waterfall development,” or “phase-gate development.”2 If they sound like a ridiculous straw-man, well, consider yourself fortunate. Not every company used a document-heavy, phase-based process in the ’90s, but it was widely recognized as a logical and sensible way to work. Of course you needed to define requirements, then design, then implement, then test. Of course you needed to document every phase. This was discipline. This was engineering. How else could you possibly succeed?

2Waterfall is often mistakenly attributed to a 1970 paper by Winston Royce. But phase-based approaches date back to the 1950’s, and Royce’s paper was largely ignored until the late 1980’s, when it was used to describe what people were already doing. Laurent Bossavit explores the mischaracterizations of Royce’s paper, along with other common software development myths, in his superb book, The Leprechauns of Software Engineering: How Folklore Turns Into Fact and What To Do About It [Bossavit 2013] (Chapter 7).

Born Out of Crisis

Big companies defined their processes in excruciating detail. Roles, responsibilities, document templates, modeling languages, change control boards... every aspect of development was strictly defined and controlled. If a project didn’t succeed—and, according to the CHAOS Report, less than a sixth of them did—it was because the process needed more detail, more documents, more sign-offs. It resulted in a massive amount of documentation. Martin Fowler called it “The Almighty Thud.” [Fowler 1997]

This wasn’t a great way to work. It was bureaucratic and dehumanizing. Skill didn’t seem to matter as much as adherence to process. Programmers felt they were interchangeable cogs in an impersonal machine. It didn’t even work all that well.

So several people created simpler, slimmer, and less prescriptive methods for developing software. They were called “lightweight methods” in contrast to the heavyweight methods used by big companies. These new methods had names like “Adaptive Software Development,” “Crystal,” “Feature-Driven Development,” “Dynamic Systems Development Method,” “Extreme Programming,” and “Scrum.”

By the late ’90s, these methods were attracting serious attention. Extreme Programming, in particular, saw an explosion of grass-roots interest among programmers. In 2001, 17 of the lightweight methodology proponents met at a ski resort in Utah to discuss unifying their efforts.

The Agile Manifesto

“I personally didn’t expect this particular group of [people] to ever agree on anything substantive,” Alistair Cockburn said later.

And, in fact, after two days, they only accomplished two things: the name “Agile,” and a statement of four values (see the “Agile Values” figure). During the following months, over email, they hashed out twelve accompanying principles (see the “Agile Principles” figure).

This was the Agile Manifesto. It changed the world. So, as Alistair went on to say, they did agree on something substantive after all.3

3Alistair Cockburn, quoted by Jim Highsmith in [Highsmith 2001]. The full quote is, “I personally didn’t expect that this particular group of agilites to ever agree on anything substantive... Speaking for myself, I am delighted by the final phrasing [of the Manifesto]. I was surprised that the others appeared equally delighted by the final phrasing. So we did agree on something substantive.”

Agile Values

Figure 1. Agile Values

Agile Principles

Figure 2. Agile Principles

If your teams embody the Agile philosophy, then they’re Agile. If they don’t, they’re not.

But there was no unified Agile method. There never has been, and never will be. Agile is three things: the name, the values, and the principles. That’s it. It’s not something you can do. It’s a philosophy. A way of thinking about software development. You can’t “use” Agile or “do” Agile... you can only be Agile. Or not. If your teams embody the Agile philosophy, then they’re Agile. If they don’t, they’re not.

The Essence of Agile

Martin Fowler has made a career out of turning complicated software topics into well-considered, even-handed explanations. His explanation of “The Essence of Agile Software Development” is one of the best:

Agile Development is adaptive rather than predictive; people-oriented rather than process-oriented.4

4Fowler has expressed this idea in many ways over the years. It originated in [Fowler 2000].

Martin Fowler

Adaptive rather than predictive

Remember the CHAOS Report, which said that only one-sixth of software projects were successful? It had a very specific definition of success:

The project is completed on-time and on-budget, with all features and functions as originally specified.
The project is completed and operational but over-budget, over the time estimate, and offers fewer features and functions than originally specified.
The project is canceled at some point during the development cycle.

These definitions illustrate the predictive mindset perfectly. They’re all about conformance to plan. If you did what you said you were going to do, you were successful. If you didn’t, you weren’t! Easy.

It makes sense at first. But look closer. There’s something missing. As Ryan Nelson wrote in CIO Magazine [Nelson 2006]:

Projects that were found to meet all of the traditional criteria for success—time, budget, and specifications—may still be failures in the end because they fail to appeal to the intended users or because they ultimately fail to add much value to the business.

... Similarly, projects considered failures according to traditional IT metrics may wind up being successes because despite cost, time or specification problems, the system is loved by its target audience or provides unexpected value. For example, at a financial services company, a new system... was six months late and cost more than twice the original estimate (final cost was $5.7 million). But the project ultimately created a more adaptive organization (after 13 months) and was judged to be a great success—the company had a $33 million reduction in write-off accounts, and the reduced time-to-value and increased capacity resulted in a 50 percent increase in the number of concurrent collection strategy tests in production.

Agile teams define success as delivering value, not conforming to a plan.

Agile teams define success as delivering value, not conforming to a plan. In fact, truly Agile teams actively look for opportunities to increase value by changing their plans.

Look back at the Manifesto (see the “Agile Values” figure and the “Agile Principles” figure). “Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.” Take a moment to really study the Agile values and principles. How many relate to delivering valuable software and adapting to feedback?

People-oriented rather than process-oriented

Heavyweight processes tried to prevent errors by carefully defining every aspect of software development. By putting the “smarts” in the process, individual skill became less important. In theory, you could apply the same process over and over, with different people, and get the same results. (Come to think of it, they kind of did. Just not the results they wanted.)

Agile says people are the most important factor in software development success.

Agile says people are the most important factor in software development success. Not just their skills, but all aspects of their humanity. How well team members work together. How many distractions they encounter. How safe they feel. Whether they’re comfortable voicing dissent, and whether they feel motivated by their work.

Agile teams have a process—every team does, even if it’s implicit—but the process is in service of the humans, not the other way around. And Agile teams are in charge of their own process. When they think of a better way of working, they change it.

Look at the Manifesto again (see the “Agile Values” figure and the “Agile Principles” figure). “Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.” Which other values and principles relate to putting people first?

Why Agile Won

In the first ten years after the Manifesto, Agile faced enormous criticism. It was “undisciplined,” critics said. “It could never work.” Another ten years after that, the critics were silent. Agile was everywhere, at least in name. Heavyweight waterfall methods were effectively dead. Some younger programmers had trouble believing anybody ever could have worked that way.

It’s not that phase-based processes are inherently broken. They have their flaws, sure, but if you keep them slim and operate in a well-understood domain, waterfall-style methods can work. The problem was the heavyweight approaches big companies used. Ironically, the processes designed to prevent problems actually caused many of the problems organizations were seeing.

Learning and responding to change are at the heart of what Agile is all about.

It’s very difficult to imagine how software will work before you actually use it, and it’s even harder to think of absolutely everything your software needs to do. This is doubly true for people who aren’t actively involved with software development. As a result, it’s critically important to get working software in front of people as soon as possible. You need to get feedback about what’s missing or wrong, then change your plans based on what you learn. As the Manifesto says, “working software is the primary measure of progress.” Learning and responding to change are at the heart of what Agile is all about.

Those heavyweight processes put so much emphasis on process controls and documentation and sign-offs that they incurred a huge amount of delay and overhead. They took years to produce working software, and they had nothing concrete to show until near the end. Instead of welcoming change, they actively worked to prevent change. They actually had a dedicated part of the process, the Change Control Board, whose primary purpose was to say “no” to change requests. (Or, more accurately, “yes, but it will cost you.”)

All of this added up to projects that spent years in development before they had anything to show. When they did, it was too late and too expensive to make changes. They ultimately shipped software that didn’t do what customers needed.

Agile teams show working software, not documents.

Although there are a variety of approaches to Agile—and some of them are more about co-opting a popular name than following the actual philosophy—one thing they all have in common is a focus on making progress visible and allowing stakeholders to make course corrections as they go. This seems like a small thing, but it’s incredibly powerful. It’s why we no longer hear about the Software Crisis. Software is still late. It’s still over budget. But Agile teams show working software, not documents. Right from the beginning. And that’s huge.

There’s more to Agile than just providing visibility. But this one thing? This was enough. It’s why everybody wanted Agile.

Why Agile Works

Agile’s first break-out success was Extreme Programming (XP), a method with the slogan “Embrace Change.” XP mixed a healthy dose of philosophizing about software development with a pragmatic emphasis on making a difference for people involved. As the preface to the first XP book states:

XP makes two sets of promises.

  • To programmers, XP promises that they will be able to work on things that really matter, every day. They won’t have to face scary situations alone. They will be able to do everything in their power to make their system successful. They will make decisions that they can make best, and they won’t make decisions that they aren’t best qualified to make.

  • To customers and managers, XP promises that they will get the most possible value out of every programming week. Every few weeks they will be able to see concrete progress on goals they care about. They will be able to change direction of the project in the middle of development without incurring exorbitant costs.

In short, XP promises to reduce project risk, improve responsiveness to business changes, improve productivity throughout the life of a system, and add fun to building software in teams—all at the same time. Really. Quit laughing. [Beck 2000]

Extreme Programming Explained, First Edition

Some people did laugh. But others tried it, and they found that—contrary to common wisdom about how software development was supposed to work—XP really did everything it promised. And so, despite the laughter, XP thrived, and Agile along with it.

XP was the original poster child of Agile, donating a backbone of ideas and terminology that are still in use today. But the strength of Agile community is that it has always been a big tent. Agile isn’t limited to any one method. It’s constantly expanding to include new people and ideas.

Kent Beck’s Extreme Programming formed the seed. Mary and Tom Poppendieck’s Lean Software Development fleshed out the philosophy. Diana Larsen and Esther Derby’s retrospectives changed the way teams improve. Ken Schwaber and Jeff Sutherland’s Scrum democratized Agile, with a straightforward prescription and a network of trainers that spread Agile far and wide. David Anderson’s Kanban broke the bounds of rigid development cycles. The DevOps movement expanded the Agile team and brought XP’s ideas into the cloud era.

There’s so much more. Alistair Cockburn’s emphasis on communicating, cooperating teams. Jim Highsmith’s work on adaptive planning. Lisa Crispin and Janet Gregory’s support for testers. Bas Vodde and Craig Larman’s exploration of large-scale Agile systems. Johanna Rothman’s guidance for managers. Jeff Patton’s story maps. Pat Reed’s financial accounting. Eric Ries’ Lean Startup. And more, and more, and more. The contributions are too numerous to mention.6

6To the many significant contributors I’ve left out: please accept my apologies.

If you take all these ideas and group them into categories, five core concepts appear.

  • Rely on People. Build processes that understand and work with people’s essential humanity. Put decisions in the hands of those who are most qualified to make those decisions. Base your work on healthy, collaborative relationships.

  • Deliver Value. Seek feedback, experiment, and adapt your plans. Focus on producing valuable results. Consider partially-done work a cost, not a benefit. Deliver frequently.

  • Eliminate Waste. Work in small, reversible steps. Embrace the possibility of failure and design your plans to fail fast. Maximize work not done. Pursue throughput rather than efficiency.

  • Seek Technical Excellence. Enable agility via technical quality. Design for what is known, not what is speculated. Start simple and add complexity only in response to actual needs. Create systems that are easy to evolve, even—or especially—in unanticipated directions.

  • Improve Your Process. Experiment with new ideas. Tune and adapt what works. Never assume the established, popular way is the way that’s best for you.

Agile works because people make it work.

Agile is defined by the Manifesto, but the Manifesto is just the starting point. Agile works because people make it work. They take Agile’s ideas, adapt them to their situation, and never stop improving.

Why Agile Fails

Agile started as a grassroots movement. Its initial success was largely driven by programmers seeking better results and better quality of life. As that success grew, Agile’s momentum shifted from the underlying ideas to hype. Rather than saying, “let’s get better results by adapting our plans and putting people first,” organization leaders started saying, “Everybody’s talking about Agile. Get me some Agile.”

The thing is, there is no “Agile” to go get. It’s just a bunch of ideas. There are specific Agile approaches, such as Extreme Programming and Scrum, that will tell you how to be Agile, but you still have to be on board with the underlying philosophy.

And for a lot of organizations, that underlying philosophy—adapting plans and putting people first—is really, really foreign.

The tragedy of the cargo cult (see the “The Cargo Cults” sidebar) is its adherence to the superficial, outward signs of some idea combined with ignorance of how that idea actually works. In the story, the islanders replicated all the elements of cargo drops—the airstrip, the tower, the headphones—but didn’t understand the vast infrastructure that enabled airplanes to arrive.

People want Agile’s cargo, but they don’t understand the underlying philosophy, and often wouldn’t agree with it if they did.

The same tragedy occurs with Agile. People want Agile’s Cargo: better results, more visibility, fewer business failures. But they don’t understand the underlying philosophy, and often wouldn’t agree with it even if they did. They want to buy Agile, but you can’t buy an idea.

What they can buy is the outward signs of Agile. Stand-up meetings! Stories! Tools! Certifications! There’s lots of stuff labelled Agile, and plenty of people eager to sell it to you. It’s often sold as “enterprise-grade,” which is a way of saying “don’t worry, you won’t have to change.” Uncomfortable ideas like “adaptive planning” and “people-centric” are ignored.

And that’s how you get a cargo cult. All the activity; none of the results. The Agile part is missing.

“At my old company they wasted a huge number of man hours in meetings.”

“[Agile] cost an entire team (30+) people their jobs as they produced almost nothing for almost a year.”

“All [Agile] means is developers get shafted when the project changes... the day before delivery.”

“It’s because of Agile there are sh—ty developers and sh—ty products being shipped.”

“I can’t take this Agile crap any more. It’s lunacy.”

comments about Agile from around the Web

The name Agile is everywhere. The ideas of Agile aren’t. It’s become self-perpetuating: for many, the only Agile they know is Cargo Cult Agile.

Icon of person in a suit wearing coconut headphones

Cargo Cult Agile

It’s time to fix that. In the rest of this book, I’ll show you how to apply Agile ideas for real. Keep an eye out for the Cargo Cult Agilists shown in the margin. They’ll show you what not to do.

Ready? Let’s go.

Share your feedback about this excerpt on the AoAD2 mailing list! Sign up here.

For more excerpts from the book, or to get a copy of the Early Release, see the Second Edition home page.

If you liked this entry, check out my best writing and presentations, and consider subscribing to updates by email or RSS.