The Art of Agile Development, Second Edition

For this week’s Tuesday Lunch & Learn livestream, I have a special show for you: the new edition of my book, The Art of Agile Development, Second Edition.

The Art of Agile Development came out in 2007. It’s a well-loved classic that’s still talked about today... and now it’s time for an update. In today’s episode, I talk about what’s new, what’s changed, and I read several excerpts from the upcoming Early Release.

If you prefer text, there’s a transcript below the video.

Edited Transcript

  1. Introduction
  2. Book Excerpt
  3. Discussion
  4. Certification
  5. Waterfall
  6. The Cover and Lost Subtitle
  7. Providing Feedback
  8. Commitments & Deadlines
  9. Changes in Second Edition
  10. Cargo Cult Agilists
  11. C-Levels
  12. Final Thoughts

Introduction

Welcome to Tuesday Lunch & Learn. I’m James Shore. Every week, we look at software engineering topics, and this week, I have a special show for you. I’m going to be reading from the new edition of my book, The Art of Agile Development. I’m really looking forward to it.

(Reading chat) UncleScientist, welcome to the show; third_man, welcome, thanks for coming back; GrumpyGameDev—(chat says: “false starts are the best starts”)—yes, we’ll just pretend that never happened. I’ll edit it right out of the final video.

Today we’re looking at the second edition of The Art of Agile Development. This, of course, is my book originally published in 2007. I originally wrote it along with Shane Warden. It’s, I think, a pretty well-loved book. It was pretty popular—people still talk about it today—and it’s overdue for an update. So I’m really happy to say that I am working on an update for it.

If you’d like to see what we’re doing with it, go right down here, jamesshore.com/s/aoad2, and you’ll see the current page. This is going to be developed entirely in the open. It’s coming out on O’Reilly Early Release later this month, which means you’ll be able to see the chapters that we’ve written so far. And then, as we write additional chapters, those are actually going to go up on my website for feedback and comments. Then we will take those comments and bring them into the actual book.

If you’d like to be informed when that happens, go to this shortlink here—let me put this up on the screen—right here, jamesshore.com/s/updates, or you can follow me on Twitter and look at the #aoad2 hashtag. I‘ll be putting announcements on Twitter as well.

Yeah, as UncleScientist mentions in the chat, I have updated the site design to go along with the upcoming release of the book and (laughs) I like it a lot better. If you saw the old design, the nicest thing I can say about it was that it was a very “classic” design. (laughs) But I’m much happier with the design now.

OFFY_98, welcome, thank’s for joining me. JitterTed, glad to see you again too.

What’s new... by the way, I want to mention, of course, that I am following with Twitch chat; your comments are very welcome. As we go along, please put them in and I’ll address them as we go.

Let me take you over to the upcoming table of contents. (Brings up book’s web page.) Now this is a draft table of contents.

The original edition of the book was aimed at the sort of “early majority” of companies who wanted to try out Agile. People had heard of Agile, but they weren’t really using it yet. So the first edition of the book was, “This is how you use Agile in your company.”

Now it’s 13 years later and everybody’s has heard of Agile. Almost everybody has worked in a company that’s called itself Agile in some way. And, whether they have or not, everybody has an opinion on what Agile is. And there’s a ton of misinformation about Agile, and a lot of bad Agile.

So Shane and I are rewriting the book for a modern audience: “Here’s what Agile really is, and here’s how to make it really work.” We’re keeping the pragmatic, down-to-earth, “here’s how to do it” approach, but we’re updating absolutely everything else.

(Scrolling through table of contents.) You can see the first part has been completely rewritten. The table of contents is brand new; we’re restructuring everything, moving it all around. There’s going to be more about how to customize Agile; more about the ideas behind Agile; and more about working with modern cloud-oriented development.

So in a moment... (pauses to read chat) GrumpyGameDev says, “Do you mean that simply labelling it Agile and still being waterfall isn’t actually Agile?” Yeah. (laughs) I would say so. In fact, we’ll get to that when I read the excerpt. Which I’ll get to in just a moment. I’m going to read the first chapter of the book, which is called “What is Agile?” But even if you think you know what Agile is, or even if you do know what Agile is, I think you might enjoy this chapter. I’m really proud of it.

Before we get going, I want to remind you all that this show—and in fact, the writing of the book—is made possible by the people who hire me for training and consulting. It turns out that writing a book—at least a technical book—is not a way to riches. It may be a way to fame... maybe... but it’s certainly not a way to riches. So, what allows me to do this—what allows me to do this show, this software development show every week, and what allows me to write this book—is the people who hire me for training and consulting.

People who have the capability for a lot of business agility, but don’t have the capacity, hire me for anything from software development training, like test-driven development training, all the way up to process design and how to get teams working together smoothly. Because when you have a large environment, teams often form bottlenecks. And that is a process problem, it’s a social problem, it’s also a technical problem. And I help out with these things.

Okay, let’s get to the book. Again, to be notified when the Early Release is available, go ahead and sign up for the updates here. There’s an RSS feed; you can sign up for email updates; jamesshore.com/s/updates.

While I’m reading the excerpt, go ahead and put your comments in the chat. It’s going to take about 20 minutes to read, and then we’ll have the rest of the time—another 55 minutes or so from now—to discuss your comments and questions. So just put them all in and we’ll have a nice conversation after.

All right, let’s get to it. Part 1, Improving Agility, Chapter 1.

Chapter 1: What is Agile?

Agile is everywhere. And, paradoxically, nowhere.

In the 20 years after the Agile freight train roared into software developers’ conscious, the number of companies calling themselves “Agile” increased by orders of magnitude. The number of teams actually taking an agile approach to their work? Not so much. ”Agile,” the easily-repeated name, is enormously successful. The ideas behind Agile—well, most of them are ignored.

Let’s fix that.

Agile’s Genesis

In the 1990s, software development was believed to be in crisis. They actually called it that: “The Software Crisis.” Software projects were over-budget, late, didn’t meet requirements, and—according to the oft-quoted and ominously named “CHAOS Report”—a third of them failed outright.

Agile wasn’t a response to this crisis. Far from it. Agile was a response to the response.

To bring software development under control, big organizations created highly detailed processes that defined exactly how software was to be created. Everything was tightly controlled so that no mistakes could be made. (In theory, anyway.)

First, business analysts would interview stakeholders and document the system requirements. Next, software architects would read the requirements documents and create detailed design documents specifying every component of the system and how they related to each other. Then programmers would convert the design documents to code. In some organizations, this was considered low-skill work—just a mechanical translation exercise. In fact, programming was considered such a mechanical activity that, for a while, there was a movement called CASE—Computer Aided Software Engineering—to automatically convert architectural diagrams into code. Meanwhile, test leads would use the same documents to generate test plans, and when coding was done, armies of QA personnel would manually follow those test plans and report variances as defects. After each phase, everything would be carefully documented, reviewed, and signed off.

This approach was called “waterfall development” or “phase-gate development.” If it sounds like a ridiculous straw-man, well, consider yourself fortunate. Not every company used a heavyweight process in the ’90s, but it was widely recognized as a logical and sensible way to work. Of course you needed to define requirements, then design, then implement, then test. Of course you needed to document every phase. This was discipline. This was engineering. How else could you possibly succeed?

Born Out of Crisis

Big companies defined their processes in excruciating detail. Roles, responsibilities, document templates, change control boards... every aspect of development was defined and controlled. If a project didn’t succeed—and according to the CHAOS Report, two-thirds of them didn’t—it was because the process needed more detail, more documents, more sign-offs. It resulted in a massive amount of documentation. Martin Fowler called it “The Almighty Thud.”

This wasn’t a great way to work. It was bureaucratic and dehumanizing. Skill didn’t seem to matter as much as adherence to process. Programmers felt they were interchangeable cogs in an impersonal machine. It didn’t even work all that well.

So several people independently developed simpler, slimmer, and less prescriptive methods for developing software. They were called ”lightweight methods” in contrast to the heavyweight methods used by big companies. These new methods had names like “Adaptive Software Development,” “Crystal,” “Feature-Driven Development,” “Dynamic Systems Development Method,” “Extreme Programming,” and “Scrum.”

By the late ’90s, these methods were attracting serious attention. Extreme Programming, in particular, saw an explosion of grass-roots interest among programmers. In 2001, 17 of the lightweight methodology proponents met at a ski resort in Utah to discuss unifying their efforts.

The Agile Manifesto

“I personally didn’t expect this particular group of people to ever agree on anything substantive,” Alistair Cockburn said later.

And, in fact, after two days, they only accomplished two things: the name “Agile,” and a statement of four values. Over the following months, over email, they hashed out twelve accompanying principles.

This was the Agile Manifesto. It changed the world. So, as Alistair said later, they did agree on something substantive after all.

But there was no unified Agile method. There never has been, and never will be. Agile is three things: the name, the values, and the principles. That’s it. It’s not something you can do. It’s a philosophy. A way of thinking about software development. You can’t “use” Agile or “do” Agile... you can only be Agile. Or not. If your teams embody the Agile philosophy, then they’re Agile. If they don’t, they’re not.

The Essence of Agile

Martin Fowler has made a career out of turning complicated software topics into well-considered, even-handed explanations. His explanation of “The Essence of Agile Software Development” is one of the best:

Agile Development is adaptive rather than predictive; people-oriented rather than process-oriented.

Martin Fowler

Adaptive rather than predictive

Remember the CHAOS Report, which said that only one-third of software projects were successful? It had a very specific definition of success:

Successful
“Completed on time, on budget, with all features and functions as originally specified.”
Challenged
“Completed and operational but over budget, over the time estimate, [with] fewer features and functions than originally specified.”
Impaired
“Cancelled at some point during the development cycle.”

These definitions illustrate the predictive mindset perfectly. They’re all about conformance to plan. If you did what you said you were going to do, you were successful. If you didn’t, you weren’t! Easy.

It makes sense at first. But look closer. There’s something missing. As Ryan Nelson commented in CIO Magazine:

Projects that were found to meet all of the traditional criteria for success—time, budget, and specifications—may still be failures in the end because they fail to appeal to the intended users or because they ultimately fail to add much value to the business.

Similarly, projects considered failures according to traditional IT metrics may wind up being successes because despite cost, time or specification problems, the system is loved by its target audience or provides unexpected value. For example, at a financial services company, a new system... was six months late and cost more than twice the original estimate (final cost was $5.7 million). But the project ultimately created a more adaptive organization (after 13 months) and was judged to be a great success—the company had a $33 million reduction in write-off accounts, and the reduced time-to-value and increased capacity resulted in a 50 percent increase in the number of concurrent collection strategy tests in production.

Agile teams define success as delivering value, not conforming to a plan. In fact, truly Agile teams actively look for opportunities to increase value by changing their plans.

Take a second look at the Manifesto (see figure 1 and figure 2). “Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.” How many of its values and principles relate to delivering valuable software and adapting to feedback?

(To camera) You can see the Agile Manifesto at agilemanifesto.org.

People-oriented rather than process-oriented

Heavyweight processes tried to prevent errors by carefully defining every aspect of software development. By putting the “smarts” in the process, individual skill became less important. In theory, you could apply the same process over and over, with different people, and get the same results. (Come to think of it, they kind of did. Just not the results they wanted.)

Agile says people are the most important factor in software development success. Not just their skills, but all aspects of their humanity. How well team members work together. How many distractions they encounter. How safe they feel. Whether they’re comfortable voicing dissent, and whether they feel motivated by their work.

Agile teams have a process—every team does, even if it’s implicit—but the process is in service of the humans, not the other way around. And Agile teams are in charge of their own process. When they think of a better way of working, they change it.

Look back at the Manifesto (again, agilemanifesto.org). “Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.” Which values and principles relate to putting people first?

Why Agile Won

In the first ten years after the Manifesto, Agile faced enormous criticism. It was “undisciplined,” critics said. “It could never work.” Another ten years after that, the critics were silent. Agile was everywhere, at least in name. Heavyweight waterfall methods were effectively dead. Some younger programmers had trouble believing anybody ever could have worked that way.

There’s actually nothing inherently wrong with the basic waterfall process. If you keep it slim and operate in a well-understood domain, waterfall can work well. The problem was the heavyweight processes big companies used. Ironically, the processes designed to prevent problems actually caused many of the problems organizations were seeing.

With software, it’s very difficult to imagine how it will work before you actually use it, and it’s even harder to think of absolutely everything your software needs to do. This is doubly true for people who aren’t actively involved with software development. As a result, it’s critically important to get working software in front of people as soon as possible. You need to get feedback about what’s missing or wrong, then change your plans based on what you learn. As the Manifesto says, “working software is the primary measure of progress.” Learning and responding to change are at the heart of what Agile is all about.

Those heavyweight processes put so much emphasis on process controls and documentation and sign-offs that they incurred a huge amount of delay and overhead. They took years to produce working software, and they had nothing concrete to show until near the end. Instead of welcoming change, they actively worked to prevent change. They actually had a dedicated part of the process, the Change Control Board, whose primary purpose was to say “no” to change requests. (Or, more accurately, “yes, but it will cost you.”)

All of this added up to projects that spent years in development before they had anything to show. When they did, it was too late and too expensive to make changes. They ultimately shipped software that didn’t do what customers needed.

Although there are a variety of approaches to Agile—and some of them are more about co-opting a popular name than following the actual philosophy—one thing they all have in common is a focus on making progress visible and allowing stakeholders to make course corrections as they go. This seems like a small thing, but it’s incredibly powerful. It’s why we no longer hear about a Software Crisis. Software is still late. It’s still over budget. But Agile teams don’t spend years building failures. And that’s huge.

There’s more to Agile than just providing visibility. But this one thing? This was enough. It’s why everybody wanted Agile.

Cargo Cult Agile

At first, Agile was a grassroots movement. It was largely driven by programmers seeking better results and better quality of life. As it became more successful, Agile’s momentum shifted from the underlying ideas to hype. Rather than saying, “let’s get better results by adapting our plans and putting people first,” organization leaders started saying, “Everybody’s talking about Agile. Get me some Agile.”

The thing is, there is no “Agile” to go get. It’s just a set of values and principles. There are specific Agile approaches, such as Extreme Programming and Scrum, that will tell you how to be Agile, but you still have to be on board with the underlying philosophy.

And for a lot of organizations, that underlying philosophy—adapting plans and putting people first—is really, really foreign.

The tragedy of the cargo cult is its adherence to the superficial, outward signs of some idea combined with ignorance of how that idea actually works. In the story, the islanders replicated all the elements of cargo drops—the airstrip, the tower, the headphones—but didn’t understand the vast infrastructure that enabled airplanes to arrive.

The same tragedy occurs with Agile. People want Agile’s Cargo: better results, more visibility, fewer business failures. But they don’t understand the underlying philosophy, and often wouldn’t agree with it even if they did. They want to buy Agile, but you can’t buy an idea.

What they can buy is the outward signs of Agile. Stand-up meetings! User stories! Tools! Consultants! There’s lots of stuff labelled Agile, and plenty of people eager to sell it to you. It’s often labelled as “enterprise-grade,” which is a way of saying “don’t worry, you won’t have to change.” Uncomfortable ideas like “adaptive planning” and “people-centric” are ignored.

And that’s how you get a cargo cult. All the activity; none of the results. The Agile part is missing.

“At my old company they wasted a huge number of man hours in meetings.”

“[Agile] cost an entire team (30+) people their jobs as they produced almost nothing for almost a year.”

“All [Agile] means is developers get shafted when the project changes... the day before delivery.”

“Its because of Agile there are sh—ty developers and sh—ty products being shipped.”

“I can’t take this Agile crap any more. It’s lunacy.”

comments about Agile from around the Web

The name Agile is everywhere. The ideas of Agile aren’t. It’s become self-perpetuating; for many, the only Agile they know is Cargo Cult Agile.

It’s time to fix that. In the rest of this book, I’ll show you how to apply Agile ideas for real. Keep an eye out for the Cargo Cult Agilists shown in the margin. They’ll show you what not to do.

Ready? Let’s go.

Discussion

So that is the first chapter of The Art of Agile Development. (Reads chat: “*claps*”) Thank you, UncleScientist.

So I think that hopefully gives you a bit of an idea of what we’re going for in this book: how the focus has changed from the first edition. The first edition, again, was really about exposing people to what Agile is and how to use it, and the second edition is for all those people who are already using Agile—or something they think is Agile—and talking to them about how they can make it better.

A lot of it is centered around the Agile Fluency® Model, which I developed with Diana Larsen. You can see that in the table of contents. Part 2 is “Focusing on Agility,” which corresponds to the Focusing zone of the Agile Fluency Model. Part 3 is about Delivering Reliably, which about the Delivering zone, more about technical practices. And Part 4 is about Optimizing Outcomes. That’s the Optimizing zone. It’s about how you can really be adaptive.

Now, we only have so many pages in the book. I’m trying to target about 400 pages total, and I’m already at risk of exceeding that. So the bulk of the book is going to be in parts two and three. We’re just going to have a little bit about adaptive planning and how you can take advantage of agility. Because in my experience, only about 5% of the teams I talk to are set up as an organization to even try for that zone.

(Reads chat: “That was a great intro. Hits so many good points.”) RegalRegex, thanks for the comment, I really appreciate it.

So, as I said, your comments and questions—now is a great time to have a discussion about what’s in the book, or just Agile in general. I’m going to go back and look through some of the comments that were made while I was reading. Go ahead and put additional comments in the chat and I will take them as they come. I’ve got about 35 minutes left scheduled for today’s session. We don’t need to use it all, but we have the time if we want it.

So let me scroll back and see what people were saying...

GrumpyGameDev says, “I’ve written four books and the fame and fortune got lost in the mail somewhere.” Yeah, I think it’s in the same dead letter box my fame and fortune is. (laughs) JitterTed says, “I think your expectations may be off.” JitterTed, I refuse to believe that. (laughs)

Tomowens says, “I don’t like the equality between waterfall and phase-gate.” That’s an interesting point, tomowens. In practice, what I have seen is that phase-gate is a way of rebranding waterfall for a lot of companies. Because “waterfall” is obviously bad and wrong now—everybody knows this—so we’re going to do the same thing and call it something else.

So if waterfall is requirements analysis, and then architecture, and design, and then coding, and then QA/testing... often with the validation V, where we do the validation in reverse... phase gate is basically the same thing, but we focus more on the sign offs between phases, and maybe we have less iteration. (laughs) I’m maybe a little cynical about these things. If you disagree with that, would love to hear your comments.

(Reads chat: “It’s only the intro, but I’m ready to buy.”) Tomowens, thanks for saying that. So, if you do want to buy the book, the Early Release is going to come out on O’Reilly in the next week or two. I will announce it right here, jamesshore.com/s/updates. That will give you announcements of all kinds of things that I’m doing, which is mostly focused on the book. Or you can just follow the #aoad2 hashtag on Twitter, which will be specifically for The Art of Agile Development.

Certification

Mrz_al_hun says, “Are you planning on mentioning certification mills?” Well, it’s funny you should mention that. Chapter 3, which is not done yet, so I can’t really share it... so we’ll just pretend I’m not sharing it here... (brings up IDE with chapter 3 text) You can’t see chapter 3, pretend there’s nothing to see here... Oh yeah, Invest in Learning. I’ll just read this real quick.

Agile practices take time to learn. Expect teams’ performance to dip by about 10-20% for about 2-3 months per zone that they’re learning. Less, with full-time coaching. Possibly more, if their learning is entirely self-directed. Delivering zone practices can take longer still, depending on the nature of the team’s codebase.

You can get faster results if you hire people to help you. There’s a wide variety of help available, ranging from occasional mentoring, to training, to help with process design and implementation, to full-time (or near-full-time) coaching. The most effective help you can get is to hire experienced people to join your teams full time and coach by contributing to your team’s real-world work.

Ignore the myriad Agile certifications. Although some are connected to excellent training courses, that’s entirely dependent on the trainer, not the certification. Don’t use Agile certifications for hiring, either; they have no teeth. Most demonstrate nothing more than the ability to connect butt to chair for a few days. Instead, do your due diligence: ask your network for recommendations, sample publicly-available materials, and ask for references.

That’s pretty much the only thing we have to say about certification.

Waterfall

GrumpyGameDev says, “I used to work at GE... they crossed off ‘waterfall’ on the label, and wrote ‘agile’ in crayon over it. (laughs) Really, it was ‘iterative waterfall.’” You know, iterative waterfall is a great method... well, it has potential to be a great method. The really big companies tend to put way too much process around it. But Spiral, which was basically iterative waterfall, or Evo, by Tom Gilb, which is not really iterative waterfall, but its own thing, and really a sort of proto-Agile method... they were all about learning and taking advantage of that learning as you went.

But iterative waterfall or Spiral is still not Agile. Agile is really characterized by its strong emphasis on adaptive planning and people-centricity. And iterative waterfall, although it is iterative, it’s not adaptive in the same way that Agile is, and it doesn’t have the people-centric part at all.

I haven’t studied Evo, Tom Gilb’s stuff, enough to say how it fits in, but I have actually been in correspondence with him in the last couple of weeks. He feels that his stuff is really very much in the Agile mindset and I see no reason to doubt him.

JitterTed says, “Nobody ever really did waterfall anyway, it was more phase-gate or just ad-hoc.” You know, waterfall is a sort of poorly-defined term. Some people say it’s Royce’s paper that originally defined waterfall, but as far as I know, he was just documenting what people were already doing.

JitterTed basically says what I just said, “Mainly because waterfall is not a defined process, very few people actually read the Royce paper where it originated.” I’m not sure that it originated there, JitterTed, and I don’t know if it was that influential at the time. Laurent Bossavit has a fantastic book—he published it on LeanPub—I strongly, strongly recommend it. It’s called Leprechauns of Software Engineering.

Leprechauns of Software Engineering is excellent. Laurent goes through and he looks at some of the commonly-cited papers that people constantly quote, and he goes back and finds the source material, and finds what they really said, and it’s often not what people say that it said. Anyway, he discusses the Royce paper. Really, really highly recommend it. You can find it on LeanPub.

The Cover and Lost Subtitle

Book cover for “The Art of Agile Development, Second Edition” by James Shore and Shane Warden. Published by O'Reilly. The cover has a large sticker on it that says “Early Release: Raw and Unedited.” The cover artwork shows a water glass containing a small sapling. The sapling has small green leaves. There is a goldfish in the glass.

Second Edition cover

Paulhenman asks, “Why is there a fish in it?” You know, that is a good question, and I thought this might come up. So I prepared a little bit to explain that.

(Shows picture of first edition cover, which shows a large water glass with a twig or sapling in it.) The first edition design was by Karen Montgomery, and I really love the design. It’s quite striking. People always ask us, “why (laughs) why do you have a water glass with a twig in it?” And the answer is maybe not that interesting, but I’m going to tell you anyway.

(Shows two cover images. One has jars of sand, the other is a plastic paint pallete with watercolors.) This is the first set of cover proposals we got from our editor, Mary Treseler, back in March 2007. The book was done at this point and it was just going through the final development process, which took about six months. And she said, “here’s four covers for you to consider. (Shows two more cover images. One has wine glasses overlaid with a mess of orange lines and the other has scraps of paper overlaid with circular bursts of lines and dots. Each cover has the title, “The Art of Agile Development: with Extreme Programming”)

Shane and I looked at these, and... we were not a fan. (laughs) Of any of these book covers. Now, The Art of Agile Development is part of O’Reilly’s “/Theory/In/Practice” line, and they all had sort of these real-world covers with random dots and lines on them.

So we went back to Mary and we said, “You know, we don’t really love any of these covers. Can we have some more options? And maybe something prettier?” (laughs) We sort of said, “this is what the book’s about, and this is what we’d like the cover to convey—adapting your plans, and so forth—and failing that, could it at least be pretty.”

(Shows two more cover images. One is the current book cover and the other is similar, but with a squiggly line on top.) And she came back to us with a second round of covers, and the cover we have now was actually the first option, and then there was one with a random squiggle and a dandelion head on it.

(Shows two more images.) And then we had this one, which I actually rather like, with a growing something—sprout? And then a version of that sprout with a random squiggle with a dandelion head. (Two more images) The water one with little wavy lines—I think that one’s really pretty, actually. Stairways, with a step. We definitely did not want that, because that would sort of imply, you know, the waterfall, except in reverse. (Two more images.) A random ball of rubber bands, and another random ball of almost rubber bands.

JitterTed asks, “‘With Extreme Programming’ was originally the subtitle?” Yeah, we actually had “with Extreme Programming” in the subtitle right up towards the end. But this was in 2007. Extreme Programming had basically fallen off the radar. And so, even though the book chose Extreme Programming as its method, we ultimately decided to take that off the subtitle because we decided we wanted this to be more a general-purpose Agile book that used Extreme Programming rather than being an Extreme Programming-specific book.

The actual original book—the very, very first intention—was to take chromatic (Shane Warden)’s book, which was the Extreme Programming Pocket Guide—the original intention was to be a little second edition to the Extreme Programming Pocket Guide. But O’Reilly, by then, had stopped doing Pocket Guides because they got lost on the shelf and didn’t sell well. And, as we got into it, I realized that I had a whole heck of a lot to say about Agile.

So the little tiny pocket guide that we had decided to do had turned into this massive book, which was still a lot about Extreme Programming, but was also about wider Agile development as well. So we ultimately took the subtitle off.

(Laughs) GrumpyGameDev says, ”Is there a ‘cover must be uninspiring’ mandate from the publisher?”

So these were the options that we got. (Shows one final image) This was the last one. It was a lamp with sort of wavy lines coming out of it. The design department said, “I think the designs speak for themselves, so I won’t bore you with ‘designer rationale’ unless you really want it.” So why did they choose this cover and these options? I don’t know.

What we came back and said, though, was, “We think #5 is prettiest [which is the waves], but felt that #1 would draw more attention on the shelf and conveys the spirit of the book slightly better.”

So at this point, this is where all your hopes and dreams about there being some deeper meaning behind the cover? This is where I as an author completely smash those hopes and dreams (laughs). But if you really want a reason for the cover—and the reason is, this is what we were given, and we chose one—but if you really want a reason: we did feel like the spirit of the book was adaptive planning. The sapling grows towards the light. So that is why we chose this cover.

(Laughs) CharliePancakes says, “This is like the ending to Lost all over again.” Yeah, I announced last week that I was doing the livestream today and Pelyt asked, “Why do you have a glass with a twig on the cover?” I said, “Well, I’ll explain it next week, but (laughs) you might be disappointed.”

So, for the second edition, the nice thing is, it’s 13 years later, and O’Reilly has a much different process. It’s much more, well, Agile. We write source code in AsciiDoc, and there’s the Early Release so you can actually see the book as it comes out. So the cover comes a lot sooner in the process. For the first edition, the cover came at the end; for this one, it’s coming pretty close to the beginning.

So my brief was, “I’d like to use the same cover, it’s really iconic, people recognize it, but I’d like there to be some growth, to sort of symbolize the second edition.” And the graphic designer, who I don’t have the name of, but I imagine will be credited in the final book, they did this beautiful job where they added a little bit of growth to the cover. A little green. And they added a fish, the goldfish, which... there is no particular meaning, but I just love the splash of color. I love the whimsy of it. And I think it’s fantastic.

(laughs) So that is the, perhaps disappointing, reason and explanation behind the cover.

JitterTed says, “I always liked the organic feel to Agile, that it grows towards something, rather than as a planned roadmap.” And yeah, that is why we chose that one over the waves. Even though we really liked the waves and the water—we thought it was super pretty—we did like that the twig could be seen as growing toward the light. So if you need a reason, that’s the reason.

Simongeering: “When did O’Reilly stop doing animals on covers?” They still do animals on covers, but there’s different lines. So the animal books are the programming reference books. And I think those are a lot less relevant today? People are getting their programming reference online, rather than getting printed books. And—I have no inside information—but I think they’re running out of animals (laughs).

It started out, they were using these Dover woodcuts from the 1800’s, I think because it was royalty-free. O’Reilly started out as sort of this scrappy upstart publisher. And you’ll notice that they’ve moved on to insects for a lot of their cover designs and I think it’s because they’ve run out of mammals. Don’t take my word for it. I don’t know.

Anyway, so this line—“/Theory/In/Practice”—and I think other lines, like the Head-First Series—it’s all these sort of fish-eye lens, looking down at people’s heads, with giant heads—I think they’ve come up with different styles for different lines. And this is the /Theory/In/Practice line, which is all real images with some sort of random lines on the cover.

GrumpyGameDev: “Upgrade: now with goldfish!” Yeah! It’s a brand-new book! Buy it! It’s got a goldfish. That’s all I got to say.

Providing Feedback

Okay, I’m going to go back up... again, keep putting your comments in. I’m loving the conversation. We’ve got about 15 minutes left before I’m going to call it a day.

So, let’s see, what do we have next up here...

Jwalter: “Is it a bad idea to report spelling errors here?” It’s not a bad idea, but I’m going to ignore them, because (laughs)—I guess I could actually go in and modify the source code. Sure! Why not! Go ahead and report spelling errors. I will update the actual source book in real time. We are going to have a proper editing process, and so that’s when the real errors will be fixed, but spelling errors are welcome as well.

The main thing I’m looking for from the Early Release process, though, and the open development, is feedback about the ideas and concepts. Does it ring true? Do you have additional material that you think should be included or that you think’s important? Is there something that we’ve said that you don’t agree with? Those are the kinds of things that we’re really looking for in the open development process.

And again, to be informed when we put up new chapters for your review, right there: jamesshore.com/s/updates, or check out the #aoad2 hashtag on Twitter.

Commitments & Deadlines

CharliePancakes says, “I think the biggest problem I see is that I’ve faced is that my team can be agile but the upper management still need estimates and budgets and milestones etc.” Yeah, that is so true. And in fact... man, I probably shouldn’t read this to you... but I’m going to anyway.

Chapter 3, which I’m still working on, pretty happy with, it’s talking about the investments that you need to make to be successful with Agile. The second part of this, “2. Get Stakeholder Buy-In”—it’s way too long, I’m going to have to cut it down—but I’m going to read a little bit of this.

Software teams have always had a contentious relationship with their stakeholders on the matter of deadlines. Sales and product management, in particular, want accurate forecasting (often called “estimating”). It’s understandable! Their jobs would be much easier if they could tell customers exactly what they’re going to get and when. It’s important for building trust, securing new customers, and conducting business planning.

No matter how much they want it, they usually don’t get it. This is a sore spot. But accurate forecasting requires things most companies can’t provide.

  1. Detailed plans that are fixed in advance. (This often results in shipping the wrong thing, as we saw in the first chapter.)

  2. Statistical modeling and probabilistic date ranges. (Stakeholders usually want simple answers.)

  3. No interference in teams’ forecasts. (Accurate forecasts are usually much bigger than stakeholders want, leading to them being “negotiated down” to politically-acceptable fantasies.)

Instead, companies put in increasingly elaborate processes to try to “control” their schedules. It takes a lot of time away from development and it usually doesn’t work. In the end, it all boils down to gut feel, lots of padding, and political maneuvering.

Agile don’t play that.

Agile gives you visibility and flexibility. It tells you what is, not what’s politically expedient. It shines the harsh light of reality on people’s hopes and dreams. In exchange, it lets you steer your plans so you can get the best results possible with the time and people you have.

That doesn’t mean you won’t have roadmaps! You will. See [that section] for details. But it does mean that your roadmaps aren’t iron-clad commitments. They’re guides, not straight-jackets.

Some stakeholders love it. Finally, they know what’s really going on, and they have the ability to influence the results. Others, especially those who have been disappointed by software developers time after time, see Agile as just another political maneuver: a complicated way to avoid giving them what they need. They fight Agile tooth and nail.

You don’t need every stakeholder to buy in to the Agile way of planning. You do need the ones with a lot of political influence to do so. If they don’t, they could quietly sabotage your efforts, pulling the rug out from under you in six months to a year, even if—or especially if—Agile is succeeding.

Then I talk about how to deal with that. One thing I really like about this chapter is that is says, “this is what you need,” and then it says, over and over again, “well, if you can’t get that...”

If concrete commitments are required...

Agile teams can work to a deadline. They can even produce a detailed plan in advance. After development begins, their forecasts are typically more accurate than non-Agile teams’ forecasts, and they get better over time.

What Agile teams can’t do is guarantee a fixed scope (set of features) by a fixed date. Almost no one can. If you need certainty, you can plan on a fixed scope and ship whenever it’s ready; or you can plan on a fixed date and ship whatever’s done on that date. Not both.

That said, if your company requires fixed scope, fixed date releases, you can keep using whatever forecasting technique you’re using today. Agile won’t make it worse, and as development proceeds, it will give you visibility into how things are really working out. Predictive plans aren’t the Agile way, but there are more benefits to Agile than adaptive planning, and you can work your way up to true agility over time.

Do be careful. Some people like finding scapegoats, and some companies have a culture of assigning blame. Even if Agile doesn’t make anything worse, it might be convenient to blame Agile for the same deadlines that would have been missed before Agile. If that’s the case at your company, introducing Agile might be a mistake.

That’s what we have to say about “your team can be Agile, but upper management still need estimates and budgets and milestones.” You can have estimates and budgets and milestones, etc., and you can even have that in an Agile way, but it’s not going to be set in stone. If you want, you can do that, but it’s not super Agile. But you can be as successful with Agile at that as with anything else, which is usually... a lot of padding.

GrumpyGameDev says, ”It’s a challenge to be agile when you have to answer to shareholders on a quarterly basis.” It’s really not. Shareholders do not need that level of detail. You can tell them, “this is the direction we’re going,” without saying, “this is exactly what we’re doing and when we’re going to be done.”

JitterTed says, “That often results in AgileFall, the team is agile, but the environment is waterfall.” Yeah. And, again, you can do that.

Tomowens says, ”There’s something inherently wrong with the basic waterfall process. I can’t think of a software development effort that is in a well-understood domain. Some are more well-understood than others, but I would say that iterative and incremental development is almost always the right solution.” Yeah, I would agree with that, Tom.

That said, I do think there are some organizations that are—you see this, for example, in web agencies, digital agencies, that are doing a lot of marketing websites. The software behind it is pretty well established. There can be a little bit of custom development, but they’re basically doing the same thing over again, with different creative details, but not a lot of development challenge. There’s also companies that basically sell white-label software. They’ve got their software that they sell to big companies, and the big companies customize it. So they’re doing kind of the same thing over and over again.

(Scans for more comments) GrumpyGameDev: “#YouHadMeAtGoldfish.” (laughs) UncleScientist: “My only snobby comment is to not cite Wikipedia.” (jokingly) What? What’s wrong with citing Wikipedia? Don’t worry, we’re going to update it.

Zispro asks, “Will #noestimates be in the book?” Yes, yes it will. Right here (shows table of contents) in the Accountability chapter. It’s currently pencilled in. Whether or not it will stay as its own standalone part, or it will have something else, it will be in there. Part of the reason for that is that I believe Focusing teams are typically incapable of providing useful forecasts. That’s why we put the Forecasting section down in the Delivering zone part.

(laughs) JitterTed says, “Execs don’t like reality, from my experience.” You know... I’m of two minds on that. The work I do tends to be with senior software development leaders. They’re the ones that hire me. And I would say that the execs that hire me—which, as I’ve said in the past, are all above-average, highly intelligent, more-attractive-than-usual people (laughs)—they do like reality. In fact, one of the things they want, is they want to know reality soon, because they hate getting blindsided by mistakes or by problems. What they want to do is be able to understand it, they want to be able to respond to it.

But they’re often in a larger ecosystem where they’re working with a marketing department. They’re working with a sales department. They’re working with a product department. Often, I actually am hired by product leaders as well as engineering leaders, but sometimes when I’m hired by engineering leaders, the product people aren’t on board.

And that’s where we get back to that thing I mentioned in the book, in chapter 3, where some people see Agile and its reality as actually a way of telling them that they can’t have what they want. Salespeople particularly feel this. Because Sales is a completely different world from software development. Salespeople often see developers as kind of lazy. They’re slackers, is how they see it. And that’s because software developers tend to come in late. They might work late, but they tend to come in late. They tend to jump around a lot. Sales tends to be really, really concrete.

So there’s often a lot of friction between Engineering, and Sales, and sometimes that bleeds over to Marketing and Product. And I wouldn’t say that there’s a dislike of reality. It’s a dislike of everything that’s behind the unpleasant reality that we have to share. That’s a real challenge, and that’s part of the challenge of bringing in adaptive planning and real Agile.

Changes in Second Edition

Toltarius says, “Sorry if I’m late in asking this, but what are a summary of changes made over the first edition?” Thanks for asking, Toltarius. The second edition is a complete restructuring of the first edition. If you look at the table of contents, you’ll see that it’s completely different. And, if you want, the the first edition table of contents is also on my site, so you can compare the two if you want.

It’s been completely restructured. It’s following the Agile Fluency Model that I created with Diana Larsen. And it’s also targeting a different audience. The first edition was for that early majority of people who weren’t working in the innovator, early adopter companies, but who wanted to really bring in Agile. So, what is Agile, how does it work, it was a really practical, pragmatic way of doing Agile.

The second edition, also, really practical and pragmatic, but now we’re targeting sort of the late majority. Everybody’s heard of Agile at this point, they think they know what it is, now we’re going to talk to you about how to really do it, and how to really do it well.

So, that’s what it’s about.

Cargo Cult Agilists

Let’s see what else we have in the chat. Little bit about Royce’s paper, which I think we’ve talked about...

GrumpyGameDev: “Oooooh, Cargo Cult Agile, that’s the one we use at work!” Yeah, you and everybody else.

Oh, I need to show you all, just because I can: I don’t know if this will be what’s used in the final book, but I have made a little icon. (Shows icon of a person in a suit wearing coconut headphones.) Throughout the book, we’re going to occasionally put this Cargo Cult Agilist with a section on what not to do. And this is the icon that I’ve proposed that we use. Their graphic designers might actually produce something nicer.

Let’s see if I have an example of one of those Cargo Cult Agilist sections...

So that’s the Cargo Cult Agilist that’s going to show up throughout the book.

And... I tend to be a little cynical when I first write, so some of the edge may be taken off these... but maybe not. We’ll see.

C-Levels

We are almost out of time, but let’s see if there’s any other comments in here.

(Simongeering:) “Now all you need is global legislation to make every CTO and CEO required to read it, getting the ideas in the brains of the folks with the power to change an organizational culture is the problem. As with all hard things it is a people problem.”

You know, when I get hired, a lot of it is because somebody read my book, often a team lead or architect or something like that, and they bring me in and we go talk to the leadership, and then the leadership hires me. Because it’s easier to get them to hire me (laughs)—and I’m not that cheap—it’s easier to get them to hire me than to read a book.

(more seriously) And also, all cynicism aside, they’re more interested in getting the ideas throughout the company and hiring me will help them do that. And then, as part of my work, it’s bringing them on board and helping them understand their role in how to make this possible and sustaining it. Which, honestly, is not that hard. It’s partly because I come in from the outside and outsiders are always, sort of, more listened to than people who are inside the company. Familiarity breeds contempt. But also, it’s a matter of knowing how to talk to folks. How to put yourself in their shoes. And also focusing on what they care about.

It’s a really interesting topic, and I would love to talk about it, but we’re running out of time.

(reading more of the chat) A lot of cynical comments about C-levels, which, sorry, I can’t really agree with.

(NazarChops:) “Is a shift to Agile possible from the bottom up, or does it require Cs and VPs to drive the change?” Best is from both ends. You want support from the C’s and VP’s and you want bottom-up. Unfortunately, we’re out of time, but I have a whole section on that in Chapter 3. It will have to come some other time.

Final Thoughts

I think that’s most of the comments and it’s also 1:00 my time, so I think this is a good place to stop. Thank you all so much for listening. I really appreciate the interest. Had a good turn out today, so I’m just thrilled that there’s so much interest in the new book.

Again, if you’d like to be informed when the Early Release comes out, which should be in the next week or two—it is going through O’Reilly’s process right now—subscribe to updates. It’s free of course, and you can subscribe by RSS or email. jamesshore.com/s/updates, or you can follow my Twitter, @jamesshore. #aoad2 is the hashtag.

Next week, we’re going back to our normal software engineering scheduling. That’s going to be noon Pacific, August 25th, and we’re going to be talking about building a microservice client using Nullable Infrastructure.

Thanks so much, again, if you want to see more about the book, check out jamesshore.com/s/aoad2, that will take you to the table of contents and more information.

Thanks again! I will see you all next time.

If you liked this entry, check out my best writing and presentations, and consider subscribing to updates by email or RSS.