AoAD2 Chapter 3: How to Be Agile

Book cover for “The Art of Agile Development, Second Edition.”

Second Edition cover

This is a pre-release excerpt of The Art of Agile Development, Second Edition, to be published by O’Reilly in 2021. Visit the Second Edition home page for information about the open development process, additional excerpts, and more.

Your feedback is appreciated! To share your thoughts, join the AoAD2 open review mailing list.

This excerpt is copyright 2007, 2020 by James Shore and Shane Warden. Do not distribute or republish without James Shore’s express written permission.

How to Be Agile

What does it mean to “be Agile?”

As we’ve seen in the previous chapters, Agile is a philosophy: a way of thinking about software development. To be Agile, you need to put the Agile philosophy into practice. Paying lip service to Agile ideas doesn’t get you anything. It’s action that matters.

Practicing Agile

Every team has a way of working—a process, or method—that they follow, even if it isn’t formally written down. It’s rarely articulated, and isn’t necessarily self-consistent, but that method reflects an underlying development philosophy.

For a team to be Agile, they need to change their method to reflect the Agile philosophy. This is both easier and harder than it sounds. It’s easy because, in most cases, they can start with one of the many off-the-shelf Agile methods, like the one in this book. It’s hard because they need to change their way of working, and that involves changing a lot of habits.

Methods consist of individual elements the Agile community calls practices. These are things like using version control, having automated builds, and giving demos to stakeholders. Most practices have been around for decades. Agile methods combine them in unique ways, accentuating those parts that support the Agile philosophy, discarding the rest, and mixing in a few new ideas. The result is a lean, powerful, self-reinforcing package.

It’s tempting to choose practices that are similar to what you’re already doing and ignore practices that are different. Avoid this temptation! The practices that are the least familiar are often the ones that involve the biggest change in philosophy. They’re the ones that your teams need the most, if they’re really going to be Agile.

In addition, Agile practices often perform double- and triple-duty, solving multiple software development problems simultaneously and supporting each other in clever and surprising ways. You won’t truly understand how it all works until you’ve seen it in action for a while.

So, although it’s tempting to customize your Agile method from the beginning, it’s best to start out with a by-the-book approach. Put it into practice—the whole thing—and spend several months refining your usage and understanding why it works. Then customize. Choose one of the rough edges, make an educated guess about how to improve, observe what happens, and repeat.

The Road to Success

The majority of this book—parts two through four—is dedicated to a curated set of Agile practices that have been proven in practice.1 To succeed with Agile, follow these steps:

1The method in this book is primarily based on Extreme Programming, but it also draws inspiration from Scrum, Kanban, Lean Software Development, the DevOps movement, and Lean Startup.

  1. Choose the zones that your organization both needs and is willing to pay for, as described in the “Choose Your Zones” section at the end of the previous chapter.

  2. Figure out which investments your organization needs to make (see the “Investing in Agility” chapter), and get your managers, teams, and stakeholders on board with making those investments, as described below.

  3. Help your teams to apply all your chosen zones’ practices together, as much as they can. The practices are self-reinforcing, so they work best when you use them all together. Parts II-IV of this book describe the practices for each zone.

  4. Follow the practices rigorously and consistently. If a practice doesn’t work, try following this book’s approach more closely. Teams new to Agile often under-apply the practices. Depending on which zones you choose, expect each team to take up to four months for the practices to start feeling comfortable and another two to six months for them to become second nature.

  5. As your teams become confident they’re applying the practices correctly—again, give it several months—they can start experimenting with changes that aren’t by the book. Each practice in this book includes a discussion of why the practice works and how it can be changed. Each time a team makes a change, they should critique what happens and make further improvements. (The team retrospective is a good place for this—see the “Retrospectives” practice.) Each team will have its own challenges and should make its own changes. Every team will end up with its own custom process.

  6. There is no last step. Agile software development is an ongoing process of learning and improvement. Never stop practicing, experimenting, and evolving.

Kaizen and Kaikaku

Kaizen (rhymes with “I win”) is a common term in the Agile community. It’s a Japanese word meaning “improvement.” In the West, it’s come to mean a process of continuous gradual improvement: constantly reviewing and improving your practices at all levels of the organization.

Continuous improvement is an integral part of being Agile, so shouldn’t you kaizen your way to Agile in the first place? Counterintuively... probably not. Kaizen is for improving your existing ways of working. If you have a document-oriented culture, kaizen will help you streamline your documents. If you have a blame-oriented culture, it will help you place blame more accurately. But it won’t help you make the leap from either to Agile.

To jump from one way of working to another, you need a different Japanese word: kaikaku (rhymes with “I rock you“). Kaikaku means “radical change.” Instead of incrementally improving your existing way of working, as kaizen does, kaikaku fundamentally changes your underlying approach.

The great Agile teams I know all started with kaikaku. They figured out what they wanted from Agile, how they were going to invest in getting those results, and went all in. This can still be done incrementally—you can change one team at a time, or one division at a time—but for any given team, the change to Agile is all-in. It’s a disruptive change, but the disruption is relatively brief: less than half a year.2

2If you have many teams, the overall time frame will be longer.

I see many more companies with mediocre Agile teams than great Agile teams. One thing the mediocre teams have in common is that they didn’t go all in. They tried to kaizen their way to Agile. It seems to work at first, but invariably stalls out. People burn out on the mismatch between Agile ideas and company values. They get tired of absorbing big new ideas. Change fatigue sets in and progress stutters to a halt after several years of effort. Ironically, the disruption from approaching Agile this way lasts much longer than the disruption from kaikaku.

If your company is new to Agile ideas, regardless of whether they already use the name, use kaikaku. It’s faster and safer. If radical change is a scary proposition, reduce risk by starting with a pilot. Choose just one team, and maybe just the Focusing fluency zone. You can use that experience to guide bigger changes in the future.

Teams that are already Agile can kaizen their way to better results within their current fluency zones. See the “Improvement” chapter for details. To move to new zones—for example, if a team is fluent in the Focusing zone, and wants to add Delivering or Optimizing fluency—kaikaku is still the best approach. New zones require new investments and major changes, and that’s best done all at once.

Successful kaikaku requires discipline and care. And it starts with...

Get Management Buy-In

If you want to be Agile, you need management support. Without it, the mismatch between your teams’ Agile practices and your organization’s non-Agile culture will cause constant friction. If you’re a manager yourself, you still need your manager to be on board, and it’s best if your peers support you, too.

1. Start with a conversation

It begins with a conversation. This is often easiest one-on-one, and you’ll be most successful if your conversations are in-person, or at least over video. Start with an influential manager you trust and recruit them as an ally. They’ll help you understand who else you need to talk to and how best to approach them.

In your conversations, starting with that first manager, talk about the challenges your organization faces with software development. Based on the benefits described in the “Choose Your Agility” chapter, talk about how you think software development could be better in your company. Don’t monopolize the conversation, though; engage your audience. Briefly share the benefits of each zone and ask which zones they think are important. Ask them why. Spend more time listening than talking.

Above all, focus on what they can get and what inaction could cause them to lose rather than pushing Agile for the sake of Agile. In fact, given the extent of misunderstandings about what Agile is, you might be better off not even mentioning the word “Agile.”

2. Get the economic buyer’s approval

Your ultimate goal is to speak to the person who has the authority to make the investments your teams need, or to have a trusted manager speak on your behalf. In sales, this person is called the “economic buyer.”

Economic buyers are often surrounded by gatekeepers who see their job as protecting the economic buyer. They’ll ask you to make your pitch to them, so they can present it to the buyer. They’re not trying to steal your idea; they’re trying to save the buyer time. Sometimes they’ll assure you that they’re the real buyer, even if they don’t actually have the authority needed.

Don’t be fooled. Although it’s helpful to get gatekeepers on your side, and often necessary, it’s not enough. Gatekeepers can’t approve the investments you need. You need to talk to the real economic buyer.

Prior to talking to the economic buyer, focus your conversations on the benefits of Agile: what’s at stake. The investments are likely to be a distraction, or even a source of concern, because the people you’re talking to won’t have the authority to make those investments.

When you finally talk to the economic buyer, your goal is to get them to agree in principle to investing in Agile. You probably won’t have a lot of time, so stay focused on the big picture. It’s also often best if you approach your meeting as a conversation, rather than a presentation with slides, but your allies will know the best approach for your specific situation.

In your conversation with the economic buyer, talk to them about what they want from their organization and how Agile will help. This works even better if a manager they trust speaks informally on your behalf.

Once the buyer is on board with Agile’s benefits, talk about the investments. Summarize the investments needed for each zone (see the “Summary of Investments” sidebar), along with how those zones map to what they want, and ask the economic buyer which of those investment/benefit tradeoffs seems most appropriate. Giving them a choice of several options, rather asking for a yes/no decision, reduces the chance that they’ll reject you outright.

Assuming the economic buyer agrees to investing in Agile in principle, or at least is worth considering further, ask for permission to create a concrete proposal. Ask what they need to see in the proposal, in general, for them to approve it. Ask for them to recommend a sponsor for you to work with, provide a date when you’ll have the proposal to them—within a day or two is best, so it’s a good idea to have a rough draft done already—and ask when you should expect to hear back.

Finally, ask for permission to follow up with them. They probably won’t get back to you when they said they would, so it’s good to be able to say, “I’m following up as you requested.”

3. Make a formal proposal

If you got this far, congratulations! You’ve passed your most important hurdle. Now you need to follow through with a proposal.

The formality of your proposal will depend on your organization. Your sponsor and other allies will help you understand how to format your proposal, help you refine it, and promote it to the economic buyer. Be prompt, polite, and persistent.

In your proposal, describe the benefits your that organization can expect to see and the investments it needs to make. Be concrete. The “Choose Your Agility” chapter describes Agile’s benefits in general; translate those benefits to your actual situation, the investments the economic buyer is willing to make, and what that realistically means for your organization.

For the investments part of your proposal, read the “Investing in Agility” chapter and translate each step into concrete requests. You may have to compromise on some investments. That chapter explains how. But avoid compromising too much. The investments are ultimately what make Agile’s benefits possible.

If this sounds like too much work...

This careful buy-in process is for when support is uncertain: when you’re working with multiple teams, asking for big investments, or working in a bureaucratic organization that’s uncomfortable about Agile ideas (even if they use the name a lot).

But sometimes, that’s not the case. Sometimes you’re just helping one small team become more Agile. If you and your manager already have the power to make the investments you need, do it!

If management thinks they’re already Agile...

Some organizations—these days, a lot of organizations—think they’re already Agile. One company told me, “We’re post-Agile!” Or you might hear, “we’re little-a agile, not big-A Agile!” But when you compare the philosophy in the “What Is Agile” chapter to how the organization acts, and the benefits in the “Choose Your Agility” chapter to what they get, you see that they’re nowhere close.

There’s nothing to be gained from arguing about terminology. If they want to say they’re agile, or post-agile, or extra-super-duper agile, let them. Instead, stay focused on the situation at hand: the challenges your teams are facing. The benefits the organization could get. The investments needed to get those benefits.

If management isn’t supportive...

If you can’t get traction with managers at first, don’t give up. Put yourself in their shoes. What does organizational success mean to them? What about personal success? How can Agile help them get what they want? If the answer is “it can’t,” then maybe Agile isn’t a good fit. Choose another approach to software development, one that properly fits your organization’s culture. Waterfall is a fine choice! Keep your documentation lightweight, your approach flexible, and your delivery horizons down to 3-6 months—shorter is better—and you’ll avoid its most common failure modes.

If you have a trusted manager you can turn to, ask for help and advice. If not, try conducting an informational interview with a manager at a company who’s been through it before. (They may try to recruit you. Win/win!) The “Change Your Organization” sidebar has more ideas.

In the early days of Agile, when it was a grass-roots movement, many teams adopted Extreme Programming (a popular Agile method) on their own, with little to no permission or support. You could try that. But I don’t recommend it. In the experience reports from teams who tried it, somebody—often the project manager—ended up having to bridge the gap between company culture and Agile philosophy. It was a thankless job and burned them out.

Some people use the Kanban method to motivate organizational change.3 Kanban wraps around existing ways of working to highlight work-in-process bottlenecks and the cost of delay. It’s easy to put in place and can motivate a more Agile approach to your work.

3Note that the Kanban method is much more than just the Kanban board some teams use for planning.

Kanban is a kaizen approach to change, so it’s slow and can only go so far, but it’s very effective and can lead to permission for kaikaku. See XXX for more information.

If nothing you do makes a difference, take a hard look at what you need for personal success. Assume the status quo won’t change, because it probably won’t. Either that’s good enough for you—and it often is—or it’s time to move to a company that better fits your needs.

Get Team Buy-In

Agile puts people first, so it shouldn’t be a surprise that you need your prospective Agile teams to agree to it as well. It is possible to force people to nod agreement with gritted teeth, but—and I speak from hard-won experience, here—that road involves a lot of employee turnover. Or fails, with a passive-aggressive whimper.

When I’m asked to help teams become Agile, I always speak to each team on their own, without managers present. (I used to include managers. No longer. That’s the “hard-won experience” part.) In your case, if you’re in a position of authority, have the teams’ coaches speak to them without you.

First, though, you’ll need to choose your prospective teams. Review the next chapter’s material about choosing teams and selecting coaches. Select a few extra candidates, just in case, because the secret to team buy-in is giving them the opportunity to refuse.

When you or the coach talk to each team, explain that their team has been chosen as a possible candidate to try Agile. I explain why their managers are interested in Agile, what benefits it will bring to the organization, and how it will affect them personally. I also explain that changing work habits can be stressful, and that they should expect a period of chaos—typically, up to three months—as everyone figures out how to make Agile work for them.

The Satir Change Model is one way of explaining how teams react to change. Steven Smith has a good article on the Satir model at https://stevenmsmith.com/ar-satir-change-model/ that includes tips for helping team members through each stage.

“If you agree,” I tell them, “I’ll ask you to follow a by-the-book approach to Agile for three months. At that point, we’ll evaluate what’s working, what’s not, and make improvements.4 After six months, you’ll have the final say on whether you continue with Agile or go back to what you have now.”

4This is a little white lie. Agile involves constant improvement, so we evaluate what’s working and what’s not within a few weeks. But we do pause for a bigger evaluation at the three month mark.

Then I open the floor for questions. Teams typically a lot of questions about the process, but invariably, one of the questions is, “What happens if we say no?” My answer is always the same: “Then it doesn’t happen.” This is important! The veto must be real. If you don’t give people the option of saying, “no,” their “yes” doesn’t mean anything. By giving people a real chance to refuse now, and a concrete opportunity to change their minds later, you make it safe to try something new.

Make sure you allow plenty of time to answer everybody’s questions. The meeting usually takes about an hour, but sometimes it goes longer. After everyone’s questions have been addressed, ask for a vote. Remind everyone that there’s no consequence for voting against Agile. Mean it. Then ask the team to vote.

If team members are skeptical...

Skepticism is normal, and you should expect it. Be straight with your teams: change is disruptive, but the results are rewarding. Be clear about the practices that you think people might find frustrating or unusual at first, such as pair programming. That will help disarm skepticism and also make it easier to introduce those practices in the future.

It helps to emphasize that this is an experiment, and that the team has the final say over whether they stick with Agile or not. I often ask the team to schedule the three-month and six-month evaluation meetings on the spot. That makes them real.

If a few team members refuse...

If a few people disagree, ask them to explain why, and see if their objections can be resolved. If they can’t, ask if they’d be willing to reserve judgement and go along with the rest of the group for six months.

If they still don’t agree, ask if they’d be okay moving to another team, subject to management approval. If that’s not okay, or if the dissenters won’t speak up (in the case of an anonymous vote), then this team isn’t a good candidate.

If the majority of the team refuses...

If the team doesn’t agree, then you’ll have to choose another team. I’ve only had this happen once, but it does happen. In my case, it was because the team didn’t trust their organization to give them the time they needed to learn. In retrospect, they were correct, and it’s a good thing we didn’t go through with the change.

If people lie about their acceptance...

Sometimes, people will vote for Agile while secretly opposing the changes. Other than making sure people don’t feel coerced, there’s nothing you can do about it. It’s not productive to second-guess people’s votes.

Even if nobody lies, the nature of change is that everyone will have second thoughts once in a while. You’ll need to work through those objections as they arise. When they do, it helps to be able to remind people that they to agreed to stick with the experiment for six months, but there is a clear end date, and if it still isn’t working by then, they’ll be able to change their mind. Be compassionate and respectful; changing habits is takes time, and people can feel like the norms they rely on have been set adrift.

In my experience, by the time the six month date rolls around, the chaos of change will be past and people will be happy with their Agile method. That’s been true for every team I’ve worked with.

If you’re introducing Agile to a lot of teams...

Changes that affect a lot of people are much more disruptive than changes that only affect a few teams. The disruption is multiplicative. Rumors start flying, people start worrying about their jobs, and the upcoming changes become part of every conversation.

Large changes—those that directly impact more than 30-70 people—require professional change management. Depending on the size of your organization, your HR department may have change management experts on staff who can help. If not, you can hire consultants. Some Agile consultants have change management experience and can guide both aspects of your Agile adoption. For very large changes, you’ll need dedicated change management experts to work alongside your Agile guides.

Organizational leaders often underestimate the importance of expert change management. This is a grave mistake! To put it in terms of the Satir Change Model, everybody goes through a period of resistance and chaos when they learn about a change. But everybody goes through it at their own pace, and leaders typically go through it first. Not because they’re better or more resilient, but because they know about it first.

By the time the rest of the organization learns about the change, organizational leaders have already experienced the “transforming idea” that resolves their feelings of resistance and chaos. To the leaders, the change now seems obvious and necessary. Why would anybody disagree?

Then they introduce the change and experience massive amounts of pushback and disruption from people going through their own resistance and chaos stages. It can kill the change entirely.

Proper change management can’t prevent all disruption, but it can reduce it. Don’t skimp. If you’re not prepared to get expert help, limit your Agile changes to a few teams at a time.

Get Stakeholder Buy-In

Your teams’ stakeholders are everyone who is affected by, or has an influence over, their work. For you and your Agile improvement efforts, your stakeholders also include anybody else who has influence over the changes you want to make.

You don’t need every stakeholder to buy in to your Agile intentions. You do need the ones with a lot of political influence to do so. If they don’t, they could quietly sabotage your efforts, pulling the rug out from under you in six months to a year, even if—or especially if—Agile is succeeding.5

5Alistair Cockburn calls this “organizational antibodies.” The more successful a change initiative is, the more people worry that it will affect them, and the more they fight against it.

The stakeholders who are most likely to resist are your teams’ business partners: product management, marketing, and sales. Agile represents a major shift in how teams interact with these groups. They’re used to a predictive approach, with a focus on commitments and deadlines. Their interactions with development teams are typically focused on documents, progress reports, and sign-offs.

Agile teams focus on feedback, frequent delivery of value, and adapting their plans. They constantly ask their stakeholders for feedback, then change their plans based on what they learn. Because their plans are always changing, they don’t make detailed release commitments. Some teams do provide release forecasts, but even then, those forecasts aren’t commitments, and they change frequently.

Some stakeholders love it. Finally, they know what’s really going on, and they have the ability to influence the outcome. Others, particularly those who have been burned by missed commitments in the past, see Agile as a political maneuver: a complicated way for teams to avoid making promises. They fight Agile tooth and nail.

Talk to your teams’ key stakeholders about Agile. This is often best done one-on-one. The topic is politically fraught, so make sure you plan your strategy with your management allies. You might not be the best person to have the conversations—a manager or person your stakeholders trust might be a better choice.

When you (or your proxy) talk to stakeholders about Agile planning, be sure to put yourself in their shoes. Demonstrate that you know what’s important to them. Perhaps they need to make marketing plans, or talk to sales prospects about competitive new features, or coordinate delivery with a third party. You want to make a connection: to dispel their preconceptions that you’re just another arrogant engineer here to tell them they can’t have what they need.

Talk about what you’re going to do better. The “Planning” chapter describes how it works. Don’t sugar-coat or over-promise. Instead, say something along the lines of, “We want to do better. We’ve found that, when we fix our plans in advance, we end up disappointing customers with delays or half-baked results. That’s not good for you or our reputation as a company. Instead of making a fixed plan and surprising you with delays, we’re going to give you more visibility and control, so you can see problems early, and we’ll adjust our plans as we go to get better results.” For many stakeholders, it’s not missed deadlines that are the biggest problem; it’s being blindsided by them.

Throughout each conversation, treat your stakeholders as trusted partners. You want them to be successful. You’ve got to balance multiple interests, and Agile teams don’t use predictive planning, but you’re there to help. You’re going to do everything you can to make their job easier and more successful.

If concrete commitments are required...

The Agile way is adaptive planning: changing your plans as you go to achieve the best possible results. That’s incompatible with making precise commitments in advance. You can commit to a specific date and steer your plans to meet that date, but you can’t predict exactly which features will be done on that date.

If that’s not good enough, fluent Delivering teams have the ability to make reliable forecasts, although it takes 4-8 weeks of development to dial them in. This allows you to predict when a particular set of features will be done. You can combine that with long planning horizons and inflexible plans to make fixed-scope, fixed-date commitments.

If your teams aren’t fluent in the Delivering zone, or if you need to make predictions before development begins, you can keep using whatever big-picture planning technique you’re using today. Agile won’t make it worse, and as development proceeds, it will give you visibility into how things are really working out. Predictive plans aren’t the Agile way, but there’s more to Agile than adaptive planning, and you can work your way up to true agility over time.

That said, predictive planning is incompatible with the Optimizing zone, which is based around taking advantage of your ability to change your plans. If you have to make commitments in advance, limit your aspirations to the Focusing and Delivering zones.

Do be careful. Some people like finding scapegoats, and some companies have a culture of assigning blame. Fixed scope, fixed date commitments are notoriously unreliable. Even if Agile doesn’t make anything worse, it might be convenient to blame Agile—and you personally—for the same deadlines that would have been missed before Agile. If that’s the case at your company, introducing Agile could be a mistake.

If stakeholders don’t buy in...

Some software teams have a contentious relationship with their stakeholders, particularly those in product management and sales. It can get pretty acrimonious. In some cases, bad blood and a lack of trust might lead stakeholders to flat out refuse to support an Agile process. They might also object to the initial slow-down involved with learning Agile (see the “Account For Learning” section in the next chapter).

If only a few stakeholders object, you can choose teams that they’re not involved with. If a lot of stakeholders object, or if their high-level leadership objects, you might be able to convince them to try out a pilot with a single team. If you can, choose a team whose stakeholders are both influential and eager to try out new ideas. It might take a while—even a year or two—but they’ll come to like the visibility and control Agile gives them, and they’ll convince their colleagues to give Agile a chance.

Sometimes software organizations try to force Agile on their stakeholders. They can even succeed, if they have enough political power, but it leads to long-term blowback. If you face widespread, active opposition, to the point that even a pilot team isn’t acceptable, Agile isn’t a good fit for your organization.

Further Reading

Once you have the buy-in you need, it’s time to make your investments. We’ll talk about that in the next chapter.

XXX Reading recommendations TBD.

Share your feedback about this excerpt on the AoAD2 mailing list! Sign up here.

For more excerpts from the book, or to get a copy of the Early Release, see the Second Edition home page.

AoAD2 Chapter 2: Choose Your Agility

Book cover for “The Art of Agile Development, Second Edition.”

Second Edition cover

This is a pre-release excerpt of The Art of Agile Development, Second Edition, to be published by O’Reilly in 2021. Visit the Second Edition home page for information about the open development process, additional excerpts, and more.

Your feedback is appreciated! To share your thoughts, join the AoAD2 open review mailing list.

This excerpt is copyright 2007, 2020 by James Shore and Shane Warden. Do not distribute or republish without James Shore’s express written permission.

Choose Your Agility

Real talk: There’s no point in Agile for the sake of Agile. The only reason to be Agile is to improve your results. So, what results will you get?

Let’s Talk Success

When I was a kid, I was happy to just play around with computers. I loved the challenge of programming. When I got a program to work, it was a major victory. Back then, even a program that didn’t work was a success of some sort, so long as I had fun writing it. My definition of success centered on personal rewards.

As I gained experience, my software became more complicated. I often lost track of how my programs worked, and I had to abandon some programs before they were finished. I began to believe that maintainability was the key to success—an idea that was confirmed as I entered the workforce and began working with teams of other programmers. I prided myself on producing elegant, maintainable code. Success meant technical excellence.

Despite good code, some products flopped. Even impeccably executed code could elicit yawns. I came to realize that my teams were part of a larger ecosystem involving dozens, hundreds, or even thousands of people. My software needed to satisfy those people... particularly the ones signing my paycheck. In fact, for the people funding the work, the value of the software had to exceed its cost. Success meant delivering value to the organization.

At each step along my journey, my understanding of the world expanded. For a while, I found myself perplexed by the companies that achieved massive organizational success despite creating technical messes and treating their people terribly. Now I've come to realize that my previous conclusions weren’t wrong. They were just incomplete.

  • Organizational success makes a company;

  • Lack of technical success breaks a company;

  • And personal success glues it together.

You need all three types of success to get the best results.

A Venn diagram showing three circles, marked “Organizational Success,” “Technical Success,” and “Personal Success.” There’s a star in the center where all the circles intersect.

Figure 1. Types of success

The Importance of Organizational Success

Organizational success is often neglected by software teams. They assume that, if they just do what they’re told, they’ll be successful.

Rest assured, however, that even if your teams aren’t taking responsibilty for success, the broader organization is judging them at this level. For senior management, it’s not enough for your software to be elegant, maintainable, or even beloved by its users. These are just means to an end. They care about results: the return on investment your teams produce. If your teams don’t achieve this sort of success, management will take steps to ensure that they do.

Unfortunately, senior managers don’t usually have the time or perspective to apply a nuanced solution to each team. They wield swords, not scalpels. They rightly expect their product teams to take care of fine details.

When senior management is unhappy with teams’ results, the swords come out. Costs are the most obvious target. There are two easy ways to cut them: set aggressive deadlines to reduce development time, or ship the work to a country with a lower cost of labor. Or both.

These are clumsy techniques. Aggressive deadlines end up increasing schedules rather than reducing them [Mcconnell 1999] (p.220), and offshoring has hidden costs of its own [Overby 2003].

Do aggressive deadlines and the threat of offshoring sound familiar? If so, it’s time for your teams to take back responsibility for success: not just personal or technical success, but organizational success as well.

Enter Agility

Will Agile help your teams be more successful? That depends on how much your company embraces its ideas. Agile development focuses on achieving personal, technical, and organizational successes. If you’re having trouble with any of these areas, Agile might help.

Organizational Success

Organizational success is crucial. Without it, the money will eventually run out. Organizational success doesn’t just mean dollars and cents, though; see the “What Organizations Value” sidebar.

Agile teams achieve organizational success by focusing on delivering value and decreasing costs. This directly translates to increased return on investment (ROI). Agile teams also validate expectations early, so if an idea won’t provide good ROI, you’ll find out early enough to stop working on it before your organization has spent much money.

Specifically, Agile teams increase value by seeking out business expertise, getting frequent user and customer feedback, and focusing development on the core value that their products provide. Agile teams release their most valuable features first and release updates frequently. When business needs change, Agile teams change direction to match. The best Agile teams actively seek out opportunities to adapt their plans.

Agile teams decrease costs as well. They do this partly via technical success; the best Agile teams generate only a few bugs per month. They also eliminate waste by cancelling bad products early and replacing expensive development practices with simpler ones. Agile teams communicate quickly and accurately, and they’re not bottlenecked by the work of a single team member. They regularly improve their process and code, making their software easier to maintain and enhance over time.

Technical Success

Technical success grants teams the ability to cost-effectively enhance and maintain their software. Without technical success, costs rise and your teams’ ability to make changes plummets. Eventually, they have to throw their software away and rewrite it. It can take years for the replacement to achieve feature parity with the old code. Meanwhile, they’re spending buckets of money just to catch up with where they used to be, all while their customers become increasingly frustrated with their lack of progress.

I highlight two Agile approaches to technical success in this book. The first is Extreme Programming (XP), an Agile method that’s particularly adept at technical successes. In XP, programmers work closely together, which helps them keep track of the nitpicky details necessary for great work. Programmers continuously integrate their code, which enables the team to release their software whenever it makes the most business sense. The whole team focuses on finishing each feature completely before starting the next, which prevents unexpected delays and allows the team to change direction at will.

In addition to the structure of development, XP includes advanced technical practices that lead to excellence. The most well-known practice is test-driven development, which helps programmers write code that does exactly what they intend. XP teams also create simple, ever-evolving designs that are easy to modify when plans change.

I also include techniques from the DevOps movement2, which brings XP’s ideas into the modern era of cloud-based software. Agile teams using XP and DevOps techniques create high-quality, low-defect software that’s easy to modify, maintain, and operate.

2Not to be confused with DevOps tools, which dominate much of the discussion around DevOps.

Personal Success

Personal success is, well, personal. Most people need more than a paycheck to feel satisfied, and they don’t always complain when they’re unhappy. In the best case, they find a new job, taking their accumulated organizational knowledge with them. In the worst case, they’ll quietly sabotage the company’s efforts as they agitate for work that’s more satisfying.

Agile development won’t satisfy everyone’s needs, and it’s different enough that it can be uncomfortable at first. However, once people get used to it, they find a lot to like about Agile:

Executives and senior management
People focused on the bottom line appreciate Agile teams’ emphasis on providing visibility, a solid return on investment, and long-lived code.
Users, stakeholders, domain experts, and product managers
People who care about what the software can do appreciate their ability to influence the direction of development; teams’ focus on delivering useful and valuable software; and increased delivery frequency.
Product and project managers
People interested in delivering a valuable product appreciate their ability to change direction as business needs change; teams’ reliability; and improved stakeholder satisfaction.
UX and graphic designers
People obsessed with user experience appreciate their ability to influence development plans; teams’ willingness to iterate on ideas; and faster, more accurate implementation of design ideas.
Programmers
People who write code appreciate the improved quality of life that results from technical excellence; their improved influence over forecasts and schedules; and their autonomy.
Testers
People who focus on quality appreciate being integrated with the rest of the development team; their ability to influence quality at all stages of development; and more challenging, less repetitive work.
Operations
People who keep software running in production appreciate being included in development decisions; having operational requirements included in development plans; programmers’ focus on providing actionable telemetry and logging; and teams taking responsibility for post-release quality and up-time.

Working on Agile teams has provided some of the most enjoyable moments in my career. Imagine the camaraderie of a team that works together to identify and deliver products of lasting value, with each team member enthusiastically contributing to a smooth-running whole. Imagine how it feels to take responsibility for your areas of expertise, whether technical, business, or management, with the rest of the team trusting your professional judgment and ability. Imagine how pleasant it is to address the frustrations of your work and see quality improve over time.

Agile development changes the game. Developing and delivering software in a new way takes a lot of work and thought. Yet if you do it consistently and rigorously, you’ll experience amazing things: you’ll ship truly valuable software on a regular basis. You’ll demonstrate progress multiple times per month. You’ll have the most fun you’ve ever had in software development.

The Agile Fluency® Model

I’ve painted a rosy picture of Agile. Is it too good to be true? That’s up to your company.

In 2014, I partnered with Diana Larsen to analyze why companies see such different results from their Agile teams. We’ve both worked with Agile teams from the beginning, and, over the years, we noticed that Agile teams tend to cluster in different “zones” of capability. We captured these observations in the Agile Fluency3 Model [Shore and Larsen 2018].

3“Agile Fluency” is a registered trademark of Agile Fluency Project LLC, founded by Diana Larsen and James Shore.

A picture of the Agile Fluency Model. It shows a path from “Pre-Agile” through a shift in team culture to “Focusing,” followed by a shift in team skills to “Delivering,” then a shift in organizational structure to “Optimizing,” and finally a shift in organizational culture to “Strengthening.” The path continues and fades away, as if there are additional zones yet to be discovered.

Figure 2. The Agile Fluency Model

Although the “Agile Fluency Model” figure shows a clear path from one zone to the next, reality is messier: each zone actually consists of multiple skills, all of which develop in parallel. A team has proficiency in a zone when they’re able to apply all of its skills. They have fluency when they can do so with effortless grace, even under pressure. Teams can develop fluency in any order, although the progression shown in the figure is most common.

The zones correspond to how companies invest in Agile ideas. In other words, the fluency your teams achieve—and the results your company gets—depends on how much it buys in to the Agile philosophy. Not just lip service, but actual, meaningful changes that cost time, money, and political capital.

Each zone has its own set of investments, and you probably won’t be able to convince your company to invest in every zone. That’s okay. Each zone also has its own set of benefits, and as long as your company fully invests in at least one zone, you’re likely to see improvements.

Which zones should you choose? It’s a matter of picking the right cost/benefit tradeoffs.

Focusing Zone

The Focusing zone corresponds to the fundamentals of Agile. For your teams to reap the benefits of Focusing fluency, your organization needs to buy in to the core Agile philosophy.

What is that core philosophy? As we saw in the “Essence of Agile” section: Agile is adaptive rather than predictive, and people-oriented rather than process-oriented.

In practice, this means that Focusing teams regularly review and change their plans. They work as a tightly-knit team to define their own work processes and self-organize around completing the work. All the while, they’re Focusing on the next piece of value for their organization.

For most teams and organizations, this requires a shift in how they think about teams. Pre-Agile organizations make plans in advance, ask teams for estimates, and expect reports about how work is progressing relative to those estimates. Focusing teams revise their plans frequently—at least every month—and demonstrate progress by showing what they’ve completed.

Pre-Agile organizations break their plans into tasks, assign those tasks to individuals on the team, and judge individuals based on how well they complete their tasks. Focusing teams do their own task breakdown, decide for themselves who will work on each task, and expect to be judged on their ability to create value as a team.

Can your organization buy into this core Agile philosophy? Again, lip service isn’t enough. They’ll need to make concrete investments in the form of changes to team structure, management, and work environment. (See the “Investing in Agility” chapter for details.) It’s a “good news, bad news” situation: The bad news is that, when the rubber meets the road, some organizations won’t be willing to invest. The good news is that, if they refuse, you’ve discovered early that they’re not really on board with the Agile philosophy. You just saved yourself years of frustration and heartache chasing Cargo Cult Agile.

If you are able to get buy-in, Focusing fluency will take each team about 2-6 months of dedicated practice to develop, with benefits appearing within 1-4 months. Part II shows how. In exchange, expect to see the improvements in value described in the “Organizational Success” section. Some aspects of personal success described in the “Personal Success” section will also improve. Specifically, people with a business and user focus will like their improved visibility into the team’s work and their ability to influence teams’ plans.

You aren’t likely to see the improved technical success described in the “Technical Success” section, though, or the organizational cost savings that come with it. In fact, technical quality could get worse. Traditional approaches to design rely on careful up-front work based on firm plans. That doesn’t mesh well with Agile’s emphasis on flexible, frequently changing plans. As a result, Focusing teams often end up with an ad-hoc approach to design, which isn’t sustainable.

Poor technical quality can be frustrating for team members, lowering their feelings of personal success. It usually prevents the team from making reliable forecasts and commitments, which can be frustrating for stakeholders.

To improve technical success and make useful forecasts, you’ll also need fluency in the Delivering zone.

Delivering Zone

Companies who invest in the Focusing zone have taken an important first step: they’ve aligned themselves with the core Agile philosophy. This is critical! It’s the secret to Agile success.

But it’s not enough for long-term success. Although organizational success makes a company, and Focusing fluency will help you get it, lack of technical success breaks a company. Software built by Focusing teams becomes unmaintainable over time. Over the course of several years, costs will rise. The team will eventually lose their ability to make cost-effective changes. They’ll tell you they need to throw away the software and rewrite. That can kill a product.

Delivering teams prevent this problem by investing in technical success. They design their code so that it can absorb frequent changes. They keep code quality high, so they don’t waste time hunting bugs. They refine their production lifecycle so releases are painless and operations are manageable. They’re capable of Delivering reliable, low-defect software whenever it makes the most business sense.

This requires a shift in team members’ skills, which requires a substantial investment in learning and code quality, as well as structural changes to integrate technical disciplines such as Testing and Ops into each team. (See the “Investing in Agility” chapter for details.)

If your company makes these investments, Delivering fluency will take each team 3-24 months to develop, in addition to the time needed for Focusing fluency, although you’ll see benefits within 2-6 months. Part III has the techniques. The exact amount of time each team will need depends on the quality of their existing code and how much coaching they receive.

In exchange, expect to see the technical success factors described in the “Technical Success” section, the cost reductions described in the “Organizational Success” section, and the team member success factors described in the “Personal Success” section. Combined with Focusing fluency, that’s all the success factors described at the beginning of this chapter.

Optimizing Zone

The vast majority of companies would be satisfied with Focusing and Delivering fluency. But Agile imagines more. In its full glory, Agile is a world in which teams twirl and dance in response to changing market conditions. They experiment and learn; develop new markets; outmaneuver the competition.

Optimizing teams achieve this level of agility. They understand what their market wants, what your business needs, and how to bridge the two. Or, as in a startup environment, they know what they need to learn and how to go about learning it. They’re constantly Optimizing their product plans to achieve the most value possible.

This requires a shift in organizational structure. Creating optimal plans requires constant attention by people with deep business and product expertise, and that means having product and market experts join development teams full-time. It also means giving those teams full responsibility for their product budgets and plans.

These structural changes require high-level permission in the organization. It can be difficult to obtain. It typically takes teams at least a year of building trust via Delivering fluency before they get permission for these investments. Once that happens, Optimizing fluency takes another 3-9 months to develop, although benefits appear within 1-3 months. That said, Optimizing is a never-ending process of experimentation, learning, and discovery. Part IV describes how to begin.

Strengthening Zone

There’s one final zone in the Agile Fluency Model. It’s largely speculative: a possible future for Agile. It’s also only appropriate for organizations on the bleeding edge of management theory and practice. That puts it out of scope for this book. Briefly, the Strengthening zone involves distilling teams’ collective insights and channeling them into organizational improvements. If you’d like to learn more, start with [Shore and Larsen 2018].

Choose Your Zones

Will Agile get you better results than you’re seeing now? It depends on which zones your organization can support. In a vacuum, the first three zones together provide the best results and the purest realization of Agile ideas. But that also takes the most investment. Investing in more than you need could incur backlash. Only invest in the zones that will be a big benefit for your organization.

Once you’ve chosen a zone, though, invest fully in it. Without sufficient investment in a zone, the skills teams need to become fluent in that zone will develop slowly, if at all, and full fluency will probably be out of reach. You’ll incur the costs of learning without getting all the benefits, and you might even see worse results than now if teams don’t have the support they need.

In other words, choose the zones that your company both needs and is willing to pay for.

So, which zones should you choose? As you’ll see in the next chapter, it starts with a conversation with management. But here’s a handy guide.

  • Every Agile team needs Focusing fluency. It’s fundamental. If your company can’t at least invest in Focusing fluency, Agile isn’t a good fit. You don’t need perfect support, as the “Investing in Agility” chapter describes, but if you can’t at least get the minimum, you’ll need a different approach.

  • Delivering fluency decreases costs and increases development speed. It takes time to learn, but you’ll see benefits within 2-6 months, and it will pay for itself within a year. Without it, your code will eventually succumb to technical debt. That makes the Delivering zone a no-brainer for most teams. That said, some organizations aren’t ready to make the big investments in learning and code quality that the Delivering zone requires. It’s okay to start with Focusing fluency first, demonstrate success, and then use that to make further investment.

  • Optimizing fluency is where Agile shines brightest. It’s also a lot to ask. For most organizations, it’s best to build trust by demonstrating fluency in the Delivering zone first, then gradually take on more responsibility. But if your organization already has a culture of delegating decision-making authority to cross-functional teams, as is often seen in startups, Optimizing fluency will give you great results.

Whichever zones you choose, invest in learning all of them simultaneously. It’s fastest and easiest to learn Agile ideas if you practice everything together, because the techniques in the later zones make the earlier zones work better. If you can’t get all the investments you want, though, that’s okay. It takes longer, but as long as you can at least invest in Focusing, you can build up to the later zones over time.

Once you’ve chosen your zones, you’re ready to get started. That’s the topic of our next chapter.

Share your feedback about this excerpt on the AoAD2 mailing list! Sign up here.

For more excerpts from the book, or to get a copy of the Early Release, see the Second Edition home page.

Complete “TDD Lunch & Learn” Series Available

Screenshot showing a slide labelled “Overlapping Sociable Tests.”

The “Testing Without Mocks” episode.

My full “TDD Lunch & Learn” series is now available! This weekly livestream series, which ran from May to September 2020, uses test-driven development (TDD) to develop a microservice and accompanying command-line application from scratch. Each episode focuses on a new challenge.

The series emphasizes real-world TDD, with a strong focus on “hard” problems such as networking, logging, and testing without mocks. If you’ve learned the basics of TDD and want to see how it works in the real world, this series is the perfect next step.

Start Here.

Microservice Architecture Without Microservice Overhead

Every week in my Tuesday Lunch & Learn livestream, we choose a useful software development skill, define a challenge related to that skill, and solve the challenge live. This week, we’re looking at microservice architecture.

Microservice architecture is a great way to partition code so that teams can work independently. It’s effective at preventing the “big ball of mud” problem that can plague large monolithic codebases.

But microservice architecture comes at a high cost. Because all communication happens via network calls, microservice architecture introduces a lot of complexity in the form of error handling and distributed transactions.

What if there was a way to get the advantages of microservice architecture without all the added complexity? In this episode, we look at “microliths,” a way to reduce overhead while still retaining the benefits of a microservice-based design. We cut the size of our codebase in half and increase performance across the board.

To follow along, download the code from GitHub and check out the 2020-09-29 tag. To see the final result, check out the 2020-09-29-end tag or view it on GitHub.

Visit the Lunch & Learn archive for more.

AoAD2 Chapter 1: What is Agile?

Book cover for “The Art of Agile Development, Second Edition.”

Second Edition cover

This is a pre-release excerpt of The Art of Agile Development, Second Edition, to be published by O’Reilly in 2021. Visit the Second Edition home page for information about the open development process, additional excerpts, and more.

Your feedback is appreciated! To share your thoughts, join the AoAD2 open review mailing list.

This excerpt is copyright 2007, 2020 by James Shore and Shane Warden. Do not distribute or republish without James Shore’s express written permission.

What Is Agile?

Agile is everywhere. And, paradoxically, nowhere.

In the 20 years after the Agile freight train roared into software developers’ conscious, the number of companies calling themselves “Agile” increased by orders of magnitude. The number of teams actually taking an agile approach to their work? Not so much. “Agile,” the easily-repeated name, is enormously successful. The ideas behind Agile—well, most of them are ignored.

Let’s fix that.

Agile’s Genesis

In the 1990s, software development was believed to be in crisis. They actually called it that: “The Software Crisis.” Software projects were over-budget, late, didn’t meet requirements, and—according to the oft-quoted and ominously named “CHAOS Report”—nearly a third of them were cancelled outright [Standish 1994].

Agile wasn’t a response to this crisis. Far from it. Agile was a response to the response.

To bring software development under control, big organizations created highly detailed processes that defined exactly how software was to be created. Everything was tightly controlled so that no mistakes could be made. (In theory, anyway.)

First, business analysts would interview stakeholders and document the system requirements. Next, software architects would read the requirements documents and create detailed design documents specifying every component of the system and how they related to each other. Then programmers would convert the design documents to code. In some organizations, this was considered low-skill work—just a mechanical translation exercise.1 Meanwhile, test leads would use the same documents to generate test plans, and when coding was done, armies of QA personnel would manually follow those test plans and report variances as defects. After each phase, everything would be carefully documented, reviewed, and signed off.

1In fact, programming was considered such a mechanical activity that, for a while, there was a movement called CASE—Computer Aided Software Engineering—to automatically convert architectural diagrams into code.

This approach was called “waterfall development” or “phase-gate development.” If it sounds like a ridiculous straw-man, well, consider yourself fortunate. Not every company used a heavyweight process in the ’90s, but it was widely recognized as a logical and sensible way to work. Of course you needed to define requirements, then design, then implement, then test. Of course you needed to document every phase. This was discipline. This was engineering. How else could you possibly succeed?

Born Out of Crisis

Big companies defined their processes in excruciating detail. Roles, responsibilities, document templates, change control boards... every aspect of development was defined and controlled. If a project didn’t succeed—and according to the CHAOS Report, less than a sixth of them did—it was because the process needed more detail, more documents, more sign-offs. It resulted in a massive amount of documentation. Martin Fowler called it “The Almighty Thud.” [Fowler 1997]

This wasn’t a great way to work. It was bureaucratic and dehumanizing. Skill didn’t seem to matter as much as adherence to process. Programmers felt they were interchangeable cogs in an impersonal machine. It didn’t even work all that well.

So several people independently developed simpler, slimmer, and less prescriptive methods for developing software. They were called “lightweight methods” in contrast to the heavyweight methods used by big companies. These new methods had names like “Adaptive Software Development,” “Crystal,” “Feature-Driven Development,” “Dynamic Systems Development Method,” “Extreme Programming,” and “Scrum.”

By the late ’90s, these methods were attracting serious attention. Extreme Programming, in particular, saw an explosion of grass-roots interest among programmers. In 2001, 17 of the lightweight methodology proponents met at a ski resort in Utah to discuss unifying their efforts.

The Agile Manifesto

“I personally didn’t expect this particular group of [people] to ever agree on anything substantive,” Alistair Cockburn said later.

And, in fact, after two days, they only accomplished two things: the name “Agile,” and a statement of four values (see the “Agile Values” figure). Over the following months, over email, they hashed out twelve accompanying principles (see the “Agile Principles” figure).

This was the Agile Manifesto. It changed the world. So, as Alistair went on to say, they did agree on something substantive after all.2

2Alistair Cockburn, quoted by Jim Highsmith in [Highsmith 2001]. The full quote is, “I personally didn’t expect that this particular group of agilites to ever agree on anything substantive... Speaking for myself, I am delighted by the final phrasing [of the Manifesto]. I was surprised that the others appeared equally delighted by the final phrasing. So we did agree on something substantive.”

Agile Values

Figure 1. Agile Values

Agile Principles

Figure 2. Agile Principles

But there was no unified Agile method. There never has been, and never will be. Agile is three things: the name, the values, and the principles. That’s it. It’s not something you can do. It’s a philosophy. A way of thinking about software development. You can’t “use” Agile or “do” Agile... you can only be Agile. Or not. If your teams embody the Agile philosophy, then they’re Agile. If they don’t, they’re not.

The Essence of Agile

Martin Fowler has made a career out of turning complicated software topics into well-considered, even-handed explanations. His explanation of “The Essence of Agile Software Development” is one of the best:

Agile Development is adaptive rather than predictive; people-oriented rather than process-oriented.3

3Fowler has expressed this idea in many ways over the years. It originated in [Fowler 2000]. As of October 2020, his most up-to-date overview could be found at https://martinfowler.com/agile.html.

Martin Fowler

Adaptive rather than predictive

Remember the CHAOS Report, which said that only one-sixth of software projects were successful? It had a very specific definition of success:

Resolution Type 1, or project success: The project is completed on-time and on-budget, with all features and functions as originally specified.

Resolution Type 2, or project challenged: The project is completed and operational but over-budget, over the time estimate, and offers fewer features and functions than originally specified.

Resolution Type 3, or project impaired: The project is canceled at some point during the development cycle.

These definitions illustrate the predictive mindset perfectly. They’re all about conformance to plan. If you did what you said you were going to do, you were successful. If you didn’t, you weren’t! Easy.

It makes sense at first. But look closer. There’s something missing. As Ryan Nelson wrote in CIO Magazine [Nelson 2006]:

Projects that were found to meet all of the traditional criteria for success—time, budget, and specifications—may still be failures in the end because they fail to appeal to the intended users or because they ultimately fail to add much value to the business.

... Similarly, projects considered failures according to traditional IT metrics may wind up being successes because despite cost, time or specification problems, the system is loved by its target audience or provides unexpected value. For example, at a financial services company, a new system... was six months late and cost more than twice the original estimate (final cost was $5.7 million). But the project ultimately created a more adaptive organization (after 13 months) and was judged to be a great success—the company had a $33 million reduction in write-off accounts, and the reduced time-to-value and increased capacity resulted in a 50 percent increase in the number of concurrent collection strategy tests in production.

Agile teams define success as delivering value, not conforming to a plan. In fact, truly Agile teams actively look for opportunities to increase value by changing their plans.

Take a second look at the Manifesto (see the “Agile Values” figure and the “Agile Principles” figure). “Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.” How many of its values and principles relate to delivering valuable software and adapting to feedback?

People-oriented rather than process-oriented

Heavyweight processes tried to prevent errors by carefully defining every aspect of software development. By putting the “smarts” in the process, individual skill became less important. In theory, you could apply the same process over and over, with different people, and get the same results. (Come to think of it, they kind of did. Just not the results they wanted.)

Agile says people are the most important factor in software development success. Not just their skills, but all aspects of their humanity. How well team members work together. How many distractions they encounter. How safe they feel. Whether they’re comfortable voicing dissent, and whether they feel motivated by their work.

Agile teams have a process—every team does, even if it’s implicit—but the process is in service of the humans, not the other way around. And Agile teams are in charge of their own process. When they think of a better way of working, they change it.

Look at the Manifesto again (see the “Agile Values” figure and the “Agile Principles” figure). “Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.” Which other values and principles relate to putting people first?

Why Agile Won

In the first ten years after the Manifesto, Agile faced enormous criticism. It was “undisciplined,” critics said. “It could never work.” Another ten years after that, the critics were silent. Agile was everywhere, at least in name. Heavyweight waterfall methods were effectively dead. Some younger programmers had trouble believing anybody ever could have worked that way.

There’s actually nothing inherently wrong with the basic waterfall process. If you keep it slim and operate in a well-understood domain, waterfall can work well. The problem was the heavyweight processes big companies used. Ironically, the processes designed to prevent problems actually caused many of the problems organizations were seeing.

With software, it’s very difficult to imagine how it will work before you actually use it, and it’s even harder to think of absolutely everything your software needs to do. This is doubly true for people who aren’t actively involved with software development. As a result, it’s critically important to get working software in front of people as soon as possible. You need to get feedback about what’s missing or wrong, then change your plans based on what you learn. As the Manifesto says, “working software is the primary measure of progress.” Learning and responding to change are at the heart of what Agile is all about.

Those heavyweight processes put so much emphasis on process controls and documentation and sign-offs that they incurred a huge amount of delay and overhead. They took years to produce working software, and they had nothing concrete to show until near the end. Instead of welcoming change, they actively worked to prevent change. They actually had a dedicated part of the process, the Change Control Board, whose primary purpose was to say “no” to change requests. (Or, more accurately, “yes, but it will cost you.”)

All of this added up to projects that spent years in development before they had anything to show. When they did, it was too late and too expensive to make changes. They ultimately shipped software that didn’t do what customers needed.

Although there are a variety of approaches to Agile—and some of them are more about co-opting a popular name than following the actual philosophy—one thing they all have in common is a focus on making progress visible and allowing stakeholders to make course corrections as they go. This seems like a small thing, but it’s incredibly powerful. It’s why we no longer hear about a Software Crisis. Software is still late. It’s still over budget. But Agile teams don’t spend years building failures. And that’s huge.

There’s more to Agile than just providing visibility. But this one thing? This was enough. It’s why everybody wanted Agile.

Why Agile Lost

At first, Agile was a grassroots movement. It was largely driven by programmers seeking better results and better quality of life. As it became more successful, Agile’s momentum shifted from the underlying ideas to hype. Rather than saying, “let’s get better results by adapting our plans and putting people first,” organization leaders started saying, “Everybody’s talking about Agile. Get me some Agile.”

The thing is, there is no “Agile” to go get. It’s just a set of values and principles. There are specific Agile approaches, such as Extreme Programming and Scrum, that will tell you how to be Agile, but you still have to be on board with the underlying philosophy.

And for a lot of organizations, that underlying philosophy—adapting plans and putting people first—is really, really foreign.

The tragedy of the cargo cult is its adherence to the superficial, outward signs of some idea combined with ignorance of how that idea actually works. In the story, the islanders replicated all the elements of cargo drops—the airstrip, the tower, the headphones—but didn’t understand the vast infrastructure that enabled airplanes to arrive.

The same tragedy occurs with Agile. People want Agile’s Cargo: better results, more visibility, fewer business failures. But they don’t understand the underlying philosophy, and often wouldn’t agree with it even if they did. They want to buy Agile, but you can’t buy an idea.

What they can buy is the outward signs of Agile. Stand-up meetings! User stories! Tools! Certifications! There’s lots of stuff labelled Agile, and plenty of people eager to sell it to you. It’s often sold as “enterprise-grade,” which is a way of saying “don’t worry, you won’t have to change.” Uncomfortable ideas like “adaptive planning” and “people-centric” are ignored.

And that’s how you get a cargo cult. All the activity; none of the results. The Agile part is missing.

“At my old company they wasted a huge number of man hours in meetings.”

“[Agile] cost an entire team (30+) people their jobs as they produced almost nothing for almost a year.”

“All [Agile] means is developers get shafted when the project changes... the day before delivery.”

“It‘s because of Agile there are sh—ty developers and sh—ty products being shipped.”

“I can’t take this Agile crap any more. It’s lunacy.”

comments about Agile from around the Web

The name Agile is everywhere. The ideas of Agile aren’t. It’s become self-perpetuating: for many, the only Agile they know is Cargo Cult Agile.

Icon of person in a suit wearing coconut headphones

It’s time to fix that. In the rest of this book, I’ll show you how to apply Agile ideas for real. Keep an eye out for the Cargo Cult Agilists shown in the margin. They’ll show you what not to do.

Ready? Let’s go.

Share your feedback about this excerpt on the AoAD2 mailing list! Sign up here.

For more excerpts from the book, or to get a copy of the Early Release, see the Second Edition home page.

Art of Agile Development, Second Edition: Open Review!

The open review for The Art of Agile Development, Second Edition has begun! As we finish sections of the book, I’ll publish them on the Second Edition home page.

We want your feedback! Join the AoAD2 open review mailing list to be informed when new sections are released and to share your thoughts.

Sign up here.

Request Cancellation

Every week in my Tuesday Lunch & Learn livestream, we choose a useful software development skill, define a challenge related to that skill, and solve the challenge live. This week, we’re cancelling network requests.

In past episodes, we built a small command-line application that uses a microservice to encode text. We put a lot of effort into error handling, including making sure that the client times out if the microservice doesn’t respond.

There’s a catch, though: although the client times out and displays an error, it doesn’t actually cancel the request, so the application doesn’t exit. In today’s episode, we fix that problem.

This turns out to be a design problem. Although cancelling requests is easy to do in Node.js—you just call request.destroy() on the Node request object—the challenge is doing it cleanly. How can we make our low-level HTTP code cancel a request without exposing the internal details? How can we make our API convenient to use and understandable? How can we make it testable, so we can test-drive our application-layer code?

To follow along, download the code from GitHub and check out the 2020-09-22 tag. To see the final result, check out the 2020-09-22-end tag or view it on GitHub.

Visit the Lunch & Learn archive for more.

Request Timeouts

Every week in my Tuesday Lunch & Learn livestream, we choose a useful software development skill, define a challenge related to that skill, and solve the challenge live. This week, we’re timing out network requests.

Not all server failures are clear-cut. Sometimes servers accept a connection, but then just leave it open without sending a response, or trickle out data one byte at a time. Your normal error-handling code won’t do anything about this. To your code, everything will appear to be working, even though the server is functionally broken.

As a result, it’s important to program your clients to recognize a “stuck” server. And, of course, if you’re going to program it, you need to be able to test it. How can you write a fast unit test that simulates a misbehaving server? How can you program your application code to time out? We answer these questions in today’s episode.

To follow along, download the code from GitHub and check out the 2020-09-15 tag. To see the final result, check out the 2020-09-15-end tag or view it on GitHub.

Visit the Lunch & Learn archive for more.

Art of Agile Development, Second Edition: Early Release!

The Early Release preview of The Art of Agile Development, Second Edition is now available through O’Reilly’s online learning platform!

Get it here.

For more information about the book, including the full table of contents, a video discussing the book, and details about the development process, see the Second Edition home page.

Microservice Clients Without Mocks, Part 2: Paranoic Telemetry

Every week in my Tuesday Lunch & Learn livestream, we choose a useful software development skill, define a challenge related to that skill, and solve the challenge live. This week, we’re finishing our two-part series on testing microservice clients.

In today’s episode, we finish off our ROT-13 tool by using last week’s generic HttpClient class to build a dedicated client for our ROT-13 service. Calling the service is easy; the real challenge is making sure that our code is robust against errors. With services, anything that can go wrong, eventually will go wrong, so we need to make sure our code can handle anything.

In addition to handling errors, we also make sure that all our code can be tested without mocks. This involves making our ROT-13 client nullable and trackable, and we also program it to generate errors on command. This lets us test how our application code deals with client errors.

To follow along, download the code from GitHub and check out the 2020-09-01 tag. To see the final result, check out the 2020-09-01-end tag or view it on GitHub.

Visit the Lunch & Learn archive for more.

Microservice Clients Without Mocks, Part 1: HttpClient

Every week in my Tuesday Lunch & Learn livestream, we choose a useful software development skill, define a challenge related to that skill, and solve the challenge live. This week, we’re starting a two-part series on testing microservice clients.

In today’s episode, we build a generic HttpClient class that can be tested without using mocks or integration tests. There are two main challenges: First, we need to build the client itself, which requires that we implement a small Spy Server so we can test that the client works properly. Second, we need to make the client testable without mocks. This involves making it nullable, configurable, and trackable.

To follow along, download the code from GitHub and check out the 2020-08-25 tag. To see the final result, check out the 2020-08-25-end tag or view it on GitHub.

Visit the Lunch & Learn archive for more.

The Art of Agile Development, Second Edition

For this week’s Tuesday Lunch & Learn livestream, I have a special show for you: the new edition of my book, The Art of Agile Development, Second Edition.

The Art of Agile Development came out in 2007. It’s a well-loved classic that’s still talked about today... and now it’s time for an update. In today’s episode, I talk about what’s new, what’s changed, and I read several excerpts from the upcoming Early Release.

If you prefer text, there’s a transcript below the video.

Edited Transcript

  1. Introduction
  2. Book Excerpt
  3. Discussion
  4. Certification
  5. Waterfall
  6. The Cover and Lost Subtitle
  7. Providing Feedback
  8. Commitments & Deadlines
  9. Changes in Second Edition
  10. Cargo Cult Agilists
  11. C-Levels
  12. Final Thoughts

Introduction

Welcome to Tuesday Lunch & Learn. I’m James Shore. Every week, we look at software engineering topics, and this week, I have a special show for you. I’m going to be reading from the new edition of my book, The Art of Agile Development. I’m really looking forward to it.

(Reading chat) UncleScientist, welcome to the show; third_man, welcome, thanks for coming back; GrumpyGameDev—(chat says: “false starts are the best starts”)—yes, we’ll just pretend that never happened. I’ll edit it right out of the final video.

Today we’re looking at the second edition of The Art of Agile Development. This, of course, is my book originally published in 2007. I originally wrote it along with Shane Warden. It’s, I think, a pretty well-loved book. It was pretty popular—people still talk about it today—and it’s overdue for an update. So I’m really happy to say that I am working on an update for it.

If you’d like to see what we’re doing with it, go right down here, jamesshore.com/s/aoad2, and you’ll see the current page. This is going to be developed entirely in the open. It’s coming out on O’Reilly Early Release later this month, which means you’ll be able to see the chapters that we’ve written so far. And then, as we write additional chapters, those are actually going to go up on my website for feedback and comments. Then we will take those comments and bring them into the actual book.

If you’d like to be informed when that happens, go to this shortlink here—let me put this up on the screen—right here, jamesshore.com/s/updates, or you can follow me on Twitter and look at the #aoad2 hashtag. I‘ll be putting announcements on Twitter as well.

Yeah, as UncleScientist mentions in the chat, I have updated the site design to go along with the upcoming release of the book and (laughs) I like it a lot better. If you saw the old design, the nicest thing I can say about it was that it was a very “classic” design. (laughs) But I’m much happier with the design now.

OFFY_98, welcome, thank’s for joining me. JitterTed, glad to see you again too.

What’s new... by the way, I want to mention, of course, that I am following with Twitch chat; your comments are very welcome. As we go along, please put them in and I’ll address them as we go.

Let me take you over to the upcoming table of contents. (Brings up book’s web page.) Now this is a draft table of contents.

The original edition of the book was aimed at the sort of “early majority” of companies who wanted to try out Agile. People had heard of Agile, but they weren’t really using it yet. So the first edition of the book was, “This is how you use Agile in your company.”

Now it’s 13 years later and everybody’s has heard of Agile. Almost everybody has worked in a company that’s called itself Agile in some way. And, whether they have or not, everybody has an opinion on what Agile is. And there’s a ton of misinformation about Agile, and a lot of bad Agile.

So Shane and I are rewriting the book for a modern audience: “Here’s what Agile really is, and here’s how to make it really work.” We’re keeping the pragmatic, down-to-earth, “here’s how to do it” approach, but we’re updating absolutely everything else.

(Scrolling through table of contents.) You can see the first part has been completely rewritten. The table of contents is brand new; we’re restructuring everything, moving it all around. There’s going to be more about how to customize Agile; more about the ideas behind Agile; and more about working with modern cloud-oriented development.

So in a moment... (pauses to read chat) GrumpyGameDev says, “Do you mean that simply labelling it Agile and still being waterfall isn’t actually Agile?” Yeah. (laughs) I would say so. In fact, we’ll get to that when I read the excerpt. Which I’ll get to in just a moment. I’m going to read the first chapter of the book, which is called “What is Agile?” But even if you think you know what Agile is, or even if you do know what Agile is, I think you might enjoy this chapter. I’m really proud of it.

Before we get going, I want to remind you all that this show—and in fact, the writing of the book—is made possible by the people who hire me for training and consulting. It turns out that writing a book—at least a technical book—is not a way to riches. It may be a way to fame... maybe... but it’s certainly not a way to riches. So, what allows me to do this—what allows me to do this show, this software development show every week, and what allows me to write this book—is the people who hire me for training and consulting.

People who have the capability for a lot of business agility, but don’t have the capacity, hire me for anything from software development training, like test-driven development training, all the way up to process design and how to get teams working together smoothly. Because when you have a large environment, teams often form bottlenecks. And that is a process problem, it’s a social problem, it’s also a technical problem. And I help out with these things.

Okay, let’s get to the book. Again, to be notified when the Early Release is available, go ahead and sign up for the updates here. There’s an RSS feed; you can sign up for email updates; jamesshore.com/s/updates.

While I’m reading the excerpt, go ahead and put your comments in the chat. It’s going to take about 20 minutes to read, and then we’ll have the rest of the time—another 55 minutes or so from now—to discuss your comments and questions. So just put them all in and we’ll have a nice conversation after.

All right, let’s get to it. Part 1, Improving Agility, Chapter 1.

Chapter 1: What is Agile?

Agile is everywhere. And, paradoxically, nowhere.

In the 20 years after the Agile freight train roared into software developers’ conscious, the number of companies calling themselves “Agile” increased by orders of magnitude. The number of teams actually taking an agile approach to their work? Not so much. ”Agile,” the easily-repeated name, is enormously successful. The ideas behind Agile—well, most of them are ignored.

Let’s fix that.

Agile’s Genesis

In the 1990s, software development was believed to be in crisis. They actually called it that: “The Software Crisis.” Software projects were over-budget, late, didn’t meet requirements, and—according to the oft-quoted and ominously named “CHAOS Report”—a third of them failed outright.

Agile wasn’t a response to this crisis. Far from it. Agile was a response to the response.

To bring software development under control, big organizations created highly detailed processes that defined exactly how software was to be created. Everything was tightly controlled so that no mistakes could be made. (In theory, anyway.)

First, business analysts would interview stakeholders and document the system requirements. Next, software architects would read the requirements documents and create detailed design documents specifying every component of the system and how they related to each other. Then programmers would convert the design documents to code. In some organizations, this was considered low-skill work—just a mechanical translation exercise. In fact, programming was considered such a mechanical activity that, for a while, there was a movement called CASE—Computer Aided Software Engineering—to automatically convert architectural diagrams into code. Meanwhile, test leads would use the same documents to generate test plans, and when coding was done, armies of QA personnel would manually follow those test plans and report variances as defects. After each phase, everything would be carefully documented, reviewed, and signed off.

This approach was called “waterfall development” or “phase-gate development.” If it sounds like a ridiculous straw-man, well, consider yourself fortunate. Not every company used a heavyweight process in the ’90s, but it was widely recognized as a logical and sensible way to work. Of course you needed to define requirements, then design, then implement, then test. Of course you needed to document every phase. This was discipline. This was engineering. How else could you possibly succeed?

Born Out of Crisis

Big companies defined their processes in excruciating detail. Roles, responsibilities, document templates, change control boards... every aspect of development was defined and controlled. If a project didn’t succeed—and according to the CHAOS Report, two-thirds of them didn’t—it was because the process needed more detail, more documents, more sign-offs. It resulted in a massive amount of documentation. Martin Fowler called it “The Almighty Thud.”

This wasn’t a great way to work. It was bureaucratic and dehumanizing. Skill didn’t seem to matter as much as adherence to process. Programmers felt they were interchangeable cogs in an impersonal machine. It didn’t even work all that well.

So several people independently developed simpler, slimmer, and less prescriptive methods for developing software. They were called ”lightweight methods” in contrast to the heavyweight methods used by big companies. These new methods had names like “Adaptive Software Development,” “Crystal,” “Feature-Driven Development,” “Dynamic Systems Development Method,” “Extreme Programming,” and “Scrum.”

By the late ’90s, these methods were attracting serious attention. Extreme Programming, in particular, saw an explosion of grass-roots interest among programmers. In 2001, 17 of the lightweight methodology proponents met at a ski resort in Utah to discuss unifying their efforts.

The Agile Manifesto

“I personally didn’t expect this particular group of people to ever agree on anything substantive,” Alistair Cockburn said later.

And, in fact, after two days, they only accomplished two things: the name “Agile,” and a statement of four values. Over the following months, over email, they hashed out twelve accompanying principles.

This was the Agile Manifesto. It changed the world. So, as Alistair said later, they did agree on something substantive after all.

But there was no unified Agile method. There never has been, and never will be. Agile is three things: the name, the values, and the principles. That’s it. It’s not something you can do. It’s a philosophy. A way of thinking about software development. You can’t “use” Agile or “do” Agile... you can only be Agile. Or not. If your teams embody the Agile philosophy, then they’re Agile. If they don’t, they’re not.

The Essence of Agile

Martin Fowler has made a career out of turning complicated software topics into well-considered, even-handed explanations. His explanation of “The Essence of Agile Software Development” is one of the best:

Agile Development is adaptive rather than predictive; people-oriented rather than process-oriented.

Martin Fowler

Adaptive rather than predictive

Remember the CHAOS Report, which said that only one-third of software projects were successful? It had a very specific definition of success:

Successful
“Completed on time, on budget, with all features and functions as originally specified.”
Challenged
“Completed and operational but over budget, over the time estimate, [with] fewer features and functions than originally specified.”
Impaired
“Cancelled at some point during the development cycle.”

These definitions illustrate the predictive mindset perfectly. They’re all about conformance to plan. If you did what you said you were going to do, you were successful. If you didn’t, you weren’t! Easy.

It makes sense at first. But look closer. There’s something missing. As Ryan Nelson commented in CIO Magazine:

Projects that were found to meet all of the traditional criteria for success—time, budget, and specifications—may still be failures in the end because they fail to appeal to the intended users or because they ultimately fail to add much value to the business.

Similarly, projects considered failures according to traditional IT metrics may wind up being successes because despite cost, time or specification problems, the system is loved by its target audience or provides unexpected value. For example, at a financial services company, a new system... was six months late and cost more than twice the original estimate (final cost was $5.7 million). But the project ultimately created a more adaptive organization (after 13 months) and was judged to be a great success—the company had a $33 million reduction in write-off accounts, and the reduced time-to-value and increased capacity resulted in a 50 percent increase in the number of concurrent collection strategy tests in production.

Agile teams define success as delivering value, not conforming to a plan. In fact, truly Agile teams actively look for opportunities to increase value by changing their plans.

Take a second look at the Manifesto (see figure 1 and figure 2). “Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.” How many of its values and principles relate to delivering valuable software and adapting to feedback?

(To camera) You can see the Agile Manifesto at agilemanifesto.org.

People-oriented rather than process-oriented

Heavyweight processes tried to prevent errors by carefully defining every aspect of software development. By putting the “smarts” in the process, individual skill became less important. In theory, you could apply the same process over and over, with different people, and get the same results. (Come to think of it, they kind of did. Just not the results they wanted.)

Agile says people are the most important factor in software development success. Not just their skills, but all aspects of their humanity. How well team members work together. How many distractions they encounter. How safe they feel. Whether they’re comfortable voicing dissent, and whether they feel motivated by their work.

Agile teams have a process—every team does, even if it’s implicit—but the process is in service of the humans, not the other way around. And Agile teams are in charge of their own process. When they think of a better way of working, they change it.

Look back at the Manifesto (again, agilemanifesto.org). “Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.” Which values and principles relate to putting people first?

Why Agile Won

In the first ten years after the Manifesto, Agile faced enormous criticism. It was “undisciplined,” critics said. “It could never work.” Another ten years after that, the critics were silent. Agile was everywhere, at least in name. Heavyweight waterfall methods were effectively dead. Some younger programmers had trouble believing anybody ever could have worked that way.

There’s actually nothing inherently wrong with the basic waterfall process. If you keep it slim and operate in a well-understood domain, waterfall can work well. The problem was the heavyweight processes big companies used. Ironically, the processes designed to prevent problems actually caused many of the problems organizations were seeing.

With software, it’s very difficult to imagine how it will work before you actually use it, and it’s even harder to think of absolutely everything your software needs to do. This is doubly true for people who aren’t actively involved with software development. As a result, it’s critically important to get working software in front of people as soon as possible. You need to get feedback about what’s missing or wrong, then change your plans based on what you learn. As the Manifesto says, “working software is the primary measure of progress.” Learning and responding to change are at the heart of what Agile is all about.

Those heavyweight processes put so much emphasis on process controls and documentation and sign-offs that they incurred a huge amount of delay and overhead. They took years to produce working software, and they had nothing concrete to show until near the end. Instead of welcoming change, they actively worked to prevent change. They actually had a dedicated part of the process, the Change Control Board, whose primary purpose was to say “no” to change requests. (Or, more accurately, “yes, but it will cost you.”)

All of this added up to projects that spent years in development before they had anything to show. When they did, it was too late and too expensive to make changes. They ultimately shipped software that didn’t do what customers needed.

Although there are a variety of approaches to Agile—and some of them are more about co-opting a popular name than following the actual philosophy—one thing they all have in common is a focus on making progress visible and allowing stakeholders to make course corrections as they go. This seems like a small thing, but it’s incredibly powerful. It’s why we no longer hear about a Software Crisis. Software is still late. It’s still over budget. But Agile teams don’t spend years building failures. And that’s huge.

There’s more to Agile than just providing visibility. But this one thing? This was enough. It’s why everybody wanted Agile.

Cargo Cult Agile

At first, Agile was a grassroots movement. It was largely driven by programmers seeking better results and better quality of life. As it became more successful, Agile’s momentum shifted from the underlying ideas to hype. Rather than saying, “let’s get better results by adapting our plans and putting people first,” organization leaders started saying, “Everybody’s talking about Agile. Get me some Agile.”

The thing is, there is no “Agile” to go get. It’s just a set of values and principles. There are specific Agile approaches, such as Extreme Programming and Scrum, that will tell you how to be Agile, but you still have to be on board with the underlying philosophy.

And for a lot of organizations, that underlying philosophy—adapting plans and putting people first—is really, really foreign.

The tragedy of the cargo cult is its adherence to the superficial, outward signs of some idea combined with ignorance of how that idea actually works. In the story, the islanders replicated all the elements of cargo drops—the airstrip, the tower, the headphones—but didn’t understand the vast infrastructure that enabled airplanes to arrive.

The same tragedy occurs with Agile. People want Agile’s Cargo: better results, more visibility, fewer business failures. But they don’t understand the underlying philosophy, and often wouldn’t agree with it even if they did. They want to buy Agile, but you can’t buy an idea.

What they can buy is the outward signs of Agile. Stand-up meetings! User stories! Tools! Consultants! There’s lots of stuff labelled Agile, and plenty of people eager to sell it to you. It’s often labelled as “enterprise-grade,” which is a way of saying “don’t worry, you won’t have to change.” Uncomfortable ideas like “adaptive planning” and “people-centric” are ignored.

And that’s how you get a cargo cult. All the activity; none of the results. The Agile part is missing.

“At my old company they wasted a huge number of man hours in meetings.”

“[Agile] cost an entire team (30+) people their jobs as they produced almost nothing for almost a year.”

“All [Agile] means is developers get shafted when the project changes... the day before delivery.”

“Its because of Agile there are sh—ty developers and sh—ty products being shipped.”

“I can’t take this Agile crap any more. It’s lunacy.”

comments about Agile from around the Web

The name Agile is everywhere. The ideas of Agile aren’t. It’s become self-perpetuating; for many, the only Agile they know is Cargo Cult Agile.

It’s time to fix that. In the rest of this book, I’ll show you how to apply Agile ideas for real. Keep an eye out for the Cargo Cult Agilists shown in the margin. They’ll show you what not to do.

Ready? Let’s go.

Discussion

So that is the first chapter of The Art of Agile Development. (Reads chat: “*claps*”) Thank you, UncleScientist.

So I think that hopefully gives you a bit of an idea of what we’re going for in this book: how the focus has changed from the first edition. The first edition, again, was really about exposing people to what Agile is and how to use it, and the second edition is for all those people who are already using Agile—or something they think is Agile—and talking to them about how they can make it better.

A lot of it is centered around the Agile Fluency® Model, which I developed with Diana Larsen. You can see that in the table of contents. Part 2 is “Focusing on Agility,” which corresponds to the Focusing zone of the Agile Fluency Model. Part 3 is about Delivering Reliably, which about the Delivering zone, more about technical practices. And Part 4 is about Optimizing Outcomes. That’s the Optimizing zone. It’s about how you can really be adaptive.

Now, we only have so many pages in the book. I’m trying to target about 400 pages total, and I’m already at risk of exceeding that. So the bulk of the book is going to be in parts two and three. We’re just going to have a little bit about adaptive planning and how you can take advantage of agility. Because in my experience, only about 5% of the teams I talk to are set up as an organization to even try for that zone.

(Reads chat: “That was a great intro. Hits so many good points.”) RegalRegex, thanks for the comment, I really appreciate it.

So, as I said, your comments and questions—now is a great time to have a discussion about what’s in the book, or just Agile in general. I’m going to go back and look through some of the comments that were made while I was reading. Go ahead and put additional comments in the chat and I will take them as they come. I’ve got about 35 minutes left scheduled for today’s session. We don’t need to use it all, but we have the time if we want it.

So let me scroll back and see what people were saying...

GrumpyGameDev says, “I’ve written four books and the fame and fortune got lost in the mail somewhere.” Yeah, I think it’s in the same dead letter box my fame and fortune is. (laughs) JitterTed says, “I think your expectations may be off.” JitterTed, I refuse to believe that. (laughs)

Tomowens says, “I don’t like the equality between waterfall and phase-gate.” That’s an interesting point, tomowens. In practice, what I have seen is that phase-gate is a way of rebranding waterfall for a lot of companies. Because “waterfall” is obviously bad and wrong now—everybody knows this—so we’re going to do the same thing and call it something else.

So if waterfall is requirements analysis, and then architecture, and design, and then coding, and then QA/testing... often with the validation V, where we do the validation in reverse... phase gate is basically the same thing, but we focus more on the sign offs between phases, and maybe we have less iteration. (laughs) I’m maybe a little cynical about these things. If you disagree with that, would love to hear your comments.

(Reads chat: “It’s only the intro, but I’m ready to buy.”) Tomowens, thanks for saying that. So, if you do want to buy the book, the Early Release is going to come out on O’Reilly in the next week or two. I will announce it right here, jamesshore.com/s/updates. That will give you announcements of all kinds of things that I’m doing, which is mostly focused on the book. Or you can just follow the #aoad2 hashtag on Twitter, which will be specifically for The Art of Agile Development.

Certification

Mrz_al_hun says, “Are you planning on mentioning certification mills?” Well, it’s funny you should mention that. Chapter 3, which is not done yet, so I can’t really share it... so we’ll just pretend I’m not sharing it here... (brings up IDE with chapter 3 text) You can’t see chapter 3, pretend there’s nothing to see here... Oh yeah, Invest in Learning. I’ll just read this real quick.

Agile practices take time to learn. Expect teams’ performance to dip by about 10-20% for about 2-3 months per zone that they’re learning. Less, with full-time coaching. Possibly more, if their learning is entirely self-directed. Delivering zone practices can take longer still, depending on the nature of the team’s codebase.

You can get faster results if you hire people to help you. There’s a wide variety of help available, ranging from occasional mentoring, to training, to help with process design and implementation, to full-time (or near-full-time) coaching. The most effective help you can get is to hire experienced people to join your teams full time and coach by contributing to your team’s real-world work.

Ignore the myriad Agile certifications. Although some are connected to excellent training courses, that’s entirely dependent on the trainer, not the certification. Don’t use Agile certifications for hiring, either; they have no teeth. Most demonstrate nothing more than the ability to connect butt to chair for a few days. Instead, do your due diligence: ask your network for recommendations, sample publicly-available materials, and ask for references.

That’s pretty much the only thing we have to say about certification.

Waterfall

GrumpyGameDev says, “I used to work at GE... they crossed off ‘waterfall’ on the label, and wrote ‘agile’ in crayon over it. (laughs) Really, it was ‘iterative waterfall.’” You know, iterative waterfall is a great method... well, it has potential to be a great method. The really big companies tend to put way too much process around it. But Spiral, which was basically iterative waterfall, or Evo, by Tom Gilb, which is not really iterative waterfall, but its own thing, and really a sort of proto-Agile method... they were all about learning and taking advantage of that learning as you went.

But iterative waterfall or Spiral is still not Agile. Agile is really characterized by its strong emphasis on adaptive planning and people-centricity. And iterative waterfall, although it is iterative, it’s not adaptive in the same way that Agile is, and it doesn’t have the people-centric part at all.

I haven’t studied Evo, Tom Gilb’s stuff, enough to say how it fits in, but I have actually been in correspondence with him in the last couple of weeks. He feels that his stuff is really very much in the Agile mindset and I see no reason to doubt him.

JitterTed says, “Nobody ever really did waterfall anyway, it was more phase-gate or just ad-hoc.” You know, waterfall is a sort of poorly-defined term. Some people say it’s Royce’s paper that originally defined waterfall, but as far as I know, he was just documenting what people were already doing.

JitterTed basically says what I just said, “Mainly because waterfall is not a defined process, very few people actually read the Royce paper where it originated.” I’m not sure that it originated there, JitterTed, and I don’t know if it was that influential at the time. Laurent Bossavit has a fantastic book—he published it on LeanPub—I strongly, strongly recommend it. It’s called Leprechauns of Software Engineering.

Leprechauns of Software Engineering is excellent. Laurent goes through and he looks at some of the commonly-cited papers that people constantly quote, and he goes back and finds the source material, and finds what they really said, and it’s often not what people say that it said. Anyway, he discusses the Royce paper. Really, really highly recommend it. You can find it on LeanPub.

The Cover and Lost Subtitle

Book cover for “The Art of Agile Development, Second Edition” by James Shore and Shane Warden. Published by O'Reilly. The cover has a large sticker on it that says “Early Release: Raw and Unedited.” The cover artwork shows a water glass containing a small sapling. The sapling has small green leaves. There is a goldfish in the glass.

Second Edition cover

Paulhenman asks, “Why is there a fish in it?” You know, that is a good question, and I thought this might come up. So I prepared a little bit to explain that.

(Shows picture of first edition cover, which shows a large water glass with a twig or sapling in it.) The first edition design was by Karen Montgomery, and I really love the design. It’s quite striking. People always ask us, “why (laughs) why do you have a water glass with a twig in it?” And the answer is maybe not that interesting, but I’m going to tell you anyway.

(Shows two cover images. One has jars of sand, the other is a plastic paint pallete with watercolors.) This is the first set of cover proposals we got from our editor, Mary Treseler, back in March 2007. The book was done at this point and it was just going through the final development process, which took about six months. And she said, “here’s four covers for you to consider. (Shows two more cover images. One has wine glasses overlaid with a mess of orange lines and the other has scraps of paper overlaid with circular bursts of lines and dots. Each cover has the title, “The Art of Agile Development: with Extreme Programming”)

Shane and I looked at these, and... we were not a fan. (laughs) Of any of these book covers. Now, The Art of Agile Development is part of O’Reilly’s “/Theory/In/Practice” line, and they all had sort of these real-world covers with random dots and lines on them.

So we went back to Mary and we said, “You know, we don’t really love any of these covers. Can we have some more options? And maybe something prettier?” (laughs) We sort of said, “this is what the book’s about, and this is what we’d like the cover to convey—adapting your plans, and so forth—and failing that, could it at least be pretty.”

(Shows two more cover images. One is the current book cover and the other is similar, but with a squiggly line on top.) And she came back to us with a second round of covers, and the cover we have now was actually the first option, and then there was one with a random squiggle and a dandelion head on it.

(Shows two more images.) And then we had this one, which I actually rather like, with a growing something—sprout? And then a version of that sprout with a random squiggle with a dandelion head. (Two more images) The water one with little wavy lines—I think that one’s really pretty, actually. Stairways, with a step. We definitely did not want that, because that would sort of imply, you know, the waterfall, except in reverse. (Two more images.) A random ball of rubber bands, and another random ball of almost rubber bands.

JitterTed asks, “‘With Extreme Programming’ was originally the subtitle?” Yeah, we actually had “with Extreme Programming” in the subtitle right up towards the end. But this was in 2007. Extreme Programming had basically fallen off the radar. And so, even though the book chose Extreme Programming as its method, we ultimately decided to take that off the subtitle because we decided we wanted this to be more a general-purpose Agile book that used Extreme Programming rather than being an Extreme Programming-specific book.

The actual original book—the very, very first intention—was to take chromatic (Shane Warden)’s book, which was the Extreme Programming Pocket Guide—the original intention was to be a little second edition to the Extreme Programming Pocket Guide. But O’Reilly, by then, had stopped doing Pocket Guides because they got lost on the shelf and didn’t sell well. And, as we got into it, I realized that I had a whole heck of a lot to say about Agile.

So the little tiny pocket guide that we had decided to do had turned into this massive book, which was still a lot about Extreme Programming, but was also about wider Agile development as well. So we ultimately took the subtitle off.

(Laughs) GrumpyGameDev says, ”Is there a ‘cover must be uninspiring’ mandate from the publisher?”

So these were the options that we got. (Shows one final image) This was the last one. It was a lamp with sort of wavy lines coming out of it. The design department said, “I think the designs speak for themselves, so I won’t bore you with ‘designer rationale’ unless you really want it.” So why did they choose this cover and these options? I don’t know.

What we came back and said, though, was, “We think #5 is prettiest [which is the waves], but felt that #1 would draw more attention on the shelf and conveys the spirit of the book slightly better.”

So at this point, this is where all your hopes and dreams about there being some deeper meaning behind the cover? This is where I as an author completely smash those hopes and dreams (laughs). But if you really want a reason for the cover—and the reason is, this is what we were given, and we chose one—but if you really want a reason: we did feel like the spirit of the book was adaptive planning. The sapling grows towards the light. So that is why we chose this cover.

(Laughs) CharliePancakes says, “This is like the ending to Lost all over again.” Yeah, I announced last week that I was doing the livestream today and Pelyt asked, “Why do you have a glass with a twig on the cover?” I said, “Well, I’ll explain it next week, but (laughs) you might be disappointed.”

So, for the second edition, the nice thing is, it’s 13 years later, and O’Reilly has a much different process. It’s much more, well, Agile. We write source code in AsciiDoc, and there’s the Early Release so you can actually see the book as it comes out. So the cover comes a lot sooner in the process. For the first edition, the cover came at the end; for this one, it’s coming pretty close to the beginning.

So my brief was, “I’d like to use the same cover, it’s really iconic, people recognize it, but I’d like there to be some growth, to sort of symbolize the second edition.” And the graphic designer, who I don’t have the name of, but I imagine will be credited in the final book, they did this beautiful job where they added a little bit of growth to the cover. A little green. And they added a fish, the goldfish, which... there is no particular meaning, but I just love the splash of color. I love the whimsy of it. And I think it’s fantastic.

(laughs) So that is the, perhaps disappointing, reason and explanation behind the cover.

JitterTed says, “I always liked the organic feel to Agile, that it grows towards something, rather than as a planned roadmap.” And yeah, that is why we chose that one over the waves. Even though we really liked the waves and the water—we thought it was super pretty—we did like that the twig could be seen as growing toward the light. So if you need a reason, that’s the reason.

Simongeering: “When did O’Reilly stop doing animals on covers?” They still do animals on covers, but there’s different lines. So the animal books are the programming reference books. And I think those are a lot less relevant today? People are getting their programming reference online, rather than getting printed books. And—I have no inside information—but I think they’re running out of animals (laughs).

It started out, they were using these Dover woodcuts from the 1800’s, I think because it was royalty-free. O’Reilly started out as sort of this scrappy upstart publisher. And you’ll notice that they’ve moved on to insects for a lot of their cover designs and I think it’s because they’ve run out of mammals. Don’t take my word for it. I don’t know.

Anyway, so this line—“/Theory/In/Practice”—and I think other lines, like the Head-First Series—it’s all these sort of fish-eye lens, looking down at people’s heads, with giant heads—I think they’ve come up with different styles for different lines. And this is the /Theory/In/Practice line, which is all real images with some sort of random lines on the cover.

GrumpyGameDev: “Upgrade: now with goldfish!” Yeah! It’s a brand-new book! Buy it! It’s got a goldfish. That’s all I got to say.

Providing Feedback

Okay, I’m going to go back up... again, keep putting your comments in. I’m loving the conversation. We’ve got about 15 minutes left before I’m going to call it a day.

So, let’s see, what do we have next up here...

Jwalter: “Is it a bad idea to report spelling errors here?” It’s not a bad idea, but I’m going to ignore them, because (laughs)—I guess I could actually go in and modify the source code. Sure! Why not! Go ahead and report spelling errors. I will update the actual source book in real time. We are going to have a proper editing process, and so that’s when the real errors will be fixed, but spelling errors are welcome as well.

The main thing I’m looking for from the Early Release process, though, and the open development, is feedback about the ideas and concepts. Does it ring true? Do you have additional material that you think should be included or that you think’s important? Is there something that we’ve said that you don’t agree with? Those are the kinds of things that we’re really looking for in the open development process.

And again, to be informed when we put up new chapters for your review, right there: jamesshore.com/s/updates, or check out the #aoad2 hashtag on Twitter.

Commitments & Deadlines

CharliePancakes says, “I think the biggest problem I see is that I’ve faced is that my team can be agile but the upper management still need estimates and budgets and milestones etc.” Yeah, that is so true. And in fact... man, I probably shouldn’t read this to you... but I’m going to anyway.

Chapter 3, which I’m still working on, pretty happy with, it’s talking about the investments that you need to make to be successful with Agile. The second part of this, “2. Get Stakeholder Buy-In”—it’s way too long, I’m going to have to cut it down—but I’m going to read a little bit of this.

Software teams have always had a contentious relationship with their stakeholders on the matter of deadlines. Sales and product management, in particular, want accurate forecasting (often called “estimating”). It’s understandable! Their jobs would be much easier if they could tell customers exactly what they’re going to get and when. It’s important for building trust, securing new customers, and conducting business planning.

No matter how much they want it, they usually don’t get it. This is a sore spot. But accurate forecasting requires things most companies can’t provide.

  1. Detailed plans that are fixed in advance. (This often results in shipping the wrong thing, as we saw in the first chapter.)

  2. Statistical modeling and probabilistic date ranges. (Stakeholders usually want simple answers.)

  3. No interference in teams’ forecasts. (Accurate forecasts are usually much bigger than stakeholders want, leading to them being “negotiated down” to politically-acceptable fantasies.)

Instead, companies put in increasingly elaborate processes to try to “control” their schedules. It takes a lot of time away from development and it usually doesn’t work. In the end, it all boils down to gut feel, lots of padding, and political maneuvering.

Agile don’t play that.

Agile gives you visibility and flexibility. It tells you what is, not what’s politically expedient. It shines the harsh light of reality on people’s hopes and dreams. In exchange, it lets you steer your plans so you can get the best results possible with the time and people you have.

That doesn’t mean you won’t have roadmaps! You will. See [that section] for details. But it does mean that your roadmaps aren’t iron-clad commitments. They’re guides, not straight-jackets.

Some stakeholders love it. Finally, they know what’s really going on, and they have the ability to influence the results. Others, especially those who have been disappointed by software developers time after time, see Agile as just another political maneuver: a complicated way to avoid giving them what they need. They fight Agile tooth and nail.

You don’t need every stakeholder to buy in to the Agile way of planning. You do need the ones with a lot of political influence to do so. If they don’t, they could quietly sabotage your efforts, pulling the rug out from under you in six months to a year, even if—or especially if—Agile is succeeding.

Then I talk about how to deal with that. One thing I really like about this chapter is that is says, “this is what you need,” and then it says, over and over again, “well, if you can’t get that...”

If concrete commitments are required...

Agile teams can work to a deadline. They can even produce a detailed plan in advance. After development begins, their forecasts are typically more accurate than non-Agile teams’ forecasts, and they get better over time.

What Agile teams can’t do is guarantee a fixed scope (set of features) by a fixed date. Almost no one can. If you need certainty, you can plan on a fixed scope and ship whenever it’s ready; or you can plan on a fixed date and ship whatever’s done on that date. Not both.

That said, if your company requires fixed scope, fixed date releases, you can keep using whatever forecasting technique you’re using today. Agile won’t make it worse, and as development proceeds, it will give you visibility into how things are really working out. Predictive plans aren’t the Agile way, but there are more benefits to Agile than adaptive planning, and you can work your way up to true agility over time.

Do be careful. Some people like finding scapegoats, and some companies have a culture of assigning blame. Even if Agile doesn’t make anything worse, it might be convenient to blame Agile for the same deadlines that would have been missed before Agile. If that’s the case at your company, introducing Agile might be a mistake.

That’s what we have to say about “your team can be Agile, but upper management still need estimates and budgets and milestones.” You can have estimates and budgets and milestones, etc., and you can even have that in an Agile way, but it’s not going to be set in stone. If you want, you can do that, but it’s not super Agile. But you can be as successful with Agile at that as with anything else, which is usually... a lot of padding.

GrumpyGameDev says, ”It’s a challenge to be agile when you have to answer to shareholders on a quarterly basis.” It’s really not. Shareholders do not need that level of detail. You can tell them, “this is the direction we’re going,” without saying, “this is exactly what we’re doing and when we’re going to be done.”

JitterTed says, “That often results in AgileFall, the team is agile, but the environment is waterfall.” Yeah. And, again, you can do that.

Tomowens says, ”There’s something inherently wrong with the basic waterfall process. I can’t think of a software development effort that is in a well-understood domain. Some are more well-understood than others, but I would say that iterative and incremental development is almost always the right solution.” Yeah, I would agree with that, Tom.

That said, I do think there are some organizations that are—you see this, for example, in web agencies, digital agencies, that are doing a lot of marketing websites. The software behind it is pretty well established. There can be a little bit of custom development, but they’re basically doing the same thing over again, with different creative details, but not a lot of development challenge. There’s also companies that basically sell white-label software. They’ve got their software that they sell to big companies, and the big companies customize it. So they’re doing kind of the same thing over and over again.

(Scans for more comments) GrumpyGameDev: “#YouHadMeAtGoldfish.” (laughs) UncleScientist: “My only snobby comment is to not cite Wikipedia.” (jokingly) What? What’s wrong with citing Wikipedia? Don’t worry, we’re going to update it.

Zispro asks, “Will #noestimates be in the book?” Yes, yes it will. Right here (shows table of contents) in the Accountability chapter. It’s currently pencilled in. Whether or not it will stay as its own standalone part, or it will have something else, it will be in there. Part of the reason for that is that I believe Focusing teams are typically incapable of providing useful forecasts. That’s why we put the Forecasting section down in the Delivering zone part.

(laughs) JitterTed says, “Execs don’t like reality, from my experience.” You know... I’m of two minds on that. The work I do tends to be with senior software development leaders. They’re the ones that hire me. And I would say that the execs that hire me—which, as I’ve said in the past, are all above-average, highly intelligent, more-attractive-than-usual people (laughs)—they do like reality. In fact, one of the things they want, is they want to know reality soon, because they hate getting blindsided by mistakes or by problems. What they want to do is be able to understand it, they want to be able to respond to it.

But they’re often in a larger ecosystem where they’re working with a marketing department. They’re working with a sales department. They’re working with a product department. Often, I actually am hired by product leaders as well as engineering leaders, but sometimes when I’m hired by engineering leaders, the product people aren’t on board.

And that’s where we get back to that thing I mentioned in the book, in chapter 3, where some people see Agile and its reality as actually a way of telling them that they can’t have what they want. Salespeople particularly feel this. Because Sales is a completely different world from software development. Salespeople often see developers as kind of lazy. They’re slackers, is how they see it. And that’s because software developers tend to come in late. They might work late, but they tend to come in late. They tend to jump around a lot. Sales tends to be really, really concrete.

So there’s often a lot of friction between Engineering, and Sales, and sometimes that bleeds over to Marketing and Product. And I wouldn’t say that there’s a dislike of reality. It’s a dislike of everything that’s behind the unpleasant reality that we have to share. That’s a real challenge, and that’s part of the challenge of bringing in adaptive planning and real Agile.

Changes in Second Edition

Toltarius says, “Sorry if I’m late in asking this, but what are a summary of changes made over the first edition?” Thanks for asking, Toltarius. The second edition is a complete restructuring of the first edition. If you look at the table of contents, you’ll see that it’s completely different. And, if you want, the the first edition table of contents is also on my site, so you can compare the two if you want.

It’s been completely restructured. It’s following the Agile Fluency Model that I created with Diana Larsen. And it’s also targeting a different audience. The first edition was for that early majority of people who weren’t working in the innovator, early adopter companies, but who wanted to really bring in Agile. So, what is Agile, how does it work, it was a really practical, pragmatic way of doing Agile.

The second edition, also, really practical and pragmatic, but now we’re targeting sort of the late majority. Everybody’s heard of Agile at this point, they think they know what it is, now we’re going to talk to you about how to really do it, and how to really do it well.

So, that’s what it’s about.

Cargo Cult Agilists

Let’s see what else we have in the chat. Little bit about Royce’s paper, which I think we’ve talked about...

GrumpyGameDev: “Oooooh, Cargo Cult Agile, that’s the one we use at work!” Yeah, you and everybody else.

Oh, I need to show you all, just because I can: I don’t know if this will be what’s used in the final book, but I have made a little icon. (Shows icon of a person in a suit wearing coconut headphones.) Throughout the book, we’re going to occasionally put this Cargo Cult Agilist with a section on what not to do. And this is the icon that I’ve proposed that we use. Their graphic designers might actually produce something nicer.

Let’s see if I have an example of one of those Cargo Cult Agilist sections...

So that’s the Cargo Cult Agilist that’s going to show up throughout the book.

And... I tend to be a little cynical when I first write, so some of the edge may be taken off these... but maybe not. We’ll see.

C-Levels

We are almost out of time, but let’s see if there’s any other comments in here.

(Simongeering:) “Now all you need is global legislation to make every CTO and CEO required to read it, getting the ideas in the brains of the folks with the power to change an organizational culture is the problem. As with all hard things it is a people problem.”

You know, when I get hired, a lot of it is because somebody read my book, often a team lead or architect or something like that, and they bring me in and we go talk to the leadership, and then the leadership hires me. Because it’s easier to get them to hire me (laughs)—and I’m not that cheap—it’s easier to get them to hire me than to read a book.

(more seriously) And also, all cynicism aside, they’re more interested in getting the ideas throughout the company and hiring me will help them do that. And then, as part of my work, it’s bringing them on board and helping them understand their role in how to make this possible and sustaining it. Which, honestly, is not that hard. It’s partly because I come in from the outside and outsiders are always, sort of, more listened to than people who are inside the company. Familiarity breeds contempt. But also, it’s a matter of knowing how to talk to folks. How to put yourself in their shoes. And also focusing on what they care about.

It’s a really interesting topic, and I would love to talk about it, but we’re running out of time.

(reading more of the chat) A lot of cynical comments about C-levels, which, sorry, I can’t really agree with.

(NazarChops:) “Is a shift to Agile possible from the bottom up, or does it require Cs and VPs to drive the change?” Best is from both ends. You want support from the C’s and VP’s and you want bottom-up. Unfortunately, we’re out of time, but I have a whole section on that in Chapter 3. It will have to come some other time.

Final Thoughts

I think that’s most of the comments and it’s also 1:00 my time, so I think this is a good place to stop. Thank you all so much for listening. I really appreciate the interest. Had a good turn out today, so I’m just thrilled that there’s so much interest in the new book.

Again, if you’d like to be informed when the Early Release comes out, which should be in the next week or two—it is going through O’Reilly’s process right now—subscribe to updates. It’s free of course, and you can subscribe by RSS or email. jamesshore.com/s/updates, or you can follow my Twitter, @jamesshore. #aoad2 is the hashtag.

Next week, we’re going back to our normal software engineering scheduling. That’s going to be noon Pacific, August 25th, and we’re going to be talking about building a microservice client using Nullable Infrastructure.

Thanks so much, again, if you want to see more about the book, check out jamesshore.com/s/aoad2, that will take you to the table of contents and more information.

Thanks again! I will see you all next time.

Email Updates Now Available

In addition to my RSS feed, you can now get my latest material in your inbox!

Colophon

After 15 years, this site has finally received a long overdue design refresh. Here’s everything I do to make it a reality.

Production

I use a 13” MacBook Pro with a large external monitor to compose all of my essays. Years ago, my Windows laptop died and I decided to see if life was better on the other side. I’ve come to like MacOS, after some initial frustrations, and now I have a hard time imagining going back.

I write my essays in hand-crafted HTML using Webstorm. The files are served using a custom content management engine (CME) I originally wrote for my Let’s Code JavaScript screencast. It’s a polyglot engine, so I could theoretically use other markup languages, such as Pug, Markdown, or AsciiDoc, but I’m used to HTML and I haven’t bothered setting up the alternatives.

Everything required to make the site go, including all the content, are stored as static files in a git repository. I run my CME locally for testing, then push to my web host when I’m ready to go live.

I’m paranoid about backups and revision control. There’s decades of work here. That’s why all of my essays are written in a text editor and stored in git. Most blogging engines save your work in the cloud. Easy, but not fault tolerant. I’ve been writing for nearly two decades and plan to continue writing for several more. My stuff has to be easily scripted, version-controlled, backed up, and trivial to deploy to another web host when (not if) my host goes under... which it already has. Twice.

My site is backed up many times over. First, in the git repo on my laptop. Second, in a copy of the git repo on my web host. The computer itself is backed up to two redundant drives with Time Machine every hour. I also make a bootable whole-disk backup using SuperDuper! every day. Once per quarter, I rotate one of the backup drives to an off-site location.

Design

The site was originally designed in 2005. For 15 years, it had a very... “classic” look, and it didn’t work well on mobile. In 2020, I finally redesigned the site. I’m very happy with it, and particularly proud of its dark mode support and custom print styling1.

1Try it out! Click the “Print” button at the top of this page, or just tell your browser to print the page.

I’m not much of a graphic designer, so I took a page from great artists: I stole whatever I could. Dave Liepmann’s Tufte CSS, based on the work of Edward Tufte, was my principal inspiration. The home page was inspired by Michael “GeePaw” Hill’s website.2

2GeePaw Hill is super smart and thoughtful about Agile engineering practices. Go read his stuff.

The site’s fairly monochromatic, but there’s bits of color here and there. The colors are based on my Let’s Code JavaScript site, which was professionally designed by the talented folks at Primate.

Icons3 are in SVG, which I want to love, and probably will someday. After they make it better. I purchased my icons from The Noun Project. Their tagline is, “Icons for Everything,” which certainly seems to be true. I also use emojis as icons4 in a few spots, mostly because they’re so much easier to work with.

3 Subscribe icon Twitter icon Search icon Print icon

4 ⭐️ 📖

Fonts come from Adobe. I’m using Utopia for my main serif font, Cronos for my sans serif buttons and some links, and good old Courier for monospace—or Menlo, if you happen to have it installed.

Finally, lots of elbow grease got me to the result I have today. MDN, the Mozilla Developer Network, was invaluable for figuring out the style sheet. CSS has improved a lot since 2005, and I think I might even... like it? Well, almost.

Nearly all of the original design has been wiped away, but one I did keep one of my favorite aspects: printed URLs. I learned that from Eric Meyer’s “Going to Print” article.

Hosting

The site is hosted by Heroku and runs on a custom content management engine I wrote in Node.js. Logging is handled by Papertrail, which also sends me email and text message alerts when things go wrong. I use Pingdom as a backup alerting system. TierraNet makes sure that my domains point to the right place. Google handles my analytics. (Sorry about that.) I use DuckDuckGo for my site search and Google Groups for update emails.

This combination of custom source code and outside services gives me total control over the code behind the site while outsourcing a lot of the complexities of modern software stacks. It’s a nice tradeoff that allows me to focus my attention on more important things. The code is very well tested, so it almost never fails. Because there’s no database—everything’s in static files—and minimal service dependencies, my uptime is the same as Heroku’s, and requires no effort from me.

History

In the very beginning, the site was rendered by a very simple static site generator I wrote in Ruby. But for 15 years, from February 2005 to July 2020, the site was rendered by the ultra-minimalistic Blosxom. That’s when I started blogging in earnest; you can see my very first entry here.

Back in the day, Blosxom was the only blogging software I could find that actually allowed me to store entries locally, in files that I could back up and put in version control, rather than on a database on the server. (Nowadays, of course, static site generators are a dime a dozen.) Blosxom ran on Perl 5. It did the job, but my volume of content caused performance problems, and it had a lot of idiosyncracies. It was low maintenance, but also constrained what I could do. I’m happy to finally be rid of that technical debt.

Colophonem adidi.

Testable Logs

Every week in my Tuesday Lunch & Learn livestream, we choose a useful software development skill, define a challenge related to that skill, and solve the challenge live. This week, we’re building a testable logger.

The Log class we’re building this week is high-level infrastructure. High-level infrastructure is just like normal infrastructure, except that it uses code we’ve already written rather than talking to the outside world directly. So for today’s episode, we’re tying together a lot of the concepts from previous episodes. We end up with a lovely utility that makes testing logging a breeze.

To follow along, download the code from GitHub and check out the 2020-08-11 tag. To see the final result, check out the 2020-08-11-end tag or view it on GitHub.

Visit the Lunch & Learn archive for more.