AoAD2 Practice: Forecasting

Book cover for “The Art of Agile Development, Second Edition.”

Second Edition cover

This is a pre-release excerpt of The Art of Agile Development, Second Edition, to be published by O’Reilly in 2021. Visit the Second Edition home page for information about the open development process, additional excerpts, and more.

Your feedback is appreciated! To share your thoughts, join the AoAD2 open review mailing list.

This excerpt is copyright 2007, 2020, 2021 by James Shore and Shane Warden. Although you are welcome to share this link, do not distribute or republish the content without James Shore’s express written permission.

Revised: July 15, 2021

Forecasting

Audience
Product Managers

We can predict when we’ll release.

“When will you release?”

Programmers dread this question. Software development has so many details, it’s impossible to know exactly what’s left to be done, let alone how long it will take. Yet stakeholders have a genuine need to know how long the work will take. They need to plan budgets and coordinate with third parties. To create trust and show accountability, you need to be able to predict when you’ll release.

Making these predictions is usually called estimating, but that’s a misnomer. Estimating is just one technique for making predictions, and not even the most important one. The real secret to predicting well is to understand uncertainty and risk. That’s why I call it forecasting instead.

Uncertainty and Risk

Allies
Capacity
Slack

At first glance, Agile offers the perfect solution to forecasting: if you’re using iterations, add up your stories (or their estimates), divide by your capacity, and voila! The number of iterations left before you’re done. After all, capacity and slack give you the ability to consistently make and meet iteration commitments. Shouldn’t that mean you can make reliable release commitments, too?

Unfortunately not. Imagine you’re on a team with 30 stories to finish before release. Your team consistently finishes six stories every week, so it will take five weeks, right? It’s January 1st, so you tell your business stakeholders you’ll release five weeks later, on February 5th. They’re enthusiastic about the new release and start telling customers. “Wait until you see what’s next! Expect it February 5th!”

Over the next week, you finish six stories, as usual. Along the way, you discover a bug. It’s not a big deal, but it needs to be fixed prior to release. You add a story to fix it in the next iteration. On January 8th, you have 25 stories remaining. You tell your stakeholders that you might be a bit later than February 5th. They urge you to speed up a bit. “Just squeeze it in,” they say. “Our customers are counting on a February 5th release!”

On January 15th, during your stakeholder demo, your stakeholders realize one of the features needs more audit controls. You add four new stories to address the need. Combined with the six stories you finished, there are 23 stories remaining, which means you definitely won’t be done on February 5th. You propose cutting a feature to bring the date back in, but stakeholders balk. “We’ve already told customers what to expect,” they say. “We’ll just have to tell them there’s been a week’s delay.”

The next week, everything goes smoothly. You finish six stories again, and on January 22nd, there are 17 stories remaining. You’re on track for releasing on February 12th.

The next few weeks don’t go as well. You’ve been waiting for another team to deliver a special UI component. They promised to have it to you in early January, but their date kept slipping. Now you’ve run out of other stories to work on. You pull in some extra “nice to have” stories to keep busy. You finish six stories, as usual, but most of them are new. On January 29th, you still have 15 stories remaining.

Then the team working on the UI component comes clean: they’ve run into an unexpected technical issue. The UI component you’ve been counting on isn’t going to be ready for another month, at minimum. You revise your plan, adding stories to work around the missing component. On February 5th, despite finishing six stories, you still have 13 stories to go. Your stakeholders are getting frustrated. “We’ll push the release out one more week, to February 19th,” they say. “You’ll just have to squeeze in that last story. And we can’t keep slipping the date! We’re getting eaten alive on Twitter.”

The next two weeks are Not Fun. You keep discovering new stories to make up for the missing UI component. Everybody works overtime to try to hit the release date, and you cut back on testing and slack. You ignore your capacity and sign up for more stories on the assumption that the extra time will work out.

It doesn’t work out. At first, everything seems fine. On February 12th, you finished nine stories! But the next week, you learned that four of them had to be reworked because of bugs and missed assumptions. Combined with all the new UI stories, there’s just too much to do. When the 19th rolls around, you still have four stories left.

Finally, the following week, you release. It’s February 26th. You never finished fewer than six stories per week. But somehow, it took eight weeks to release the 30 stories in your original plan. These sorts of delays are called schedule risks.

Predefined Release Dates

The best way to forecast is to define when you’ll release, but not what you’ll release.

Schedule risks are unpredictable. If you could predict them, they wouldn’t be risks—they’d be part of your plan. They’re also unavoidable. So the best way to forecast is to define when you’ll release, but not what you’ll release. That way, you can adjust your plans when you encounter surprises. In the example, if stakeholders hadn’t told customers exactly what to expect, the team could have cut features and still released on time.

Telling people what and when you’ll release also reduces your agility. Agility involves seeking out new information and changing your plans in response. If you tell people exactly what you’re going to do, you’ll have to change your forecast every time you change your plan. At best, this means the time and effort that went into the previous forecast was wasted. More often, people treat your forecasts as commitments, and get upset when you change them.

Ally
Adaptive Planning

Instead, only forecast your release date. Steer your plans so you’re ready to release your most valuable increments on that date. That way, you can release on time, no matter you’ve finished. A common variant of this idea is the release train, which is a predefined series of release dates. (See “Release Early Release Often” on page XX.)

How to steer your plans

The secret to being ready on a predefined release date is to slice your work into the smallest valuable increments you can. Focus on getting to a releasable state as quickly possible. To do so, set aside every story that isn’t strictly necessary to release.

That bare minimum is your first increment. Once you’ve identified it, take a look at the stories you set aside and decide which can be done on their own, as additional increments. Some of those increments might be as small as a single story—in fact, that’s the ideal. In a perfect world, you want every story to be something that can be released on its own, without having to wait for any additional stories. This gives you the maximum flexibility and ability to steer your plans. “Keep Your Options Open” on page XX has more details.

At a minimum, your increments need to be small enough that you can easily finish at least one before your release date. As you finish work, keep an eye on how much time is left and use it to decide what to do next. If there’s a lot of time, you can build a big new increment that takes the software in a new direction. If there isn’t much time left, focus on smaller increments that add polish and delight.

You may be able to judge increment size based on gut feel. If you need more rigor, use a temporary date and scope forecast (described later) to see what will fit, but don’t share those forecasts. That will give your team the flexibility to change their plans later.

Feasibility Forecasts

Sometimes you just want to know if an idea is worth pursuing, without the time and expense of detailed planning. Any approach that doesn’t involve detailed planning will just be based on gut feel, but that’s okay. People with a lot of experience can make good gut decisions.

To make a feasibility forecast, gather the team’s sponsor, a seasoned product or project manager, and a senior programmer or two—preferably ones that will be on the team. Choose people with a lot of experience at your company.

Ask the sponsor to describe the development goals, when work would start, who would be on the team, and the latest release date that would still be worth the cost. Then ask the product manager and programmers if they think it’s possible.

Note that you aren’t asking how long it will take. That’s a harder question to answer. What you’re looking for here is a gut reaction. Phrasing the question in terms of a solid expectation makes the gut reaction more reliable.

If the answer is an unqualified “yes,” then it makes sense to invest in a month or two of development so you can make a real plan and forecast. If the experts waffle a bit, or say “no,” then there’s some risk. Whether or not that risk is worth investing in a better forecast is up to the sponsor.

Date and Scope Forecasts

Although it’s best to predict when, but not what you’ll release, sometimes you need to forecast both. Doing so accurately requires accounting for schedule risks. You’ll essentially add some padding, called a risk adjustment, to absorb problems. Here’s the formula:1

1Many thanks to Todd Little for his feedback on this technique.

number of weeks remaining = number of stories (or estimate total) remaining ÷ weekly throughput × risk adjustment

You can also forecast how many stories will be done by a predefined release date:

number of stories (or estimate total) done = number of weeks remaining × weekly throughput ÷ risk adjustment

Here’s what each of those terms means:

  • Number of weeks remaining: The amount of time between now and your release date.

  • Number of stories (or estimate total): The number of “just right” stories that need to be completed before release, or that will be done by your release date. If you use estimates, it’s the sum of those stories’ estimates.

  • Weekly throughput: If you use iterations, it’s the number of stories finished last iteration (or the sum of their estimates) divided by the number of weeks per iteration. If you use continuous flow, it’s the number of stories finished last week (or the sum of their estimates). Don’t average multiple iterations or weeks; that’s taken care of by the risk adjustment.

  • Risk adjustment: See table “Risk Adjustment Rules of Thumb”.2 Use the “high-risk team” column unless your team has both Focusing and Delivering fluency. Choose the row corresponding to your desired likelihood of meeting or beating the predicted date. For example, forecasts made using the “90%” row will meet or beat the predicted date nine out of ten times.

2These risk numbers are an educated guess. The “high risk” numbers are based on [Little 2006] and subsequent conversations with Todd Little. The “low risk” numbers are based on DeMarco and Lister’s RISKOLOGY simulator, version 4a, available at https://systemsguild.eu/riskology. I used the standard settings but turned off productivity variance, as capacity automatically adjusts for that risk.

Table 1. Risk adjustment rules of thumb

LikelihoodLow-risk teamHigh-risk team
10% (almost impossible)11
50% (coin toss)1.42
90% (very likely)1.84
Ally
The Planning Game

Date and scope forecasts depend on stories that are sized “just right” using the planning game. If you haven’t broken all the stories in your release down to that level of detail, you won’t be able to forecast the release. You’ll need to use the planning game to size all your stories first.

Similarly, if the release includes any spike stories, you’ll have to finish all of them before you can make a forecast. This is why spike stories are separate stories in your plan; sometimes it’s valuable to schedule them early so you can resolve risks and make forecasts.

Update your forecast at the end of every iteration, or once per week if you use continuous flow. As your release date approaches, the forecast will “narrow in” on the actual release date. Graphing the forecasted release dates over time will help you see trends, especially if your throughput isn’t stable.

A date and scope example

Let’s revisit the example from the introduction. To calculate the number of weeks remaining, start with the number of stories remaining and the team’s throughput. As before, the team has thirty stories remaining and finishes six stories per week.

Next, determine the risk adjustment. I usually forecast a range of dates that’s 50-90% likely. It gives a relatively narrow range I can beat about half the time. If I don’t think my audience can handle a range-based forecast, I’ll use the 90% number alone.

The specific risk adjustment depends on whether the team is high risk or low risk, which depends on their fluency. In this case, let’s say the team is fluent in both the Focusing and Delivering zones, so they’re low risk. That gives risk adjustments of 1.4 and 1.8 for 50% and 90% likelihoods, yielding the following forecasts:

  • 50% Likely: 30 stories ÷ 6 stories per week × 1.4 risk adjustment = 7.0 weeks

  • 90% Likely: 30 stories ÷ 6 stories per week × 1.8 risk adjustment = 9.0 weeks

If the team makes this forecast on January 1st, they tell their stakeholders they’ll release “between February 19th and March 5th.” Every week, they provide an updated forecast, as follows:

  • Jan 1: 30 stories remain; forecast: Feb 19 - Mar 5. (7.0 - 9.0 weeks.)

  • Jan 8: 25 stories remain; forecast: Feb 19 - Mar 5. (5.8 - 7.5 weeks. I always round up.)

  • Jan 15: 23 stories remain; forecast: Feb 26 - Mar 5. (5.4 - 6.9 weeks.)

  • Jan 22: 17 stories remain; forecast: Feb 19 - Mar 5. (4.0 - 5.1 weeks.)

  • Jan 29: 15 stories remain; forecast: Feb 26 - Mar 5. (3.5 - 4.5 weeks.)

  • Feb 5: 13 stories remain; forecast: Feb 26 - Mar 5. (3.0 - 3.9 weeks.)

  • Feb 12: 9 stories remain; forecast: Mar 5. (2.1 - 2.7 weeks.)

  • Feb 19: 4 stories remain; forecast: Feb 26 - Mar 5. (1.0 - 1.4 weeks.)

  • Actual release: February 26th.

Although the team in the example encountered a lot of problems, the risk adjustment allowed them to release on time. Figure “Example Iterated Forecast” shows the forecast in graph form.

A two-axis line chart. The x-axis is labelled “Date forecast made” and shows dates, in one week intervals, from January 1st to February 26th. The y-axis is labelled “Forecasted release date” and shows dates, in one week intervals, from January 29th to March 12th. The body of the graph shows three lines, labelled “10%,” “50%,” and “90%.” On January 1st, they show a forecast ranging from February 5th, to February 19th, to March 5th. Moving from left to right, they gradually converge on a release date of February 26th.

Figure 1. Example iterated forecast

Reducing risk

High-risk teams have trouble making useful forecasts. Three months of stories yields a 50-90% forecast of 6-12 months. That much uncertainty is hard for stakeholders to accept.

The easiest way to reduce risk is to make your increments smaller. Instead of releasing three months’ worth of stories, release two weeks’ worth. That results in a 50-90% forecast of 4-8 weeks. Not too bad.

The harder, but more effective way is to improve your development practices. You don’t actually need to have perfect Focusing and Delivering fluency to use the “low-risk” column of the risk adjustment table. You just need to be able to answer each of the following questions in the affirmative:

  • Did you have the same throughput in each of your last four iterations? (Or, if you’re using continuous flow, did you finish the same number of stories in each of the last four weeks?)

  • If you use iterations, were all of your stories in the last four iterations “done done?”

  • Did you add no new bug-fix stories in the last four iterations (or weeks)?

  • For your most recent release, when your stories were done, were you able to release to production immediately, without additional work, waiting for QA, or other delays?

Allies
Slack
Test-Driven Development
Continuous Integration

Address the first two questions by introducing slack, as described in “Stabilizing Capacity” on page XX. To address the last two questions, introduce Delivering zone practices, starting with test-driven development and continuous integration.

Custom risk adjustments

The risk adjustments in table “Risk Adjustment Rules of Thumb” are just educated guesses. If you want to be more accurate—and possibly less pessimistic—you can create your own risk adjustment table. It takes a lot of data, though, and may not be worth the effort.

To create your own risk adjustment table, you’ll need historical release estimates. Every week, or iteration, make a baseline release estimate. The baseline release estimate is “weeks remaining = stories remaining (or estimate total) ÷ weekly throughput.”

Then, when the release actually happens, go back and calculate how long, in weeks, the release actually took from the date of each estimate. If you were pressured to release early, or had a lot of bugs or hotfixes, use a release date that represents your real release—when the software was actually done—so your forecasts will represent the time your team actually needs to create a viable release.

You should end up with several pairs of numbers: an estimate, in weeks, and the actual time required, also in weeks. Divide the actual by the estimate to get an actual/estimate ratio for each pair. Table “Historical Release Estimates” shows an example.

Table 2. Historical release estimates

DateBaseline estimateActualActual/Estimate
Jan 15.00 weeks8 weeks1.60
Jan 84.17 weeks7 weeks1.68
Jan 153.83 weeks6 weeks1.57
Jan 222.83 weeks5 weeks1.76
Jan 292.50 weeks4 weeks1.60
Feb 52.17 weeks3 weeks1.38
Feb 121.50 weeks2 weeks1.33
Feb 190.67 weeks1 week1.50
Feb 26(Release)

Finally, sort the ratios from smallest to largest. Add another column with the position of each row as a percentage. In other words, if you have eight rows, the percentage for the first row would be “1 ÷ 8 = 12.5%.” This is your custom risk adjustment table. The percentage is the risk likelihood and the actual/estimate ratio is the risk adjustment. Table “Example Custom Risk Adjustments” continues the example.

Table 3. Example custom risk adjustments

LikelihoodRisk adjustment
12.5%1.33
25.0%1.38
37.5%1.50
50.0%1.57
62.5%1.60
75.0%1.60
87.5%1.68
100.0%1.76

The more release data you include, the more accurate the risk adjustments will be. For best accuracy, every team should track their data independently, but you can combine data from several similar teams to get started.

Questions

Our throughput changes a lot, so our forecast is constantly bouncing around. Can we average the throughput so it’s more stable?

Allies
Capacity
Slack

It’s best to use capacity and slack to stabilize your throughput. If that’s not an option, you can average the last three weeks (or iterations) to stabilize your forecast, or graph your forecasts and draw trend lines.

Our forecast shows us releasing way too late. What should we do?

You have to cut scope. See “When Your Roadmap Isn’t Good Enough” on page XX for details.

Your rule-of-thumb risk adjustments are too large. Can we use a lower ratio?

When your forecast gives you bad news, it’s tempting to play with the numbers until you feel happier. Speaking as somebody who’s been there and has the spreadsheets to prove it: this is a waste of time. It won’t change when your software actually releases. If you have historical data, you can make a custom risk adjustment table, but if you don’t, your best option is to face the unpleasant news head-on and cut scope.

Prerequisites

Predefined release dates and feasibility forecasts are appropriate for any team.

Ally
The Planning Game

To make date and scope forecasts, you need a team that’s working on the actual software being forecasted. You should have at least four weeks of development history, and you can only forecast increments with stories that have been sized “just right” in the planning game.

More importantly, make sure you really need a forecast. Too many companies ask for forecasts out of habit. Forecasting takes time away from development. Not just the time required to make the forecast itself, but the time required to manage the many emotional responses that surround forecasts, both from team members and stakeholders. It also adds resistance to adapting your plans.

As with everything the team does, be clear about who date and scope forecasts benefit, why, and how much. Then compare that value against the other ways your team could spend their time. Predefined release dates are often a better choice.

Indicators

When your team forecasts well:

  • You can coordinate with external events, such as marketing campaigns, that have long lead times.

  • You’re able to coordinate with business stakeholders about upcoming delivery dates.

  • You understand when your team’s costs will exceed its value.

  • You have data to counter unrealistic expectations and deadlines.

Alternatives and Experiments

There are many approaches to date and scope forecasting. The one I’ve described has the benefit of being both accurate and easy. However, its dependency on “just right” stories makes it labor-intensive for pre-development forecasts. Another downside is the amount of historical data required to use custom risk adjustments, although the rules of thumb are often good enough.

An alternative is to use Monte Carlo simulations to amplify small amounts of data. Troy Magennis’ “Throughput Forecaster” spreadsheet at https://www.focusedobjective.com/w/support/ is a popular example.

The downside of Magennis’ spreadsheet, and similar estimating tools, is that it asks you to estimate sources of uncertainty rather than using historical data. For example, Magennis’ spreadsheet asks the user to guess the range of stories remaining, as well as a range of how many stories will be added (or “split,” to use its terminology). These guesses have a profound impact on the forecast, but they’re just guesses. The spreadsheet would be stronger if it used actual data for scope increases, rather than guesses.

Before you experiment with alternative date and scope forecasts, remember that the best way to forecast is to pick a predefined release date and steer your plans to meet that date exactly.

Further Reading

XXX Further reading to consider:

  • Agile Estimating and Planning (Cohn)

  • Software Estimation: Demystifying the Black Art (McConnell)

  • Software Estimation Without Guessing (Dinwiddie)

  • When Will It Be Done? (Vacanti)

Share your feedback about this excerpt on the AoAD2 mailing list! Sign up here.

For more excerpts from the book, or to get a copy of the Early Release, see the Second Edition home page.

If you liked this entry, check out my best writing and presentations, and consider subscribing to updates by email or RSS.