Hi there.

This is the epilogue to my series on assessing the impact of AI-assisted software development. It brings all the concepts together into one whole. It’s also good for sharing with your CFO and other members of your leadership team.

You can find the beginning of the series here: Are You Getting Your Money’s Worth from AI? The rest of the articles are linked below.

Comfy? Great. Let’s get started.

Give Me the Shortest Possible Version

AI-assisted software development is expensive, and while the potential returns are high, they’re not yet proven. The field is changing rapidly, so learning the right approach requires experimentation and measurement. Spending varies by an order of magnitude: from hundreds of dollars per person per month, to thousands. To know which approach is best, you need to measure impact as well as costs.

The fullest view involves modeling the ROI of your software initiatives, which requires leadership support and cross-departmental participation. For a faster and more targeted view, ask your engineering leaders to measure changes in software development speed. But don’t just look at what you’re getting now; consider the long-term effects of unintended consequences and the possibility of lock-in.

Say More About AI Program Fundamentals

Measuring the costs of an AI-assisted software development program is fairly easy, but measuring benefits is harder. It’s tempting to use easy, shallow approaches that don’t consider the full picture.

Some token costs are heavily subsidized, so prices could go up. Examples include subscriptions with generous token allowances and temporary promotions. On the other hand, competition could drive prices down. As you forecast ROI, consider optimistic and pessimistic cost scenarios.

Consider that the fastest approaches might not be the most cost-effective. The best investment might involve lower per-person delivery speed, but much lower token costs, especially if the money for AI comes at the cost of headcount.

Read more about why you should measure impact, and what you shouldn’t measure, in part 1.

What Was That about Modeling ROI?

Most organizations primarily think in terms of cost and predictability, with minimal attention to value. Modeling ROI will not only help you understand AI’s impact, it will give you a better understanding of your software development efforts in general.

Product development can create a model that calculates an initiative’s net present value. This requires cross-departmental participation, backing from Leadership and Finance, and some educated guesses. It will never reflect real value perfectly—it involves too many guesses and assumptions—but it makes those assumptions concrete and focuses people on what you’re getting for your money.

Read more about modeling software’s business impact in part 4.

I Just Want to Know If AI Is Faster

Engineering leaders can get a narrower read on AI’s impact by measuring how it changes software delivery speed. They can also compare one approach to another, which will allow them to find the optimal blend of speed and costs for your company.

Simplistic speed measurements abound. When evaluating claims, look for evidence of sophistication, such as randomized comparisons, and look for sustained results across numerous teams. Beware of measurements that simply count deliverables or changes in “velocity.” They’re not standardized.

The pressure to demonstrate results from AI is immense, and the stories of success that are shared in conferences and executive circles grow with each retelling. Calibrate your skepticism accordingly. Listen for how the storytellers measured impact. Look for evidence of sustained results, not cherry-picked examples.

Read how to measure changes in delivery speed in part 2.

What About Unintended Consequences and Lock-In?

Speed is only part of the picture. If AI makes software development faster, but increases cost of change, it could end up costing you more than it saves. Lock-in is a risk, too: If you’re locked in, and AI token costs increase, you could be stuck with a choice between a big bill or cutting back on development.

Look for evidence that your team is considering unintended consequences. This guide recommends tracking cost of change over time. The data is noisy, so you should also look for models that explore scenarios around topics such as maintenance costs and turnover, as well as management of associated risks.

Read how to measure and model unintended consequences in part 3.

Be Careful of Measurement Dysfunction

Performance metrics always risk measurement dysfunction, and some of these metrics are particularly prone to misuse. Don’t establish a performance measurement program and leave it running. Instead, use these metrics to determine how well a particular approach to AI is working. Whatever you do, don’t use them to measure the performance of teams or individuals. Measure the approach instead, learn from it, and stop measuring.

Thanks for Reading

This concludes my series on assessing the impact of AI on software development. As you’ve seen, it’s a lot more complicated than just throwing some numbers up on a dashboard. As always with software development, you need to consider the system as a whole, and how changes here affect outcomes there.

Doing this right takes some work. Given the stakes, though—a substantial percentage of your budget, but also the potential for a substantial increase in productivity—it’s worth doing right.

This series is based on my work on introducing AI while I was VP of Engineering. Ultimately, I wanted to measure impact so I could make smarter decisions about how we used AI. If you’d like my help with your organization, send me an email. If you’d rather go it alone, this series will get you off to a good start. Either way, best wishes, and let me know how it goes.