James Shore

Testing Private Methods

19 Aug 2008

James Shore/Blog

One of my Mentor Program participants just asked me about testing private methods. Here's my response:

Generally, you should be able to test private methods by calling a public method in such a way that it calls the private method. Remember, when you're writing a private method, you're engaging in encapsulation and information hiding. You're saying "this method is an implementation detail, not an important part of the class's interface." And tests are supposed to document the public behavior, not the internal implementation. So testing a private method can be a sign of a problem.

Sometimes you really need to test a private method. The quick-and-easy solution is to make the method public, and I'll admit that I've done so on occasion. (Often, I'll mark it "for testing only!") But that makes your class's interface more confusing and reduces its self-documentation. It's harder to understand a class's responsibilities if there's no way to tell public behavior from implementation detail.

If you need to make private methods public, it may be a sign that your class has become too big and complex. Often these sorts of classes need to be split into multiple, smaller classes with more clearly defined responsibilities. Once you do that, private methods often become public, because now they're part of the interface that defines the class's behavior, not just an implementation detail.

Comments

29% Discount at Amazon

18 Aug 2008

James Shore/Blog

Amazon has bumped their discount for The Art of Agile Development from 14% to 29%, which is a pretty good deal. If you've been thinking about picking up a few extra copies for your team, now's a good time.

(Why the sudden change? I have no idea. The Amazonians are a mysterious tribe, impossible for mere mortals to understand, and not to be trifled with. The last anthropologist to venture into that trackless jungle emerged six months later, eyes bulging and brain addled, with nothing to say but a mindless repeated cry: "Tapioca! Tapioca!" Six servings later, he fell into a vegatative state, and nothing could be done except wipe off the pudding and place him in the care of Dr. Finkle's Sanitarium and Public Exposition (Some Exhibits Not Suitable for Small Children or the Faint of Heart). There he remains today.)

Comments

The Art of Agile Development: Continuous Integration

13 Aug 2008

James Shore/Agile-Book

in 99 words

Keep code integrated and build release infrastructure with the rest of the application. The ultimate goal is to be able to deploy all but the last few hours of work at any time.

To do so, integrate every few hours and keep your build, test, and other release infrastructure up to date. Each integration should get as close to a real release as possible.

Prefer synchronous integration, in which you wait for the integration to succeed, to asynchronous integration, in which a tool tests the integration for you. Synchronous integration requires fast builds, but ensures that they never break.

Inside The Book

Continuous Integration
Why It Works
How to Practice Continuous Integration
Never Break the Build
The Continuous Integration Script
- To update from the repository
- To integrate
Sidebar: Continuous Integration Servers
Introducing Continuous Integration
Dealing with Slow Builds
Multistage Integration Builds
Questions
- I know we're supposed to integrate at least every four hours, but what if our current story or task takes longer than that?
- What should we do while we're waiting for the integration build to complete?
- Isn't asynchronous integration more efficient than synchronous integration?
- Are you saying the asynchronous integration will never work?
- Our version control system doesn't allow us to roll back quickly. What do we do?
- We rolled back our check-in, but the build is still failing on the integration machine. What do we do now?
- Why do we need an integration machine? Can't we just integrate locally and check in?
- I seem to always run into problems when I integrate. What am I doing wrong?
- I'm constantly fixing the build when other people break it. How can I get them to take continuous integration seriously?
Results
Contraindications
Alternatives

Brain Dump: Forces Affecting Continuous Integration

Ultimately, continuous integration is about a very simple business idea:

Mr. Customer, you can release your software at any time and receive value from it reflecting your investment to date.

paraphrased from Planning Extreme Programming, by Jeffries and Fowler

From that core idea, we get the two rules of continuous integration:

The code in the version control repository always builds, passes its tests, and is ready to release.
The code in the repository represents all of our work up to a few hours ago.

It's not about integration servers, or automated builds, or any of that stuff. That's all incidental... technique, not essence. The essence is simple. Release the software at any time. Recoup your investment.

With that core idea firmly in mind, let's explore some of the forces that affect continuous integration. I have to warn you: this list is long, aimed at advanced readers, and not particularly well written. It's sort of a rambling brain dump.

Business Priorities

Ideally, the code in the repository is always ready to release. But even though we can make it technically capable of release (nearly bug-free, builds, passes tests), it's not always going to be ready to release from a business perspective.

There's two scales here. First, you have iteration cycles. During the iteration, the team's allowed to have the software in pieces. It still needs to be technically capable of release, but features can be half-done and bugs can be unfixed. Everything is supposed to be wrapped up by the end of the iteration--stories "done done" or removed, bugs fixed, and so forth.

Second, you have release cycles. Although all stories are supposed to be "done done" at the end of each iteration, those stories may not represent complete minimum marketable features (MMFs). So the software may not hang together well, or it may not have the polish necessary to succeed in the market, and so forth. That stuff is supposed to be wrapped up by the end of the release.

Really great teams don't have these problems. Their repository really is ready to release at any time. I think part of the reason is that they tend to be mature teams with a lot of experience... and they have a mature codebase, too, which allows them to add small features rather than big ones. It's a lot easier to release daily when you're improving existing features rather than creating a brand new piece of software from scratch.

That said, it is possible to practice real continuous integration even with a brand-new project. It requires that the continuous integration mindset extend to product planning as well as programming. In the past, I've described it as building a mansion by first building a doghouse... then stre-e-e-tching it out into a lean-to... then stre-e-e-tching it into a shed... then adding an out-house... and so on. That works really well in the Web 2.0 era, but wouldn't work so well for a company like Apple that likes to release polished, seamless obelisks.

Of all of the ways that continuous integration can be compromised, this is the one that I'm most comfortable with. Sometimes, it just doesn't make sense to release a piece of software because it isn't functionally ready. As long as release cycles are kept short and the team keeps producing live demos, I'm okay with that, for the most part.

Number of Concurrent Code Streams

By "concurrent code streams," I mean the number of development efforts you have going on at any given time. For example, if you have a classic XP environment with ten programmers, and they're all working, you'll have five code streams. The programmers will be working in pairs at five different workstations. If you aren't pairing, you'll have ten code streams.

Each of those code streams is, in fact, a short-lived branch of the code. In a distributed version control system, that fact is made explicit. In traditional version control systems, the branches are called "sandboxes," but they're still a sort of branch. When we integrate, what we're actually doing is merging the branch back in with the mainline.

The number of code streams is a force on continuous integration because the more code streams you have, the more likely you are to have non-trivial integrations. (A trivial integration is one where nobody else has checked anything in, hence nothing to integrate with.) In other words, more code streams means more integration effort and more potential for integration problems.

The number of concurrent code streams interacts with...

Frequency of Integration

The more frequently you integrate, the more likely you are to have a trivial integration, which is a good thing. Think of it this way. Imagine you and Bob the Schlub from down the hall are the only programmers. Bob integrates every two weeks and you integrate ten times a day. When you integrate, 99 out of 100 times, Bob won't have checked in and you'll only be integrating with yourself. (Kinky.)

Frequent integrations also makes the amount of code you have to merge smaller, too, which reduces the likelihood of conflict and makes your integrations easier. So, all in all, frequent integrations are a Good Thing.

It's not all beer and pork rinds. Frequent integrations interact with the number of code streams and...

Speed of Build

By "speed of build," here, I really mean "the amount of time it takes to build the software and run all of the tests."

A slow build leads to all kinds of evils. One of the things it does is to interact with the number of concurrent code streams to control the frequency of your integrations. It works like this: if you have ten concurrent code streams and people check in every hour, then you will average ten check-ins per hour. That means that your build has to take (60 minutes divided by ten equals) six minutes, max. If your build takes longer than six minutes, then your integrations will pile up.

In actuality, people don't check in perfectly like that. It's actually an application of queueing theory, although I can't check that right now. If I remember correctly that means we need about 30% slack in order to allow the queue to flow smoothly, assuming frequent small check-ins. So in this scenario, our maximum build time is actually closer to four minutes.

You can see how this interacts with all kinds of XP practices. Pair programming cuts the number of concurrent code streams in half, for free. The ten-minute build keeps builds small. XP teams tend to integrate every two to four hours, and XP recommends teams that are no larger than 10-12 programmers.

It works out perfectly: 12 programmers pairing and integrating every two hours is six concurrent code streams. Do the math (120 min / 6 streams * 70% utilization) and you get a maximum build length of 14 minutes. Cool.

Most teams I work with have very slow builds (it's because of their tests), so they compensate by using...

Multi-Stage Builds

A multi-stage build was one of the original workarounds presented for slow builds. (Continuous integration servers were another.) It breaks my heart, because rather than fixing their build problems, teams invest in these workarounds. Sigh... "easy" wins out over "good" once again.

That's not to say a multi-stage build is always bad. Like many easy solutions, though, it's over-used. The idea is quite simple: if your tests are slow, split your build into two parts: the "fast" build which runs whenever you integrate, and the "slow" build which runs daily, weekly, or manually just prior to release.

Sometimes the slow build is a manual regression test carried out by armies of disgrunted testers. (You'd be disgruntled, too, if you had to do manual script-based regression testing every day.) This takes days or even weeks. One company I worked with took days just to set up the test environment.

A multi-stage build provides fast feedback, but it isn't actually continuous integration. Remember the core idea: we want to be able to release software that reflects our investment. The software isn't ready to release until it's passed all of the tests in the slow build, so from a CI perspective, the frequency of integration is actually determined by the frequency of the least frequent build.

If you can't release until you've had an army of disgruntled testers run through an array of manual test scripts, then you aren't practicing continuous integration. You're integrating frequently, sure, but you aren't verifying that the integration succeeded. You're only doing half of it.

Slow builds force another common choice...

Synchronous vs. Asynchronous Integration

Even if a team works around their slow builds by using multi-stage builds, their "fast" build is often quite slow. Again, thirty minutes is common. This build is run after every check-in, which means that the team has to use asynchronous integration.

I talked about synchronous and asynchronous integration in the book, so I'll just summarize: asynchronous integration delays build feedback until after the person or pair has started another task. As a result, build failures force a context switch and interrupt flow. In practice, team members tend to delay fixing these build failures.

Asynchronous integration requires an automated CI tool, and it so happens that most of these tools do not require that the build pass before code is checked into the mainline. As a result, build errors propagate to the entire team and lead to delays and cascading failures.

In a team practicing CI well, the team's build should never break. An individual person might have trouble checking in, but those build failures should never affect the rest of the team. Never! (Well, once every month or two, maybe, due to cross-machine incompatibilities.) It's trivially easy to do with synchronous integration, and much more difficult with asynchronous integration.

Number of Collectively-Owned Codebases

Up until now, when I've been talking about a "team," I've been referring to a group of programmers that are working on the same codebase and using Collective Code Ownership.

There's another option, of course, and that's code ownership models. On agile teams, these are most prevalent when you have large groups working on a single product. For example, you might have five teams collaborating on a single codebase, with each team responsible for a particular part of the code. This is a common approach, but it's not required. Some groups choose to use collective ownership, even with larger groups.

In theory, this sort of code ownership model has clearly defined responsibilities and interfaces, allowing each team to work in their own area without worrying about integrating with the rest of the product. The reality is a bit less genteel. APIs never behave quite as expected, so continuous integration is still a good idea.

Continuous integration in this environment runs into the problems that occur with large numbers of concurrent code streams. However, it's possible to give each team its own branch for local integrations. The branches are integrated with the mainline on a regular basis--every two hours, for example.

By doing this, and assuming that code ownership prevents many of the team's changes from affecting others, each team acts as a single code stream. This reduces the number of concurrent changes from overwhelming the integration queue.

Code ownership has its downside. It requires that modules, integrations points, and APIs be clearly defined, which requires a level of up-front design. Teams will be organized around the design, which will make it very difficult to change... and some of that design will, inevitably, be wrong. This calcification of design is why some organizations prefer to have collectively-owned code, even with large teams.

End Transmission

So... some of the forces affecting continuous integration:

Easier	Harder
mature feature set	new product
small number of concurrent code streams	large number of concurrent code streams
frequent integration	infrequent integration
fast build	slow build
synchronous integration	asynchronous integration
one team	many teams

And, as I've discussed, each of these forces affects the others in various ways.

In the end, though, continuous integration is about one simple idea: we should be able to release our software at any time and get value proportional to our investment. That's the forest. Don't let this meander through the trees (or the shiny baubles that are CI tools) distract you from that underlying goal.

Comments

November 20th in Portland, Oregon: Life Inside an Iteration (Panel)

08 Aug 2008

James Shore/Calendar

I'll be participating in the Software Association of Oregon's panel discussion, "Life Inside an Iteration," on November 20th. We'll be discussing the nuts and bolts of what it's like to participate in an Agile iteration from a variety of perspectives. I'll post more once I see the SAO's program notes.

Comments

November 4th, 5th, or 6th at Agile Vancouver: Agile Planning in Action (Tutorial)

08 Aug 2008

James Shore/Calendar

I'll be reprising my popular Agile 2008 tutorial, "Agile Planning in Action," at the "Much Ado About Agile" conference in Vancouver, BC. I'll post the specific date and more information once I see the conference program.

Comments

October 29-31 in Portland, Oregon: The Art of Agile Delivery (Training)

08 Aug 2008

James Shore/Calendar

Save the date: Diana Larsen and I will be presenting a public training course on Agile execution and delivery October 29th through the 31st in Portland, Oregon. This will be a cross-functional class: we'll have sessions for the whole team as well as specialty-specific sessions on subjects such as test-driven development, incremental requirements, and customer testing.

Watch this space for information and signup details by the end of August... or email me to be added to the notification list.

Comments

October 27-28 in Portland, Oregon: The Art of Agile Planning (Training)

08 Aug 2008

James Shore/Calendar

Save the date: Diana Larsen and I will be presenting a public training course on Agile planning on October 27th and 28th in Portland, Oregon. Watch this space for information and signup details by the end of August... or email me to be added to the notification list.

Comments

October 15th at PNSQC: Collaborative Quality (Panel)

08 Aug 2008

James Shore/Calendar

I will part of a panel be at the Pacific Northwest Software Quality Conference (PNSQC) on October 15th. The subject of the panel is "Collaborative Quality: Does It Help?" (Um... yes?) It's a lunchtime panel, tentatively scheduled for 12:30pm. I'll post more information here when I see PNSQC's program notes.

Comments

September 19th in Portland, Oregon: Come Hang With the Agilists (Lunch)

08 Aug 2008

James Shore/Calendar

The local Portland Agile community gets together every third Friday to eat, drink, and make merry. Oh, and sometimes we talk about agile stuff, too. Come join us! I don't always attend, but I come to as many sessions as my schedule permits. You can find us at the Broadway McMenamins (on the corner of NE Broadway and 15th) on the third Friday of every month at 1:00pm.

Comments

The Art of Agile Development: Ten-Minute Build

30 Jul 2008

James Shore/Agile-Book

in 99 words

Build, test, and deploy your entire product at any time with the push of a button.

Your build should be comprehensive but not complex. Make it compile source code, run tests, configure registry settings, initialize database schemas, set up web servers, launch processes, build installers, and deploy. Your IDE won't do all this, so learn to use a dedicated build tool. Make sure your build works when disconnected from the network, too.

Builds should be fast. If not, look at your tests. End-to-end integration tests are the typical culprit. Replace them with faster, more maintainable unit tests.

Inside The Book

Ten-Minute Build
Automate Your Build
How to Automate
When to Automate
Automating Legacy Projects
Ten Minutes or Less
Questions
- Who's responsible for maintaining the build script?
- We have a configuration management (CM) department that's responsible for maintaining our builds. We aren't allowed to modify the script ourselves. What do we do?
- How do we find time to improve our build?
- Should we really keep all of our tools and libraries in version control?
- Does the build have to be under 10 minutes? We're at 11.
- We use an IDE with an integrated build system. How can we automate our build process?
- We have different target and development environments. How do we make this build work?
- How can we build our entire product when we rely on third-party software and hardware?
- How often should we build from scratch?
Results
Contraindications
Alternatives

Commentary: Living in the Punch Card Era

I'm obsessed with fast builds. There's a good reason for this: fast builds means quicker feedback. Quicker feedback means fewer bugs. If the feedback gets fast enough (less than ten seconds), and if you have comprehensive regression tests, then your entire approach to coding changes. You stop worrying about whether your changes work or not and just program. Thoughts flow from brain to program without stopping to worry about bugs. It's an amazing way to work... and I'm willing to bet you've never experienced it because your build is too slow. (Heck, half of you have given up on me as an idiot at this point. It's okay. I'm used to it.)

Allow me to illustrate. Once upon a time, people had to worry about syntax errors. You know, when you accidentally mistype something, or leave out a semicolon? They obsessed over them, tediously desk-checking their work, to make sure that their programs had no errors, before they even thought about putting it in the computer!

"Ridiculous!", you scoff? No, just old. This happened in the punch-card era of computing, in the 60's and 70's. It could take 24 hours to get your stack of punch cards input into the computer, compiled, and run. Compiler errors meant your job was rejected and you had to find the error, fix it, resubmit, and wait another 24 hours. Painful, yes. Thus with the desk-checking.

You don't desk-check, do you. Nope... you type in your program, and if you make a typo (happens all the time, doesn't it?), then a little red squiggly appears in your IDE and you fix it! No brainer. In fact, it's so ingrained, you probably don't even notice half the syntax errors you make.

A fast build with good tests is like that. Except that the errors you find and fix are semantic errors--bugs--not syntax errors.

Good builds are game-changing.

So, how do you do it?

It's not hard, but it's not obvious, either. The obvious part is that you have to automate your build. Your IDE's build almost certainly won't cut it. It's also pretty obvious that you have to have a comprehensive regression test suite. (This is easy if you're creating a new system and you use test-driven development from the start. It's less easy when you're working with existing code.)

The part that's not obvious is that you don't just need a regression test suite, you need an awesome regression test suite. By "awesome," I mean that it has to be fast. You're going to have thousands of tests and you want this puppy to finish in less than ten minutes, and the majority of the tests need to finish in less than ten seconds. That means you have to be able to run hundreds of tests per second.

You can see why this is unobvious. Most people create tests that set up data in the database, stimulate some part of the application, then check that the database changed correctly. It's slow. For complex systems with many tiers, it's hella slow. It's also brittle and prone to breakage, and creating test environments is usually a real pain in the keister.

If you're doing the database thing, you're probably running less than ten tests per second. For some of you, your tests are taking tens of seconds each, and they're breaking all the time.

Yup, fast builds: not obvious.

Here's the key. You have to let go of the integration tests. Figure out how to replace end-to-end integration tests with unit tests. You don't have to get rid of all of them, but the better you get, the fewer you need.

Part of the challenge here is that fast tests require a different design mindset. If you're creating end-to-end tests rather than unit tests and focused integration tests (the good form of integration test: they're like unit tests, but they cross a process boundary), you can get away with all kinds of sloppy, tightly-coupled design. If you primarily write unit tests (and focused integration tests), you have to decouple, create strong responsibilities, and have ways to instantiate objects independently of each other.

Rise to the challenge. Fast builds with awesome tests will change your life. For programmers, they're the best thing you can do for your codebase, your productivity, and your enjoyment of work.

Go ahead. It's time to stop living in the punch-card era.

Comments