James Shore

One of my Mentor Program participants just asked me about testing private methods. Here's my response:

Generally, you should be able to test private methods by calling a public method in such a way that it calls the private method. Remember, when you're writing a private method, you're engaging in encapsulation and information hiding. You're saying "this method is an implementation detail, not an important part of the class's interface." And tests are supposed to document the public behavior, not the internal implementation. So testing a private method can be a sign of a problem.

Sometimes you really need to test a private method. The quick-and-easy solution is to make the method public, and I'll admit that I've done so on occasion. (Often, I'll mark it "for testing only!") But that makes your class's interface more confusing and reduces its self-documentation. It's harder to understand a class's responsibilities if there's no way to tell public behavior from implementation detail.

If you need to make private methods public, it may be a sign that your class has become too big and complex. Often these sorts of classes need to be split into multiple, smaller classes with more clearly defined responsibilities. Once you do that, private methods often become public, because now they're part of the interface that defines the class's behavior, not just an implementation detail.

Amazon has bumped their discount for The Art of Agile Development from 14% to 29%, which is a pretty good deal. If you've been thinking about picking up a few extra copies for your team, now's a good time.

(Why the sudden change? I have no idea. The Amazonians are a mysterious tribe, impossible for mere mortals to understand, and not to be trifled with. The last anthropologist to venture into that trackless jungle emerged six months later, eyes bulging and brain addled, with nothing to say but a mindless repeated cry: "Tapioca! Tapioca!" Six servings later, he fell into a vegatative state, and nothing could be done except wipe off the pudding and place him in the care of Dr. Finkle's Sanitarium and Public Exposition (Some Exhibits Not Suitable for Small Children or the Faint of Heart). There he remains today.)

From the "Better Late Than Never" category:

We are soliciting nominations for the 2008 Gordon Pask Award for Contributions to Agile Practice.

Each year, the Agile Alliance awards the Gordon Pask Award on the last day of the Agile 200X conference. It recognizes two people whose recent contributions to Agile Practice make them, in the opinion of the Award Committee, people others in the field should emulate. In order to grow the next generation of Agile thought leaders, the award is given to people whose reputation is not yet widespread.

Details here.

PS: Brian says the late announcement is his fault, but he's just hogging the spotlight. Everyone knows it's Chet's fault.

I returned from the long weekend to discover that my ISP, Comcast, is blocking all emails forwarded from Domain Discover, who hosts jamesshore.com. This means that any email you sent me went to the great bit bucket in the sky. (It probably explains the intermittent email failures I've been experiencing over the past month, too.)

So... if you have emailed me in the last month and didn't get a reply, please email me again. I've responded to all of the emails I have.

I apologize for the inconvenience, and I've taken steps to prevent this from happening again. Domain Discover now archives my email before forwarding it to Comcast, so I won't lose anything if this happens again.

(To top it all off, Domain Discover inadvertently turned off my domain name when they added archiving to my email forward. Sigh... it's fixed now.)

Update, 8 July 2008: Mark Casem of Comcast just called me to make sure my problem was resolved. Apparently, Comcast uses Google Alerts to follow up on blog posts like this one. Impressive. I've heard that Comcast has a reputation for poor customer service, but my experience has always been positive, and this was certainly above and beyond.

I recently discovered that our book is available on a popular textbook torrent tracker. This isn't the first time I've seen it pop up on a pirate site; given the nature of digital files, it's both inevitable and unsurprising.

What amused me, though, was the hypocrisy displayed. The front page of the tracker says, "We're fighting the high price of textbooks." They actually might have a point, considering what textbooks go for. But we wrote our book for professionals, not a captive student audience, and it's priced accordingly. The cover price is a reasonable $40 and Amazon has it for $34.50. There's no price gouging here.^* If you're fighting the evils of price-gouging publishers, why is our book on your site?

(^*Yes, even at $40 we're more expensive than a paperback, but it's not price gouging. Computer books have very low sales compared to mass-market books and I'd bet they have higher production costs.)

Look, guys. You're taking something without paying for it. You're doing it because you don't want to pay, or because it's more convenient than paying. Current law calls that "stealing," the vernacular calls it "piracy." Whatever. I'm not going to lecture you--duplication of digital content is unstoppable and us content creators have to figure out other revenue streams. (Hey, hire me!) I lose more sleep over the RIAA's lobbyists and thugs than I do over lost sales. But please... don't pretend you have some high-minded aim here. You're pirates. You're not being noble, you're being cheap. Or lazy. Just admit it, already.

PS: I'm glad you like the book. 36 seeders and 257 downloads? That makes us the second-most popular by seed and seventh-most popular by download at the time I checked. Yay us!

PPS: To really show your appreciation, review the book for Slashdot.

The Project Management View blog has put their recording of my Wednesday webinar, "Maximizing Value with Agile Release Planning," up on their site. I thought it turned out really well. If you were interested but couldn't attend for some reason, now's your chance.

Thanks to Chelsea Woodhead and Ryma Technologies for putting together such a well-organized session.

A client in Poland recently emailed me about his new branch office. They're applying agile development from day one. Here's my response:

Thanks for the update, P----. I'm glad things are working well.

Here's some things to keep an eye on as you go forward:

• The team should be able to make and meet iteration/Sprint commitments almost every time. If they don't, it reflects planning problems or engineering problems.
• Your QA/testing burden should not grow over time. If it does, it reflects engineering problems.
• Bugs and incorrect features should be rare in iteration/Sprint demos. If you see them, it reflects communication problems or engineering problems.

These are the most common issues I see new teams struggle with.

I'm going to be in Toronto, with some time free, this Friday and Saturday (May 30th & 31st). Is anybody interested in getting together for beer and/or food? I'm available Friday evening and Saturday morning with no other plans, and I can drive anywhere in the Toronto area.

If you're interested, email me and we'll make some plans.

In case you don't feel like listening to all ninety minutes of my Secrets of Agile Success presentation, Mike Polen has a nice summary of the talk on his blog. He also links some of the papers I mentioned. Thanks, Mike!

I do want to make one correction: I wasn't trying to beat up on Scrum. I state my position on Scrum better in my post, "Should We Adopt Scrum or XP?"

I gave a presentation on "The Secrets of Agile Success" at the Dallas chapter of the APLN on May 12th. Susan Fojtasek recorded the session and posted it online. It was an interactive session with lots of discussion and audience stories. I thought it went really well. It's about 90 minutes long, so if you have a long commute or flight, take a look.

InfoQ just posted a video interview with me. Deborah Hartmann interviewed me at Agile 2007 last year just before the book came out. It came out great, and InfoQ also included a transcript for those of you who prefer reading to watching.

This interview joins a few other good recent interviews: Bob Payne's Agile Toolkit interview, also conducted at Agile 2007, and Clarke Ching's Agile Thinkers interview.

My understanding of success has evolved over the years. At first, I thought of it as a pyramid of levels, a la the CMM:

• Level 1: The software works
• Level 2: It's maintainable
• Level 3: It provides value
• Level 4: It provides more value than it costs
• Level 5: It provides the most value possible with the resources used

Later, I realized that there are actually multiple aspects to success:

• Organizational success is necessary for a product continue to receive funding.
• Technical success is needed to keep maintenance costs under control
• Personal success is required to receive support from stakeholders, prevent sabotage, and gain team member commitment.

Recently, Jurgen Appelo pointed me at an essay he wrote that discusses (among other things) the nature of success. He takes a completely different approach. In Jurgen's view, success is the absence of failure. One thought that occurred to me as I read is that we could measure success as the length of time a product is in production. An intriguing idea.

It's a fascinating essay, long, but worth the read. Take a look.

A reader recently asked:

I have learned a bit about Scrum and Crystal and found that things are looser and they do not prescribe practices as XP seems to and that they want you to work toward finding the practices that work for your team. An example is Mike Cohn saying that he does not like to prescribe practices, like TDD which he believes in, but wants the team to discover for themselves their need of it as they do root-cause analysis and retrospectives. Does this go against y'alls advice on trying to do all the practices and later adjusting things to meet your needs as a project? Or is it something different?

[For example, see] Mike Cohn's thoughts about Scrum and XP. In particular starting with Scrum and creating your own XP.

My response:

I agree with Mike Cohn's description of the differences between XP and Scrum. I would emphasize that Scrum and XP are very similar, with the exception that XP includes more "how to" practices. There's also a bit of an focus difference; I feel like Scrum's focus is "let's create a self-organizing team" and XP's focus is "let's create excellent software." In other words, when I talk to Scrum people, they tend to talk more about self-organization and enabling the team; when I talk to XP people, they tend to talk more about software and delivering value. But despite that difference in attitude, the actual practices are very similar.

Mike's closing statement, that Scrum brings big benefits by itself, matches my experience. Scrum is also easier and less threatening than XP, so I see a lot more people starting out with Scrum. On the downside, the teams that start with Scrum tend to struggle more than the teams that start with XP. The XP teams experience more pain starting out, but then get to a high performance state within the first year. The Scrum teams I know have a much longer ramp-up time, generally having code quality problems for several years. Actually, I have yet to meet one that wasn't having code quality problems. None have gotten to the same high performance result that good XP teams do.

Granted, these statements are based on my experience, and my experience is formed by the people I meet at conferences and the companies who hire me. High performance teams generally don't hire me, so it could be that I'm not meeting the high performance Scrum teams.

Nonetheless, my experience is that Scrum teams have an initial success and then struggle. XP teams struggle to get started, but once they figure it out they have more success. Sometimes they don't figure it out, fail, and give up. With Scrum, teams seem less likely to realize that haven't figured it out; I've met teams who said they were using Scrum but weren't doing anything of the sort; they were just using the terminology. That happens with XP, too, but to a lesser extent, and usually of the form "we're doing XP but not..."

To boil it down to simple statements (perhaps too simple):

• Scrum: easy to adopt, very hard to master, fails quietly. You're more likely to successfully adopt Scrum, but the benefits won't extend to your codebase and you'll struggle with legacy code for many years. If you're missing pieces, you may not realize it.

• XP: hard to adopt, easier to master, fails noisily. You're less likely to successfully adopt XP, but you'll be well positioned for long-term success and mastery. If you're missing pieces, you'll probably be able to tell.

I have one complaint about Mike's essay. XP doesn't prescribe engineering practices, as Mike says; it includes them. For teams new to agile development, I do recommend that you use all of the practices from the start. However, a team that knows what it's doing will change its practice of XP over time. No two experienced XP teams practice XP in the exact same way, and there's no requirement that you do exactly what the books say. For example, one long-running XP team I know has evolved away from using iterations entirely, but I'd still think of them as practicing XP.

However, in order to know what to change, you have to know how the techniques work in practice. That's why I recommend starting out with all of the practices. Otherwise, it's like deciding to be an avant-guarde violinist without first mastering the violin.

Should you start with Scrum or XP? I recommend starting with XP, but Mike's right--you won't succeed if you force the practices down people's throats. The team has to agree to try them. If they're not ready to do that, you might be better off starting with Scrum. Then you can use that success, along with some of the code problems you'll encounter, to guide them to XP.

Finally, if you have the opportunity to get a coach to help you adopt your method, I recommend that you do. A good coach will help the transition go a lot more smoothly.

I haven't been this excited about a programming language since I tried to invent my own.

Programming in Subtext

Subtext is a programming language invented by Jonathan Edwards. It's based on schematic tables (see figure) which display logic and computation in two dimensions. In contrast, textual languages show just one dimension (a stream of characters), and it's usually computation. Introducing conditionals requires you to puzzle out control flow.

Subtext makes logic and computation obvious by placing them on two axes in a table. To understand how this works, I recommend watching Jonathan's video demonstrating Subtext. (If you saw the video for the original version of Subtext, watch the new one--it's come a long way. I didn't even recognize it.)

Subtext nails a bunch of things that I want. To explain, I'll need to digress for a moment.

Example-Driven Development

My background is in commercial software development in a wide variety of industries. I'm passionate about seeing software lead to real-world successes, which involves organizational success (return on investment, to oversimplify), technical success (maintainable code), and personal success for the systems' community. I've dedicated my career to improving my understanding of those three components of software development. I still have a long way to go.

Along this journey, I discovered Agile methods and I've became fairly proficient with them. Two Agile techniques that I've found to be particularly powerful are Test-Driven-Development (using a tool like JUnit) and Example-Driven Requirements (using a tool like Fit).

A Fit document

JUnit and its brethren are great. TDD works well. Example-driven requirements also works well, but I've become increasingly dissatisfied with Fit. Fit, in case you haven't heard of it, is a tool created by Ward Cunningham and maintained, in a laissez-faire sort of way, by me. It allows you to provide domain-level examples in HTML tables, then automatically test those tables via fixtures that execute the table against your production code.

The great thing about Fit is that it enables example-driven requirements. This is incredibly powerful and useful, and Fit is the only tool I know of that focuses on this. (Most clones of Fit get it wrong, focusing on test automation rather than customer collaboration.)

The problem with Fit is that it's a maintenance burden. You have to write your examples in HTML and none of the existing tools (not even MS Word) make editing and revising HTML tables easy. You have to create fixtures. When done well, fixtures are small and simple, but they're easy to get wrong. You have to deal with Fit's primitive and difficult-to-use tool support. (Okay, I could fix this one, and should.) And, to top it off, almost everybody misunderstands and misuses Fit.

Okay, back to Subtext and Jonathan Edwards. TDD and example-driven requirements (EDR) can both be considered cases of example-driven development. In other words, your development efforts are focused around creating examples and making them work. The examples form sort of an incomplete specification of your work that are fleshed out along with your program. TDD operates at the level of programmer intent and EDR operates at the level of customer intent.

Well, Jonathan shares a passion for examples in programming. So it's no surprise that Subtext looks like it will support the Agile conception of example-driven development quite well. The best thing about it is that it looks like it will support TDD, example-driven requirements, and software development within a single paradigm. Brilliant.

Critique

In addition to its support for example-driven development, I like Subtext's focus on making programmers' intent more obvious. The schematic tables are a great way to show what happens computation is mixed with complex conditionals. I particularly liked Subtext's ability to automatically discover gaps and overlaps in conditions. A minor nit: I think Jonathan could decrease the verbosity of his language by allowing more complex statements on each line, perhaps allowing a click to change the view between the simple, linked form shown in the video and more complex compound statements. For example, the Fibonacci example could show the entire >= 2 case in one statement: fib(in - 1) + fib(in - 2). I think I'd actually find that more readable than the current display.

A little light reading

Given that Subtext is a work in progress, I liked almost everything I saw in Jonathan's video. The one thing I disliked was towards the end, as he was talking about polymorphism. I got a sense of disdain towards object-oriented programming. That's not uncommon in the academic world--OOP is anything but well-defined from a formal perspective.

But although OOP is messy, it has an important characteristic that Jonathan also values: it makes programming languages more usable. Objects are an important way of organizing large programs so that the programmer can ignore irrelevant details. Although polymorphism makes control flow more complex, as Jonathan observes, it also enables design techniques that make programs easier to understand.

In a small program, an experienced programmer relies on well-named functions/methods with carefully-constructed semantics. When that's the case, she may use the context of the program to infer the behavior of those functions. For example, if you saw that a mathematical calculation called an "exponent" function, you could guess that it calculated the exponent of a number and move on, not bothering to look at the body of the function unless it was directly related to your work.

Classes and methods perform a similar service in OOP. In a large program, an experienced programmer also relies on well-named classes with carefully-constructed responsibilities. The noun-verb combination of classes and methods allows her to infer the behavior of those classes. For example, if you saw a call to monster.attack(character), you could guess that it calculated attack and damage probability and adjusted character's hit points accordingly.

Subtext's damage calculation gets complicated

Subtext could face challenges dealing with larger, more complicated systems. You can see in the figure how even minor complications in the damage calculation start to make the table unwieldy. Subtext provides a focusing mechanism to hide unnecessary details, but I'm skeptical of how useful the current incarnation is for this purpose. I think it will work well for simple algorithms, but not for the enormously complex, interrelated computations that large systems deal with.

Those enormously complex, interrelated computations are exactly what object-oriented programming languages are good at managing. (With good design, at least, which is by no means a given.)

I'd like to see Subtext give objects a more prominent role in the language. It's hard to tell from the video exactly what's going on, but it looks like the programmers' use of objects is largely limited to defining types and subtypes. I'd like to see more. Specifically, I'd like to see support for named methods attached to objects. They could be expanded and edited in-place, or they could be edited in the context of the object.

Simplifying damage calculation with polymorphic methods

For example, a lot of the complexity of the damage calculation comes from the polymorphic 'power' constants. Magic attacks have a power of 5, melee attacks have a power of 4, and ranged attacks have a power of 3 (not shown in the figure). If Subtext showed that polymorphism as a reference to a method, then the diagram would be simpler. The effect of defense could be similarly collapsed into a 'damage' method.

To continue the example, if the programmer wished, he could open up the 'Attack' class and see 'power' and 'damage' defined there. Perhaps subclasses and/or methods could be shown as columns. That would allow him to contemplate the relationships between the classes and their methods--a common need--and make changes to them together.

This is actually a fairly minor tweak to an impressive system, and it's quite possible that Jonathan has already thought of or even included these improvements. If he hasn't, I hope he'll consider them. Either way, Subtext is impressive. I want it.

(Thanks to Keith Braithwaite for the link.)

Every so often, I describe what I do to make this site a reality. For the four of you who like to watch me gaze at my navel, I have good news: today is one of those days. For the rest of you, I'm sorry. It's one of those days. You can move on. (I won't be offended. You can even give this essay a nasty rating if you want. I have skin like leather, and I do pay attention to the ratings.)

Okay, here goes.

Production

I use a MacBook to compose all of my essays. My old Toshiba laptop (which was running Windows) died and I decided to see if life was better on the other side. I have come to like Mac OS X, after some initial frustrations, but I'm not a fanatic or anything.

I write the body of my entries in hand-crafted HTML using Smultron, a text editor. Smultron is not a great text editor, but I've grown used to it and haven't felt compelled to seek out something better. I also use OmniGraffle when I need to make pretty diagrams. Mmm... lickable.

Everything required to make the site go is stored locally, on my MacBook. I deploy the site to a local copy of Apache for testing, then to my web host when I'm ready to go live. I use a Ruby Rakefile to build the site, and then rsync the whole mess to the server. The exact same site goes on the server as on the local test site; all that changes is the rsync destination.

I am absolutely paranoid about backups and revision control. There's years of work here. That's why all of my essays are written in a text editor and stored on my laptop. Most blogging software I know of saves your work in a database on the web server. Easy, but not fault tolerant. I plan to be writing for the next ten, twenty, thirty years... minimum. My stuff has to be easily scripted, version-controlled, backed up, and trivial to deploy to another web host when (not if) my current host goes under.

The source for my site (including rakefiles) is versioned by Subversion. The repository is local, on my laptop. The laptop is backed up several times over: first, the entire site is on the web host, of course. Most of my files are also backed up automatically to my Mirra continuous backup server, running on my wireless network. The entire computer is backed up with Time Machine every hour. (Time Machine is better than Mirra, so when the Mirra server dies, I won't replace it. Mirra seems to have been killed by Seagate anyway.) I also make a bootable whole-disk backup using SuperDuper! and store it off-site.

Design

The biggest challenge for me in making the site was my complete lack of web design skills. (Software engineering, yes. Graphic design, no.) Sitepoint got me started and A List Apart carried me through. I borrowed liberally from Jeffrey Zeldman in creating the side menu, as permitted by comments in his CSS.

For the color scheme, Jason Beaird's "Color for Coders" article helped me get started and Visibone's Color Laboratory allowed me to pick web-safe colors. (I have no idea if we're supposed to limit ourselves to 216 "web-safe" colors any more, but it seemed like a safe bet. I cheated a bit for quotes, though.)

I also took advantage of some royalty-free icons. The RSS feed icon () came from Feed Icons; the print icon () came from graphicPUSH; the star icon () came from 1ClipArt; and the spinner () came from Andrew Davidson.

The nicely self-contained JS-Kit handles my comments and ratings. JS-Kit is very straightforward, but I do have a few small (very small) lessons learned that might be useful.

Finally, lots of trial and error got me to the result I have today. W3Schools' CSS Reference was invaluable for making it work. Eric Meyer's "Going to Print" article provided the finishing touch for my print stylesheet by showing me how to automatically insert URLs after printed links.

During this process, I learned how badly Internet Explorer 6 supports CSS. Rather than struggle with IE's problems, I settled for graceful degradation. IE will work, but Firefox renders the site better, will give you URLs in the print results, and has a more responsive "print" button.

Hosting

The site is hosted by TextDrive (now owned by Joyent) and runs on the Apache web server. I use mod_rewrite to make sure public-facing URLs are implementation-independent and to ensure that you can link to a page forever (for sufficiently small values of forever). Domain Discover makes sure that my domains (jamesshore.com and titanium-it.com) point to the right place. Mint handles my analytics, along with the plug-ins Fresh View, Trends, Download Counter, Doorbell, Outbound, and Secret Crush.

The site is rendered by the ultra-minimalistic Blosxom. Blosxom was the only blogging software I could find that actually allowed me to store entries locally, in files that I can back up and put in Subversion, rather than on a database on the server. Blosxom runs on Perl 5. Blosxom does almost nothing by itself, so I use these plug-ins to help out: blok, meta, default_flavour, prefs, breadcrumbs, directorybrowse, entries_index_tagged, interpolate_fancy, menu, plain_text, postheadprefoot, readme, static_file, urltranslate, and wordcount.

I run my whole site with Blosxom, not just the blog portion. Getting this working was quite a headache and I had to make some custom mods to several of the plug-ins as well as Blosxom itself. It's been three years since I got it set up the way I like it, though, and the site has been pretty much hands-off and trouble-free ever since.

Colophonem adidi.

(I ran across that little gem while using Google to define "colophon." It means "I have put the finishing touch to it.")