Winning with Tech Practices

People want—need, in fact—to know if Test-Driven Development really works in the long run.

There are academic and industry studies out there, indicating that TDD reduces defects, and makes future changes easier (the first two tiers of value obtained). I used to quote those findings in my courses and in my writing, but since people can simply google for them, I no longer do this. Plus, some who see these studies are still not convinced, and I prefer not to waste time on academic debate.

What follows are the stories from my own professional experience as a software developer and XP coach. For every team I’ve been on that stuck diligently to TDD for six months or more, a third tier of value was “unlocked.” These are the Tales of the Black Swan User Stories.

A “Black Swan User Story” Defined

“Black Swan” is the term used by Nassim Nicholas Taleb in his book The Black Swan, to describe a particular type of event. A Black Swan Event is rare, unforeseen, unpredictable, and very disruptive (in a positive or negative way). And though they are unplanned and unpredictable, they are—given an appropriate length of time—inevitable.

Recent examples include the Fukushima quake/tsunami/nuclear-meltdown triple-gut punch, and COVID-19.

Depending on the state of “the system” that receives a Black Swan, they can be extremely costly, extremely valuable, or both. Taleb describes systems that improve when clobbered with such an event as “antifragile.” Contrast this with resilience, were the attacked system rebounds back to a relatively steady state and is “healed.” An antifragile system goes one step further and will adapt and “learn” from the event.

So a “Black Swan User Story” (my term, combining Taleb’s term with Kent Beck’s “user story”) is a request for a change to the product that is unplanned, unexpected, potentially “game-changing,” possibly very expensive to implement, and potentially even more valuable than anyone is aware of at the time. Plus, once it has been implemented, the code is more flexible, and potentially able to handle even more change in the future. The source of this last benefit cannot be attributed to a single practice: likely it’s partly the improved safety net of tests, partly the code becoming more extensible in the areas affected, and perhaps mostly due to the learning obtained by the team.

From 1998 to 2004, I worked on at least 6 products as a full-time TDD developer/coach. That may not sound like many, but these were long-lived products, not short-lived projects. This was six years of as-close-to-full-time TDD as one can get (and still take vacations, or deliver occasional XP/TDD courses). Each client engagement required immersion into a team dedicated to building a viable and successful product.  The shortest time I spent on any of the following was six months.

And, what I’ve experienced: After months of comprehensive test-first development and diligent refactoring, teams often find a user story introduced at the top of their queue that would have taken the average development team months to build (if at all), but it takes their team mere days.

 A “Major Architectural Change!”

In brief: Internationalize an entire suite of servlet applications? Three days, and about a dozen lines of code.

In 2001 James Shore and I were working on the “Denali Project,” Novell’s Developer Portal. It was a series of applications deployed through a few Java servlets. When I got there, Jim and his team had been using XP practices for a while, and I was very impressed with the results.  In fact, as I told him then, it was the easiest code I had ever had the pleasure to work in.

Right before joining Jim’s team, I had been subjected to EJBs, JSP pages, and a host of other awkward attempts to scale web apps. The brilliance of Jim Shore, I was to discover, was that he never took a framework at face-value. He would explore, and he would often come up with a simpler, home-brewed way to accomplish the same goals that those frameworks often failed to deliver.

At first, Jim and I were the whole Portland portion of the team, and most of the team was in the Salt Lake area. We would fly to Salt Lake for planning every two weeks. I clearly recall the day this particular Black Swan user-story flew into the planning meeting.

“I know that we agreed we’d only support languages represented by the Latin American character sets,” said the Product Advocate, “but we have an opportunity to make inroads into Asian markets, if only we could support Unicode character sets. So…?”

All static text in our app was already internationalized. That was, if I recall, a configuration issue, technologically: The browser would tell the app what character set it wanted, and the app would serve up a page from the appropriate folder. Our static text was already translated by professional translators.

The challenge exposed by this user-story was particularly noticeable in our “survey” app. Like an early version of today’s well-known online survey-creation-and-sharing apps, this app allowed the user to build survey questions and multiple-choice answers. Only, Asian markets were limited to writing the surveys in English (or French, or Spanish…)

New to the team, I felt my heart sink. “Oh, no!” I thought to myself, “What have I gotten myself into?! Internationalization! A major architectural change, that is!”

The team took a brief look at the code. Jim suggested an estimate of three days. I nearly fell out of my chair.

“Are you kidding?!”  I started, “To retrofit internationalization into all our servlets and all our pages and…???” I had flashbacks of JSP-heavy dot-BOMB apps (prior to developers learning how to properly use tag-libraries), and having to change every single textbox, and every bit of dynamic text.

“Relax, Rob,” he said in that metered, matter-of-fact voice of his, “It’s just an estimate.”

At that time, many XP teams were estimating in what were called “Ideal Days.” An Ideal Day is a day of uninterrupted, unfettered work. So our “velocity” was measured in Ideal Days. (You can just replace all occurrences of “Ideal Days” with “Story Points” if it makes you feel better.) We two external consultants, working in Jim’s office near Portland, and not having to go to any internal Novell meetings, could pretty much work six or seven hours uninterrupted on product development.

So when Jim said three days, on the one hand, he meant three days. On the other hand, it could take us three times that and still fit into our two-week iteration. But even two weeks seemed far too short to me!

Jim and I got to take that Black Swan Story back to Portland with us. We (he!) volunteered. The rest of the small team seemed happy to not have to deal with this particular user story. At first, I felt as they did: “Why didn’t we just turn this one down? This is going to be a disaster!” But Jim’s unexplained confidence was contagious, and by the time I was back in Portland, I was happy for the challenge.

We did it in three days!  In fact, we spent about 2.5 of those days researching.

In 2001, there weren’t very many web apps that could do what we were hoping to do. We experimented by registering with various well-known sites using some random Chinese, Japanese, or Korean phrase (e.g., “감사합니다”), and the app would respond with something like “Thank you for registering, ☐☐☐☐☐☐☐☐☐☐!”

We found one (and only one) site that could do it. (Remember BabelFish.com? It’s still out there, though it looks and feels a bit dated.)

This proved that it was possible. Then we set out to figure out exactly how to do it in Java. All our web searches resulted in nothing. Despite the fact that Java was touted as “all Unicode, all the time!” apparently no one at that time had publicly documented the way to preserve the whole Unicode character from browser to database and back to browser again. 

We read carefully through the Java J2EE documentation regarding input and output streams and chaining them together (modeled as the Decorator Pattern, if I recall), and got our first unit test to pass. We then thoroughly tested that code to make sure it worked with all sorts of character sets, old and new, 8-bit and 16-bit.

Three days.  Oh, we had some luck on our side, too.  It turned out that it was relatively easy to convert an Oracle database from 8-bit to 16-bit. You shut it down, alter a single configuration parameter, and restore all the data from your backups.  So three days of development time, plus a bit of weekend maintenance downtime, and we were up and running the next week.

For me, that’s not the amazing part of this story! There’s a reason why we were able to make all the changes to all the pages and servlets in about half a day.

Not a Miracle, Just Diligent Refactoring

It only took about a dozen lines of code, in one or two classes.

Part of doing TDD well is diligent, thoughtful refactoring.  We (and they, the whole team, long before I showed up) continuously refactored away duplication and other code smells as needed, rather than waiting until “we have the time.” The design was nearly ideal for what the system had been asked to do up until that point.

At some point, refactorings had led the team to isolate all user input and all generated output through one or two classes, and two methods.

So when it came time to internationalize all user input and all stored data back to the user, well, that was already isolated in those two methods.

It had not been an up-front architectural decision. I’m not even sure it had been a conscious decision. It was a nice example of “emergent design”: simple guidelines and efficient practices, repeated with discipline, can result in better outcomes than rigorous planning and untestable UML diagramming. (Apparently this particular design was driven by a desire to make unit testing easier and more concise.)

About a dozen lines of code in three days? Doesn’t sound all that productive, does it? Of course, it’s not the number of lines of code that mattered to the customers in those new potential markets (China, Japan, S. Korea…).

Once in a Lifetime?

Afterwards, I clearly recall thinking, “That was neat! But I doubt I will ever see something like that happen again in my career!” Thankfully, I was wrong. Black Swans may be rare, unpredictable, and expensive; but they’re also inevitable.

UTF8-to-Unicode: About ten lines of Java code

OTIS 2

In brief: Add a functioning "Display in PDF" button to every one of dozens of HTML reports after having committed 6 months to HTML-only? Less than a week.

Menlo Innovations assigned me to a work with the small team at the University of Michigan on a product called OTIS2. We were to rewrite the University of Michigan Organ Transplant Center’s “Organ Transplant Information System,” aka OTIS. OTIS was written in FoxPro, and had reached many of the awkward limitations of that technology.

FoxPro is not even close to a self-documenting language, and OTIS was reaching its 10th year in production.  Extracting all the “business” rules (better described as “critical patient-survival” rules) was going to be a challenge. Menlo had actually suggested at one point that it might be easier to rewrite FoxPro (since Microsoft had abandoned it) and remove the quirky limitations, than to risk missing a critical piece of domain-specific knowledge. Instead, Menlo and the U of M had agreed to rewrite OTIS in Java, partly to coach the U of M developers in Java, object oriented design, and Agile methods (specifically, Extreme Programming).

To test OTIS2, we would always run all rewritten reports in parallel on both systems, and compare the report output, letter for letter.  In that way, as we developed OTIS2, good old trusted OTIS(1) would itself provide detailed functional test scenarios for OTIS2.

One day, after the first early release, the first (of two!) beautiful Black Swan Stories arrived, courtesy the users (aka transplant surgeons). They really liked what we were doing with OTIS2. Like OTIS1 before, the reports were all in HTML. Some of the reports would be printed and placed in patient records.

Think back to the World Wide Web in 2001. Whenever most printers printed a web page, that page would come out on two separate sheets of 8.5x11 paper, because the screens were wider than they were long, and apparently the idea of fitting them to 8.5 inches, or printing them landscape, was not an interesting feature to browser developers of the day. So a nursing assistant would take the two sheets (or more, if it was a long report), staple or tape them together so the left and right sides of the report were together, fold the report back into a size that would fit in the patient records folder, and place it in the folder.

“We really like OTIS2, but there’s just one small change we’d like to request: On each report, we’d like you to add a button that says ‘Print me in PDF format’. We realize that we’re going to have to rearrange the next release plan, but it’s that important to us.” In fact, more important than all the stories in the next release.

So: A whole new output format, in addition to HTML, on dozens of unique reports! What would that be to most of the apps built before 2003? Say it with me: “A major architectural change.”

As the XP coach (and de facto architect), I knew that we had already done the unthinkable, and hard-coded HTML tags into our Java code.

(Aside: “Why?!” you ask? Why not use JSP pages? Sun touted JSP pages as a “solution” to the embedded HTML problem, when really it was a solution to the “Microsoft beat us to market with ASP pages” problem. After all, how is Java embedded in HTML tags any better than HTML—which is data, by the way—embedded in Java? I’ll grant you, tag libraries were pretty cool, and testable. But few developers wrote their own tags, and fewer unit-tested them. But I digress...)

Over time we had realized that our reports tended to differ by columns. Patient data would appear as rows from a database query, but the columns were specific to which organ, or organ-combo, was being transplanted: Heart. Heart-Lung. Both Lungs. Heart-Plus-Both-Lungs. Liver. Liver-Kidney. Brain.[1]

There were other dimensions, as well. The reports would satisfy a particular general query. Who is likely destined to receive the next transplant? Who survived the transplant for over 10 years? And, presumably, there was enough data in these large reports to help create a hypothesis as to which of the thousands of variables were important to their survival.

Eventually, surgeons could customize the reports by adding special sets of extra columns.

HTML naturally holds cells within rows. But the domain asked us to orient cells within columns. So we had this “business model” of Columns that would reference the correct “Cell” of data from each record in the query results.

When the app generated the HTML output, Report code would walk through the collection of Columns and ask them for the Cell corresponding to the correct row number.

In summary: Before this Black Swan arrived, we were taking database rows, organizing them into columns, and then formatting them into rows. Insane, I know.  (And if you’re a fan of code optimization, you may be squirming in your seat. It was inefficient, I suppose, but it was all happening in memory. No one ever complained about response time. The doctors’ bottleneck was always printer speed.)

So, when we heard this Black Swan Story, the team asked to have an hour or so to do some research. We quickly learned two very important things: (1) Whereas HTML is row-centric, PDF is column-centric, like the business-model. (2) There was already a Java library available for generating PDF output programmatically.

“Three days,” I said.  Many jaws dropped to the floor; both team jaws, and stakeholder jaws.

We spent the first day refactoring. We took all the HTML formatting code and extracted it away from the Report code, into something called the HTMLEmitter.  Ran all the tests, and pushed to CI.

Day two, we used TDD to create the PDFEmitter, and to allow the Report to switch emitters. Probably based on something as simple as “?format=PDF” on the URL.  The Report was then instantiated with the correct Emitter (an abstract class) injected into the constructor.

(Getting nerdly: Some of you might recognize this as “Branch by Abstraction,” or what I used to call “refactoring towards extensibility.” Also, it resulted in a Template Method design pattern, which has long been my favorite pattern to use as example in all technical courses, and is pretty much built into the Ruby programming language as “modules.” All this is fancy nerd-talk for “divide and conquer.”)

It worked, almost perfectly, right out of the box. I recall that we had to tweak the PDFEmitter (with TDD, of course) to get the cell padding and fonts just right. That was the first few hours of Day Three. We finished “early” (in quotes, because three days was “just an estimate” after all!)

Note that we had to actually generate reports, look at them, study the PDF protocol in a little more detail, then go back and use TDD to alter the PDFEmitter until it was just right. So, this is not only a success story for TDD and diligent refactoring, but also for Exploratory Testing. Not all testing can, or should, be automated. Someone (developer, tester, Product Advocate, customer, and/or user) always has to fire up the app and look at what it’s producing.

OTIS2, The Sequel 

In brief: We re-purpose a tool we built for our own data-and-schema conversion for a department head who was spending 80% of her time manually converting data for another reporting system. This saves (present tense intentional) the hospital hundreds of thousands of dollars per year. We were done in less than a week.

In addition to rewriting OTIS in Java, we needed to port all the data and schema from FoxBase to Oracle.

In order to speed up responsiveness in the original OTIS app, some critical FoxBase tables had been extremely (and excruciatingly) de-normalized. We wanted to be able to re-normalize some tables, and also convert some data formats to a more standardized SQL database type.

So we had built a custom FoxBase-to-Oracle database conversion app, which started out simple but quickly became such a field-by-field customization nightmare, that it morphed into a processor of human-editable matrices of per-database-column conversion rules.

I’ll give you a moment to parse that last sentence. I’m not sure how to simplify it further without adding a curse-word.

By the time we were done, it probably could have been used to convert any schema into any other, and convert any data. It was XSLT for relational databases; only even uglier, and cognitively much less intuitive. (Yes, I just referred to XSLT as “intuitive.” It is, once you’ve been working with it for a thousand hours or so…)

Another division of the U of M Hospital heard about what we were working on, and asked whether or not we could customize this database converter.

“Actually,” we said, “It can probably do what you need without any changes in code. We’ll just give you the application as-is, and show you how to configure it for your schema.”

Somehow, over time, the director of this division of the U of M transplant hospital—a highly skilled and highly paid hospital director—had become overwhelmed with a growing backlog of manual data conversion-and-entry from another old FoxBase database. When she came to us, she was spending 4 days each week manually entering data and creating a single set of reports.  Yes, 80% of her time! (Presumably there was a reason, perhaps HIPPA-related, as to why an intern could not be hired to do the work. Or perhaps this task had very gradually grown out–of-control, like deadly bacteria in a Petri dish.)

We gave her the app, and configured the conversion instruction mappings for her (not an easy task unless you were an OTIS2 insider), and reduced her time spent on this task from 4 days per week to an hour per week.

Once we had worked through her schema, I think this director saw us as wizards of arcane magic, though her conversions were far more straightforward than those of OTIS1-to-OTIS2.

We weren’t proud of the way the OTIS2 conversion map looked (the worst user-experience I’ve ever helped develop!), but the Java code that did the heavy lifting was readable, concise, elegant, easily extensible, and very thoroughly tested.

And rescuing 80% of her time made us heroes, and made her a hero in her department.

Pain-Avoidance

By the way, the conversion app was a key player in allowing us to go live 18 months earlier than the 24 month plan. The original plan was to convert the whole database at the end, after OTIS2 had been thoroughly tested with no real data. You know, Waterfall Style. I can only imagine the suffering that would have caused us (or them, since it would have occurred after my time with Menlo had expired - who needs an XP coach when the coding is “all done”?).

Fortunately, the great folks at Menlo were able to convince the leadership at the U of M that ongoing data conversion was much safer.

The converter ran nightly, would detect changes in the FoxBase database, and would convert only updates or inserts (nothing was ever deleted from that database).  It could also be told to start over from scratch, particularly for significant schema changes, but we eventually stopped using that feature.

That, and giving OTIS2 the ability to recognize what it couldn’t do and to forward any such request to OTIS1, allowed us to retire OTIS1 gently and gradually. (Nerdly aside #2: Yep, the dreadfully-named Strangler Pattern, which uses a Façade Pattern to allow such migrations to happen without having to “throw the big switch!” We just thought of it all as necessity-driven.)

Update on OTIS2 

We built every part of OTIS2 and the data-conversion background application Test-Driven.  At the end of two years, OTIS2 had about 20,000 tests. Perhaps 2 to 5 percent of those were the functional tests, and tests that connected to a database to perform focused integration testing of the database-access layer.

All 20,000 ran within 15 minutes.  Imagine! In the time it would take us to stroll down to the nearby café and grab an Americana, we would know whether we had broken any of the approximately 12 person-years of investment we had made in the rewrite, while preserving at least another 20 person-years of life-critical business-rules.

We had such faith our hand-crafted safety-net of unit tests that we would make changes days before going live.  (Which we started doing every two weeks, after 6 months of initial development.)

We would eagerly invite Menlo newbies onto the project, to do actual work. “Welcome to OTIS2. Be aware, if you introduce a defect, a patient might subsequently die. (No pressure!) We’re going to implement this feature together, you and I, and we can do whatever we need to do to any of the code, so long as we run all the tests before pushing to the CI server. If one single tiny test breaks, we don’t push.”

In June 2016 I went back to Ann Arbor to chat with Richard Sheridan. He mentioned that the last “software emergency” at Menlo (requiring weekend or overtime efforts by someone on the team), including ongoing enhancements to OTIS2, was in 2004.

His teams use TDD on every software product they build, and the last show-stopper was over a decade ago.

CMBS Pipeline 

In brief: TDD & pair-programming with a no-nonsense college grad reduces a critical 6 month rewrite down to a few lively and educational weeks.

In 2003 I worked with a small team responsible for enhancing some very mission-critical software.  The entire organization was oriented around one simple revenue stream:  We took commercial mortgages and securitized them. In other words, we took batches of loans, helped rate them, and then bundled them together into securities to be re-sold to investors.

This company had started as a small start-up, and the software was originally built by a single developer, who—I later learned—had also been teaching himself Java while writing the system that supported the brokers. It was a mess. He had learned all the clever syntax of Java, and not one iota of real object-oriented programming.

(Aside: I’m not blaming this unknown developer. He wasn’t unique in his mistaken understanding of objects. Before I received XP coaching in 1998, I decided to take a course in “Object Oriented Programming” from the nearby college, in order to be prepared for that XP training. I know exactly where those bad OO habits come from.)

When I started with this client, the manager showed me one class in particular: the infamous AbstractDAOLoanBean. This was the parent class for all forms, business objects, and database objects; and the “helper” class (in other words, a home for misplaced procedural code) for those behaviors and everything else.  Yes, “One Class to Rule them All, and in the Darkness Bind them.”[2]

The manager said, “It’s the heart of the system. So don’t touch it!”

“Heh! Right…” I said (inwardly). There was no way to make any significant enhancements to the system without changing AbstractDAOLoanBean!

The system had few tests, and those were higher-level, slow-running, broad-scenario tests. Helpful, yes; but cumbersome, fragile, and slow.

Gradually, mostly out of necessity, we had refactored that abomination, and its tests. After about six months, the code was starting to look better. But not great.

Our first big challenge had been to add a new type of loan. Though AbstractDAOLoanBean did everything, the concept of a single “Loan” was spread out over numerous subclasses.  There was AbstractLoanView, a LoanDAOImpl[3], an AbstractLoanAction…[4]

And then the Black Swan arrived. The bank that handled the actual loans for us saw an opportunity to streamline the whole process, if only we could exchange loan information in their own XML format. Less data-entry, more time in the field evaluating actual real estate. Everyone wins!

We were already using XML to represent loans when we generated reports. But our XML format (or “schema”) was different than theirs.

At the time, I didn’t see what—to many of you now—was an obvious solution. The small team and I estimated that it would take months to alter the Java code to create and use a different schema.

Enter the Brilliant College Grad!

At about the same time, we had decided to grow the team, organically, by one or two. For our technical interview, I devised a programming puzzle—not that the candidate had to solve, but that we would solve together, by pair-programming and using TDD.

The developer we hired had never done TDD, or pairing, but he did a lot of things that let us know he fit into our team: he looked up something on the Internet without asking permission or apologizing; he disagreed, politely, and (in the case where the Internet proved him wrong) he accepted the new learning. Bottom line: It was just like doing the work. It didn’t feel like an interview to either of us, and that was what I was hoping to set up. I wanted to know if he could learn new practices, and apply his own intelligence.

We hired him as quickly as I could get the request through HR, and got him started as soon as possible.

So I sat down to pair with him on his first day, and explained that we needed to rewrite a lot of our XML-emitting Java code, and I explained why. Blah blah blah…

He stopped me and said, bluntly, and with mild incredulity, “Why not just use XSLT?!”

I answered, “Well, mostly because I have no idea what you’re talking about.” (Hey, it was 2003 and XSLT was the hot new tool! I can’t always keep up with every industry breakthrough…)

He explained that XSLT could transform text from one format into another. E.g., XML into HTML. He was suggesting that we continue to use our format, and to build a transformation layer right at the boundary between our system and the bank’s.

Together we found that there was a very simple XSLT-processing object for Java, and there was already a single convenient place in our code to pipe XML through that transformer. Sound familiar? Yep, this was so, thanks to all that previous refactoring driven by the need to add types of loans.

There was another problem: XSLT is code, in the form of XML (bleh!). It is also very fussy. Without tests, you could fix one problem, and another might be created, manifesting deep within an XML tree. I imagined erroneous values placed into incorrect fields, resulting in false bond ratings, lost reputation and revenues, and crushing lawsuits.

Fortunately, we were able to unit-test our XSLT code. The transformation code all lived in a production XSLT text file. In production, the Java translator would read the file and process a stream created by our existing system. In the unit tests for the XSLT code—written in JUnit—that same production Java code would read the same XSLT file, but would process a sample of hand-coded XML from a string written right into the unit test. Then the test would assert that the output string matched our expectations.

The college grad learned that XSLT could be unit-tested, and that the tests made it so much easier to write good XSLT. And I was able to sleep peacefully at night.

We turned that estimated 6 month rewrite into a 3-6 week dynamic (and fun) experience. This Black Swan was won over not only due to the well-factored state of the code, but also due to the collaborative practice of pair-programming.

And because XSLT is such a fussy declarative language, which does exactly what you tell it to do, I cannot imagine how we could have built it without Test-Driven Development techniques. Only by keeping all the previous XSLT-related unit tests running while working on a current failing test, could we have written it without a lot of very frustrating manual testing (XML…bleh!).

Or—had we tried to write tests afterwards—we would have encountered numerous interrelated defects that would, when fixed, spawn other defects. Only in the aggregate of a growing suite of tests could the XSLT be compelled to do all the things we needed it to do, without doing something wrong, or something ”extra” (also very bad, but far more difficult to spot).

Successfully Taming the Black Swans 

The arrival of a Black Swan User Story is probably inevitable on any software program (Agile or otherwise) running for more than six months. Of course, I can't guarantee it, but that’s been my experience.

But if your team and your code aren’t ready, the cost of implementing the Black Swan will be prohibitive, and it will either get repeatedly pushed lower on the queue, or will never make it into customers’ hands. Can you imagine if we had told the OTIS2 surgeons, “A button? That’ll take six months…” It would likely have never been scheduled in, and medical interns would still be paying for our negligence with papercuts and stapler accidents.

Summarizing the conditions common to all cases:

• The user-story was a surprise to everyone, including the Product owner/advocate who invented it. Ergo, "Black Swan” user-stories: Surprising, disruptive, potentially quite valuable, and potentially quite expensive. On a long-standing Agile team, such a story results from the Product advocate's knowledge of the software's existing capabilities, the team's skills and reliability, and the needs of the market. A Black Swan is frequently the innovative pivot, the wildly insightful customer request, or a clever repurposing. Once implemented, the Black Swan opens a whole new market segment, improves the flow of value in another area of the organization, or greatly aids in retaining critically important customers.

• The user-stories seemed at first to be nearly impossible to implement in any reasonable amount of time. Many team members saw the Black Swan as a major architectural change, based on their decades of experience on pre-Agile teams. Yet every one of these requests was completed in less than two weeks. In each case, the teams were able to prepare the code to facilitate the new enhancements by using rapid, often dramatic, refactorings.

• In each case, upon reflecting on root causes, the team involved determined that the code's existing flexibility was key to completing the story.

• Furthermore, that design was already in place due to disciplined efforts to refactor away "code-smells" (i.e., design problems); and to keep a design that was appropriate for the current behaviors of the application. There was no effort to prepare for some unknown future eventuality. By definition, you cannot prepare for a particular Black Swan. You can only “keep the lake clean” (to continue the metaphor).

• That design, in turn, was only possible due to a comprehensive and very fast suite of regression tests built continuously over all iterations.

• That suite was the result of disciplined Test-Driven Development. Every developer acknowledged that the suite would not have been as robust, or as fast, if unit tests had been retrofitted rather than written test-first.

Not a Guarantee

Though we expect Black Swan Stories to be rare, in my experience they have arrived roughly once or twice per year on every long-term software development team I’ve worked on, coached, or observed.

So, the arrival cannot be guaranteed, and the successful deployment of a Black Swan User Story that arrives cannot be guaranteed.

But the opposite is a near certainty. Based on a decade of pre-Agile software development experience, plus my experience coaching hundreds of Agile teams who “chose” not to use TDD, I have observed: Without very fast, comprehensive testing, teams' Black Swan stories were either repeatedly rejected as too expensive and dangerous, or took many months to complete, sometimes missing a critical competitive window. Some of those companies no longer exist.

Diligent Refactoring, facilitated by TDD, and resulting in emergent design, will keep the code good enough for what it does now, and ready to change clearly and easily. That’s how the Black Swans come to your clean, sparkling lake, land softly, and enrich your life.

[1] Bad joke. Unoriginal, too.

[2] We used to call them—in the politically incorrect days of yore—“God Objects”: A “God Object” knows how to do everything, and when you look at the code, you couldn’t help but exclaim, “Oh…God!”

[3] Wherever there’s an “AbstractFoo” there has to be a “FooImpl”, right??? (Yes, that’s sarcasm.)

[4] The original novice Java programmer had used Struts, presumably to give the application a Model-View-Controller architecture. Unfortunately, he had misused Struts by coupling every object to the Struts framework, and to each other, via the AbstractDAOLoanBean.