Technical Debt Isn’t Really a Metaphor
Many years ago, Ward Cunningham (who coined the term “technical debt” as far back as the late-1990’s, as far as I can determine) posted an excellent video on YouTube regarding refactoring and “debt”. If you haven’t seen it, I have it for you, below. But is Technical Debt simply a metaphor, or is he describing a real fiscal debt associated with software development activities?
Real debt?
I think of technical debt in terms of three categories:
Design Debt
The form of technical debt that Ward is referring to is noted by developers as poor, brittle design. Since a poor design is one that’s difficult to change or maintain, it will take developers longer to add features, and their changes will more likely result in defects. They will tip-toe slowly and cautiously within (or around) untested code, and even the best developer will inadvertently introduce the occasional defect.
Isn’t developer time expensive? Of course. So, in effect, this Design Debt is real debt. We may not be able to easily quantify this debt in terms of dollars (unless we routinely use Cost-of-Delay/Duration to aid our high-level decision-making process) which makes it even more risky.
Quality Debt
Design Debt feeds into another form of debt which is easier for the business to see: Quality Debt. This can be quantified as the number and severity of defects experienced by users. This metric has a fairly direct relationship with the financial impact of lost sales, time spent on support calls, the impact of negative word-of-mouth (I found a book with the bone-chilling title of Satisfied Customers Tell Three Friends, Angry Customers Tell 3,000), developer time spent debugging the product, and so on.
Often teams and whole organizations assume that Quality Debt is “the nature of software development.” That may have been so in the past, but we now have modern development practices that greatly reduce those numbers; if the teams are permitted to learn, use, and discover the whole-team benefits of those practices and skills.
Testing Debt
Unfortunately, what we usually try to do may just make it worse…
The functionality grows, the complexity grows, and it all has to be tested. We hire more testers, or give ourselves longer for the test “phase” of an iteration/sprint. This gated approach has been noted on many “agile” teams. If waterfalls are bad, why are more of them better???
As the stack of manual test-cases grows and grows, the poor test team works more overtime to run all the tests: Point-click-type-point-click-type… And even the most professional testers make occasional mistakes.
Later, the team will selectively run only the most “critical” tests each iteration, and save the whole suite for the pre-release shake-down cruise. But which tests–which product features–are not critical? Plus waiting until the end to do full testing creates a process gate (a la waterfall) resulting in delayed feedback and painful rework.
This is the third form of technical debt, Testing Debt. The anecdotal measurement of Testing Debt is the height of the stack of printed manual test-cases. A modern measure would be the time it takes to run the entire regression suite: How long after a change does it take the team to know whether or not they’ve broken something?
Is Testing Debt real financial debt? Well, there’s testers’ time and salary. Also, Testing Debt again feeds into Quality Debt. If you don’t assure that there are no defects, your customer will find them for you.
You know what you get when you add more features to existing defective code? Yeah, more broken features. It’s the software development equivalent of compounded interest. At 18% APR. Not good.
What can be Done?
First, in the immortal words of Douglas Adams: Don’t panic.
In 2002, on the University of Michigan Transplant Center’s two-year “OTIS2” rewrite of their aging Organ Transplant Information System; and after bi-weekly reminders that “a mistake could kill a patient”; we would release changes into production with only one day of additional (i.e., gated) exploratory testing.
After 15 years, this team is still active, still making enhancements, and–according to Richard Sheridan of Menlo Innovations–there hasn’t been a software-related emergency requiring developer overtime since 2004!
The majority of testing is built into an automated suite of unit-tests and story-tests that execute with each build. After only two years this team had about 20,000 such tests.
They all ran within 15 minutes.
Because the build/test feedback loop was so fast, multiple developers would confidently commit changes (“trunk-based”) every 2-4 hours.
New members of the team start working on code in the first two days on the team. They are told, effectively, “You can make any change to the system, as long as you run all the tests and they all pass. Oh, and you are always working with someone else on the team.”
In the story above, I’ve alluded to quite a few of the Essential Agile Developer Skills:
Test-Driven Development: Creating that fine-grained safety-net of thorough, comprehensive tests.
Pair-Programming: Working with one other developer to sustain quality, adherence to agreed-upon practices and idioms, enthusiasm, curiosity, and courage.
Continuous Integration: All pairs integrate changes into the release candidate branch (we used master) every 0-4 hours.
Sustainable Pace: We avoid coding when tired. People make mistakes when they are tired, even with Red Bull supplements.
Too Little Too Late?
Most teams I encounter have not been doing these things. Is it too late?
Consider: How does anyone get out of debt?
Acknowledge the existence of the debt.
Make regular payments.
Stop accruing more debt.
Examples of how agile teams can pay down technical debt:
Design Debt: When tasking and estimating stories, the team should consider the refactorings necessary to get any new story designed correctly. Those refactorings become explicit tasks for the first story that needs them.
Quality Debt: Set aside time within each iteration (Buffer Time!*) to explore and fix high-priority defects, or create a story for a group of related bugs. On a well-running agile project, each reported defect becomes a story of its own.
Testing Debt: Set aside time within each iteration for testers to automate those “critical” tests, or a particularly onerous set of tests. If you’re not ready for that, create a story (or a timeboxed “spike”) for the team (testers and developers working together) to explore automation tools, including record/playback tools as well as Behavior Driven Development tools such as Cucumber.
You and your teams will have to slow down the introduction and completion of additional stories/features. Think of this as consolidating debt under a lower interest rate, or simply paying down principal.
Ways to avoid new debt:
Design Debt: Developers, adopt Test-Driven Development (TDD) and refactor in tiny increments as you identify seemingly insignificant trends towards poor design. Whenever a task or story requires a new variation in your design, first refactor the appropriate piece of the design into one that follows the Open-Closed Principle for that variation. (That’s as concise an intro I can fit into a blog post about another topic. Perhaps another day. I’ll first need to scrape up a handful of old UML diagrams I use to describe OCP.)
Quality Debt: Pin down desired behavior immediately with fast, automated tests. Adopt TDD and Behavior Driven Development (BDD) to keep away any new defects. Avoid feeding Quality Debt from the other two forms of debt.
Testing Debt: Stop writing manual test cases, perhaps today. As soon as practical, build test scenarios using Cucumber (or something similarly flexible and readable by your Product folks). It’s also up to the team to find ways to keep all tests running fast. Usually, this requires Test-Doubles.
