What are the ‘Agile Engineering Capabilities,’ ‘Scrum Developer Practices,’ or ‘Software Craftsmanship practices’?

These are three (of many) common names for a set of practices assembled in the mid-1990s to support and improve Agile software development efforts. They are a collection of practices that help teams work in an incremental, iterative, and simultaneous way (i.e., all functional departments working towards a clear goal on a continuous basis, and not waiting on each other at explicit or implicit process gates).

Most were first combined as a complete collection of practices within Extreme Programming (XP), a popular Agile method, around 1996.

The first set of questions are for a general audience, the second set (after the “break”) are developer-specific.

Why do we need them? How do they solve my problems?

What I’ve observed repeatedly on numerous Scrum teams is a slowing of value delivery, sometimes after a mere 5 or 6 sprints (even on greenfield projects). Developers find it more and more challenging to provide new releasable-quality features, and the burden of testing every feature—preferably every single one since the inception of the product—also increases, and usually falls heaviest upon testers. 

I call this the Agilist’s Dilemma:  The very thing that helps us deliver highest value continuously can also have a systemic “back-pressure” that negatively impacts the quality and maintainability of the product. This further reduces the ability to deliver quickly. If left unchecked, progress could cease entirely. 

It’s neither because of a shortcoming in Agile methods, nor is it a great mystery. The very act of software development, itself, must change alongside our development methodologies.  Software development teams that use Scrum or Kanban get the most benefit out of their Agile methods when they replace older, gated development activities with practices designed to work within shorter feedback loops.

The Agilist’s Dilemma is due to the nature of software: It becomes more and more difficult to add new features onto old without modifying the older code, or creating a lot of unmaintainable duplication and unneeded complexity in the system.

We need ways to keep our code truly well-designed. In other words:  Easily changeable.

Would you describe a few? Which are most critical?

Refactoring

In order to adapt to changing market forces, Agile teams cannot try to plan or predict all features up-front. In order to add new features to code designed “for the moment” by an Agile team, we must be able to reshape the design so that it supports the new functionality.

Thus, we must have the ability to refactor. Over the years, I’ve come to see refactoring as the central Agile developer skill: This is the practice that helps whole teams innovate. All other development practices support refactoring in some way. 

BDD and TDD

Test-Driven Development (TDD) is a novel way to write code incrementally. This practice creates a safety-net of automated, comprehensive tests that allow for rapid, fearless refactoring; just the sort of thing that’s needed to maintain and support innovative features, without compromising quality.

Behavior-Driven Development is a similar practice for writing business-facing tests. It can also help build confidence in the quality of the code, but its true power lies in helping the Agile team clearly circumscribe a feature for the understanding and agreement of all Agile team members. The conversations that occur, and the ubiquitous language that develops via BDD are at least as important as the automated testing, itself. 

Whereas either practice would suffice to create a safety net needed to support refactoring, the difference in emphasis is important:  BDD encourages critical conversations about features and scope, whereas TDD is a core developer practice used to craft, change, and design the product’s code with greater confidence and far fewer defects. 

Complex and interdependent business models and business rules are best supported by both BDD practices together, starting as early in the development process as possible. Within a complex business domain, TDD can be used to break up dependencies between business rules and their objects, and thus completely avoid the need for numerous slow, combinatorial test scenarios. Teams that use both find a sweet spot: Rather than relying solely on an explosion of BDD examples, the team can drop into a TDD cycle to more easily cover all permutations, or to build a supportive substructure of the code that has no immediate mapping to the customer-facing ubiquitous language.

Simpler business domains and smaller products (e.g., an early version of Twitter) would be fine with one practice or the other. If your company plans to grow this area of the business in the future, plan to try both in concert with each other. 

Continuous Integration

Teams larger than two people, when building software in a rapidly-changing, ever-growing code-base, also need to provide their changes to others on the team; developers and testers.

This is particularly important when teams “swarm” on the one or two highest priority Product Backlog Items (i.e., PBIs, e.g. “user stories”) in the sprint backlog.

This is done via Continuous Integration, the practice of fully integrating development changes into source-control and build systems, many times per day. This provides a single unified snapshot of the direction, quality, and fit-for-purpose of the product. This is also a key “DevOps” practice, and Continuous Delivery and Continuous Deployment both rely on this core developer practice.

Pair Programming

We ask two people to work together—truly together at one computer whenever possible—on the product’s programming tasks. In my experience, this practice provides far more than any academic study has been able to ascertain. Interestingly, this—certainly the most contentious Agile developer skill—is the one that best facilitates the adoption and mastery of all other developer skills and practices.

And there are quite a few other Agile developer skills, some typically adopted more contextually than the sample of core practices described above.

Why don’t developers already do these things? They know how to write software, right?

Yes, but not all have experience doing so in a highly iterative and incremental multi-person environment.

I’ve encountered perhaps one in a hundred developers who have experienced TDD or pair-programming in college.  Most university curricula still tend to focus on data structures, algorithms, compiler theory, automata theory, and other (very fascinating!) topics. Students are often expected to work independently on labs. Formal software testing is, at best, done by the professor or TA while grading the assignment.

Then, when folks graduate, they discover that most software development does not involve the invention and design of fresh new algorithms, or even the implementation of good old algorithms.  Many key algorithms are built into the libraries of our modern programming languages.

Whereas we’d all like to be the next Zuckerberg, the reality is that even the most talented developers can expect to spend many years implementing business rules and customer features for apps or APIs. These are often interesting, innovative, and lucrative products, but they rarely push the boundaries of computer science, itself.

Though our projects, products, and programs are often very interesting work, if developers are seen as fungible resources and unconsciously incentivized to do counter-productive work (e.g., copy/paste/modify), or increasingly repetitive tasks; morale will suffer, performance will suffer, craftsmanship and quality will suffer, and your product will suffer. Everyone, including you, will be miserable.

One of the important skills I convey in my technical courses is how to establish a disciplined practice. These skills require self-discipline, and about three or four weeks to establish good habits.

Many teams try to learn on their own.  Some developers will have success with this approach, but they don’t always have the influence over the whole team required to sustain a viable team practice. Even seasoned lead developers typically do not have experience with effective training techniques.  

I like to think of my courses as pragmatic, intensive, in-depth, challenging, and immensely fun post-grad/continuing-education courses for the professional developer.

What do I, as leader and manager, need to do to help my teams be successful?

It can take a team one or two months after training in these practices to get used to them and to see some clear, measurable benefit.  That first month or two will slow the team down quite a bit.  Firstly, be prepared for that.

Time and Trust

Unfortunately, most teams, and their leaders, give up before then.  When teams abandon the practices, it’s usually because they don’t feel they have the time or the autonomy necessary to learn and explore these practices. Please trust your team to act professionally.

Scrum already asks us to work towards transparency, empiricism, honesty, and trust. Continue to foster those values in your teams and at all levels of leadership. All important values grow in both directions, as relationships between people.

Co-define Success

As an Agile leader, you likely see more of the “big picture” than the team has time to investigate.

By clarifying and frequently referring back to the product vision, all parties will remain enthusiastic, focused, and professional. Great software teams love what they do, and they also appreciate the transparency around planning, priorities, and profits.

Additionally, some of these practices—Behavior Driven Development in particular—require your, or a product advocate’s, frequent participation.

Clearly there are real costs to implementing these practices. What are those costs, and how do I recoup the investment?

Many of these practices help shorten critical feedback loops, often from months to days, or even minutes. Even without formal metrics, you should be able to perceive marked improvements in value delivered, quality, and team morale.

Of course, there are also costs. TDD, for example, generates unit-testing code at a ratio of about 2/1 or even 3/1 compared to shippable implementation code. Pair programming clearly appears to use two people to do a single task, so how could the benefits outweigh the costs?

Increases Throughput of Quality and Value

Both TDD and pair programming greatly reduce rework. TDD has been shown to reduce defect density by 40-90%, with about 15% more time to complete tasks, initially. That much reduction in defects avoids broken features (and unhappy customers), time spent searching for the defect, fixing, and then re-testing the defective area (some developers tell me this is what they do half the time!).

I’ve found through experience that TDD is only costly at the start of the learning curve. After developers get used to it, TDD becomes their de facto tool for building code, and the accelerative capabilities outpace the costs. 

Pair Programming is a practice that’s a bit more subtle. It, too, has been known to reduce defects by almost the same amount as TDD.  If you’re already doing TDD, would you need both?

It’s a valid question.

Academic studies of pair programming miss the longer-term systemic benefits of having an idea, idiom, design, or clever technique spread like wildfire through the team. Teams develop real on-the-job proficiency from each other; commitment and support in the other practices; and pragmatic, work-related, team-building social interaction.

If you manage a team of people in cubes, you’ve probably noticed it’s never easy to tell whether folks are working, or mentally “checked-out.” When you observe a team who routinely pairs, you can tell the difference. You will see a marked improvement in team communication, camaraderie, and morale. A happy team is a productive team is a happy team.

Also, as I suggest to the developers:  Try it for a month or two. Then ask yourself, “Did that experiment really cost me twice as much?” Probably not.

Reduces Other Costs

Often the benefits of Agile developer practices are gained by the reduction or replacement of some other practice that doesn’t fit quite as well in an iterative, incremental, and parallel-work methodology like Scrum.

For example, let’s examine the practice of holding code reviews. This is one of the best tried-and-true ways to find defects in a heavily-gated or hierarchical methodology. I used it for the ~15 years prior to my introduction to Agile methods. I resisted giving up such a powerful, collaborative practice! (We used to do code reviews in a room of about 4-6 people.)

Yet many teams discover their constraint is in the code reviews, particularly if all reviews must involve the architect or tech lead. If only one or two people must review all code before a PBI is complete, then they will find themselves in the unpleasant position of being the limiting factor of team productivity. Also, they must never take a sick day or vacation! If they did, productivity would stop.

Instead, I recommend that tech leads pair-program with developers, use TDD and BDD practices, and coach the team’s developers on the tools, architectural direction, idioms, and expectations for the product.

If a team pairs up on all production code (that is, the code that actually goes into the product), and they adopt Collective Responsibility (also known as “collective code ownership”), then traditional code reviews will become less and less necessary, because traditional defects and standards-violations will become extinct.

Avoids Lost Opportunity Costs

Primarily, the Agile developer skills reduce the cost-of-delay of lingering defects and delayed time-to-market. They all support refactoring, which allows Agile teams to adapt their existing design to fit new, sometimes unexpectedly innovative features into the existing product, without breaking any existing functionality.  This ability to rapidly implement an innovative change is critical to the success of any modern software development project, program, or product.

As a developer, I’m concerned that these practices will take more time. I don’t have more time! Are there benefits to me?

You may notice that most of the benefits described above are product/organization -oriented. Those can impact developers in a positive way, but there’s often a significant lag. Fortunately, all of these practices have a variety of immediate benefits for developers, as well.  (And, of course, these developer benefits are often the cause of the organizational benefits.)

Refactoring

I’ve noticed that some of the best-designed, easiest-to-maintain code I’ve ever seen (or written, myself) has been as a result of continuous, just-in-time, refactoring.

By “deferring” certain design decisions until they’re needed, you’re not doing as much up-front design that may or may not be needed. This is often called out as its own skill: Incremental Design.

Test-Driven Development

This skill can seem really weird at first, but it’s a very logical discipline. We developers ask: What new thing does the code need to do? What inputs and circumstances will give what outputs? How do I expect client code to call this object or function?

You design the call to the new code by asking those questions first, and creating a small engineering specification (it’s not really a “test” until it passes) for the new functionality.

This practice gives a lot of instantaneous feedback, catching tiny logic errors right away, so we don’t have to over-design (or argue over) the implementation.  And we avoid having to return to the implementation months later to find the defect that was found by the customers.

Also, the implementation changes in response to one simple request at a time, rather than having us juggle multiple concerns while writing or changing code. (That juggling was possible with the simple code I wrote in the 80s and early 90s. But software has become far more complex, particularly when there’s more than one person writing code.)

It’s eventually much easier to simply “think in tests” rather than labor over all possible contingencies without any trace of executable examples (the old way);  and now you end up building that fast safety-net of tests that will allow for more confident refactorings later on.

TDD and refactoring, alone, have saved me and my teams days per week in (a) time spent searching for defects; (b) tracing them in the debugger; (c) figuring out what we had originally intended for that code, because we can barely remember why we wrote it that way, ultimately because months had gone by; (d) and the re-work required to get that code right (without breaking anything else!). All of this debugging is a significant waste of your time, plus it’s not all that much fun, is it?

Behavior Driven Development

BDD gives the whole team the opportunity to collaboratively clarify business requests. Fewer debates over scope-creep, “works as designed,” and “that’s not what I meant” occur on a team that starts building software with given/when/then examples.

Pair Programming

“I can’t focus with all that noise,” “What if the other person slows me down?” “I’m an introvert…” Yep, I’ve heard it all.

Truth is, most developers find topical conversation—i.e., “talking it out”—helps them focus, helps them see alternative solutions, helps them deal with various interruptions, and helps them keep up their energy and enthusiasm for the work.

I suggest trying it for a month. Most people see the benefits, and even enjoy it. You may be an introvert, but regarding pairing, that means you don’t want to be sitting in a noisy crowd. If everyone on your team pairs, no one seems to notice the noise.

And you’ll be able to recuperate from a fun but exhausting day at the office using this next practice…

Sustainable Pace

We work until our focus starts to fade, then we leave the office, both physically and mentally. This avoids defects, burn-out, health problems, family troubles, and an ever-increasing pressure to sling more and more lines of code.

There may be rare occasions for overtime or a few work-from-home hours, but on well-running and successful Agile teams, they are rare.  One client I worked with recently reported that they haven’t had a single incident requiring a developer to work in the evening or on a weekend since 2004. More than a dozen years without a “developer emergency.” And that’s on an application that helps surgeons make life-or-death triage decisions.

Where to, next?

Take a look at the brief descriptions of my courses on the home page, or perhaps consider some coaching for your team(s).