UW Agile CP202 Class 3 Managing Software Debt

UW Agile CP202 Class 3:Managing Software DebtContinued Delivery of High Value as Systems Age

Chris SterlingTechnology Consultant / Agile Coach /

Certified Scrum TrainerSterling Barton, LLC

Web: www.SterlingBarton.comEmail: [email protected]

Blog: www.GettingAgile.com Follow Me on Twitter: @csterwa

Hash Tag for Presentation: #swdebt

1Monday, May 10, 2010

http://www.SterlingBarton.com

http://www.SterlingBarton.com

mailto:[email protected]

mailto:[email protected]

http://www.GettingAgile.com

http://www.GettingAgile.com

Topics Being Covered

Problems Found with Aging Software

Software Debt Explained

Technical Debt

Quality Debt

Configuration Management Debt

Design Debt

Platform Experience Debt

The Wrap Up

A Story of What is Possible


Problems Found with Aging Software

Software gets difficult to add features to as it ages

Business expectations do not lessen as software ages

Software must remain maintainable and changeable to meet needs of business over time


Lack of emphasis on software quality attributes contributes to

decay


Like-to-Like Migrations

“It will be easy since we worked on the original version” - although we understand the domain we will be fighting with new features, technology, tools, and processes

“We don’t have any other options” - Refactoring and test automation are potential alternatives to like-to-like migrations.


Limited Platform Expertise

Risk and costs increase as expertise becomes more limited for aging software platforms.


Costs for Release Stabilization

0

125000

250000

375000

500000

Release 1Release 2

Release 3Release 4

Release 5Release 6

Cost of Fixing Defects Cost for Feature Dev


Regression Costs - Manual vs.


Extreme Specialization

Knowledge and capability to maintain legacy software decays with time

Costs to maintain rarely used software platforms are higher

Leads to waiting for people in specialized roles to finish their tasks in support of development effort


Software DebtCreeps into software slowly and leaves organizations with liabilities


Software Debt Creeps In

Shows a relatively new system with little software debt accrued.



An aging software system slowly incurs significant debt in multiple functional areas.



An older system has accrued significant debt in all functional areas and components.


Managing Software Debt – an


Managing Software Debt

It is impossible to stop software debt from creeping into our software

Managing debt in software is based on putting frequent feedback mechanisms in place for code, quality assurance, configuration management, design, and organization of people on project team

Feedback mechanisms should be frequent, automated, and easy to use to support their continued use or modified to new needs


Types of Software Debt

Technical Debt

Quality Debt

Configuration Management Debt

Design Debt

Platform Experience Debt


Technical DebtIssues in software implementation that will impede future development

if left unresolved


* Ward Cunningham on “Technical Debt”

Technical Debt includes those internal things that you choose not to do now, but which will impede future development if left undone. This includes deferred refactoring.

Technical Debt doesn’t include deferred functionality, except possibly in edge cases where delivered functionality is “good enough” for the customer, but doesn’t satisfy some standard (e.g., a UI element that isn’t fully compliant with some UI standard).

* Ward Cunningham - “Technical Debt” - http://c2.com/cgi/wiki?TechnicalDebt


http://c2.com/cgi/wiki?TechnicalDebt

http://c2.com/cgi/wiki?TechnicalDebt

My Definition of “Technical Debt”

“Technical debt is the decay of component and inter-component behavior when the application

functionality meets a minimum standard of satisfaction for the customer.”


Test-Driven Development

Test-Driven Development (TDD)

The “Flow”

Integrating with Team

Difficulties with TDD

Example TDD Session

Refactoring


Questions for You...

How many of you are practicing TDD?

What issues are you having currently with TDD?


TDD - Basic “Flow”



Write Failing Test



Write Failing Test

Make Test Pass



Write Failing Test

Make Test Pass

Refactor to Acceptable Design



Write Failing Test

Make Test Pass



Steps of TDD “Flow”

1. Write a programmer test

2. Run the programmer test and it should fail (red bar)

3. Write just enough code to make failing test pass

4. Run programmer test successfully (green bar)

5. Refactor code to an acceptable design (green bar)

RISK: Pushing out refactoring to add more code first. Don’t forget to refactor frequently to an acceptable design.


Be a User --“What should the software do next

This question helps you to decide what the next programmer test should model





Write Failing Test



Write Failing Test

Make Test Pass



Write Failing Test

Make Test Pass

Integrate with Team



Write Failing Test

Make Test Pass


Integrate with Team



Write Failing Test

Make Test Pass


Integrate with Team



Write Failing Test

Make Test Pass


Integrate with Team


Need-Driven Design



What Causes TDD to be so Difficult to Implement Well?


Impediments to TDD

This simple technique for developing software, when used in a disciplined manner, will enable individuals and teams to improve software quality. The discipline necessary to do TDD is not easily attainable. Following is a list of environmental issues that lowers chances of effectively implementing TDD approach.


Schedule Pressure

Pressure from management and stakeholders to release based on an unreasonable plan. Integrity of the software is always sacrificed when the plan is inflexible and unwilling to incorporate reality.


Lack of Passion

Lack of passion for learning and implementing TDD effectively on Team. The amount of discipline required makes passion extremely helpful.


Insufficient Experience

Insufficient team member ratio who have experience doing TDD well. When there is little experience on or coaching for the team success is minimal.


Legacy Code

If the software’s design is poor or is difficult to test then finding a starting point could seem impossible. If legacy code is large and contains no, or only minimal, test coverage then disciplined TDD will not show visible results for some time.


Introducing TDD Approach

To successfully adopt TDD, it is important to manage these environmental issues. This could include managing expectations, providing the Team support from a coach, and enabling sufficient learning of tools and techniques while working with an existing code base.



Quick Example


Jitter – Example TDD SessionFake micro-blogging tool named “Jitter” is made by Seattle-based fictitious company that focuses on enabling coffee injected folks to write short messages and have common online messaging shorthand expanded for easy reading. The user story we are working on is:

So it is easier to read their kid’s messages, Mothers want to automatically expand common shorthand notation

The acceptance criteria for this user story are:

LOL, AFAIK, and TTYL are expandable

Expand lower and upper case versions of shorthand


Expand LOL to “laughing out loud”

public class WhenUsersWantToExpandMessagesThatContainShorthandTest {

@Test

public void shouldExpandLOLToLaughingOutLoud() {

JitterSession session = mock(JitterSession.class);

when(session.getNextMessage()).thenReturn("Expand LOL please");

MessageExpander expander = new MessageExpander(session);

assertThat(expander.getNextMessage(),

equalTo("Expand laughing out loud please"));

}

}

public class MessageExpander {

public String getNextMessage() {

String msg = session.getNextMessage();

return msg.replaceAll("LOL", "laughing out loud");

}


But wait…what if…?

What if LOL is written in lower case?

What if it is written as “Lol”? Should it be expanded?

What if some variation of LOL is inside a word?

What if characters surrounding LOL are symbols, not letters?

Write these down as upcoming programmer tests as comments so I don’t forget them.

// shouldExpandLOLIfLowerCase

// shouldNotExpandLOLIfMixedCase

// shouldNotExpandLOLIfInsideWord

// shouldExpandIfSurroundingCharactersAreNotLetters


Expand LOL If Lower Case

@Test

public void shouldExpandLOLIfLowerCase() {

when(session.getNextMessage()).thenReturn("Expand lol please");



equalTo("Expand laughing out loud please"));

}

This forced use of java.util.regex.Pattern to handle case insensitivity.



return Pattern.compile("LOL”, Pattern.CASE_INSENSITIVE)

.matcher(msg).replaceAll("laughing out loud");

}


Don’t Expand “Lol” – Mixed-Case

@Test

public void shouldNotExpandLOLIfMixedCase() {

String msg = "Do not expand Lol please";

when(session.getNextMessage()).thenReturn(msg);


assertThat(expander.getNextMessage(), equalTo(msg));

}

This forced me to stop using Pattern.CASE_INSENSITIVE flag in pattern compilation. Only use “LOL” or “lol” for replacement criteria.



return Pattern.compile("LOL|lol").matcher(msg)

.replaceAll("laughing out loud");

}


Don’t Expand “LOL” If Inside Word

@Test

public void shouldNotExpandLOLIfInsideWord() {

String msg = "Do not expand PLOL or LOLP or PLOLP please";

when(session.getNextMessage()).thenReturn(msg);


assertThat(expander.getNextMessage(), equalTo(msg));

}

The pattern matching is now modified to use spaces around each variation of valid LOL shorthand.

return Pattern.compile("\\sLOL\\s|\\slol\\s").matcher(msg)



Expand “LOL” If Not Inside Word

@Test

public void shouldExpandIfSurroundingCharactersAreNotLetters() {

when(session.getNextMessage()).thenReturn("Expand .lol! please");



equalTo("Expand .laughing out loud! please"));

}

The final implementation of pattern matching code looks as follows.

return Pattern.compile("\\bLOL\\b|\\blol\\b").matcher(msg)



Merciless Refactoring

Refactoring: a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior.*

Merciless: having or showing no [mercy - showing great kindness toward the distressed]

Relieve your distressed code through kindness and disciplined restructuring

* From http://www.refactoring.com/


http://www.refactoring.com

http://www.refactoring.com

Where to Start Refactoring?Does this change directly affect feature I am working on?

Would change add clarity for feature implementation?

Will change add automated tests where there are none?

If “yes” to any question above, ask following question to decide if you should work on it now:

At first glance, does refactoring look like a large endeavor involving significant portions of the software’s components?


When to Stop Refactoring?

Am I refactoring code not directly affected by feature?

Is other code directly affected by feature I am working on that has not been refactoring sufficiently?

If refactoring is exploding feature estimate given to Customer then I should bring it up to Team to decide how we should progress

If Team decides that refactoring can be absorbed into current iteration without affecting delivery on our commitments then continue refactor


Automate Testing to Support

Refactoring cannot be done effectively without automated tests surrounding code

Start by creating automated test which fails

If difficult to create at unit level look at automated acceptance tests from functional perspective

Over time look for ways to create automated unit tests


Automated Tests

Application


Automated Tests

New Feature

Application


Automated Tests

New Feature

Application

Design Changes


Automated Tests

New Feature

Application

Design ChangesAutomatedRegressionTest Run


Automated Tests

New Feature

Application

Design ChangesAutomatedRegressionTest Run


Principles of Agile Software Quality

The system always runs

No code is written without a failing test

Zero post-iteration bugs

* From “Flawless Iterations” presented by Alex Pukinskis at Agile 2005


Quality DebtA lack of quality, either technical or functional, will lessen value per

feature added over time


Accrual of Quality Debt with Releases


Break/Fix Only Prolongs the Agony


Effect of Project Constraints on


Effect of Project Constraints on


Acceptance Test-Driven Development


A Fit Case StudyCost reduction using Fit for test automation and data conversion


Manual Regression Testing

Testing was taking 75 person hours during 2 full test runs consisting of:

Comprehensive manual regression testing

Data conversion and validation

Cost for testing was $17,000 each iteration


Introducing Fit into Testing Process

After 8 iterations team had introduced healthy amount of Fit fixtures and automated tests

Reduced 70+ hour test runtime down to 6 hours which now included:

Fit automated regression testing

Data conversion and validation automated with Fit fixtures

Reduced cost of testing each iteration from $17,000 to $7,000


Configuration Management DebtCreating unpredictable and error-prone release management


Traditional Source Control



Main Branch



Main Branch

Version 1Branch

Integrate forVersion 2

CodeComplete



Main BranchDebt

Death March

Version 1Branch


CodeComplete



Main BranchDebt

Death March {Debt accrues quickly within stabilization periods

Version 1Branch


CodeComplete


Flexible Source Control Management



Main Branch



Main Branch

Version 1



Main Branch

Version 1 Version 2



Main Branch

Version 1 Version 2{Not Easy! Must have proper infrastructure to do this.


Continuous Integration


Scaling to Multiple Integrations


Design DebtDesign decays when not attended to so design your software

continually


* Abuse User Stories

Implement Securityfor User Information

* From “User Stories Applied” presented by Mike Cohn Agile 2006


* Abuse User Stories

Implement Securityfor User Information

As a Malicious Hacker I want to steal credit card information so that I can make fraudulent charges

* From “User Stories Applied” presented by Mike Cohn Agile 2006


Platform Experience DebtSilos of knowledge and increased specialization will increase cost of

maintenance over time


Team Configuration Patterns

Virtual Architect Pattern

Integration Team Pattern

Component Shepherd Pattern

Team Architect Pattern



EnterprisePlanning



Pros

Share architecture ideas and needs across teams

Based on verbal communication

Cons

Usually singles out special Team Member role

Could lead to top-down architecture decisions

IT may gain extensive influence and begin to run Product



Feature Development

Integrate Features

All features are implemented

and integrated every iteration



Pros

Reduces complexity on Feature Teams

Forces delivery from Integration Team instead of interface and deployment designs

Cons

Perpetuates specialized roles

Don’t always work on highest value Product Backlog items





Pros

Share more knowledge within organization to minimize platform experience debt

Work on highest value Product Backlog items

Cons

Not always optimal as using individual knowledge

Difficult to learn multiple systems across Teams





Pros

Team owns architecture decisions

Decisions are made close to implementation concerns

Cons

May not have appropriate experience on Team

Team could get “stuck” on architecture decisions


What is possible?High quality can be attained and should enable accelerated feature

delivery at same time.


A Story: Field Support Application

2000+ users access application each day

Application supports multiple perspectives and workflows from Field Support Operations to Customer Service

Team of 5 people delivering features on existing Cold Fusion platform implementation

Migrating to Spring/Hibernate in slices while delivering valuable features

36 2-week Sprints, 33 production releases, and only 1 defect found in production

So, what was the defect you say? Let me tell you…


Lets wrap this up...What should I take away from this?


Principles for Managing Software Debt

Maintain one list of work

Emphasize quality

Evolve tools and infrastructure continually

Improve system design always

Share knowledge across the organization

And most importantly, get the right people to work on your software!
