Upload
kibo-nelson
View
62
Download
5
Embed Size (px)
DESCRIPTION
How Agile Ended Our Defect Report-Fix-Check-Rework Cycle. Richard Leavitt Rally Software Development [email protected]. Objectives for Today. Observation: Traditional workflows and project metrics evolved to manage large defect inflows during long testing phases Question: - PowerPoint PPT Presentation
Citation preview
Copyright 2003-2005, Rally Software Development Corp
How Agile Ended Our Defect Report-Fix-Check-Rework Cycle
Richard LeavittRally Software [email protected]
Slide 2 Copyright 2003-2005, Rally Software Development Corp
Objectives for Today
Observation: ∙ Traditional workflows and project metrics evolved to
manage large defect inflows during long testing phases
Question: ∙ How relevant are these management processes in
dramatically shorter development cycles?
Agenda:∙ Defect Workflow & Metrics in Test-Last∙ Our Agile Transition∙ New Dev & Test Processes, Workflows and Reports∙ Benefits and Bruises
Slide 3 Copyright 2003-2005, Rally Software Development Corp
Agile Doesn’t Change Our Goals
∙ Customers judge our software as high value and high quality
∙ Our business judges our efforts as timely and providing high ROI
∙ We like our jobs
Slide 4 Copyright 2003-2005, Rally Software Development Corp
Agile Doesn’t Change Our Questions
We desire:∙ A small set of useful and valid measures,
provided by ∙ Standard reports that everyone anticipates
and understands, with∙ The ability to drill down and roll up
ReadinessWhen can
we release?
StatusHow’s feature X
doing?ProgressWhat’s
blocking?
Slide 5 Copyright 2003-2005, Rally Software Development Corp
Before: Phased Development Process
Advantages:• Requirements and design
“up front”• Logical• Prescriptive• Plan based
Disadvantages:• Requirements “freeze” paradigm• Dreaded system integration phase• High defect counts and “surprises”• Hard to adapt to changing requirements• Plan based
Slide 6 Copyright 2003-2005, Rally Software Development Corp
One of My Test-Last Environments
∙ Release size: 16 FTEs x 14 Months ∙ Detailed work breakdown with task complete
dates∙ Unit testing? What’s that? ∙ QA drops with high bug counts (100’s)∙ Islands of information
– Project plan– Requirement docs
(MRD->PRD->SRS)– Test cases in Excel– Build results
– Test effort & results– Traceability matrix– Defect/Issue/Request
lists and metrics
Slide 7 Copyright 2003-2005, Rally Software Development Corp
High Defect Counts Add Weight
∙ “Push” workflow systems with complex rule engines for routing & signoffs
∙ Fine-grained permissions to control ownership and changes
∙ Lengthy bug reviews & management burdens
∙ DELAY!
Slide 8 Copyright 2003-2005, Rally Software Development Corp
But Defects are Easy to Count & Graph!Statistical gymnastics∙ Defect rates∙ Densities, cycle times∙ By phase∙ By owner, type,
status, severity, module, how found…
∙ In general, lots of totals, percentages, averages and ratios
Slide 9 Copyright 2003-2005, Rally Software Development Corp
And how else can we answer our questions?
After all, our typical experience goes like this:
“I’ve never heard of any direct measure of project status, program readiness for release, or progress toward meeting milestones, etc.
Instead, we’ve used surrogate measures and metrics.”
The Darker Side of MetricsDouglas Hoffman, Software Quality Methods, LLC
Slide 10
Copyright 2003-2005, Rally Software Development Corp
Problems with Defect Metrics
QA Drops
ReadinessWhen can
we release?
Slide 11
Copyright 2003-2005, Rally Software Development Corp
Problems with Defect Metrics
Bug Review Meetings
Close Rate1
Close Rate2Defects “closed” as• Deferred• Duplicate• Low priority• Can’t Reproduce• Enhancement Request
ReadinessWhen can
we release?
Slide 12
Copyright 2003-2005, Rally Software Development Corp
So, Phased Development Processes Encourage a Defect-Driven Approach
And its accompanying problems:
∙ Large defect counts with heavy process burdens, high management effort and delay
∙ Defect-based metrics that don’t relate well to the real questions we’re trying to answer₋All metrics programs cause unintended side
effects on our behaviors, but can we do better?
Slide 13
Copyright 2003-2005, Rally Software Development Corp
Why Agile Development?
∙ Agile promotes rapid delivery and feedbackover elaborate upfront planning
Short iterations give all stakeholders timely and regular visibility into readiness, status and progress
∙ Shared commitments simplify workflows…but everyone needs real-time status information!
∙ Agile handles change better ₋ Accepts late-binding requirements
₋ Continuous integration and early testing means less “integration hell” and associated defects & surprises
In ShortMinimum Process, Maximum Value
Slide 14
Copyright 2003-2005, Rally Software Development Corp
The Agile Paradigm Shift
Constraints
Estimates
Schedule
ScheduleCost
Features
Value /VisionDriven
The Plan drives cost & schedule The Vision drives features
Requirements
Cost
PlanDriven
Waterfall Agile
Slide 15
Copyright 2003-2005, Rally Software Development Corp
Moving to Iterative & Incremental
ProjectMgmt
DevelopmentProcess
Requirements& Design
Test & FixProcess
Critical Path through Phases
Critical Drop Schedule
Time Boxes
Freeze & Signoff
Last Phase Only
“Test What’s Working”
Acceptance Test Inside the Iteration
Manage Scope Creep
Detailed Design Inside Iteration
All Features in Parallel
Multiple Drops to QA
Increment at a time
Waterfall IterativeIterative andIncremental
ParallelAcceptanceTest Driven
Agile Development
Slide 16
Copyright 2003-2005, Rally Software Development Corp
Agile Team Practices
∙ Time box everything∙ Two levels of planning: rough for releases,
fine-grained for iterations∙ Just-in-time requirements elaboration∙ Focus on the highest priorities∙ Early and continuous testing∙ Try to remove roadblocks and defects
immediately
Slide 2
Slide 17
Copyright 2003-2005, Rally Software Development Corp
The New Planning Process
∙ Release cycle: 8 or 10 weeks (4 or 5 iterations)₋Prod Mgmt defines themes & features/stories
₋Dev provides rough “planning” estimates
₋Release Backlog is then negotiated & prioritized
₋“Hardening” iteration at the end tackles technical debt
∙ Iteration cycle: 2 weeks with one ½ day planning meeting₋Recap previous iteration
₋PM presents new and elaborated stories & use cases
₋Dev details tasks & estimates just-in-time
₋Priorities are set and the team commits
Slide 18
Copyright 2003-2005, Rally Software Development Corp
Story Acceptance is Common Goal
∙ One schedule, one budget, one team, but independent dev & test people and tasks
∙ Features are squeezed, NOT testing and schedule
∙ Dev codes highest priority requirements all the way to acceptance
∙ QA elaborates & automates acceptance tests in parallel with coding₋Dev-QA collaboration versus separation improves
code robustness and testability right away.
₋Quality no longer “tested in”
Slide 19
Copyright 2003-2005, Rally Software Development Corp
Early & Continuous Testing Drives Down Defect Counts
∙ Unit tests run on code check-in ∙ Nightly build runs automated acceptance
tests∙ Team tries to resolve failures immediately
∙ When combined with short iterations, we get∙ Tiny defect counts (dozens, max) with little
management burden and team overhead
In Short, We Stay Close To Shippable Code
Slide 20
Copyright 2003-2005, Rally Software Development Corp
Iteration Decisions & Metrics
Meeting iteration intent and schedule means responding to changing realities∙ Should we cut or adjust any features?
₋What progress have we made on each story?
₋How much is there left todo?
₋What’s our remaining budget?
∙ What’s standing in our way of acceptance?₋Blocks?
₋Which story card tests are failing?
₋Are any defects open against our iteration stories?
Slide 21
Copyright 2003-2005, Rally Software Development Corp
New Iteration Status ViewsReadinessWhen can
we release?
StatusHow’s feature X
doing?ProgressWhat’s
blocking?
24
Slide 22
Copyright 2003-2005, Rally Software Development Corp
Tie Acceptance Test Results To Story
ProgressWhat’s
blocking?
Slide 23
Copyright 2003-2005, Rally Software Development Corp
So, Failing Tests Replace Many Defects for Simple, Pull-based Workflow
∙ Dev and Test now communicate via failing tests instead of routing defects
Test-driven Advantages₋Failures are easy to reproduce
₋Failures are transient, less management overhead
₋We’re done when the tests pass
₋Lighter process with less delay
Slide 24
Copyright 2003-2005, Rally Software Development Corp
Summary: Instead of Focus on Defects
Slide 25
Copyright 2003-2005, Rally Software Development Corp
We focus on Acceptance
Slide 26
Copyright 2003-2005, Rally Software Development Corp
Summary: Instead of Complex, Push-based Workflow
Slide 27
Copyright 2003-2005, Rally Software Development Corp
We use Simpler, Pull-based Workflows
Slide 28
Copyright 2003-2005, Rally Software Development Corp
Benefits and Bruises
∙ Agile has dramatically changed our project planning, tracking and workflow. We are more responsive to new requests and late changes.
>Everyone needed education on paradigm shift
∙ Everyone focused on incremental, meaningful functionality, not defect find/assign/fix/regress
>Demands high collaboration and real-time status. >Upstream analysts must provide steady flow of
business needs and priorities
Slide 29
Copyright 2003-2005, Rally Software Development Corp
More Benefits and Bruises
∙ Time boxes mean we always know when we’re going to release
>Though the scope may be slightly different!
∙ QA plays large role in requirement elaboration and defect prevention.
>Early resistance to TDD – required a change agent (Mike Cohn)
>Struggles to automate tests inside iteration>New skills required in test design and automation.
Slide 30
Copyright 2003-2005, Rally Software Development Corp
More Benefits and Bruises
∙ Tiny bug lists (dozens) with little management overhead>Most “bugs” found during exploratory testing in
hardening iterations. >Re-factoring tests is critical to keep up velocity.
∙ Simple, pull-based workflow with team members actively grabbing the next most important item off the stack
>Need culture of shared ownership that “takes responsibility”
Slide 31
Copyright 2003-2005, Rally Software Development Corp
Suggestions for Getting Ready
∙ Suggested Process₋ Brown bag discussions
₋ Guest lectures & webinars
₋ Local Agile/XP user groups
₋ Agile Conferences∙ Agile2005 – Denver, July
05
∙ Where to start on the web₋ Agile Alliance Web Site
www.agilealliance.org
₋ Rally Ecosystem Pages www.rallydev.com/ecosystem
∙ Suggested Reading₋ Agile Project Management
w/ SCRUM www.controlchaos.com/
₋ Agile & Iterative Development www.craiglarman.com
₋ Lean Software Development www.poppendieck.com/
₋ Agile Project Management www.jimhighsmith.com/
₋ Tactical Management of Iterative Development www.rallydev.com
Slide 32
Copyright 2003-2005, Rally Software Development Corp
Questions?
Slide 33
Copyright 2003-2005, Rally Software Development Corp
Acknowledgments
∙ The Darker Side of Metrics, Douglas Hoffman, PNSQC 2000
∙ It’s Not About the Bugs, Harry Robinson, Stickyminds 2003
∙ Managing Software Requirements: A Use Case Approach, Dean Leffingwell, 2003
∙ The Test Manager at the Project Status Meeting, Brian Marick, Steve Stukenborg, 1997