Business Value of
Agile MethodsBenefits of Testing Early & Often
Dr. David F. Rico, PMP, ACP, CSM
Twitter: @dr_david_f_ricoWebsite: http://www.davidfrico.com
LinkedIn: http://www.linkedin.com/in/davidfricoFacebook: http://www.facebook.com/profile.php?id=1540017424
Dave’s Agile Articles: http://davidfrico.com/agile-message.doc
Author Background DoD contractor with 28+ years of IT experience B.S. Comp. Sci., M.S. Soft. Eng., & D.M. Info. Sys. Large gov’t projects in U.S., Far/Mid-East, & Europe
2
Published six books & numerous journal articlesAdjunct at George Wash, UMBC, UMUC, ArgosyAgile Program Management & Lean DevelopmentSpecializes in metrics, models, & cost engineeringSix Sigma, CMMI, ISO 9001, DoDAF, & DoD 5000Cloud Computing, SOA, Web Services, FOSS, etc.
Today’s Whirlwind Environment
3
OverrunsAttritionEscalationRunawaysCancellation
GlobalCompetition
DemandingCustomers
OrganizationDownsizing
SystemComplexity
TechnologyChange
VagueRequirements
Work LifeImbalance
InefficiencyHigh O&MLower DoQVulnerableN-M Breach
ReducedIT Budgets
81 MonthCycle Times
RedundantData Centers
Lack ofInteroperability
PoorIT Security
OverburdeningLegacy Systems
ObsoleteTechnology & Skills
Pine, B. J. (1993). Mass customization: The new frontier in business competition. Boston, MA: Harvard Business School Press. Pontius, R. W. (2012). Acquisition of IT: Improving efficiency and effectiveness in IT acquisition in the DoD. Second Annual AFEI/NDIA Conference on Agile in DoD, Springfield, VA, USA.
Traditional Projects
4
Big projects result in poor quality and scope changes Productivity declines with long queues/wait times Large projects are unsuccessful or canceled
Jones, C. (1991). Applied software measurement: Assuring productivity and quality. New York, NY: McGraw-Hill.
Size vs. Quality
Def
ect
Den
sity
0.00
3.20
6.40
9.60
12.80
16.00
0 2 6 25 100 400
Lines of Code (Thousands)
Size vs. Productivity
Cod
e P
rodu
ctio
n R
ate
0.00
1.00
2.00
3.00
4.00
5.00
0 2 6 25 100 400
Lines of Code (Thousands)
Size vs. Requirements Growth
Per
cent
age
0%
8%
16%
24%
32%
40%
0 2 6 25 100 400
Lines of Code (Thousands)
Size vs. SuccessP
erce
ntag
e
0%
12%
24%
36%
48%
60%
0 2 6 25 100 400
Lines of Code (Thousands)
Global Project Failures
5Standish Group. (2010). Chaos summary 2010. Boston, MA: Author.Sessions, R. (2009). The IT complexity crisis: Danger and opportunity. Houston, TX: Object Watch.
Challenged and failed projects hover at 67% Big projects fail more often, which is 5% to 10% Of $1.7T spent on IT projects, over $858B were lost
16% 53% 31%
27% 33% 40%
26% 46% 28%
28% 49% 23%
34% 51% 15%
29% 53% 18%
35% 46% 19%
32% 44% 24%
33% 41% 26%
0% 20% 40% 60% 80% 100%
1994
1996
1998
2000
2002
2004
2006
2008
2010
Year
Successful Challenged Failed
$0.0
$0.4
$0.7
$1.1
$1.4
$1.8
2002 2003 2004 2005 2006 2007 2008 2009 2010
Trill
ions
(US
Dolla
rs)
Expenditures Failed Investments
Requirements Defects & Waste
6Sheldon, F. T. et al. (1992). Reliability measurement: From theory to practice. IEEE Software, 9(4), 13-20Johnson, J. (2002). ROI: It's your job. Extreme Programming 2002 Conference, Alghero, Sardinia, Italy.
Requirements defects are #1 reason projects fail Traditional projects specify too many requirements More than 65% of requirements are never used at all
Other 7%
Requirements47%
Design28%
Implementation18%
Defects
Always 7%
Often 13%
Sometimes16%
Rarely19%
Never45%
Waste
What is Agility? A-gil-i-ty (ə-'ji-lə-tē) Property consisting of quickness,
lightness, and ease of movement; To be very nimble The ability to create and respond to change in order to
profit in a turbulent global business environment The ability to quickly reprioritize use of resources when
requirements, technology, and knowledge shift A very fast response to sudden market changes and
emerging threats by intensive customer interaction Use of evolutionary, incremental, and iterative delivery
to converge on an optimal customer solution Maximizing BUSINESS VALUE with right sized, just-
enough, and just-in-time processes and documentationHighsmith, J. A. (2002). Agile software development ecosystems. Boston, MA: Addison-Wesley.
7
What are Agile Methods?
8
People-centric way to create innovative solutions Product-centric alternative to documents/process Market-centric model to maximize business value
Agile Manifesto. (2001). Manifesto for agile software development. Retrieved September 3, 2008, from http://www.agilemanifesto.orgRico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods. Ft. Lauderdale, FL: J. Ross Publishing.Rico, D. F. (2012). Agile conceptual model. Retrieved February 6, 2012, from http://davidfrico.com/agile-concept-model-1.pdf
Customer Collaboration
Working Software
Individuals & Interactions
Responding to Change
valuedmore than
valuedmore than
valuedmore than
valuedmore than
Contracts
Documentation
Processes
Project Plans
Frequent comm. Close proximity Regular meetings
Multiple comm. channels Frequent feedback Relationship strength
Leadership Boundaries Empowerment
Competence Structure Manageability/Motivation
Clear objectives Small/feasible scope Acceptance criteria
Timeboxed iterations Valid operational results Regular cadence/intervals
Org. flexibility Mgt. flexibility Process flexibility
System flexibility Technology flexibility Infrastructure flexibility
Contract compliance Contract deliverables Contract change orders
Lifecycle compliance Process Maturity Level Regulatory compliance
Document deliveries Document comments Document compliance
Cost Compliance Scope Compliance Schedule Compliance
NetworkComputer
Operating SystemMiddlewareApplications
APIsGUI
How Agile Works Agile requirements implemented in slices vs. layers User needs with higher business value are done first Reduces cost & risk while increasing business success
9Shore, J. (2011). Evolutionary design illustrated. Norwegian Developers Conference, Oslo, Norway.
Agile Traditional1 2 3 Faster
Early ROI
Lower Costs
Fewer Defects
Manageable Risk
Better Performance
Smaller Attack Surface
Late
No Value
Cost Overruns
Very Poor Quality
Uncontrollable Risk
Slowest Performance
More Security Incidents Seven Wastes1.Rework2.Motion3.Waiting4.Inventory5 .Transportation6.Overprocessing7 .Overproduction
MINIMIZES MAXIMIZES
JIT, Just-enough architecture Early, in-process system V&V Fast continuous improvement Scalable to systems of systems Maximizes successful outcomes
Myth of perfect architecture Late big-bang integration tests Year long improvement cycles Breaks down on large projects Undermines business success
What is Agile Testing? Traditional testing is a late, manual process Agile testing is an early and automated process The goal of agile testing is to deliver early and often
10Rico, D. F. (2012). Agile testing resources. Retrieved Sep. 9, 2012, from http://davidfrico.com/agile-testing-resources.txtCrispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.
Traditional Testing
Combining source files
Combining software and environment
Combining software and data
Combining software and tests
Combining developers
Agile Testing
Code is frequently checked in
Code is automatically retrieved
Compilation is done automatically
Tests are done automatically
Code reports are generated
Developers get instant feedback
Code is automatically deployed or packaged for delivery
Basic—Test Driven Development Term coined by Kent Beck in 2003 Consists of writing all tests before design Ensures all components are verified and validated
11Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.
Thousands of TestsContinuously Executed
No More Late BigBang Integration
Advanced—Continuous Integration User needs designed & developed one-at-a-time Changes automatically detected, built, and tested System fully tested and deployed as changes occur
12Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.
BuildIntegration
Server
VersionControlServer
BuildScripts
UsesWatches
BuildStatus
ProvidesDeveloper A
Developer B
Developer C
CommitsChanges
CommitsChanges
CommitsChanges
Builds
Database
Analysis
Testing
Reporting
Documentation
Deployment
Early, Automated, Fast,Efficient, & Repeatable
Constant ReadinessState & CM Control
Lean, Waste Free, Low WIP,No Deadlocked Test Queues
Rapidly & SuccessfullyDev. Complex Systems
Agile Testing Done Early & Often Eliminates big-bang integration in the 11th hour Creates a repeatable and reliable testing process Evaluates system-wide changes throughout project
13Maeda, M. K. (2009). Agile testing: Early, often, and Smart. Arlington, MA: Cutter Consortium.
Agile Testing Practices Agile testing consists of seven broad practices Includes automated builds, testing, inspections, etc. Also includes reporting, documentation, deployment, etc.
14
Practice
Building
Database
Inspections
Testing
Feedback
Documentation
Deployment
Description
Frequently assembling products and services to ensure delivery readiness
Frequently generating/analyzing database schemas, queries, and forms
Frequently performing automated static analysis of product/service quality
Frequently performing automated dynamic product and service evaluation
Frequently generating automated status reports/messages for all stakeholders
Frequently performing automated technical/customer document generation
Frequently performing automated delivery of products/services to end users
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.
Agile Testing Workflow
15
Traditional vs. Agile Cumulative Flow
Wor
k (S
tory
, Poi
nt, T
ask)
or E
ffor
t (W
eek,
Day
, Hou
r)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Wor
k (S
tory
, Poi
nt, T
ask)
or E
ffor
t (W
eek,
Day
, Hou
r)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Traditional Cumulative Flow Agile Cumulative Flow
Late big bang integration increases WIP backlog Agile testing early and often reduces WIP backlog Improves workflow and reduces WIP and lead times
Anderson, D. J. (2004). Agile management for software engineering. Upper Saddle River, NJ: Pearson Education.Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.
Agile Testing Costs & Benefits
Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.Fredrick, J. (2008). Accelerate software delivery with continuous integration and testing. Japanese Symposium on Software Testing, Tokyo, Japan.
Most agile testing tools are “free” open source A build server is no more than a commodity PC 10x more efficient/effective than traditional testing
16
Agile Testing Economics Traditional testing finds a defect in about 10 hours Manual code inspections find a defect in 1 hour Agile testing finds a defect every 6 minutes
17Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.
Agile Cost of Quality (CoQ) Agile testing is 10x better than code inspections Agile testing is 100x better than traditional testing Agile testing is done earlier “and” 1,000x more often
18Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.
Agile Testing Statistics Fewer builds leave in higher bug counts A high number of builds eliminates the defects Goal is to have as many, early builds as possible
19Lacoste, F. J. (2009). Killing the gatekeeper: Introducing a continuous integration system. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 387-392.
How Google Tests Software Google early adopter of agile methods and Scrum Google also uses agile testing at enterprise scale 15,000 developers run 75+ million tests per day
20Micco, J. (2013). Continuous integration at google scale. Eclipse Con, Boston, MA.Whittaker, J., Arbon, J., & Carollo, J. (2012). How google tests software. Upper Saddle River, NJ: Pearson Education.
15,000+ developers in 40+ offices 4,000+ projects under active development 5,500+ submissions per day on average Single monolithic code tree with mixed language code Development on one branch - submissions at head All builds from source 20+ sustained code changes per minute with 60+ peaks 50% of code changes monthly 75+ million test cases run per day
Scaling Agile Testing Agile testing slows down with very large systems Slow testing slows integration and increases bugs Agile testing can speed back up with proper attention
21Kokko, H. (2009). Increase productivity with large scale continuous integration. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.
Wide Impact Tuning
Fast builds – less changes – more green
Parallelization of test runs
ClearCase to subversion
Pre-installing as much as possible
Removal of randomness
Compilation in memory
Installation starting parallel with system build
Focused Impact Tuning
More memory and CPUs
Parallelize builds
Replace 3rd party test libraries
Reduce/remove timeouts in tests
Select different tests
Refactor code & components
Tune the network & software
Tune the database
General Agile Metrics Agile methods are based on traditional measures Velocity, burnup, and burndown are basic measures Top-notch shops use business value, Agile EVM, etc.
22Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods. Ft. Lauderdale, FL: J. Ross Publishing.
Basic Metrics Velocity Productivity Cycle time Effort Cost Schedule System quality Customer Satisfaction
Business ValueBurnup & Burndown
Advanced MeasuresAgile Earned Value
Epic burnup & burndown Feature burnup & burndown Release burnup & burndown Iteration burnup & burndown Weekly burnup & burndown Daily burnup & burndown
Business value per epic Business value per feature Business value per release Business value per iteration Business value per week Business value per day Business value per story
Story points per release Planned sprints per release Planned budget per release Current to planned sprint Current to planned points Planned release points Release points completed
Customer collaboration Customer trust & loyalty Teamwork cohesion & quality Purpose, autonomy, & mastery Workflow capacity & throughput Workflow efficiency & reliability Organizational culture & agility Information systems flexibility
Agile Testing Metrics Agile testing also based on traditional metrics Agile testing measures include basic volumetrics Key metrics are RTF, code coverage, complexity, etc.
23Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods. Ft. Lauderdale, FL: J. Ross Publishing.
Volumetrics Number of user stories Number of story points Number of lines of code Number of function points Number of check-ins Number of unit tests Number of builds Number of deployments
Code CoverageRunning Tested Code
Defect DensityCode Complexity
Running tested epics Running tested features Running tested releases Running tested iterations Running tested builds Running tested user stories
Epic coverage Feature coverage Release coverage Iteration coverage Build coverage Statement coverage Function coverage Other coverage types
Epic complexity Feature complexity Release complexity Iteration complexity Build complexity Cyclomatic complexity Actual complexity Other complexity types
Defects per epic Defects per feature Defects per release Defects per iteration Defects per build Defects per user story Defects per story point Defects per other types
Agile Cost & Benefit Analysis Costs based on avg. productivity and quality Productivity ranged from 4.7 to 5.9 LOC an hour Costs were $588,202 and benefits were $3,930,631
24Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods: Maximizing ROI with just-in-time processes and documentation. Ft. Lauderdale, FL: J. Ross Publishing.
d1 = [ln(Benefits Costs) + (Rate + 0.5 Risk2) Years] Risk Years, d2 = d1 Risk Years
5
1i
Studies of Agile Methods Dozens of surveys of agile methods since 2003 100s of Agile and CMMI case studies documented Agile productivity, quality, and cost better than CMMI
25Rico, D. F. (2008). What is the return-on-investment of agile methods? Retrieved February 3, 2009, from http://davidfrico.com/rico08a.pdf Rico, D. F. (2008). What is the ROI of agile vs. traditional methods? TickIT International, 10(4), 9-18.
Benefits of Agile Methods Analysis of 23 agile vs. 7,500 traditional projects Agile projects are 54% better than traditional ones Agile has lower costs (61%) and fewer defects (93%)
Mah, M. (2008). Measuring agile in the enterprise: Proceedings of the Agile 2008 Conference, Toronto, Canada.
Project Cost in Millions $
0.75
1.50
2.25
3.00
2.8
1.1
Before Agile
After Agile
61%LowerCost
Total Staffing
18
11
Before Agile
After Agile
39%LessStaff
5
10
15
20
Delivery Time in Months
5
10
15
20
18
13.5
Before Agile
After Agile
24%Faster
Cumulative Defects
625
1250
1875
2500
2270
381
Before Agile
After Agile
93%Less
Defects
26
Agile vs. Traditional Success Traditional projects succeed at 50% industry avg. Traditional projects are challenged 20% more often Agile projects succeed 3x more and fail 3x less often
Standish Group. (2012). Chaos manifesto. Boston, MA: Author.
27
Agile Traditional
Success42%
Failed9%
Challenged49%
Success14%
Failed29%
Challenged57%
Hoque, F., et al. (2007). Business technology convergence. The role of business technology convergence in innovation and adaptability and its effect on financial performance. Stamford, CT: BTM Institute. 28
Study of 15 agile vs. non-agile Fortune 500 firms Based on models to measure organizational agility Agile firms out perform non agile firms by up to 36%
Benefits of Organizational Agility
Agile Enterprise Delivery Model
Beck, K., & Fowler, M. (2001). Planning extreme programming. Upper Saddle River, NJ: Addison-Wesley.Highsmith, J. A. (2010). Agile project management: Creating innovative products. Boston, MA: Pearson Education.Larman, C., & Vodde, B. (2010). Practices for scaling lean and agile development. Boston, MA: Addison-Wesley.Leffingwell, D. (2011). Agile software requirements: Lean requirements practices for teams, programs, and the enterprise. Boston, MA: Pearson Education.
Begins with a high-level product vision/architecture Continues with needs development/release planning Includes agile delivery teams to realize business value
29
Agile Adoption
30House, D. (2012). Sixth annual state of agile survey: State of agile development. Atlanta, GA: VersionOne.
VersionOne found 80% using agile methods today Most are using Scrum with several key XP practices Lean-Kanban is a rising practice with a 24% adoption
ContinuousIntegration
●●
●
●●
●
●●●
●
●●
Agile Proliferation
Scrum Alliance. (2012). Scrum certification statistics. Retrieved February 6, 2013, from http://www.scrumalliance.org/resource_download/2505Taft, D. K. (2012). Agile developers needed: Demand outpaces supply. Foster City, CA: eWeek. 31
Number of CSMs have doubled to 200,000 in 2 years 558,918 agile jobs for only 121,876 qualified people 4.59 jobs available for every agile candidate (5:1)
Agile Industry Case Studies 80% of worldwide IT projects use agile methods Includes regulated industries, i.e., DoD, FDA, etc. Agile now used for safety critical systems, FBI, etc.
32
Industry
ShrinkWrapped
ElectronicCommerce
HealthCare
LawEnforcement
Org 20 teams 140 people 5 countries
Size
15 teams 90 people Collocated 4 teams 20 people Collocated 10 teams 50 people Collocated 3 teams 12 people Collocated
U.S.DoD
Primavera
Stratcom
FBI
FDA
Project
Primavera
Adwords
SKIweb
Sentinel
m2000
Purpose
ProjectManagement
Advertising
KnowledgeManagement
Case FileWorkflow
BloodAnalysis
1,838 User Stories 6,250 Function Points 500,000 Lines of Code
Metrics
26,809 User Stories 91,146 Function Points 7,291,666 Lines of Code 1,659 User Stories 5,640 Function Points 451,235 Lines of Code 3,947 User Stories 13,419 Function Points 1,073,529 Lines of Code 390 User Stories 1,324 Function Points 105,958 Lines of Code
Rico, D. F. (2010). Lean and agile project management: For large programs and projects. Proceedings of the First International Conference on Lean Enterprise Software and Systems, Helsinki, Finland, 37-43.
Structure, reward, decision, staffing, leadership, etc. Top-down, individualism, regulation, compliance, etc. Focus on reforming acquisition & procurement system
33
Type/Kind Common DoD Agile Perceptions
Discipline
Reality with Respect to Agile Methods
Scalability
Domain
Management
Requirements
Architecture
Quality
Inspections
Security
Undisciplined Cowboy Coding
Only Applies Small Projects
Only for Protoperational Systems
Flexible Scope/Can't Use EVM
Doesn't Use Requirements
Spaghetti Code from Iterations
No Documents/Unmaintainable
High CoQ from No Inspections
Vulnerabilities from Hacking
Rigorous process, plans, requirements, QA, CM, testing, documents etc.
Used by 100, 500, 1,000, 10,000+ person person projects & organizations
Used in DoD, medical devices, avionics, autos, electronics, etc.
Lightweight EVM model is used with its release planning methodology
Always begins with valuable, well-defined, & prioritized requirements
Begins with lean architecture or create waste-free emergent design
Electronic plans, requirements, designs, tests, manuals, documents, etc.
One or two orders of magnitude more inspections & tests performed
Security practices result in smaller attack surface & fewer vulnerabilities
Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods: Maximizing ROI with just-in-time processes and documentation. Ft. Lauderdale, FL: J. Ross Publishing.
Perceptions of Agile Methods
Agile World View “Agility” has many dimensions other than IT It ranges from leadership to technological agility The focus of this brief is program management agility
Agile Leaders
Agile Organization Change
Agile Acquisition & Contracting
Agile Strategic Planning
Agile Capability Analysis
Agile Program Management
Agile Tech.
Agile Information Systems
Agile Tools
Agile Processes & Practices
Agile Systems Development
Agile Project Management
34
Agile Recap Agile methods DON’T mean deliver it now & fix it later Lightweight, yet disciplined approach to development Reduced cost, risk, & waste while improving quality
35Rico, D. F. (2012). What’s really happening in agile methods: Its principles revisited? Retrieved June 6, 2012, from http://davidfrico.com/agile-principles.pdfRico, D. F. (2012). The promises and pitfalls of agile methods. Retrieved February 6, 2013 from, http://davidfrico.com/agile-pros-cons.pdfRico, D. F. (2012). How do lean & agile intersect? Retrieved February 6, 2013, from http://davidfrico.com/agile-concept-model-3.pdf
What How ResultFlexibility Use lightweight, yet disciplined processes and artifacts Low work-in-process
Customer Involve customers early and often throughout development Early feedback
Prioritize Identify highest-priority, value-adding business needs Focus resources
Descope Descope complex programs by an order of magnitude Simplify problem
Decompose Divide the remaining scope into smaller batches Manageable pieces
Iterate Implement pieces one at a time over long periods of time Diffuse risk
Leanness Architect and design the system one iteration at a time JIT waste-free design
Swarm Implement each component in small cross-functional teams Knowledge transfer
Collaborate Use frequent informal communications as often as possible Efficient data transfer
Test Early Incrementally test each component as it is developed Early verification
Test Often Perform system-level regression testing every few minutes Early validation
Adapt Frequently identify optimal process and product solutions Improve performance
Conclusion
36
Agility is the evolution of management thought Confluence of traditional and non-traditional ideas Improve performance by over an order of magnitude
“The world of traditional methods belongs to yesterday”“Don’t waste your time using traditional methods on 21st century projects”
Agile methods are …
Systems development approachesNew product development approachesExpertly designed to be fast and efficientIntentionally lean and free of waste (muda) Systematic highly-disciplined approachesCapable of producing high quality systemsRight-sized, just-enough, and just-in-time tools
Scalable to large, complex mission-critical systems Designed to maximize business value for customers
Wysocki, R.F. (2010). Adaptive project framework: Managing complexity in the face of uncertainty. Boston, MA: Pearson Education.
Books on ROI of SW Methods Guides to software methods for business leaders Communicates business value of software methods Rosetta stones to unlocking ROI of software methods
http://davidfrico.com/agile-book.htm (Description) http://davidfrico.com/roi-book.htm (Description)
37