Upload
xbosoft
View
164
Download
6
Embed Size (px)
Citation preview
When Agile Is a Quality Game Changer:
Predictive Forecasting of Defect Profiles
with QSM Associates’ Michael Mah
#AgileGameChanger
XBOSoft, Inc. All Rights Reserved.
About XBOSoft
Dedicated to Software Quality ImprovementFounded in 2006
We speed products to market with our expert:• Software QA consulting• Software testing
Global team with offices in San Francisco & Beijing“Thorough, accurate and fast”
XBOSoft, Inc. All Rights Reserved.
House Rules
Participants other than the speakers are muted Ask questions in the GoToWebinar control on the
right side of your screen or through Twitter @XBOSoft
Questions may be asked throughout the webinar - we’ll try to answer them at the end
You’ll receive info on recording after the webinar
Webinar Hashtag: #AgileGameChanger
XBOSoft, Inc. All Rights Reserved.
Meet the XBOSoft Team
Philip LewCEO and Founder, XBOSoft•Relevant specialties and passions
o Software quality process, evaluation, measurement and improvement
o Software quality in use / UX design
o Mobile User Experience and usability
o Cycling and travel
XBOSoft, Inc. All Rights Reserved.
Rachel GeibigMarketing Manager, XBOSoft•Emails you about our industry hot topic webinars•Invites you to download our latest trends in software testing white papers
With over 25 years of experience, Michael and his partners have derived productivity patterns for thousands of software projects collected worldwide across engineering and business applications.His current work examines time-pressure dynamics of teams, and its role in project success and failure. In addition to his background in physics and electrical engineering, he is a mediator specializingin dispute resolution for technology projects.
• Director of the Benchmarking Practice at the Cutter Consortium, a Boston-based IT think-tank
• Past editor of IT Metrics Strategies publication• Managing partner at QSM Associates Inc. based in Massachusetts• Teaches, writes, and consults to companies on measuring, estimating and
managing software projects, whether in-house, offshore, waterfall, or agile
He has a degree in electrics engineering from Tufts University. His training on dispute resolution, mediation, and participatory processes is from the Program on Negotiation at Harvard Law School and the Radcliffe Institute for Advanced Study. Michael is also a private pilot and lives in the mountains of Western Massachusetts.
Michael Mah
(#6)
(#9)
Rayleigh Curve Defect Rate
0
20
40
60
80
100
120
140
160
Defect Type (All)
Count of Severity*
Create Date Status Mode Status Severity* TR-Version
Product+*
(#11)
(#12)
(#13)
(#14)
(#15)
The QSM SLIM Database
QSM maintains the world’s largest benchmarking database of 12,000+ completed software projects collected worldwide. We put industry productivity statistics on the desktop.
The QSM SLIM database contains projects in all industries, waterfall, Agile, offshore/outsourced, in-house, new development, and maintenance.
SLIM tools enable managers to measure and estimate and in-flight forecast Agile and waterfall projects.
Partial List of Clients
British TelecomSAP MicrosoftIntel AT&T/BellSouth Nationwide Motorola Verizon WirelessRoche Diagnostics
Fiserv Corp IBM Global Misys Healthcare JPMorganChase Boeing Bank of New York Mellon Lockheed Martin Progressive Insurance DirecTV
(#18)
Input to SLIM
Size
Time
Defects
Effort
Trendline Assessment – Defects/Quality
Defects During Test
100 1,000
Effective SLOC (thousands)
10
100
1,000
10,000
Errors (S
ysInt-Del)
Rel 5.0
Rel 6.0 Rel 6.5Rel 7.5 Rel 7.0
Rel 8.0
Rel 5.0
Rel 6.0 Rel 6.5Rel 7.5 Rel 7.0
Rel 8.0
Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 BusinessAv g. Line Sty le 1 Sigma Line Sty le
Far Fewer Defects: 50% - 66% Below Industry
Trendline Assessment – Build Phase Schedule
Main Build Phase Duration vs Size
100 1,000
Effective SLOC (thousands)
1
10
100
Time (M
onths)
Rel 5.0
Rel 6.0
Rel 6.5
Rel 7.5Rel 7.0
Rel 8.0Rel 5.0
Rel 6.0
Rel 6.5
Rel 7.5Rel 7.0
Rel 8.0
Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 BusinessAv g. Line Sty le 1 Sigma Line Sty le
Schedules are Half Industry
Industry Average
Current Performance
Delta
Project Cost $3.5 Million $2.2 Million -$1.3M
Schedule 12.6 months 7.8 months -4.8 mos
QA Defects 242 121 -50%
Staffing 35 35 n/a
Follett vs. Industry Average
Short Feedback Loops
Paired programmers
Instantaneous code reviews
Accelerated learning and execution
Face to face communication channel
Time Boxing
Short iterations
Clear and discernible progress
Anticipation of the next important feature
Efficiency
Avoiding Burnout
XP = Sustainable pace
40 Hour Work WeeksPrevent productivity collapse for overworked teams
Transparency
“Transparency is a great floodlight. People who thrive in political maneuvering hate SCRUM…”
- Ken Schwaber
High-Bandwidth Communication
The best teams have “wide-open pipes”
Domain knowledge moves among the team
Information flows rapidly and accurately
Avoiding Waste and Costly Rework
Rework has high costRework takes timeRework creates defectsRework is badRefactoring can be a cover up
Norbert Wiener, 1894-1964
“He noticed that the Feedback principle is a key feature of life forms from the simplest plants to the most complex animals, which change their reactions in response to their environment.”
Keywords: Shannon’s Information Theory, Cybernetics
Norbert Wiener, 1894-1964
Example Outsourced Project – Whiteboard Sketch
Input to SLIM
Size
Time Effort
SLIM Replica – Release 9.5Staffing & Probability Analysis
Avg Staff (people)<Single Goal - MBI 4.433>
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21Jan'06
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan'07
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
0
20
40
60
80
100
Avg Staff (people)876543210
CDR&DC&T
Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R
Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R
SOLUTION PANEL <Single Goal - MBI 4.433>
DurationEffortCost
Peak StaffMTTD
Start Date
C&T14.5716
4153.274.80.1
8/4/2006
Life Cycle19.6919
5328.574.80.1
3/1/2006
MonthsPM$ (K)peopleDays
PI=21.3 MBI=4.4 Eff SLOC=686,000
CONTROL PANEL <Single Goal - MBI 4.433>
PI 17.0 25.5
21.3
Peak Staff 59.8 89.7
74.8
Eff SLOC (K) 549 823
686
Main Build Phase Schedule vs. Size
1 10 100 1,000 10,000
Effective SLOC (thousands)
1
10
100
1,000
DC
&T D
uration (Months)
AB Rel 1.0
AB Rel 9.5AB Rel 9.0
AB Rel 1.0
AB Rel 9.5AB Rel 9.0
AB Medical All AB Medical Sy stems Special Project Misy s - All Projects Av g. Line Sty le 1 Sigma Line Sty le
Trendline Assessment – Main Build Schedule
Company – All Projects
Deadline target for Rel 9.0 is aggressive but consistentwithin the band of history. Rel 9.5 and Rel 10scope/dates are off the chart.
Main Build Phase Staff vs Size
1 10 100 1,000 10,000
Effective SLOC (thousands)
1
10
100
1,000
DC
&T P
eak Staff (P
eople)
AB Rel 9.5
AB Rel 10
AB Rel 9.0AB Rel 9.5
AB Rel 10
AB Rel 9.0
AB Medical All AB Medical Sy stems Special Project Misy s - All Projects Av g. Line Sty le 1 Sigma Line Sty le
Planned staffing profiles (RED projects)are much more aggressive than history. Tight deadlines and time-to-marketare the drivers.
Trendline Assessment – Main Build Staffing
Company – All Projects
Trendline Assessment – Defects/QualityDefects During Test vs Size
1 10 100 1,000 10,000
Effective SLOC (thousands)
0.001
0.01
0.1
1
10
100
Errors (S
ysInt-Del) (thousands)
AB Medical All AB Medical Sy stems Special Project Misy s - All Projects Av g. Line Sty le 1 Sigma Line Sty le
Ambitious scope, aggressive staffing, and tight deadlinesresult in (higher/lower) defects. (Fill in answer)[WATCH YOUR WARRANTEES]
Company – All Projects
Defects Found
6 12 18 24 30 36 42 48 54 60 66 72 78 84 9005/17
'0306/28 08/09 09/20 11/01 12/13 01/24
'0403/06 04/17 05/29 07/10 08/21 10/02 11/13 12/25 02/05
'05
0
50
100
150
200
250
300
350
Defects
109876543101098765
Current Plan Actuals Green Control Bound Yellow Control Bound Project: CEP
Defects vs. Industry Average
Defects Found
6 12 18 24 30 36 42 48 54 60 66 72 78 84 9005/17
'0306/28 08/09 09/20 11/01 12/13 01/24
'0403/06 04/17 05/29 07/10 08/21 10/02 11/13 12/25 02/05
'05
0
100
200
300
400
500
Defects
109876543101098765
Current Plan Actuals Green Control Bound Yellow Control Bound Project: CEP
Scaled Up Defect Curve – Outsourced Project
Nov 19
Forecast to Complete (4 Month Delay)
Industry Average
Offshore Average
Delta
Project Cost $3.5 Million $3.2 Million -$0.3M
Schedule 12.6 months 9.6 months -3.0 mos
QA Defects 242 677 +280%
Staffing 35 50 +15
* Normalized to same project size
Offshore vs. Industry Average
Industry Average
Current Performance
Delta
Project Cost $3.5 Million $2.2 Million -$1.3M
Schedule 12.6 months 7.8 months -4.8 mos
QA Defects 242 121 -50%
Staffing 35 35 n/a
Follett vs. Industry Average
The force of concentration, or “clustering” of human creativity and talent… the powerful (economic) gains that come when smart and talented people (co-)locate in close proximity to one another…
Robert Lucas Revisited
Some Practical Advice
The Source of Our Troubles
Take a Little More Time
Build a Little Less
Use Small A-Teams
Give Them the Best Tools
(#60)
(#61)
Curve Fitting and Creating Agile TrendsAgile Trends - Iterations/Build Phase
Time-to-Market
10 100 1,000
New + Modified Size (thousands)
1
10
100
Months
Effort
10 100 1,000
New + Modified Size (thousands)
1
10
100
1,000
10,000
Person-M
onths
Average Staff
10 100 1,000
New + Modified Size (thousands)
1
10
100
1,000
Headcount
Bugs During QA
10 100 1,000
New + Modified Size (thousands)
1
10
100
1,000
10,000
Defects
All Sy stems Av g. Line Sty le 1 Sigma Line Sty le
New Agile Benchmark TrendsAgile Trends - Iterations/Build Phase
Time-to-Market
10 100 1,000
New + Modified Size (thousands)
1
10
100
Months
Effort
10 100 1,000
New + Modified Size (thousands)
1
10
100
1,000
10,000
Person-M
onths
Average Staff
10 100 1,000
New + Modified Size (thousands)
1
10
100
1,000
Headcount
Bugs During QA
10 100 1,000
New + Modified Size (thousands)
1
10
100
1,000
10,000
Defects
The S
LIM-S
uite
SLIM-Estimate:Size, Schedule, Cost & Quality
Estimating
SLIM-DataManagerSoftware Project Metrics Repository
SLIM-Control:Variance Analysis
& Adaptive
Forecasting
SLIM-Metrics: Industry
Benchmarking & Process
Improvement
SLIM-MasterPlan: Incremental
Development & Project
Aggregation
The SLIM-Suite
Software Lifecycle Management
(#66)
QSM SLIM Agile Benchmark TrendsAgile Trends - Iterations/Build Phase
Time-to-Market
10 100 1,000
New + Modified Size (thousands)
1
10
100
Months
Effort
10 100 1,000
New + Modified Size (thousands)
1
10
100
1,000
10,000
Person-M
onths
Average Staff
10 100 1,000
New + Modified Size (thousands)
1
10
100
1,000
Headcount
Bugs During QA
10 100 1,000
New + Modified Size (thousands)
1
10
100
1,000
10,000
Defects
(#68)
Contact Michael
Michael MahManaging PartnerQSM Associates, Inc.Pittsfield MA USA
email: [email protected]: www.qsma.comtwitter: @michaelcmahtel: 1 413-499-0988
Post your webinar questions on Twitter @XBOSoft
Registrants will receive an email with information on where to view the recording and slides from today’s webinar.
Join us to keep updated on all our webinars, reports and white papers:
facebook.com/xbosoft
+xbosoft
linkedin.com/company/xbosoft
Check out our blog: http://xbosoft.com/software-quality-blog/
Download our free white papers: http://xbosoft.com/knowledge-center/
Email us with ideas for future webinars or questions regarding our services! [email protected]
Thank you!
Q&A
www.xbosoft.comXBOSoft, Inc. All Rights Reserved.