Upload
harold-van-heeringen
View
211
Download
1
Tags:
Embed Size (px)
DESCRIPTION
How to prove Cost reductions after outsourcing Test activities - a framework. Software Measurement European Forum (Milan, may 2008)
Citation preview
Outsourcing Test activities
How to prove cost reductionsHarold van Heeringen, Metrics Consultant
Sogeti
Wednesday May 28, Milan, Italy
Outline
• Problem definition• Factors that influence Test
Performance• Model CASE• Results & Conclusions
Problem definition
• More and more outsourcing test activities
• Contracts - x% of cost reductions for the client in 1,2,3 years
• So far: Impossible to prove!>Compare bills ?>What about volume increase/decrease ?>What about defects found… and defects not
found?
Test activities
Functional design
Technical design
Coding + Unit test
Systems test
User Acceptance Test
Client
Client
Sogeti
Client
Factors that influence Test Performance
• The most important factors:>Number of defects found (in relation to
number of defects present)Many defects found – did we test well?Few defects found – did we test badly?
>Number of hours spentWe have spent 1.000 hours testing this project. Is that
OK?
>Time interval availableBuild phase ended late, but end date is fixed. What
can we do in the time interval available?
The relation between these factors
• Extra short duration...>How many extra
hours do we need?
• Extra long duration...>How many extra
defects will we find?>How many defects
are not found?
Impossib
le
Unpractical
Eff
ort
(h
our)
DurationD
efe
cts
rem
ain
ing
When do we test well?
• More defects found than expected?• Less hours spent than expected?
• But how do we know the number of defects and hours expected??
• We need an estimation tool!
Defects found < expected
Defects found = expectated
Defects found > expected
hours spent < expected
? + ++hours spent = expected
- ± +hours spent > expected
-- - ?
QSM SLIM Estimate
• Project Estimation> Hours, duration, fte, costs, etc
> But also: defects !!!
Benchmark test performance
• Benchmark supplier performance against client performance – but how??>Every project is different >Duration is critical variable>Test hours and defects found are highly
dependent on the Build performance
• How can we know the client’s performance?
• Hours spent• Defects found• Defects not found
Model proposed: 3 stages
>1: Baseline analysis – assess the client situation
• PI (productivity duration)• Cost per test hour• Defect Tuning factor
>2: Operational execution – Estimate, measure and benchmark performance
• Simulate project with actual duration, but with client baseline parameters
• Measure actuals• Calculate Cost reductions
>3: Evaluation – Evaluate whether the cost reduction targets have been met
Model step 1: Baseline analysis
• Select 4 – 5 representative projects>Size the projects in COSMIC/FPA/slocs>Administrate results in SLIM Datamanager
• Results:>Detailed analysis per project>SLIM Estimate WBS>Average/median Productivity index>Defect tuning factor>Costs per test hour
CASE: Baseline analysis
• Contract: outsourcing systems testing • Target: 8% Cost reductions 1st year
• Baselinedata:
• 1 test hour = 80 euro> activities?> Fixed costs?> Waiting hours?
Project Size in
CFP
Hours S-test
Defects S-test
Defects A-test /1st M production
A 310 554 12 8
B 512 937 27 5
C 212 508 22 12
D 224 732 52 18
E 487 878 35 8
Model step 2: Operational execution
• Choice: Measure everything, or measure predefined set of projects1 – Measure size 2 – Estimate project with client parameter
set and actual duration3 – Compare client estimation with Sogeti
actual data4 – Calculate Test Performance5 – Calculate Cost Reduction
CASE: Operational execution
Step 1 - 5 projects selected to measureStep 1 - Size measurement in COSMICStep 2 - Estimation in SLIM, using client baseline parameters and actual durationProject Q
Scenario Defects S-test FS
Defects A-test + 1M Prod. FAP
Defects Total FT
FS/FT
Client baseline
30 31 61 0,49
Scenario Test hours TU
Hour rate R
Costs C (TU*R)
Client baseline
2.045 80 163.600
CASE: Operational execution
Step 3 – Compare Sogeti actuals against baseline scenarioProject Q
Scenario Test hours TU
Hour rate R
Costs C (TU*R)
Client baseline
2.045 80 163.600
Sogeti actuals
1.800 82 147.600
Scenario Defects S-test FS
Defects A-test + 1M Prod. FAP
Defects Total FT
FS/FT
Client baseline
30 31 61 0,49
Sogeti actuals
35 30 65 0,53
CASE: Operational execution
Step 4 – Assess Test Performance projects> Outsourcing Effectiveness
Did we test better than the client would have done?
> Outsourcing EfficiencyDid we test more efficiently than the client would have done?
Outsourcing Efficiency OF
Baseline C – Actual C Baseline C
Outsourcing Effectiveness OV
(Actual FS/FT) – (Baseline FS/FT) Baseline (FS/FT)
*100%
*100%
CASE: Operational execution
• Project Q>Outsourcing Effectiveness:
OV = (0.53-0.49)/0.49 ≈ 8,2 %
>Outsourcing Efficiency: OF = (163.600–147.600) / 147.600 ≈ 9,7%
Outsourcing effectiveness
Outsourcing efficiency
- 100 %
100 % 0 %
100 %
- 100 %
0 %
No Cost Reduction
Cost Reduction
CASE: Operational execution
• Step 5 – Calculate Cost Reduction>Maybe client rates OV higher than OF or
vice versa>Contract – agree on weight factors
OV = Outsourcing EffectivenessOF = Outsourcing Efficiencyw1 = weight factor OVw2 = weight factor OF
Cost Reduction CR
(w1*OV) + (w2*OF) w1+w2
Model step 3: Evaluation
• After contract period: Analyze results
Outsourcing effectiveness
Outsourcing efficiency
- 100 %
100 % 0 %
100 %
- 100 %
0 %
No Cost Reduction
Cost Reduction
8%
8%
CASE: Evaluation
• Evaluation client X, 1st year
• Total CR: € 129.143 / € 1.383.570 ≈ 9,3%• Substantial cost reductions, even more than
was promised!!
Project OV OF C Baseline CR % CR €
Q 9,7% 8,2% € 163.600 8,95% € 14.642
R 12,3% 6,1% € 312.150 9,2% € 28.718
S 8,2% 3,0% € 122.650 5,6% € 6.868
T 6,4% 13,3% € 718.120 9,85% € 70.735
U 15,6% 8,8% € 67.050 12,2% € 8.180
Total € 1.383.570 € 129.143
First results
• Clients reactions are very positive• Model should be evaluated in
practice and should be developed further
• Please help us further develop the model:>Metrieken.sogeti.nl>[email protected]
Conclusions
• The model proposed is objective and can be used in all organizations and all projects
• The client can now see the real cost reductions they achieve by outsourcing
• The supplier can also use the model as a commercial instrument and is able to support claims of cost reduction based on experience
metrieken.sogeti.nl