Over the last few years, the expectations on the implementation of high maturity practices of CMMI have gone up several notches. It has been a difficult and exciting journey for many organizations in revamping their measurement systems and their approach to metrics. Organizations that consider the new requirements as a "delta" over what have been earlier doing have struggled and not been able to transition smoothly. On the other hand, organizations that have "unlearned" their old habits and ways of thinking, kept an open mind and aligned themselves to the new way have been able to make the transition smoothly. The talk focuses on the new way of statistical thinking, and the typical mistakes that organizations make in implementation. Key areas covered are: * Sub-process control * Process Performance Models and composing the defined process * Quantitatively managing process improvements The talk concludes with a summary of the expectations for a true and robust implementation of high maturity practices. This presentation was in CSI-SPIN forum on July 14 as well as Sept 27, 2010.
Citation preview
1. High Maturity Implementation: Pitfalls and Misconceptions At
CSI-SPIN (Mumbai), Sept 27, 2010 Rajesh Naik QAI India Ltd Page
1
2. Agenda Process Performance Models Sub-Process Control
Managing Process Improvements Typical misconceptions and pitfalls
Page 2
3. CMMI Levels Source: How Does High Maturity Benefit the
Customer? Rick Hefner, Northrop Grumman Page 3
4. Source: SEI Webinar A Mini Tutorial for Building CMMI
Process Performance Models Stoddard, Schaaff, Young & Zubrow
Page 4
5. PPMs are complex - because reality is complex I want to go
from my residence to my friends place I have many options (have you
heard - we dont have options?) With a little thought we can come up
with options all seem valid Taxi Bus Auto My Friends terminus House
House Bus Bus Auto Bus Page 5
6. PPMs are complex - because reality is complex (contd.) There
are combination of resources that I would like to optimize Energy
level (physical, emotional) [Quality] Money [Cost/ effort] Elapsed
time [Schedule] (some may be more important than others, some may
start pinching when they cross a threshold) I may also have
constraints on some of the resources (e.g., I can spend a max of 3
hours elapsed time; or I dont want to spend more than Rs 500 on the
journey) Page 6
7. PPMs are complex - because reality is complex (contd.) Each
step of the journey (each process) would consume (or sometimes add
back) some of the resources From To Mode Energy Money Time My Res
Friend's Res Taxi 0.5 unit 400 Rs 1 hour My Res Terminus Bus 1.0
unit 50 Rs 1 hour My Res Terminus Auto 1.0 unit 120 Rs 45 mins
Terminus Friend's Res Bus 1.0 unit 50 Rs 1 hour Terminus Friend's
Res Auto 1.0 unit 120 Rs 45 mins What is the simplification in the
above table? Page 7
8. PPMs are complex - because reality is complex (contd.) Many
simplifications, significant enough to make a difference in the
choices made 1. Not taking into account wait times to get the
transport 2. Assuming that all values are invariant, fixed and
deterministic Look at the table in the previous slide and examine
whether the above two factors could have a significant impact on
your choice Page 8
9. Outcome of Complex Process is difficult to predict
intuitively Source: SEI Webinar A Mini Tutorial for Building CMMI
Process Performance Models Stoddard, Schaaff, Young & Zubrow
Page 9
10. Outcome of Complex Process is difficult to predict
intuitively Source: SEI Webinar A Mini Tutorial for Building CMMI
Process Performance Models Stoddard, Schaaff, Young & Zubrow
Page 10
11. Outcome of Complex Process is difficult to predict
intuitively Source: SEI Webinar A Mini Tutorial for Building CMMI
Process Performance Models Stoddard, Schaaff, Young & Zubrow
Page 11
12. Source: SEI Webinar A Mini Tutorial for Building CMMI
Process Performance Models Stoddard, Schaaff, Young & Zubrow
Page 12
13. Issues seen in PPM Implementation PPMs used only as
forecasting tools We do not have ANY choices PPMs used for a single
parameter assumption is that we have unlimited other resources PPMs
used in a stand alone manner one for defect prediction, one for
effort, one for schedule in reality every choice potentially
impacts all three simultaneously (everything is interdependent)
Page 13
14. Issues seen in PPM Implementation (contd.) Separate,
unrelated PPMs used in each phase ignoring the fact that phases
depend on each other (defect density found may be dependent on the
defect density present) Variation of processes and sub-processes
not taken into account Skill of people/ team not considered in the
PPM as a factor that impacts cost, schedule, defects Ignoring the
process tailoring done while evaluating PPMs Not re-evaluating the
process composition after some progress in the project Page 14
15. Issues seen in PPM Implementation (contd.) Assuming normal
(symmetric) distribution no real phenomena with human beings has a
normal distribution only gambling situations and computer games
have a normal distribution Page 15
16. Issues seen in PPM Implementation (contd.) Assuming that
changing the values of some process parameters will change process
behavior (without actually changing the process). Here is a classic
one if we increase the review effort, we will find more defects. if
you dont change the review process, why will it take more effort?
Underlying data in PPMs not based on true process/ sub-process
performance baselines PPMs trying to optimize Schedule Variance and
Effort Variance (Thankfully, we dont try to optimize defect
variance) Page 16
17. Sub-Process Control Choosing sub-processes and parameter to
control High contribution to the overall project for one or more
parameters (effort, schedule, quality) High contribution to the
variation in the overall project for one or more parameters
(effort, schedule, quality) The sub-process and parameters are
appropriate for statistical process control You have control on the
parameter - you can change something in the process Statistical
tool SPC charts Page 17
18. Issues seen in Sub-process Control Implementation Confusing
sub-process with parameter We are controlling schedule variance
sub-process Sub-process at a very high level (not really a
sub-process, but an aggregate) Trying to control output, instead of
the controllable input/ process You only monitor the output But you
can control the inputs and the process E.g., You cannot control
your weight (output) But you can control your diet and exercise
Page 18
19. Issues seen in Sub-process Control Implementation (contd.)
Data that is used is not actually from the same sub-process. E.g.,
speed of running is plotted but from races of different distances
(100 meters to marathon) Coding productivity from programs of
different sizes and complexity Coding productivity - taken from the
performance of people with different skill levels Page 19
20. Issues seen in Sub-process Control Implementation (contd.)
Accepting huge variation (wide range of process control limits)
because all data points follow the rules of process stability
(missing the woods for the trees) Using an arbitrary sequence in
the control chart (e.g., should we sequence by start date, or end
date?) Ignoring the fact that points with a large base have a
smaller variation by its very nature Page 20
21. Issues seen in Sub-process Control Implementation (contd.)
Discarding outliers, till all remaining data points show stability
of the sub-process Using baseline control limits, without
qualitatively determining that the sub- process continues to be the
same Ignoring the phenomenon that measurement and focus has an
impact on the stability Page 21
22. Managing Process Improvements OID & CAR Involves
Specifying improvement objectives Identifying processes/
sub-process to be improved Piloting proposed process improvements
Checking the impact; refining the improvement Deploying the change
Measuring the impact (after large scale deployment) Page 22
23. Issues seen in Process Improvement Implementation Drawing
cause-effect relationship from correlation (higher the review
effort -> higher defects found) Measuring the improvement in
just one parameter (defects found) while ignoring the impact on
other parameters (effort, schedule) Not trying to ensure that
conditions for before and after are same (except for the change
that is being tried) Is the skill level the same Is the input the
same? Page 23
24. Issues seen in Process Improvement Implementation (contd.)
Taking an isolated view of the improvement (not looking downstream)
Ignoring the impact of measurement and attention that is being
focused on the improvement Not checking over long durations Not
setting the right hypotheses for testing; and not using the right
tool for testing the hypotheses Page 24
25. Issues seen in Process Improvement Implementation (contd.)
Assuming that changing a quantitative parameter will bring about
the improvement (without changing the input or process. E.g., If we
increase the test effort then more defects will be found (but if we
use the same test process, how can we fruitfully utilize the
increased test effort?) Page 25
26. What we should see in future High Maturity Implementations
More comprehensive / holistic analysis Models should be factoring
in important soft influencers Skills/ Cross-skills (IPPD?) Team
work/ gelled teams (IPPD?) Impact of empowerment (IPPD?) Impact of
measurement Impact of management focus Page 26
27. About this Presentation Author: Rajesh Naik Consulting
Partner QAI India Limited [email protected][email protected] +91 9845488767 More resources on the
subject are available from the creator of this presentation at:
http://www.rajeshnaik.com Rajesh Naik, 2010 This work is released
under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0
Unported License license. This means you can use it for
non-commercial purposes so long as you include the copyright line
Rajesh Naik, 2010". If you create derivative works using this work,
they should also be made available under a similar license. For
further information go to
http://creativecommons.org/licenses/by-nc-sa/3.0/ For uses outside
the scope of the license, contact Rajesh Naik at
[email protected] PPT Template Copyright 2010 PowerPoint
Styles from http://www.powerpointstyles.com Page 27
28. Thank You Rajesh Naik Rajesh Naik Consulting Partner
Consulting Partner QAI India Limited QAI India Limited Email Email
[email protected][email protected] OR OR
[email protected][email protected] Mobile Mobile
+91 9845488767 +91 9845488767 Website Website www.rajeshnaik.com
www.rajeshnaik.com Also, have a look at the latest business novel:
Aligning Ferret: How an Organization Meets Extraordinary Challenges
By Swapna Kishore & Rajesh Naik Website:
http://www.postscript-impressions.com Page 28