SCAMPI Sampling Rules1
Sampling the SCAMPI Sampling Rulesor
Trying to Explain the Unexplainable
Pat O’Toole, PACTwww.PACTCMMI.com
May, 2012
SCAMPI Sampling Rules3
SCAMPI Class A v1.3
SCAMPI–Standard CMMI Appraisal Method for Process Improvement
SCAMPI MDD– SCAMPI Method Definition Document
250+ page process document for SCAMPI A appraisals
Specific source of angst–SCAMPI MDD v1.3 Section 1.1.4
–It’s only 7 pages, but do NOT drive or operate heavy equipment after reading it!
SCAMPI Sampling Rules4
New Terminology with MDD v1.3
Basic Units
Sampling Factors
Subgroups
Coverage Rules.
SCAMPI Sampling Rules5
New Terminology with MDD v1.3
Basic Units
Sampling Factors
Subgroups
Coverage Rules.
SCAMPI Sampling Rules6
Basic Unit
For CMMI-DEV, “Basic Unit” is roughly equivalent to “Project”–However, “project” doesn’t work as well for the
Services or Acquisition constellations
–“Basic Unit” is much more generic
–Therefore, for purposes of this presentation, I will use
“PROJECT!”
<of Work>
SCAMPI Sampling Rules7
New Terminology with MDD v1.3
Basic Units
Sampling Factors
Subgroups
Coverage Rules.
SCAMPI Sampling Rules8
Sampling Factors – Why?
Project work may be performed in different ways depending on various characteristics– Small projects may be conducted differently than large ones
– Project work conducted by your USA-based team may be done differently than your offshore development group
– Scrum projects are run differently than Waterfall projects
Sampling factors ensure that the organizational scope provides a robust (and complete) view of the various ways work is performed by the organization.
SCAMPI Sampling Rules9
Mandatory Candidate Sampling Factors
The Sampling Factors that MUST be considered:–Location (e.g., Minneapolis vs. Las Vegas)
–Customer (e.g., DoD vs. commercial industry)
–Size (e.g., small vs. medium vs. large vs. 4XL)
–Organizational Structure (e.g., different divisions)
–Type of Work (e.g., development vs. maintenance)
For each candidate, the question to be answered is:–Do different settings for this factor affect the way that
work is performed in the organization to be appraised? Yes – Relevant Sampling Factor No – Irrelevant Sampling Factor.
SCAMPI Sampling Rules10
Other Potential Sampling Factors
Other Relevant Sampling Factors may exist and, if so, must also be considered–Life Cycle (e.g., Scrum projects vs. Waterfall projects)
–Funding Source (e.g., customer vs. internal funding)
–Criticality (e.g., nice-to-have vs. life critical)
–Etc.
Your organization may have unique characteristics that significantly affect how work is performed–These would be included as Relevant Sampling Factors
in a SCAMPI A appraisal.
SCAMPI Sampling Rules11
ExampleProject Location Customer Size Type of Work
1 MSP Internal Small Maintenance2 MSP Internal Medium Development3 MSP External Medium Development4 MSP External Medium Maintenance5 MSP External Small Maintenance6 MSP External Large Development7 MSP External Medium Development8 MSP External Medium Development9 MSP External Medium Development
10 MSP External Large Development11 MSP External Small Maintenance12 MSP Internal Medium Development13 MSP External Medium Maintenance14 MSP External Small Maintenance15 MSP External Medium Development
SCAMPI Sampling Rules12
New Terminology with MDD v1.3
Basic Units
Sampling Factors
Subgroups
Coverage Rules.
SCAMPI Sampling Rules13
Subgroup
“Cluster” of projects that share common relevant sampling factor values and exhibit similar process implementations–If Customer, Size, and Type of Work are relevant
sampling factors, how many POSSIBLE subgroups are there? Customer: Internal / External Size: Small / Medium / Large Type of Work: Development / Maintenance
2 values for <Customer> x3 values for <Size> x2 values for <Type of Work> = 12 possible subgroups.
SCAMPI Sampling Rules14
How Many ACTUAL Subgroups?Project Location Customer Size Type of Work
1 MSP Internal Small Maintenance2 MSP Internal Medium Development3 MSP External Medium Development4 MSP External Medium Maintenance5 MSP External Small Maintenance6 MSP External Large Development7 MSP External Medium Development8 MSP External Medium Development9 MSP External Medium Development
10 MSP External Large Development11 MSP External Small Maintenance12 MSP Internal Medium Development13 MSP External Medium Maintenance14 MSP External Small Maintenance15 MSP External Medium Development
3-tuple #Internal , Smal l , Dev.
Internal , Medium, Dev.
Internal , Large, Dev.
Internal , Smal l , Maint.
Internal , Medium, Maint.
Internal , Large, Maint.
External , Smal l , Dev.
External , Medium, Dev.
External , Large, Dev.
External , Smal l , Maint.
External , Medium, Maint.
External , Large, Maint.
3-tuple #Internal , Smal l , Dev.
Internal , Medium, Dev.
Internal , Large, Dev.
Internal , Smal l , Maint.
Internal , Medium, Maint.
Internal , Large, Maint.
External , Smal l , Dev.
External , Medium, Dev.
External , Large, Dev.
External , Smal l , Maint.
External , Medium, Maint.
External , Large, Maint.
3-tuple #Internal , Smal l , Dev.
Internal , Medium, Dev. 2Internal , Large, Dev.
Internal , Smal l , Maint.
Internal , Medium, Maint.
Internal , Large, Maint.
External , Smal l , Dev.
External , Medium, Dev.
External , Large, Dev.
External , Smal l , Maint.
External , Medium, Maint.
External , Large, Maint.
3-tuple #Internal , Smal l , Dev.
Internal , Medium, Dev. 2Internal , Large, Dev.
Internal , Smal l , Maint.
Internal , Medium, Maint.
Internal , Large, Maint.
External , Smal l , Dev.
External , Medium, Dev.
External , Large, Dev.
External , Smal l , Maint.
External , Medium, Maint.
External , Large, Maint.
3-tuple #Internal , Smal l , Dev.
Internal , Medium, Dev. 2Internal , Large, Dev.
Internal , Smal l , Maint. 1Internal , Medium, Maint.
Internal , Large, Maint.
External , Smal l , Dev.
External , Medium, Dev.
External , Large, Dev.
External , Smal l , Maint.
External , Medium, Maint.
External , Large, Maint.
3-tuple #Internal , Smal l , Dev.
Internal , Medium, Dev. 2Internal , Large, Dev.
Internal , Smal l , Maint. 1Internal , Medium, Maint.
Internal , Large, Maint.
External , Smal l , Dev.
External , Medium, Dev.
External , Large, Dev.
External , Smal l , Maint.
External , Medium, Maint.
External , Large, Maint.
3-tuple #Internal , Smal l , Dev.
Internal , Medium, Dev. 2Internal , Large, Dev.
Internal , Smal l , Maint. 1Internal , Medium, Maint.
Internal , Large, Maint.
External , Smal l , Dev.
External , Medium, Dev. 5External , Large, Dev. 2External , Smal l , Maint. 3External , Medium, Maint. 2External , Large, Maint.
SCAMPI Sampling Rules15
Number of projects in the given subgroup
Number of projects in the given subgroup
How Many Projects per Subgroup?
Minimum # of projectsfrom a given subgroup
=
Number of subgroups
X
Total number of projects
SCAMPI Sampling Rules16
Back to Our Example…Subgroups #
InternalMedium
Development2
InternalSmal l
Maintenance1
ExternalMedium
Development5
ExternalLarge
Development2
ExternalSmal l
Maintenance3
ExternalMedium
Maintenance2
# of IMD = (6 x 2) / 15 = 0.8 => 1
# of ISM = (6 x 1) / 15 = 0.4 => 1
# of EMD = (6 x 5) / 15 = 2.0 => 2
# of ESM = (6 x 3) / 15 = 1.2 => 1
# of ELD = (6 x 2) / 15 = 0.8 => 1
# of EMM = (6 x 2) / 15 = 0.8 => 1.
SCAMPI Sampling Rules17
What if…
# of IMD = (6 x 200)/1500 = 0.8 => 1
# of ISM = (6 x 100)/1500 = 0.4 => 1
# of EMD = (6 x 500)/1500 = 2.0 => 2
# of ESM = (6 x 300)/1500 = 1.2 => 1
# of ELD = (6 x 200)/1500 = 0.8 => 1
# of EMM = (6 x 200)/1500 = 0.8 => 1
Subgroups #InternalMedium
Development200
InternalSmal l
Maintenance100
ExternalMedium
Development500
ExternalLarge
Development200
ExternalSmal l
Maintenance300
ExternalMedium
Maintenance200
SCAMPI Sampling Rules18
New Terminology with MDD v1.3
Basic Units
Sampling Factors
Subgroups
Coverage Rules.
SCAMPI Sampling Rules19
Coverage Rules
Three flavors:–Coverage Rules for Process Areas
–Coverage Rules for Basic Units (whoops! – Projects)
–Coverage Rules for Support Functions.
SCAMPI Sampling Rules20
Cover Rules for Process Areas
PA Coverage Rule 1 (in for a penny, in for a pound)–Evaluation of a given PA must cover all practices, both
specific and generic
–There is a wee bit of wiggle room here as the rule includes the words, “…for which they supply data”
PA Coverage Rule 2 (no need to keep pounding)–If a PA is performed in a consistent way across
subgroups, evaluation of that PA may be consolidated
–For example, if CM, DAR, PPQA, and RSKM are performed in the same manner across all projects, then they need not be evaluated for all 6 subgroups.
SCAMPI Sampling Rules21
Project Coverage Rules
Project Coverage Rule 1– For each subgroup, both artifacts and affirmations shall be
provided for at least one project for every process area This sampled project shall provide data for all process areas
Project Coverage Rule 2– For at least 50% of the sampled projects in each subgroup,
both artifacts and affirmations shall be provided for at least one process area
Project Coverage Rule 3– For all sampled basic units in each subgroup either
artifacts or affirmations shall be provided for at least one process area.
SCAMPI Sampling Rules22
ATLAS #19a
Can you apply what you’ve learned?–Well, heard?
Let’s find out…
Work individually or together to answer the questions in ATLAS #19a–There is no need to provide comments.
SCAMPI Sampling Rules23
Cover Rules for Process Areas
PA Coverage Rule 1–Evaluation of a given PA must cover all practices, both
specific and generic
–HINT: Start with the question, “Which projects contribute to its fulfillment?”
SCAMPI Sampling Rules24
Project Coverage Rules
Project Coverage Rule 1– For each subgroup, both artifacts and affirmations shall be
provided for at least one project for every process area This sampled project shall provide data for all process areas
Project Coverage Rule 2– For at least 50% of the sampled projects in each subgroup,
both artifacts and affirmations shall be provided for at least one process area
Project Coverage Rule 3– For all sampled basic units in each subgroup either
artifacts or affirmations shall be provided for at least one process area.
SCAMPI Sampling Rules25
ATLAS #19a Results
ATLAS #19a was distributed to all SCAMPI lead appraisers and only SCAMPI lead appraisers
61 SCAMPI lead appraisers responded–That’s about a 12% response rate
–But it’s still more than “one expert’s opinion”
Let’s see how YOU did compared with those responsible for REALLY understanding this stuff…
SCAMPI Sampling Rules26
Question 1
1. Is Process Area Coverage Rule 1 fulfilled?<Evaluation of a given PA must cover all practices, both specific and generic.>
36% Yes / 64% No
Which project(s) contribute to its fulfillment:
P1? 82% Yes / 17% No
P2? 90% Yes / 10% No
P3? 74% Yes / 24% No
P4? 7% Yes / 93% No.
SCAMPI Sampling Rules27
Question 2
2. Is Project (BU) Coverage Rule 1 fulfilled?“For each subgroup, both artifacts and affirmations shall be provided for at least one project for every process area”
34% Yes / 66% No
Which project(s) contribute to its fulfillment:
P1? 42% Yes / 56% No
P2? 14% Yes / 86% No
P3? 4% Yes / 96% No
P4? 4% Yes / 96% No.
SCAMPI Sampling Rules28
Question 3
3. Is Project (BU) Coverage Rule 2 fulfilled?“For at least 50% of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for at least one process area”
51% Yes / 49% No
Which project(s) contribute to its fulfillment:
P1? 54% Yes / 46% No
P2? 97% Yes / 3% No
P3? 22% Yes / 78% No
P4? 18% Yes / 82% No.
SCAMPI Sampling Rules29
Question 4
4. Is Project (BU) Coverage Rule 3 fulfilled?“For all sampled basic units in each subgroup either artifacts or affirmations shall be provided for at least one process area.”
48% Yes / 52% No
Which project(s) contribute to its fulfillment:
P1? 94% Yes / 6% No
P2? 98% Yes / 2% No
P3? 93% Yes / 7% No
P4? 25% Yes / 75% No.
SCAMPI Sampling Rules31
Modified Coverage Rules - #1
PA Coverage–1: Objective evidence provided for a PA must address all practices that are part of that PA, both specific practices as well as all generic practices. The Basic Unit and Support Function coverage rules indicate the types of objective evidence that must be provided for each practice.
Project Coverage–1: For each subgroup, both artifacts and affirmations shall be provided for each practice for at least one project.
Project Coverage–2: For at least 50 percent of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for each practice for at least one PA
Project Coverage–3: For all sampled projects in each subgroup either artifacts or affirmations shall be provided for each practice for at least one PA.
SCAMPI Sampling Rules32
Modified Coverage Rules - #2
PA Coverage–1: Objective evidence provided for a PA must address all practices that are part of that PA, both specific practices as well as all generic practices. [No change]
Project Coverage–1: For each subgroup, some combination of both artifacts and affirmations shall be provided for at least one project.
Project Coverage–2: For at least 50 percent of the sampled projects in each subgroup, some combination of both artifacts and affirmations shall be provided for at least one PA
Project Coverage–3: For all sampled projects in each subgroup either artifacts and/or affirmations shall be provided for at least one PA.
SCAMPI Sampling Rules33
Conclusion
The new sampling and coverage rules are intended to provide a truly representative set of projects that reflect how work is performed in the organization–There are some issues, however:
They only solve the “proportional problem,” not the “how much is enough” problem
There are multiple ways to interpret the rules as written, arriving at significantly different results
Based on ATLAS #19a, the SEI has committed to clarify, or even re-write the coverage rules
With respect to brain function, covering this topic is equivalent to drinking 6 beers; please drive carefully!
SCAMPI Sampling Rules34
Questions???
Pat O’TooleProcess Assessment, Consulting & Training (PACT)
If anyone is interested in receiving my monthly “Do’s and Don’ts” articles, either:
Give me a business card
Send me an email.