Upload
dangcong
View
226
Download
1
Embed Size (px)
Citation preview
CMMI Working Group 1
CMMI® Surveillance Appraisals: A Modest Proposal
Michael Campo, Raytheon Company NDIA CMMI® Working Group
NDIA Systems Engineering Division
2011 NDIA CMMI Technology Conference and User Group
November 16, 2011 ® CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
CMMI Working Group 2
Note The CMMI Surveillance Appraisal method described in this presentation represents the thoughts and recommendations of the NDIA CMMI Working Group, with inputs from selected stakeholders. The method has not been approved or endorsed by the CMMI Program. You feedback will help determine if this is a viable concept. Suggestions are welcome.
CMMI Working Group 3
Agenda Introduction
General Concepts
Operational Concept
Methodology • Organizational Unit Change Constraints • Organizational Scope and Sampling
Case Study Example
Characterization & Rating
Team Qualifications
Open Issues
Summary
CMMI Working Group 4
Introduction SCAMPI A appraisal costs have been problematic for many
businesses seeking to use CMMI and SCAMPI A to benchmark their organizational processes. • Appraisals are expensive and often perceived as a burden to CMMI
adoption.
After a SCAMPI A is completed, an organization must then conduct additional SCAMPI A appraisals every 3 years to maintain the previously achieved rating. • Diverts valuable process improvement funding to a benchmarking activity
that may offer little value in return.
Goal: Develop a CMMI Surveillance Appraisal • Remove the need for a full SCAMPI A appraisal every 3 years by defining
a low cost surveillance appraisal method to extend the lifespan of a SCAMPI A rating without compromising rating integrity.
• Align with ISO 9001/AS9100, TL9000 surveillance benchmarking activities.
CMMI Working Group 5
General Concepts (1 of 2)
A CMMI surveillance appraisal should be not be more than 25-30% of the cost of a SCAMPI A.
• This should be a design constraint on the method. A CMMI surveillance appraisal must be rigorous enough to ensure full
confidence in extending the results of a SCAMPI A appraisal. A CMMI surveillance appraisal will provide an extension of SCAMPI A
results for 2 years from the completion date of the surveillance appraisal.
Maximum time between SCAMPI A appraisals is 7 years.
• CMMI surveillance appraisals cannot extend SCAMPI A rating(s) beyond 7 years. • Two CMMI surveillance appraisals could be conducted in that timeframe
CMMI surveillance appraisals can be used for all CMMI constellations
and the People CMM. Both staged and continuous representations will be supported.
CMMI Working Group 6
General Concepts (2 of 2)
Model scope cannot be greater than the SCAMPI A baseline model scope.
• Can’t add PAs that weren’t in baseline, including previously not applicable PAs. • Can’t increase maturity levels or capability levels using a CMMI surveillance appraisal.
Organizational unit change must be within CMMI surveillance appraisal method constraints.
If a CMMI surveillance appraisal identifies problems (e.g., goal failures) the SCAMPI A rating cannot be extended.
• The SCAMPI A rating would not be immediately revoked. • Additional CMMI surveillance appraisals would not be permitted until after another SCAMPI A.
If a SCAMPI A rating (or successful surveillance appraisal extension) expires before a surveillance appraisal is successfully conducted, the rating cannot be restored via a surveillance appraisal.
Maximum duration of surveillance appraisal is 45 days.
CMMI surveillance appraisal ratings should be noted as such in PARS.
CMMI Working Group 7
T+1 Year
CMMI ML rating validity
Extended CMMI ML rating validity
T+2: Conduct
Surv. Appraisal
Example Appraisal Timelines T0:
SCAMPI-A
Extended CMMI ML rating validity
T+4: Conduct
Surv. Appraisal
Use 2 Surveillance Appraisals to extend rating 3 years
T+2 Years
T+7 Years
T+6 Years
T+5 Years
T+4 Years
T+3 Years
CMMI ML rating validity
Extended CMMI ML rating validity
T+3: Conduct
Surv. Appraisal
Extended CMMI ML rating validity
T+5: Conduct
Surv. Appraisal
Use 2 Surveillance Appraisals to extend rating 4 years
CMMI Working Group 8
Surveillance Appraisals - Operational Concept
SCAMPI-A •Appraisal scope (model, basic units, sampling) •Ratings •Findings (strengths, weaknesses, opportunities)
•Surveillance appraisal plan (preliminary) •Schedule, timelines, constraints, risks •SEI notification 30 days in advance •Appraisal scope, initial data collection
Surveillance Appraisal Planning
OU
•Basic Units •Support Functions
Min
i- Pl
anni
ng Stage 1: Contextual Overview
•Business objectives, process plans •Interviews (sponsor, EPG, QA leads) •Performance measures •Process deployment/maintenance •QA audit analysis
•Performance measures •Process deployment/maintenance •QA audit analysis •Scope: initial areas of emphasis (PAs, units, etc.) •Data collection plan for Stage 2
OU
•Basic Units •Support Functions
Min
i- Pl
anni
ng Stage 2: High-Yield Artifacts
•Work products (basic units, support); high leverage artifacts •Interviews (basic unit/support leads as needed)
•Final model sample scope: investigative and random sampling (SGs, GPs) •Final data coverage •Data collection plan for Stage 3
OU
•Basic Units •Support Functions
Mni
- Pl
anni
ng
•Practice characterizations •Goal ratings •Surveillance appraisal final findings •Extension of Maturity Level rating (+2 yrs x 2) •Package and archive appraisal assets
Stage 3: Model Sampling •Artifacts, affirmations (interviews) •Sampling of model SGs, GPs
T-0:
< T+3 yrs:
Time, coverage, confidence
CMMI Working Group 9
Operational Concept Stage 1: Contextual Overview
Goals: Contextual understanding of the organization’s process state, process deployment and maintenance status, possible noncompliance trends
Interviews: Sponsor, EPG leads, Quality leads
Activities: Review performance measures from the organization • Business objectives, process performance objectives • Measurement and analysis related to achieving objectives • Corrective actions and process improvements
Review status of process deployment and maintenance • Status of deployment to basic units, support functions • Results of process implementation/compliance monitoring • Status of improvement activities, appraisal findings
Review PPQA audit-type analysis • Trend reports of audit non-compliances • Relate significant areas of concern to CMMI process areas
Output: Potential areas for further investigation in Stages 2 and 3
OU
•Basic Units•Support Functions
Min
i-Pl
anni
ng
Stage 1: Contextual Overview•Business objectives, process plans•Interviews (sponsor, EPG, QA leads)•Performance measures•Process deployment/maintenance•QA audit analysis
•Performance measures•Process deployment/maintenance•QA audit analysis•Scope: initial areas of emphasis (PAs, units, etc.)•Data collection plan for Stage 2 Model Coverage
(Conceptual)
Typical PAs: •MA, PPQA •OPF, OPD, OT, OPM, OPP •Probe selected PAs based on data collected
GPs
SP
s
CMMI Working Group 10
Operational Concept Stage 2: High Yield Artifacts
Goals: Determine Stage 3 model sample scope
Interviews: Optional interviews with basic unit and support function leads as needed for high yield artifact analysis
Activities: Review example high yield artifacts • Plans, documents, reviews, financial reports, etc. (MDD Table18) • Collected from basic units and support functions in org scope •Identify areas of concern, initial Stage 3 model sample scope
Use random sampling to fill Stage 3 model sample scope • Random sampling of SGs/GPs to supplement evidence review • Must meet method-defined minimum model sampling reqts • Status of improvement activities, addressing appraisal findings
“Finalize” Stage 3 model sample scope • Appraisal team can identify additional SGs/GPs • Determine data coverage reqts based on model scope and applicable MDD coverage rules • Update Data Collection Plan for Stage 3
Output: Final Stage 3 model scope, updated data collection plan
OU
•Basic Units•Support Functions
Min
i-Pl
anni
ng
Stage 2: High-Yield Artifacts•Work products (basic units, support);high leverage artifacts•Interviews (basic unit/support leads as needed)
•Final model sample scope: investigative and random sampling (SGs, GPs)•Final data coverage•Data collection plan for Stage 3 Model Coverage
(Conceptual)
•Map OE to model SPs, GPs •Identify model sample scope for Stage 3 (see sampling rules)
-Targeted sampling (investigative) -Random sampling (supplementary)
GPs
SP
s
CMMI Working Group 11
Operational Concept Stage 3: Model Sampling
Goals: Rate SGs and GGs included in model sample scope to determine if SCAMPI A rating can be extended
Interviews: As needed for affirmations based on applicable MDD coverage rules
Activities: Verify objective evidence • Review data collected by OU based on model sample scope and MDD coverage rules • Conduct interviews for affirmations
Perform practice characterizations as described in MDD • Characterize all SPs for SGs in model sample scope • Characterize GPs in model sample scope
Derive findings and rate goals • Document preliminary findings (strengths and weaknesses) • Validate preliminary findings (optional) • Rate goals in model sample scope, determine ratings extension
Report results •Deliver final findings. Package and archive appraisal assets.
Output: Final findings and ratings
Model Coverage (Conceptual)
•Collect and verify OE for SPs and GPs in model sample scope •See details for sampling and rating methods •Follow MDD rules for practice characterization and aggregation
GPs
SP
s
OU
•Basic Units•Support Functions
Mni
-Pl
anni
ng •Practice characterizations•Goal ratings•Surveillance appraisal final findings•Extension of Maturity Level rating (+2 yrs x 2)•Package and archive appraisal assets
Stage 3: Model Sampling •Artifacts, affirmations (interviews)•Sampling of model SGs, GPs
CMMI Working Group 12
Organizational Unit Change Constraints
If the Organizational Unit has evolved since the previous SCAMPI A… • The determinant on whether a surveillance appraisal is appropriate is the amount of
change in the sampling factors used to scope the previous SCAMPI A.
Change in Sampling Factors/Values Surveillance Appraisal Candidate* No change. Yes # sampling factors/values has been reduced, with no new sampling factors/values added
Yes
# sampling factors/values have grown or sampling factors/values have changed
No
Internal reorganizations or external mergers/acquisitions
Maybe •Sampling factor/value changes? – see above •If no process implementation impact - Yes
* - The lead appraiser makes the final decision on whether a CMMI surveillance appraisal may be performed. Document rationale in plan/ADS.
CMMI Working Group 13
Organizational Scope and Sampling (1 of 2)
Basic Unit and Support Function Sampling
• Organizational scope is determined using SCAMPI A V1.3 MDD rules. • Sampling factors beget subgroups which beget sample basic units and
support functions.
• Sampled basic units and support functions may or may not be the same as those sampled in baseline SCAMPI A.
• All sampled basic units in each subgroup and support functions undergo Stage 2 high yield artifact review and Stage 3 sampling.
• Stage 3 data coverage is based upon applicable SCAMPI A V1.3 MDD rules applied to Stage 3 model sample scope.
• Coverage Rule 1 for Process Areas is not applicable by design of the CMMI surveillance appraisal method.
• Process Area practice coverage is determined by SGs and GPs identified during Stages 1 and 2.
CMMI Working Group 14
Organizational Scope and Sampling (2 of 2)
Model Sampling Methodology • Specific Goals and Generic Practices are identified in Stage 1 and 2 for
inclusion in Stage 3 sample scope (investigative sampling). • A minimum of 33% of SGs and GPs in model scope must be included in Stage
3 sample scope. • Random sampling of SGs and GPs in model scope supplements
investigative sampling performed in Stage1 and Stage 2. • At least two SGs must be selected for random sampling (even if Stage 1
and Stage 2 identified SGs are ≥ 33%. • At least one SG from each Process Area Category (within model scope)
must be included in the Stage 3 sample. • GPs identified for Stage 3 sample are sampled for all Process Areas with
SGs in Stage 3 sample scope. • Appraisal team can always request additional evidence beyond original sample
scope.
CMMI Working Group 15
Case Study Example (1 of 5)
Reference MDD V1.3 Appendix F, Case Study 1 (expanded to Maturity Level 5). OU undergoes a CMMI Surveillance Appraisal 2 years after achieving Maturity Level 5.
• Stage 1 • Team meets with Sponsor
• Observes business objectives unchanged since last appraisal • Team meets with EPG Lead, Quality Leads
• Observes some updated OSSP standards and quality issues with suppliers • Stage 2
• Team reviews OSSP, project plans, tailorings, status reporting packages • Observes QPPOs unchanged and some projects not meeting objectives
without appearing to take action or use CAR • Observes supplier quality, cost, and schedule problems in status. Supplier
stakeholder involvement questionable. • Stage 3
• Team identifies all SAM, OPP, QPM, CAR, OPM SGs, and IPM SG2 (11 SGs), and GP 2.2 and GP 2.7 for Stage 3 sample
• For 33% SG coverage (16), randomly select 5 more SGs • For 33% GP coverage (4), randomly select 2 more GPs
CMMI Working Group 16
Case Study Example (2 of 5)
SGs and GPs identified during Stages 1 and 2: • All SGs in SAM, OPP, QPM, CAR, OPM, and also IPM SG2 • GPs: GP 2.2, GP 2.7
GPsGP 2.2GP 2.7
PA SG # SPsSAM SG 1 3SAM SG 2 3IPM SG 2 3OPP SG 1 5QPM SG 1 4QPM SG 2 3CAR SG 1 2CAR SG 2 3OPM SG 1 3OPM SG 2 4OPM SG 3 3
CMMI Working Group 17
Case Study Example (3 of 5)
Use Random Number Generator to select 5 additional SGs and 2 additional GPs.
GPs1 GP 2.12 GP 2.33 GP 2.4 4 GP 2.55 GP 2.66 GP 2.87 GP 2.98 GP 2.109 GP 3.110 GP 3.2
PA SG # SPs1 CM SG 1 32 CM SG 2 23 CM SG 3 24 MA SG 1 45 MA SG 2 46 PPQA SG 1 27 PPQA SG 2 28 PMC SG 1 79 PMC SG 2 3
10 PP SG 1 411 PP SG 2 712 PP SG 3 313 REQM SG 1 514 DAR SG 1 615 IPM SG 1 716 OPD SG 1 717 OPF SG 1 318 OPF SG 2 219 OPF SG 3 4
PA SG # SPs20 OT SG 1 421 OT SG 2 322 PI SG 1 323 PI SG 2 224 PI SG 3 425 RD SG 1 226 RD SG 2 327 RD SG 3 528 RSKM SG 1 329 RSKM SG 2 230 RSKM SG 3 231 TS SG 1 232 TS SG 2 433 TS SG 3 234 VAL SG 1 335 VAL SG 2 236 VER SG 1 337 VER SG 2 338 VER SG 3 2
11 investigative sample SGs + 5 random sample SGs = 16 = 33% SG model scope. 2 investigative sample GPs + 2 random sample GPs = 4 = 33% GP model scope.
CMMI Working Group 18
Case Study Example (4 of 5)
MDD Appendix F Case Study 1 Table 27 expanded to ML5 SCAMPI A Practice Instantiations
Subgroups Sample SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GPLA, Commercial, Large, Short
1 5 12 14 12 10 12 8 12 7 12 6 12 10 12
LA, Commercial, Small, Short
1 5 12 14 12 10 12 8 12 7 12 6 12 10 12
LA, DoD, Large, Long 1 5 12 8 12 7 12 10 125 12 8 12 0 0 10 125 12 0 0 0 0 0 0
Dayton, DoD, Large, Long 1 5 12 8 12 7 12 10 125 12 8 12 0 0 10 120 0 8 12 0 0 0 0
Enterprise Process Group 9 12 7 12 7 1235 84 42 36 30 36 56 84 28 48 8 24 12 24 9 12 7 12 7 12 60 72
Subgroups Sample SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GPLA, Commercial, Large, Short
1 8 12 9 12 8 12 5 12 10 12 7 12 6 12 7 12 5 12
LA, Commercial, Small, Short
1 8 12 9 12 8 12 5 12 10 12 7 12 6 12 7 12 5 12
LA, DoD, Large, Long 1 8 12 9 12 8 12 5 12 6 12 5 128 12 9 12 8 12 5 12 6 12 5 120 0 0 0 0 0 0 0 0 0 0 0
Dayton, DoD, Large, Long 1 8 12 9 12 8 12 5 12 6 12 5 128 12 9 12 8 12 5 12 6 12 5 120 0 0 0 0 0 0 0 0 0 0 0
Enterprise Process Group 5 12 10 1248 72 54 72 48 72 30 72 30 36 21 36 36 72 5 12 21 36 30 72 10 12
4 12
REQM PP PMC MA CM PPQA SAM OPF OPD OT RD
4 12Dayton, DoD, Small, Long 2
TS PI VER VAL IPM RSKM
14 12 10 12LA, DoD, Small, Short 2
DAR OPP QPM CAR OPM
12LA, DoD, Small, Short 2
Dayton, DoD, Small, Long 2
10 12 7 12 7
62710081635
SCAMPI A SPs SCAMPI A GPs Total
CMMI Working Group 19
Case Study Example (5 of 5)
MDD Appendix F Case Study 1 Table 27 expanded to ML5 Surveillance Appraisal Practices
Subgroups Sample SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GPLA, Commercial, Large, Short
1 * * * * * * 4 4 2 4 6 4 * *
LA, Commercial, Small, Short
1 * * * * * * 4 4 2 4 6 4 * *
LA, DoD, Large, Long 1 * * * * * * 4 4 2 4 * ** * * * * * 4 4 * * * ** * * * * * * * * * * *
Dayton, DoD, Large, Long 1 * * * * * * 4 4 2 4 * ** * * * * * 4 4 * * * ** * * * * * 4 4 * * * *
Enterprise Process Group 2 4 7 4 * *0 0 0 0 0 0 28 28 8 16 0 0 12 8 2 4 7 4 0 0 0 0
Subgroups Sample SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GP SP GPLA, Commercial, Large, Short
1 * * 4 4 * * * * 3 4 * * * * 7 4 5 4
LA, Commercial, Small, Short
1 * * 4 4 * * * * 3 4 * * * * 7 4 5 4
LA, DoD, Large, Long 1 * * 4 4 * * * * * * 5 4* * 4 4 * * * * * * 5 4* * * * * * * * * * 5 4
Dayton, DoD, Large, Long 1 * * 4 4 * * * * * * 5 4* * 4 4 * * * * * * 5 4* * * * * * * * * * * *
Enterprise Process Group 5 4 10 40 0 24 24 0 0 0 0 9 12 0 0 0 0 5 4 21 12 35 28 10 4
* - reviewed in Stage 1 and 2 only
OT RD
* *
REQM PP PMC MA SG1 CM SG2 PPQA
LA, DoD, Small, Short 2
SAM OPF SG2 OPD SG1
* *Dayton, DoD, Small, Long 2
TS PI SG3 VER VAL IPM SG2 RSKM DAR OPP QPM CAR OPM
4LA, DoD, Small, Short 2
Dayton, DoD, Small, Long 2
3 4 * * 7
161144305
Total Stage 3 SPs SampledTotal Stage 3 GPs SampledTotal Stage 3 Sample
Support: MA, CM, CAR
Proj. Mgt.: SAM, IPM, QPMProc. Mgt.: OPD, OPF, OPP, OPMEng.: PI
26%14%19%
SP % of SCAMPIGP % of SCAMPITotal % of SCAMPI
Investigative Sample Random Sample
CMMI Working Group 20
Characterizations and Ratings Characterizations are done on the Stage 3 sampled practices using
SCAMPI A MDD characterization rules. Goals that have Stage 3 sampled practices are rated based on the
characterizations of the sampled practices and associated weaknesses (if any).
If all sampled goals are satisfied, the SCAMPI A rating can be
extended. If any sampled goals are not satisfied, the SCAMPI A rating cannot
be extended.
CMMI Working Group 21
Team Qualifications (1 of 2)
CMMI Surveillance Appraisal Team member qualifications are more stringent than SCAMPI A requirements • Experienced team members able to reach accurate conclusions with less evidence
SCAMPI A Surveillance Appraisal Minimum team size is 4, including the lead appraiser
Minimum team size is 2, including the lead appraiser •Members of the appraised organization are allowed to be team members but the team must have a minimum of 2 team members external to the OU (including the lead appraiser).
Team members complete Introductory to model course
Team members must have previous experience as appraisal team members on at least two SCAMPI A appraisals
Average of 6 years field experience wrt reference model, aggregate of 10 yrs management experience, 1 team member with 6 yrs mgt. experience
Each team member must have minimum of 6 yrs field experience, (including experience performing practices from the process areas that the team member is reviewing)
Aggregate of 25 years field experience N/A
CMMI Working Group 22
Team Qualifications (2 of 2)
High Maturity
SCAMPI A Surveillance Appraisal Certified High Maturity Lead Appraiser (HMLA)
Certified High Maturity Lead Appraiser (HMLA)
All members of high maturity mini-team have high maturity experience
Team members reviewing high maturity process areas must have been on a previous SCAMPI A high maturity appraisal
HMLA or ATM with statistical analysis & other high maturity training assigned to high maturity mini-teams
At least one team member reviewing high maturity process areas must: • have been on a previous SCAMPI A high maturity appraisal as part of a high maturity mini-team or •be a certified HMLA who has been on a high maturity appraisal as lead or team member
Team as a whole has collective experience implementing high mat activities
N/A
CMMI Working Group 23
Open Issues V1.2 Model and Method Sunset issue
• Should CMMI V1.2 model appraisals be candidates for surveillance appraisals? • Under discussion • If no, there may be no surveillance appraisal market until 2013 • If yes, outdated model ratings are being extended
• Should surveillance appraisals be done with MDD V1.2? • Recommendation – NO • The CMMI Surveillance Appraisal method requires use of the SCAMPI A
V1.3 MDD methods.
• Appraisal scoping, characterizations, ratings, etc. must be done in accordance with MDD V1.3
• For example, an OU that used MDD V1.2 would have to convert critical factors to sampling factors, and apply the MDD V1.3 subgroup/sampling process, data coverage rules, etc.
• The lead appraiser must ensure that the OU in the CMMI Surveillance Appraisal falls within the OU change constraints described earlier in this presentation, and document the rationale in the CMMI Surveillance Appraisal plan.
CMMI Working Group 24
Summary
Goal: Define a low cost CMMI Surveillance Appraisal method to extend the lifespan of a SCAMPI A rating without compromising rating integrity.
Lower costs… …without compromising
integrity Less evidence collection and review by the organizational unit and appraisal team
Organizational Unit has already undergone a recent baseline SCAMPI A appraisal
Fewer, but more experienced team members
Team member qualifications are more stringent than SCAMPI A V1.3 MDD team member qualifications
Fewer SCAMPI A appraisals performed just to maintain ratings
Combines investigative sampling and random sampling of specific goals and generic practices
CMMI Working Group 25
For More Information….
NDIA CMMI Working Group http://www.ndia.org/Divisions/Divisions/SystemsEngineering/Pages/CMMI_Working_Group.aspx
Jim Armstrong Stevens Institute
Steve Austin Lockheed Martin
Dan Blazer SAIC
Michael Campo Raytheon Company
Juliet Davis The Boeing Company
Geoff Draper Harris Corporation
Jeffrey L. Dutton Jacobs Technology
Lisa Ming BAE Systems
Wendell Mullison General Dynamics, Land Systems
Kathy Smith Hewlett-Packard
Rick Welch Northrop Grumman