Current Trends in the Adoption of the CMMI

Current Trends in the Adoption of the CMMI Product Suite

Dave Zubrow, PhD Senior Member of the Technical Staff, Software Engineering Institute

[email protected]

Abstract

This paper examines the current state of

adoption of the Capability Maturity ModelIntegration (CMMI) Product Suite. It is based on

appraisal reports submitted to and processed by the Software Engineering Institute (SEISM) as of

July 10, 2003. Relevant appraisal data— including organization type, geographic

distribution, and appraisal history—are used to

describe CMMI adoption and use patterns. This paper also explains some characteristics of the

appraisal process and obtained results.

1. Background

The CMMI Product Suite [1] is composed of process improvement models, appraisal methods, and training materials. Benchmark versions of the models and appraisal method were first released early in 2002. Throughout that year, organizations using the models and appraisal methods migrated from the earlier versions of the CMMI Product Suite to more stable versions: CMMI v1.1 and the Standard CMMI Appraisal

Method for Process Improvement (SCAMPISM)v1.1 [2]. During this transition, organizations used a variety of appraisal methods with the v1.0X versions of the models. Transition to these versions is important for organizations using the legacy models and appraisal methods, as these will no longer be supported, recognized, or authorized for use after 2005.

This paper provides an early, in-depth look at the appraisal data reported to the SEI on the use of the CMMI Product Suite. It characterizes the organizations that have performed CMMI appraisals, provides some descriptive statistics on the appraisals, and summarizes the appraisal results. The data for this paper come from 93 CMMI-based appraisals using the benchmark versions of the model, CMMI v1.1, and appraisal method, SCAMPI v1.1. Unlike a previous version of this paper, results of appraisals using early versions of the model and appraisal method have been excluded.

2. CMMI Adoption

Insight can be gained into the types of organizations that are using CMMI models by looking at those organizations that have performed appraisals. According to SEI data, 87 organizations (some more than once) have performed a CMMI appraisal. The classification is shown below:

• Commercial 41 • DoD Contractor 30 • Civil Contractor 8 • Civil Organization 3 • In House Development 3 • Military Organization 2

Of some interest is the relatively rapid adoption of CMMI and the SCAMPI method among commercial organizations. This is somewhat surprising as some critics have claimed that the CMMI models were only useful to large government contracting organizations.

Another view of adoption is to look at the geographic distribution of the organizations. The countries and the number of organizations appraised using the CMMI models are listed below.

• USA 39 • Japan 21 • India 9 • United Kingdom 5 • Australia 3 • China 2 • France 2 • Taiwan 2 • Canada 1 • Russia 1 • South Korea 1 • Switzerland 1

Reviewing this list in light of the earlier introduction of the CMM for Software (SW-CMM) [4], it is clear that international interests are building on the existing foundation of the SW-CMM rather than starting anew. Indeed, 48,

Proceedings of the 27th Annual International Computer Software and Applications Conference (COMPSAC’03) 0730-3157/03 $ 17.00 © 2003 IEEE

Authorized licensed use limited to: UNIVERSITY OF NOTTINGHAM. Downloaded on December 22, 2008 at 00:42 from IEEE Xplore. Restrictions apply.

or 55%, of the organizations appraised against the CMMI models are outside of the United States.

3. Appraisal Characteristics

CMMI offers organizations multiple models from which to choose. Currently, the models cover the following disciplines: software engineering (SW), systems engineering (SE), integrated process and product development (IPPD), and supplier sourcing (SS). Organizations can scope their appraisal using any combination of the disciplines.

Likewise, organizations may choose to use the continuous or staged representation of the selected model. The following table reflects these choices.

Table 1. Model representation by model scope

Scope Staged Continuous SW + SE 34 8

SW + SE + IPPD

1 1

SW + SE + IPPD + SS

1 2

SW + SE + + SS

5 2

SW 20 6

SE 8 2

SW+IPPD+SS 1

SE+IPPD+SS 1

SE+SS 1

As can be seen, the most common scope is the software and systems engineering and staged is the most commonly used representation. In the staged representation, process areas are organized by maturity levels (1-5). The SW-CMM uses a staged representation. Despite this legacy factor, approximately one-quarter of the reported appraisals use the continuous representation. In the continuous representation, each process area is rated in terms of a capability level (0-5). The continuous representation has also been used in the ISO/IEC 15504 reference model and in the Systems Engineering Capability Model (EIA 731) [3]. Another observation regarding this table is that each of the disciplines offered in the CMMI models has

been used in an appraisal; however, not all combinations of model scope have been used. Assuming that an appraisal will include software or system engineering, 8 of the 12 possible combinations have been reported. It appears from this data that the community is indeed taking advantage of the flexibility and options offered by CMMI.

4. Description of the Results

Perhaps the most familiar depiction of appraisal results is the maturity profile [5], a simple bar chart showing the number of organizations at each maturity level. Organizations that choose to use the continuous representation do not get rated at a maturity level; therefore, their results are shown in a different format (process area capability profile). However, even those organizations using the staged approach may choose to not be rated.

0%

5%

10%

15%

20%

25%

30%

35%

40%

1 2 3 4 5

Figure 1. Maturity profile

The maturity profile in Figure 1 presents maturity level ratings from 70 appraised organizations. Overall, the profile depicts a relatively mature set of organizations when compared to the larger set of organizations assessed against the SW-CMM in the maturity profile. For instance, approximately 23% of the above ratings were for maturity level 4 and maturity level 5. This contrasts with the high maturity ratings reported in the April 2003 release of the maturity profile, which were only 15%.

Further analysis shows a difference in the pattern of ratings between those organizations reporting their results to the SEI for the first time and those reporting a reappraisal. First-appraisal organizations tend to have a less mature profile (as shown in Figure 2) with 18% reporting a rating of maturity level 1 and only 14% reporting a rating of level 4 or 5.



0%

5%

10%

15%

20%

25%

30%

35%

40%

1 2 3 4 5

Figure 2. Maturity profile for first-appraisal organizations

By way of contrast, 53% of the 17 organizations that had performed previous assessments using SW-CMM were rated at maturity level 5 against the CMMI. Furthermore, all reappraisals, except one, against the CMMI resulted in the same or an improved maturity level rating. One exception occurred when the scope of the appraisal was extended beyond software engineering. Another interesting finding of this analysis is that nearly one-third of the first-appraisal organizations chose to not be rated.

Another question regarding the use of the CMMI Product Suite concerns the way new process areas are appraised. The CMMI models have more process areas than the SW-CMM and some organizations have voiced concern that they might not be rated as highly as they had been previously. This, in fact, may be probable if the organization had not been addressing a particular process area in the past. For example, the CMMI models now contain a process area for Measurement and Analysis. As noted above, reviewing the pattern of reappraisal results does not support a “backsliding” hypothesis. Additionally, an earlier analysis of 33 process area profiles showed a success rate for the Measurement and Analysis process area that was comparable to that for process areas that had been represented in the SW-CMM.

Another factor to be taken into account with respect to the concern that the “bar has been raised” is the option to perform appraisals where no formal rating is made. This option to perform appraisals as a readiness review with no formal rating allows organizations the ability to better prepare for and time their appraisals where ratings will be given. Indeed, 31% of the first-time appraising organizations took advantage of this option. While this undoubtedly occurred in

the past, greater visibility into this behavior is now possible since these results are now reported to the SEI.

5. Summary

This paper presented an initial analysis of the organizations and results of CMMI appraisals. It provided some summary descriptions as well as comparisons with results from assessments using the SW-CMM. Notable among the results is the relatively rapid adoption of the model in general and the apparent embracing of it by commercial organizations. Also notable, compared to the SW-CMM, is the relatively rapid international adoption of the model. As a point of comparison, 55% of the CMMI appraising organizations included in this study are outside of the United States while only 47% of the organizations included in the latest SW-CMM maturity profile are from outside of the United States.

The concern among some organizations that there would be significant backsliding in maturity level ratings has not materialized. Also, the users of the model are in fact utilizing the variety of choices in representation, model scope, and whether to do a formal rating. These are initial observations based on a relatively small number of appraisals. Time will tell whether these observations hold true as more organizations adopt and appraise their practices against the CMMI models.

6. References

[1] CMMI Product Team. CMMI for Systems Engineering/Software Engineering/Integrated Product and Process Development/Supplier Sourcing, Version 1.1, Continuous Representation (CMMI-SE/SW/IPPD/SS, V1.1) (Continuous: CMU/SEI-2002-TR-011) and Staged Representation (CMU/SEI-2002-TR-012). Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon University, 2002. http://www.sei.cmu.edu/cmmi/models/models.html

[2] Members of the Assessment Method Integrated Team. Standard CMMI Appraisal Method for Process Improvement (SCAMPI), Version 1.1: Method Definition Document (CMU/SEI-2001-HB-001). Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon University, 2002. http://www.sei.cmu.edu/publications/documents /01.reports/01hb001.html

[3] Electronic Industries Association. Systems Engineering Capability Model (EIA/IS-731). Washington, D.C.: Electronic Industries Association, 1998. http://geia.org/sstc/G47/page6.htm



[4] Mark C. Paulk, Bill Curtis, Mary Beth Chrissis, and Charles V. Weber, Capability Maturity Model for Software, Version 1.1, Software Engineering Institute, CMU/SEI-93-TR-24, DTIC Number ADA263403, February 1993. http://www.sei.cmu.edu/publications/documents /93.reports/93.tr.024.html

[5] Software Engineering Institute. Maturity Profile. http://www.sei.cmu.edu/sema/profile.html

Capability Maturity Model and CMMI are registered in the

U.S. Patent and Trademark Office by Carnegie Mellon

University.SM SEI and SCAMPI are service marks of Carnegie Mellon

University.



Documents

Current Trends in the Adoption of the CMMI