Convener: Houman Younessi

1

Convener: Houman Younessi

Convener: Houman Younessi

Software Engineering Management

Software Engineering Management

Course # CISH-6050Course # CISH-6050

Lecture 7: Lecture 7:

Software Process AssessmentSoftware Process Assessment

07/09/201207/09/2012

2 CISH-6050 - Software Engineering Management

AGENDA …AGENDA …

• Process Assessment- SEI SW-CMM Improvement Approach- Do organizational assessments pay off?- Whistleblowers- Reference: SPICE Assessment

Documents


Documents


SW-CMM Process AssessmentSW-CMM Process Assessment

• SEI SW-CMM Improvement Approach: IDEAL- Initiating- Diagnosing- Establishing- Acting- Leveraging

• Diagnose process, establish priorities, & act on them

• SEI SW-CMM Improvement Approach: IDEAL- Initiating- Diagnosing- Establishing- Acting- Leveraging

• Diagnose process, establish priorities, & act on them


SW-CMM: IDEALSW-CMM: IDEAL

• Initiating:- Stimulus for improvement- Set context & sponsorship- Establish improvement infrastructure

• Diagnosing:- Appraise & characterize current practice- Develop recommendations- Document phase results

• Initiating:- Stimulus for improvement- Set context & sponsorship- Establish improvement infrastructure

• Diagnosing:- Appraise & characterize current practice- Develop recommendations- Document phase results


SW-CMM: IDEAL …SW-CMM: IDEAL …

• Establishing:- Set strategy & priorities- Establish process action items- Plan actions

• Acting:- Define processes and measures- Plan & execute pilots- Plan, execute, & track installation

• Establishing:- Set strategy & priorities- Establish process action items- Plan actions

• Acting:- Define processes and measures- Plan & execute pilots- Plan, execute, & track installation


SW-CMM: IDEAL …SW-CMM: IDEAL …

• Leveraging:- Document & analyze lessons- Revise organizational approach

• Leveraging:- Document & analyze lessons- Revise organizational approach


SW-CMM Process AssessmentSW-CMM Process Assessment

• General Classes of SW-CMM Appraisal1. Software Process Assessment

Determine state of organization’s software process

2. Software Capability Evaluations Identify contractors qualified to

perform software work

• General Classes of SW-CMM Appraisal1. Software Process Assessment

Determine state of organization’s software process

2. Software Capability Evaluations Identify contractors qualified to

perform software work


SW-CMM Process Assessment …SW-CMM Process Assessment …

• Software Process Assessment:- Identify improvement priorities within

organization- Assessment team uses CMM to guide

identifying & prioritizing findings- Findings & KPA guidance used to plan

improvement strategy for organization

• Software Process Assessment:- Identify improvement priorities within

organization- Assessment team uses CMM to guide

identifying & prioritizing findings- Findings & KPA guidance used to plan

improvement strategy for organization



• Software Capability Evaluations:- Identify risks associated with a project

or contract to build high quality on schedule & budget

- During acquisition process, capability evaluation may be performed on bidders

- Findings of an evaluation may be used to identify risk with using a contractor

- Performed on existing contracts to monitor process performance

• Software Capability Evaluations:- Identify risks associated with a project

or contract to build high quality on schedule & budget

- During acquisition process, capability evaluation may be performed on bidders

- Findings of an evaluation may be used to identify risk with using a contractor

- Performed on existing contracts to monitor process performance


Software Process Assessment & Capability Evaluation Steps:

Software Process Assessment & Capability Evaluation Steps:



• Common Steps:- Team Selection

Select team trained in CMM Knowledgeable in SE & mgmt

- Maturity Questionnaire Site reps complete questionnaire

- Response Analysis Analyze results of questionnaire Investigation areas = KPAs

• Common Steps:- Team Selection

Select team trained in CMM Knowledgeable in SE & mgmt

- Maturity Questionnaire Site reps complete questionnaire

- Response Analysis Analyze results of questionnaire Investigation areas = KPAs



• Common Steps …- On-site Visit

Using results analysis, conduct on-site visit to view process areas

Using KPAs as guide, question, listen, review & synthesize info

Apply professional judgment Document rationale for situations

where KPAs not met

• Common Steps …- On-site Visit

Using results analysis, conduct on-site visit to view process areas

Using KPAs as guide, question, listen, review & synthesize info

Apply professional judgment Document rationale for situations

where KPAs not met



• Common Steps …- Findings

At end of on-site period, team produces list of findings

Identifies strengths & weaknesses of org’s software processes

Software Process Assessment -> Basis for PI recommendations

Software Capability Evaluation -> Findings part of risk analysis

• Common Steps …- Findings

At end of on-site period, team produces list of findings

Identifies strengths & weaknesses of org’s software processes

Software Process Assessment -> Basis for PI recommendations

Software Capability Evaluation -> Findings part of risk analysis



• Common Steps …- KPA Profile

Team prepares KPA profile, showing where KPAs satisfied / not satisfied by organization

KPA can be satisfied and still have associated findings, as long as findings don’t identify major problems achieving goals of KPA

• Common Steps …- KPA Profile

Team prepares KPA profile, showing where KPAs satisfied / not satisfied by organization

KPA can be satisfied and still have associated findings, as long as findings don’t identify major problems achieving goals of KPA



• Differences: Process Assessments & Capability Evaluation- Results of process assessment or

capability evaluation may differ - Assessment / evaluation scope may

vary: Different definitions of ‘Organization’ Org may be based on senior

management, geo location, common app, profit/loss center, etc.

Sample of selected projects

• Differences: Process Assessments & Capability Evaluation- Results of process assessment or

capability evaluation may differ - Assessment / evaluation scope may

vary: Different definitions of ‘Organization’ Org may be based on senior

management, geo location, common app, profit/loss center, etc.

Sample of selected projects



• Differences …- Motivation, objective, outcome & results

ownership differ These factors lead to differences in

dynamics of interviews, scope of inquiry, info collected, & results

Assessment & evaluation methods are different

Assessment training doesn’t prepare team to do evaluation, vice versa

• Differences …- Motivation, objective, outcome & results

ownership differ These factors lead to differences in

dynamics of interviews, scope of inquiry, info collected, & results

Assessment & evaluation methods are different

Assessment training doesn’t prepare team to do evaluation, vice versa



• Differences …- Process Assessment – performed in

open, collaborative environment Commitment from mgmt & staff to do

process improvement Objective: surface problems & help

improve organization Emphasis on interviews as tool for

understanding organization’s software process

• Differences …- Process Assessment – performed in

open, collaborative environment Commitment from mgmt & staff to do

process improvement Objective: surface problems & help

improve organization Emphasis on interviews as tool for

understanding organization’s software process



• Differences …- Capability Evaluation – performed in

audit-oriented environment Objective tied to monetary

considerations Emphasis on documented audit

trail that reveals software process actually implemented by organization

• Differences …- Capability Evaluation – performed in

audit-oriented environment Objective tied to monetary

considerations Emphasis on documented audit

trail that reveals software process actually implemented by organization



• Process Assessment- SEI SW-CMM Improvement Approach- Do organizational assessments pay

off?- Whistleblowers- Reference: SPICE Assessment

Documents

• Process Assessment- SEI SW-CMM Improvement Approach- Do organizational assessments pay

off?- Whistleblowers- Reference: SPICE Assessment

Documents


Software Process AssessmentsSoftware Process AssessmentsSoftware Process AssessmentsSoftware Process Assessments

• Why do process assessment?- “Why Do Organizations Have

Assessments? Do They Pay Off?”, CMU SEI Technical Report, July, 1999

- Panel discussion from SEPG ’99 in Atlanta, GA on March 10, 1999

- Each panelist is one of most active lead assessor for SEI for CMM-Based Appraisals for Internal Process Improvement (CBA IPI)

• Why do process assessment?- “Why Do Organizations Have

Assessments? Do They Pay Off?”, CMU SEI Technical Report, July, 1999

- Panel discussion from SEPG ’99 in Atlanta, GA on March 10, 1999

- Each panelist is one of most active lead assessor for SEI for CMM-Based Appraisals for Internal Process Improvement (CBA IPI)


Software Process Assessments …Software Process Assessments …Software Process Assessments …Software Process Assessments …

• Electronic Data Systems- Why do assessments?

Driven by customer: higher maturity to keep customers; extra bidding points for higher maturity; DoD requirements

- Do assessments pay off? Yes! Need exec sponsorship, funding Everyone learns, helps org culture

• Electronic Data Systems- Why do assessments?

Driven by customer: higher maturity to keep customers; extra bidding points for higher maturity; DoD requirements

- Do assessments pay off? Yes! Need exec sponsorship, funding Everyone learns, helps org culture



• Reuters- Why do assessments?

Framework for continuous process improvement; maturity levels build on previous level

Raises awareness in group- Do assessments pay off?

Clear view of cost, effort, resource Measure key performance items Method for introducing global change

• Reuters- Why do assessments?

Framework for continuous process improvement; maturity levels build on previous level

Raises awareness in group- Do assessments pay off?

Clear view of cost, effort, resource Measure key performance items Method for introducing global change



• U. S. Navy- Why do assessments?

Early SEI affiliate Beta test group for Maturity

Questionnaire Continued with SEI assessments

- Do assessments pay off? Invaluable to organization In 1997 determined 4.1 to 1 ROI Education for new team members

• U. S. Navy- Why do assessments?

Early SEI affiliate Beta test group for Maturity

Questionnaire Continued with SEI assessments

- Do assessments pay off? Invaluable to organization In 1997 determined 4.1 to 1 ROI Education for new team members



• Panel Q&A- Why pay cost of doing CBA IPI?- How often do you do full CBA IPI?- How small of organization to do CBA IPI?- How to reduce impact/stress on org when

doing CBA IPI?

• Panel Q&A- Why pay cost of doing CBA IPI?- How often do you do full CBA IPI?- How small of organization to do CBA IPI?- How to reduce impact/stress on org when

doing CBA IPI?




Documents


Documents



• Blowing the Whistle on Troubled Software Projects- Article by Mark Keil & Daniel Robey- Controversy behind going forward to

management when auditors find problems with projects

- Case study of IS Auditors 1999 Study 75 Auditors

• Blowing the Whistle on Troubled Software Projects- Article by Mark Keil & Daniel Robey- Controversy behind going forward to

management when auditors find problems with projects

- Case study of IS Auditors 1999 Study 75 Auditors



• Key Points- Auditor obligation vs. risk- Reluctance to blow the whistle- Reluctance to hear the whistle- Conditions that encourage or influence

whistle blowing

• Key Points- Auditor obligation vs. risk- Reluctance to blow the whistle- Reluctance to hear the whistle- Conditions that encourage or influence

whistle blowing




Documents


Documents


SPICE Assessment: ReferenceSPICE Assessment: Reference

• SPICE Software Process Assessment Information- SPICE documentation available free of

charge from Software Quality Institute web site: http://www.sqi.gu.edu.au/spice/suite/download.html

• SPICE Software Process Assessment Information- SPICE documentation available free of

charge from Software Quality Institute web site: http://www.sqi.gu.edu.au/spice/suite/download.html


SPICE Assessment: Reference …SPICE Assessment: Reference …

• SPICE Part 3 – Rating Process- Section 4.2: Defining Assessment Input - Section 4.4: Assessing and Rating

Process- Section 4.5: Recording the Assessment

Output

• SPICE Part 3 – Rating Process- Section 4.2: Defining Assessment Input - Section 4.4: Assessing and Rating

Process- Section 4.5: Recording the Assessment

Output



• SPICE Part 4 – Guide to Conducting Assessment- Section 4.2: Process Rating Scheme- Section 4.3.1.1: Team-based Assessment- Section 4.4: Assessment Stages (8)- Section 4.5: Success Factors for Process

Assessment- Section 5.1: Reviewing Assessment

Inputs

• SPICE Part 4 – Guide to Conducting Assessment- Section 4.2: Process Rating Scheme- Section 4.3.1.1: Team-based Assessment- Section 4.4: Assessment Stages (8)- Section 4.5: Success Factors for Process

Assessment- Section 5.1: Reviewing Assessment

Inputs



• SPICE Part 4 …- Section 5.2: Selecting Process Instances- Section 5.3: Preparing for Team-based

Assessment- Section 5.4: Collecting and Verifying

Information- Section 5.5: Determine Actual Ratings for

Process Instances- Section 5.6: Determining Derived Ratings- Section 5.7: Validating the Ratings- Section 5.8: Presenting Assessment Output

• SPICE Part 4 …- Section 5.2: Selecting Process Instances- Section 5.3: Preparing for Team-based

Assessment- Section 5.4: Collecting and Verifying

Information- Section 5.5: Determine Actual Ratings for

Process Instances- Section 5.6: Determining Derived Ratings- Section 5.7: Validating the Ratings- Section 5.8: Presenting Assessment Output



• SPICE Part 5 – Construction, Selection, & Use of Assessment Tools- Section 4.1: Form and Purpose of an

Assessment Instrument- Section 4.2 (and Annexes A, B, C, D):

Implementation of Standard Indicators- Section 4.3: Tailoring of Indicators

Contained in Assessment Instrument- Section 4.4: Modular Assessment

Instruments

• SPICE Part 5 – Construction, Selection, & Use of Assessment Tools- Section 4.1: Form and Purpose of an

Assessment Instrument- Section 4.2 (and Annexes A, B, C, D):

Implementation of Standard Indicators- Section 4.3: Tailoring of Indicators

Contained in Assessment Instrument- Section 4.4: Modular Assessment

Instruments



• SPICE Part 5 …- Section 4.5: Capturing and Processing

Assessment Data- Section 4.6: Using an Assessment

Instrument

• SPICE Part 6 – Qualifications & Training of Assessors- Section 4.1: Role of the Assessor- Section 5: Assessor Competence

• SPICE Part 5 …- Section 4.5: Capturing and Processing

Assessment Data- Section 4.6: Using an Assessment

Instrument

• SPICE Part 6 – Qualifications & Training of Assessors- Section 4.1: Role of the Assessor- Section 5: Assessor Competence


ReferencesReferences

• M. Paulk, B. Curtis, M. B. Chrissis, C. V. Weber, "Capability Maturity Model for Software", Version 1.1, Software Engineering Institute, Carnegie Mellon University, February, 1993. Available at http://www.sei.cmu.edu/publications/documents/93.reports/93.tr.024.html

• W. S. Humphrey, Managing the Software Process, Addison-Wesley, Reading, MA, 1989

• M. Paulk, C. V. Weber, B. Curtis, M. B. Chrissis, The Capability Maturity Model: Guidelines for Improving the Software Process, Addison-Wesley, Boston, MA, 1995

• “Process Maturity Profile of the Software Community 2002 Mid-Year Update”, Software Engineering Institute, Carnegie Mellon University, August, 2002. Available at http://www.sei.cmu.edu/sema/pdf/SW-CMM/2002aug.pdf

• M. Paulk, B. Curtis, M. B. Chrissis, C. V. Weber, "Capability Maturity Model for Software", Version 1.1, Software Engineering Institute, Carnegie Mellon University, February, 1993. Available at http://www.sei.cmu.edu/publications/documents/93.reports/93.tr.024.html

• W. S. Humphrey, Managing the Software Process, Addison-Wesley, Reading, MA, 1989

• M. Paulk, C. V. Weber, B. Curtis, M. B. Chrissis, The Capability Maturity Model: Guidelines for Improving the Software Process, Addison-Wesley, Boston, MA, 1995

• “Process Maturity Profile of the Software Community 2002 Mid-Year Update”, Software Engineering Institute, Carnegie Mellon University, August, 2002. Available at http://www.sei.cmu.edu/sema/pdf/SW-CMM/2002aug.pdf


References …References …

• D. Dunaway, R. Berggren, G. des Rochettes, P. Iredale, I. Lavi, G. Taylor, “Why Do Organizations Have Assessments? Do They Pay Off?”, Software Engineering Institute, Carnegie Mellon University, July, 1999. Available at http://www.sei.cmu.edu/publications/documents/99.reports/99tr012/99tr012abstract.html

• M. Keil, D. Robey, “Blowing the Whistle on Troubled Software Projects”, Communications of the ACM, Vol. 44, No. 4, April 2001, pp. 87-93. PDF file (ACMp87-Keil.pdf) available online via ACM Digital Library when

accessed from the RPI at Hartford Library • ISO/IEC TR 15504:1998 - Software Process Assessment (SPICE)

documentation suite. Available at http://www.sqi.gu.edu.au/spice/suite/download.html

• D. Dunaway, R. Berggren, G. des Rochettes, P. Iredale, I. Lavi, G. Taylor, “Why Do Organizations Have Assessments? Do They Pay Off?”, Software Engineering Institute, Carnegie Mellon University, July, 1999. Available at http://www.sei.cmu.edu/publications/documents/99.reports/99tr012/99tr012abstract.html

• M. Keil, D. Robey, “Blowing the Whistle on Troubled Software Projects”, Communications of the ACM, Vol. 44, No. 4, April 2001, pp. 87-93. PDF file (ACMp87-Keil.pdf) available online via ACM Digital Library when

accessed from the RPI at Hartford Library • ISO/IEC TR 15504:1998 - Software Process Assessment (SPICE)

documentation suite. Available at http://www.sqi.gu.edu.au/spice/suite/download.html