Lessons Learned Lessons Learned on the way to Level 3on the way to Level 3

If we knew then what we know now…

Dave Groening ARA SED QA & Process Improvement LeadBeth Layman Principal, Layman & LaymanAllen York ARA SED Division Manager

The process improvement journey of ARA’s Southeast Division (SED)

NDIA 8th Annual CMMI® Technology Conference and User GroupNovember 17 – 20, 2008

2

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

22

About ARA and Southeast DivisionAbout ARA and Southeast DivisionAn engineering and science services company with distributed officesDiverse DoD and Federal contractor1,200+ employee ownersFY 08 Sales of ~$200 million

Southeast DivisionRaleigh, NCSuffolk, VASanta Barbara, CAAberdeen , MDOrlando, FL (FY06)

3

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

33

ARA Southeast DivisionARA Southeast DivisionHQ in Raleigh, NCSales of approx $23M in FY08Mix of MS and PhD-level engineers/scientistsand senior-level software system developersAreas of Expertise

Large-scale software system developmentHigh-fidelity and fast-running physics-based weapon-target interaction modelingSensor exploitation & target recognition technologiesSynthetic environment generation forsemi-automated forces and visual simulatorsConventional and nuclear (low andhigh altitude) weapon effectsRapid site/facility modeling toolsC4ISR and decision support technologiesCounter-IED technologies

MilitaryPlanning

Tools

SimulationTechnologies

C4ISRTechnologies

4

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

44

PEO STRI

Our Customers IncludeOur Customers IncludeMDA TSWG

RDECOM

AFRL

DIA

PMS 408JIEDDO

JTCG

STRATCOM

5

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

55

Our Process Improvement HistoryOur Process Improvement HistoryFY06:•Additional CMMI training •Creation of Process Asset Library (PAL)•SCAMPI Class C Document Review

FY07:•EPG reviewed & approved processes•Produced & began delivering process training•Project rollout started

Oct-06 Dec-07

Nov 06 Dec 06 Jan 07 Feb 07 Mar 07 Apr 07 May 07 Jun 07 Jul 07 Aug 07 Sep 07 Oct 07 Nov 07

11/27/06EPG Kickoff

12/15/2006EPG CMMI Training

1/3/2007OPT Convened

1/5/2007EPT Convened

1/9/2007PMPT

Convened

1/5/2007SPT Convened

6/8/2007EPG

2nd-half Kickoff

2/9/2007OPD Approved

2/26/2007PPQA Approved

3/1/2007VER Approved

4/2/2007MA

Approved

4/29/2007RSKM, VAL,

DAR, CM & OT Approved

4/8/2007PI & SAMApproved

5/17/2007PP/PMC/IPM

Approved5/30/2007

RD/REQM Approved

8/5/2007OPF Approved

10/2/2007TS Approved

6/8/2007PPQA Training

10/23/2007OT Training

10/21/2007PMP Training

11/15/2007RD/REQM Training

10/8/2007OPD Process Audit

10/12/2007Project Rollout / PMR Kickoff

6

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

66

Process Improvement FY08 Process Improvement FY08 October – December

Process training (PAL 1.0 )January - February

Contracted with Layman & LaymanSCAMPI Class CRevised FY08 Process Improvement planNew PI organizational structure

March – May PAL 1.1 process revision

5 working groups (PATs)More unified, detailed and prescriptive set of processes -Easier to use

Overview sessions for all staff

June – AugustNew EPGPI Plan updateProject transition workshopsRole-based trainingPAL 1.1 process audits

SeptemberSCAMPI B

7

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

77

Establish an effectiveEstablish an effectiveProcess Improvement organizationProcess Improvement organization

Dedicate at least some of the resourcesDefine Management’s role and keep them engagedGaining buy-in across the organization is criticalRun process improvement like a projectConsider using a consultant

8

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

88

On PI = Organizational ChangeOn PI = Organizational ChangeProcess definition is easy, process deployment and institutionalization is hard!

Lesson Learned: Don’t spend all your energy on task work

The ARA team initially underestimated their role and the powerful dynamics of change

Org. change leadership is now practiced w/great results!

9

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

99

MSG Role in Process ImprovementMSG Role in Process ImprovementWhat are your main responsibilities?

(Hint: There are 4 of them)1. DRIVE PI PLANS

create vision and desire for changeapprove all PI strategies, plans, roadmaps…including Measurement & Training Plan development and execution

2. COMMUNICATE regularly about PI program to your teamstie business objectives of the organization to PI programreview plans, scope, upcoming events, expectations

3. Provide CONSEQUENCES for “not doing” itENFORCE compliancetie to CDP/annual reviews

4. MONITOR & CONTROL PI Programtreat it like any other project remove roadblocks

DesiredState

TransitionStates

ChangeStrategy

ManagingChange

PresentState

Prod

uctiv

ity

DesiredState

TransitionStates

ChangeStrategy

ManagingChange

PresentState

Prod

uctiv

ity

10

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

1010

SEDSED’’s Communication Plans Communication Plan

11

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

1111

PI Program FundingPI Program FundingOur ongoing software projects could not be burdened with our PI/CMMI start-up expenses –expenses not planned for the projects

We did not pursue asking the government to fund our PI program through existing software projects

Funding is a critical constraint becauseSED is essentially a small businessReceived little corporate supportCannot use “profit” - must fund within our cost structure

Required an Internal Research and Development (IR&D) effort

IR&D funds are limited and are shared with costs to prepare proposals trading PI for business up front

12

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

1212

Funding StrategyFunding StrategyTook out a multi-year IR&D

Had to be approved by CEOIR&D covered expenses for

Creating PAL, processes, templates, forms, plansPIL, SCAMPIs and appraisal team, consultant

Projects covered expenses forAdopting processes – process trainingCreating artifactsMolding contractual documents to fit PI/CMMI standardsProcess audits

13

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

1313

How much did it cost?How much did it cost?Over $1M !

Over 3 yearsWe have tracked about $950k through IR&D projectEstimate some IR&D expenses went on overhead and some went on “personal”timeBy far, the longest and most expensive IR&D in corporate history

$0

$100,000

$200,000

$300,000

$400,000

$500,000

$600,000

$700,000

$800,000

$900,000

$1,000,000

Budget

Actual

14

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

1414

On Good Process ArchitectureOn Good Process ArchitectureLessons Learned:

– The model’s just a model– Terminology does matter– Avoid process silos – what’s needed is a

cohesive set of processes– Prescriptivism is better than vagaries– Allow legacy variances (but not for preferences)– Connect processes and training through roles– Integrate process and measurement– Ease the implementation burden with simple tools

15

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

1515

Specific Process AreasSpecific Process AreasProject Management

tying everything together with a lifecycle modelMeasurement & Analysis

the benefits of using a measurement information modelOrganizational Training

linking training with strategic planning and performance management

Process & Product Quality Assurance monitor process improvement progress

16

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

1616

Project ManagementProject Managementtying everything together with a lifecycle model

Scoping &Planning

Scoping &Planning DesignDesignAnalysisAnalysis Implementation

& IntegrationImplementation

& IntegrationAcceptance& Release

Acceptance& ReleaseTestingTesting

ProjectAward

ProjectAward

SoftwareRequirements

Review

SoftwareRequirements

Review

Prelim.DesignReview

Prelim.DesignReview

CriticalDesignReview

CriticalDesignReview

TestReadiness

Review

TestReadiness

ReviewAcceptance

TestAcceptance

Test ReleaseRelease

17

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

1717

Project ManagementProject Managementtying everything together with a lifecycle model

Scoping &Planning Analysis

ProjectAward

SoftwareRequirements

Review

Loop 1

Loop 2Loop 3

18

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

1818

Project ManagementProject Management

Scoping &Planning

Scoping &Planning DesignDesignAnalysisAnalysis Implementation

& IntegrationImplementation& Integration

Acceptance& Release

Acceptance& ReleaseTestingTesting

ProjectAward

ProjectAward

SoftwareRequirements

Review

SoftwareRequirements

Review

Prelim.DesignReview

Prelim.DesignReview

CriticalDesignReview

CriticalDesignReview

TestReadiness

Review

TestReadiness

ReviewAcceptance

TestAcceptance

Test ReleaseRelease

Configuration ManagementConfiguration ManagementDecision Analysis & ResolutionDecision Analysis & Resolution

Measurement & Analysis Measurement & Analysis Process Audit Process Audit

Product IntegrationProduct Integration

Supplier Agreement ManagementSupplier Agreement Management

Technical SolutionsTechnical Solutions

VerificationVerificationValidationValidation

Process ManagementProcess ManagementOrganizational TrainingOrganizational Training

Requirements Development & ManagementRequirements Development & Management

Project ManagementProject Managementtying everything together with a lifecycle model

19

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

1919

SEDSEDMeasurement Measurement Information Information ModelModel

Entity

AttributeAttributeAttributeAttribute

Indicator & Decision Criteria

InformationNeed

Analysis Model

Base Measure

Base Measure

Application Guidance & Decision Criteria

Measurement Method

Measurement Method

Base Measure

Derived Measure

Base Measure

Measurement Method

Measurement Method

Measurement Function

Projections are based on earned value techniques

-

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

Oct

…N

ov…

Dec

…

Jan…

Feb…

Mar

…

Apr

…

Ma…

Jun…

Jul-0

6

Aug

…

Sep

…

Oct

…

Nov

…D

ec…

Jan…

Feb…

Mar

…

Apr

…

Ma…

Jun…

Jul-0

7

Aug

…

Sep

…

Oct

…

Nov

…

Dec

…Ja

n…

Feb…

Mar

…

Apr

…

Ma…

Jun…

Jul-0

8A

ug…

Sep

…

Oct

…

Nov

…

6,750

7,000

7,250

7,500

Jul-08 Aug-08 Sep-08`

Cost Variance

CurrentFunding

TABBCWS

BCWS

ACWP

BCWPScheduleVariance

BCWP

-$176k

-$90k

CPI=0.987SPI=0.976

BAC

EAC

20

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

2020

0

10

20

30

40

50

60

70

80

90

100

PM SAM MA CM RD TS PI VER VAL DAR

Audit Scores by Process

Project MeasuresProject Measures

‐40

‐20

0

20

40

60

80

100

120

S&P Analysis Design I&I Testing Release

Customer Requirements By Phase

Added

Dropped

Modified

Unmodified

21

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

2121

Bottom-UpEmployee Assessment

+ CDP input

Top-DownGroup

Assessment

GroupTraining

Requirements& Cost

Estimates

GroupTraining

Requirements& Cost

Estimates

$DivisionAnnualTacticalTraining

Plan&

Budget

DivisionAnnualTacticalTraining

Plan&

Budget

Training linked to strategic planning Training linked to strategic planning & performance management& performance management

OT Scope &PriorityCut Line

Support GroupTraining

Requirements

TrainingDatabase

22

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

2222

Artifacts & Work Products AuditedScoping & Planning Analysis Design

Implementation & Integration Testing

Acceptance & Release

Contract or other document showing fundingWork Breakdown Structure PMCustomer Requirements Spreadsheet SAMEvidence of team mtgs and IPRs MAProject Management Plan (w/ Stakeholder Matrix) CMIntegrated Master Schedule RDConfiguration Management Plan TSProduct Acquisition List PIMeasurement and Analysis Plan VERRisk Management (sources, parameters, strategy) VALProduct Integration (test) Plan RSKMRisk Management Plan (complete) DAREvidence of Measurement & Analysis as Planned PPQAPreliminary Design Review (PDR) Meeting Minutes UnspecifiedPreliminary Design DocumentEvidence of DAR process during analysis decisionsCritical Design Review (CDR) Meeting MinutesAction Items from PMRs, PQRsDesign Doc (complete)Interface Control DocumentVerification Plan w/ STPValidation Plan w/ STPSubcontractor Delivery LogCode review recordsDraft Product Support DocumentationPeer Review ChecklistCode test procedure (STR)VER ReportEvidence of Validation Activities (STR, PRs, CRs)VAL ReportEvidence of Risk Mitigation ActionsRecord of successful unit test casesIntegration DocumentationDCRs for process improvementEvidence of corrective actionAAR Minutes and Action Items

PROCESS LEGEND

PROJECT LIFE-CYCLE AND AUDIT PHASES

Process Audits throughout Project LifeProcess Audits throughout Project Life--CycleCycle

23

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

2323

Process Audits throughout Project LifeProcess Audits throughout Project Life--CycleCycleProcess

AreaChecklist Meets

Req'mtsPM Project Management Plan:PM PMP is under revision control YesPM PMP is peer reviewed and updated YesPM Estimation Spreadsheet created (per task) YesPM Project Schedule:PM Work Breakdown Structure updated and reviewed YesPM Schedule is reviewed monthly and updated YesPM Schedule is under revision control YesPM Project Budget:PM Work Breakdown Structure updated and reviewed YesPM Budget is reviewed monthly and updated YesPM Budget is under revision control YesPM Project Risk Spreadsheet:PM List of Project Risks updated YesPM Priority Rankings updated YesPM List of Project Risks under revision control YesPM Risk Information Sheet updated YesPM Risk Mitigation Plan updated NoPM Risk status reviewed and updated YesPM Tasking Spreadsheet under revision control YesPM Project Reviews (IPRs) conducted:SAM Project Acquisition List included in PMP (if applicable) YesSAM Acquisition List reviewed at least monthly YesSAM If this is a COTS product, a PO or supplier agreement/contract must be included YesSAM If this is a consulting service, project should have an approved Agreement for Consulting Services N/ASAM If this is a subcontracting product:RD Accepted Customer Requirements and Software Requirements added to RequisitePro YesRD Software Requirements Specification generated YesRD SRS under revision control YesRD SRS peer reviewed YesRD Interface Requirements added to RequisitePro YesRD Customer commitment at milestone (meeting minutes) YesRD Customer commitments are under version control Yes

24

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

2424

Process Audits throughout Project LifeProcess Audits throughout Project Life--CycleCycle

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

PM SAM MA CM RD TS PI VER VAL DAR

Process Audit ScoreScoping & Planning  Analysis Design Implmnt & Intgrt Testing Release

CreatedKilled

Assigned(for corrective

action)

Resolved

Verified

Create

Kill

Re-Create

Assign

Re-assign

Resolve

Verify

Reject

Process Anomaly (PA)state transition diagram

Terminal State

Transient State

Decline

Modify

Modify

Modify

Modify

25

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

2525

Summary of Lessons Learned Summary of Lessons Learned Establish organizational infrastructure with adequate resourcesDon’t overlook organizational change aspect

(writing processes is the easy part)Overall process design/architecture is importantManaging process improvement project is critical

If we had known these things at the outset, it would have taken less time and less money to achieve Maturity Level 3

26

Cop

yrig

ht 2

007.

All

right

s re

serv

ed. A

pplie

d R

esea

rch

Asso

ciat

es, I

nc.

2626

Thank You Thank You –– Questions? Questions? Dave Groening

Applied Research Associates, Inc.Southeast Division QA & Process

Improvement Leaddgroening@ara.com

Allen R. York II, PhD, P.E.Applied Research Associates, Inc.Southeast Division Managerayork@ara.com

Beth LaymanPrincipal , Layman & LaymanBeth.Layman@LaymanandLayman.com