AIAA Space 2009, Harrison and Scheinin

AIAA Space 2009 Conference

14–17 September 2009

Cecilia HarrisonWarren Scheinin

Northrop Grumman Corporation

Integration and Test in a CMMI Level 5 Environment

AIAA Space 2009, Harrison and Scheinin

2

The Steps to Success: An Agenda in Reverse Order

Success

The Result: Managed Integration

The Details: How it is Done

The Solution: Engineering for Integration

The Problem: Some Assembly Required

The Environment: CMMI Level 5

The Set Up

AIAA Space 2009, Harrison and Scheinin

3

The Set Up

• Some products were not being delivered in as timely a manner or within budget as planned

• Defects injected early in the product life cycle were not being detected until late in the life cycle

• Components did not always plug together as easily as hoped

• Unanticipated and unwanted outputs needed to be eliminated

• Project personnel often experienced stress as milestones drew near

• Smooth running processes often hit bumps when external stakeholders provided inputs

We knew we could be more efficient

AIAA Space 2009, Harrison and Scheinin

4

The Environment: CMMI Level 5

• Institutionalized processes, tools and stakeholder involvement are focused on success factors

• Selected processes are optimized using results based, quantitative methods

• Early life cycle processes are not allowed to crowd out later life cycle processes at project start-up and early execution

• Staged representation lifts all process areas in parallel

• Continually improving performance through incremental and innovative changes

AIAA Space 2009, Harrison and Scheinin

5

Climbing the CMMI Level 5 Ladder

• Each CMMI Level is a step to Higher Project Maturity– Starts with the foundations for a maintainable system– Gets your head above water– Clears the fog of fighting fires– Engages the supercharger Level 4

QuantitativelyManaged

Level 1Initial

Level 2Managed

Level 5Optimizing

Level 3Defined

The Ad Hoc Sink Hole

AIAA Space 2009, Harrison and Scheinin

6

Level 4: Understand Your Processes and Subprocesses

• Targets defined processes for quantitative control

• Take advantage of organizational knowledge– Identify the implementations of processes which perform best – Know that processes are performing within natural bounds that

are consistent across teams – Surgical precision of Six Sigma methods– Co-ordinate with other projects Defect Detection Profile

0.00

20.00

40.00

60.00

80.00

100.00

120.00

140.00

160.00

180.00

Req'mts Design Code Unit Test Integrate Sys Test Del 90 Days

Phase

All Projects

New Process

• Better understanding and use of Level 3 metrics– Goal setting becomes more important

for Levels 4 and 5– Need better definitions of the

measures and measurement processes– Lower level metrics of subprocesses

Pro

cess

Cap

abil

ity

Met

ric

Life Cycle Phase

AIAA Space 2009, Harrison and Scheinin

7

Level 5: Adding the Edge

• Get ahead of the curve by catching problems before they attack the project

• Reduce the variation– Initiate late cycle processes early– Create procedures/checklists– Strengthen process audits

• Increase the effectiveness (increase the mean)– Train people– Create checklists– Reduce waste and re-work– Replicate best practices

Revolutionary Process Capability Improvement

AIAA Space 2009, Harrison and Scheinin

8

The Problem: Some Assembly Required

• Things do not always work as intended

• Needed two-pronged attack on the classical weakness of integration: surprise and delay

• Many projects disempowered integration and test until it is too late

• Major flaws are generally “baked into the cake” before integration efforts even start

• Integration and Test teams are often brought onto the project far too late to influence and/or contribute to the system design

• I&T Team can have negative affect on the dynamics of the project team if not on board from “Day One”

• Risks need to be identified before they become problems

• Let software do it is not a good answer

AIAA Space 2009, Harrison and Scheinin

9

Worse Case Scenario

Requirements

Process Development

Design

Program Program FailureFailure

Product Development

Test and Evaluation

Operations

DefectDefectss

Acquisition

Missed defect in early product realization processes provide opportunities for potential failure modes to be created and propagated, causing Mission failure

• Early Defects Cause Late Term Problems

• Need eyes of Integration and Test Managers and leads early in product life cycle

AIAA Space 2009, Harrison and Scheinin

10

Experience at NGIS

• Lessons Learned show that as the product development life proceeds, the more positive the impact of early integration and test management

High leverage points

0

1

2

3

4

5

6

7

8

9

10

Pre-Proposal ProgramInitiation

ProgramExecution

Business Acquisition Phase

Imp

act

on

Su

ccess

IntegrationManagement

Risk Management

Tech PerformManagement

ScheduleManagement

SubcontractManagement

Scope Management

Cost Management

CommunicationsManagement

Quality Management

Issue Management

AIAA Space 2009, Harrison and Scheinin

11

The Solution: Engineering for Integration

• Focuses on managing the integration and test efforts

• Mechanism to assess and improve system reliability starting from “Day One”

• Empowers the Integration and Test team by involving them at the earliest part of the project lifecycle

• Supports performance analysis to optimize throughput, capacity, and product quality

• Proactively make demands of the system design to suit Managed Integration and Performance Analysis needs

• Implemented in parallel to the System Development effort

Byproducts are satisfied customers and users

AIAA Space 2009, Harrison and Scheinin

12

The Details: Managed Integration

• Process of integrating hardware and software products in frequent, small, additive steps into a full system environment

• Typically characterized by: – Deliberately conceived architectures that enable parallel,

component-level integration– Test tools sufficient to analyze weaknesses in submitted products,

so as to enable their rapid correction/improvement– Emphasis on evaluation of both internal and external interfaces– Concentration on a meaningful set of basic functionality (Core

Function and Long Mission Threads)– Early capacity for full operational stress conditions– Potential for rapid turnaround of products and tasks in integration,

such as in “continuous integration” approaches

AIAA Space 2009, Harrison and Scheinin

13

10 General Elements of Managed Integration

• Early Infrastructure Selection

• System Architecture Skeleton Strategy

• Layered Service Provisioning

• Long Mission Threads Emphasis

• Strategic Component Selection

• Minimize Interdependencies

• Specify, Organize, and Control System States

• Automate Regression Testing

• Design for Performance Tuning

• Control Configurations and Versions

AIAA Space 2009, Harrison and Scheinin

14

The Details: Performance Analysis

• Targets details of implementation and obscure characteristics of complex systems that can have extraordinary impact on overall performance

• Solve issues that could become major factors impacting delivery timelines

• Ensures that flow down of general performance requirements into specific requirements are realistic and testable

• Uses system architecture skeleton to analytically extrapolates overall understand of components on the system collectively to mitigate overall performance risk

• Allows focusing of limited resources to components of the system which will experience the greatest performance stress

AIAA Space 2009, Harrison and Scheinin

15

4 General Elements of Program for Performance Analysis

“Program for Performance Analysis” is a well-planned effort to examine performance and apply a variety of strategies to gain appropriate, timely understanding of the system.

• Component Benchmarking

• Communication Channel Characterization

• Long Mission Thread Exercise

• Interface performance Description

AIAA Space 2009, Harrison and Scheinin

16

The Result: Managed Integration From Day One

• Integration and Test managers and leads working side by side with System Engineers and Architects to define scenarios that will drive a design that meets the required performance

• Early establishment of systems engineering and architecture elements which most directly pertain to and predict integration success

• Proactively exploiting the emerging system in integration to achieve higher quality, higher confidence results

AIAA Space 2009, Harrison and Scheinin

17

ProgramProgram

••

Systems Systems Safety Safety

ReliabilityReliability

• MaintainabilityMaintainability

Mission Mission AssuranceAssurance

QualityQualityQuality Quality AssuranceAssuranceSoftwareSoftwareAssuranceAssurance

SupplierSupplierQualificationQualification

SelectionSelection•AssessmentAssessment

•

• IndependentIndependentReviews andReviews andAssessmentsAssessments

••

HardwareHardwareAssuranceAssurance

•••

• AvailabilityAvailability

•••

Risk Risk ManagementManagement

Indicators of Success

• More up front thinking means less work later

• More value-added products produced with reduced effort and time– Instead of overrunning budgets and schedules, products are

delivered early and on budget – Extra deliveries are possible without strain

• Less “help” from senior management but more resources available

• Lots of new work pours in

• Communications with other groups is easier

AIAA Space 2009, Harrison and Scheinin

18

Better metrics

More stakeholder involvemen

t

More efficient

statusing

Add more subprocesse

s

A family of

integration

strategies

A Level 5 Organization is Never Satisfied….

AIAA Space 2009, Harrison and Scheinin

19