Embed Size (px)
Citation preview
1/30
Patrick K. BarkerMCR, LLC
(703) – 898–6354
Glen B. AllemanLewis & Fowler
(303) – 241 9633
Connect Quality with Performance
Knowing what DONE looks like begins with the Integrated Master Plan.
Recognizing what DONE looks likewhen it arrives means measuring the planned Technical Performance.
Measuring Physical Percent Completetells us how far we have moved toward DONE by calculating the “Earned Value” we’ve achieved.
Connecting Earned Value, Technical Performance, and Physical Percent Complete in the IMP establishes a credible measure of Progress to Plan.
Without these four elements we don’t have a clue to what DONE looks like or if we’ll ever get there as planned.
2/30
Evidence of DONE Measures of DONE Beneficial Outcomes
IMP
Accomplishment
Criteria
(AC)
The collection of TPM, EV, and
PPC measures defined as “exit
criteria” for the Work Packages
performing the work.
The IMP then defines
the measurable path to
the planned maturity.
Technical
Performance
Measure
(TPM)
The actual technical
performance measured at a
defined date compared to
planned technical performance.
Evidence that the
product or service is
meeting the planned
maturity or quality.
Physical Percent
Complete
(PPC)
Unit of measure to assess the
planned technical performance,
evidenced by tangible outcomes.
Evidence of completion
of the planned work in
units meaningful to the
customer.
Earned Value
(EV)
Cost and schedule needed to
deliver the planned technical
performance on baseline.
The performance
measured in units of
dollars.
3/30
No Matter How Great and Destructive Your
Problems May Seem Now, Remember, You’ve
Probably only Seen the Tip of Them
4/30
5/30
A Technical Performance Measures …
State how well the program is achieving the planned performance requirements at the planned delivery time.
Use actual or predicted values from:– Engineering measurements– Tests– Experiments– Prototypes
For Example:– Response time– Flight range– Power consumption– Static takeoff weight – Product Quality
6/30
Document, Baseline,
IMS, EVM Parameter
IMP, Functional Baseline Measures Of Effectiveness (MOE)
IMP, WBS, Functional Baseline Measures Of Performance (MOP)
IMP, Allocated Baseline Technical Performance Measure
IMSTPM Milestones And Planned
Values
Work Packages TPM% Complete Criteria
7/30
Physical Percent Complete Requires A Tangible
Measure of “Progress to Plan”
Tangible evidentiary
materials measure
progress to plan.
Pre–defined
existence of this
evidence in
meaningful units of
measure established
before starting work.
Progress is defined in
these same units of
measure.
No Hand Waving Allowed
“Show Me The Money”
8/30
Work activities performed to produce the deliverables that fulfill the requirements that enable the capabilities
Specified result, substantiating a Milestone or Event, that indicates maturity or progress for each product or process
Definitive measures substantiating the Accomplishment maturity level. Completion of specific work that ensures closure of a specified Accomplishment
Major program milestones or assessment events that substantiate system maturity (initial, progress, or final). These milestones or assessment events deliver the specific capabilities for the system on planned dates.
IMS
IMP
Describes how the
capabilities will be
delivered and
how these
capabilities will
be recognized
Supplemental Schedules
Work Packages and Tasks
Criteria
Accomplishment
Events
Milestones
9/30
Program
Events
Statement of
WorkCWBS
Significant
Accomplishments
Accomplishment
Criteria
CDRLs and
Deliverables
Tasks Contained
in Work Packages
Measure the progress to plan using Physical % Complete at the
Accomplishment Criteria (AC) and CWBS level
DefinesAligned
Aligned
AlignedAligned
AlignedCompleted SA’s are
entry criteria for
Program Events
Completed Work
Packages are exit
criteria for Tasks
Describes increasing
product maturity as 0/100 or
EVMS SD guidance
Documents the product
maturity that is aligned with
SOW and CWBS
Work necessary to
mature products
grouped by CWBS
Work structure
aligned to
SOW
10/30
PlannedProfile
AchievedTo Date
Tolerance Band
Current
Estimate
Threshold
Milestones
PlannedValue
Variation
10
5
15
Tech
nic
al P
ara
mete
r V
alu
e
(e.g
., V
eh
icle
Weig
ht,
lb
)
0.0 TOTAL BRILLIANT EYES PROGRAM
1.0 SPACE BRILLIANT EYES SYSTEM
1.1
1.2
1.2.1
1.2.2
1.2.2.1
1.2.2.2
1.2.2.2.1
1.2.2.2.2
1.2.2.2.3
1.2.2.2.4
1.2.3
1.2.4
1.2.5
1.2.6
1.2.7
1.3
1.4
1.5
System-Level Costs
Space Vehicle (SV) Segment
SV Program Level
Space Vehicle Prime Mission Equipment
Space Software
Space Vehicle
Space Vehicle IA&T
Sensor Payload
Insertion Vehicle
Survivability
Prototype Lot
Spare Parts
Technology and Producibility
Aerospace Ground Equipment
Launch Support
Engineering Change Orders (ECOs)
Other Government Costs
Risk
A WBS can be displayed as a “linear” List. Program Cost is often Calculated by summing the Costs of All Items on That List … but that ignores correlation and basic math: distributions don’t add that way.
Technical performance measurement have often been treated as an engineering function only, not typically linked with EVM, although it shares a common, almost identical, language structure with EVM.
The Schedule Network is Nonlinear, and Program Duration Cannot be Calculated by Adding Together the Durations of All Activities in the Network. And since time is money it is important for schedule slips to reflect cost
11/30
2 1 6 4 3 5
3 1 6 9 2 4
6 9 2 1 4 2 1
12/30
Neither cost performance nor schedule performance nor technical
performance measurement can be fully evaluated by itself.
How to Define DONE
Determine Physical Percent Complete based in performance measurement– Compare actual performance against an integrated baseline planned performance of the
integrated cost, schedule and technical goals.
This measurement is possible within a performance measurement system– Use the organization’s internal management control system to provide decision makers with
specific performance information about the progress and expenditures against the planed progress and expenditures.
In many projects where EVM systems are implemented, what a CAM determines (“eyeballing”) as physical percent complete often differs from what is reported via the EV techniques. – This disconnect in measurement of “done–ness” is a clear sign that a cost, schedule and/or
technical performance problem is going to materialize before the PM can properly react.
13/30
Risk
SOW
Cost
WBS
IMS
TPM
PMB
14/30
15/30
Cost
Performance
Baseline
Schedule
Performance
Baseline
Technical
Performance
Baseline
Perform
Functional
Analysis
Determine
Scope and
Approach
Develop
Technical
Logic
Develop
Technical
Baseline
Develop
WBS
Define
Activities
Estimate
Time
Durations
Sequence
Activities
Finalize
Schedule
Identify Apportioned Milestones
Determine
Resource
Requirement
Prepare
Cost
Estimate
Resource
Load
Schedule
Finalize
Apportioned
Milestones
Determine
Funding
Constraints
Approve
PMB
Making Trades
Between
Cost, Schedule,
and
Performance
Baselines is a Ponzi
Scheme
When we’re on
baseline, the algebraic
relationship between
C,S,P, means when
there is a change
everyone loses
16/30
A Really Simple Example of Physical Percent Complete
17
Painting a room is simple example to focused on a notion of percent complete.
What are the measures of physical percent complete?
‒ Design basis complete and verified
‒ Hardware platform increasing capabilities
‒ Software system increasing capabilities
‒ Integration and test of 1st flight article
‒ Operational verification and validation complete
Where can we look for these measures of physical percent complete?It’s Obvious – in the Integrated Master Plan’s Accomplishment Criteria
But how about a flight avionics system?
18/30
The notion of measuring physical percent
complete is at the core of every project
management method.
It answers the question of what DONE is
in units of measure meaningful to the
program.
This helps us determine answers to
questions that include, but are not limited
to:– What does DONE look like for today, for this
week, or for this month?
– What does DONE look like for entry/exit into
the technical review?
– What does DONE look like for quality control?
– What does DONE look like for the customer?
No stretching the truth
is allowed once we
start using Technical
Performance
Measures with
tangible evidence.
19/30
Criteria Represent Work Packages thatfulfill Requirements.
20/30
Program EventsDefine the availabilityof a Capability at a point in time.
AccomplishmentsRepresent requirements that enable Capabilities.
Work Package
WorkPackage
Work Package
Work Package
Work Package
Work Package
Work Package
Work package
The increasing maturing of a product or service is described through Events or Milestones, Accomplishments, Criteria, and their Work Packages or Planning Packages.
Each Event or Milestone represents the availability of one or more capabilities.
The presence of these capabilities is measured by the Accomplishments and their Criteria.
Accomplishments are the pre–conditions for the maturity assessment of the product or service at each Event or Milestone.
This hierarchy decomposes the System Capabilities into Requirements, Work Packages, and the activities which produce deliverables. This hierarchy also describes increasing program maturity resulting from the activities contained in the Work Packages.
Performance of the work activities, Work Packages, Criteria, Accomplishments, and Events or Milestones is measured in units of “physical percent complete” by connecting Earned Value with Technical Performance Measures.
The Structure of a Credible IMP
Work Package
Each WP must
have a measure
of Physical
Percent
Complete
21/30
IEEE 1220: 6.8.1.5, Performance–based
progress measurement
EIA–632: Glossary CMMI for Development
Requirements Development
TPMs are key to
progressively assess
technical progress
Predict future value of key
technical parameters of the
end system based on
current assessments
Specific Practice (SP) 3.3,
Analyze Requirements
Typical work product:
Technical Performance
Measures
Establish dates for
– Checking progress
– Meeting full conformance
to requirements
Planned value profile is time–
phased achievement projected
– Achievement to date
– Technical milestone where
TPM evaluation is
reported
Sub Practice (SP):
Identify TPMs that will be
tracked during development
22/30
23/30
24/30
EV TPM What this tells the PM Derived Measurements
RequirementsHow well the system is maturing as compared to expectations
Number of requirements approved, percentage requirements growth, number TBD/TBR closure per plan, estimated impact of changes, defect profile, defect density, cycle time for requirements changes
System Definition Change Backlog
If changes are being made in a timely manner
Approval rates, closure rates, cycle times, priority density
InterfaceRisk associated with interface development and maturity
Number interfaces approved, percent interface growth, number TBD/TBR closure per plan, estimated impact of interface changes, defect profile, defect density, cycle time for interface changes
Requirement ValidationIf requirements are being validated with applicable stakeholders appropriate to the level of requirement.
Requirements validation rate. Percent requirements validated
Requirement VerificationIf requirements are being verified appropriate to the level of requirement.
Requirements verification rate. Percent requirements verified
Work Product Approval Work progress and approval efficiencyApproval rate, distribution of dispositions, approval rate performance
Technical/Design Review Action Closure
Progress in closing significant action items from technical reviews
Closure rates, action item closure performance, variance from thresholds
Technology MaturityMaturity of key components, subsystems, elements to expectations
Component –Subsystem– Element –System TRL, Technology opportunity exposure, technology obsolescence exposure
Technical Risk HandlingEffectiveness of the risk management process
Percentage of risk handling actions closed on time, percent overdue, percent of risks meeting handling expectations
Technical Staffing And Skills
Adequacy of technical effort and dynamics of actual staffing mix
Technical effort staffing, variance. Efficiency by labor category
Technical PerformanceCurrent performance status, projections and associated risk
Delta performance (planned versus actual).Delta performance to thresholds, objectives
Control Account
WP1: Chassis WP2: Array
Completion Criteria & Taking “Value” for Work Packages
Chassis June July August Sept
Current 100 150 200 250
Completion
Criteria
Requirement
Quality
Design 80%
(drawings)
Completed; TPM
2, 5 and 6 within
PDR tolerances
via basic
performance
model
All TPM within
pre–build
tolerances via
refined
performance
model; TPM 1
and 6 verified
via bench testing
Build complete;
All TPM
validated by
actual
measurement
Cummulative 100 250 450 700
25/30
Using Periodic Program Events to Calibrate “Doneness”
Date: 3/17/2009 8:35:32 AM
Samples: 2000
Unique ID: 114
Name: FIST GEO-1
Completion Std Deviation: 71.23 days
95% Confidence Interval: 3.12 days
Each bar represents 25 days
Completion Date
Fre
qu
en
cy
Cu
mu
lative
Pro
ba
bili
ty
3/17/20117/6/2010 3/16/2012
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.16 Completion Probability Table
Prob ProbDate Date
0.05 10/21/2010
0.10 11/11/2010
0.15 12/2/2010
0.20 12/20/2010
0.25 1/3/2011
0.30 1/14/2011
0.35 1/28/2011
0.40 2/9/2011
0.45 2/22/2011
0.50 3/8/2011
0.55 3/24/2011
0.60 4/7/2011
0.65 4/25/2011
0.70 5/10/2011
0.75 5/27/2011
0.80 6/16/2011
0.85 7/5/2011
0.90 7/26/2011
0.95 9/9/2011
1.00 3/16/2012
Cumulative Distribution Function
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
250000 300000 350000 400000 450000 500000 550000 600000 650000
Total Cost
Pro
ba
bil
ity
Lvl 1 WBS
CA–level risks (100’s)
INPUT
Program Requirements & Scope
PROCESS
IPT–level risks (10’s)
Program–level Risk Assessment of the technical baseline
Ro
llup
of
qu
anti
fied
pro
cess
& p
rod
uct
ris
ks
OUTPUT
When it comes right down to it, the IBR is all about the technical baseline.
IBRs should accompany each major baseline evolution.
In the end, the question implicitly asked of the PM each time is:
“What is your confidence level in your program’s ability to hit cost, schedule, and scope targets
associated with your technical baseline”
Program Cost & Schedule Estimate
IBR Guidance
26/30
Risk: CEV-037 - Loss of Critical Functions During Descent
Planned Risk Level Planned (Solid=Linked, Hollow =Unlinked, Filled=Complete)
Ris
k S
core
24
22
20
18
16
14
12
10
8
6
4
2
0
Conduct Force and Moment Wind
Develop analytical model to de
Conduct focus splinter review
Conduct Block 1 w ind tunnel te
Correlate the analytical model
Conduct w ind tunnel testing of
Conduct w ind tunnel testing of
Flight Application of Spacecra
CEV block 5 w ind tunnel testin
In-Flight development tests of
Damaged TPS flight test
31.M
ar.
05
5.O
ct.05
3.A
pr.
06
3.J
ul.0
6
15.S
ep.0
6
1.J
un.0
7
1.A
pr.
08
1.A
ug.0
8
1.A
pr.
09
1.J
an.1
0
16.D
ec.1
0
1.J
ul.1
1Risk Response
and Risk ID in
IMS
Milestone Date
traceable between
RM Tool and IMS
Connecting Risk Management with Measures of DONE
27/30
Risk
SOW
Cost
WBS
IMS
TPM
PMB
28/30
A Credible Performance Measurement SystemAssures the information is available to the decision maker, needed to deliver
on‒time, on‒budget, and on‒specification
Technical Performance Measures
Cost Schedule
Conventional Earned Value
+
=
The Master Schedule
used to derive Basis of
Estimate (BOE) not the
other way around.
Probabilistic cost
estimating uses past
performance and cost risk
modeling.
Labor, Materiel, and other
direct costs accounted for
in Work Packages.
Risk adjustments for all
elements of cost.
Cost Baseline
Earned Value is diluted by
missing technical
performance.
Earned Value is diluted by
postponed features.
Earned Value is diluted by
non compliant quality.
All these dilutions require
adjustments to the
Estimate at Complete
(EAC) and the To
Complete Performance
Index (TCPI).
Technical Performance
Requirements are
decomposed into physical
deliverables.
Deliverables are produced
through Work Packages.
Work Packages are
assigned to accountable
manager.
Work Packages are
sequenced to form the
highest value stream with
the lowest technical and
programmatic risk.
Schedule Baseline
29/30
Connecting Measures of Cost, Schedule, and Technical Performance
Measuring Actual Physical Percent Complete against the Planned Physical Percent Complete at the time that Physical Percent Complete was planned to be achieved is the foundation of determining a program’s performance.
Doing anything else leads to disappointment.
30/30
means the difference between …
