Upload
lexuyen
View
229
Download
0
Embed Size (px)
Citation preview
www.QinetiQ.com
Burchelli Consulting Limited
© 2011 Burchelli Consulting Limited
Maximising Parametric Model Utilisation by Developing Excellent Cost Estimating Relationships
Dr Spencer Woodford, Burchelli Consulting Ltd
Society for Cost Analysis & Forecasting
8 February 2011
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Contents
• Introduction
– Parametric Cost Estimating Relationships
– General Method
• Attributes of an „Excellent‟ CER
– Data
– Model Structure
– Algorithm Development
– Accuracy
– Documentation
– Users
• Example – Aero Gas Turbine Development
• Conclusions
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Introduction
• Parametric Cost Estimating Relationships (CERs):
– are algorithms that forecast costs based upon the examination and
validation of the relationships that exist between a project's technical,
programmatic, and cost characteristics as well as the resources consumed
during its development, manufacture, maintenance, and/or modification;
– can be Simple (single parameter) or Complex (multiple parameters);
– are inconsistently used for Defence projects;
– suffer from a lack of support at higher levels due to poor understanding and
lack of exposure to the levels of accuracy available from the tools.
• Similar algorithms are widely used in numerous industries to estimate
outputs based on a number of technical and/or programmatic inputs, e.g.
– size/mass estimation of aircraft components/systems;
– reliability and maintainability metrics for complex systems;
– manpower numbers required to operate and support complex systems, etc.
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Introduction – General Method
Input Parameters
• System parameters; size, mass, technology, performance, complexity, stealth, etc.
Real World Data
• Acquisition data
• Operational data
• Cost data
Parametric Equations
• Effort Consumed
• Labour rates
• Equipment Costs
Predicted
Whole Life
Cost
Predicted
Total
Effort
Correlation & (Multiple) Regression
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Attributes of an „Excellent‟ CER (1) – Data
• Quantity
– Non-linear equations require a minimum of 11 data points for each
parameter added to the equation
– Never dismiss data for being „too old‟ – technology (improvement) factors in
the model could/should account for this
• Normalisation
– All input data must be in consistent units, e.g. kg, m2
– All costs must all be converted to the same currency in the same financial
year (preferably using recognised & authoritative currency exchange rates)
• Visibility
– A lack of total transparency of the data values, sources, missing/estimated
input parameters, etc. will exacerbate the lack of confidence that others
have in any model
• „Trust me‟ is not good enough to convince others that you have a good model
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Attributes of an „Excellent‟ CER (2) – Model Structure
• There must be a direct (or strongly correlated) relationship between the
inputs and the output of the model
– If there is no good engineering link between parameters, they should be
removed
– Often, there is a theoretical link between input parameters & the output
• This can be used to validate that the model coefficients are of the right order
• Complex systems will likely require more complex models to enable
suitable fidelity to be captured
– Consider splitting the overall output into several sub-CERs
• Increases visibility and ability to correlate/validate individual elements
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Attributes of an „Excellent‟ CER (3) – Model Algorithms (1)
• Simple models can make use of algorithms with single factors and
coefficients, e.g.:
– Aero Engine Inlet Diameter (m) = A × [mass flow rate]b
– „A‟ is a correlation factor; „b‟ is a correlation coefficient: A = 131.5, b = 0.5
Note that „b‟ aligns with the
theoretical relationship between
air mass flow rate and intake
diameter for a fixed (i.e. „ideal‟)
compressor entry velocity
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Attributes of an „Excellent‟ CER (3) – Model Algorithms (2)
• Complex models may contain numerous inputs & coefficients, e.g.
Empty Mass = Wing +
Fuselage + Empennage +
Undercarriage +Engine +
Cockpit + Fuel System +
APU + Flying Controls +
Hydraulics + Pneumatics +
Electrics + Env. Control +
Avionics + Sensors +
Trapped Fuel & Oil + Gun
Calculated vs Actual Empty Masses (kg)
X-35A JSFX-35A Deriv
X-35A JSF X-32A JSF
Tornado F3Tornado GR4
SU-27 Flanker B
Sea Harrier II
Rafale D
Mirage 2000C
MiG-31 Foxh'nd D
MiG-29 Fulcrum K
MiG-25 Foxbat P
MiG-23 Flogger K
MIG-21 Fishbed N
Sepecat JaguarIAI Kfir C.2
Hawk 200Hawk 100
FMA 63 Pampa
F-22 Raptor
F-20 T'shark
F-18E Super Hornet
F-18C Hornet
F-16C Falcon
F-15E Eagle
F-14A Tomcat
F-5E Tiger II
F-4E Phantom
EF Derivative
Eurofighter TyphoonBAe EAP
Aermacchi MB-339CD
A-10 Thunderbolt
y = 0.9906x
R2 = 0.9829
0
5000
10000
15000
20000
25000
0 5000 10000 15000 20000 25000
Actual Empty Mass (kg)
Calc
ula
ted E
mpty
Mass (
kg)
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Attributes of an „Excellent‟ CER (3) – Model Algorithms (3)
• When conducting the regression analysis, choose the regression
approach carefully
Logarithmic scales
can produce VERY
misleading results
Results of Culy Log-Log Model
y = 1.084x + 0.4228
R2 = 0.8905
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 8 9
Ln(Actual Development Cost ($M1990))
Ln
(Ca
lcu
late
d D
eve
lop
me
nt
Co
st
($M
19
90
))
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Attributes of an „Excellent‟ CER (3) – Model Algorithms (4)
• When conducting the regression analysis, choose the regression
approach carefully
Gradient is more
than twice that
expected &
intercept is poor
Culy Model Results - All Gas Turbines
y = 2.5847x + 68.918
R2 = 0.7874
0
2000
4000
6000
8000
10000
12000
0 500 1000 1500 2000 2500 3000 3500 4000
Actual Development Cost ($M1990)
Ca
lcu
late
d D
eve
lop
me
nt
Co
st
($M
19
90
)
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Attributes of an „Excellent‟ CER (3) – Model Algorithms (5)
• When conducting the regression analysis, choose the „objective function‟
(of the internal optimisation) carefully
• R2 is a very commonly used indicator of „goodness of fit‟
– R2 is the „Coefficient of Determination‟ and is defined mathematically as:
where is the mean of the observed data.
– Therefore, using R2 as part of the regression will „skew‟ the observed error
to reduce with increasing values of x
– This is not a problem for some programmes, but across a range of results, it
may be preferable to have a consistent percentage error for all outputs
• Many programmes will be measured on a „% error basis‟, not on total error
• Consider using absolute percentage error instead of R2
– Generates consistent percentage errors across all values
i
i
i
ii
yf
fy
R2
2
2
)(
)(
1 y
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Attributes of an „Excellent‟ CER (4) – Accuracy
• Displaying/exposing model accuracy is a „two-edged sword‟
– Telling a customer that a model is „inaccurate‟ can prevent its use (or
significantly reduce their willingness to use it)
– It is vitally important that we explain that no model can be 100% accurate
• Must expose the level of modelling error in the modelling algorithm
• Use multiple approaches to detail the inaccuracy and how it has been captured
• Example contained in later slide
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Attributes of an „Excellent‟ CER (5) – Model Documentation
• Model documentation must exist to explain the data, sources,
normalisation, model structure, rationale, and applicability
– Lack of documentation will guarantee that the model will not be used
• No-one likes producing model documentation, but it is a necessary evil
– Must be complete, accurate, and synchronised with model developments
• Should be treated as a „log-book‟ for the model
– Can be a powerful source of evidence for use with Customers, CAAS,
Scrutiny, etc.
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Attributes of an „Excellent‟ CER (6) – Expert Users
• Parametric models can very quickly generate highly inaccurate outputs in
the hands of an inexperienced/poorly trained user
– All parametric models have a limited applicability
– Understanding the model and its applicability is key to generating realistic,
accurate outputs
– The user must understand:
• the data, sources, normalisation, and validation
• the model structure and the factors and coefficients
• the optimisation approach and whether the model was generated with R2,
|%| error, or some other objective function
• The units and economic conditions of the model output, and how to translate that
value in to the value required to answer the question
• A „poor user‟ will almost guarantee poor results
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Example – Generation of an (almost) Excellent Aero Gas Turbine Development Cost Estimating Relationship
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Data Sources and Processing
• Historical database obtained from US source
– Contains 32 technical and programme parameter values for each engine,
plus normalised development cost (1990 e.c.)
– 307 data points made up of 10 engine types
• Removed piston, rotary, auxiliary power units, automotive & rocket engines,
leaving 236 data points
• 17 mis-classified or highly outlying engines removed
– 219 aero gas turbine engines remaining
• 115 turbofan, 34 turbojet, 26 turboshaft, 44 turboprop
• 6 parameters difficult to estimate prior to start of development, therefore
not used
– Mass, Part count, Programme duration, Number of test hours, Number of
stages, Number of rotating spools
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Parametric CER Derivation
• Focused on the development of a model of the form:
Cost = x (Parameter A)a x (Parameter B)b x … x y(Parameter ) x z(Parameter )
– = Constant x Type x Problem
– , = dummy variable parameters
– a, b, y, z coefficients calculated using Generalised Reduced Gradient non-
linear optimisation code (GRG2)
• Possible variables identified using a combination of engineering
knowledge and statistical evidence
• Iterative process to identify significant variables to be included within the
model & derive coefficient values
– Relevance of variables checked using statistical checks, e.g. p-test, t-test…
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Common „Errors‟ with Engine CERs
• Technical Parameter Separation (Engineering)
– Thrust as a Technical Parameter
• Thrust is a function of both engine size and technical advance. As such, it is not
a good technical parameter for this model, despite statistical evidence
• Cross-correlation (Statistics)
– Bypass Ratio (BPR) and Mass Flow Rate (WA)
• Large civil aircraft have high WA and BPR. Small turbojets have low WA and
BPR. Small engines are unlikely to have high BPR, as this results in a small,
inefficient engine core
– Turbine Entry Temp. (TET) and Overall Pressure Ratio (OPR)
• Engines require high OPR to make use of efficiencies generated through high
TET. Both are measures of technological advance, and should not appear in the
same equation together
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Results – Model Overview
• Best fitting model was found to contain:
– Engine Type, Problem, WA, OPR, Joint Venture, No: test engines, Mach
• Includes factors for:
– Complex Programmes, Military Engines, Country of Origin, “Skunk Works”
Programmes
• „Type‟ factor gives relative costs of Turboprop and Turboshaft engines,
and Turbofan & Turbojet engines, with and without reheat
• „Problem‟ factor gives cost variation according to novelty of proposed
engine design
– 11 levels from „All-New Design‟ to „De-Rate‟
• All factors are „trade-able‟ – which is most beneficial?
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Results & Comparison with Published ModelsComparison of Engine Development CERs
y = x + 9E-07
R2 = 0.9698
y = 0.2917x + 164.76
R2 = 0.3719
y = 0.4771x + 431.79
R2 = 0.3172
y = 2.5847x + 68.918
R2 = 0.7874
y = 1.3335x + 29.350
R2 = 0.8754
0
500
1000
1500
2000
2500
3000
3500
4000
0 500 1000 1500 2000 2500 3000 3500 4000
Actual Development Cost ($M 1990ec)
Calc
ula
ted
De
ve
lop
me
nt
Co
st
($M
19
90
ec
)
QinetiQ Younossi Birkler Culy 1995 Culy 2
Best Fit Best Fit Best Fit Best Fit Best Fit
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Accuracy & Comparison with Published ModelsModel Accuracy by Error Band
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Error (%)
% o
f D
ata
wit
hin
Sp
ec
ifie
d B
an
d
QinetiQ Younossi Birkler Culy 1995 Culy 2
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Accuracy – Capture & Addition to Model Algorithm
• Model „inaccuracy‟ captured using
„best-fit distribution‟ tools within COTS
software product
– Gives clear link between „accuracy‟
(e.g. R2) and confidence levels
• Model algorithm updated to include
„inaccuracy‟ as part of the stochastic
modelling process
– Allows inclusion of model
„inaccuracy‟ as well as input
„uncertainty‟
Engine Development Error Data & Best-Fit Distribution
0.0
0.5
1.0
1.5
2.0
2.5
3.0
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
5.0% 5.0%90.0%
-0.248 0.664
Mean = 0.10651
Mean = 0.10651
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Conclusions
• Parametric CERs are used inconsistently across MoD projects
• The generation of Excellent CERS requires:
– a large database of high-quality data points;
– accurate and consistent data „cleansing‟ and normalisation;
– inclusion of parameters that are demonstrate theoretical/engineering links;
– an appropriate level of complexity to reflect the engineering reality;
– adoption of a powerful regression technique that does not hide error;
– selection of an „objective function‟ that is relevant for the question/model;
– capture and usage of model „inaccuracy‟ to enhance the model output;
– comprehensive documentation that explains all of the above (and more!);
– trained, expert users to ensure that the model is applied correctly.
• Generating Excellent CERs may increase their use within MoD projects
– Doing so would increase quality and confidence in resulting cost forecasts
A presentation date, a presentation title
www.QinetiQ.com
© 2011 Burchelli Consulting Limited
Burchelli Consulting LimitedBurchelli Consulting LimitedFew things in life are as simple as Black and White