Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
Mission Operations Cost Estimation Tool (MOCET)
2019 UpdatesMarc Hayhurst, Elliott Tibor, Brian Wood
The Aerospace Corporation
Cindy Daniels, Lissa Jordin, Washito Sasamoto, Waldo RodriguezNASA Langley Research Center
Science Office for Mission Assessments (SOMA)
August 14, 2019
Approved for public release OTR 2019-01048.
© 2019 The Aerospace CorporationAerospace acknowledges NASA’s sponsorship under Contract NNL11AA01B Task Order 80LARC18F0021
2
Topics
• MOCET Overview & Status
• Updates in MOCET v1.4
• Planned Reserves Process
• User Community
• Summary & Future Work
3
MOCET Overview
• The Mission Operations Cost Estimation Tool (MOCET) – A capability for Phase E estimation jointly developed by The Aerospace Corporation and NASA
Science Office for Mission Assessments (SOMA)– Based on actual costs of historical missions with emphasis on competed missions– Constructed by breaking the mission operations cost into the various phases– Has few subjective inputs– Estimates total Phase E mission cost– Implemented entirely in Excel and requires no additional software or tools– Also includes a user manual which provides additional instruction and background
Mar-34 Oct-34 Apr-35 Nov-35 May-36 Dec-36 Jun-37
Cruise
Approach/EDL
First Landed Month
Landed Prime Operations
Mars 2034 RoverMission Schedule
Download from ONCE Model Portal https://oncedata.msfc.nasa.gov
Available external to NASA via https://software.nasa.gov
For more information:Email: [email protected]
4
Bold = New Data in v1.4 Potential New Data for possible version 1.X update
MOCET Mission Types Database Overview
Mission/CER Type Program Missions
Planetary
Discovery MESSENGER, Stardust, Deep Impact, GRAIL, NEAR, Dawn
Mars Scout PhoenixRobotic Lunar Exploration LRO, LADEENew Frontiers New Horizons, Juno (Orbital Ops), OSIRIS-REx (Cruise, Approach/Orbit Insertion)Mars Exploration MRO, Odyssey, MER, MSL, MAVEN, Insight
Earth ScienceEarth System Science Pathfinder (ESSP) GRACE, CloudSat, CALIPSO, Aquarius, OCO-2, CYGNSS, OCO-3, GEDI
Earth Systematic Missions (ESM) Aqua, Aura, Terra, Jason-1, OSTM, ICESat, GPM, SMAP, TSIS-1, ICESat-2, GRACE-FO
Explorers
Mission of Opportunity (MO) Suazku (ASTRO-E2), TWINS, CINDI, NICER, GOLD
Small Explorers (SMEX) NuSTAR, IRIS, IBEX, AIM, GALEX, RHESSI
Medium Explorers (MIDEX) THEMIS, Swift, WISE, TESS
Near Earth Discovery Helio-Astro
Discovery Genesis, KeplerSolar Terrestrial Probes (STP) STEREO, TIMED, MMSLiving With a Star (LWS) RBSP, SDO, PSPCosmic Origins Spitzer
Physics of the Cosmos Fermi, Chandra
Data Sources Utilized and Consulted
– NASA SAP BusinessWarehouse• Monthly Expenditures
– Monthly Flight Project Reviews (FPR)and Monthly Status Reviews (MSR)• Monthly Expenditures, Mission Events
& Schedule
– NASA Cost Analysis DataRequirements (CADRe)• Project development costs used
to assign mission class
5
MOCET CER UpdatesPlanetary
CER Name New Data Updated Inflation
Comments
Nominal/Quiescent Cruise CER X X OSIRIS-REx data incorporated
Checkout Cruise CER X Landed Missions Broken Out
Mars Landed Cruise CER X X New CER with Phoenix, Insight, and MSL
Orbital Operations CER X X Juno & OSIRIS-REx data incorporated.
Flyby/Encounter CER X
Approach/Orbit Insertion/Reduction CER X X OSIRIS-REx data incorporated
Approach EDL CER X X Insight data incorporated
Landed Checkout CER X X Renamed; Insight data incorporated
Landed Prime Operations X X Insight data incorporated
Data/Sample Analysis & Archiving X
Extended Orbital Ops CER X
Extended Landed Ops CER X
Earth/Near-Earth/OtherCER Name New Data Updated
InflationComments
Earth/Near-Earth Orbiting Checkout CER X X
Earth Science Prime Operations CER X X CYGNSS, SMAP
Explorer Prime Operations CER X X NICER, GOLD, TESS data incorporated
Near Earth Discovery Helio Astro Prime Operations CER X
Instrument Only Operations CER X X New CER, Satellite and ISS Hosted
6
Explorers Prime Operations Phase CER• CER Function:𝑌𝑌 = 0.5338 + 0.3740 ∗ 𝐼𝐼𝐼𝐼𝐼𝐼 − 0.3583 ∗ 𝐼𝐼𝐼𝐼𝐼𝐼 + 0.1225 ∗ 𝐼𝐼𝐼𝐼
Variable Name Units Definition
Y Cost FY18 $M Average monthly checkout phase cost
IMe IsMedium? 0 or 1 0 No, 1 Yes, Is this a medium mission class mission?
IMi IsMicro? 0 or 1 0 No, 1 Yes, Is this a micro mission class mission?
IA IsAstro? 0 or 1 0 No, 1 Yes, Is this an astrophysics mission?
• Goodness of FitMeasure Value
Regression Method OLSStandard Error (RSS) 0.07
Average Percentage Bias 0.0%Coefficient of Determination (R^2) 0.96
Adjusted R^2 0.95Pearson's Correlation Sqd (r^2) 0.96
Number of Observations 15Number of Input Variables 3
$1.2
$1.0
$0.8
edtc $0.6
edi
rP $0.4
$0.2
$- $- $0.2 $0.4 $0.6 $0.8 $1.0 $1.2
Actual
• Database
1. IBEX (IMe 0, IMi 0, IA 0) 6. Swift (IMe 1, IMi 0, IA 1) 11. Suzaku (IMe 0, IMi 1, IA 1)
2. AIM (IMe 0, IMi 0, IA 0) 7. WISE (IMe 1, IMi 0, IA 1) 12. IRIS (IMe 0, IMi 0, IA 0)*
3. NuSTAR (IMe 0, IMi 0, IA 1) 8. RHESSI (IMe 0, IMi 0, IA 0) 13. NICER (IMe 0, IMi 1, IA 1)
4. GALEX (IMe 0, IMi 0, IA 1) 9. CINDI (IMe 0, IMi 1, IA 0) 14. TESS (IMe 1, IMi 0, IA 1)
5. THEMIS (IMe 1, IMi 0, IA 0) 10. TWINS (IMe 0, IMi 1, IA 0) 15 GOLD (IMe 0, IMi 1, IA 0)
Additional Notes• *IRIS was a SMEX mission but the development cost grew beyond small mission class boundary. In operations the IRIS average monthly cost is also higher than any other SMEX mission.• **Copies of identical instruments (THEMIS, TWINS) or suites of similarly functioned sensors (CINDI) are considered to be a single instrument since the same data processing tools can be used for them.• ***Earth & near Earth orbiting multiple spacecraft missions such as THEMIS may be estimated with MOCET without further adjusting the output estimate.
7
Mars EDL Cruise Prime Phase CER• CER Function: 𝑌𝑌 = 0.0056 ∗ 𝑅𝑅𝑅𝑅𝐼𝐼1.1388
Variable Name Units Definition
Y Cost FY19 $M Planetary mission average monthly prime checkout cruise phase cost
RLM Rover/Lander Mass kg > 0Mass of rover or lander plus payload instruments for which NASA will fund operations (exclude contributed instruments).
• Goodness of FitMeasure Value
Regression Method GERM ZMPEStandard Error of the Estimate 14.2%
Average Percentage Bias 0.0%Pearson's Correlation Sqd (R^2) 0.98
Number of Observations 3Number of Input Variables 1
$16
$14
$12
ed $10
tc $8
redi
P $6
$4
$2
$- $- $2 $4 $6 $8 $10 $12 $14 $16
Actual
• Database
1. Phoenix (342.9) 2. MSL (919.9) 3. Insight (318.6)
4. MER (A/B)*
Additional Notes• *MER was not included in the model due using multiple spacecraft. At this time there is not sufficient data to recommend an adjustment for multiple planetary spacecraft and the model does not support
estimation of planetary multiple spacecraft missions
8
Landed Operations Prime Phase CER• CER Function: 𝑌𝑌 = 0.0591 ∗ 𝑅𝑅𝑅𝑅𝐼𝐼0.7034
Variable Name Units Definition
Y Cost FY19 $M Planetary mission average monthly prime landed operations phase cost
RLM Rover/Lander Mass kg > 0Mass of rover or lander plus payload instruments for which NASA will fund operations (exclude contributed instruments).
• Goodness of FitMeasure Value
Regression Method GERM ZMPEStandard Error of the Estimate 7.2%
Average Percentage Bias 0.0%Pearson's Correlation Sqd (R^2) 0.98
Number of Observations 3Number of Input Variables 1
$8
$7
$6
ed $5
tc $4
redi
P $3
$2
$1
$- $- $1 $2 $3 $4 $5 $6 $7 $8
Actual
• Database
1. Phoenix (342.9) 2. MSL (919.9) 3. Insight (318.6)
4. MER (A/B)*
Additional Notes• *MER was not included in the model due using multiple spacecraft. At this time there is not sufficient data to recommend an adjustment for multiple planetary spacecraft and the model does not support
estimation of planetary multiple spacecraft missions
9
MOCET Reserves Approach (1 of 2)
• When calculating the reserve level for operations phase estimates, a percent reserve of the base operations budget is often considered– Rule of thumb used by some considers a minimum 10% up to 25% for Phase E– Aerospace publication* showed that on average from launch estimate to final actual
growth was around 10%
• Not a simple matter of just looking at the offeror’s proposal and seeing if they are carrying between 10% and 25% reserve on their estimate– Offerors’ estimates and MOCET estimates vary significantly– Requires a new method of computing reserves for each MOCET estimate
• To determine best method of computing reserves for MOCET– Obtained data from available Step 1 and Step 2 proposals as well as other milestones
for comparison– Generated MOCET estimates using input parameters as given at Step 1, Step 2, etc.– Also, collected the offerors estimates for comparison
*Phase E Cost Analysis for NASA Science Missions, Robert Bitten, Marc Hayhurst, Debra Emmons, Claude Freaner, and Voleak Roeum, AIAA SPACE 2012 Conference & Exposition. Pasadena, California.
10
MOCET Reserves Approach (2 of 2)
• With the MOCET estimates generated at each milestone the percent difference from the final actual was computed for each mission– The following equation was used to compute the percent difference
𝐹𝐹𝐼𝐼𝐹𝐹𝐹𝐹𝐹𝐹 𝐼𝐼𝐴𝐴𝐴𝐴𝐴𝐴𝐹𝐹𝐹𝐹 𝐶𝐶𝐶𝐶𝐶𝐶𝐴𝐴 − 𝐼𝐼𝑀𝑀𝐶𝐶𝑀𝑀𝑀𝑀 𝑀𝑀𝐶𝐶𝐴𝐴𝐼𝐼𝐸𝐸𝐹𝐹𝐴𝐴𝐼𝐼 @ 𝐼𝐼𝐼𝐼𝐹𝐹𝐼𝐼𝐶𝐶𝐴𝐴𝐶𝐶𝐹𝐹𝐼𝐼𝐼𝐼𝑀𝑀𝐶𝐶𝑀𝑀𝑀𝑀 𝑀𝑀𝐶𝐶𝐴𝐴𝐼𝐼𝐸𝐸𝐹𝐹𝐴𝐴𝐼𝐼 @ 𝐼𝐼𝐼𝐼𝐹𝐹𝐼𝐼𝐶𝐶𝐴𝐴𝐶𝐶𝐹𝐹𝐼𝐼 ∗ 100% = 𝑃𝑃𝐼𝐼𝑃𝑃𝐴𝐴𝐼𝐼𝐹𝐹𝐴𝐴 𝐷𝐷𝐼𝐼𝐷𝐷𝐷𝐷𝐼𝐼𝑃𝑃𝐼𝐼𝐹𝐹𝐴𝐴𝐼𝐼
• Then the overall average percent difference and standard deviation were computed
• Various methods to compute reserve level were attempted and these metrics were compared at each milestone to determine which methods were most effective– Methodologies driven by historical data were considered to maintain objectivity
• Concluded that the best performing method is to employ a triangle distribution with the Formal Risk (FRISK) process driven by historical percent differences of similar missions at a particular milestone– Similar missions are determined by mission category
11
Percent Difference for all Explorer Missions
• As compared to Explorer Project estimates, average percent difference for MOCET is significantly less– Interesting to note that on average Step 1 estimates were better than those at Step 2
Project Estimate % Difference (Including Reserves)250%
Mission 1
Mission 2
Mission 3200% Mission 4
Mission 5
Mission 6150% Mission 7
e Mission 8
enc
er Average
ff D
i 100%
e ent
c 72%t 71%
a erm Pit 48%
es 50% 34%
nder 24%
U 0%
te 0%
amtisere
Ov -50%
Proposal CSR PDR CDR Launch EOM
MOCET Estimate % Difference (Without Reserve)250%
Mission 1
Mission 2
Mission 3200% Mission 4
Mission 5
Mission 6150% Mission 7
e Mission 8
enc
er Average
ff D
i 100%
e ent
t ca erm Pites 50%
nder
U 10%6% 9% 9% 8% 2%
te 0%
amtisere
Ov -50%
Proposal CSR PDR CDR Launch EOM
12
Percent Difference for all Planetary Missions
• As compared to Planetary Project estimates, average percent difference for MOCET is significantly less– Interesting to note that on average Step 1 estimates were better than those at Step 2
Project Estimate % Difference (Including Reserves)
Mission 1
Mission 280%Mission 3
Mission 4
Mission 560%50% Average
44%40%
e 40%
enc 32%
et era ffm
ii Dt
es ent 20%
10%
nder c
erPU 0%
0%
teamtisere -20%
Ov
-40%Proposal CSR PDR CDR Launch EOM
MOCET Estimate % Difference (Without Reserve)
Mission 1
Mission 280%Mission 3
Mission 4
Mission 560%Average
een
ec 40%
e erfta fim D it 20%
ent 14%
es 9% 10%nd
erc
er 6%
P 3%U -2%
0%
teamtiser -20%
eO
v
-40%Proposal CSR PDR CDR Launch EOM
13
MOCET Reserves Process Overview
Cost Estimates
LM
H
Note: Most-
Likely (M) is average
of all estimates
Triangular Distribution of Cost Estimates
ytilbi L M H
oba
rP
CostLower Most-Likely Upper Limit Limit
Example Cost Distribution• The Formal Risk (FRISK) process driven by
historical percent differences of similar missions at a particular milestone
• Similar missions are determined by mission category
• Adjusted mission estimates are fed into a triangular distribution
• Triangular distribution is fed into FRISK to produce an s-curve and determine the appropriate reserve level
70th Percentile
UFE Estimate
MOCET Estimate
14
ONCE Downloads and Users• To date MOCET has been downloaded from ONCE 183* times since the initial release
• *Downloads include those from inactive users and duplicate downloads – ONCE output generally only shows active users and unique downloads
• v1.0 30 downloads; v1.1 60 downloads; v1.2 34 downloads; v1.3 59 downloads as of June 2019• As of June 2019, 86 unique users have downloaded MOCET from ONCE
• Since the release of v1.0 the number of users has increased steadily
0
20
40
60
80
100
120
140
160
180
200
10/6/2015
1/4/2016
4/3/2016
7/2/2016
9/30/2016
12/29/2016
3/29/2017
6/27/2017
9/25/2017
12/24/2017
3/24/2018
6/22/2018
9/20/2018
12/19/2018
3/19/2019
6/17/2019
Cumulative Downloads
v 1.0 v 1.1NCS 2016
v 1.2NCS 2017
v 1.3NCSS 2018 v 1.0 v 1.1 NCS 2016 v 1.2 NCS 2017 v 1.3 NCSS 2018
Downloads by Month
18 17
16
1412
12 11 1110
10 9 9
8 7 7 7 76
6 5 5 5 5 5 54 4
4 3 3 3 3 32 2 2 2 2 2
2 1 1 1 1 1 10 0 0 0 0 0 0 0
0 O N D J F M Ap M J J Au Se O N D J F M Ap M J J Au Se O N D J F M M J Ja u ul a u a uc a a c a a ul c aAp a ulAu Se O N D J F M
c
Ap M J
t ov ec n-eb y n- - g p t ov ec n-eb y n- - g p t- - -ov ec n-eb y n- - g p t-ov eca eb ua a
- - - n- n-r-1r-1 y- 1 -1 -1 - - - r-1r-1 1 -1 -1 - - - r-1r-1 115 16 16 16 6 16 -1 -1 - - - r-1r-115 15 66 16 6 6 16 16
17 17
- 777 17 7 717 17 1718 18
-17 88 1818 8 8 818 18 1819 19
-99 1919
15
Public Release on software.nasa.gov• MOCET has been approved for public release via
software.nasa.gov– The direct link can be found here:
https://software.nasa.gov/software/LAR-18894-1
• Available to almost anyone* worldwide
• User must create an account on software.nasa.gov
• Fill out the request form
• *A signature authority must be provided for those representing businesses– Cannot be an engineer or those with similar titles– An authorized signature authority is considered
• president of the company• general counsel• or contract officer
• If representing self as a private individual, an additional signature is not required
• Rules may be different for other user types, Academic etc.
• Contact MOCET help staff at [email protected] with any issues you may encounter
16
software.nasa.gov Downloads and Users• There are MOCET 31 users on software.nasa.gov since its release there in April of 2017
• Most users are from Industry (14) and from NASA (7)• There are also 5 Representing Self, 3 Other Civil, and 2 Academic• Of the 31 users, 11 are also international coming from Africa, Asia, Australia, Europe, and South America
17
Summary & Next Steps
• New data is being added to the model which will be released as MOCET v1.4
• MOCET will continue to be periodically updated with new mission data
• Model is currently being used by both evaluators and proposers
• We will continue to engage and grow the user community
Download MOCETONCE Model Portal https://oncedata.hq.nasa.govAvailable external to NASA via https://software.nasa.gov
For More InformationDemo session (see agenda for time and room)Email: [email protected]
Publications• Mission Operations Cost Estimation Tool (MOCET) Version 1.3 and Beyond, 2019 IEEE Aerospace Conference, Big Sky, MT• The Mission Operations Cost Estimation Tool (MOCET): Development History and 2018 Updates, 2018 AIAA SPACE and Astronautics Forum and Exposition. Orlando FL• Mission Operations Cost Estimation Tool (MOCET) v1.3, 2018 NASA Cost and Schedule Symposium, August 2018, Greenbelt MD• Mission Operations Cost Estimation Tool (MOCET), 2017 IEEE Aerospace Conference, Big Sky, MT• Mission Operations Cost Estimation Tool (MOCET) FY17 Update, 2017 NASA Cost and Schedule Symposium, August 2017, Washington DC• Mission Operations Cost Estimation Tool (MOCET) Update, 2016 NASA Cost Symposium, August 2016, Cleveland OH• Mission Operations Cost Estimation Tool (MOCET), 2015 NASA Cost Symposium, August 2015, Mountain View CA