Preconstruction Planning for Large Science Infrastructure ...sites.nationalacademies.org/cs/groups/bpasite/...SRB Standing Review Board LFO Large Facilities Office NSB National Science

Preconstruction Planning forLarge Science Infrastructure Projects

William L Miller, Ph.D.AAAS Science and Technology Policy Fellow

National Science Foundation

[email protected], 703-292-7886

A B

A comparative analysis of practices and challenges

at DOE, NASA and NSF

dSdt

≥ 0

NRC Board on Physics and Astronomy April 24, 2010

Large Science Infrastructure: a highly varied class…

W. Miller 2Preconstruction Planning at DOE, NASA and NSF

Working definition and portfolios studied

DOE Office of Science

• Science user facilities (mainly at national labs)

NASA Science Missions Directorate (SMD)

• Robotic science missions (mainly GSFC, JPL)

NSF Science & Engineering Directorates, OPP

• Large facilities (receive special construction funds)

• Ground-, sea-, air-, or space-based

• Defined projects (not ongoing fleets, programs)

Large multi-user research platforms intended to serve significant

segments of scientific disciplines for discovery-oriented research.


Preconstruction Planning at DOE, NASA and NSF 4W. Miller

Major science platforms – key differences from research programs

• Expensive investments: ~ $100 millions to several billions

• Long time scales from planning (10-15 years) to

construction (3-5 yr) to operations (10 - ∞ years)

• Out-year “committed” budgeting over decades = mortgage

• Big Science Big $$ Big decisions

On the OMB/Congress/Community radar screens

Often need to prioritize (across disciplines)

Require a lot of oversight policy and process

Often need to partner


Many stakeholders are engaged…

Large ScienceInfrastructure

ScienceEnterprise

TechnologyEnterprise

NationalPriorities

InteragencyPartnerships

InternationalCooperation

Lots of interest in process, performance

and outcomes…

Year-long cross-agency study: Approach

Desired outcomes:

Indications for partnering?

Improvements to NSF practices?

Useful document for developers and stakeholders at NSF & beyond.


Questions:

1. How do agencies compare in

their infrastructure enterprises?

2. How do their development

processes align?

3. Commonalities and differences

in practices and challenges?

Sources:

• Agency documents,

other reports & analyses

(GAO, NRC, Rand, industry…)

• Look “under the hood”:

~45 stakeholder interviews,

design reviews, site visits,

advisory committee meetings, …

Comparative anatomy: Differences and similarities…


Comparative anatomy: Agency structures for science

FPD Federal Project Director

OECM Office of Engineering and Construction Mgmt

OPA Office of Project Assessment

SRB Standing Review Board

LFO Large Facilities Office

NSB National Science Board

S&E Science and engineeringPolicy and process

Independent review

$26.6B

$4.9B $4.5B

$18.7B

$2.1B

$6.9B

$5.7B

Budgets: FY 2010 approps, from FY2011 Congressional Requests

Oversight bodies

DOESecretary

Program Offices

Laboratories

Projects

OPA

Programs

Centers

Projects

Science Missions

Directorate

NSFDirector

Programs

Awardees

Projects

S&E Directorates

& Offices

NASAAdministrator

Chief

Engineer

NSB

SRBs PanelsDivisions

Divisions

UnderSec

Divisions

42 U.S.C.§1873b

(“shall not operate

laboratories”)

Ad

min

istr

atio

nP

rog

ram

sP

roje

cts

FPDs

Ofc of Science

Management

Support Office

OECM

Budget, Finance

& Award Mgmt

LFO


Key agency character differences

Internal Resources

• DOE and NASA have infrastructure-intensive MISSONS, and in-house project

resources (Labs, Centers).

• NSF mainly funds research grants, and “shall not operate laboratories” (by statute,

42 U.S.C.§1873b).

Project-related factors

• DOE has extensive legacy infrastructure (National Labs): can leverage for new

projects, but cost a lot to maintain….

• NASA spacecraft operate remotely in harsh environments, with high risk of

catastrophic failure ~ Drives extremes of planning, test … and oversight.

• NSF large facilities are highly varied – giant telescopes to distributed sensor

networks – and in disciplines new to multi-user facilities (e.g. Seismology, Biology).



Development process framework focus on planning

Early

acquisitions

Early

operations

Project

CloseoutTerminate

Facility

Development

Conceptualization Construction DOperations

Begin major

Investment

Conceptual

Preconstruction Planning

PreliminaryFinal

design

Ongoing community input, horizon planning,

reprioritization, and portfolio adjustment

Preconstruction planning (also called definition & design, formulation…)

• Plan Management/governance plans, WBS assemble project team

• Design Goals, requirements iterative design bring to readiness

• Invest R&D, necessary technologies bring to readiness

• Estimate Effort, cost, schedule, reserves, risks refine to believability

• Govern Stand-up and start oversight & decision-making structures/processes

Development policies & procedures are captured formally

Title ID Originator Released

DOE

“Program and Project

Management for the

Acquisition Of Capital Assets”

Order

413.3A

Office of Engineering and

Construction ManagementJuly 2006

NASA

“Space Flight Program &

Project Management

Requirements”

NPR

7120.5D

Office of the

Chief Engineer

Mar 2007(under

revision)

NSF “Large Facilities Manual”NSF

10-12Large Facilities Office

Nov 2009(update from

May 2007)

Top-level “best practices” for science platform development,

issued by project policy/assurance offices

All are recent and still evolving (e.g. NASA, NSF)

Processes interpreted and implemented by the sponsoring programs


12Preconstruction Planning at DOE, NASA and NSF

ImplementationFormulation

Concept

Studies

Prelim Design &

Tech Completion

Final Design &

Fabrication*

Assembly,

Integ & Test,

LaunchOperations

Concept &

Tech Devel

Pre-Phase A Phase A Phase B Phase C Phase D Phase E

KDP-A KDP-B KDP-C KDP-D KDP-ENASA-SMD Ref: NASA NPR 7120.5D

Development process crosswalk: Terminology and alignment

MCR MDR

SRR

PDR CDR SIR

ORR

FRR

LRRPLAR

CERRCDR Critical Design Review

CERR Critical Events Readiness Review

FRR Flight Readiness Review

KDP Key Decision Point

LRR Launch Readiness Review

MCR Mission Concept Review

MDR Mission Definition Review

ORR Operational Readiness Review

PDR Preliminary Design Review

PLAR Post-Launch Assessment Review

SDR System Definition Review

SIR System Integration Review

SRR System Requirements Review

*elongated to visually align NASA, NSF & DOE

equivalent events. Sequences on this chart do not

represent typical or relative phase durations.

Approve Implementation at KDP-C

CDR PDR FDR Ops

NSB Approved

Preliminary

DesignFinal Design Construction Operations

Horizon planning and

Conceptual Design

Readiness

NSFRef: NSF 0738

CDR Conceptual Design Review

FDR Final Design Review


Ops Operations Review

Approvals

Post-CDR Approve advance to Readiness

Post-PDR Approve submission to Nat. Science Board (NSB)

Post-FDR Congress appropriates MREFC funds

Post-Ops Approve Operations start

IPR** CDR EIR

PDR

FDR

IPR/EIR**

ORR/RA

**CD-0 IPR and CD-3 EIR for >$750M projects

CDR Conceptual Design Review

EIR External Independent Review (OECM)

FDR Final Design Review

IPR Independent Project Review (SC)

ORR Operations Readiness Review


RA Readiness Assessment

Critical Decision (CD) approvals

CD-0 Approve mission need

CD-1 Approve Alternatives selection & cost range

CD-2 Approve Performance baseline

CD-3 Approve Construction start

CD-4 Approve Operations start

DOE-SCRef: DOE O 413.3A

Execution

Pre-conceptual

Planning

Trans/Closeout

OperationsConceptual

Design

CD-0 CD-1 CD-2 CD-4CD-3

Initiation Definition

Preliminary

DesignFinal Design Construction

DecisionReview

W. Miller

Process On-Ramps (when does a project start?)

DOE &

NASA

• Projects may be identified in long-range strategic/science plans1,2

based on community indications (e.g. Decadal Surveys, roadmaps)

• Establish “mission need” & feasibility at milestone #1 (CD-0, KDP-A)

NSF• Peer-reviewed (un)solicited proposals, workshops, studies, etc.

• Evolved concepts may be brought to development: ~ CDR

• “Mission-driven” – projects determined via strategically-defined goals and priorities

1. Facilities for the Future of Science, A twenty year outlook, DOE/SC-0078, Dec 2003; and Four Years Later: an Interim

Report on Facilities for the Future…, Aug 2007. 2. NASA Strategic Plan, 2006; and NASA Science Plan 2007–2016

• “Community-driven” – projects “bubble up” from the scientific disciplines

Horizon concepts,

prioritization Conceptual Preliminary Final

Preconstruction Planning

?


• Agencies usually “commit” after PDR (but momentum is needed to get there…)

Governance, oversight and approval

Independent

Review

Internal decision

support

Sign Off

(e.g. Implement Project)

NASA

PDR

by Standing Review

Board

Program Mgmt Council

• Program Manager

• Center Mgmt Council

• Technical Authority

• Project Manager

KDP-C

Decision Authority (DA)(AA for Cat 1, otherwise MDAA)

Approve Implementation

(Enter Phase C – Final Design & Fab)

NSF

PDR

by Review Panel

NSF Director

MREFC* Panel

• Program Officer

• Directorate/Division

• DDLFP, CFO

NSB Approval

for inclusion in a future

budget request

DOE

PDR

by OPA

“Lehman Review”

(also: EIR by OECM)

Energy Systems Acquisition

Advisory Board (ESAAB)

• OPA briefs AE, calls ESAAB

• Federal Project Director

CD-2

Acquisition Exec (AE)

(DepSec, US Sci, SC Dir/AD)

Approve Performance Baseline

(Enter Final Design Stage)

* Major Research Equipment and Facilities Construction W. Miller 14Preconstruction Planning at DOE, NASA and NSF

Funding streams

MO&DAProgram Funds, project line-item

Mission Operations

and Data Analysis

Execution

Pre-conceptual

PlanningTrans/Closeout

Operations

Conceptual

Design

CD-0 CD-1 CD-2 CD-4CD-3

Initiation Definition

Preliminary

DesignFinal Design Construction

NSB Approved

Preliminary

DesignFinal Design Construction Operations

Horizon planning and

Conceptual Design

Readiness

R&RAProgram Funds (Research & Related Activity, R&RA) MREFC

Major Research Equipment

and Facilities Construction

Concept

Studies

Prelim Design &

Tech Completion

Final Design &

Fabrication

Assembly,

Integ & Test,

LaunchOperations

Concept &

Tech Devel

Pre-Phase A Phase A Phase B Phase C Phase D Phase E

KDP-A KDP-B KDP-C KDP-D KDP-E

Formulation Implementation

Requires separate appropriation

Program Ops FundsLIC Prelim Engr & Design (PED)

Program Funds

Two modes for intramural projects: Line Item Construction (LIC) or

Major Item of Equipment (MIE, no major construction)

Major Item of Equipment (MIE)

LIC Construction

NASA-SMD

NSF

DOE-SC

• Separate funds support research (Research and Analysis, R&A)

• Same funds support research (Research & Related Activities, R&RA)

• Separate funds support research


Challenges: Estimating project cost

• When to state:

Recent reports

on NASA data

by GAO, NASA

and Industry

Desire to state

costs LATER

Pressure to state

costs EARLIER

• Strategic Plans, Surveys etc.

• Manage portfolios

• Request appropriations

• Understand project

• Improve estimates

• Track realistically

Planning stage

• Unpredictable external factors: Funding cuts and delays. Partner delivery and contribution issues. Fuel costs for distributed systems. Exchange rates.

• Unpredictable internal factors: Reprioritizations, overruns of other projects can impact available funds and resources. (Delays may add useful time but at a cost).

• Substantial cost increases can occur late in project:– Overoptimistic technology readiness

– Unmaterialized savings from use of “heritage” designs

– Underestimated difficulties during system I&T and verification

– instrument development slow or unmanaged


Challenges: Portfolio management analysis

• For large portfolios, problems on one project can ripple through others, and out-years are hard(er) to predict.

• Challenge increases with portfolio size and project complexity.

• Need to do quantitative (retrospective and prospective) analyses, e.g.:

– Spending by stage?, is this changing?

– Funding profile dependence on project type, complexity, cost….

– Outcomes vs. initial baselines, cost estimates

Portfolio analysis can

– more fully describe the portfolio

– inform decisions on future projects, and

– serve as comparison with other agencies.


Portfolio analysis: Advantages of a good database…

Source: David Bearden, Presentation to NASA/GSFC, June 3, 2008


Other challenges…

• Workforce training new generation

o Another Federal dip (many retirements after years of flat budgets –

recall the mid-1980s) Losing corporate memory.

o Scarcity of seasoned program/project managers for large projects*

o NASA APPEL and annual PM Challenges are great models…

• Collecting and disseminating Lessons Learned

o NASA – required for all development activities (NPR 7120.6);

public database: http://llis.nasa.gov/offices/oce/llis/home/

o DOE – on-line LL repositories, project closeout reports

o NSF – some activities by programs and awardees

• Can these activities be leveraged through sharing?

*e.g. DOE whitepaper on Federal Project Directors, www.er.doe.gov/opa/pdf/SC%20FPD%20Leadership%20Skills%20rev5.pdf


http://llis.nasa.gov/offices/oce/llis/home/

http://www.er.doe.gov/opa/pdf/SC FPD Leadership Skills rev5.pdf


Challenges: Partnering

Inter-agency partnering: Joint Oversight Groups (JOGs)

• DOE-NASA: FERMI, Joint Dark Energy Mission

• NSF-DOE: U.S. Large Hadron Collider (LHC)

International partnering:

• A higher dimension of “opportunity cost”


-Source: R. Staffin, 14Feb2006, FY06 presentation to HEPAP, www.er.doe.gov/hep/files/pdfs/HEPAPFeb142005Staffin.pdf

Challenges: Partnering impedance (mis)matches

Strategic

valuation

Technical

domains

Management

practices

Capabilities

& Practices

Better planning could involve assessment of COMPLEMENTARITY across

capabilities, practices, lessons learned, etc…)


Synthesis of best practices for partnering…

• Early engagement. Agree on clear goals, timeframe, and effective membership rules and governance structure.

– Clear leadership. Having a dominant partner may work best…

– Complementarity assessment. Learn partner enterprises and realities. Identify technical domains required and respective partner(s) capabilities.

• Clear (shared) process. Agency processes align well, but ideally want an integrated review/decision process (i.e. more than a JOG).

– Strong project management team. Aim for a single team, in place before funding begins, with effective budget authority (not just a coordinator).

– Transparent budgeting. Adopt standard costing and budgeting techniques.

• Open knowledge exchange. Identify effective interfaces at all levels. Maintain open, frequent and honest communication.

Cross-agency training? Shared lessons learned?

Reference material

DOE Lifecycle Process – from O413.3A

Source: S. Meador, Feb 2009, based on DOE Order 413.3A


25W. MillerPreconstruction Planning at DOE, NASA and NSF

Conceptual Design Stage Readiness Stage Board Approved Stage Construction

Concept development – Expend approximately

1/3 of total pre-construction planning budget

Develop construction budget based on

conceptual design

Develop budget requirements for advanced

planning

Estimate ops $

Preliminary design

Expend approx 1/3 of total pre-

construction planning budget

Construction estimate based on

prelim design

Update ops $ estimate

Final design over ~ 2 years

Expend approx 1/3 of total pre-

construction planning budget

Construction-ready budget &

contingency estimates

Preliminary Design

Develop site-specific preliminary

design, environmental impacts

Develop enabling technology

Bottoms-up cost and contingency

estimates, updated risk analysis

Develop preliminary operations cost

estimate

Develop Project Management Control

System

Update of Project Execution Plan

Final Design

Development of final construction-

ready design and Project Execution

Plan

Industrialize key technologies

Refine bottoms-up cost and


Finalize Risk Assessment and

Mitigation, and Management Plan

Complete recruitment of key staff

Conceptual design

Formulation of science questions

Requirements definition, prioritization,

and review

Identify critical enabling technologies and

high risk items

Development of conceptual design

Top down parametric cost and


Formulate initial risk assessment

Initial proposal submission to NSF

Initial draft of Project Execution Plan

Construction per

baseline

Pro

ject e

vo

lutio

nB

ud

ge

t e

vo

lutio

nO

ve

rsig

ht e

vo

lutio

n

Merit review, apply 1st and 2nd ranking

criteria

MREFC Panel briefings

Forward estimates of Preliminary Design

costs and schedules

Establishment of interim review schedules

and competition milestones

Forecast international and interagency

participation and constraints

Initial consideration of NSF risks and

opportunities

Conceptual design review

NSF Director approves Internal

Management Plan

Formulate/approve Project

Development Plan & budget;

include in NSF Facilities Plan

Preliminary design review and

integrated baseline review

Evaluate ops $ projections

Evaluate forward design costs

and schedules

Forecast interagency and

international decision

milestones

NSF approves submission to

NSB

Apply 3rd ranking criteria

NSB prioritization

OMB/Congress budget

negotiations based on Prelim

design budget

Semi-annual reassessment of

baseline and projected ops

budget for projects not started

construction

Finalization of interagency and

international requirements

Final design review, fix

baseline

Congress appropriates

MREFC funds & NSB

approves obligation

Periodic external review during

construction

Review of project reporting

Site visit and assessment

MREFC $

Expenditure of budget and

contingency per baseline

Refine ops budget

MR

EF

C P

an

el re

co

mm

en

ds a

nd

NS

F D

ire

cto

r

ap

pro

ve

s a

dva

nce

to

Re

ad

ine

ss

NS

F a

ppro

ves s

ubm

issio

n to N

SB

Congre

ss a

ppro

priate

s funds

Funded by R&RA or EHR $

NSF oversight defined in Internal Management Plan, updated by development phase

Proponents development strategy defined in Project Development Plan Described by Project Execution Plan

Source: NSF Large Facilities Manual

NSF vs. Doe Process

Conceptual Design

Preliminary Design

Final Design

Construction

Operations

R&RA $ R&RA $ R&RA $ R&RA $

Approximate DOE Translation:

CD 0 CD 1 CD 2 CD 3 CD 4

Approve

mission need

Approve

alternate

selection and

cost range

Approve

performance

baseline

Approve

construction

start

Approve

operations

start

CDR PDR FDROperations

Review

Science

Review

Renewal

Review,

etc.

MREFC $

-- Source: M. Coles, NSF


NASA Project Life Cycle Process – from NPR 7120.5D


Authorities and bodies engaged in project oversight


Oversight: Project categorization sets governance level

DOE Cost categories: 5-20, 20-100, 100-400, 400-750, >750 ($M)

• Approval level depends on cost category

• Highest category requires DOE Deputy Secretary approval

NASA Cost Categories: based on Life Cycle Cost Estimate (LCCE):

(1) > $1B (or nuclear powered or human spaceflight)

(2) $250M to $1B

(3) < $250M (High Priority projects may be Cat 2)

• Approval level depends on project category

• Highest category requires NASA Administrator approval

NSF Either MREFC (eligibility based on cost*) or non-MREFC

• MREFC approvals by Director, National Science Board

• Non-MREFC projects governed within Directorates (but may

require NSB approval depending on cost)

*estimated cost exceeds 10% of a directorates budget


Governance and Oversight: Independent review

DOE Expert Panels

• Run by SC Office of Program Assessment (OPA)

• Chair always a DOE federal employee (usually from OPA)

• Evolving membership of DOE staff, contractors, other experts; usually a core

group, but introduce “new blood” as project evolves

NASA Standing Review Board (SRB)

• Coordination: Independent Program Assessment & Oversight Office (IPAO)

• Chair vetted and agreed by project, program, Decision Authority

• Fixed membership of NASA staff, contractors, other experts (selected by chair)

follows entire lifecycle, including subsystem reviews. Add experts as needed.

NSF Expert Panels

• Assembled for each review by Program Officer with assistance of LFO

director.

• Changing membership of external technical and scientific experts, but often try

to keep a core group for institutional memory.


Funding and acquisitions

DOE NASA NSF

Ownership Primarily

owner/operator

Primarily

owner/operator

Not owner/operator (except

polar)

Expenditure vehicles

In-house

projectsContracts

Contracts.

CAs often for ops

Contracts

(Polar Programs)

External

projects

Recent CA with MSU for

FRIB developmentSame as in-house

CAs for development and

operations

Carryover Funds are “no year”Funds are 2-year

(reduced for FY 2010?)

No R&RA carryover

MREFC construction funds

are “no year”

Acquisition

Start

DOE and NASA can acquire “long-lead items” during

initial development (after CD-1 and KDP-B,

respectively), per approved project plans.

No acquisition until MREFC

funds are appropriated, NSB-

approved and awarded

(after FDR)

CA – Cooperative Agreement


Acronyms

CD Critical Decision (DOE)

CDR Conceptual Design Review (DOE, NSF)

Critical Design Review (NASA)

DDLFP Deputy Director for Large Facility Projects, head of LFO (NSF)

EIR External Independent Review (DOE OECM)

IPAO Independent Program Assessment & Oversight Office (NASA)

JOG Joint Oversight Group

KDP Key Decision Point (NASA)

LFO Large Facilities Office (NSF)

MO&DA Mission Operations and Data Analysis funding account (NASA)

MREFC Major Research Equipment and Facilities Construction funding account (NSF)

NAR Non-Advocate Review (PDR, NASA)

OECM Office of Engineering and Construction Management (DOE)

OPA Office of Program Assessment (DOE/SC)

PED Preliminary Engineering and Design funding account (DOE)


PNAR Preliminary Non-Advocate Review (MDR, NASA)

R&RA Research and Related Activities funding account (NSF)

R&A Research and Analysis funding account (NASA)

SC Office of Science (DOE)

SRB Standing Review Board (NASA)

SMD Science Missions Directorate (NASA)


Documents

Preconstruction Planning for Large Science Infrastructure ...sites.nationalacademies.org/cs/groups/bpasite/...SRB Standing Review Board LFO Large Facilities Office NSB National Science