Upload
stuart-parsons
View
218
Download
0
Tags:
Embed Size (px)
Citation preview
Bill ValdezDirector, Planning and AnalysisSeptember 2005Email: [email protected]
U.S. Department of EnergyU.S. Department of EnergyOffice of ScienceOffice of ScienceBudget and PlanningBudget and Planning
Defining the Problem
“How much should a nation spend on science? What kind of science? How much from private versus public sectors? Does demand for funding from potential science performers imply a shortage of funding or a surfeit of performers?”
To answer these and related innovation policy questions we need agreement on:
• Terms & Conditions – Fed vs. private investment– Definition of “value”
• Level of Analysis– Project/program/agency/systems– U.S./International
• Resources vs. Expectations– Manhattan Project or After School Project– This year, 2010, or when fusion energy goes
commercial?
R&D Evaluation – Current State
Evaluation Types:
•Peer Review
•Output Metrics
ProjectsProjects ProgramsPrograms PortfoliosPortfolios OrganizationsOrganizations SystemsSystems<$5M
~$200M~$3B
~$25B
~$300B
Evaluation Types:
•Committees of Visitors
•Output & Outcome
Metrics•Case
Studies•Randomize
d Trials
Evaluation Types•National Academy
reviews•Econometric
Modeling•Committee Reviews•Case Studies
Evaluation Types:•Advisory
Committee reviews•Econometric
Modeling•Risk/Options
Modeling•Case Studies
Evaluation Types•NAS/COSEPUP
International Benchmarking
•Longitudinal Studies•Innovation Indexes•Case Studies
Evaluation Results– Current State
Evaluation Results:
•Cost•Schedule•Milestone
s•Data
Rates•Human
resources•Quality &
Relevance of project
ProjectsProjects ProgramsPrograms PortfoliosPortfolios OrganizationsOrganizations
Detector MIE at RHICTPC $5M
Nuclear Physics$400M
Office of Science$3.5B
U.S. R&D ~$312B
Evaluation Results:
•Characterize quark-gluon plasma
•New PhD’s produced
•Improve facility operations
Evaluation Results:•Achieve scientific
breakthroughs•Meet Administration
goals•Improve overall
management efficiency
•Advance energy efficiency
•Nobel Prizes
Evaluation Results:
•New scientific directions
•Meet DOE goals•Improve scientific
workforce•Increased
operational efficiencies
Evaluation Results:
•Increased Life Expectancy
•GDP Growth•New Knowledge•Increased National
Security•New Industries•Energy
Independence
Department of Energy
$23.4B
“Systems Level Analysis”
“Systems Level Analysis” Asks & Answers Fundamentally New
Questions:
Is it possible to compare the outcomes of one innovation system to another?
• Universities vs. National Labs?• Public vs. Private?• NSF vs. SC?• Applied vs. Basic R&D?• Japan vs. EU?
Would comparisons of performance of systems lead to:
• Management efficiencies?• New policies?• Greater funding for R&D?
Systems Level Outcomes
Examples include:• The 3% Solution• Regional Fallacies• Paper Chases• Predicting Prestige
(a.k.a. fortune telling)http://www.oecd.org/
http://www.compete.org/
Existing “Innovation Indexes” suffer from a host of problems, primarily a lack of context, causality, and comparability.
System Performance
Highly performing innovation systems should have the following attributes:
•Competition for Resources– (Money, Ideas, People, Facilities)
•An open market place for ideas– (Patents, Papers, Copyrights, IP)
•Resources sufficient for system growth– (People, Equipment, Money, Land, Energy)
•Checks & Balances– (Transparency, Multiple Funding Sources, External
Review)
An absence of any of these will seriously impair the effectiveness & efficiency of any innovation system.
Pre-Conditions for Systems Level Analysis
Before you can start, you need thefollowing:
1) Knowledge of who/what you are analyzing; i.e., the “system” that you are analyzing.
2) A review of past efforts.3) A working theory.4) An answer to the data challenge.5) New tools and methodologies.6) Partnerships to share the cost, set
comparable standards, and bring new ideas to the table.
Innovation Fits within a Dynamic System
Figure 1. National Innovation EcosystemFigure 1. National Innovation Ecosystem
TALENT•World Class Innovators•Adaptable Workforce
•Science & Engineering Skills•Magnet for Global Talent
Innovation Demand•Macro Demand
•Consumer•Business•Government
•National Priorities•Market Access•Industry Structure•Technology Diffusion•Standards•Profitability•Stock Valuation
INVESTMENT•Valuing long term innovation•Multiple disciplinary research
•Early stage investment•Service sector innovation
Innovation •Policy
•Strategy•Process•Insight
Accelerate level, quality, efficiency and profitability of US innovation
(overall success metrics)
Growth, Jobs, Standard of Living, Wealth, Comparative Advantage
INFRASTRUCTURE•World-class infrastructure•Innovative public sector
•Regulatory and legal system•21st Century IP system
Innovation Inputs•Creativity•Research•Knowledge •Information
Review of Past Efforts
We have worked for the past six years to develop a body of knowledgethat will inform our efforts:
• A review of theory, evaluation and management literature
• Partnerships with the private sector (IRI, Council on Competitiveness, Santa Fe Institute) and academia (GWU, USC, etc.).
• Ongoing interactions with the evaluation community through FedEval, AAAS, AEA, & WREN.
Theory-Based Analysis
An absence of theory hinders any effort to develop systems-level evaluations.We are developing:• Management/Economic/Social Science Theory
– What is the “net benefit” of public investment?– What differentiates Federal R&D agencies from
Bell Labs and Microsoft?– Is there such a thing as a “knowledge multiplier”?
• Evaluation Theory– What is the definition of “value”?– How can the innovation system be characterized?– What are the appropriate research questions?“All theory depends on assumptions which are not quite
true.” Robert Solow, 1957
Some Emerging Theories
We are beginning to see the emergence of a consensus on theoretical underpinnings that are relevant to Federal agencies.
• Management Theory– Gretchen Jordan’s “R&D Profiles” work.– DOE management benchmarking study.
• Social Science/Economic Theory– Science as a “co-evolving ecological community” (USC).– Models of knowledge flow (Ventana).
• Evaluation Theory– COSEPUP International Benchmarking Report– Jerry Hage’s “Innovation Systems” book.
Tackling the Data Challenge
The first problem confronting any attempt to do systems-level analysisis the absence of comparable data.There are at least five reasons:
1. R&D data is largely disaggregated.2. Aggregating R&D data is horrifically
expensive.3. Aggregating R&D data is horrifically intrusive.4. “Standard” definitions for R&D data do not
exist.5. Complexity is daunting.
Good Data is a Barrier
Complexity is Daunting– U.S. Economy is $12.2 Trillion, w/50 States &
3,066 counties.– Federal Budget is $2.6 Trillion, w/1,400 Programs– $764 Billion Global R&D investment. $312 Billion
U.S. R&D Investment - $132 Federal plus $180 Industry
– 3,700 degree-granting Colleges & Universities with 15.6 Million Students.
– 329,300 High Tech Establishments employ more than 5 Million High Tech Workers
– 4.7 Million Scientists, Engineers and Technicians – R&D data is typically found in journals, conference,
workshops, pre-print servers, and scientific databases
Sources: OMB FY06 Budget Request - Federal Budget, number of programs, U.S. Economy and Federal R&D American Association of Counties – U.S. counties OECD - Global R&D for 2003NSF Statistical Research Services - Industry R&D in 2003 (projected); U.S. Colleges, Universities and students for 2000, S&E workforce for 2003.AeA - High Tech “establishments” and High Tech Workers for 2003
Full U.S. Patent Office Database• Scientific Research Papers (Journal Papers,
Technical Reports, “Gray Literature,” Websites, Workshop Proceedings, etc.)
• Copyright submissions• International Data (OECD, EU, Japan, Korea, etc.)
Data being generated is huge:• 6 Million DOE records of papers, technical reports, etc.
since 1945• 17 Million hits annually on DOE scientific databases• 2.3 Million abstracts (65,000 new ones/year) from NIH• 670,000 scientific articles published annually in open
literature• 160,000 annual U.S. Patents (6.5 million total U.S.
Patents)
We are collecting vast quantities of information from extremely diverse sources/databases, including:
Data Mining/Data Visualization
IT tools that could be used for S&Tevaluation purposes have progressedover the past decade
• National Security concerns have sparked tremendous innovation in data mining/data visualization.
• Pacific Northwest National Laboratory is a leader in developing new tools.
• SC is exploring the use of these tools for a wide variety of applications and to solve the problem of “information overload.”
“Galaxy View” of 52,000 DOE Researchers (1976-present)
New Policy and Strategic Analysis Tools
The next generation of innovation evaluation tools will create opportunities fornew analyses to assess policy and strategic choices.
• Growth Accounting—economists will be able to better estimate the nation’s productivity performance in terms of contributing factors and outputs.
• Knowledge Economy—composite knowledge indicators will improve investment decisions for R&D, education and capital resources.
• Financial Reporting—financial reports could provide a balanced scorecard of physical as well as intangible assets.
• Valuation of Innovation—business executives and financial markets could better value R&D activity and related intangibles, estimate financial results, improve long term stock market valuations and predict outcomes.
• System Dynamics—expanding the range of “real-time” innovation metrics would help build more robust systems dynamics models and policy simulations. .
• General Purpose Technology (GPT) — improved analysis of the strategic contribution of GPT’s which set the stage for incremental innovation and have the inherent potential for pervasive application in a wide variety of industries.
• Tech-led Regional Development and Clusters—shift the emphasis from strengthening inputs to the innovation infrastructures toward improving the efficiency, rate and output of innovation.
New Tools & Methodologies
Because we are a mission agency, our focus is on tools that will help solve problems. Three general tools are being explored:
• Advanced Bibliometrics/Patent Analysis
• Network Analysis
• Modeling/Simulation
SC Patent Analysis Tool - Data
Abstracts and background data from 12,869 patents originating from work sponsored by DOE• 4,390 SC Patents
Abstracts and background data from 50,263 patents that cite DOE patents as prior art representing 82,737 citations• 27,699 citations of SC patents
1,231 Distinct organizations have attributed one or more of their patents to work sponsored by DOE• 453 organizations attribute to SC
13,345 Distinct organizations have cited one or more of DOE patents as prior art.• 4,172 organizations have cited SC
Distribution of SC-Sponsored Patents
Distribution of Citations to SC-sponsored Patents
Surgical Application of Pulsed Laser Technology
DOE holds 5 patents related to the surgical application of pulsed laser technology:
– 4,381,007, Multipolar corneal-shaping electrode with flexible removable skirt.
– 4,326,529, Corneal-shaping electrode. – 4,686,979, Excimer laser phototherapy for the dissolution of
abnormal growth. – 4,349,907, Broadly tunable picosecond IR source. – 5,720,894,: Ultrashort pulse high repetition rate laser system for
biological tissue processing.
Although most of the underlying research was originally conducted in the early to mid 1980's, these patents continue to generate broad interest within the medical community
To date, these patents have been cited in over 350 patents from some of the world's leading innovators in surgical equipment and techniques.
We are building upon work done by private industry that identifies undervalued companies and develops “non-economic” valuations of research portfolios.
• Technique examines the entire U.S. patent record in combination with scientific papers.
• “Hot” technology areas are identified.
• Individual organizational performance can be assessed and compared (Federal agencies, universities, companies, nations).
• Has the potential to be a predictive tool of performance.
“Technology Hotspots”
Federal Agency PerformancePercentage of Patents Citing Papers Funded by Different
Agencies that are in Hotspot Technologies
3.2%
2.5%
4.0%
4.3%
5.0%
0% 2% 4% 6%
All Agencies
NIH
NASA
NSF
DOE
Pap
er F
un
din
g S
ou
rce
Percentage of Citing Patents in Hotspots
“Successor Patents” Could Prove Predictive
0%
10%
20%
30%
40%
50%
60%
Any of 4Agencies
All Patents DOE NASA NSF NIH
Funding Agency
Percentage of Patents Citing Scientific Papers Funded by Different Government Agencies that Become Next Generation Patents
1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001
Coherent light generators
Superconductor technology
Drug, bio-affecting compositions
Coating apparatus
Power plants
Measuring and testing
Electricity - measuring and testing
Chemistry - electrical and wave energy
Chemistry of inorganic compounds
Stock materials
Catalysts and solid sorbents
Synthetic resins or natural rubbers
Semiconductor devices
Image analysis
Optics - systems
Surgery
Chemistry - electrical current
Chemical apparatu
Coating processes
Organic compounds
Optics - measuring and testing
Communications
Radiant energy
Chemistry - molecular and microbiology
University (Patents -- Citations)Boston University (14 -- 22)Brown University (18 -- 10)Cal Tech (30 -- 122)Columbia University (10 -- 14)Emory University (6 -- 8)Johns Hopkins University (10 -- 28)Michigan State University (21 -- 46)Northeastern University (6 -- 39)Northwestern University (37 -- 34)Princeton University (8 -- 37)Stanford University (42 -- 21)University of California (1316 -- 689)University of Dayton (20 -- 27)
DOE Support at Research Universities Has Produced Patents and Citations in a Wide Range of Areas*
University of Delaware (18 -- 21)University of Kentucky (5 -- 21)University of Michigan (14 -- 44)University of Minnesota (12 -- 35)University of Missouri (20 -- 19)University of New Mexico (10 -- 17)University of Pennsylvania (6 -- 19)University of Pittsburgh (5 -- 26)University of Rochester (9 -- 23)University of Texas (7 -- 102)University of Utah (13 -- 27)University of Washington (13 -- 35)Yale University (5 -- 16)
*Width of color band indicates relative number of patents in each classification.
Network Analysis – Tracing Ideas
Theory is that science is conducted through networks of people and their interactions are key to understanding the “value” of public investments.
Network analysis builds on:• “Value chain” analysis used by industry
• “Social Network Analysis”
• Simulation & modeling techniques
• Case study techniques
• Patent & paper analysis
Definition of Value/Outcomes
Focusing on “knowledge” as the key outcome of basic research. We are systematically tracking knowledge
outcomes and how they flow.
• Applied Math: First study looked at algorithms, software products, generic mathematical approaches, etc., of SC’s Applied Math Program.
• Nanoscience: Second study, just begun, will look at SC’s five nanocenters and nanoscience evolution as a discipline.
• European Union/FP6: Complementary study, done by European Union, examined how funding affected network formation in Europe.
Initial Results
First results indicate:1. It is possible to assign “value” to
knowledge as it travels through the ideas marketplace.
2. Factors influencing the transfer of knowledge can be identified.
3. We should be able to compare innovation systems and model their behavior.
4. Knowledge multipliers are real.
Emergence of the Term “Nano” in Open Literature*Showing Representative DOE Papers and Patents
*Terms with at least 10 occurrences in at least one year. Width of color band indicates relative number of occurrences‡ Papers identified by the Institute for Scientific Information as among the Top 25 Highly Cited Papers in Nanotechnology.
System Dynamics Modeling for a Dynamic System
What would the models look likeand what will they tell us?• System Dynamics Modeling is an alternative to
Optimization/Econometric Modeling.• Builds on work done by Jay Forrester, MIT, over
past 45 years.• Uses “soft data” and “system attributes” to
model non-linear systems, such as R&D and knowledge.
• Results are “scaleable” and can go from project to program to organization to system.
Where Are We Headed?
• European Commission is cooperating on joint research.
• WREN members are slowly coalescing around the problem.
• International workshops will lead to “bureaucrat-to-bureaucrat” cooperative frameworks.
• Private sector and academia are heavily engaged.
• OSTP will launch an effort.
We need help to leverage scarce resources…and we are working to get that help.