Center for Scientific ReviewNational Institutes of HealthDepartment of Health and Human Services

Toni Scarpa

NIH Peer Review: Continuity and Change

NIDA CouncilBethesda, Feb 8, 2006

Peer Review: An N.I.H. “Conception”

• Is the heart and soul of NIH• Has produced an effective partnership between the

federal government and research institutions• Has created the best academic medical centers, the best

biomedical/behavioral research and biotechnology• Has made possible the best cures and the best prevention• Has been admired and imitated here and abroad• Has protected NIH against outside influence

Center for Scientific Review

This Is Not Amazon.com

This is CSR

0

20,000

40,000

60,000

80,000

1998 1999 2000 2001 2002 2003 2004 2005

Fiscal Year

Nu

mb

er

of

Ap

plic

ati

on

sApplications Received for all of NIH and Applications

Referred for CSR Review, FY 1998 - 2005

NIH Applications

Applications for CSR Review

CSR Mission Statement

To see that NIH grant applications receive fair, independent, expert, and timely reviews -- free from inappropriate influences -- so NIH can fund the most promising research.

NIDA Applications Reviewed by CSR in FY 2002 and 2005

2002 2005Number of applications reviewed by CSR (for all ICs) 33,469 51,690

Number of NIDA applications reviewed by CSRPercent of total reviewed by CSR

969(2.9%)

1,480(2.9%)

Number of CSR study section meetings 1,387 1,798

Number of CSR study sections meetings that reviewed NIDA applications(Percent of total)

225(16.2%)

30416.9%

Fiscal Year of Review

Review Outcomes for NIDA Applications in 2002 and 2005

R01 Applications Only

FY N%

Unscored

Median Priority Scores

% Scoring at the 10th percentile or

better

% Scoring at the 20th percentile or

better

2002 654 32.4% 244.0 7.8% 18.6%

2005 948 41.1%* 239.0 9.8% 20.5%

* FY comparison, p < .001

Time

C

om

ple

xity

an

d I

mp

act

CSR Operations

C

urrent S

ystems

New Systems?

Necessary Changes in CSR Peer Review Operations

• Increase communications between CSR, the ICs, our reviewers and applicants

• Increase uniformity

• Increase efficiency

• Facilitate work of IC program staff

Changes in CSR Operations

Increased Communication and Transparency

• Within CSR• With NIH and other Agencies• With the Scientific Community

Changes in CSR Operations 1

Increase uniformity

Slate Nomination

Summary Statements• Posting all within one month of Study Section meetings• Posting Summary Statements of new investigators within

one week• Producing more complete and structured resumes

Unscoring• Common practice• Unscoring 50%

Changes in CSR Operations 2

Increase Efficiency

• Electronic Submission

• Text Fingerprinting, Artificial Intelligence Software

Changes in CSR Operations 3

Potential of Knowledge Management Tools for Peer Review

Collexis Software or Others

• Knowledge management solutions• Fingerprinting and text retrieving• Disease coding

Benefits for Peer Review

• Assigning applications to Integrated Review Groups or Study Sections

• Selecting reviewers (one application, multiple applications)

Nine pilots are underway to begin to assess these benefits

• Facilitate work of IC program staff

Possible Changes in CSR Operations

Study Section Realignment

• Review of one IRG every month• Total review every 2 years

Required Changes in Current Systems

• Shorten the review cycle

This is Not an Ford Assembly Line

Receipt ReferEvaluate Scientific

Merit of Applications

EnterpriseArchitecture@mail.nih.gov

Shortening the NIH Review Cycle, Initial Steps

For most research grants, we are posting summary statements within one month after the study section meeting instead of two to three months after the meeting (effective Oct 05)

We are conducting a pilot study to speed the review process for new investigators so they may revise and resubmit for the very next review cycle 4 months earlier than before (effective Feb 06)

Possible Changes in Current Systems

• Shorten the review cycle

• Address concern that clinical research is not properly evaluated

• Improve the assessment of innovative, high- risk/high-reward research

• Do more to recruit and retain more high-quality reviewers

Expanding Peer Review’s Platforms

Electronic Reviews• Telephone Enhanced Discussions• Video Enhanced Discussions• Asynchronous Electronic Discussions

Study Sections

Necessity ● Clinical reviewers Preference ● Physicists, computational biologists

New Opportunities ● Fogarty, International Reviewers

Applications Received for All of NIH FY 1998-2004

0

20,000

40,000

60,000

80,000

1998 2000 2002 2004Fiscal year

Nu

mb

er o

f ap

pli

cati

on

s

Number of Research Grant Applications/Applicant

1.05

1.1

1.15

1.2

1.25

1.3

1.35

1.4

1.45

1980

1982

1984

1986

1988

1990

1992

1994

1996

1998

2000

2002

2004

CSR Applications Reviewed, Regular and SEP May Council Only

0

2000

4000

6000

8000

10000

12000

14000

16000

1999 2000 2001 2002 2003 2004 2005

Applications Regular

Applications SEP

Study Section Application/Reviewer Ratio October Council Only

0

0.5

1

1.5

2

2.5

3

3.5

4

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Regular

SEP

CENTER FOR SCIENTIFIC REVIEW FY 2004-2006 Non-Discretionary vs. Discretionary Spending

■ Non-Discretionary

■ Discretionary

FY 2004 FY 2005 FY 2006

77%

23%18%

82% 88%

12%

If we didn’t have any peer-review system and we had to design one from scratch, what would it look like?

Possible New Systems

This is CSR