1
CSR and NIH Peer ReviewMaintaining Highest Quality & Fairness
René Etcheberrigaray, M.D.
Deputy Director, CSR
Health Research Alliance Members’ Meeting
September 18, 2014 - Durham, NC.
National Institutes of Health
We seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability.
2
National Institutes of Health
Much of the biomedical research in the United States is supported by the Federal Government, primarily the National Institutes of Health of Health (NIH)
Why Has NIH Been So Successful?
Peer Review
• Focus is on funding ideas not institutions.
• Ideas spring from local researchers.
• Researchers must compete – like entrepreneurs – for funding.
• Scientific experts do the judging.
• Institutions receive funds only when their scientists submit successful applications.
• NIH program and review staff are separated.
• Scientists manage the peer review process.
3
Spending at NIH
NIH Extramural & Intramural FundingFY 2012 Enacted: $30.9 Billion
83%
17%
Spending Outside NIH$25.7 B
– Supports over 300,000 Scientists & Research Personnel
– Supports over 2,500 Institutions
– $3.4 B Intramural Research – $1.5 B Research Management & Support – $0.3 B Buildings and Facilities, Other
$5.2 B
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Suc
cess
Rat
e
Fiscal Year
Grant Success RatesFY 1978-2013
* FY 2013 success rate is a preliminary estimate
4
NIH Program Level in Appropriated Dollars and Constant 1998 Dollars
0
20,000
40,000
60,000
80,000
100,000
120,000
App
licat
ions
Actual Fiscal Year
ARRA included
ARRA excluded
*2013 estimate based on applications received through July
Number of Applications Received by Fiscal Year
5
Scientific Research & Development Spending Change from 2012 to 2013
“To Maintain Our Edge . . . ”
“we’ve got to protect our rigorous peer review system and ensure that we only fund proposals that promise the biggest bang for taxpayer dollars . . . that’s what’s going to maintain our standards of scientific excellence for years to come.”
Remarks by the President on the 150th Anniversary of the National Academy of Sciences, April 29, 2013
6
Takes Final Action
National Institutes of Health
Center for Scientific Review
Study Section
Institute
Advisory Councils and Boards
Institute Director
Assigns to IC & IRG/Study Section
Reviews for Scientific Merit
Evaluates for Relevance
Recommends Action
ResearchGrant Application
School or OtherResearch Center
InitiatesResearch Idea
ConductsResearch
Allocates Funds
Submits Application
Review Process for a Research Grant
Core Values of NIH Peer Review
• Expert Assessment
• Transparency
• Impartiality
• Fairness
• Confidentiality
• Integrity and Ethical Conduct
7
Center for Scientific Review
To see that NIH grant applications receive fair, independent, expert, and timely reviews – free from inappropriate influences – so NIH can fund the most promising research.
CSR Mission
8
CSR Peer Review – Fiscal Year 2013
• 84,000 applications received by CSR
• 73% of NIH grant applications reviewed by CSR
• 173 standing study sections
• 236 Scientific Review Officers
• 1,500 review meetings
• 17,000 reviewers
Goals of CSR
• Improve continuously:
– Fairness of review
– Quality of review
– Efficiency of review
– Morale of staff and reviewers
• Create a science of peer review
9
Separation of Peer Review and Grant Award
NIH Peer Review System for Grant Applications
First Level of Review
Scientific Review Group (Study Section)
Second Level of ReviewNIH Institute/Center Council
About 80,000 applications and 18,000 reviewers
10
Scientific Review Officer
The overall peer review process is managed by a Designated Federal Official, who has doctoral-level scientific expertise relevant to the field
The Best Reviewers
11
Review by experts: Recruiting the Best ReviewersAcademic Rank of CSR Standing Reviewers
0%
10%
20%
30%
40%
50%
60%
70%
80%
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011
PROFESSOR ASSOCIATE PROFESSOR ASSISTANT PROFESSOR
Recruiting the Best Reviewers
o Move a meeting a year to the (west coast) other citieso Additional review platformso Expansion of no submission deadlines
o No submission deadlines for chartered members of study sections & frequent reviewers
o Provide flexible time for reviewerso Choice of 3 times/year for 4 years oro 2 times/year for 6 years
Year Regular Freq. Flyer Total Unique
12/2009 4799 1090 5889
12/2010 5068 3317 8385
12/2011 4978 1853 6831
12/2012 4941 1856 6797
12
The Study Section Meeting
The Study Section Meeting
Summary Statement
• Summary of Review Discussion• Essentially Unedited Critiques• Budget Recommendations• Administrative Notes• Priority Score and Percentile Ranking
Results are documented by SRO in a summarystatement and forwarded to the PI and the assigned NIH Institute or Center, where a funding decision is made.
The Summary Statement Contains
13
Confidentiality!
Review materials, application reviewers and the content of review are confidential!
• Protects you and applicant
• Do not discuss or share applications with anyone outside of the review meeting.
• If you need more expert advice to evaluate an application, discuss it with your SRO
• If an applicant asks you a question or wants to discuss review, ask them to talk to the SRO; after review they can talk to their assigned program officer.
14
Speed of Award
Days to 90% of Summary Statement
AIDS: 102
Non-AIDS (R01): 154
Days to 90% of Award
AIDS: 363
Non-AIDS (R01): 414
AIDS (All)
Non-AIDS (R01)
Group
SRG Assign+
Award
SS Release
Days to
IRG Assign
Meeting
15
Current Timeframe for R01s -- Submission to Award
Three Main Overlapping Cycles per Year
http://grants1.nih.gov/grants/funding/submissionschedule.htm
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Cycle I
Cycle II
Cycle III
AIDSAIDSAIDS
R&R Assign Review Meeting SS Council Award
Council Award R&R Assign Review Meeting SS
Review Meeting SS Council Award R&R Assign
Fairness in peer review
16
Probability of NIH R01 award by race and ethnicity, FY 2000 to FY 2006 (N = 83,188)
D K Ginther et al. Science 2011;333:1015-1019
Concerns of Reviewer Bias
Award Probabilities by NIH Funding Rank and Race/Ethnicity
17
Ginther et al. (2011) Findings
• African American applicants were 10 percentage points less likely to receive NIH research funding compared to Whites
• Suggested explanations:
– Bias in peer review
– Deficits in applicants’ grant writing abilities
• Applications with strong priority scores were equally likely to be funded regardless of race
− This suggests that problems occur at the peer review stage or earlier
NIH Response
ACD Working Group on Diversity in the Biomedical Research Workforce met throughout 2012 and developed recommendations
• Create a subcommittee on peer review
• Research to examine causes of award disparities
– Implicit Bias
– Cumulative advantage or disadvantage
• Exploration of interventions to reduce possibility of bias
18
CSR Early Career Reviewer Program Goals
• Train and educate qualified scientists to become critical, competent and fair reviewers
• Provide peer review experience to help improve applicant competitiveness
• Enrich the existing pool of NIH reviewers
Early Career Reviewer Program (ECR)
3524 ECRs have been accepted into the program
1069 have served on at least one study section
30 percent of ECRs who have served are from under-represented groups
Software developed to allow for online applications
ECR video for outreach
Outreach webinars for R15 schools
19
How to Apply for the Early Career Reviewer Program
• Instructions are at www.csr.nih.gov/ECR
• If eligible, your name will be placed into our ECR database
• You will be invited to serve as an ECR when your expertise is needed for particular applications
Methods for strengthening reviewer training methods to maximize fairness and impartiality in peer review with regards to gender, race/ethnicity, institutional affiliation, area of science, and/or amount of research experience of applicants.
Launched on May 5, 2014. Submission deadline June 30, 2014 Awards September 30, 2014
• Prizes
First - $10,000
Second - $5,000
Challenge #2 - Strategies to Strengthen Fairness and Impartiality in Peer Review
20
Detection of Bias:Text Analysis of Reviewer Comments
Task 1: Development of lexicon/thesaurus for text analysis
Task 2: Test thesaurus for ability to detect text differences by applicant race and gender
Task 3: Full Scale study of bias in reviewer comments due to investigator characteristics
Contract to be awarded Summer 2014
Bias Detection:Alter Identity of Grant Applicants
• Grant applications from White Males will be altered by:
– Gender (male vs. female)
– Race (black vs. white)
– Institutional affiliation (low vs. high research intensive)
• Comparison of scores
• Analysis of written and verbal comments of reviewers
21
America COMPETES Challenges
Challenge #1 - New Methods to Detect bias in Peer Review
How to detect bias among reviewers due to gender, race/ethnicity, institutional affiliation, area of science, and/or amount of research experience of applicants.
Launched on May 5, 2014 Submission deadline June 30, 2014 Awards September 30, 2014
Judging criteria Best Empirically Based Submission Most Creative Submission
Prizes First - $10,000 Second - $5,000
Investigator Perceptions: Survey of New Investigators and Focus Groups
Survey launched May 28, 2014
Topics
Perception of fairness of NIH peer review
Stereotype threat
Self-Efficacy for Grant Writing
Access to mentors
Institutional supports and resources
Interaction with NIH
Response to critiques
22
Pilots and Studies
• Post meeting survey pilot
• More intra-IRG ranking studies
• Innovation and conformity study
• Post meeting ranking pilot
• Anonymization pilot
• Reliability
• Replicability
This Is CSR
23
Key NIH Review and Grants Web Sites
NIH Center for Scientific Review
http://www.csr.nih.gov
NIH Office of Extramural Research
http://grants.nih.gov/
Overall Impact of Research Applications