CSR Advisory Council Meeting

May 19, 2014

Editorial Board ReviewA Few Good Reviewers

Don Schneider, Ph.D.

• Reviewer pools

• Past experience

• Cost considerations

Toward Review by the Best

Reviewer Status Numbers

HHMI 300

NIH R37 700

NAS 2,000

R01 26,000

CSR needs 16,000

Reviewer Pools

• Format modeled on journal manuscript review

• First piloted in 2008 with 6 SBIR panels, just in time for TR01s, Challenge grants, DP1 etc.

• Two stages– First Stage – Mail reviewers– Second Stage – Editors

Past Experience

First Stage/Mail Reviewers

• Subject matter experts– Provide depth in review

• Focus on scientific and technical merit• 2-3 first stage mail reviewers per application• Submit full critiques• Give overall impact and criterion scores

– Overall impact scores not factored into final priority score

• Hold face-to-face meeting• Recruit broad experts

– Provide perspective in review (assign about 15 applications each)

• Focus on impact and significance• Assign 3 second stage reviewers per application• Consider first stage critiques in review• Write overall impact paragraph• Give overall impact score

– Final priority score based on second stage only

Second Stage/Editors

• Provides both depth and breadth in review

• Optimizes use of the best reviewers

• Scales well for large numbers of applications(second stage discusses a fraction of the applications)

Rationale

Perceived Advantages

• Involves no travel/teleconference for first stage reviewers

• Allows small, interactive face-to-face meetings

• Promotes better scoring and assessment of impact

• Lessens travel and lodging expenses and inconveniences

Review # of Applications Cost/application

Regular R01 F2F $518

DP1/Pioneer EB+I 244 $280

DP2/New Innov EB 593 $124

DP5/Early Indep EB+I 84 $875

Cost Considerations(Alicia Caffi)

• Recruitment of large numbers of reviewers

• Timeline– Tight, two sequential reviews – (in the 17 week cycle)

• More staff time required (SROs)

• Some sense of isolation by first stage reviewers

Challenges

• Each application examined by at least 5 reviewers

• Interactive, thoughtful discussions

• Overall scoring by second stage members

• Reviewers and staff like final review products

Review Outcomes

• Survey conducted by A Kopstein of reviewers participating in SBIR pilots 2008

• Outcomes were generally positive

– Majority willing to participate in either review stage in future

– Editorial Board Review:• Increases expert review

» 3/4ths of respondents• Preferred for their own applications

» 2/3rds of respondents

Survey

Hopes for a few good reviewers?

Discussion