Peer Review of ARS Research Projects

Peer Reviewof ARS Research Projects

Peer Reviewof ARS Research Projects

Office of Scientific Quality ReviewSteven Huber, Scientific Quality Review

Officer

A Brief Overview of the Process

OSQR Web Sitehttp://www.osqr.ars.usda.gov

OSQR Web Sitehttp://www.osqr.ars.usda.gov

National ProgramsSchedules of ReviewsForms and Resources (TIPS!)Reviewer InformationPeer Review statusManual (online comments soon)Staff

Topics to CoverTopics to Cover

BackgroundRole of the Research LeaderThe Prospectus & Project PlanGeneral writing tipsThe panel processOutcomes

New Mandate for this processNew Mandate for this process

The Agricultural Research, Extension and Education Reform Act of 1998 (public law 105-185):

ARS projects be evaluated by panels composed predominantly of non-ARS scientists

With oversight from the REE Advisory Committee

Intent of Peer ReviewIntent of Peer Review

To obtain constructive, independent, expert feedback and advice on ways to improve the scientific and technical merit of each project plan

NOT funding issues

Fundamental QuestionsFundamental Questions

Panelists are asked to evaluate:

MERIT AND SIGNIFICANCE (Relevance to NP) APPROACHES AND PROCEDURES LIKELIHOOD OF SUCCESS

(feasibility)

Suggestions for improvements or solutions to problems are solicited

Steps in the ProcessSteps in the Process

RL and Scientists develop Prospectus

RLs Prepare Project Plan with LSsSubmit Plan for approval

Peer Review

Response and RevisionSubmit revised project plan

Re-review?

NPS

AD

OSQR

AD/NPS

OSQR

16 wk(24 wk)

18 wk(14 wk)

6 wk

6 wk

6 wk

Scientist’s ResponsibilitiesScientist’s Responsibilities

• Communicate with the NP Team.• Scope of the project.• Coordination with other projects.

• QUALITY of the Project Plan.• Many are ‘poorly’ written.

Pro

ject

Pla

n T

em

pla

tePro

ject

Pla

n T

em

pla

te Title and Investigator(s) TOC (coming soon!) Project summary Objectives (< ½ p) Need for research (1p) Scientific Background (5-6 p) Approach & Procedures (6-12 p) Milestones & Outcomes ( 1 p) Literature Cited Past Accomplishments Issues of Concern statements Appendices

15 - 25 pages

Scientific Background Scientific Background

Focus on the pertinent literature

Show linkage/coordination with other similar projects in the NP (or other NPs) and with minority-coded projects in the MU.

GENERAL TIPGENERAL TIP

Use illustrations (figures, schemes, etc) to enhance the Plan. (Up to 2 pages does not count against page limit).

Preliminary dataHow your project fits with othersWorking modelsExperimental design/treatments

Approach and Procedures—TipsApproach and Procedures—Tips

Lack of necessary detail is the most common criticism.

Tell what will be done, by whom, and what will result. Identify staff involved with key portions of the project.

If an objective is supported by a grant, mention that.

The PanelThe Panel

PanelistsPrimary reviewersSecondary reviewers

Panel Chair

Ad-hoc reviewers may be used

OSQR Officer

Provides orientation ObservesDebriefsReceives results

Panel Chair SelectionPanel Chair Selection

SUGGESTIONSfor Non-ARS Panel Chair

from NPS, ADs, etc.

OSQR OfficerSelects Chair

OSQR OfficerConsults inside

and outsideARS

Panelist SelectionPanelist Selection

SUGGESTIONS for Panelistsfrom NPS, ADs, RLs, Lead Scientists, University Scientists, Customers, etc.

OSQR OfficerApproves Panel*

Panel Chair (PC)Makes

recommendations toOSQR Officer

*Based on PC recommendations and diversity requirements

Suggest

ed P

anelis

tsSuggest

ed P

anelis

tsSUGGESTED PANEL MEMBER

Full Name of Prospective Panelist:

Full Title:

Business Address:

Tele: FAX: Email:

Scientific Area(s) of Expertise:

National Program(s):

101 Animal Genomes, Germplasm, Reprod. 205 Grazinglands Management 102 Animal Production Systems 206 Manure and Byproduct Utilization 103 Animal Health 207 Integrated Farming Systems 104 Arthropod Pests of Animals & Humans 301 Plant Microbial & Insect Germplasm... 105 Animal Well-Being & Stress Contr. Syst. 302 Plant Biological & Molecular Processes 106 Aquaculture 303 Plant Diseases 107 Human Nutrition Reqs., Food Cons... 304 Crop & Commodity Pest Biology... 108 Food Safety 305 Integrated Crop Production & Protection 201 Water Quality 306 New Uses, Quality, & Marketability of 202 Soil Quality & Management Plant & Animal Products 203 Air Quality 307 Bioenergy & Energy Alternatives 204 Global Change 308 Methyl Bromide Alternatives

Please provide further pertinent information on this prospect (such as prior panels served, etc.):

PICReturn this form with your Project Outline to your National Program Team LeaderNPLCForward this form with the Project Outline to the OSQRADCForward this form to the OSQR

Use this form!

The Panel Meeting Process

Panel Discussion

Primary reviewer presents the plan input from

secondary reviewer, panelists, and ad hoc reviewer(s).

Panelists and Panel Chair assign an action class to the project.

Turn into SQR Officer

Primary reviewer modifies the review recommendation on disk

1 hour

On breaks, evenings

Panel Chair Approval

Turn into SQR Officer

Action Classes*Action Classes*

Each panelist individually provides overall ‘Action Class’ assignment: No revision required. Minor revision might be made.

Minor revision required. Objectives fit the NP action plan; approaches to all objectives are sound. Project is feasible.

Moderate revision required. Moderate revision of an objective and/or one approach needed. Project is feasible.

Major revision required. Should be sound and feasible after major revision. (May be re-reviewed).

Not feasible. Deficiencies in expertise or facilities or major flaw.

*Action Class Matrix will be provided

After the Panel MeetingAfter the Panel Meeting

If the composite Action Class is: No revision required Minor revision required Moderate revision required

Major revision required

Not feasible

Scientists solicit comments and revise the Plan within 6 weeks

Revised Plans will be revised AND re-reviewed

Revise for immediate re-review; delay revision until major resource changes (e.g., personnel) are made; or terminate

Responding to the Review Recommendations

Project Title:_____________________________________________________________

CRIS #_______________________ National Program__________________________

Lead Scientist:_________________________________ Reviewer #_________________

PEER REVIEW OF ARS RESEARCH PROJECT

2. Adequacy of Approach and Procedures:

Please comment on the following: Are the hypotheses and/or plan of work well conceived? Are the experiments, analytical methods, and approaches and procedures appropriate and sufficient to accomplish the objectives? How could the approach or research procedures be improved?

Please list and number each significant recommendation being made. Be sure to briefly state the rational or basis for suggestions made or questions raised. Each recommendation can include several specific questions you believe should be addressed by the lead scientist.

1. Objective 1--The hypothesis being tested is only one of several that should be considered. For example……

ARS Response:

ARS Responses to Review Recommendations

ARS Responses to Review Recommendations

ARS scientists must respond to each major recommendation.

• TONE of the response: should be receptive, not defensive or condescending.

•CONTENT: make all reasonable efforts to accommodate suggestions made. Lack of adoption must be justified. Should ‘stand alone.’

The completed forms go back to the panelists!

‘Re-Review’ of Project Plans‘Re-Review’ of Project Plans

Purpose:

1) independent evaluation of response, and

2) maintain credibility with community.

NP108, NP302, NP202 Panel StatisticsNP108, NP302, NP202 Panel Statistics

7 Panel Chairs and 44 Panelists (51 total)

1 ARS employee 33 employed by universities 7 employed by private industry, trades groups, or

self-employed 10 Other federal agencies (U.S. and Canada)

NP108, NP302, NP202 Panel results

NP108, NP302, NP202 Panel results

0

10

20

30

40

50

NO MIN MOD MAJ NOT

Action Class

Num

ber

of P

roje

cts

Comments from PanelistsComments from Panelists

Think the peer review process is important for the agency

Appreciated the positive approach of the ARS peer review process.

Enjoyed the experience (opportunity for them to discuss science with peers).

Hope the review recommendations will be useful and incorporated.

Benefits of Peer ReviewBenefits of Peer Review

Better short- and long-term thinking and planning of research (if done properly!).

Increased communication among scientists, ADs and NPLs.

Increased collaboration (especially among ARS labs).

Increased quality—perspectives from peer reviewers.

Greater ability to compete for competitive funds.

OSQR Web Sitehttp://osqr.ars.usda.gov/

OSQR Web Sitehttp://osqr.ars.usda.gov/

• National Programs

• Schedules of Reviews

• Forms and Resources

• Reviewer Information

• Peer Review Status

• Manual

• Staff