ONS Foundation

Dissertation &Research Grant

REVIEWER ORIENTATION

Assigned Applications• Reviewers will “identify” the applications they would be able to review, based

on the match with their content and/or methods expertise, identifying any

conflicts of interest. The total number of applications per reviewer will

depend on the number of applications received. A Reviewer Proposal

Identification Worksheet, will be submitted by each reviewer with the

proposals identified for their review. The completed worksheet will be sent to

the ONS Foundation Research Department.

• The chair will finalize all assignments, ensuring that each application has a

primary, secondary and collateral reviewer.

• Assigned applications will be reviewed through a link to the online review

area.

• All components of the assigned applications should be read

• Major strengths and weaknesses should be identified for each criteria, using

the Reviewer Comment-Scoring Form

• A preliminary overall impact score is to be assigned as well as scores to

each of the five core criteria on the Reviewer Comment-Scoring form

Reviewer Comment-Scoring Form • A Reviewer Comment-Scoring Form will be used by the three

assigned reviewers (primary, secondary and collateral) for each

application

• Emails will be sent with the link to the online review website, the

reviewer’s passwords and other review materials needed.

• All reviewers should use the Reviewer Comment-Scoring form

when reviewing the applications assigned.

• The completed Reviewer Comment-Scoring Forms are to be

emailed to the ONS Foundation Research Department by a

designated date. These forms must be submitted on time as

they will be combined and emailed to the review team in

preparation of the review call.

Preparation of Critique Comments

• Use bulleted points to make succinct, focused

comments

• Short narratives may occasionally be appropriate,

but should be rare

• Focus on major strengths and weaknesses (ones

that impacted your overall rating of the application)

• Limit text to ¼ page per criterion

Reviewer Comment-Scoring

Form Layout

Criteria – Description of scoring criteria (i.e., significance,

investigators, etc.)

Score: ___

Strengths:

- Bulleted Comments

Weaknesses:

- Bulleted Comments

Scoring DescriptionsImpact Score Descriptor Additional Guidance on Strengths/Weaknesses

High

1 Exceptional Exceptionally strong with essentially no weaknesses

2 Outstanding Extremely strong with negligible weaknesses

3 Excellent Very strong with only some minor weaknesses

Medium

4 Very Good Strong but with numerous minor weaknesses

5 Good Strong but with at least one moderate weakness

6 Satisfactory Some strengths but also some moderate weaknesses

Low

7 Fair Some strengths but with at least one major weakness

8 Marginal A few strengths and a few major weaknesses

9 Poor Very few strengths and numerous major weaknesses

Minor Weakness: An easily addressable weakness that does not substantially lessen impact

Moderate Weakness: A weakness that lessens impact

Major Weakness: A weakness that severely limits impact

Scoring Criteria1. Significance - Does the project address an important problem or a

critical barrier to progress in the field? If the aims of the project are

achieved, how will scientific knowledge, technical capability, and/or

clinical practice be improved? How will successful completion of the

aims change the concepts, methods, technologies, treatments, services,

or preventative interventions that drive this field?

2. Investigator(s) - Are the PIs, collaborators, and other researchers well

suited to the project? If Early Stage Investigators or New Investigators,

or in the early stages of independent careers, do they have appropriate

experience and training? If established, have they demonstrated an

ongoing record of accomplishments that have advanced their field(s)? If

the project is collaborative or multi-PD/PI, do the investigators have

complementary and integrated expertise; are their leadership approach,

governance and organizational structure appropriate for the project?

Scoring Criteria (Continued)3. Innovation - Does the application challenge and seek to

shift current research or clinical practice paradigms by

utilizing novel theoretical concepts, approaches or

methodologies, instrumentation, or interventions? Are

the concepts, approaches or methodologies,

instrumentation, or interventions novel to one field of

research or novel in a broad sense? Is a refinement,

improvement, or new application of theoretical

concepts, approaches or methodologies,

instrumentation, or interventions proposed?

Scoring Criteria (Continued)4. Approach - Are the overall purpose, aims, strategy, methodology,

and analyses well-reasoned and appropriate to accomplish the

specific aims of the project? Are potential problems, alternative

strategies, and benchmarks for success presented? If the project is

in the early stages of development, will the strategy establish

feasibility and will particularly risky aspects be managed?

If the project involves clinical research, are the plans for

1) protection of human subjects from research risks, and 2)

inclusion of minorities and members of both sexes/genders, as well

as the inclusion of children, justified in terms of the scientific goals

and research strategy proposed?

Scoring Criteria (Continued)

5. Environment - Will the scientific environment in which the

work will be done contribute to the probability of success?

Are the institutional support, equipment and other physical

resources available to the investigators adequate for the

project proposed? Will the project benefit from unique

features of the scientific environment, subject populations,

or collaborative arrangements?

Where criteria are addressed

• Significance: Project narrative

• Investigator(s): Biographical sketch(es)

• Innovation: Innovation field, project

narrative and overall

• Environment: Facilities and resources

• Approach• Project narrative

• Protection of human subjects or animals

• Women and minority inclusion

• References

• Timetable

• Letters of support

• Instruments

• Consent form

• Miscellaneous field

• Project NarrativeLength: >$25,000 (12 pages) / $25,000 or less (6 pages)

• Purpose and specific aims

• Significance, framework and review of literature

• Preliminary work

• Methods and design

Design

Sample and settings

Experimental variables, if applicable

Instruments

Data collection schedule and procedures

Data analysis and interpretation

Additional Review Criteria

• As applicable, reviewers will consider additional items in

the determination of scientific and technical merit, but will

not give separate scores for these items.

• Responses for Protections for Human Subjects and/or

Vertebrate Animals are required for all applications

• A response for Inclusion of Women, Minorities and

Children is required for applications proposing Human

Subjects Research.

Additional Review Criteria

Protection for Human Subjects

Protections for Human Subjects – Did the application

describe how informed consent will be obtained and the

steps taken to protect participants’ rights or the welfare of

animals? Did the application identify any potential risks

associated with participation in the project?

Comments –

Data and Safety Monitoring Plan (Clinical trials only)

Comments -

Additional Review Criteria

Inclusion of Women, Minorities and Children

Inclusion of Women, Minorities and Children (Applicable Only for

Human Subjects Research) – Did the application address the inclusion

of women, minorities and children in developing a research design

appropriate to the scientific objectives of the study. Inclusion is

required unless a clear and compelling rationale shows that inclusion is

inappropriate with the respect to the health of the subjects or that

inclusion is inappropriate for the purpose of the study. Did the

application provide information on the composition of the proposed

study population in terms of sex/gender and racial/ethnic group and

provide a rationale for selection of such subjects in terms of the

scientific objectives and proposed study design.

Comments -

Additional Review Criteria

Vertebrate AnimalsVertebrate Animals – Did the application address the involvement of

live vertebrate animals as part of the scientific assessment according to

the following five points: 1) proposed use of the animals, and species,

strains, ages, sex, and numbers to be used; 2) justifications for the use

of animals and for the appropriateness of the species and numbers

proposed; 3) adequacy of veterinary care; 4) procedures for limiting

discomfort, distress, pain and injury to that which is unavoidable in the

conduct of scientifically sound research including the use of analgesic,

anesthetic, and tranquilizing drugs and/or comfortable restraining

devices; and 5) methods of euthanasia and reason for selection if not

consistent with the AVMA Guidelines on Euthanasia.

Comments (Required unless not applicable)

Additional Criteria

Budget and Period of Support(Not part of the scientific review)

Budget and Period of Support – Is the budget and the

requested period of support fully justified and reasonable in

relation to the proposed research. For more details, please

see Budget Information on the application.

Recommended budget modifications or possible overlap

identification -

Overall Impact Reviewers will provide an overall impact score to reflect their assessment of the

likelihood for the project to exert a sustained, powerful influence on the research

field(s) including consideration of the following five scored review criteria, and

additional review criteria. An application does not need to be strong in all categories

to be judged likely to have major scientific impact.Proposal #: _____

Principal Investigator: ________________

Reviewers will provide an overall impact score to reflect their assessment of the likelihood for the project to exert a

sustained, powerful influence on the research field(s) involved in consideration of the following five scored review criteria,

and additional review criteria. An application does not need to be strong in all categories to be judged likely to have major

scientific impact.

Overall Impact - After considering all of the review criteria, briefly summarize the significant

strengths and weaknesses of the application and state the likelihood of the project to exert a

sustained powerful influence on the field.

Preliminary Score: _____

Final Score: _____

Strengths –

Weaknesses -

Overall Impact ScoringImpact Score Descriptor Additional Guidance on Strengths/Weaknesses

High

1 Exceptional Exceptionally strong with essentially no weaknesses

2 Outstanding Extremely strong with negligible weaknesses

3 Excellent Very strong with only some minor weaknesses

Medium

4 Very Good Strong but with numerous minor weaknesses

5 Good Strong but with at least one moderate weakness

6 Satisfactory Some strengths but also some moderate weaknesses

Low

7 Fair Some strengths but with at least one major weakness

8 Marginal A few strengths and a few major weaknesses

9 Poor Very few strengths and numerous major weaknesses

Minor Weakness: An easily addressable weakness that does not substantially lessen impact

Moderate Weakness: A weakness that lessens impact

Major Weakness: A weakness that severely limits impact

Additional Comments to Applicant

Reviewers may provide guidance to the applicant or

recommend against resubmission without fundamental

revision.

Additional Comments to Applicant (Optional) – Please

provide any additional guidance to the application or recommend

against resubmission without fundamental revision

Online Overall Impact Scores

• After completing your review of each

application you will need to enter a

preliminary impact score in the online

review system by clicking on one of

the numerical ratings at the bottom of

each application reviewed

• Once you have selected your

preliminary impact score, click

“Submit Ratings” to finalize your

score

My Application Score

0 / 90.0

My Ratings

Overall Impact Score1 = Exceptional (High impact)

2 = Outstanding (High impact)

3 = Excellent (High impact)

4 = Very Good (Medium impact)

5 = Good (Medium impact)

6 = Satisfactory (Medium impact)

7 = Fair (Low impact)

8 = Marginal (Low impact)

9 = Poor (Low impact)

1 2 3 4 5 6 7 8 9

O O O O O O O O O

Prior to Review Conference Call• Completed worksheets are to be emailed to

research@onsfoundation.org by the designated due date

• Primary, secondary and collateral comment-scoring forms are

combined by the ONS Foundation Research Department for each

application

• The combined forms for the applications each reviewer has been

assigned to review are emailed several days before the call for your

preparation for the call

• Primary, secondary and collateral preliminary impact scores are

averaged and the Preliminary Mean Scores are ranked and emailed

to all reviewers

• The Grant Review Team Chair contacts any reviewers with variant

scores and determines any proposals that are non-competitive and do

not need to be discussed

ONS Foundation

Dissertation and Research

Grant Program

ONLINE Review

Instructions

Logging In with the Temporary Password Provided

• Access the website (https://admin.closerware.net/gm_onsf/

page.jsp?pagename=evaluatorarea )

• Enter the username

provided in the email

• Enter the password

provided in the email• (please do not change the

password provided)

• Click “Login”

“My Applications”

• All of the applications you

have been assigned to review

will appear below the

“Applications Pending your

Review” area

• Click on the applicant’s name

to view the application

• Applications will remain in this

area until a score is assigned

My

Applications

Pending

Applications

Approved

Applications

Research Grants Program

(RE##): Type of Grant

Applicant Name (Username)

Applicant Name (Username)

Applicant Name (Username)

Admin

© Copyright 2003-2010 Closerware, LLC. All rights reserved.

Evaluator Extranet

Applications Pending

Your Review

Applications you have

reviewed

“Pending Applications”

• As you complete your review

of each application and the

score is assigned, the

application will move to the

“Pending Applications” area

• To view these applications

again, simply click on the

applicant name in that area to

access that application again,

as needed

My

Applications

Pending

Applications

Approved

Applications

Applicant Name (Username) Research Grants Program

(RE##): Type of Grant

Admin

© Copyright 2003-2010 Closerware, LLC. All rights reserved.

Evaluator Extranet

Application Label Grant: Grant Opportunity

Accessing Applications• The applicant name

and application

number will appear in

the blue area followed

by:

– Type of Grant

– User Queue Number

– Application Status

(Pending)

– Research Team

Evaluator Extranet

•Grant Opportunity Main >

•Grant Application Main >

•Grant Application

* Administrative Index

* Grant Opportunity Main

* Grant Application Main

* Grant Application

Grant Application

Created: 9/6/10 5:29:36 PM EDT

Grant Opportunity: Research Grants Program (RE##): Type of Grant

User Queue: #1

Application Status: Pending

Research Grants: Research Team

Research Team: Name Credentials Institution Res Team Role

Name Credentials Institution PI

Name Credentials Institution Co-investigator

Applicant Name (Username)

Conflict of Interest Review

• Check the names shown in the “Research Team”

area of the Applicant Login Worksheet to see if you

have any conflicts of interest with anyone on the

team or their institution

• If you identify anyone or any institution that you may

be perceived as having or have a conflict with,

notify research@onsfoundation.org immediately

Uploaded Documents

Abstract: Abstract.pdf

Project Narrative: Project Narrative.pdf

Reference List: Reference List.pdf

Timetable: Timetable.pdf

Human Protection

Education Certificates

of Completion: Certificates.pdf

Support Letters: Letters of Support.pdf

Biographical Sketches: Biosketches.pdf

Instruments: Instruments.pdf

Itemized Budget: Budget Worksheet.pdf

Budget Justification: Justification.pdf

• Documents uploaded by the

applicant will appear as links

which can be opened and either

reviewed online or printed

• The uploads will be in PDF

formats

• The system will not allow the

applicant to upload more than

one document in any area (i.e.,

letters of support, biosketches,

instruments, etc.) so the

applicants have been asked to

scan the multiple documents into

one document and upload in a

PDF format.

Resubmission / Special Grants

Is this application a resubmission

from a previous ONS Foundation

research grant cycle:– Year of Grant

– Type of previous grant

• Resubmission – from a previously

non-funded ONS Foundation

research grant cycle - year and type

of grant will be shown

– Resubmission Cover Letter-uploaded

– The previous reviewer’s critiques will

have been added to the end of the

resubmission cover letter by the

Research Department.

• Special Grants ($25,000 or less

Only) – Applicant will indicate if they are

applying for one of the ONS Foundation special

grants

Special Grants ($25,000 or less Only):

– ONS/STTI Research Grant

– Trish Greene Pain Research Grant

– Certification, Education and Outcomes

Grant

Project Title and PI Information

Title of Project:

PI First Name:

PI Last Name:

PI Credentials:

PI Title:

PI Institution:

• Following the project title is

the principal investigator’s

name, title and institution

• Review both the research

team members (at the top

of the application page) as

well as the PI information to

be sure you do not have a

conflict of interest with this

applicant

IRB Submission

• Human Subjects?:

Yes / No / Pending

• Animal Subjects?:

Yes / No / Pending

• IRB to be submitted

upon notification of

funding?: Yes

• If human or animal subjects will be

included, the applicant must obtain IRB

approval either before or after funding

notification

• If submitted or approved, IRB Assurance

ID number and proof of approval or

submission will be uploaded in a PDF

format

• If submitting upon notification of funding,

the applicant will indicate this

Protection of Human Subjects and

Inclusion of Women and Minorities• Protection of Human Subjects or

Animals Used for Research:– How informed consent will be obtained

– Steps taken to protect participant’s

rights or the welfare of animals

– Potential risks associated with

participation in the project

• Women and Minority Inclusion in

Clinical Research:– Information on the composition of the

proposed population in terms of

sex/gender and racial/ethnic group

– Rationale for selection of subjects in

terms of scientific objectives and design

• The applicant is asked to

submit 1-2 paragraphs

on the following areas

– Protection of human

subjects or animals used

for research

– Women and minority

inclusion in clinical

research

There are currently no comments.

Innovation and Facilities and Resources

• Innovation– How the project challenges existing

paradigms or clinical practice

– How the project addresses an innovative

hypothesis or critical barrier to progress in the

field

– If applicable, how it develops or employs

novel concepts, approaches, methodologies,

tools or technologies in the area

• Facilities and Resources (Environment):

– Facilities and resources available to carry out

the project at all research sites such as:

• Computers

• Statistical and data management support,

• Office space

• Equipment, etc.

• The applicant is asked

to submit 1-2

paragraphs on the

following areas

– Innovation

– Facilities and

Resources

Implications for Practice

Implications for Practice and Research:

– Implications for oncology nursing

practice

– Identify future research that may

develop from this project

– Describe how this project will provide

the groundwork for seeking additional

funding in the future

– Describe when and how the study

findings will be disseminated

• The applicant is asked

to submit 1-2

paragraphs on the

Implications for

Practice and Research

Online Overall Impact Scores

• After completing your review of each

application you will need to enter a

preliminary impact score in the online

review system by clicking on one of

the numerical ratings at the bottom of

each application reviewed

• Once you have selected your

preliminary impact score, click

“Submit Ratings” to finalize your

score

My Application Score

0 / 90.0

My Ratings

Overall Impact Score1 = Exceptional (High impact)

2 = Outstanding (High impact)

3 = Excellent (High impact)

4 = Very Good (Medium impact)

5 = Good (Medium impact)

6 = Satisfactory (Medium impact)

7 = Fair (Low impact)

8 = Marginal (Low impact)

9 = Poor (Low impact)

1 2 3 4 5 6 7 8 9

O O O O O O O O O

Review Call Discussion

1) Primary, secondary, and collateral reviewers will begin by

stating their preliminary impact scores for the proposal.

2) Primary reviewer provides a brief summary of the project

3) Primary reviewer presents a brief 2 –3 minute critique

highlighting strengths and weaknesses in each criteria.

4) Next, Secondary Reviewer adds additional strengths and

weaknesses.

5) Collateral reviewer follows with any additional comments

not previously made.

Review Call Discussion

6) After the critiques have been presented, the review

team will discuss the proposal for up to 2 – 3 minutes.

7) Primary, secondary and collateral reviewers are then

asked if their preliminary impact scores stand as

submitted

8) All remaining reviewers will assign a final Overall

Impact Score to the application in private on the

online application review page.

9) Any concerns about the budget are discussed at this

time.

Comment Revisions

• You may have changes to your Reviewer Comment-

Scoring form after the conference call. Please make

these changes and e-mail the revised form to

research@onsfoundation.org within 48 hours of the

conference call.

• If you do not have changes to your critiques, please

e-mail the ONS Foundation Research Department to let

them know.

• The chair or co-chair will read the critiques. You may be

contacted if there are any questions.

Summary Statements

• Overall impact scores of applications will be the

mean of scores entered by all eligible reviewers,

multiplied by 10

• Final scores will range from 10-90, in whole

numbers

• The primary reviewer’s summary for ALL

applications will include the criterion scores and

critiques posted by assigned reviewers

Recipient Selection• After the review all of the proposals, the group’s average

scores will be rank ordered and used to guide the

assignment of grant awards.

• The ONS Foundation Executive Director, the ONS

Research Director and the review team chair and co-chair

will identify the recommended impact score funding range

and the applications within that range.

• The group’s recommendations are presented to the ONS

Foundation Board of Trustees for their approval.

• Once the award and non-award letters are mailed, the list

of funded proposals will be emailed to you.

THANK YOU!

Questions?

Please e-mail research@onsfoundation.org

with any questions during the review process