1

Approach—Methods:Not an Afterthought!

Blair T. JohnsonCHIP | Psychology | UConn

2

Wow! Methods and Statistical Analysis Sections!!•After lunch??•Really???!!

3

My background• Assistant-Associate Professor of Psychology, Syracuse University (1988-1999)• Professor of Psychology, UCONN (1999-present)• Background: Social influence, Social Structure, Statistics (especially Meta-Analysis)• Served on numerous NIH, CDC, and UCONN grant panels• Served on IRB, two terms (2003-5; 2013-present)• Funded by NIH continuously since 1995

• 3 R01 awards from NIMH (current R01 is in year 15!)• 2 K awards, 1 from NIMH (5-years, K01) and 1 from NIAAA (1-year, K18)• This work focused on HIV prevention

• Currently proposing new research on HIV prevention and on exercise and blood pressure (with Linda S. Pescatello)

• New collaborations? Sure! Use my geotemporal databases to illuminate important health problems!

4

Overview, Goals, Assumptions• Big picture: The place and importance of the Method section in your

grant proposal: Not an afterthought!• 12 13 14 15 16 Tips for how to write a clear Method section that

inspires confidence in your project.• Discussion throughout the session (and the next focused on the

statistical analysis portion of the proposal).• Assumptions:

• NIH-style grant proposals (e.g., R01, R03, R21) (though much advice generalizes to other funders).

• Much of my advice generalizes to the entire grant-writing process.• Mea culpa: I’ve not yet written a perfect grant proposal.

5

The Big Picture• What are the most important components of the grant proposal?

• Summary• Resources• Budget justification• Biosketch(es)• Specific Aims• Approach (Method/Statistical Analysis)• Literature Cited• Data Safety and Monitoring Plan• Multiple PI

6

The Big Picture• Tip 1: The Method section should flow from the preceding sections;

inconsistencies increase doubt and decrease confidence in the proposal.• “Consistency is contrary to nature, contrary to life. The only completely

consistent people are the dead.” (Huxley)

7

The Big Picture• Tip 2: Write in such a way that you can convince your most severe, over-

worked, stress-crazed critic.• Corollary: Know the traditions and personnel of the SRG (look at what

research it has supported in the recent past).• Explain in detail how you will carry out the research.

• Demonstrate careful, thorough research plan.• Convey feasibility.

• Describe all study procedures.• Acknowledge potential setbacks and how you will overcome them.

Tip 3, an “Über-Tip”: Don’t Make the Reader Do Your Work

For You! (Or it will hurt your score!)

8

Research PlanThe research plan describes the proposed research, stating its significance and how it will be conducted. Remember, your application has two audiences: the majority of reviewers who will probably not be familiar with your techniques or field and a smaller number who will be familiar.• All reviewers are important to you because each reviewer gets one vote.• To succeed in peer review, you must win over the assigned reviewers .

They act as your advocates in guiding the review panel’s discussion of your application.

• Write and organize your application so the primary reviewer can readily grasp and explain what you are proposing and advocate for your application.

http://grants.nih.gov/grants/writing_application.htm#developPlan

INSIDER TIP: Appeal to the reviewers and the [program officers] by using language that stresses the significance of your proposed work.

9

How Will Your Approach Be Judged?Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project? Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed?

http://grants.nih.gov/grants/funding/sbirsttr_ReviewCriteria.htm

10

Methods Section: Not an Afterthought!

Tip 4: Detail matters!

s Corollary: Proof read your proposal!

Corollary: Carefully read all instructions in the application and application guide!

11

SRG Reaction• “the submitted application contained a number of typos… raised

concerns that lack of attention to detail may produce sloppy research…“ (paraphrased)

12

Give an Overview, Use Headings• Tip 5: Your introductory paragraph should give an overview to the entire

approach.• Basic sections:

• Introduction• Justification and Feasibility• Work design/activities• Expected outcomes/results• Anticipated problems and their solutions (alternative strategies)

• Tip 6: Using headings for the sections will help reviewers find the information.

13

Use Your Specific Aims• Tip 7: Use your Specific Aims to organize the details in the Method.• A Method section should be as linear and straightforward as possible. • Organizing the Method by specific aim helps the reviewer judge

whether your research is possible, and, at best, that it is highly likely to succeed.

14

Nuance the Method Meaningfully• Tip 8: Anticipate concerns that the SRG might have about particular

methods.• Corollary: Your own knowledge of the substantive domain should be

high!• Write brief justifications why a certain procedure is more trusted than

another.• Or, describe how you will use the data from the project to justify

particular assumptions. (More on this issue in the Statistical Analysis portion!)

• Showing this level of expertise helps to inspire confidence in the proposal.

15

Must Be Justifiable and Feasible

• Tip 9: Your approach must be justifiable and feasible.• Corollary: Don’t promise too much.• Corollary: Preliminary research should strongly support the approach.• Make sure your approach is consistent with current scientific literature

related to the subject.

16

Include a Detailed Timeline• Tip 10: Include a detailed timeline

across the entire funded project.• Shows attention to detail,

thoroughness.• Helps to show how busy you and

your team will be.• Convince reviewers the proposal is

reasonable with allotted time.

17

Use Proactive and Confident Language• Tip 11: Describe your methods proactively and confidently, as though

you will receive the award.• Corollary: Do not describe your project tentatively. • In other words: You and your team are poised and ready, just waiting for

the grant money to begin the project.• Good example: “Our team will perform these procedures…”• Bad example: “Our team would like to perform these procedures…”

Convey Excitement About YOUR PROPOSAL

No one will be excited about the proposal if You Are NOT!!!

19

Give Method Section Sufficient Space• Tip 12: Save yourself enough space to write a compelling Method

section!• Corollary: Write the Methods early in the process (e.g., once Specific

Aims solidify), even though most SRG members will read from front to back.

20

Illustrations Can Save Space, Enhance Interest

• Tip 13: To the extent that grant instructions permit it, use Tables and Figures to save space and enhance interest.

• Corollary: Proposals can be too text-dense.

Which Grant Would You Want to Read ?

22

• Attention to detail—thoroughness—inspires confidence in the team and the Methods proposed.

• Attractive graphics imply high skill.

23

• Even an image that takes a full-page can be presented as a fraction of a page.

• SRG members can zoom in if they want to see the details.

24

Savvy Software Expertise Comes in Handy• Tip 14: Use creative illustrations.• Corollary: Graphics and illustrations lacking clarity will only frustrate a

reader and decrease confidence.• Make images larger than they need to be and then resize them. • Images saved in png format generally have better scalability than other

common formats (e.g., jpg, gif, bmp).

USE FIGURES TO ILLUSTRATEPATHWAYS AND SPECIFIC AIMS

Aim 1: Characterize nuclearprotein complex

Aim 2: TRA-1, TRA-2interaction

Aim 3: Nuclear-cytoplasmic trafficking NXF-1 and TRA-2

26

Practical Matters• Tip 15: The budget is implicit in the Method section. • Corollaries:

• Make certain that the roles for all team personnel are clearly stated (and match what is said in the budget justification); highlight competence and expertise of yourself/collaborators to carry out research.

• If a reviewer had doubts about your budget after reading the justification, the Method is your last chance to make an implicit pitch that your budget is appropriately scaled to the tasks that your team will perform.

• The Method should implicitly sell what a great value the research would be to conduct.

27

But Your Method Is about Science, Not Budget!

• Tip 16: Leave arguments about budget per se in the Budget Justification section.

• Your Method section should mention money only in connection to practical matters (e.g., incentive payments to participants).

• But the details in the Method should make clear why particular personnel and their time are needed for the success of the project.

28

Wow! Statistical Analysis Section!!•After the coffee break!•Really!!

Approach: Statistical Analysis—Not an Afterthought!

Blair T. JohnsonCHIP | Psychology | UConn

29

30

31

Goal of the Statistical Analysis Section• State how the Specific Aims will be addressed in your statistical

analyses. • Makes a case that the study has a good chance to detect the proposed

effects (viz., “is sufficiently powered”).• Assumes sampling individual-level observations.• In meta-analysis: Examine all available qualifying observations.

• The concern is more with whether there are enough studies to permit tests of the hypotheses.

• Power analysis is less well developed here.• NIH now requires a power analysis for most proposed research.

32

Tips for a Successful Analysis Section1. Examine multiple models of analysis sections from high-quality

research similar to your own.2. Justify the analysis method as the best in practice.3. Early consultation with a statistician is recommended, but there are

many statistical software programs available (ensure statistical analysis plan is sound).

4. For power analysis, select effect size standards carefully (ideally from a meta-analysis of relevant trials). If you select too large an effect size, you will be underpowered to detect the effect!

33

Power Analysis• In order to minimize the reporting of false-negative data, a power

analysis should be performed for sample size determination. Power is the capability of a study to detect a difference if the difference really exists. A type II error occurs when a true difference exists between study populations but there are insufficient numbers of subjects to detect this difference.

Power/Sample Size ExampleExpected standardized mean difference between treatment and control groups

Power 1.25 1.00 0.75 0.50 0.25

75% 20 30 52 114 446

80% 22 34 58 128* 504

85% 26 38 66 146 576

90% 28 44 76 170 674

*Required sample size to address study aims, if a medium effect size is assumed.

35

G*Power 3.1.9.2

36

More Tips for a Successful Analysis Section5. Generally, statistical analysis sections should use conservative

estimates of effect size to ensure that study is well-enough powered. SRG members are likely to be savvy about standards and easily suspicious, so it pays to sweat the details.

6. Discuss how you will use sensitivity analyses to examine the robustness of your main analyses.

37

Critique?“To establish an estimate of power and to project proper sample size, a previous study quantifying the dependent variable was used using GPower software 8. For a 2-sided within-group design at a 0.05 alpha level and desired power level of 0.95, the estimated sample size was 11 for each gender. We will enroll up to 14 subjects for each group in order to increase statistical power as well as take into account potential drop outs. Dependent T-tests will be used to assess statistically significant differences between conditions.”

38

Example 2With n=400, alpha2 = .05, and a medium effect size (.30) the power will be >.99 for analyses of variance and .96 for

zero order correlations120. The power for a regression analysis that includes 11 variables (i.e. sex, ethnicity, positive symptoms, negative symptoms, aggression, impulsivity, depression, premorbid adjustment, gene marker, family history and substance abuse) with n=400, alpha=.05 and a medium effect size (R2=.10) will be greater than .90. We do not anticipate that all 11 variables will contribute significantly to the model. With 6 variables, a more likely model, the power will be > .90. For the exploratory regression analyses conducted within the group of attempters (n=200) the power will be at least .80 to detect a medium effect size (R2=.10). For correlational studies among the genetic and biochemical measures (approximate n=100) the power will be .80 for a medium effect size (R2=.10) and .95 for a larger medium effect size (R2=.25). The increased sample size will now provide the power necessary to consider attempters and nonattempters separately. Analyses examining single attempters, multiple attempters and nonattempters (approximate sample sizes: 120, 80, 200) will still maintain adequate power (power>.70).

Grant application by Jill M. Harkavy-Friedman, PhD

Good Luck!If at first you don’t succeed –

Revise and resubmit