Evaluation of Large Initiatives of Scientific Research at the National Institutes of Health

Mary KaneConcept Systems Inc.

William M. TrochimCornell University

American Evaluation AssociationNovember 4, 2006

The Context

Changing nature of science Interdisciplinary, collaborative Large initiatives for complex problems Expansion of use of large center grants as research

funding mechanism Similar issues reported in the European Union (EU) in

connection with the evaluation of Science, Technology, and Innovation (STI) policies

Government wide accountability expectations GPRA PART ExpectMore.gov

Good science requires good management

Evaluation of Large Initiatives

National Cancer Institute Transdisciplinary Tobacco Use Research

Centers (TTURCs), (2001 – 2003) Centers for Disease Control

Prevention Research Centers Network, 2003-2005

National Institute of Allergies and Infectious Diseases AIDS Clinical Trials Network, Division of AIDS,

National Institutes of Health (2005 – present)

Evaluation Approach Culture change Collaboration and involvement of

researchers, funders, consultants Understand initiative life-cycle Develop initiative logic model Link comprehensive measures and tools to

model Keep costs and respondent burden low Assure scientific objectivity and credibility Address multiple purposes and audiences Design for re-use where possible Report and utilize results Provide an opportunity for reflection and

learning

Initiative Life Cycle Model

Conceptual Model

Measures

Questions

Stakeholders

Context• Motivation• Capacity• Structure• Expertise• Support

The context includes the organizational structures and organizational constraints that delimit evaluation activities. Issues include:

At each stage a wide variety of stakeholders need to be involved both in helping determine what questions should be addressed in evaluation and in providing their assessments of initiative performance and outcomes.

At each stage there are a variety of evaluation questions with more prospective questions earlier in the life-cycle and more retrospective ones later. Processes are needed for for prioritizing which questions will be addressed at each stage.

Evaluation is an empirical activity. Consequently, measures that are related to the constructs in the conceptual model needed at every stage.

Structured Conceptualization

Evaluation Methods

Needs Assessment

Evaluability Assessment

ImplementationEvaluation

ProcessEvaluation

OutcomeEvaluation

ImpactEvaluation

Cost-Effectiveness & Cost Benefit Evaluation

Secondary Analysis

Meta-Evaluation

Conceptual

Model

Formative/Ex Ante

Summative/Ex PostMethods

PolicyContext

New Initiatives

Strategic Impact

Policy Implications

StrategicGoals

Plan DisseminateImplementDevelop

Plan

DevelopImplement

Disseminate

The TTURC Case Study

Transdisciplinary Tobacco Use Research Centers

History RFA released 12/98 Grants reviewed 7/99 First award 9/99 Reissuance 9/04

Approximately $75 million in first phase

TTURC Life Cycle Model

Model Development

Concept Map

Engage the Community

Diversity & SensitivityRelationships & Recognition

Active Dissemination

Technical Assistance

Training

Research Methods

Research Agenda

Core Expertise & Resources

Evaluation System Plan

1234567…….

1234567…….

1234567…….

Logic Model

Inputs Outputs OutcomesActivities

Active Dissemination

TrainingEngage the Community

Technical Assistance

Core Expertise & Resources

Research Agenda

Community Health change

AnalysesMeasures

Measures & AnalysesConceptual Map & Logic ModelConceptual Map & Logic Model

FinancialReport (SF259a)

FinancialReport (SF259a)

Budget & JustificationBudget & Justification

BibliometricsBibliometrics

ProgressReport Summary

ProgressReport Summary

PublicationsPublications

Expenditures& CarryoverExpenditures& Carryover

ResearcherFormResearcherForm

Peer EvaluationPeer Evaluation

Peer EvaluationPeer Evaluation

Financial AnalysisFinancial Analysis

PersonnelReportPersonnelReport Personnel AnalysisPersonnel Analysis

EvaluationAnalysis

EvaluationAnalysis

Content AnalysisContent Analysis

Survey AnalysisSurvey Analysis

ProgressReport (PHS2590)

ProgressReport (PHS2590)

1. How well is the collaborative transdisciplinary work of the centers (including training) accomplished?

2. Does the collaborative transdisciplinary research of the centers lead to the development of new or improved research methods?

3. Does the collaborative transdisciplinary research of the centers lead to the development of new or improved scientific models and theories?

4. Does TTURC research result in scientific publications that are recognized as high-quality?

5. Does TTURC research get communicated effectively?6. Are models and methods translated into improved

interventions?7. Does TTURC research influence health practice?8. Does TTURC research influence health policy?9. Does TTURC research influence health outcomes?

Evaluation Questions

Evaluation Questions

1. How well is the collaborative transdisciplinary work of the centers accomplished?

• What are TTURC researcher attitudes about collaboration and transdisciplinary research?

• How do researchers assess performance of their centers on collaboration, transdisciplinary research, training, institutional support and center management?

• What are examples of collaboration, transdisciplinary and training activities of the centers?

• What is the quality and impact of the collaboration, transdisciplinary and training activities of the centers?

• Do TTURC research publications provide evidence of collaboration and transdisciplinary research, and how do they compare with “traditional” research?

• How effective and efficient is the management of the TTURCs?

Subquestions:

Evaluation Questions

1. How well is the collaborative transdisciplinary work of the centers accomplished?

Data Sources:• Researcher Form

• Attitudes about Transdisciplinary Research Scale (15 items)• Center Collaboration Scale (15 items)• Attitudes about Collaboration in Research Scale (8 items)• Institutional Support Index (12 items)• Overall Ratings of collaboration, transdisciplinary integration, training,

institutional support• Content Analysis of annual progress reports for activities, results and barriers

(code on collaboration, transdisciplinary integration, training, institutional support)

• Peer evaluation• Annual progress reports• Publications

• Bibliometric analysis of publications• Collaboration within and across institutions and centers• Numbers of fields represented by publications, cited and citing articles,

weighted by impact of journals• Management analysis

• Personnel• Budget and Financial

Researcher Form Each center responsible for

generating measures for 3-4 clusters on the map (at least two centers reviewed each cluster)

Compiled into measure development database, draft measure produced

25 closed-ended questions each with multiple subquestions

Overall performance ratings by outcome area

Open-ended Comments

244 specific measurement items proposed across the 13 content clusters

Scales and Indexes Attitudes about Transdisciplinary Research Scale (15

items) Center Collaboration Scale (15 items) Attitudes about Collaboration in Research Scale (8 items) Institutional Support Index (12 items) Methods Progress Scale (7 items) Science and Models Scale (17 items) Barriers to Communications Scale (8 items) Center-to-Researcher Communications (5 items) Center External Communications (2 items) Progress on Development of Interventions Index (12 items) Policy Impact Index (4 items) Translation to Practice Index(9 items) Health Outcome Impact Scale (6 items)

Researcher Survey8. Collaboration within the center

o.

n.

m.

l.

k.

j.

i.

h.

g.

f.

e.

d.

c.

b.

a.

95% CI

4.64.44.24.03.83.63.4a. Support staffing for the collaboration.b. Physical environment support (e.g., meeting space)

for collaboration.c. Acceptance of new ideas.d. Communication among collaborators.e. Ability to capitalize on the strengths of different

researchers.f. Organization or structure of collaborative teams.g. Resolution of conflicts among collaborators.h. Ability to accommodate different working styles of

collaborators.i. Integration of research methods from different fields.j. Integration of theories and models from different

fields.k. Involvement of collaborators from outside the center.l. Involvement of collaborators from diverse disciplines.m.Productivity of collaboration meetings.n. Productivity in developing new products (e.g., papers,

proposals, courses).o. Overall productivity of collaboration.

Content Analysis

Code approximately 80-90 project reports per year by the 13 outcome clusters

Did three rounds of reliability testing and refinement of coding definitions

Final reliability > .9

Progress Report Content Analysis – Years 1-3

Collaboration

Transdisciplinary Integration

External Recognition and Support

Science & Models

Publications

Interventions

Communication

Policy Implications

Translation to Practice

Health Outcomes

Training

Internal Recognition And Support

Methods

(data from Content Analysis of Annual Progress Report Form PHS2590)

Peer Evaluation – Years 1-3

Training

Collaboration

Transdisciplinary Integration

Internal Recognition And Support

External Recognition and Support

Methods

Science & Models

Publications

Interventions

Communication

Policy Implications

Translation to Practice

Health Outcomes

Bibliometric Analysis

What is a TTURC publication? Results from TTURC research Cites TTURC Grant Number Independent peer evaluation would

identify the influence Components of bibliometric analysis

Publications, citations, cited (references)

Journals of publication, citing, cited Field (Current Contents) Year

Bibliometric Analysis Indicators

Journal Impact Factor (JIF) – average number of citations of a journal of all articles published in previous two years

Journal Performance Indicator (JPI) – average number of publications to date for all publications in a journal in a particular year

Field Journal Performance Indicator – JPI for all journals in a field

Adjusted Journal Performance Indicator (Expected Citations) – JPI for a specific type of publication

5-year Impact – Average number of citations to publications over a five year period

Bibliometrics

Paired Samples Test

.2007 1.97978 .25348 -.2228 .6241 .792 60

.6367 2.66952 .34180 .0657 1.2077 1.863 60

.5961 2.58795 .33135 .0426 1.1497 1.799 60

Citations -Expected Citations

Pair1

Citations - JournalPerformanceIndicator (JPI)

Pair2

Citations - FieldPerformanceIndicator (FPI)

Pair3

Mean Std. DeviationStd. Error

Mean Lower Upper

90% ConfidenceInterval of the

Difference

Paired Differences

t df

On average, there were .64 more citations of TTURC publications than for other publications in the same journal.

On average, there were .6 more citations of TTURC publications than for other publications in the same field.

Citation of TTURC publications is significantly higher than for journal and field comparison groups.

Bibliometrics

Descriptives

Difference (Citations - Expected Citations)

6 -1.9633 .53114 .21684 -2.5207 -1.4059 -2.43 -1.2317 1.0835 3.15161 .76438 -.5369 2.7039 -1.74 11.0023 .2887 3.02642 .63105 -1.0200 1.5974 -2.43 11.00

20002001Total

N MeanStd.

Deviation Std. Error Lower Bound Upper Bound

95% Confidence Interval forMean

Minimum Maximum

ANOVA

Difference (Citations - Expected Citations)

41.170 1 41.170 5.392 .030160.333 21 7.635201.503 22

Between GroupsWithin GroupsTotal

Sum ofSquares df Mean Square F Sig.

Only the two complete years were used in this analysis.

Citations lower than expected in year 1, higher in year 2.

Citation of TTURC research publications is significantly increasing over time relative to expectation.

Financial Analysis

(data from Financial Status Reports of grantees)

Cumulative Percent of Federal Funds Spent by Grantee

80%

62%

99%

78%

72%

46%

77%

91%

74%

68%

57%

93%

82%

94%

87%

66%

40%

50%

60%

70%

80%

90%

100%

Period 1 Period 2

Center 1

Center 2

Center 3

Center 4

Center 5

Center 6

Center 7

Total

Carryover

(data from Budget Justification, Annual Progress Report Form PHS2590)

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

Year 1 Year 2 Year 3

% s

ub

pro

ject

s

1

2

3

4

5

6

7

Mean

Percent of subprojects by center and year that reported a carryover.

Delay of projectstart

Unanticipatedobstacles

Changes inprocess -practical

Other-Specify Not stated

Reasons for Carryover

Reasons for Carryover

0

0.1

0.2

0.3

0.4

0.5

0.6

Causes of Delay or Unanticipated Obstacles

Sta

ffing

issu

eIm

plem

enta

tion

or lo

gist

ical

issu

eR

esea

rch/

met

hods

issu

eG

rant

ing

agen

cy is

sue

Infra

stru

ctur

e is

sue

Oth

er-S

peci

fy

Not

sta

ted

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

(data from Budget Justification, Annual Progress Report Form PHS2590)

What Worked

Less Promising Researcher Survey – one wave Content Analysis – costly, time

consuming Peer Evaluation of publications

More Promising Researcher Survey scales Peer evaluation of progress reports Financial Analysis Bibliometrics

Conclusions Sustainability Challenges

Funding challenges Researcher motivation

Methodological Challenges Peer review Bibliometrics Integrating results

Organizational Challenges Agency resources Grantee resources External contractors

Utilization Challenges Building over multiple time points Building over multiple initiatives