EVALUATING NSF EPSCOR INITIATIVES:TRACKING AND ASSESSING THE DEVELOPMENT OF EPSCOR-SUPPORTED RESEARCH CAPACITY

Julia Melkers, Associate ProfessorSchool of Public Policy, Georgia Institute of Technology

Eric Welch, Associate ProfessorDepartment of Public Administration, University of Illinois at Chicago

Prepared for presentation at the National Research CouncilWashington, DC

December 17, 2012

Evaluation frames are based on the understanding that:

Building research capacity is a complex and continuous process. Tracking magnitude of changes over time is critical to

capturing effects (interim effects). Given its mission and the structure of the

awards, EPSCoR has the potential to affect research capacity in range of ways.

Capacity is reflected in (and affected by): State politics and science objectives Institutions Infrastructure Human capital Collaborative relationships Resources Among others....

InstitutionalizatiInstitutionalizationon

Development of Development of early collaborative early collaborative outputsoutputs

Increased Increased production & other production & other impactsimpacts

Movement toward Movement toward sustainabilitysustainability

InstitutionalizatiInstitutionalizationon

Development of Development of early collaborative early collaborative outputsoutputs

Increased Increased productionproduction

Movement toward Movement toward sustainabilitysustainability

Dimensions of Research Capacity

Capacity as infrastructure Lab equipment Cyberinfrastructure Institutional processes and policy

Capacity as human capital Scientists Postdocs Graduate Students

Capacity as integration Integrating different expertise and perspectives on the

research problem (disciplinary, translational, etc) Capacity as productivity

Grant activity that extends and builds upon award work

Using Theories of Change in Evaluation:Example Project Logic Model

Sample Guiding Evaluation Questions: Collaboration

Collaboration: What are the barriers and facilitators to collaboration in [state] EPSCoR? within and between sites/groups? interdisciplinary and cross institutional? early career engagement? mid career role? Native and rural community partners? private and non profit sector integration? international partners? critical success factors?

Sample Guiding Evaluation Questions: Impacts

Research Impacts: What are the knowledge impacts of [EPSCoR] research and activities?

Stakeholder Impacts: How has [EPSCoR] developed interactions /tangible relationships with external partners/stakeholders over the course of the project?

Faculty and Staff Impacts: What research and management capacity is increasing as a result of [EPSCoR] work?

Student Impacts: What benefits and capacities do students accrue from [EPSCoR] ?

Institutional Impacts: How has the research capacity of [state] institutions changed via [EPSCoR] ?

Data Sources

Example: Some Observable Outcomes Science

Knowledge advancement and integration

Research productivity

Research impacts Increased know-how

Faculty/Researchers Production Advancement Enhanced or new

collaborative ties Student engagement Leadership Visibility

“Next generation” Student learning

and other impacts STEM interests and

advancement Institutional

capacity New programs,

processes Systems Support staff Culture

Measuring Research Activities & Outcomes

Collaborative Development

Intermediate outcomes that precede academic production

Emerging collaborative relationshipsIntegration of students in researchKnowledge transfer and ImpactsResearch-relevant behavioral changesNature of linkages and interactions

Operationalized through Network Analysis

Pre-publication interaction Faculty-faculty & faculty-student

collaboration Changed conceptual/ theoretical/

methodological approaches Changed research outlets and

communities Mechanisms/modes for boundary-

spanning linkages and outcomesMeasurable

through network analysis

Example: Student Interaction

All Collaborative Ties – Graduate Students Only Statistics – All Ties Number of nodes Number of ties Average degree centrality E-I Discipline

48 77 .42 -.299

Legend Square = Core institution Circle = Other institutions Green = environmental science Blue = computer science/engineering

Orange = social science Black lines = within discipline Red lines = between disciplines Node size = larger nodes reflect more semesters supported by EPSCoR

Collaboration Ties # ties Worked with on a grant proposal 14 Presented research to external stakeholders or external partners 20 Presented research to an academic audience 69 Presented research to a nonacademic audience 29 Co-authored a working paper 20 Co-authored an academic paper that has been published or is under review 5

Knowledge Impacts

Alaska-Based Orgs (Federal, State, Private, NGO)

International

Top Ten Subject Categories

(2007-12 EPSCoR Publications# Records

Subject Category

1 37 Environmental Sciences & Ecology

2 27 Ecology3 18 Geology4 17 Environmental

Sciences5 12 Environmental

Studies6 12 Zoology7 11 Physical

Geography8 10 Geosciences,

Multidisciplinary

9 8 Evolutionary Biology

10 7 Biochemistry & Molecular Biology

Other EPSCoR States

Other States

Rafols, Porter and Leydesdorff (2009)

Cognitive Sci.

Agri Sci

Biomed Sci

Chemistry

Physics

Engineering

Env Sci & Tech

Matls Sci

Infectious Diseases

Psychology

Social Studies

Clinical Med

Computer Sci.Business & MGT

Geosciences

Ecol Sci

Econ. Polit. & Geography

Health & Social Issues

Global Map of Science, 2007221 SCI-SSCI Subject Categories

Knowledge Impacts: Disciplinary Placement & Knowledge Sources

Reflections on EPSCoR Outcomes Research

Integration across disciplines – new questions, approaches Added value of collaborative engagement -- generation of new ideas and

new collaborative relationships Knowledge impacts -- National and global visibility and impacts

STEM workforce Early career affiliates – early grants, network reach, production, time to

tenure, mobility and competitiveness, mentoring and advice relationships Students – research networks, apprenticeship structures, mentoring

relationships, production, next career step Research – related support staff, sustained research relationships, new

grant awards Institutions

EPSCoR offices – communication, administrative structures, management Smaller institutions – research activity, developed capacity and

integration, statewide representation in decision making (eg. state S&T plan)

Curricular issues – new courses, content Social Impacts

engaged stakeholder organizations – economic development, social and economic objectives, support for education and research by stakeholders

Issues Relevant to Evaluation of EPSCoR EPSCoR projects and settings vary.

Disciplinary variance Partner relationships Under-represented populations Changes over time (historically)

Multiple and competing programmatic goals Lack of clarity/changing evaluation needs from EPSCoR Data consistency, across states and over time. Increased attention to strategic plans Unpacking role/impacts of senior leads vs other

faculty/researchers Institutional variation in data availability (esp. small

schools). Institutional effects not always evident Futures are unclear -- difficulty/lack of support to track

students and faculty post-award. Issue of relationships across awards and cumulative effects

on a state are not captured.