View
215
Download
0
Category
Tags:
Preview:
Citation preview
Evaluating the Alaska EPSCoR Phase III: Resilience and Vulnerability in a Rapidly Changing North: The Integration of Physical, Biological and Social Processes
Dr. Julia Melkers, Associate Professor of Public PolicyGeorgia Institute of Technology
Dr. Eric Welch, Associate Professor of Public AdministrationUniversity of Illinois at Chicago
Year 3 Evaluation PresentationAlaska EPSCoR All Hands Meeting
May 2010Fairbanks, Alaska
Stating the Obvious Collaboration is difficult, problems of:
disciplinary language, distances, funding, methodological, conceptual and theoretical approaches lack of clarity about why research questions are
“important”. Requires commitment to learn from other fields Variation in goals and interests of stakeholders and rural
campuses Institutional barriers and facilitators are key to success
Three Year Evaluation of Alaska EPSCoR Phase III A formative and interactive approach.
Can also provide lessons learned for future EPSCoR proposals and initiatives.
An summative evaluation process to serve as an accountability mechanism.
A research driven evaluation, based on studies of science, collaboration, networks, among others.
Year 1 Year 2 Year 3 What are the
foundational capacities of AK EPSCoR faculty and students?
What foundational relationships exist?
What early outcomes can be observed?
How are collaborative interactions and related outputs developing in AK EPSCoR?
How integrative is AK EPSCoR and what are the related barriers?
What benefits are EPSCoR students experiencing?
How are collaborative interactions and related outcomes continuing to develop among students and faculty?
How integrative are these interactions and outcomes?
How are rural campuses and native communities benefiting from EPSCoR? What are the key issues in this?
How are institutional and administrative factors relevant to the ability of AK EPSCoR to meet its goals?
Evaluation Questions
Year 1 Year 2 Year 3Evaluation Focus
Collaboration Familiarity Interdisciplinary
orientation
Collaboration Productivity Factors affecting
productivity Integration Student impacts
Continuation of year 2 issues
Collaboration Administrative issues Rural campus
Data Collection
Baseline Data Collection: Survey of
faculty and students
Interviews with AK EPSCoR leadership
Interim Data Collection: Student
interviews Survey of faculty
and students
Final Data Collection: University
administrator interviews
Native organizations interviews
Rural campus interviews
Survey of faculty and students
Social Networks & Evaluation
Social network analysis can add to the program evaluation methodological toolbox.
Why use SNA in program evaluation? Allows us to capture knowledge development, social and
human capital, and other interim outcomes that cannot be captured in other data.
Adds rigor to attitudinal and self reported data regarding behavior, outputs, outcomes, and relationships.
Adds detailed dyadic data on specific relationships and exchange.
Can be combined with other data to provide a comprehensive picture. Data may be drawn from surveys, interviews and existing documentation.
Some Network Characteristics Network Size
Number of nodes that are connected to others
Number of Ties Number of ties that link
these nodes individuals can have multiple ties)
Network Density measures the percentage
of ties that exist compared the number of possible ties.
This measure provides one way of describing the extent to which potential ties are unexploited.
Number of Ties Density Network centralization index Average Degree Centrality E-I Index (Organization) E-I Index (Specialty)
680 .09 22.36% 16.34% -.43 -.36
Legend: Organization Circle = Org1 Square= Org2 Triangle = Org3
Rank 1=Rank1 2= Rank2 3= Rank3 4= Rank4 5=Other
Specialties Red= Field1 Blue= Field2 Green= Field3
Cross Specialty Ties Green Line = Across Specialty Purple Line = Within Specialty
Some Network Characteristics
Average Degree Centrality measures the average
number of immediate connections that each individual has in the network.
Provides a way to examine the level of participation in network activity by the ‘average’ person in the network.
Number of Ties Density Network centralization index Average Degree Centrality E-I Index (Organization) E-I Index (Specialty)
680 .09 22.36% 16.34% -.43 -.36
Legend: Organization Circle = Org1 Square= Org2 Triangle = Org3
Rank 1=Rank1 2= Rank2 3= Rank3 4= Rank4 5=Other
Specialties Red= Field1 Blue= Field2 Green= Field3
Cross Specialty Ties Green Line = Across Specialty Purple Line = Within Specialty
Some Network Characteristics
(E-I) Index the extent to which the
network is made up of individuals outside as compared to inside a particular environment or context.
The EI index is calculated as: (external ties – internal ties) / (external ties + internal ties) and ranges from negative one to one.
Number of Ties Density Network centralization index Average Degree Centrality E-I Index (Organization) E-I Index (Specialty)
680 .09 22.36% 16.34% -.43 -.36
Legend: Organization Circle = Org1 Square= Org2 Triangle = Org3
Rank 1=Rank1 2= Rank2 3= Rank3 4= Rank4 5=Other
Specialties Red= Field1 Blue= Field2 Green= Field3
Cross Specialty Ties Green Line = Across Specialty Purple Line = Within Specialty
Our Final Report
Today’s Presentation Collaborative interactions – Year 3 and Overtime Integration with rural campuses and communities Student Impacts Observations and some lessons learned
Written Report Collaborative Interactions and Outcomes Student benefits and learning Integration with native communities Rural Campus issues Attitudes and feedback regarding EPSCoR issues Lessons Learned Recommendations for future EPSCoR efforts
Overview of Survey Respondents
Year 1 Faculty: 86% response rate (n=59) Students: 85% response rate (n=57)
Year 2: Faculty: 75% response rate (n=79) Students: 72% response rate (n=48)
Year 3 Faculty: 75% response rate (n=111) Students: 57% response rate (n=52)
Year 3 Faculty/Researcher Respondents (n=111)
UAA30%
UAS
6%
UAF64%
biological science
component38%
social science compo-
nent42%
physical science
component20%
Rank:Assistant Professor = 35%Associate Professor = 26%Full Professor = 21%Research faculty & other = 24%
Year 3 Student Respondents (n=52)
84% respondents at UAF
Bache-lors29%
Mas-ters25%
Doc-toral46%
Comparing Developments Across Time Useful to track the EPSCoR community
General observations on community of researchers.
More meaningful to track core group of individuals who have been involved over time. Longitudinal Faculty Comparison Groups Survey Respondents from Years 1 & 3 (n=44) Year 1 respondents and “active” individuals who
responded to Year 3 survey (n=65) (59% of Year 3 respondents)
Changes in Year 2 & 3
COLLABORATION & INTEGRATION
Sole Authoring by Component Y3 (mean number of faculty reporting these products)
.00
.20
.40
.60
.80
1.00
biological component social science componentphysical science component
Sole Authoring by Institution Y3(mean number of faculty reporting these products)
.00
.20
.40
.60
.80
1.00
UAA UAS UAF
What became of Desired Collaboration Year 1?
Respondents // Named desired collaborators 48 // 60Percent collaborators named in same discipline 44/60 (73.3%)Percent collaborators named in other disciplines 16/60 (26.7%)
Same disciplineBio to biological science 25
Phys to physical science 4Soc to social science 15
Different disciplineBio to physical science 3
Bio to social science 10Phys to social science 3
Percent disciplinary collaborations 2/16 (12.5%)
Percent interdisciplinary collaborations 10/44 (22.7%)
Who would you like to collaborate with, but have not done so to date?
Reasons for Successful Collaborations
Sharing common research interests in permafrost related arctic engineering and science issues.
Common interest in phylogenetic methodology, especially methods of DNA sequence alignment.
We are both interested in future trends in moose and caribou populations.
We are both interested in developing programs that link arts with the sciences.
Collaborative style and our complementarity [That person’s] energy and enthusiasm for doing
challenging empirical research.
Production: Journal Articles
Collaborative Outputs 2009-10: Journal Articles 2009-10 Number of Ties Density Average Degree Centrality E-I Index (Campus) E-I Index (Component)
93 .004 .72 -.720 -.462
2008-09 Number of Ties Density Average Degree Centrality E-I Index (Campus) E-I Index (Component)
71 .006 .92 -.52 -.40
Red=Biological ScienceBlue= Social ScienceGreen=Physical ScienceSquare= Year 1 & 3 participant
Impacts on Communication
Changed the way in which you communicate your research 2009-10 Number of Ties Density Average Degree Centrality E-I Index (Campus) E-I Index (Component)
127 .005 1.06 -.591 -.150
2008-09 Number of Ties Density E-I Index (Campus) E-I Index (Component)
113 .009 1.85 -.55 -.42
Red=Biological ScienceBlue= Social ScienceGreen=Physical ScienceSquare= Year 1 & 3 participant
Impacts on Research
Changed the data collection or analysis methods you use in your research 2009-10 Number of Ties Density Average Degree Centrality E-I Index (Campus) E-I Index (Component)
108 .005 .95 -.685 -.481
2008-09 Number of Ties Density Average Degree Centrality E-I Index (Campus) E-I Index (Component)
82 .006 1.34 -.71 -.77
Red=Biological ScienceBlue= Social ScienceGreen=Physical ScienceSquare= Year 1 & 3 participant
Growing Collaboration Around Key Research Themes
Disturbance Regimes 2009-10 Number of Ties Density Average Degree Centrality E-I Index (Campus) E-I Index (Component)
120 .005 1.01 -.78 -.33
2008-09 Number of Ties Density Average Degree Centrality E-I Index (Campus) E-I Index (Component)
78 .006 1.2 -.64 -.44
Red=Biological ScienceBlue= Social ScienceGreen=Physical ScienceSquare= Year 1 & 3 participant
Coalescing of Research Themes
Species response to climate change
Disturbance regimes
Human migration & climate change
Perma-frost
Social Aaspects of
ground engineering
Ecosystem services
2009-10 100 120 86 95 52 95
2008-10 127 78 69 79 57 95
Change in the Number of Collaborative Ties over Time
Do Networks Matter for Continued Activity? Dependent Variable – Indicator of Continuation
Have you developed any new ideas or plans for research as a result of your interaction with EPSCoR Phase III? (1/0)
Independent Variables Familiarity “I understand this person’s knowledge and
research skills. This does not necessarily mean that I have these research skills or am knowledgeable in these domains, but that I understand the skills this person has and domains in which they are knowledgeable.” (Strongly Disagree to Strongly Agree)
Size of Collaboration Network: Since July 2007, with which of the following EPSCoR faculty members did you work with on a journal article. (1/0)
Other controls: Tenured, Biology, UAF
Example Findings
B SE Significance Exp(B)
Familiarity 1.647 0.778 ** 5.191
Collaboration Network Size 1.341 0.687 ** 0.262
Biology 1.962 1.421 0.141
UAF -1.474 1.089 4.366
Tenured -1.223 1.016 3.399
Constant 3.479 1.920 * 0.031
Have you developed any new ideas or plans for research as a result of your interaction with EPSCoR Phase III?
(1/0) *= p<0.10, ** = p<0.05)
INTEGRATION WITH RURAL CAMPUSES AND COMMUNITIES
Stakeholder Interaction
Faculty report limited importance of interaction with stakeholders.
Faculty report few meetings with stakeholders over the course of EPSCoR Phase III (1= never, 2= once, 3= 2 to 3 times, 4= more than three times)
UAA UAS UAF
Private Companies1.86 1.50 1.96
Rural campuses 1.37 1.57 1.50
K-12 education 1.64 2.43 2.05
Native communities 2.22 1.71 2.16
Native Alaskan corporations 1.46 1.00 1.62
State government officials 2.07 1.86 2.26
Local government officials 2.10 1.57 1.95
Federal government officials 2.11 2.00 2.29
Research in Native Communities
Native organizations and local contacts critical. Provide access Experiences with researchers are varied EPSCoR can facilitate the capacity of faculty
to interact with native communities Nature of outreach matters
Input / agreement of community Follow-up is key
Relevant research has implications for attraction and retention in STEM
Rural Campuses
Play key role as liaison to native communities Faculty are important nodes for connection Student are important nodes for connection
Serve multiple roles Skills and training Research for understanding local context and for
ensuring community security Capacity building in community
EPSCoR is an important supporter of research dissemination through Western Alaska Interdisciplinary Science Conference (WAISC).
STUDENT IMPACTS
EPSCoR Faculty and Students
28% of faculty report that they are currently working with an EPSCoR supported student.
50% of those report that the research would not have been conducted without EPSCoR funding.
Students Report Positive Impacts:Research Interests 75% of student respondents are required to
do a thesis. Of these:
65% report that their thesis or dissertation includes some aspects of climate change research.
55% report that their thesis or dissertation topic “has been influenced by their EPSCoR work.”
To what extent has your Alaska EPSCoR experience helped you to develop skills and knowledge in the following areas?(1= not at all, 2= some, 3= a great deal)
coordinating research across institutions
understanding climate change research
analyzing or interpreting data
working on a multi-disciplinary team
communicating technical concepts to others
writing professionally
understanding how to incorporate perspectives from difference disciplines
making academic presentations
designing research
.00 .50 1.00 1.50 2.00 2.50 3.00
PhD
Masters
Bachelors
PROSPECTS FOR THE FUTURE
Faculty Collaboration Interests
UAA UAS UAF
I would like to collaborate more with faculty in different disciplines.
3.19 3.43 3.17
I would like to collaborate more with faculty on other UA campuses.
3.32 3.57 3.00
I would like to collaborate more with students in different disciplines.
3.00 3.17 2.94
I anticipate continuing to collaborate with individuals I have worked with in Alaska Phase III
3.28 3.86 3.31
Generally positive attitudes across institutions regarding collaboration. (4 point scale, 1=strongly disagree)
Faculty Collaboration Interests
Faculty component-based responses show similar variation in interest in interacting with faculty and students in other disciplines.
Biological Social Physical
I would like to collaborate more with faculty in different disciplines.
3.05 3.28 3.30
I would like to collaborate more with faculty on other UA campuses.
3.13 3.14 3.10
I would like to collaborate more with students in different disciplines.
2.84 3.02 3.10
I anticipate continuing to collaborate with individuals I have worked with in Alaska Phase III
3.39 3.28 3.38
Production: Grant Proposals to Extend Phase III Research
Grant Proposals to Extend Phase III Research Number of Ties Density Average Degree Centrality E-I Index (Campus) E-I Index (Component)
93 .004 .75 -.570 -.247
Red=Biological ScienceBlue= Social ScienceGreen=Physical ScienceSquare= Year 1 & 3 participant
Capacity Development in EPSCoR Phase III Research:
Evidence that collaborative networks have increased in size and diversity over Phase III.
Research theme-based networks are gaining more research ties across components.
Production and grant activities to sustain the capacity developed through Phase III are evident.
Faculty report positive impacts of EPSCoR engagement:
Production Connections Research ideas Changes in research processes
Capacity Development in EPSCoR Phase III Education :
Students report positive impacts of EPSCoR Phase III
Level of student research interests in climate change, as evidenced by thesis and dissertation topics, indicates potential capacity impacts on new generation of climate change researchers.
Strong student attribution of research skill development to EPSCoR Phase III, particularly among undergraduates.
Capacity Development in EPSCoR Phase III
Rural Campuses and Communities: Significant interest in developing stronger ties to bring
research and education opportunities and resources to rural campuses.
Researchers in the larger campuses express concern about “access to” rural communities.
Increasing exposure of climate change research in rural communities may attract and retain students there.
There are key individuals in rural campuses and communities that can help catalyze productive interaction.
Structural and Other Issues Relevant to Future EPSCoR Importance of support of EPSCoR at all levels of
administration and management. In smaller institutions it is not always obvious
how to best benefit from EPSCoR. Administrative cultures constitute barriers to
effective participation in and management of EPSCoR.
Ambitious efforts to cross disciplines and institutions require strong institutional support. Student network development across institutions
can also assist in this.
Questions and Comments
Recommended