Embed Size (px)
Citation preview
www.edvantia.org
Organizational Consultant, Critical Friend,
and Evaluator: The Value and Challenge of Flexible
Roles
Kristine L. Chadwick, Ph.D.Evaluator for
The Spatial Intelligence and Learning Center (SILC)The Science of Learning Center on Visual Language
and Visual Learning (VL2)
www.edvantia.org
SILC and VL2 Evaluations
• NSF encouraged centers to change evaluators
• Have served as SILC’s evaluator since its third year of operation (2008-2009)
• Served as VL2’s evaluator during its fourth year of operation (2009-2010) (and will continue in Years 6-10)
2
www.edvantia.org
What is SILC? (http://spatiallearning.org)
“The Spatial Intelligence and Learning Center (SILC) brings together scientists and educators from Temple University, Northwestern University, the University of Chicago, the University of Pennsylvania, and the Chicago Public Schools (CPS) to pursue the overarching goals of
– understanding spatial learning– using this knowledge to develop programs and technologies that will
transform educational practice, helping learners to develop the skills required to compete in a global economy
SILC participants include researchers from cognitive science, psychology, computer science, education, and neuroscience, as well as practicing geoscientists and engineers who are particularly interested in spatial thinking in their fields, and teachers in the CPS.”
3
www.edvantia.org
What is VL2?(http://vl2.gallaudet.edu)
“The purpose of VL2 is to gain a greater understanding ofthe biological, cognitive, linguistic, sociocultural, and pedagogical conditions that influence the acquisition of language and knowledge through the visual modality. Gallaudet University houses VL2, and brings together deaf and hearing researchers and educators from a variety of disciplines and institutions to study how language and literacy develop in deaf individuals.”4
www.edvantia.org
SILC and VL2 Evaluation Priorities
• Satisfy NSF evaluation “requirements”
• Present well at Year 4 high-stakes site visit
• Consult on center management issues
• Gather and report data on aspects of center PIs knew less about (e.g., outreach for SILC, training at affiliate IHEs for VL2)5
www.edvantia.org
Articulating Center Theory• Theory of the center, not its research per
se– Value added of center-mode research– Challenging for the center scientists
• Immediate and intermediate outcome categories– Centerness– Dissemination– Interdisciplinarity– Outreach– Training– Tools and resources
6
www.edvantia.org
Program Theory / Logic Model• Long-term Outcomes (> 10 years)
– Social Capital (reputation, networks)– Intellectual Capital (research advancement)– Physical Capital (tools and resources)– Human Capital (diverse interdisciplinary
researchers)• Impacts based on NSF goals
– advancing science– applying science to specific technological,
educational, and workforce challenges– enabling science to capitalize on new
opportunities and discoveries and respond to new challenges
7
www.edvantia.org
Evaluation Approach
• Operationalizing, measuring (or documenting) resources, activities, outputs, immediate outcomes in logic model
• Reflect their growth as centers• Consulting on center processes• Outsider viewpoint, critical friend
8
www.edvantia.org
NSF Requirements
• Center-level evaluation requirements not very clear– Show value added of center-mode
functioning– Present outcomes of center at site
visits• NSF need: To demonstrate to the
National Science Board the worth or value of the Science of Learning Centers
9
www.edvantia.org
Limited Resource Tensions
External Evaluator• Show merit or worth of
the center• Demonstrate how
center-mode functioning better than individual RO1 grants
• Focus on outcomes
Consultant & Critical Friend• Fix some process gaps• Work through the logic
model (are activities happening and resulting in expected outputs?)
• Attend to developmental stage of center
• Focus on process and some immediate outcomes10
www.edvantia.org
Not enough resources to attend to all needsCaption for this vacation photo: “Up a Tree”
11
www.edvantia.org
Conflicting Role Example
• SILC Year 3 Evaluation– Process focused– Organizational consulting– Conversation and informal feedback
and suggestions to Pis– SILC leadership satisfied– Site visit team wants outcome data
12
www.edvantia.org
Example
• SILC & VL2 Year 4 Evaluation– Outcomes oriented– Bibliometric analyses
• Limited corpus of publications• Not useful to center at this early stage
– Less feedback mid-year– PIs less satisfied– Fewer questions/critiques from site
visit team
13
www.edvantia.org
Balancing Tensions in a Most Unusual Way
• Years 6-7 will bring back focus on process (new 5-year funding cycle)
• Update logic model and key measures• Focus also on intermediate outcomes
(now appropriate in center’s lifecycle)• “SLC Evaluation Network” economies
of scale may result in better– center-specific evaluation– SLC program-wide evaluation
14
www.edvantia.org
Thank You!
Kristine Chadwick, Ph.D.Executive Director
Evaluation and Applied Research Initiatives
15
