Embed Size (px)
Citation preview
PRACTICE MAKES PRACTICE MAKES PROGRESS: REFLECTIONS PROGRESS: REFLECTIONS OF EVALUATORS IN OF EVALUATORS IN TRAINING AND TRAINING AND SIGNIFICANCE TO THE SIGNIFICANCE TO THE FIELDFIELDKarl Lipovsek, Steven Christiansen, Cortney Sabin, Jack Karl Lipovsek, Steven Christiansen, Cortney Sabin, Jack Lazorik, and Tiffany SmithLazorik, and Tiffany Smith
University of Wisconsin—Stout University of Wisconsin—Stout
REFLECTIVE PRACTICEREFLECTIVE PRACTICE• One of six essential competenciesOne of six essential competencies• Relates to an evaluator’s awareness of personal Relates to an evaluator’s awareness of personal
expertise, need for growth, and engagement in expertise, need for growth, and engagement in professional developmentprofessional development
““We learn more about We learn more about ourselves throughout the ourselves throughout the
process [of reflective process [of reflective practice] about our practice] about our
strengths, weaknesses, strengths, weaknesses, biases, etc.”biases, etc.”
——Student Student
COLLABORATIVE REFLECTIVE COLLABORATIVE REFLECTIVE PRACTICEPRACTICE• Useful in classroom contextUseful in classroom context• Students share ideas with each otherStudents share ideas with each other• Explore ideas outside of classroom settingExplore ideas outside of classroom setting
““Reflective practice involves a willingness to actively participate in Reflective practice involves a willingness to actively participate in a perpetual growing process requiring ongoing critical reflection a perpetual growing process requiring ongoing critical reflection on both classroom practices and core beliefs” (Larrivee, 2010, p. on both classroom practices and core beliefs” (Larrivee, 2010, p.
293-307).293-307).
OUR PROJECTOUR PROJECT• Two classes over the course of one yearTwo classes over the course of one year• Online discussion forumsOnline discussion forums• Themed responses for reflective practice behaviorsThemed responses for reflective practice behaviors
““Patton is saying [that] we need to Patton is saying [that] we need to prevent stagnant thoughts. Doing this prevent stagnant thoughts. Doing this reflective practice, thinking about what reflective practice, thinking about what we have done, and how to be stronger we have done, and how to be stronger
and all around better.”and all around better.”——StudentStudent
OUR PROJECTOUR PROJECT• What we will be What we will be
discussing…discussing…• Themes discoveredThemes discovered• Relation to reflective Relation to reflective
practicepractice• Importance to student Importance to student
learninglearning• All quotes from this All quotes from this
point on are from fellow point on are from fellow students unless students unless otherwise notedotherwise noted
META EVALUATIONMETA EVALUATION““Without checking on our own Without checking on our own
evaluation skills, there would be evaluation skills, there would be no accountability for error. No no accountability for error. No
accountability would be accountability would be devastating to the field, devastating to the field,
because if we were no longer because if we were no longer viewed as evaluation experts, viewed as evaluation experts,
who would be?”who would be?”““The struggles of keeping true to The struggles of keeping true to
the research results, and the research results, and conveying them to a group who conveying them to a group who
has stake in the project is a topic has stake in the project is a topic I'll continue to investigate, as I I'll continue to investigate, as I
know my future work will involve know my future work will involve these moments of concern or these moments of concern or
disagreement.”disagreement.”
META EVALUATIONMETA EVALUATION
““Nobody will want to hire an Nobody will want to hire an evaluator whose knowledge evaluator whose knowledge won't help solve a problem. won't help solve a problem.
This is why meta evaluation is This is why meta evaluation is super important: to stay super important: to stay
current with changing times.”current with changing times.”
““As knowledge seekers, we are As knowledge seekers, we are continuously learning newer, continuously learning newer,
innovative approaches so that innovative approaches so that we will be the best evaluators we will be the best evaluators we can be for our stakeholders, we can be for our stakeholders,
through professional through professional development opportunities, development opportunities,
reflective practices, and meta-reflective practices, and meta-evaluations.”evaluations.”
COLLABORATIONCOLLABORATION
““My group has come up with My group has come up with a pretty KICK ASS evaluation a pretty KICK ASS evaluation plan and project only to have plan and project only to have
it knocked down and it knocked down and changed probably over 40 changed probably over 40
times and sometimes we go times and sometimes we go back to the original way or back to the original way or
even come up with new even come up with new ideas we thought were ideas we thought were
impossible in the first place.”impossible in the first place.”
““The most positive aspect of this The most positive aspect of this project so far is having teammates project so far is having teammates to bounce ideas off of and keep our to bounce ideas off of and keep our attitudes upbeat even when things attitudes upbeat even when things
take a turn for the worst.”take a turn for the worst.”
““It has been fun, although at times It has been fun, although at times challenging, to collaborate and challenging, to collaborate and
really listen and understand one really listen and understand one another in order to make a decision another in order to make a decision
on how to move forward. This is on how to move forward. This is one of my favorite parts of group one of my favorite parts of group
work.” work.”
COLLABORATIONCOLLABORATION “ “I always feel that when we I always feel that when we
really listen to each other, and really listen to each other, and build off each others ideas, we build off each others ideas, we ultimately end up with a better ultimately end up with a better
idea than one of us could idea than one of us could have come up with on our have come up with on our
own.”own.”
““It is cool to be working with It is cool to be working with individuals who have strong individuals who have strong
work ethics and WANT to work ethics and WANT to succeed and gain as much succeed and gain as much knowledge as possible.” knowledge as possible.”
““Our group has done a good job delegating Our group has done a good job delegating work to individuals and then coming together work to individuals and then coming together
to finalize things with consensus. I feel like to finalize things with consensus. I feel like we've learned a lot about the benefits of this we've learned a lot about the benefits of this
collaboration in planning an evaluation.”collaboration in planning an evaluation.”
LEARNINGLEARNING““The most positive The most positive
experience I have gotten experience I have gotten out of the evaluation has out of the evaluation has
been going from a position been going from a position of confusion and disarray, of confusion and disarray,
to getting a hold of the to getting a hold of the situation and developing a situation and developing a
plan that is do-able.”plan that is do-able.”
““I now see the importance I now see the importance evaluation utilization has evaluation utilization has
for the evaluator, for the evaluator, stakeholders, program, stakeholders, program, organization, and pretty organization, and pretty much anyone else who much anyone else who
comes in any proximity of comes in any proximity of the project.”the project.”
QUESTIONS/UNCERTAINTIESQUESTIONS/UNCERTAINTIES
““What do you think could happen if the description you
What do you think could happen if the description you
come up with for the evaluand is not what *some*
come up with for the evaluand is not what *some*
stakeholders expected? How do we facilitate that
stakeholders expected? How do we facilitate that
discussion?”discussion?”
““How involved SHOULD
How involved SHOULD [stakeholders] be in this
[stakeholders] be in this process?”process?”
““Now
Now
what?”
what?”““How are evaluators
How are evaluators
certain of what to
certain of what to
measure, or that what we
measure, or that what we
are measuring is
are measuring is correct?”correct?”
INTROSPECTIONINTROSPECTION““I love running stats and I love running stats and
making interpretations out making interpretations out of them. I am however not of them. I am however not
looking forward to data looking forward to data collection because it's not collection because it's not fun, based off my previous fun, based off my previous experiences, and the high experiences, and the high
attrition rate of e-mail attrition rate of e-mail surveys really scares me.”surveys really scares me.”
““I was naive to think that I was naive to think that evaluations are almost always evaluations are almost always going to be straightforward. going to be straightforward.
Sometimes the client and the Sometimes the client and the evaluator will not be on the evaluator will not be on the
right track.”right track.”
““I think I have always had the frame of I think I have always had the frame of mind to use the results of an evaluation.”mind to use the results of an evaluation.”
LOOKING BACKLOOKING BACK
““It would be difficult for us as It would be difficult for us as professionals to learn and professionals to learn and
grow if we refuse to reflect on grow if we refuse to reflect on what we could have done to what we could have done to make the process smoother make the process smoother
and so on.”and so on.”
““If I were to travel back in time I If I were to travel back in time I would impart some of the would impart some of the
perspective that we all have perspective that we all have gained up to this point about gained up to this point about what realistic expectations for what realistic expectations for
the project are. - Our the project are. - Our expectations, the client expectations, the client
expectations, how to collect data expectations, how to collect data from this group, time from this group, time
management skills, proper management skills, proper distribution of the work, clearer distribution of the work, clearer plans for group work, perhaps a plans for group work, perhaps a
system to encourage system to encourage participation. There are dozens participation. There are dozens
of things.”of things.”
LOOKING BACKLOOKING BACK““In retrospect, simulating the use of the In retrospect, simulating the use of the evaluation findings would have been a evaluation findings would have been a
great idea to help us bridge any gap that great idea to help us bridge any gap that might have been missing between what our might have been missing between what our stakeholder might have wanted out of the stakeholder might have wanted out of the evaluation and what we think she wants.”evaluation and what we think she wants.”
““Retrospectively, I think a Retrospectively, I think a conversation with PIUs would conversation with PIUs would have been helpful, but it may have been helpful, but it may
have widened the scope again to have widened the scope again to outside of what we are able to outside of what we are able to
do under time pressure.”do under time pressure.”
QUANTITATIVE DATAQUANTITATIVE DATA
Data showed a Data showed a 22% increase in 22% increase in posts read posts read
QUANTITATIVE DATAQUANTITATIVE DATA
38% increase in 38% increase in replies made replies made between the 2 between the 2 semesterssemesters
BUT WHY?BUT WHY?““When building a space When building a space ship you want precise ship you want precise instruments. When instruments. When evaluating the team that evaluating the team that makes the spaceship, makes the spaceship, you will also need you will also need precise tools. Serious precise tools. Serious reflective practice and reflective practice and meta evaluation help meta evaluation help hone these tools.”hone these tools.”
Karl LipovsekKarl [email protected]
Steven ChristiansenSteven [email protected]
Cortney SabinCortney [email protected]
Jack Lazorik Jack Lazorik [email protected]
