Upload
tamsin-day
View
218
Download
4
Tags:
Embed Size (px)
Citation preview
UMCP Educational Technology Outreach On-line Course Assessment
and Evaluation Model
2005 EDUCAUSE Mid-Atlantic Regional Conference
Baltimore, Maryland January 12, 2005
University of MarylandEducational Technology OutreachDirector: Davina Pruitt-Mentle
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 2
Statement of the ProblemStatement of the Problem
Technologically mediated instruction offered at a distance has rapidly become an important component of higher education. However, faculty/instructors often find it difficult to assess student learning and outcomes when they have little, if any face to face interaction with participants.
While student assessment through distance learning poses a challenge, a focus on interactivity, design strategies that help decrease student anxiety level, and multi-pronged activities and assessment measures can provide unique opportunities for detailed feedback to learners.
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 3
ObjectivesObjectives
Discussion of lessons learned from the evolution of the development and delivery of the University of Maryland, College of Education, Educational Technology Outreach On-line Course Assessment and Evaluation Model.
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 4
ETO Online Course Assessment ETO Online Course Assessment and Evaluation Modeland Evaluation Model
1. How will our courses provide evidence of student content knowledge and skill competency to meet the NCATE review board requirements? and
2. Have the course content and activities actually changed participant behaviors in the classroom/workplace?
Resulted from two overarching questions:
NCATE=National Council on Accreditation of Teacher Education
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 5
Evaluation StrategyEvaluation Strategy
• Based the conceptual framework on Kirkpatrick's Training Model (level III)
• Assessment and evaluations of the on-line courses are systematic and ongoing
• Are used to improve the content and delivery of the on-going and future courses
• Both formative and summative evaluation protocols and instruments are used to guide the content and delivery
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 7
All Courses IncludeAll Courses Include
• An on-line pre-profile “assessment”
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 8
PT3 Pre-Assessment SurveyPT3 Pre-Assessment Survey
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 9
All Courses IncludeAll Courses Include
• An introduction discussion thread - helps the instructor find out more about the particular interest and background of the participants
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 10
All Courses IncludeAll Courses Include
Throughout the course real time chat room dialogue allows participants to express comments and concerns regarding the course
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 12
Formative Content Knowledge Formative Content Knowledge Assessments Assessments
• On-going
• Include a variety of formats– on-line discussion threads– mini-assignments– group work– chat room discussions– projects and papers– on-line self tests and quizzes – graded quizzes and tests
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 13
Formative Content Knowledge Formative Content Knowledge AssessmentsAssessments
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 14
Importance of FeedbackImportance of Feedback
Feedback gathered from assignments has helped in the redesign of course activities (both in the on-going course and in redesign of the same course offered later)
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 15
Multiple Summative Multiple Summative EvaluationsEvaluations
• Multiple summative evaluations are also used. – the post-profile assessment, – the University’s evaluation instrument which
has been recreated in the online format and can be submitted anonymously,
– Two (now one) open ended discussion questions and
– Completion of the How I as a Student Meet Standards Matrix
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 16
Post-Profile AssessmentPost-Profile Assessment
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 17
Post-Profile AssessmentPost-Profile Assessment
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 18
Post-Profile AssessmentPost-Profile Assessment
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 19
University’s Evaluation Instrument University’s Evaluation Instrument
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 20
Open Ended Discussion Open Ended Discussion QuestionsQuestions
• Subject: Experience Reflection Message no. 378 Author: Davina S. Pruitt-Mentle (EDUC477)Date: Sunday, November 30, 2003 9:50pm
• Please share your experiences with this course this semester. What did you expect? What did you get out of the course? What will you remember months from now? (you can either post to the discussion thread, or send an email to [email protected]).
• Please give a critical analysis of this course, course content, course experience (including the online format). What formatting features in the instructional design would make it easier to navigate (given the WebCT format)? How many "discussions, mini exercises" etc… do you feel are appropriate each week? (remember this is a graduate course and we do not meet the 3 full hours F2F or online each week). Did you prefer the assignments separate from the discussion or embedded within the discussion section? Please indicate anything else that would be helpful towards presenting this information to others. It is understood that everyone here is an adult and professional and therefore critical analysis is appreciated and in no means will be reflected in the grade. (You can either post to the discussion thread, or send an email to [email protected])
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 21
Open Ended Discussion Open Ended Discussion QuestionsQuestions
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 22
How I as a Student Meet How I as a Student Meet Standards MatrixStandards Matrix
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 23
Changed Behaviors?Changed Behaviors?
• To answer the second question, “Have the course content and activities actually changed participant behaviors in the classroom/workplace”?
• ETO uses follow-up surveys to see if and how students have continued to apply the knowledge/skills after course conclusion. This is done 1-2 semesters after completion of the course (via email).
EDUCAUSE MARC Conference 2005 (c) Davina Pruitt-Mentle
January 12, 2005 24
• End slide
Contact Information:Davina Pruitt-Mentle
Director: Educational Technology Policy, Research and Outreach
University of Maryland2127 TAWES
College Park, MD 20742(301) 405-8202
Questions?Questions?