Upload
laurel-horton
View
226
Download
4
Embed Size (px)
Citation preview
IDEA Student Ratings of Instruction
Shelley A. Chapman, PhD
InsightImprovementImpact®
University of Alabama BirminghamSeptember 11, 2012
Plan for this Session• What Makes IDEA Unique• Conditions for Good Use• Reflective Practice Framework• Student Learning Framework• Faculty Information Form• Interpreting Reports• Questions and Answers
• Individual Development and Educational Assessment
• Kellogg Grant in 1975• Non-profit Organization 2000• Mission
InsightImprovementImpact®
To help colleges and universities as they seek to improve teaching, learning, and leadership
What makes IDEA unique?
1. Focus on Student Learning
2. Focus on Instructor’s Purpose
3. Adjustments for Extraneous Influences
4. Validity and Reliability
5. Comparison Data
6. Flexibility
Conditions for Good Use
The instrument • Focuses on learning • Provides suggested action steps for teaching improvement
Conditions for Good Use
The Faculty• Trust the process• Value student feedback• Are motivated to make improvements
Conditions for Good Use
Campus Culture•Teaching excellence - high priority•Resources to improve - provided•Student ratings - appropriate weight
Conditions for Good Use
The Evaluation Process• 30-50% of evaluation of teaching• 6-8 classes, more if small (<10)• Not over-interpreted
(3-5 performance categories)
Reflective Practice using Individual Reports
IDEA resources that are keyed to reports
Talk with colleagues
Try new ideasOnline, Paper
What the reports sayand what they mean
Student Learning Framework: 2 Assumptions
Assumption 1:
Types of learning must reflect the instructor’s purpose.
Student Diagnostic Form
Assumption 2:
Effectiveness determined by students’ progress on objectives stressed by instructor
Student Learning Model
Specific teaching behaviors are associated with certain types of student progress under certain circumstances.
Student LearningTeaching Behaviors
Circumstances
Student Learning Model: Diagnostic Form
Student LearningItems 21-32
Teaching Behaviors
Items 1-20
CircumstancesStudents: Items 36-39, 43
Course: Items 33-35
Summary Items: 40-42Research Items: 44-47Up to 20 extra items
Student Learning Model: Short Form
Summary Measures: Items 16-18
Experimental Questions: Items 14
20 Additional Questions
Student LearningItems 1-12
Teaching Behaviors
CircumstancesStudents: Items 13-15
Faculty Information Form(FIF)
FIF: Selecting Objectives
• 3-5 as “Essential” or “Important”
• Is it a significant part of the course?
• Do you do something specific to help students accomplish the objective?
• Does the student’s progress on the objective influence his or her grade?
Be true to your course.
The Average Number of Objectives Selected by UAB: Spring 2012
Common Misconception #1
Students are expected to make significant progress on all 12 learning objectives in a given course.
Common Misconception #2
Effective instructors need to successfully employ all 20 teaching methods in a given course.
Relationship of Learning
Objectives to Teaching Methods
Common Misconception #3
The 20 teaching methods items should be used to make an overall judgment about teaching effectiveness.
Faculty Evaluation
Course Description Items (FIF)• Used for research• Best answered toward end of term• Do NOT influence your results
Bottom of Page 1
Top of page 2
IDEA Online
• Email delivery/reminders• Start/end dates determined by Institution
• Access is unlimited while available
• Questions can be added to student survey
• Objectives can be copied from previously completed FIFs
IDEA Online: FIF Delivery
Copying Objectives
• Email/Course embedded URL
• Blackboard Building Block• Email reminders• Start/end dates Determined by Institution
• Submission is confidential and restricted to one
IDEA Online: Student Survey Delivery
Online Response Rates – Best Practices
• Create value for student feedback
• Monitor and Communicate through multiple modalities:• Twitter• Facebook• Other
• Prepare Students• Talk about it• Syllabus
Example: Course Syllabus
Objective 3: Learning to apply course material (to improve thinking, problem solving, and decisions)
Students will be able to apply the methods, processes, and principles of earth science to understanding natural phenomenaStudents will think more critically about the earth and environment
Objective 8: Developing skill in expressing myself orally or in writing
Students will be able to present scientific results in written and oral forms
IDEA Center Learning Objective
Course Learning Outcomes
Reflective Practice with IDEA
IDEA Student Ratings of Instruction
Individual ReportsGroup Summary ReportsBenchmarking Reports
Diagnostic Report Overview
Page 1 – Big Picture How did I do?
Page 3 – Diagnostic What can I do differently?
Page 2 – Learning Details What did students learn?
Page 4 – Statistical Detail Any additional
insights?
Your Average (5-point Scale)
Raw Adj.
A. Progress on Relevant Objectives1
Four objectives were selected as relevant (Important or Essential—see page 2)
4.1 4.3
1If you are comparing Progress on Relevant Objectives from one instructor to another, use the converted average.
The Big Picture
ProgressOnRelevantObjectives
4
4.3 + 4.34.14.23.6
5
Summary Evaluation: Five-Point Scale
Report Page 1
Your Average Score
(5-point scale)
Raw Adj.
A. Progress on Relevant ObjectivesFour objectives were selected as relevant (Important or Essential—see page 2)
4.1 4.3
Overall Ratings B. Excellent Teacher 4.7 4.9
C. Excellent Course 4.1 4.4
D. Average of B & C 4.4 4.7
Summary Evaluation(Average of A & D) 4.3 4.5
50%
25%
25%
Adjusted Scores
Adjusted Scores
• Student Work Habits (#43DF, #13SF)• Student Motivation (#39DF, #15SF)• Class Size (Enrollment, FIF)
Understanding Adjusted Scores
Work Habits
(Item 43)
Student Motivation (Item 39)
HighHigh Avg.
Avg.LowAvg.
Low
High 4.48 4.38 4.28 4.13 4.04
High Avg. 4.38 4.29 4.14 3.96 3.76
Average 4.28 4.14 4.01 3.83 3.64
Low Avg. 4.15 4.05 3.88 3.70 3.51
Low 4.11 3.96 3.78 3.58 3.38
Impact of Extraneous Factors• Gaining Factual Knowledge – Average Progress Ratings
Technical Report 12, page 40
Work Habits
(Item 43)
Student Motivation (Item 39)
HighHigh Avg.
Avg.LowAvg.
Low
High 4.48 4.38
High Avg. 4.38 4.29
Average 4.01
Low Avg. 3.70 3.51
Low 3.58 3.38
Impact of Extraneous Factors• Gaining Factual Knowledge – Average Progress Ratings
Technical Report 12, page 40
Doraw
scores meet or exceed
expectations?*
Are adjusted
scores lower or higher than
raw scores?
Use adjusted
scores
Useraw scores
Lower Yes
Higher
When to Use Adjusted Scores for Personnel Decisions
*Expectations defined by your unit.
No
Comparisons (Norms): Converted Averages
• Able to compare scores on the same scale• Use T Scores
• Average = 50• Standard Deviation = 10
• They are not percentiles or percentages
Comparisons (Norms): Converted Averages
Comparison Scores Distribution
40%
Similar
Lower20%
10%Much Lower
10%Much Higher
20%Higher
Gray Area
Comparison Scores
Using the Report to Improve Course Planning and Teaching
Page 2: What did students learn?
Suggested Action Steps
#16#18#19
POD-IDEA NotesIDEA Website
POD-IDEA Notes
• Background• Helpful Hints• Application for online learning
• Assessment Issues• References and Resources
References and Linksto Helpful Resourcesare Provided
IDEA Papers
Resources for •Faculty Evaluation•Faculty Development
Reflective Practice
POD-IDEA Notes IDEA Papers
Meet with colleaguesto reflect
Interpret Reports
Paper or Online
Try something new
Questions ?
www.theideacenter.org
Visit our IDEA Help Community!