Interpreting IDEA reports

Diagnostic FormShort Form

www.lvc.edu/idea

Page 1 of report

Page 1 of report

Guide says: “Report Responds toThese Questions”

1. Overall, how effectively was this class taught?2. How does this compare with the ratings of other teachers?3. Were you more successful in facilitating progress on some class objectives than on others?4. How can instruction be made more effective?*5. Do some salient characteristics of this class and its students have implications for instruction?* Diagnostic Form Report only.

Our Modest Ambitions Today

• Understand summative scores– Raw averages– Converted scores– Adjusted scores– How used at LVC

• Formative use of p.2 – relevant objectives– POD papers on objectives

• Formative use of p.3 for Diagnostic only– POD papers on strategies

Summative scores ….

How “Summary Evaluation” Computed

• 41. Overall, I rate this instructor an excellent teacher. (25%)

• 42. Overall, I rate this course as excellent. (25%)

• Objectives 1-12 (21.-32) that are ranked as Important or Essential (50%)

“Converted averages”

• Transform group’s mean to 50, s.d. to 10.• Examples:

– Score 57 means .7 group s.d.’s above average– That’s at 75.8 %-tile of Normal Distribution– … but these groups of IDEA scores not Normal

25 30 35 40 45 50 55 60 65 70 750

1

2

3

4

5

6

NormalIDEA

Adjusted Scores:Effects that are removed

P. 2 of Diagnostic Form report:

Adjusted Scores:Effects that are removed

P. 2 of Diagnostic Form report:

Only these on Short Form

Is unadjustedScore on

Progress towardObjectives high?

(≥4.2)

Are adjusted

scores lower or higher than unadjusted

scores?

Use adjusted scores

Useunadjusted

(raw) scores

Lower Yes

Higher

When to Use Adjusted Scores for Administrative Decisions:Where is there evidence of effective teaching? (old)

*Expectations defined by your unit.

No

Dounadjusted (raw) scores meet or

exceed expectations?*

Are adjusted

scores lower or higher than unadjusted

scores?

Use adjusted scores

Useunadjusted

(raw) scores

Lower Yes

Higher

When to Use Adjusted Scores for Administrative Decisions:Where is there evidence of effective teaching? (new)

*Expectations defined by your unit.

No

College Use for Evaluating Faculty

• Current:– Summative: Pay attention to extremes– Assess formative use of IDEA by instructor

• IDEA recommendation– Summative: Establish standards

• Faculty set standards• Department level or College level• Assess progress in meeting them

Page 2: Objectives

P. 3 of Diagnostic Form Report:Directions….

www.theideacenter.org/podidea

Follow the links…

Teaching Methods Relevant toTarget Objectives

Teaching Methods Relevant toTarget Objectives

Try it1. Identify objective…

2. Find related teaching method…

3. Follow link: www.theideacenter.org/podidea

Last page: Raw response frequencies… especially additional questions

We can retrieve the additional questions so you know which are which.

Effects of Evals being On-Line?

• Study of on-line vs paper by IDEA published in 2010:– http://

www.theideacenter.org/sites/default/files/Technical Report 16.pdf

• Online avg response rate 57%– LVC response rates 70-75%

• No significant change in means or s.d.’s of responses– If loss of middle group, s.d. would go up.– On 12 objectives, s.d.’s slightly lower for on-line!

Key questions for LVC (from IDEA)

• How much emphasis will you give student ratings in the evaluation of teaching?

• How many classes need to be rated? [LVC: all]• Performance categories: How many should you use?• Do you plan to authenticate objectives?• Will you weight the importance of items or groups of

items in the decision-making process? If so, how?• Will you use averages on a five-point scale or converted

averages?• Will you use adjusted or unadjusted scores?

References

• www.lvc.edu/idea – Campus IDEA page• POD notes:

– The 20 methods: www.theideacenter.org/podidea– The 12 objectives:

www.theideacenter.org/podidea/PODNotesLearning.html

• IDEA: www.theideacenter.org• Technical: http://

www.theideacenter.org/sites/default/files/techreport-12.pdf - Data about adjusting, etc.