John Ogier

Making summative unit surveys totally summative - risks or opportunities

Survey Timing

• Collings AHEEF 2007

• I misheard David’s method!!

• What are the implications of a long survey window?

• Into and after exams?

• Post grades?

• Response demographics??

• Immediate response

• Post exams

• Post grades

• General - gender, grades etc

• Online Survey Experiment

• Wanted a wide range of large “units”

• Divide each randomly into 4 groups

• But high female response rate AUSSE 2007 - so maintain gender balance

• Got small selection of units

• Group 1 - normal pre-exams 15 days

• Group 2 - day after exam 15 days

• Group 3 - day after grades 13 days

• Group 4 - open entire 7 weeks

• Day 1 - personalised invitation email

• Week before close - reminder email

Also some other units - MATH200, ECON200 and 300

Responses

Group 1 Group 2 Group 3 Course W1 Rem Non Total W1 Rem Non Total W1 Rem Non Total

AFI S204 11 6 69 86 21 9 56 86 20 5 61 86 ECON104 24 16 126 166 39 11 116 166 32 13 120 165 ECON105 2 3 33 38 3 4 31 38 5 2 32 39

Total 37 25 228 290 63 24 203 290 57 20 213 290 Group 4 W1 Mid Rem Non Total ECON104 36 12 8 112 168 ECON105 6 0 2 30 38

Total 42 12 10 142 206 Group 1 Group 2 Group 3

Others W1 Rem Non Total W1 Rem Non Total W1 Rem Non Total Excluded 100 74 515 689 45 26 217 288 53 20 213 286

(Whole Yr Group 4 Pass-Fail W1 Mid Rem Non Total

Teaching) 108 21 64 552 745

Table 1: Response Count by Group & Response Period

Variables

C_FdBck “I received helpful feedback on my progress”

C_Asmt “The assessments in this course measured my learning effectively”

C_OAll “Overall, this was a good quality course”

T_OAll “Overall, the lecturer is an effective teacher”

Gender F, M

Group 1, 2, 3, 4 (Survey Group)

NumGrade -1 to 9 (“A+” = 9 through to “E” = -1)

PostGrades PG (responded post grades/mid-year test return), NA (prior)

Week W1 (responded Week 1), Rem (post-reminder email), NA (between)

Also knew: -

Ethnicity, International Student status, Citizenship, Age, intended Qualification, Overall GPA, Response Day & Time and Time taken

First analysis

C_O

All

1

2

3

4

5

1 2 3 4

Group

Group Number Mean Std Error Lower 95% Upper 95% 1 178 3.80337 0.06818 3.6695 3.9372 2 157 3.93631 0.07260 3.7938 4.0788 3 150 4.02000 0.07427 3.8742 4.1658 4 254 3.90551 0.05708 3.7935 4.0176

Table 2: C_OAll Means for One-way ANOVA

What about grades?C_F

dB

ck

1

2

3

4

5

1 2 3 4

Group

Each PairStudent's t 0.05

C_A

smt

1

2

3

4

5

1 2 3 4

Group

Each PairStudent's t 0.05

C_O

All

2

3

4

5

1 2 3 4

Group

Each PairStudent's t 0.05

T_O

All

1

2

3

4

5

1 2 3 4

Group

Each PairStudent's t 0.05

Figure 4: ANOVA – C_FdBck, C_Asmt, C_OAll and T_OAll by Group

The dangers of “means”

• Dommeyer, Baum, Hanna & Chapman, 2004 - online surveys “do not produce significantly different mean evaluation scores” from paper

• UC Paper and Online survey history support this

• Not just means - also Std Deviations

• But are the responses “representative of the whole group”?? (Nulty, 2008)

Gender - a response bias?

p N p N N p N p N N p N p N NGroup 1 48 7 38 10 17 96 14 70 26 40 22 2 16 3 5Group 2 48 12 38 18 30 98 29 68 21 50 20 3 18 4 7Group 3 47 11 39 14 25 89 13 76 32 45 30 5 9 2 7Group 4 N/A N/A N/A N/A N/A 94 30 74 26 56 24 4 14 4 8

Total 143 30 115 42 72 377 86 288 105 191 96 14 57 13 27

p% N% p% N% N% p% N% p% N% N% p% N% p% N% N%Group 1 19% 3% 15% 4% 7% 14% 2% 11% 4% 6% 14% 1% 10% 2% 3%Group 2 19% 5% 15% 7% 12% 15% 4% 10% 3% 8% 13% 2% 12% 3% 5%Group 3 18% 4% 15% 5% 10% 13% 2% 11% 5% 7% 20% 3% 6% 1% 5%Group 4 N/A N/A N/A N/A 14% 5% 11% 4% 8% 16% 3% 9% 3% 5%

Total 55% 12% 45% 16% 28% 57% 13% 43% 16% 29% 63% 9% 37% 8% 18%

N x Gender 21% 37% 23% 36% 15% 23%p (M/F) 1.24 1.31 1.68N (M/F) 0.57 0.63 0.64

AFIS204 (258) ECON104 (665) ECON105 (153)

Survey Response Numbers and Perecentages by Gender

Male FemaleMale Female Male Female

No significant diff @ 95% CI in Likert means or across Groups by Gender

What about grades?

Across the main units analysed:

Bivariate fits

Correlations for “Overall” Qtns

• Course R2 = 0.047

• Teaching R2 = 0.0045

A+ is 9 ………… E is -1

1

2

3

4

5

C_O

All

-1 0 1 2 3 4 5 6 7 8 9NumGrade

1

2

3

4

5

T_O

All

-1 0 1 2 3 4 5 6 7 8 9NumGrade

Broken into Groups

Survey timing now apparent!

Group 2 - post exams

Group 3 - post grades

Why not the full picture?

Week=W1 Distributions GradeGrp

A B C D

Frequencies Level Count Prob A 69 0.34673 B 64 0.32161 C 35 0.17588 D 31 0.15578 Total 199 1.00000

Week=Rem Distributions GradeGrp

A B C D

Frequencies Level Count Prob A 14 0.17722 B 23 0.29114 C 27 0.34177 D 15 0.18987 Total 79 1.00000

Where to from here?

Oct 6 - online surveys - 40 units

Group 2 runs until 15th Dec. Grades released 5th Dec

Pre-exam Post-exam Post-grades

Group 1 - Control 11487 Standard Course Survey - no demographic questions

Group 2 - hint3469 Standard Course Survey - no demographic questions

Group 3 - no hint 3 3RGroup 4 - no hint 4 4RGroup 5 - no hint 5R

1472 Course Survey plus demographic questions1626 Course Survey plus demographic questions1305 Course Survey plus demographic questions

M=5660, F=3699, T=9359

5

2