Upload
matthew-apple
View
215
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Testing & evaluation (TEVAL) SIG workshop at JALT 2014 (Tsukuba), November 22.
Citation preview
+Don’t walk:
Rasch to
join
the
questionnai
re trend!
Ritsumeik
an
University
Keita
Kikuchi
Kanagaw
a
University
J. W.
Lake
Fukuoka
Women’s
University
+
Get ready to be Rasched…
(This is supposed to be a joke. Laugh.)
Structure of
this
workshop
Ritsumeik
an
University
Keita
Kikuchi
Kanagaw
a
University
J. W.
Lake
Fukuoka
Women’s
University
+
Rasch concepts
Terminology and explanations
Structure of
this
workshop
Ritsumeik
an
University
+
Steps to create a
questionnaire
Constructs, concepts, items, piloting, evaluation,
revision…
Structure of
this
workshop J. W.
Lake
Fukuoka
Women’s
University
+
Demotivation: An example
The flaws of traditional FA and why Rasch can
help questionnaire creation
Structure of
this
workshop
Keita
Kikuchi
Kanagaw
a
University
Teacher
Behavior
GETA
e1
.69
MET
e2
.85
ONE
e3
.75
PRON
e4
.69
EXPL
e5
.77
Environment
AUD
e14
TOPC
e13
INTR
e12
.76.74.67
VID
e15
.70
GRAM
e6
Experience
of difficulties
VOCI
e7
VOC
e8
TEST
e9
SELF
e10
.72 .68 .66.68
Lack of
Interest
NOP
e22
NON
e21
NOG
e20
NOI
e19
NOF
e18
.82
.83.77.78.81
NOTU
e11
.74
demotivation
.74
.68
.69
d1
d2
d3
d4
.66
FRN
e16
.64
MST
e17
.61
.50
GFI=.911
NFI=.905
CFI=.920
RMSEA=.064
SRMR = .055
+
Matthew Apple
Ritsumeikan University
Department of Communication
International Communication Program
Rasch
concepts
+
Rasch terms
Rasch log-odds (logits)
Rasch measures (logit scores)
Infit/Outfit (means sq. and z-score)
Item difficulty / endorsability
Person/item reliability / separation
Construct validity and
unidimensionality (not strictly
speaking Rasch, but…)
Rasch Principal components
analysis (Rasch PCA)
Loadings
Contrasts and Residuals
A probabilistic model
+Logits (log-odds)
The probability of a person correctly
answering an item 50% of the time
+Fit
0.75 to 1.3 logits
0.60 to 1.4 logits
“Within 2 standard deviations of the mean”
“Only Outfit z-scores of 3.0”
+Separation
The number of groups distinguishable by the
measurement instrument
(Wilson, 2005;
Wright, 1996)
The ratio of error-free variance and
observed variance
(Fisher, 1992)
+Item map
Persons and items on the same
linear logit scale
+
The “Line”Items and persons on the same scale
Item-person map (or)
Wright map
Ben Wright
+
The “Line”Items and persons on the same scale
Item-person map (or)
Wright map
+
How Rasch works
for questionnaires
Likert-scale data
Likert-type category data
Questionnaires do not produce
true interval but ordinal data
The steps in the “scale” can be
conceived as thresholds (τ)
named after Thurston, originator
of factor analysis)Refer to Andrich (1977, 1978)
+
How Rasch works
for questionnairesRefer to Andrich (1977, 1978)
+
How Rasch works
for questionnairesRefer to Andrich (1977, 1978)
N
1
2
3
4
5
+
How Rasch works
for questionnairesRefer to Andrich (1977, 1978)
N
1
2
3
4
5
1 + 3 = 4
+
How Rasch works
for questionnairesRefer to Andrich (1977, 1978)
N
1
2
3
4
5
1 + 3 = 4?
SD + N = A?
+
J W Lake
Fukuoka Women’s University
Steps to
create a
questionnaire
Steps in scale development: Issues to
consider (Netemeyer, Bearden, Sharma,
2003)
Step 1: Construct definition and content domain
The importance of clear construct definition,
content domain, and the role of theory.
Construct dimensionality: unidimensional,
multidimensional, or a higher-order construct?
Determine the purpose of the scale:
measurement or correlational analysis or
model building
Step 2: Generating and judging measurement items
Theoretical assumptions about items (e.g., domain
sampling)
Generating potential items and determining the
response format
How many items as an initial pool
Dichotomous vs. polytomous response formats
Item wording issues
The focus on “content” validity in relation to
theoretical dimensionality
Item judging (expert and layperson) --- the focus on
“content” and “face” validity
Step 3: designing and conducting studies to develop
and refine the scale
Pilot testing as an item-trimming procedure
The use of several samples from relevant populations for scale development
Designing the studies to test psychometric properties
Initial item analyses via exploratory factor analyses (EFAs)
Initial item analyses and internal consistency estimates
Retaining items for the next studies
EFA may be useful for correlational analysis or model building
Step 4: Finalizing the scales
The importance of several samples from relevant populations
Designing the studies to test the various types of validity
Item analysis via EFA
The importance of EFA consistency from Step 3 to Step 4
Deriving an initial factor structure—dimensionality and theory
Item analyses and confirmatory factor analyses (CFAs)
Testing the theoretical factor structure and model
+
Keita Kikuchi
Kanagawa University
Demotivatio
n: An
example
Teacher
Behavior
GETA
e1
.69
MET
e2
.85
ONE
e3
.75
PRON
e4
.69
EXPL
e5
.77
Environment
AUD
e14
TOPC
e13
INTR
e12
.76.74.67
VID
e15
.70
GRAM
e6
Experience
of difficulties
VOCI
e7
VOC
e8
TEST
e9
SELF
e10
.72 .68 .66.68
Lack of
Interest
NOP
e22
NON
e21
NOG
e20
NOI
e19
NOF
e18
.82
.83.77.78.81
NOTU
e11
.74
demotivation
.74
.68
.69
d1
d2
d3
d4
.66
FRN
e16
.64
MST
e17
.61
.50
GFI=.911
NFI=.905
CFI=.920
RMSEA=.064
SRMR = .055
Example Study
focused on specific external /internal forces that Japanese high school students may experience which might cause their motivation to be reduced or diminished
administered the questionnaire asking high school students to report what diminished their motivation to study in their high school days, which contained 40 Likert-scale questions (4-points)
analyzed the quantitative data using a confirmatory factor analysis using Amos and Rasch PCA of the residuals using Winsteps.
If you’d like to read this process thoroughly, please locate Kikuchi (forthcoming).
Demotivation
Dörnyei (2001)
Definition of demotivation
“specific external forces that reduce or
diminish the motivational basis of a
behavioral intention or an ongoing action” (p.
143).
I expand this definition and explore
demotivators including both internal and
external forces.
Previous Studies (Dörnyei, 1998)
Based on interviews with 50 secondary school students, he identified following as demotivators, the reason to get demotivated.
1. Teachers’ personalities, commitments, competence, teaching methods.
2. Inadequate school facilities (very big group, not the right level or frequent change of teachers).
3. Reduced self-confidence due to their experience of failure or success.
4. Negative attitude toward the foreign language studied.
5. Compulsory nature of the foreign language study.
6. Interference of another foreign language that pupils are studying.
7. Negative attitude toward the community of the foreign language spoken.
Previous Studies (Kojima, 2004, p.42)
English
Learning
Demotivation
Languag
e Level
Learning
situation
Amoun
t of
study
.58
.86
.71
.46
.41
Gramm
ar
Readin
g
Self-
confidenc
e
Writing Learning
method
Memori
zing
Vocab.
Learner
Level
Listening
problem
Class
atmospher
e
Change
of teaching
style
Teaching
approach
Teacher
personality.89
.77
.89
.89
.85 .90 .92 .80 .83 .85
GFI = 0.906
AGFI = 0.890
RMSEA = 0.052
Previous Studies
Kikuchi (2009)
47 university students
open-ended questionnaires
reflection on high school days
Kikuchi and Sakai (2009)
112 university students
a 35-item questionnaire with a 5-point scale
Sakai and Kikuchi (2009)
676 high school students
a 35-item questionnaire with a 5-point scale
Common demotivation factors Sakai and Kikuchi(2009)
F1: Learning Contents and Materials,
F2: Teachers’ Competence and Teaching Styles
F3: Inadequate School Facilities [Classroom Environment]
F4: Lack of Intrinsic Motivation
F5: Test Scores [Experience of Inferiority]
Kikuchi and Sakai (2009)
F1: Course Books
F2: Inadequate School Facilities
F3: Test Scores
F4: Non-Communicative Methods
F5: Teachers’ Competence and Teaching Styles
Both studies used a principal axis factor analysis using the direct
oblimin rotation
Six Original Constructs Teachers: Teachers’ attitude, teaching competence,
language proficiency, personality, and teaching style
Characteristics of classes: Course contents and pace,
focus on difficult grammar or vocabulary, monotonous and
boring lessons, a focus on university entrance exams and
the memorization of the language
Experiences of failure: Disappointment due to test scores,
lack of acceptance by teachers and others, and feeling
unable to memorize vocabulary and idioms.
Class environment: Attitude of classmates, compulsory
nature of English study, friends’ attitudes, inactive classes,
inappropriate level of the lessons, and inadequate use of
school facilities such as not using audio-visual materials
Class materials: Not suitable or uninteresting materials (e.g.,
too many reference books and/or handouts)
Lack of interest: Sense of English used at schools is not
practical and not necessary. Little admiration toward English
Participants (N=1,266)
MethodMaterials
background questions
a 40-item questionnaire “We would like to study the situations of English study in high schools.
The following statements are possible demotivating factors for English learning. To what extent are these statements true for you? Answer based on your experience.”
Questions are revised from what I used in my previous studies (Kikuchi and Sakai, in-press; Sakai and Kikuchi, 2009).
Example of Items (1=Strongly disagree, 2=disagree, 3=agree, and 4=Strongly agree) Teachers made one-way explanations too often.
The number of students in classes was large.
A great number of textbooks and supplementary readers were assigned.
I lost my understanding of the purpose of studying English.
I could not do as well on tests as my friends.
Results of EFA
An Exploratory Factor Analysis
40 items
principal axis factor analysis with a promax rotation procedure
a four-factor solution
teachers behaviors
class environment
experiences of difficulties
Lack of interest
Only 22 items left to be included in Confirmatory Factor Analysis.
Table 2:Factor Analysis of DemotivationNo. Item descriptions F 1 F 2 F 3 F 4
Factor 1: Experience of difficulties(α = .87)
i16 There were too many vocabularies that I did not understand in reading. 0.81 0.03 -0.14 -0.11
i15 I had difficulty in memorizing words and phrases. 0.80 -0.13 -0.01 -0.08
i13 I got low scores on tests (such as mid-term and final examinations). 0.79 -0.17 0.06 -0.09
i8 I did not understand grammar even though I studied. 0.75 -0.02 -0.19 0.05
i39 I started not to understand the content of the class. 0.71 0.09 -0.09 0.08
i7 I could not do as well on tests as my friends. 0.62 -0.11 0.10 -0.09
i33 I got lost in how to self-study for English lessons. 0.60 -0.02 0.03 0.09
Factor 2: Teacher behavior(α = .84)
i18 I thought that the approach that teacher used was not good. -0.05 0.93 -0.14 0.00
i5 Teachers' explanations were not easy to understand. 0.03 0.86 -0.22 0.02
i17 Teachers made one-way explanations too often. 0.06 0.78 -0.07 -0.02
i6 Teachers' pronunciation of English was poor. -0.10 0.73 0.02 -0.07
i34 I could not get along with teachers. -0.10 0.68 0.12 0.05
i31 The pace of lessons was not appropriate. 0.04 0.63 0.06 -0.03
Factor 3: Class environment(α = .85)
i28 Audio materials (such as CDs and tapes) were not used. -0.08 -0.01 0.82 -0.06
i23 The Internet was not used. -0.12 -0.16 0.81 0.04
i27 Topics of the English passages used in lessons were old. 0.04 0.02 0.71 -0.11
i35 Visual materials (such as videos and DVDs) were not used. -0.03 0.05 0.69 0.04
i10 My friends did not like English. 0.02 -0.05 0.67 -0.07
i40 The number of students in classes was large. -0.03 -0.11 0.65 0.13
Factor 4: Lack of Interest(α = .90)
i3 I lost my understanding of the purpose of studying English. -0.13 0.00 0.01 0.91
i2 I lost my goal to be a speaker of English. -0.08 -0.04 0.03 0.85
i26 I think that I will not use English in my future. 0.03 0.01 -0.10 0.81
i11 I don’t have specific goals for studying English. 0.04 -0.04 0.01 0.79
i24 I lost my interest in English. 0.12 -0.02 0.01 0.71
Method
Analysis
Rasch PCA of the residuals/Confirmatory
factor analysis of these six factors
Rasch PCA factor analysis /Confirmatory
factor analysis of these four factors were
conducted once again…
Conventional factor analysis confirmed
only four factors!
+
Enter the RaschWith Matt
+
Category utility
Measures the distance between
thresholds among the Likert-type
categories (“steps” of the scale)
+
Rasch PCA output
Loading
Measure
Infit means squared
Outfit means squared
Principal components analysis
+
Item fit analysis
Measure
Standard error
Infit Outfit means squared & z
+
Keita Kikuchi
Kanagawa University
Demotivatio
n, Part
Deux: The
Rasched
Teacher
Behavior
GETA
e1
.69
MET
e2
.85
ONE
e3
.75
PRON
e4
.69
EXPL
e5
.77
Environment
AUD
e14
TOPC
e13
INTR
e12
.76.74.67
VID
e15
.70
GRAM
e6
Experience
of difficulties
VOCI
e7
VOC
e8
TEST
e9
SELF
e10
.72 .68 .66.68
Lack of
Interest
NOP
e22
NON
e21
NOG
e20
NOI
e19
NOF
e18
.82
.83.77.78.81
NOTU
e11
.74
demotivation
.74
.68
.69
d1
d2
d3
d4
.66
FRN
e16
.64
MST
e17
.61
.50
GFI=.911
NFI=.905
CFI=.920
RMSEA=.064
SRMR = .055
Results
Rating Scale Instrument Quality Criteria (based on Fisher, 2007)
Criterion Poor Fair Good Very Good Excellent
Item Model Fit Mean-Square < 0.33 - >3.0 0.34 - 2.9 0.5 - 2.0 0.71 - 1.4 0.77 - 1.3
Person and item measurement
reliability <.67 .67-.80 .81-.90 .91-.94 >.94
Variance in data explained by
measures <50% 50-60% 60-70% 70-80% >80%
Unexplained variance in 1st
contrast of PCA residuals >15% 10-15% 5-10% 3-5% <3%
Table 1: Variance in measure explained by each demotivator
construct.
Six Demotivator
Constructs
Variance
explained by
measure
Unexplained
variance by
measure
Unexplained
variance
explained by 1st
contrast
Teachers 57.0% 43.0% 12.3%
Characteristics of classes 45.4% 54.6% 9.7%
Experiences failure 57.1% 42.9% 12.7%
Class environment 40.4% 59.6% 13.3%
Class materials 55.1% 44.9% 10.8%
Lack of interest 62.1% 37.9% 12.8%
No. Item description Logit
Score
Infit
MNSQ
Outfit
MNSQ
Factor
loadings
1. teachers(k=6, Rp=0.99, Gp=11.02)
15 Teachers shout or got angry 0.97 1.42 1.38 0.68
5 Teacher asked us to use accurate grammar 0.54 1.26 1.25 0.68
14 Teachers explanation not easy -0.85 0.79 0.85 -0.63
11 Teachers bad pronunciation 0.16 0.98 0.91 -0.45
40 Teachers’ bad teach method -0.31 0.75 0.72 -0.44
13 Teachers one-way explanation -0.51 0.83 0.84 -0.03
2. Characteristics of Classes (k=9, Rp=0.98, Gp=6.53)
10 Inappropriate pace of lesson -0.10 1.00 1.04 0.58
41 Monotonous class -0.22 0.98 1.04 0.58
1 Rare chance of communication -0.14 1.19 1.24 0.45
2 Focused on translation 0.06 0.81 0.83 -0.57
3 Focused on grammar -0.35 0.96 1.01 -0.56
42 Amount to study for mid-term/final tests -0.32 1.19 1.29 -0.20
6 Required memorizing passages in textbooks 0.26 0.94 0.95 -0.15
43 Amount of handout distributed 0.11 0.94 0.95 -0.13
4 Focused on college entrance exam. Prep. 0.69 0.90 0.89 -0.12
Results –Rasch PCA- Loadings of
Contrasts
No. Item description Logit
Score
Infit
MNSQ
Outfit
MNSQ
Factor
loadings
3. Experiences of Failure (k=6, Rp=0.99, Gp=8.89)
27 Did not do well on tests compared w friends 0.64 1.04 1.06 0.76
8 low scores on school test 0.25 0.84 0.84 0.68
37 Did not understand grammar -0.15 1.07 1.07 -0.64
36 Did not understand class 0.23 1.01 1.01 -0.56
9 Get lost in self-study -0.26 1.05 1.07 -0.12
7 Could not memorize vocabulary and idiom -0.71 0.95 0.96 -0.05
4. Class Environment (k=6, Rp=0.98, Gp=7.40)
31 English being compulsory subject -0.73 1.08 1.20 0.82
26 Too many students in class 0.00 1.05 1.00 0.23
22 Video and DVDs not used -0.41 0.91 0.90 0.05
25 Audio not used 0.04 0.87 0.85 -0.59
29 Friends did not like English 0.67 0.97 1.03 -0.42
23 Internet not used 0.43 1.08 0.99 -0.31
No. Item description Logit
Score
Infit
MNSQ
Outfit
MNSQ
Factor
loadings
5. Class Materials (k=6, Rp=1.00, Gp=7.30)
16 Topics of Passages uninteresting 0.14 1.06 1.09 0.57
20 Topics of Passages old 1.42 0.96 0.95 0.56
35 Unclear answers to Questions 0.22 1.01 1.03 0.41
19 too much reading 0.44 0.97 1.00 0.19
44 Many difficult Vocabulary -1.21 0.98 1.00 -0.56
18 Sentences were difficult to read -0.97 0.98 1.03 -0.55
17 Passages too long -0.04 1.00 1.01 -0.47
6. Lack of Interest (k=5, Rp=0.98, Gp=6.87)
34 No goal for being a person who can use Eng 0.31 0.92 0.89 0.75
39 No need studying English 0.63 0.85 0.84 0.73
33 Lost interest in English -0.30 1.15 1.14 -0.56
32 Lost purpose of study English -0.59 1.06 1.06 -0.42
46 No use of English in the future -0.05 0.97 0.95 -0.31
Figure 1: CFA of 6 factor models of demotivators
Notes.
GFI=800
CFI=808
RMSEA=
.072
teacher
i36e1
.56
i34e2
.73i30e3
.60
i18e4.82
i17e5 .76
i6e6.68
i5e7
.74
lesson
i29e8
i25e9
i22e10
i21e11
i20e12
i19e13
i4e14
.61
.58
.60
.59
.55
.60
.47
i31e15
.59
i32e16
.46
environment
i10 e17
i23 e18
i28 e19
i35 e20
i37 e21
i40 e22
.63
.67
.71
.70
.65
.62
material
i1 e23
i9 e24
i12 e25
i14 e26
i16 e27
i27 e28
.53
.57
.50
.56
.59
.57
interest
i2 e29
i3 e30
i11 e31
i24 e32
i26 e33
.82
.83
.77
.79
.81
failure
i39
e35
i33
e36
i15
e37
i13
e38
i8
e39
i7
e40
.74.68.67.71.67.60
i38 e41
.68
.79
.37
.27
.59
.54
.70
.56
.93
.79
.54
.88
.52
.73
.63
.57
Figure 2.
CFA of
4 factor
models
Notes.
GFI=916
CFI=926
RMSEA=
.062
Teacher
Behavior
GETA
e1
.69
MET
e2
.85
ONEW
e3
.75
PRON
e4
.69
EXPL
e5
.77
Environment
AUD
e14
TOPC
e13
INTR
e12
.76.74.67
VID
e15
.70
GRAM
e6
Experience
of difficulties
VOCI
e7
VOC
e8
TEST
e9
SELFS
e10
.71 .67 .66.68
Lack of
Interest
NOP
e22
NON
e21
NOG
e20
NOI
e19
NOU
e18
.82
.83.77.79.81
NOTU
e11
.75.67
FRN
e16
.64
MST
e17
.61
.46
.51
.44
.53
.23
.35
Teacher
Behavior
GETA
e1
.69
MET
e2
.85
ONE
e3
.75
PRON
e4
.69
EXPL
e5
.77
Environment
AUD
e14
TOPC
e13
INTR
e12
.76.74.67
VID
e15
.70
GRAM
e6
Experience
of difficulties
VOCI
e7
VOC
e8
TEST
e9
SELF
e10
.72 .68 .66.68
Lack of
Interest
NOP
e22
NON
e21
NOG
e20
NOI
e19
NOF
e18
.82
.83.77.78.81
NOTU
e11
.74
demotivation
.74
.68
.69
d1
d2
d3
d4
.66
FRN
e16
.64
MST
e17
.61
.50
GFI=.911
NFI=.905
CFI=.920
RMSEA=.064
SRMR = .055
Figure 3.
My tentative
model of
demotivation
Results –Rasch PCA of four factor
models-
Four Demotivator
Constructs
Variance
explained by
measure
Unexplained
variance by
measure
Unexplained
variance explained
by 1st contrast
Teachers 58.5% 41.5% 10.7%
Experience of difficulties 58.0% 42.0% 12.2%
Class environment 39.9% 60.1% 13.3%
Lack of interest 54.8% 45.2% 15.1%
This is how the poor factor was
working…
An activity for this workshop
Let’s try to make item bank of questionnaire items together for your practice.
Topic is Demotivating factors in English education for communication in Japan.
With your partner, think of constructs first and write items for each construct. How many constructs? How many items for each of them? Remember “the Line”!
Please use the questionnaire items that you have in your handout about demotivating factors in English education in high school English classroom to generate your discussion.
References
Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum.
Dörnyei, Z. (2001). Teaching and researching motivation. Harlow: Longman.
Fisher, W. P. (2007). Rating Scale Instrument Quality Criteria. Retrieved November 25, 2007, from http://www.rasch.org/rmt/rmt211m.htm
Kikuchi, K. (2009). Student demotivation in Japanese high school English classrooms: Exploring with qualitative research methods. Language Teaching Research, 13(4), pp.453-471.
Kikuchi, K. (forthcoming). What are possible demotivators in SLA? –An insight from English teaching contexts in Japan. Multilingual Matters
Kojima, S. (2004). English learning demotivation in Japanese EFL students: Research in demotivational patterns from the qualitative research results of three different types of high schools. Unpublished master thesis, Kwansei Gakuin University, Hyogo, Japan.
Linacre, J. M. (1997). Guidelines for rating scales. Retrieved November 25, 2007, from http://www.rasch.org/rn2.htm.
Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). Scaling procedures: Issues and applications. Thousand Oaks, CA: Sage Publications.
Sakai, H., & Kikuchi, K. (2009). Japanese learners' demotivation to study English: A survey study? JALT Journal, 31 (2), pp.183-204.
Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum.
Q & A and discussion
The Rasch model
(Rasch, 1960)
The Rating Scale Model
(Andrich, 1978)
+Don’t walk:
Rasch to
join
the
questionnai
re trend!
Ritsumeik
an
University
Keita
Kikuchi
Kanagaw
a
University
J. W.
Lake
Fukuoka
Women’s
University
Send future inquires to Keita: