Autonomy as a predictor
of English proficiency
Matthew T. AppleNara National College of Technology
Introduction keep it short!
Personal interest in autonomy since 2003, autonomy conference in
HK and Zhejiang U, paper in MAYA
Japanese educator's interest in autonomous learning
NNCT interest in autonomy, jishu
But does autonomy actually help students get better at English?
Let's find out.
Background
Participants
Methods
Results
Four main sections, hope to spend roughly 4 minutes per
section.
Any questions, please keep until the end of the presentation
Run out of time? Send me an email.
Participants
Methods
Results
Background
First start with the background of the study
In particular, the theories about autonomy and its relationship
to proficiency
Autonomy = ?
First question: what is autonomy?
Holec (1981) primarily from a political view
i.e., who has control in the classroom
David Little (1991), Leni Dam and others produced a series of
small books (most of which are out of print) theorizing that
autonomy as learner control of their own goals, materials, and
reflections.
Benson (2001) neatly summarized all the autonomy ideas to date
(sheep?)
Materials
Learningprocesses
Metacognitiveprocesses
Benson's 2001 synthesis of learner autonomy.
3 aspects.
What7s the difference between learning processes and
metacognitive processes?
Problem: autonomy is a multidimensional construct.
No, not a construct, but a concept.
Constructs must be unidimensional. Where they intersect is the
latent variable (statistcially speaking). And we need at least
three (any two constructs can correlate w/ little meaning).
People and chairs = construct?
Qualitative
Quantitative
Benson himself noted a lack of evidence about studies connecting
autonomy w/ proficiency
Two main type sof studies (gross simplication!);
The great qual-quan debate (perhaps a false dichotomy, but
certainly true in autonomy studies.)
Qual = mostly case studies, perhaps as many as 3 or 4, usually
highly advanced learners in ESL universities or gradu programs.
Quan = large numbers of students, again typically at
universities, but relatively few studies in autonomy (Cotterall,
1997, was a learner belief study, not strictly autonomy. Yang
(2006) was also beliefs.
The good learner
The good learner of Naiman (1985), Wen & Johnson (1995),
etc.
The garden path:
They were successful learners who used autonomous strategies,
therefore autonomous students will become proficient.
Ignores other individual differences such as language aptitude,
motivation, anxiety, educational backgrounds, gender, age,
socioeconomic factors, etc.
Not generalizable across learner populations.
Generalizability?
Generaliability needs greater numbers.
Case studies can't demonstrate that most students will follow
the same path.
We need more quan studies to improve the generalizability that
autonomy can predict proficiency for ALL ESL or EFL learners.
Hypothesis:Autonomy leads to English proficiency
So, let7s start with the hypothesis above.
Let's assume that it7s true, autronomy will lead directly to
greater English proficiency.
All we need now is a group of students, measuring instruments,
and analytic techniques. (oy)
Background
Methods
Results
Participants
Next, participants in the study.
The study mostly consisted of high school-age studnets; however,
the learning context is somewhat complicated, as well as unique to
Japan.
185 technical college Ss
The participants were students at a national technical
college.
The college consisted of 7 grades.
1 through 3 grade are equivalent of the three years of hs in
Japan, with a marked increase in technical classes (i.e., math,
physics, engineering prep) and a corresponding decrease in general
ed classes (i.e., English and japanese).
164 students in the study were in the 3rd grade, or high sch
senior equivalent. (I hesitate to say equivalent, b/c they tend to
take classes at the uni sopho or junior level, at least!)
21 were in the 6th grade, or senkoka 1st year. This is the
equivalent age of uni 3rd yr. However, they tend to take classes
close to grad school level.
M
E
S
C
I
5 departments represented by the participants in roughly equal n
umbers
400
Magic number for engineering students
Why?
J
A
B
E
E
TOEIC 400 is required to receive JABEE nintei, or certification
as an engineer
Japan
Accreditation
Board for
Engineering
Education
JABEE means...
The technical college joined the JABEE program in 2005 and has
been struggling to meet the English requirement, mostly due to a
combination of lack of student motivation and the few number of
classes...
Studnets only take 5 English classes of 45 minutes in grades 1
and 2, and only 3 classes per week in 3rd year.
Sen 1 only take 2 classes per week.
400
Magic number is --
315
Participant TOEIC average was this...a bridge too far?
How can they get to 400?
67.5
In only 67.5 hours per year?
100 points = 100 hours?
100 points = 400+ hours (Boldt & Ross, 1997)
No reliable figures, but regardless, many studnets have
difficulty increasing 85 points on average.
Background
Participants
Results
Methods
Methods = once more, into the breach
Reasons for ethods, as opposed to actual methods
Those interested in the hw of stats stick around later or send
an email!
Autonomy = ?
Let's go back...
Autonomy is what?
Materials
Learningprocesses
Metacognitiveprocesses
A
Autonomy has three aspects.
IOW, there are three measured constructs that form the concept
or latent variable of autonomy
Autonomy is a multidimensional construct.
Benson (2001)
Again, the terminology is not entirely accurate as far as stats
are concerned.
Rubik's has multiple faces, but we don't consider it to be
multidimensional. We do however, consider each side separately when
solving the cube...and yet, all sides are obviously related.
Trying to solve all faces at the same time leads to an
incomoplete cube...
Construct = ?
Another metaphor...weight lifting...
Each part of the body can be worked on separetly, despite the
fact that all part sof the body are obviously related.
For example, here is somebody usig the benchpress.
What's the construct?
Yes, chest or pectoral muscles. And yet obviously, we need our
arms and shoulders as well. But the pecs are the focus.
Construct = ?
How about now?
Yes. The biceps, or arm muscles. Yet we also need forearm
muscles, triceps, and shoulders. Biceps are the focus.
Construct = ?
Now? Yes, the shoulders. I think you see the point.
Construct = ?
Now what are we working on?
We don't know for sure, b/c we are using legs, back, arms, and
shoulders...leading to a severe back strain eventually! And we
don't develop what we want...ie., the biceps.
So, it order to properly evaluate, and improve, the construct we
want to measure, we need to isolate as much as possible a single
costruct, BEFORE we put all the costructs together to form a
related whole.
Materials
Learningprocesses
Metacognitiveprocesses
A
Thus, autonomy may consist of RELATED constructs, howver EACH
construct must be measured separately/
So, how do we do this?
Rasch item fit analysis
First, check the items for item fit to the intended construct.
Does each iterm measure a a different level of difficulty on the
contsurct? i.e., can it distinguish between participants who have
more or less autonomy?)
Factor analysis
(factor scores)
Then, double check to confirm that there are separet, but
related constructs in the data. FA works well here, and factor
scores will make the raw data into stanardized scores for later
analysis.
Multiple regression analysis
Here's the meat of the analysis: do the independent variables of
autonomy predict the dependent variable of English proficneyc?
Materials
Learningprocesses
Metacognitiveprocesses
A
A quick model: here are the three proposed constructs that
measure autonomy
English Proficiency
Learning
Metacognitive
Materials
And here's the model of how they predict English proficneyc
TOEIC
Shimo (2008)
And here are the mesuring intsurments.
Shimo (2008) has 18 items...she itended originally to find 4
factors, but she found 2 factors in one study and 3 in another. How
many are there, rteally
TOEIC was used for proficiecy measurement...more on that
later...
Background
Participants
Methods
Results
You can see the statistical results on your handout. Table 1
shows the item fit stats for the Rasch analysis
Table 2 shows the monte carlo parallel analysis to help confirm
how many factors there were in the data
Table 3 shows the traditional FA as another confirmation (you
can see some items were mixed)
Table 4 shows the final regression analysis
Here7s the summary!
Lack of item fit to intended constructs
5 items did not fit the intended constructs. 4 of the five items
contained words such as English ability, mistakes, or tests.
Students distracted by these words? Maybe another construct
COULD have existed with a few more items to tease it out...
But in the end the three hypothesized constructs...
Materials
Learningprocesses
Metacognitiveprocesses
A
Of Benson (2001)...turned into...
Materials
Metacognitiveprocesses
Asomething resembling the Minshuto symbol..
learning processes and metacognitive processes fell into the
same construct.
In Shimo this was termed Orientation toward reflective learning,
and included goal setting, plannig, and reflectig on mistakes.
So, in a sense, it make sense...are control over learning
processes different from metacognitive processes?
In the end, the FA also bore this out: only 2 factors.
Orientation towards improving learning environment
Orientation towards improving learning processes
Asomething resembling the Minshuto symbol..
learning processes and metacognitive processes fell into the
same construct.
In Shimo this was termed Orientation toward reflective learning,
and included goal setting, plannig, and reflectig on mistakes.
So, in a sense, it make sense...are control over learning
processes different from metacognitive processes?
In the end, the FA also bore this out: only 2 factors.
Orientation towards improving learning environment
11% of TOEIC
Autonomy predicted 14% of TOEIC differences, and 11% alone was
finding materials.
(3% was control over learning processes.)
Increasing 1 SD of autonomy would lead to 24 points on
TOEIC.
Not bad, right?
Uh huh...one problem...
400
Second problem:
Magic number of TOEIC 400...however...
350 450
Error!
TOEIC error margin is AT LEAST 25 points per section (some say
35).
So, TOEIC 400 really is a range of scores between 350 ane
450...
And this is only a 68% chance as well..
hmmm...not exactly a good measurement of English proficney
perhaps...
So is the study fatally flawed?:
1. TOEIC-IP is only passive.
2. All students were engineering majors.
3. 92% were male.
4. Other possible causes of TOEIC differences:
Language aptitude (particularly linguistic coding?)Multiple
intelligencesMotivationField IndependenceTest anxietyetc.
Generalizability
Planning for improvement
Good first step.
More students. Close to 300 or 400 would be ideal.
Refine the constructs.
Pre and post...AFTER encourahing autonomy?
i.e., I am now using the language portfolio provided by the FLP
SIG, will this encourage autonomy? Will this mprove English
proficiency?
Stayed tuned!
Autonomy as a predictor of
English proficiency
Matthew T. [email protected]
3rd yrSenkoka 1
Row 11641.#NAN
Row 2211.#NAN
Column 1
170
15
