Upload
ffranceschina
View
498
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Learning needs self assessment in International Foundation Programme students
Citation preview
Assessing IFP students’ needs assessment
Florencia Franceschina
Manchester Metropolitan University
Inform Conference,
University of Reading, 17 July 2010
1. Introduction
• Increased weight of the ‘student voice’ in HE (e.g., NSS, International Student Barometer, etc.)
• The ‘student voice’ is valued in different ways:– Students as a customers– Students as individuals capable of self-directed
learning
• Learning needs self-assessment relevant to either view
Defining self-assessment
“Self-assessment refers to the involvement of learners in making judgements about their own learning […]”.
Boud and Falchikov (1989: 529)
Why assess learning needs?
Because it can help us to identify:
• learners’ strengths and weaknesses
• learners’ goals and expectations
Issue #1: How useful is
learning needs self-assessment?
• Useful for policy makers
• Useful for curriculum developers
• Useful for teachers
• Useful for students
“Whether it be a junior high school student wondering whether to crack open his social studies textbook one more time before the test or a medical student deciding whether to practice her intubating technique before her next shift, students make more effective decisions about where to apply their learning efforts when they can actually discern their strengths and weaknesses.”
(Dunning et al., 2004: 85)
Issue #2: How valid is
learning needs (self-)assessment?
Do students know what they need?
Do even experts know which needs we should explore and aim to address? For example, the relevance of some needs may not be apparent at the time of assessment (Grant, 2002: 157).
Issue #3: How reliable is
learning needs self-assessment?
• Are all learners equally able to make accurate judgements about their learning needs?
• How good are IFP students at it?
2. Brief review of the literature
Self-assessment …
• … is difficult.
Meta-studies reveal typically low correlation between self-views and actual performance:
.21 (Hansford and Hattie, 1982)
.39 (Falchikov and Boud, 1989)
Similar findings by Dunning et al., 2003; Dunning et al., 2004; Powers, 2002; Ross, 1998.
Self-assessment …• … is difficult.• … is not always valued by learners
(e.g., compared to summative assessment)
Self-assessment …• … is difficult.• … is not always valued by learners
(e.g., compared to summative assessment)
• … is modulated by culture.
Self-regulation is subject to the influence of educational/cultural background factors
(e.g., Purdie and Hattie, 1996; Olaussen and Bråten, 1999)
Self-assessment …• … is difficult.• … is not always valued by learners
(e.g., compared to summative assessment)
• … is modulated by culture.• … is modulated by level of study.
Students in advanced courses tend to be more accurate self-assessors than students on introductory courses (Falchikov and Boud, 1989).
Self-assessment …• … is difficult.• … is not always valued by learners (e.g.,
compared to summative assessment)• … is modulated by culture.• … is modulated by level of study.• … is modulated by academic ability level.
“High-performing students were accurate, with accuracy improving over multiple exams. Low-performing students showed moderate prediction accuracy but good postdiction accuracy. Lowest performing students showed gross overconfidence in predictions and postdictions. Judgments of performance were influenced by prior judgments and not prior performance. Performance and judgments of performance had little influence on subsequent test preparation behavior.”
Hacker et al. (2000: 160)
Self-assessment …• … is difficult.• … is not always valued by learners (e.g.,
compared to summative assessment)• … is modulated by culture.• … is modulated by level of study.• … is modulated by academic ability level.• … is differently accurate depending on area
of study.
Students in science tend to be more accurate self-assessors than students in other areas (Falchikov and Boud, 1989).
Self-assessment …• … is difficult.• … is not always valued by learners (e.g.,
compared to summative assessment)• … is modulated by culture.• … is modulated by level of study.• … is modulated by academic ability level.• … is differently accurate depending on area of
study.• … can be developed as a skill.
Although student self-assessment remains inaccurate after training, it does seem to be subject to some improvement. For example:
McDonald (2010) has shown that involving students in meta-assessment training can help mathematics students to improve their test performance.
Chen (2008) found that through feedback and practice students’ self-assessments of English speaking skills become more aligned with tutor assessment.
Wide range of data collection methods available
• Formal / informal
• Planned / opportunistic
• Individual / group
• By self / peer / tutor / outside expert
Some commonly used tools
• Questionnaires• Focus groups• Online discussion fora• Staff-student liaison committees• Diaries• Objective tests• Observation• Action research
3. Methodology
• Cohort of 51 IFP students
• Min. language entry requirement: IELTS 5.5 or equivalent
• Followed up for the duration of IFP (2009-2010)
Mixed nationalities
• Saudi Arabia (22%)• Bahrain (14%)• Nigeria (14%)• Pakistan (8%)• Turkey (8%)• UAE (8%)• China (4%)• India (4%)
• Iran (4%)• Sri Lanka (4%)• DR Congo (2%)• Hong Kong (2%)• Kenya (2%)• Turkish Cyprus (2%)• USA (2%)• Zimbabwe (2%)
Mixed linked degrees
Science / Engineering
53%Business31%
HLSS14%
Other2%
Data collected
• Learning needs questionnaire– Student version (individual self-assessment)– Staff version (group assessment)
• Diagnostic language test• Attendance records• Coursework and exam results (all IFP
modules)• Staff interviews• End-of-year module feedback (EAP module)
Analyses
• Perceived needs vs. observed academic performance
• “Objective” diagnostic measures vs. observed academic performance
4. Findings
Overview
1. “Objective” measures
2. “Subjective” measures
3. Improvements in judgements?
Correlations between ‘objective’ measures
Lang test at entry
EAP unit mark
IFP mark
EAP attend.
IFP attend.
Lang test at entry
- .22 .36
EAP unit mark
- - .81 .67
IFP
mark
- - - .65
EAP attend.
- - - -
IFP attend.
- - - - -
Correlation between language test at entry and EAP unit marks
0
10
20
30
40
50
60
70
80
0 20 40 60 80 100 120 140 160 180
OPT scores at entry
EAP unit marks
Correlation between language test score at entry and overall IFP mark
0102030405060708090
100
0 20 40 60 80 100 120 140 160 180
OPT score at etnry
Overall FY mark
Correlations between ‘objective’ measures
Lang test at entry
EAP unit mark
IFP mark
EAP attend.
IFP attend.
Lang test at entry
- .22 .36
EAP unit mark
- - .81 .67
IFP
mark
- - - .65
EAP attend.
- - - -
IFP attend.
- - - - -
Correlation between EAP unit mark and IFP overall mark
0
10
20
30
40
50
60
70
80
0 10 20 30 40 50 60 70 80 90
EAP unit mark
IFP overal mark
Correlations between ‘objective’ measures
Lang test at entry
EAP unit mark
IFP mark
EAP attend.
IFP attend.
Lang test at entry
- .22 .36
EAP unit mark
- - .81 .67
IFP
mark
- - - .65
EAP attend.
- - - -
IFP attend.
- - - - -
Correlation between EAP attendance and EAP mark
0102030405060708090
100
0 10 20 30 40 50 60 70 80
EAP mark
EAP attendance
Correlation between IFP attendance and overall IFP mark
0102030405060708090
100
0 10 20 30 40 50 60 70 80 90
IFP overall mark
Attendance %
How did “objective” measures work?
• Language test at entry was poor predictor or later performance. This could be due to a number of reasons. (Similarly surprising patterns reported for UCAS tariff points/FY performance elsewhere.)
• Attendance was reasonably good predictor. This confirms findings in previous studies (e.g., Hughes, 2009).
• EAP unit was best predictor of IFP performance.
Staff perceptions about students’ needs at entry
Tutors most frequently identified general language weakness as an issue to be addressed.
“when marking essays, it is obvious that many students have a very weak grasp of syntax sometimes, and this will hold them back in the future.”
“Sometimes there have been some difficulties in their ability to express themselves effectively in written work but this is not always the case. The main problem I have found with international students who do have language difficulties is their lack of commitment in addressing the issue or them denying the fact there is a problem despite overwhelming evidence and continuous advice to attend support classes. Some students believe their English skills are better than they truly are.”
“Required much repetition on basic tasks appear not to understand instructions.”
“there are others who have a very poor command of the English language and struggle with their research and assignments.”
Tutors and support staff thought the IFP students’ numeracy skills and subject background knowledge was on the whole the same or better than in UK students.
Relative difficulty of IFP modules (MMU ‘units’)
Mean unit marks per unit type
0
10
20
30
40
50
60
70
L-type units X-type units N-type units
Mean unit marks
Spread of unit marks per unit type
0102030405060708090
100
1 8 15 22 29 36 43 50 57 64 71 78 85 92 99 106 113
Individual unit marks
L-type unit marks X-type unit marks N-type unit marks
Students’ perceptions about their needs at entry
Indirect method of calculating match between students’ perceived needs and actual performance:
1. Self-assessment questionnaire answers coded as 1 if student ticked ‘I need to study more of this’/’I need help with this’, and 0 if student ticked ‘I don’t want much of this’/’I don’t need help with this’
2. EAP unit summative assessment marks were coded as 0 if mark was above mean and 1 if below mean.
3. Match between 1 and 2 above was checked.
Maximum possible score for successful matching of needs and actual performance: 5
(W, R, S, L, IT)
IFP students’ average score: 2.5
Perceived vs observed order of difficulty of subskills
• Predicted by students:W < IT < L < S/R
• Observed:W (38%) < R (43%) < S (50%) < L (52%) < IT (80%)
Correlation between correct prediction index for subskills and actual IFP overall marks
0
1
2
3
4
5
0 10 20 30 40 50 60 70 80 90
IFP overall mark
Correct prediction index
Students’ perceptions about their needs at exit
Needs questionnaire (at exit) answers match performance in subskills at an average score of 2.3 (no improvement)
End-of-year student feedback on EAP unit
0
1
2
3
4
5
6
7
8
9
10
Q1a. vocabulary quizzes
Q1b. glossaries and essay planning notebook
Q1c. vocabulary notebook
Q1d. textbook
Q1e. IT (orange) bookletQ1f. materials on WebCT
Q1g. speaking practice and assessmentQ1h. listening practice and assessmentQ1i. reading practice and assessmentQ1j. writing practice and assessment
Q1k. IT practice and assessment
Question
Average score
How useful?
How enjoyable?
Learning need as judged by students at the end of the IFP
Higher Lower
Speaking Listening
Reading
Writing
IT
EAP unit performance
Assessment area
Mean scores
SD Median
IT 80 27 87
Listening 52 14 53
Speaking 50 15 52
Reading 43 16 47
Writing 38 21 40
5. Summary and Conclusion
• Not all objective measures that seem relevant make useful predictions.
• Tutors and support staff’s global, subjective assessment of students’ learning needs seems to be accurate.
• These IFP students do not seem to be very good at assessing their learning needs in detail, but they are relatively better when their judgements are considered at a more general, group level.
• Given how unreliable IFP students’ judgements seem to be, we should be careful when using these judgements to inform our practice.
• The highest value of learning needs self-assessment is probably for the students themselves.
6. Practical suggestions
• Bear in mind that learning needs self-assessment is difficult, especially for IFP students.
• When interpreting IFP students’ needs self-assessment, remember that it can be influenced by:– cultural factors – academic ability level (good vs. weak students)– area of study (science vs. other)– assessment context (formative/summative; high/low
stakes)
• Students become more accurate with training and practice, so provide them with opportunities to develop this skill.
• Close the feedback loop with staff and students, for example through:– As part of regular assessment feedback– Websites, VLEs, notice boards– Programme committee meetings– Informal reporting by staff in class
• Whenever possible, combine self-assessment with other sources of information about your students’ learning needs.
• Take advantage of the wide range of data collection tools available.
• Methodological tip: ask students to make self-assessment judgements using same scales as ‘objective’ performance measures you will use to compare. (This has its own problems, though.)
ReferencesBoud, David, and Falchikov, Nancy. 1989. Quantitative studies of student self-assessment in higher
education: a critical analysis of findings. Higher Education 18:529-549. Chen, Yuh-Mei. 2008. Learning to self-assess oral performance in English: a longitudinal case study.
Language Teaching Research 12:235-262. Dunning, David, Heath, Chip, and Suls, Jerry M. 2004. Flawed self-assessment. Implications for health,
education and the workplace. Psychological Science in the Public Interest 5:31-54. Dunning, David, Johnson, Kerri, Ehrlinger, Joyce, and Kruger, Justin. 2003. Why people fail to recognize
their own incompetence. Current Directions in Psychological Science 12:83-87.Falchikov, Nancy, and Boud, David. 1989. Student self-assessment in higher education: a meta-analysis.
Review of Educational Research 59:395-430.Grant, Janet. 2002. Learning needs assessment: assessing the need. British Medical Journal 324:156-
159. Hacker, Douglas J, Bol, Linda, Horgan, Dianne D, and Rakow, Ernest A. 2000. Test prediction and
performance in a classroom context. Journal of Educational Psychology 92:160-170.Hughes, Nicola T. 2009. Attendance as a measure of student motivation and engagement. InForm 3:7-8.McDonald, Betty. 2010. Improving learning through meta assessment. Active Learning in Higher
Education 11:119-129. Olaussen, Bodil S, and Br 蚯 en, Ivar. 1999. Students' use of strategies for self-regulated learning: cross-
cultural perspectives. Scandinavian Journal of Educational Research 43:409-432.Powers, Donald E. 2002. Self-assessment of reasoning skills. ETS Research Report RR-02-22.
Princeton, NJ: Educational Testing Service (ETS). Purdie, Nola, and Hattie, John. 1996. Cultural differences in the use of strategies for self-regulated
learning. American Educational Research Journal 33:845-871. Ross, Stephen John. 1998. Self-assessment in second language testing: a meta-analysis and analysis of experiential factors. Language Testing 15:1-20.