UKFIET International Conference on Education and Development

15-17 September, 2009

Oxford, England

The Politics of Formal Examinations:

language policy and progress

ConvenorsPauline Rea-Dickins

Guoxing Yu

2 The symposium

• Rationale: all researching in contexts where:– English as an L2/L3 is used in high stakes

formal examining contexts: SSA, Malaysia, USA & UK

– Politics & policy impacts on issues of fair & ethical assessment

– Policy is intended to lead to progress & equitable distribution of resources & opportunities for ALL learners

– Progress?

The symposium

Dr Ong Saw Lan (Universiti Sains Malaysia )

Narrowing the science achievement gap with the

dual-language test-booklet

Professor Jamal Abedi (University of California,

Davis)

Impact of language factors on the assessment

outcomes of English language learners

4UKFIET International Conference on Education and Development

15-17 September, 2009

Oxford, England

The Politics of National Examining Processes:

the case of languages in Zanzibar

Abdulla H. Mohamed, Pauline Rea-Dickins, Guoxing Yu

Student Performance in National Examinations: the dynamics of language in school achievement (SPINE)

www.bristol.ac.uk/spine (ESRC/DfID RES-167-25-0263)

5 Introduction

• Empirical data & documentary evidence reflecting some of the

realities of national examining processes:– Student performance on exam items– Statistical data on student exam performance– Implementation of classroom assessment

• The policy & the politics: historical perspectives: focus on CWA

• Conclusions : policy & politics & issues of justice in examining– progress?

5

6 Student performance on exam items

English RC Qn: How whales resemble man

45 students took this item:

• 35.6% = no answer

• 26.7% = wrong answer

• 28.9% = partially correct answer

• 8.8% = correct answer

7Interview: D1 who didn’t answer Q3 explains

D1: “because I did not understand by this this … resemble” (lines 115-117)

Int: “If I tell you that resemble means ‘to look like’ … can you do the question now?

D1: “Yes”Int: OK so what’s the answer?D1: “Man … is warm blooded … and

whales also … whales have lungs and man also have lungs …” (122-133)

8 Biology: responses to original item

• No answer = 67.4%

• Wrong answer = 21.7%

• Partially correct answer = 6.5%

• Correct answer = 4.4%

9 Biology: on locusts

10

Biology: on locusts

11 Biology: on locusts

Questions:

a) In which picture do you think the locust will/may

die?

b) Why do you think it will/may die?

12 Original item modified:

• Greater contextualisation

• Simplification of instruction

• 2 structured parts: A & B

• Visual clues to support information retrieval

• Rephrasing of the item

• Altering item layout

13 Results

• Original item– Only 32.6% of students wrote an answer – Just under 11% gave a partially correct or

correct answer

• Modified item– 100% responded to this item– 42% gave a partially correct answer to Part A– 53% gave a partially correct answer to Part B

14Changes in student response

OR

Sc

Written responses on Modified Item

MOD

Score

Comment

(Neil Ingram: biologist)

H3 0 In picture A the locust may/will die

I think it will/may dies because the locust get its breathing by using its body

2 + 2 Understands that locusts breathe using the body

C2 1 I think it will/may die because the boy is dipping the locust in the water to all the bodies with its trachea that used to respiration as a respiratory surface of a locust

2 + 2 Getting very close to a very complex answer “I think a very able pupil indeed”. The right answer but with great difficulty in expressing this in English

Means of Form II Exam Results (MoEVT) 15

KISW Islamic

studies

BIO CHEM MATH PHYS

2004 49 37.1 18.4 30.8 21.1 25.8

2005 46 44.3 22.6 34.2 15.3 25.4

2006 36 47.9 21.5 34.3 15.5 25.1

2007 35.5 44.5 21.5 32.0 14.3 23.9

2008 50.6 47.5 24.5 34.8 16.4 29.1

16 ENG and MATHTotal: 64.717, school=29.18%,

pupil=70.82%

ENGLISH explains (90.4-

64.717)/90.4=28.41% of the

maths total variance

Total=90.4, School=30.75%,

Pupil=69.25%

17 ENG and BIO17

Total=65.646, school=18.17%,

pupil=81.83%

ENGLISH alone explains

(114.968-65.646)/114.968=42.90%

of the total variance in BIOLOGY

Total=114.968, school=18.79%,

pupil=81.21%

18 English and CHEMTotal=158.281, school=23.06%,

pupil=76.94%

ENGLISH explains (275.993-

158.281)/275.993=42.65% of the

total CHEM variance

Total=275.993, school=22.05%,

pupil=77.95%

19 How about KISWAHILI & ARABIC?

• Although other two languages (Kiswahili and Arabic) are also significant

predictors of the students’ performance in maths, biology and chemistry,

it is noted that they are less capable of explaining the variances than

ENGLISH.

• KISWAHILI explains (275.993-88.954)/275.993=31.54% of CHEM total

variance, (114.968-76.585)/ 114.968=33.39% of BIO total variance,

(90.4-75.678)/90.4=16.29% of MATH total variance.

• ARABIC explains (275.993-203.452)/275.993=26.28% of the CHEM total

variance, (114.968-88.569)/114.968=22.96% of the BIO total variance,

(90.4-71.885)/90.4=20.48% of the maths total variance.

20 Summary of the multilevel models (a)

• It is very clear that ENGLISH is a significant and substantial predictor of the students’ performance in MATH, BIO & CHEM.

• The school-level variances explained in the cons models as well as in the models including ENGLISH as the single explanatory variable demonstrated a substantial proportion of the variance is attributable to school factors

21 Summary of the multilevel models (b)

• Not much improvement in terms of fitness of the models (measured

by the change of % of school-level variance in the total variance).

• Therefore, essential to collect further school- and pupil-level data to

examine what factors (e.g. English language learning opportunities at

home and at school, academic English proficiency) and how much

they account for the variances (in the tradition of school effectiveness

studies) – our plan for nationwide data collection using pupil and

headteacher questionnaires and vocabulary knowledge test

Coursework assessment as part of national examining: implementation issues

In Form II, Teacher Assessment (TA) was implemented

based on the following assessment procedures:

• Class work

• Oral questions

• Homework

• Weekly tests

• Terminal tests

•

22

Continuous Assessment Format & Guidelines

• Term: --------- Class: --------- subject---------

Pupils’

name

Month

1

10%

Month

2

10%

Month

3

10%

Month

4

10%

Total

C/work

40%

Exam

60%

Grand

Total

100%

X 5 6 3 8 22 30 52

Coursework assessment as part of national examining: implementation issues

• The first three activities make up 40% and the fourth

makes 60% which when combined make 100%

• The final score (out of 100% ) obtained is sent to the

Department of Curriculum and Examination for

further

processing (CWA is 25% of overall exam score)

Coursework assessment as part of national examining: implementation issues

However: significant variability among teachers:

• the types of specific activities that constitute teacher

assessment

• the number of activities from which they take marks for

each month

• decisions on how to handle student absenteeism

• how to get 10% of marks for each month; some picked

the one that students did better on, some picked

randomly, and some put the average of all marks

Coursework assessment as part of national examining: implementation issues

• Understanding of what assessment is to be done

• Awareness of guidelines

• Percentage of marks that go to the final examination

• Sharing what teachers know and practice about

assessment guidelines among teachers within and

across schools

• Level of monitoring of Teacher Assessment by head

teachers.

Continuous Assessment Format & Guidelines

• Note: Coursework assessment will be obtained from

class works, home works and weekly tests or as it will

be directed otherwise by the Department or the

Ministry. All the marks scored by the pupil from the

mentioned activities within a particular month, will be

combined and transformed into percentage of ten (10%)

before filled in the relevant space in this form.

• Source: Ministry of Education Working Guidelines: Guideline

No. 9 (1995)

Continuous Assessment format & Guidelines

Additional notes from another version

• If the term is less or more than four (4) months, four activities

chosen by the teacher him or herself should be picked at equal

intervals. The teacher is advised to give as many tests as he can

in a term but should choose only four for recording.

• In getting pupil’s marks for terminal exam for each subject, the

exam itself should contribute 60% and classroom assessments

(coursework assessments) should contribute 40%

Coursework assessment as part of national examining: implementation issues

• The language to be used is as indicated in the lesson plan.

• These notes appear to contradict with the first version which

might be leading to confusion. This version requires the

teacher to find average of marks in a month while the

second requires the teacher to choose one activity from

which the marks can be recorded.

• Source: Ministry of Education Working Guidelines: Guideline

No. 9 (1995)

The Policy & the Politics: historical perpsectives

• The need to make Assessment procedures more friendly,

less competitive and practically focused

• The prescription that student assessment must:– Include works that can enable students to function

effectively in their environments– Cover both student performance in theoretical works and

in practical productive works– Be done throughout the year

The Policy & the Politics: historical perpsectives

• The National Examination Council of Tanzania (NECTA),

formalised the new examination system by including

projects, character assessment, exercises and tests in the

assessment system

Teachers were then given responsibility to:

• measure students’ ability to use knowledge and skills

acquired, in their local environments and,

• keep ongoing records of their progress

The Policy & the Politics: historical perpsectives

Tanzania and Zanzibar Education Policies• Both The Tanzania Education and Training Policy (1990:79) & The

Zanzibar Education Policy (2006:28) emphasised that continuous assessment combined with the final exam paper would be the basis for awarding of certificates at Secondary Education levels

• For selection purposes and quality control, consistency between teacher assessment and that of the National examinations was emphasized in The Zanzibar Education Master Plan of 1996-2006

Language Policy and Assessment

• Students assessment is affected by Language policy

because students have to demonstrate knowledge and ability

through a language accepted by the system.

• The Zanzibar Education Policy of 1995, Kiswahili version

Section 5.72 (28), The Zanzibar Education Master Plan of

1996 (43), the Education Policy of 2006, all emphasized the

use of English Language as a language of instruction for all

subjects in secondary education except for Kiswahili and

Islamic studies.

Language Policy and Assessment

So, English, as a language of instruction in

teaching science, maths and so forth at secondary

level must be a language of assessment

Does this have to be the case?

Policy & Politics: progress for learners & learning?

Given that English is politically& policy wise the language of the examinations:

• “No test taker shall be harmed by the test (Hamp-Lyons 1989:13)

• No learner should be disadvantaged, in the interests of justice for ALL learners

• Ethical issues re. fairness: need to give individual a fair chance

35

36 Policy & Politics: What progress towards social justice?

Impact/Potential Disadvantage (examples)

Consequences/Injustice: (examples)

•Learners do not engage or respond poorly in examinations•Subject area (e.g. Biology, maths) construct can only be assessed where a linguistic construct has been successfully negotiated

•Loss of self-esteem & motivation for learning•Learners fail to reach their potential (glass ceiling effect) or fail altogether•Leave school as unsuccessful (e.g. at end of Basic Education)•Unequal access to available resources, educational experiences & work opportunities •Unskilled & unable to join the workforce in turn contributing to social & economic deprivation

37Policy & Politics: What progress towards social justice?

Impact/Potential Disadvantage

Consequences/Injustice

•Teachers do not use full range of LTA procedures & processes•Use of inappropriately constructed assessment frameworks•Inaccurate CWA of learners•CWA implemented as a series of tests

•Learners not fully supported in their language & content knowledge development & fail to reach potential •Test performance valued over learning •Leave school with poor educational outcomes•Inadequate d-base for decision making about student & learning progression