1

DEPARTMENT FOR LANGUAGE TEACHING AND INTERNATIONAL STUDENTS

ALTE Member

Saint Kliment Ohridski University of Sofia

2

A Developmental PerspectiveonStandardized Language Testing

Andonyan M., Borissova A., Yontcheva G., Todorinova J.

3

This presentation covers some important points regarding the examination through the Standardized Tests of Bulgarian:

4

Points of interest:

Test development Test takersTest resultsImpact of the ALTE audit 2007

5

The BeginningThe design and development of Bulgarian

standardized tests gradually evolved from 2000 onwards.

Forty-five years lasting experience of the Department in teaching Bulgarian to foreign students and citizens and the sharp increase in the interest towards its studying and certifying, have been the main factors that affected the process.

6

The aim of the tests is: to meet the needs of the international

audience for a test of Bulgarian according to the standards of ALTE and corresponding to the Common European Framework (CEFR)

to serve as an educational or job entryrequirement

to serve as a standard outcome for the international students taught at the Department

to be a part of the procedure for acquiring Bulgarian citizenship.

7

Proficiency LevelsBulgarian standardized tests cover 5 levels of language

proficiency: A2, B1, B2, C1 and C2, which are strictly aligned with the Common European Framework (CEFR) and ALTE Framework.

BASIC LEVEL – broadly corresponds to the Common European Framework of Reference for Languages, level A2

INTERMEDIATE LEVEL – B1 and B2 CEFR levels

ADVANCED LEVEL – C1 and C2 CEFR levels

8

Test Construction

Test construct, task types and format that measure four essential receptive, productive and interactive skills are based on the model of communicative competence for Bulgarian languageknown as Bachman-Tacheva’s model.

9

Test Developers A team of experienced Bulgarian

language teachers from the Department has been established to develop the design and content of the exam.

The team members are teachers and lecturers with scientific interests in the field of language testing. Most of them have academic degrees and are authors of textbooks, handbooks and manuals for language testing.

10

Test ContentTest content has been collected from

a great number of authentic sources: newspapersmagazines reference or history books brochures, booklets fiction, etc. Key principles for text selection are given

in the guidelines for item writers.

11

Content Selection CriteriaThe main selection criteria are: external context and the language use

for different levels according to CEFR the level of skills to be tested typical subject or topic areas for different

levels socio-cultural context vocabulary range grammar and structures specific difficulties and obstacles for

foreign learners of Bulgarian

12

Pre-testing

Due to some objective factors, related to test population and administration, real-time pre-testing has been difficult to be conducted.

The quality and validity of the tasks and items are evaluated by in-class testing and expert analyses, organized by the Examination Board.

13

New Materials

Prior to using, the new examination materials are checked and approved by two working committees:

The Content revising committeechecks and approves the linguistic and theme content.

The Ethics committee monitors for avoiding the insensitive language usage.

14

Training the Personnel

The Examination Board holds seminars, training workshopsand sessions on a regular basis addressed to:

test constructors and item writers examiners and supervisors experts, involved in the language

testing process.

15

Examination CentresTwo operating centres administer the test: Testing Centre at the Department Foreign Examination Centre at IMXA

Institute – Thessaloniki (Greece)We have also contracts for establishing new

centres in: Pizza (Italy) and Nantes (France)but they have not operated yet.

16

MarkingMarking is carried out according to very

clear and detailed assessment instructions both for the oral and written parts of the exam.

These instructions describe in detail: procedures of marking assessment criteria cut-off scores for different grades grading scale.

17

Scoring

Multiple choice items are scored by a prescribed number of points, given for the right answer only. No score reduction is made for

either the wrong or missing answers.

18

Scoring

Speaking is assessed and assigned a grading mark according to a special assessment instruction for the oral part of the examination only.

This part of the examination is fully audio-recorded.

19

AccuracyAll components are marked immediately

after the examination. To assure the accuracy and quality of the

marking, only professional examiners are assigned to conduct the examinations. They have been selected among the most qualified full-time language teachers at the Department.

The team of trained examiners is annually approved by the Examination Board.

20

Training the examiners

During the marking process examiners have to follow strictly defined rating criteria. These criteria and grading procedures are clarified in special training sessions, which take place at least once a year.

21

Examination supervisor

The accuracy of marking is monitored by an examination supervisor, who has the right to confirm or disprove the marks assigned by the examiners.

In case of disapproval, the supervisor is authorized to correct the assigned marks and to report the case to the Examination Board.

22

The Overall Grade

Speaking25%Writing

38%

Reading15%

Listening22%

The overall grade of the examination is formed by the results from different parts as follows:

23

Test timing and cut-off score

The test has to be completed within 3 hours.

The cut-off score is set to 60% of the total.

Candidates are provided with leaflets describing the scope of the examination and a user guide how to apply and how to interpret the results.

24

Post-test studies

Several post-test bias analyses are carried out to make sure the results are not influenced by any extraneous factors.

These studies also reveal test-takers’ profile and draw conclusions based on the data, collected from both the pre-test and post-test questionnaires.

25

QuestionnairesResponses to the questionnaires enable

the Examination Board to gather information about:

test-taker’s educational background the purpose and use of the exam pre-testing study and preparation for

the exam immediate impression and degree of

satisfaction from the examination

26

Distribution of the tested population over the period 2000-2007

Total = 419

43 5062

90

174

0

20

40

60

80

100

120

140

160

180

2000/2003 2004 2005 2006 20070

20

40

60

80

100

120

140

160

180

200

27

Purpose of the examination

General3%

Work34%

Citizenship15%

Education48%

The examination is mostly used for education, workand citizenship.

28

B2 level is mostly selected for education

A2

B1

B2

C1

C2

workgeneraleducationcitizenship

29

B1 level is mostly selected for citizenship

A2

B1

B2

C1

C2

workgeneraleducationcitizenship

30

B1, B2, C1 and C2 levelsare selected for work

A2

B1

B2

C1

C2

workgeneraleducationcitizenship

31

Purpose of the examination distributed by age

2126

3540

education general work citizenship

32

Purpose of the examination distributed by gender

7

53

4

4332

75

4

4639

128

8

89

0

20

40

60

80

100

120

140

citizenship education general work

female male totalSignificantly more men than women apply for citizenship

33

Purpose of the examination distributed by nationality

Turkish, 39%

Greek, 64%

Greek, 35% Other, 38%

Other, 26%

Education Work

34

Tested population distributed by age groups

26 - 4035%

- 188%

40 +17%

19 - 2540%

35

Tested population distributed by gender

male53%

female47%

36

Which are the most popular test information sources?

Advertissment

Test Book

Friends

IMHA Institute

Internet

None

Office

Staff

40+ years26-40 years19-25 years-18 years

37

Is preliminary test information sufficient for the test-takers?

YES50%NO

19%

No answer18%

Neither13%

38

Which factors best supporttest preparation?

0 10 20 30 40 50 60

-18 years

19-25 years

26-40 years

40+ years

TeacherSelf-practiceText book

Teacher is the main test preparation factor.

39

How do the test-takers rate the effectiveness of their test preparation?

-18 years old

19-25 yearsold

26-40 yearsold

40+ years old

positivenegative

Young people demonstrate negative attitude towards test preparation.

40

What is the average grade for different testing levels?

4.75

4.72

4.60

5.18

5.43

4.00 4.50 5.00 5.50 6.00

A2

B1

B2

C1

C2

41

Statistics of the overall grade by age groups

5.1

4.6

4.99

4.93

4.84

5.38

4.75

5

5

5

4 4.5 5 5.5 6

-18

year

s19

-25

year

s26

-40

year

s40

+ye

ars

All

Mean Median

Teenagers demonstrate higher results

42

Statistics of the overall grade by gender

5.01

5

4.68

4.75

2 3 4 5 6

Mea

nM

edia

n

MaleFemale

43

Average grade and L1

3.00 3.50 4.00 4.50 5.00 5.50 6.00

Turkish

Asian

Greek

Hebrew

Arabic

French

German

English

Italian

Spanish

Slavic

44

How do the test-takers qualify their examination result?

19%

15%

54%

12%

no answer

worse

as expected

better

45

Self-rating of the results distributed by levels

0% 20% 40% 60% 80% 100%

A2

B1

B2

C1

C2

worseas expectedbetter

27% of C1 level candidates qualify their result as worse than expected

46

How do the test-takers rate the quality of examination?

no answer17%

very good39%

good8%

satisfactory1%

unsatisfactory1%

excellent34%

81% of the test-takers appreciate the quality of examination

47

ALTE audit 2007

B2 level test was audited in June 2007.

This ALTE audit revealed some drawbacks and also inspired immediate actions for improving the quality and administration of the Standardized Tests of Bulgarian.

48

Increasing the number of candidates The Council of the Department decided

to allow the international students free of charge examination as a regular outcome of their studies at the end of the academic year 2006/2007.

90 students took advantage of this proposal (the sponsorship cost the Department nearly € 13 500).

49

Improving pre-testing

Pre-testing in real-examination conditions is to be carried out parallel with in-class pre-testing from 2008.

Preliminary test is to be introduced to help pre-testing and avoid misleading the candidates by identifying their approximate level.

50

Reliability

Insertion of anchor itemsacross different variants to handle the reliability of the test.

Increasing the numberof test-takers.

51

Calibration

To continue the process of developing calibrated examples for evaluation of various models for the different levels of the test.

52

Thank you for your attention