DIAGNOSTIC TESTING – A TALE OF TWO INITIATIVESDIAGNOSTIC TESTING – A TALE OF TWO INITIATIVES

Using diagnostic tests to assist in Using diagnostic tests to assist in (i)(i) widening access and (ii) post hoc curriculum designwidening access and (ii) post hoc curriculum design

NAN YELDNAN YELD

The Alternative Admissions The Alternative Admissions ProjectProject

The Exit-level Testing The Exit-level Testing ProjectProject

To identify talented students To identify talented students whose school results would not whose school results would not qualify them for entryqualify them for entry

To provide useful To provide useful information for curriculum information for curriculum design design

To place these students into To place these students into appropriate programmes of appropriate programmes of studystudy

To explore the effectiveness To explore the effectiveness (for entry level curriculum (for entry level curriculum design) of diagnostic design) of diagnostic information on graduating information on graduating studentsstudents

To deliver meaningful To deliver meaningful diagnostic information for diagnostic information for curriculum designcurriculum design

To assess the suitability of To assess the suitability of using existing entrance using existing entrance selection/diagnostic tests in selection/diagnostic tests in this attemptthis attempt

Selected TIMSS 1999 performance Selected TIMSS 1999 performance indicatorsindicators

BenchmarkBenchmark Description of benchmarkDescription of benchmark % students reaching % students reaching benchmarkbenchmark

Highest Highest performing performing group (e.g. group (e.g. Singapore, Singapore,

Korea, Japan)Korea, Japan)

Lowest Lowest performing performing

group (South group (South Africa, Africa,

Philippines, Philippines, and Morocco)and Morocco)11

Top 10%Top 10% Students can organise information, Students can organise information, make generalisations, and explain make generalisations, and explain solution strategies in non-routine solution strategies in non-routine problem solving situations.problem solving situations.

≥ ≥ 33 %33 % ‹ ‹ 1 %1 %

Upper Upper quarter quarter

Students can apply their Students can apply their understanding and knowledge in a understanding and knowledge in a wide variety of relatively complex wide variety of relatively complex situations.situations.

± 66 %± 66 % 1 %1 %

Median Median Students can apply basic Students can apply basic Mathematical knowledge in Mathematical knowledge in straightforward situations.straightforward situations.

± 90 %± 90 % ‹ ‹ 10 %10 %

Lower Lower quarterquarter

Students can do basic computations Students can do basic computations with whole numbers.with whole numbers.

± 99 %± 99 % 31 %31 %

Percentage of students scoring below benchmark Percentage of students scoring below benchmark levelslevels

1 %1 %Note 1: South Africa scored the lowest of this

group.

69 %

South African School- Leaving Cohort

2001 (estimate)

endorsement

8%

certificate

22%

failed

19%

rest of age

cohort

51%

School-leaving examination data 2001 - black candidates obtaining exemptions

0

10

20

30

40

50

60

70

80

40-50 50-60 ≥ 60

Examination results in aggregate categories

Per

cent

age

cand

idat

es in

ag

greg

ate

cate

gori

es

23,430

7,260

2,310

Some factors impacting on test developmentSome factors impacting on test development

Issues to be consideredIssues to be considered Consequences for test Consequences for test developmentdevelopment

Affordability and Affordability and feasibilityfeasibility

Pen and paperPen and paper‘‘mark-ability’ format issuesmark-ability’ format issues

Impact on system (e.g. Impact on system (e.g. importance of not importance of not undermining the Sen. undermining the Sen. Certificate)Certificate)

Tests not in school subject Tests not in school subject areasareasPerformance on SC remains Performance on SC remains importantimportant

Impact on individuals Impact on individuals (importance of not (importance of not creating another barrier creating another barrier to access)to access)

Writing of tests voluntaryWriting of tests voluntaryCandidates assessed on Candidates assessed on EITHER SC results OR project EITHER SC results OR project test resultstest results

The tests need to do The tests need to do something different something different (avoid ‘predicting the (avoid ‘predicting the past’)past’)

Develop and implement Develop and implement innovative ways of ‘getting at’ innovative ways of ‘getting at’ underlying ability - scaffoldingunderlying ability - scaffolding

The tests must not The tests must not perpetuate the perpetuate the ‘revolving door’ ‘revolving door’ tendency seen in many tendency seen in many access projectsaccess projects

The tests need to be able to The tests need to be able to inform curriculum inform curriculum development – must have development – must have some diagnostic power.some diagnostic power.

In a higher education context, students are required to:

• Make meaning from what they read;• Understand and interpret conceptual and metaphorical

language;• Identify and track academic argument;• Follow discourse structure in text;• Make inferences about and extrapolate on what they

read;• Demonstrate familiarity with and understanding of the

conventions of visual literacies, such as reading and interpreting graphs, flow-charts and diagrams; and

• Cope with basic numeracy demands.

Getting to grips with these requirements is seriously impeded by approaches to texts and epistemic practice

such as:

• A tendency towards verbatim reproduction or plagiarism in essays

• A tendency to describe rather than analyse, and to offer tautologies in place of justification

• A tendency to focus on examples (tokens) rather than on principles (types), and the relation between them

• A tendency to write from a highly subjective viewpoint without depersonalising

• A tendency to be prescriptive or normative when asked to be analytic

(Slonimsky and Shalem 2005)

Did the approach provide a greater Did the approach provide a greater range of scoresrange of scores (spread), so that capable students could be more clearly (spread), so that capable students could be more clearly

differentiated from weaker students?differentiated from weaker students?

Did the approach increase the Did the approach increase the predictive validitypredictive validity of the of the test (that is, did the test correctly distinguish between test (that is, did the test correctly distinguish between

weaker and stronger students)?weaker and stronger students)?

Did the approach Did the approach improveimprove (raise) (raise) the level of stronger the level of stronger students’ scoresstudents’ scores??

Assessing the effectiveness of the scaffolding approachAssessing the effectiveness of the scaffolding approach

ELPT and ALPT ScoresELPT and ALPT Scores

0

100

200

300

400

500

600

No

. of

Ca

nd

ida

tes

10 20 30 40 50 60 70 80 90 100

Test Results (%)

ELPT

ALPT

Task Preparation Study

Question 6.1: In note form (i.e. not full sentences), list the main points the author makes about the Japanese educational system. For this exercise, do not include comparisons or references to the American system.

Question 6.2: Drawing on the text of the whole article, but using your own words, summarise the points the author makes about the Japanese system of education. Note: your summary should not include comparisons or references to the American system.

Top 30% of candidates

(n=114)

Bottom 30% of candidates

(n=114)

Points made in notes and carried over to summary

.62 - 0.35

Semester- specific hazard rates for the Admissions

Project group (511) and the control group (563)

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

6 12 18 24 30 36 42

semesters

haza

rd r

ate Admissions Group

Control group

'Challenge levels' of papers

0

10

20

30

40

50

60

70

Level 1 Level 2 Level 3

I tem challenge levels

% it

ems

at c

halle

nge

leve

ls

1992 JMB 2001 1st National Paper 2003 2nd National Paper

ASSUMPTIONS UNDERLYING EXIT-LEVEL TESTING ASSUMPTIONS UNDERLYING EXIT-LEVEL TESTING PROJECTPROJECT

11 By and large, and with many notable exceptions, By and large, and with many notable exceptions, academics …. do not take seriously/ are not well academics …. do not take seriously/ are not well informed about / are not very interested in …. informed about / are not very interested in ….

information about the learning needs of incoming information about the learning needs of incoming students. students.

22 Academics are more likely to take information Academics are more likely to take information about the learning needs of their students about the learning needs of their students

seriously if these needs are demonstrably still seriously if these needs are demonstrably still present in the students they are about to present in the students they are about to

graduate.graduate.

33 In order to demonstrate these needs, a valid and In order to demonstrate these needs, a valid and credible (and feasible) instrument is needed. credible (and feasible) instrument is needed.

1.1. By and large, academics …. do not take seriously/ are not well By and large, academics …. do not take seriously/ are not well

informed about / are not very interested in …. information about informed about / are not very interested in …. information about

the learning needs of incoming students. the learning needs of incoming students. Why might this be so?Why might this be so?

• They believe it would be lowering standards to ‘pander to’ needs?They believe it would be lowering standards to ‘pander to’ needs?• They believe it’s the responsibility of the schooling sector to They believe it’s the responsibility of the schooling sector to

prepare students, not theirs?prepare students, not theirs?• They assume that massification will entail higher failure rates, it’s They assume that massification will entail higher failure rates, it’s

“only natural”?“only natural”?• They are really only interested in students who will be majoring in They are really only interested in students who will be majoring in

their disciplines, and so it’s the survivors from the first-year their disciplines, and so it’s the survivors from the first-year

shakedown that they are focused on?shakedown that they are focused on?• Because doing so requires a shift from an inward looking, Because doing so requires a shift from an inward looking,

introjected orientation, to an outward looking, ‘projective’ introjected orientation, to an outward looking, ‘projective’

orientation (Barnett 2000), and a willingness to negotiate curricula orientation (Barnett 2000), and a willingness to negotiate curricula

across departmental boundaries?across departmental boundaries?• The outsider status of those providing the information?The outsider status of those providing the information?

DESCRIPTIVE STATISTICSDESCRIPTIVE STATISTICS

Entering Entering GroupGroupN = 95N = 95

Graduating Graduating groupgroupN = 67N = 67

MeanMean 36.936.9 45.145.1

Standard ErrorStandard Error 1.31.3 1.21.2

MedianMedian 35.435.4 46.146.1

ModeMode 39.939.9 39.339.3

Standard DeviationStandard Deviation 13.013.0 9.59.5

MinimumMinimum 11.811.8 27.027.0

MaximumMaximum 71.971.9 65.265.2

RangeRange 60.160.1 38.238.2

Quintile boundaries: Entering and graduating

'disadvantaged schools' groups

0

10

20

30

40

50

60

1 (top) 2 3 4 5(bottom)

Quintile lower boundaries

Perc

enta

ge

Entering

Graduating

Skill cluster descriptions (examples)

Skill AssessedSkill Assessed Explanation of Skill AreaExplanation of Skill Area

Extrapolation, Extrapolation, application and application and inferencinginferencing

Students’ capacities to draw conclusions and Students’ capacities to draw conclusions and apply insights, either on the basis of what is apply insights, either on the basis of what is stated in texts or is implied by these texts.stated in texts or is implied by these texts.

Understanding Understanding the the communicative communicative function of function of sentencessentences

Students’ abilities to ‘see’ how parts of Students’ abilities to ‘see’ how parts of sentences / discourse define other parts; or are sentences / discourse define other parts; or are examples of ideas; or are supports for examples of ideas; or are supports for arguments; or attempts to persuadearguments; or attempts to persuade

Separating the Separating the essential from essential from the non-the non-essentialessential

Students’ capacities to ‘see’ main ideas and Students’ capacities to ‘see’ main ideas and supporting detail; statements and examples; supporting detail; statements and examples; facts and opinions; propositions and their facts and opinions; propositions and their arguments; being able to classify, categorise arguments; being able to classify, categorise and ‘label’and ‘label’

Understanding Understanding basic numerical basic numerical conceptsconcepts

Students’ abilities to make numerical Students’ abilities to make numerical estimations; comparisons; calculate estimations; comparisons; calculate percentages and fractions; make chronological percentages and fractions; make chronological references and sequence events / processes; do references and sequence events / processes; do basic computationsbasic computationsCliff, A., Hanslo, M. and Visser, A-J. (2003). Faculty of Engineering and the Built Environment: AARP

tests and first-year students’ academic performance for the 2002/3 cycles. Unpublished report: University of Cape Town, December 2003.

Skills clusters: graduating group 'disadvantaged

schools'

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

Test t

otal

%Vo

cab

Met

apho

r

Extr

apola

te

Sente

nces

Relat

ions

Genre

Visu

al

Esse

ntial

Numer

ical

Own Voic

e Tot

al

Own Voic

e Co

nten

t

Own Voic

e Org

Own Voic

e Vo

cab

Own Voic

e La

ng

Own Voic

e Mec

hanic

s

Skills clusters

Perf

orm

ance

(%

) on

ski

lls c

lust

ers

REFERENCES

• Barnett, R. (2000). Supercomplexity and the curriculum. Studies in Higher Education, 25, 255-2665.

• Cliff, A., Hanslo, M. and Visser, A-J. (2003). Faculty of Engineering and the Built Environment: AARP tests and first-year students’ academic performance for the 2002/3 cycles. Unpublished report: University of Cape Town, December 2003.

• Slonimsky, L. and Shalem, Y. (2004). Pedagogic responsiveness for academic depth. In Griesel, H. (Ed.)(2004). Curriculum responsiveness – case studies in higher education. South African Universities Vice-Chancellors’ Association: Pretoria, 81 - 101.