1

OWLTSOnline World Language Testing Software

Pittsburgh Public Schools

Prismatic Consulting LLC

2

Why create an online language assessment tool?

There can be no standards without assessments to measure student progress.

3

History

Used CAL’s, SOPI tape mediated test Created own tape mediated test Tape administration was difficult and

cumbersome Obtained FLAP grant to develop test software OWLeTS current version 2.5

– Working on version 3.0

4

OWLTS

“Authorable” Software Key Features

o User friendly student testo User friendly test administrationo Easy for teachers to rate / scoreo Improved rating reliabilityo Data aggregationo User friendly task / test development

5

User Friendly Student Test

Easy to understand student user interface Practice tests to increase student comfort

level Accommodations for special needs Students are more comfortable with computer

than tape recorder

6

User Friendly Test Administration - District

Flexibility in test size/grouping - Small or large groups from single student to large scale district wide testing

This year 1800 students tested – With audio tape only able to test 900 per year

Testing at widely distributed multiple sites – Tested at 36 sites this year

No distribution of testing equipment such as tape recorders, tapes etc.

7

User Friendly Test Administration - Teacher

Testing manual for teacher use Practice test increases teacher comfort Can test any language

– ESL– Non-traditional languages

Can test different languages at same time in one room

8

Easy for Teachers to Rate / Score

Teachers can rate from anywhere Teachers can do 1 rating in a sitting or 100 Can listening to same sample repeatedly

without fast forwarding or rewinding Easy to re-rate speech sample (second or

third rating) Teachers can easily provide rating / score

and feedback comments

9

Improved Rating Reliability

To follow ACTFL/OPI protocol OWLeTS can be configured to prevent teachers from rating own students

Has been validated with SOPI and OPI Easy to compare ratings between raters – Inter-rater

reliability checks Provides means to give feedback to teachers on

ratings Enhanced training – Can pull speech samples and

ratings for comparison in training sessions

10

Data Aggregation

Tied to student information system Easy to create new reports on collected data Data collection allows analysis of student

ratings by:– class, section, school, district, language, program,

race, socio-economic, etc.

Can compare data from ratings by raters – rater reliability

11

2003 – 2004 ComparisonY

ea

r

Te

ste

d

No

vic

e L

ow

No

vic

e M

id

No

vic

e H

igh

Inte

rme

ida

te L

ow

Inte

rme

dia

te M

id

Inte

rme

dia

te H

igh

No

-Ra

tin

g

%N

ov

ic L

ow

% N

ov

ice

Mid

% N

ov

ice

Hig

h

% In

term

eid

ate

Lo

w

% In

term

ed

iate

Mid

% In

term

ed

iate

Hig

h

% N

o-R

ati

ng

2004 1798 512 390 307 262 57 32 238 28% 22% 17% 15% 3% 2% 13%

2003 1335 310 276 226 228 0 3 292 23% 21% 17% 17% 0% 0% 22%

Difference 463 202 114 81 34 57 29 -54 5% 1% 0% -3% 3% 2% -9%

12

2003 – 2004 Comparison

28%

22%

17%15%

3%2%

13%

23%21%

17% 17%

0% 0%

22%

0%

5%

10%

15%

20%

25%

30%

%Novic Low % Novice Mid % Novice High % IntermeidateLow

% IntermediateMid

% IntermediateHigh

% No-Rating

2004

2003

13

User Friendly Task Creation - Authorable

Easy to create and edit tasks Records and plays multiple audio formats

– WAV– MP3

Accepts any image file– Clip art – Photos (jpg, bmp)– Movies (mpeg, wma)

14

User Friendly Test Creation - Authorable

Easy to create and edit tests including length Tests can be created for any

– Level -Novice, Intermediate, Advanced and Superior– Age - 5th grade, university admission, teacher candidates– Language – traditional, non-traditional, ESL

Can change test from day to day or hour to hour Tests can be created in any structure

– This test is structured like CAL (Center for Applied Linguistics) / SOPI– Can be structured for ESL - TESOL (Teaching of English to Speakers of

other Languages)– Other

15

Future – Version 3.0

Each teacher will have own task / test bank Students will have online practice from home Addition of writing and reading assessment Additional test item types; cloze, multiple

choice, short answer, etc. Improved student user interface .NET

16

Thank You