TSL705: Report

Format: 2,000 word report

Objective/Outcome: Revise an adapted test, Pilot the test, report the results of the test and insights from the test design process.

Task Details: The specific testing instrument is the Inside Out Full Placement Test (attached copy on Week 7 Moodle)Students will be required to revise the adapted test and pilot it, present a written report describing, explaining and evaluating the processes undertaken.

The adapted test will need to be piloted on the target test-taker audience (i.e. students of the appropriate language level). Students will be required to administer the test to a minimum of five non-native English speakers.

Marking Guide: See Rubric in TSL 705 Course Outline

Subject Learning Outcomes Contribution to Course Graduate Attributes

a. Critically analyse the differences between formative and summative evaluation and discuss the possible uses of each in language teaching.

b. Analyse the validity, reliability and uses of a method of language assessment.

c. Analyse and interpret test scores.

Revise the Adapted TestUse feedback from the tutor’s marking of assessment 2 in order to revise and further improve the test before piloting.

Pilot the Adapted TestPilot the adapted test with at least 5 test-takers that would take this test. Analyse & interpret the test scores and test-taker feedback. Conduct item analysis to determine which items differentiate the most effectively.

Report to Explain & Evaluate the processDescribe the results of the pilot test -student results, student feedback, exit surveys or comments, distribution of results. Suggest how the test could further improve in the future to accommodate any problems, again with academic sources to back you up. Comment on insights the pilot test results can provide about the test.

Bonus: Great ideas from past assignments:

- use a combination of written summaries and visual representation – graphs, tables, charts.- Integrate specific examples as evidence each time you make a comment or critique about the

test. - Executive summary – gives an overview of the process and results/recommendations – not an

introduction!! Should give spoilers, don’t be vague! Don’t give much background information, keep it short.

- Summarizing test piloting process – presented written and visually - what you did in each stage, and why, and what that means – revision of test, selecting participants, administration (how does this differ from real test conditions and is this significant?), collection and marking – use of rubrics. Use research to support/explain aspects of the test design process.

- Analysis of test scores – section by section – presented written and visually – give lots of specific examples. did the section or test item differentiate effectively? Is it working? Item analysis, facility index, discrimination index, tricky questions or tricky answers which made you re-think that section.

- Suggestions for improvements – support everything with research to help justify your positions and suggestions – how could you fix any bits that didn’t work? E.g. adjusting the question, weighting, scoring, rubric, timing? Be critical, use research evidence which specifically supports your position.

- Formatting - integration of written and visual representation, table of contents professionally done, section headings, labelled diagrams, appendices are listed numerically

- Language/referencing - spellcheck, avoiding run ons and fragments, references are Harvard and follow the rules – use websites to create them if you have issues.

1. Plotting the test scores on a graph to show the distribution

2. Item analysis – facility index (how hard) and discrimination index (how does it separate students)

Difficulty of questions beginner Pre-intermediate Pre-intermediate incorrect Section1 listening Q5 * * * 3 Section2 MCQ Q7 * * * 3 Section2 MCQ Q8 * * * 3 Section 2 reading Q 19 * * * 3

3. Breakdown of marks awarded and overall placement level

Students from Low to High Scorebeginner Pre-intermediate Pre-intermediate advanced advanced

Section1

listening 5 9 8 10 10MCQ 8 10 10 10 10reading 10 10 10 10 10Section 2

listening 5 10 8 10 10MCQ 4 7 8 10 10reading 4 8 6 10 10Section3

listening 2 10 9 10 10MCQ 3 9 7 10 10reading 4 8 10 10 10

Samp

les of w riting at different levels with marking break down:

Test taker feedback:

Questionnaire Test-taker feedback 1.Perception of test difficulty

A. difficult B. easy 2. Clearness of instructions

A. clear B. unclear 3. what’s your feelings about the test and the materials or tasks.