Computer-Based Assessments

Computer-Based Assessments

CBA Pilot for NILA & NINA

Autumn 2013

Assessment Coordinator Training

September / October 2013

1

Learning Intentions

At the end of the session you will be aware of: •the context for computer-based assessment (CBA) in autumn 2013;•developments to NILA and NINA; •next steps for the pilot schools; and•guidance and support available from CCEA and C2k. 

Will have an opportunity to :•access the CBA online training course; •explore NILA and NINA; and •contribute to discussion about using the NILA and NINA outcomes.

2

Overview of the Seminar

9.30 Registration

10.00 Welcome

Context of computer-based assessment in NI

10.45 Tea/Coffee

11.00 CBA context (contd.)

Developments - NILA and NINA

13.00 Lunch

13.30 Reporting Developments

Practical Activities

Next Steps for CBA Pilot

Guidance and support available from CCEA

14.45 Tea/Coffee

16.00 Questions, Evaluation & Close3

Part 1

Setting the Context for the CBAs in 2013

4

Legislative Context

In his letter to schools in May 2013 the Minister for Education outlined that:

•CBA assessment will not be specified as mandatory in 2013/14;•Use of CBA assessments (NILA and NINA) in 2013/14 will be optional for schools.

“Although CBA will not be specified in 2013/14, the Department expects schools to carry out diagnostic assessment using an assessment tool of its own choice and that the results of this assessment should be shared with the pupil’s parent/guardian at a mutually convenient time.”

“I would like to encourage as many schools as possible to participate in this pilot.”

5

Moving Forward with the CBA Pilot

Online CBA Training live as of 9 September

Initial Pre-pilot 11 – 13 September

Principal Information Seminars 11 – 25 September

CBA Pre-pilot (10 schools) 16 – 20 September

Assessment Coordinator Training 27 September  - 11 October

CBA Pilot:­ Phase 1­ Phase 2­ Phase 3­ Phase 4*

 7 October – 18 October21 October – 8 November11 November – 22 November25 November – 13 December

Questionnaires issued to schools November

6

NILA - Northern Ireland Literacy Assessment

NINA - Northern Ireland Numeracy Assessment

•Assessments are available in English & Irish.

•Assessments are adaptive and not time-bound.

•Assessments are web-based and pupil responses to questions are saved automatically online.

•Assessments may be completed by Year 4 to Year 7 pupils from the pilot schools during autumn 2013.

•Assessments are processed immediately following completion and schools can access results after 48 hours.

The purpose of the computer-based assessments is to help teachers plan the learning and teaching during the school year to meet the needs of pupils in their classrooms.

Computer-Based Assessments

7

The question items in the Literacy / Numeracy Assessments have been developed using the following criteria:

•Definition of Literacy / Numeracy as set out by CCEA;

•The expectations of the NI Language & Literacy / Mathematics & Numeracy Curricula;

•CCEA’s revised Lines of Development;

•Advice from ELBs regarding expectations at the end of each academic year; and

•Levels of Progression.

Computer-Based Assessments

8

Focus on Raising Standards

“The use of data from computer-based assessments remains a key aspect of the Department’s school improvement policy”

Paragraph 3 – DE Letter to Principals Nov 20119

Computer-Based Assessments

10

NILA Overview

11

The Literacy Assessment

• Assesses some crucial aspects of Listening, Reading and Writing.

• Draws on topics from across the Northern Ireland Curriculum.

• Is varied in form, ranging from everyday texts, such as advertisements and signs, to high quality literary passages.

• Uses 8 different answer techniques (cloze, drag and match, drag into order, drag punctuation, drag to hot spot, pick multiple, pick one and select word).

• Uses the potential of the on-screen presentation to provide interactive and engaging experiences in different ways.

12

The Literacy Assessment

• There are two age related bands in the assessment, one for Years 4 and 5 and one for Years 6 and 7.

• The questions cover a range of difficulty within each band, designed to allow pupils show what they can do whatever their ability.

• The assessment adapts to easier or more challenging questions based on pupils’ responses.

• The assessment should take the majority of pupils around 35 minutes to complete.

• Provides teachers with diagnostic information about the skills and understanding of their pupils.

13

Focus on Skills

Aspect of Literacy Skills Focus Pupils’ ability to …

Listening Information Retrieval

Listen to and understand information and ideas.

Reading Sequence Understand information and ideas

Information Find and use information

Features of Texts

Understand different types of texts

Opinions Respond to different layers of meaning

Writing Punctuation Punctuate text

Spelling Recognise accurate spellings

Grammar Use grammar accurately14

Adaptive Routing in NILA

15

NINA Overview

16

The Numeracy Assessment

• Samples the key concepts of Numeracy through two assessments / strands:

Strand 1 Understanding & Using Number Strand 2: Shape & Space, Measures; Handling Data

• Each strand will be assessed using an adaptive approach to assessment.

• Assessment for each strand will take approximately 20 minutes.

• Only mental arithmetic questions are timed.

• English Medium database is multi-instance.

17

Focus on Skills

Skills Focus Pupils ability to …

Strand 1 Number Counting & Understanding Number;

Facts Facts and Mental Arithmetic

Operations Operations with Whole Numbers, Integers and Decimals

Fractions Fractions, Decimals, Percentages, Ratio & Proportion

Problems Problems – including Money, Function Machines & Algebra

Strand 2 Shape and Space

Shape and Space

Measures Measures

Handling Data Handling Data

Skills Focus Pupils’ ability to …

Strand 1 Number Counting & Understanding Number;

Facts Facts and Mental Arithmetic

Operations Operations with Whole Numbers, Integers and Decimals

Fractions Fractions, Decimals, Percentages, Ratio & Proportion

Problems Problems – including Money, Function Machines & Algebra

Strand 2 Shape and Space

Shape and Space

Measures Measures

Handling Data Handling Data18

NINA Assessment Process

Column Information Provided

What is being assessed? Skills Focus Specific Area of the curriculum

Objective Specific aspect of numeracy

How difficult are the questions?

Year Curriculum year of the question

Band Difficult, medium or easy

How the pupil performed? Outcome Ticks and crosses

19

Adaptive Assessment

• All pupils start in their target year and are presented with questions which are one year below their current school designated year e.g. Year 3 questions for a Year 4 pupil.

• On the basis of their responses, pupils are promoted, demoted or stay in their curricular year.

• A promotion may be reversed by a subsequent demotion and vice versa.

• The overall result is then computed as a weighted average of the two strand scores using the number of threads as the weighting.

• Outcomes from the assessment give both numerical and qualitative diagnostic information about individual pupils. These will become available 48 hours after completion of Strand 2.

.

20

Adaptive Assessment

21

Part 2

Developments to NILA and NINA

22

2012 CCEA Review

Evaluation of:

•online CBA training (users);

•CBA implementation during autumn 2012 (principal, teacher and pupil);

•AROs and SSs (principal and teachers); and

•parent perspective.

Feedback from:

•helpdesk calls;

•ELB support teachers seminars;

•Principals’ focus groups; and

•written and verbal feedback from special schools

23

Developments

• Revised Pupil Demos and associated resources;

• NINA Practice Area;

• Item Analysis and Question Banks;

• SEN Improvements;

• Monitoring Assessment Status;

• Improved Reports (simpler, sortable, printable and saveable);

• Charts and Tables showing progress over time;

• Improved IM Language selection;

• Modified Pupil Checklist; and

• Streamlined Helpdesk.

24

Pupil Demos

Pupil Demos for autumn 2013

•Pace of the voice will be slowed

•Optional sub-titles available

•Scripts will be available.

NILA Practice Assessment Demo / NINA Practice Area Demo:

Introduces pupils to NILA / NINA and the techniques they need to be aware of when completing their assessment.

NILA / NINA Real Assessment Demo:

Reinforces this knowledge and prepares the pupils for logging in to complete their actual assessment.

25

Revised Practice Area

26

Revised Practice Area

27

Question Banks and Item Analysis

NILA•Additional questions added to NILA at the lower and upper end.•Drag punctuation questions:

– Improved ‘grab’ of the box; and– More accurate placement of the punctuation mark.

NINA•Reviewed small number of questions with high proportion of incorrect answers.

28

General SEN improvements

• Specific guidance included in the NILA / NINA Guidance Documents;

• Setting an SEN entry point for individual pupils;• Setting an HI entry point (removes Listening questions from NILA);• Improved ‘drag punctuation’ questions in NILA;• Voice in NILA and NINA Demos slowed down;• Improved pupil demos (for the practice areas and the real

assessment);• Streamlined NINA Practice area; and• Simplified pupil checklists.

29

Introduction of Special Entry Points

What qualifies a pupil to use the SEN entry point?

•Teacher and SMT must:– reasonably expect that the pupil is performing 2 or more years below their chronological

age; and– be in agreement that an SEN entry point is appropriate for the pupil.

What qualifies a pupil to use the HI entry point?

•Teacher and SMT must:– have evidence that the pupil is receiving special support for HI; and– be in agreement that an HI entry point is appropriate for a pupil.

N.B: Only the administrator can select the SEN and HI entry point.

30

NILA SEN and HI Entry Point

SEN Entry Point

Pupils will:•have access to the same number of questions as their peers; •allows pupils quicker access to easier and shorter questions; and•aims to deliver a more positive and potentially shorter assessment.

Hearing Impaired Entry Point

Pupils will:•only be assessed in Reading and Writing; and•use an appropriate checklist.

31

Setting an SEN Entry Point for a Pupil (NILA)

32

Setting an SEN Entry Point for a Pupil (NINA)

33

NINA SEN Entry Point

Pupil’s Curriculum Year

First Non-SEN Entry Point questions will be from:

First SEN Entry Point questions will be from:

Year 4 Year 3 Year 1

Year 5 Year 4 Year 2

Year 6 Year 5 Year 3

Year 7 Year 6 Year 4

34

NILA: Monitoring Assessment Progress for the School

Shows that the pupil has started NILA in English

Shows that the pupil has completed NILA in English

Shows that the pupil has started NILA in Irish

Shows that the pupil has completed NILA in Irish

35

NINA: Monitoring Assessment Progress for the School

Shows that the pupil has started NINA

Shows that the pupil not started NINA

Shows that the pupil has completed NINA

36

CBA Roles and Responsibilities

• NILA / NINA Administrator - Administrators manage NILA / NINA* at school level.

• Teacher - Teachers manage the delivery of NILA / NINA* for their class.  

*Note for NINA: there is an Admin Zone and a Teacher Zone

37

Administrator Role

The Administrator can:

•view the details for all teachers, pupils and classes registered in the school;•print / create teacher usernames and passwords for the school;•print pupil usernames and passwords for each class;•set an SEN / HI Entry Point for a pupil;•monitor assessment progress for all classes in the school; and•view and print tables and reports for individual pupils and classes across the school.

38

Logging in as the NILA Administrator

39

Logging in as the NINA Administrator

40

Demonstrating NILA

• Demonstration of NILA administration

41

Part 3

Developments in Reporting

42

What NILA / NINA Tables and

Reports are available?

• NILA / NINA Results Tables provide schools with Age-Related Outcome (ARO) Ranges and Standardised Scores (SS). These are available for individual pupils, classes, year groups and at whole school level. These build over time to show additional data as it becomes available for new academic years.

• NILA / NINA Parent Reports contain the results table for an individual pupil, showing only the Age-Related Outcome (ARO) Range.

• NILA / NINA Objectives Reports provide specific information about the questions asked and whether the pupil answered correctly or incorrectly.  

43

Printing

Teachers can view and print tables and reports for:

•their own class; or •any of their own pupils.

Administrators can view and print tables and reports for:

•the school;•any year group; •any class; or •any pupil.

44

Results Table (Individual Pupil)

45

Results Table (School, Year Group and Class)

46

Parent Report

47

NILA Objectives Report

• NILA Objectives Report focuses on the assessment performance of an individual pupil.

• It is designed to answer three key questions:– what skills have been assessed?– what was the difficulty of the questions?– how did the pupil perform during the assessment?

48

NILA Objectives Report

What skills have been assessed?

What was the difficulty of the

questions?

CY is now Age

> above what is expected in the pupil’s curriculum year

= within what is expected in the pupil’s curriculum year

< below what is expected in the pupil’s curriculum year

How did the pupil

perform?

Columns re-ordered so that the Outcomes are on the right.

49

NINA Objectives Report

• NINA Objectives Report focuses on the assessment performance of an individual pupil.

• Three different views of the Objectives Report designed to answer three key questions:- what skills have been assessed?- what was the difficulty of the questions?- how did the pupil perform during the assessment?

50

NINA Objectives ReportQuestion Order

Question Order

Single report which includes both Strand 1 and Strand 2.

Band

E – Easy

M – Medium

D - Difficult

Printable individually or in batches within 48

hours for current or previous years.

51

NINA Objectives ReportSkills Focus

Printable individually or in batches within 48

hours for current or previous years.

Number of Questions will be included with the

pupil details at the top of the page.

Teacher will know the pupil’s target

year.

Skills Focus

Single report which includes both Strand 1 and Strand 2.

Clear where pupil has answered multiple

questions from an objective.Difficulty of

Questions answered is clear.

52

NINA Objectives ReportCurriculum Year

Year

Single report which includes both Strand 1 and Strand 2.

Difficulty of Questions

answered is clear.

Clear where pupil has answered multiple

questions from an objective.

Printable individually or in batches within 48

hours for current or previous years.

53

Demonstrating NINA

• Demonstration of NINA administration

54

Activity: Using the ‘Outcomes’

In your pack you have a selection of NINA and NILA reports.

How could you and your colleagues use this information within your school?

55

Using Outcomes

For example,

•To monitor progress and for benchmarking performance within the school and across Northern Ireland.

•To support clear and effective communication with parents in line with feedback from schools in 2012.

•To aid communication with parents as to how they can help to support learning at home.  

•To support the planning for learning and teaching for individual pupils.

•To inform individual education plans for pupils receiving targeted support for special educational needs.

56

Reporting CBA Outcomes to Parents

In his letter to schools in May 2013 the Minister for Education outlined that:

“Although CBA will not be specified in 2013/14, the Department expects schools to carry out diagnostic assessment using an assessment tool of its own choice and that the results of this assessment should be shared with the pupil’s parent/guardian at a mutually convenient time.”

57

Practical Activities

• CBA online training course (10 mins)

• Exploring NILA (10-15 mins)

• Exploring NINA (10-15 mins)

58

Part 4

Next Steps

59

Key Dates for CBA Pilot

Prepare pupils for CBAs (practice area/assessment)

September 2013 onwards

Online teacher training available 9 September 2013 onwards

Principal training 11 - 25 September 2013

Assessment Coordinator training 27 September – 11 October 2013

Upload pupil data from SIMS to WebXchange

Not later than 30 September 2013

Phased Approach : Pupils complete live assessments in non-transformed schools.

7 October – 22 November 2013

Pupils complete live assessments in transformed schools.

25 November – 13 December

Questionnaires issued to schools November 2013

60

CBA Pilot

• ‘Pre-Flight Checks’.• Both NINA and NILA assessments should be administered to as

many pupils as possible across years 4, 5, 6 and 7; and• CCEA, C2k/Capita, and both CBA providers will be monitoring

assessment progress.• CCEA and C2k/Capita will fully support you throughout the CBA

pilot.

You can contact the CBA helpdesk at any stage on: (028) 9026 1274 or email: cba@ccea.org.uk.

For technical support contact C2k on: 0870 6011666.

NB – It is essential that only C2k PCs and laptops are used for the assessments.

61

Assessment Preparation

Teachers should:

•use the online course CBA section of www.nicurriculum.org.uk; •refer to the NILA / NINA Information for Schools Guidance Documents;•be familiar with the NILA / NINA pupil demos;•explore the NILA practice assessment and the NINA practice area;•check pupil usernames and passwords;•ensure that pupils can have a break during the assessment/s;•ensure that pupils know that NINA allows them to have the question/s read aloud if required;•consider resources required; and•check the equipment.

62

Assessment Preparation

Teachers should ensure that pupils:

• view the pupil demos as part of a class discussion;

• know how to pause or exit an assessment by clicking on the red x at the top right hand side of the screen and closing the window;

• know how to resume an assessment by clicking “Start Assessment button”. Pupil should log in again with the same username and password and see “Continue (first) assessment”; and

• understand the assessments they will be completing and have had sufficient opportunity to use the NILA Practice Assessment / NINA Practice Area.

NB - Pupils should be provided with pencils and paper for the NINA assessment.

63

The Practice Assessment / Practice Area

Be sure to:

• take your pupils through the various question/response types;

• draw attention to the instructions for each question type;

• demonstrate how to answer using a data projector or smart board;

• give your pupils plenty of opportunity to use the practice assessment (NILA) / practice area (NINA);

• show pupils how to undo/change an answer;

• evaluate how pupils cope with question types and advise; and

• show that an answer can be deleted or changed any number of times before the ‘Next’ (NILA) / ‘Submit’ (NINA) button is clicked.

NB - Pupils can access the Practice Assessment / Practice Area without usernames/password. No log-ins required.

64

Part 5Guidance & Support for CBA

65

CCEA Support and Resources for Autumn 2013

• Principals’ Information Seminars and Assessment Coordinator Training

• Online Training Course for all new Year 4-7 Teachers

• C2k Instructions re: SIMS data export file

• CCEA email re: accessing NILA/NINA Admin passwords

• NILA / NINA Information for Schools Guidance Documents, including guidance for pupils with SEN

• Website, online support & guidance materials at www.nicurriculum.org.uk

• CCEA - CBA Helpdesk: 028 9026 1274

66

CBA On-line Training

www.nicurriculum.org.uk/cbaonlinetraining

67

68

Website URLs

• NILA www.niliteracyassessment.org.uk

• NINA http://www.ninumeracyassessment.org.uk

• Online Course www.nicurriculum.org.uk/cbaonlinetraining

• NIC Website www.nicurriculum.org.uk

• NILA Training https://training.niliteracyassessment.org.uk

69

On line Training site

• www.training.ccea.org.uk

70