SPECIFICATIONS IN RELATION TO PANEL OF PREFERRED …€¦ · 2.1 Deliverable 1 – Project Plan 3 2.2. Deliverable 2 – Project Management and Reporting 4 ... original tender will

PART A - SPECIFICATIONS

SPECIFICATIONS SPECIFICATIONS IN RELATION TO PANEL OF PREFERRED PROVIDERS (RFA 2015/01) NATIONAL ASSESSMENT PROGRAM - LITERACY AND NUMERACY ITEM DEVELOPMENT (READING, NUMERACY, WRITING AND CONVENTIONS OF LANGUAGE)

Australian Curriculum, Assessment and Reporting Authority ABN 54 735 928 084

Tenderers should refer to PART B – PROPOSED CONTRACT AND PART C – REQUEST TO ESTABLISH A PANEL (INCLUDING APPLICATION RESPONSE SCHEDULES) which sets out the conditions of application and application requirements.

CONTENTS

SPECIFICATIONS 1 1. Services 1

1.1. Introduction 1 1.2. Background 1 1.3. Work available for the panel of preferred providers 2 1.4. Role of a Preferred Provider 3

2. Key Project Deliverables 3 2.1 Deliverable 1 – Project Plan 3 2.2. Deliverable 2 – Project Management and Reporting 4 2.3 Deliverable 3 – Item Development 4

3. Project Guidelines by Domain 6 4. Intellectual Property Management 6 5. Required Contract Material 7 6. Secure Deletion of Data 7 7. Policies, Standards and Guidelines 7 8. Time-frame 7

Specifications in relation to panel of preferred providers (RFA 2015/01) National Assessment Program - Literacy and Numeracy Item Development (Reading, Numeracy, Writing and Conventions of Language) Page 1

http://www.acara.edu.au/

SPECIFICATIONS

1. Services

1.1. Introduction

The Australian Curriculum, Assessment and Reporting Authority (ACARA) is responsible for the development of a rigorous, world class national Australian curriculum from Foundation to Year 12.

To complement the development of an Australian curriculum, ACARA is also responsible for developing and administering a national assessment program that measures students’ progress. ACARA also provides information, resources, support and guidance to the teaching profession.

The National Assessment Program — Literacy and Numeracy (NAPLAN) commenced in 2008 in all Australian schools. Since 2008, all students in Years 3, 5, 7 and 9 have been assessed annually in the second full week of May using national tests in Reading, Writing, Conventions of Language (Spelling, and Grammar and Punctuation) and Numeracy. Test Administration Authorities (TAAs) in each state and territory are responsible for the printing, distribution, administration, marking and reporting of the NAPLAN tests in their jurisdictions.

Performance of students in NAPLAN is reported on the NAPLAN scale for each domain, enabling comparison of results across Years 3, 5, 7 and 9. The NAPLAN scales also allow for the longitudinal tracking of performance by students, schools and systems.

Information about the performance of each NAPLAN cohort of students is released to the public in the annual NAPLAN summary information, NAPLAN National report and on the My School website. Test Administration Authorities also provide performance data to schools and publish individual student reports for parents/guardians.

1.2. Background

Federal, state and territory education ministers agreed that NAPLAN will move online from 2017, over a two-to-three year period. State/territory education authorities will make decisions about the logistics and timing to move NAPLAN online for their state/territory, and during transition years some schools will sit NAPLAN tests online and some schools will sit NAPLAN tests on paper.

In 2012 ACARA developed a research agenda to provide findings on a range of issues and provide evidence that would allow decisions to be made regarding the transition of NAPLAN from a paper-based test to a computer-based assessment. In 2013, ACARA trialled online delivery of a multistage computer adaptive test, or 'tailored test' design. The research study also investigated the cognitive and behavioural engagement of students with the tailored test and the effectiveness of the proposed test design in better targeting underperforming students. A third phase of research was conducted in 2014 which focused on the finalisation of the tailored test design and implementation of the enhancements to NAPLAN tests (including student engagement with technically enhanced items (one for reading and one for numeracy) and another with audio files (for spelling)).

ACARA’s research has also investigated autoscoring technology and technically enhanced items that are designed to assess particular aspects of literacy and numeracy aligned with the Australian Curriculum. Additional research studies are planned for 2015 and 2016 to further inform development of NAPLAN online tests. The studies will look at the interaction between lower levels of computer literacy, reading ability and lack of confidence in affecting the validity of the test for some students, and whether computer literacy skills form part of the measurement construct.

NAPLAN online tailored test design



The introduction of online testing provides the opportunity to administer tests that are better targeted to students’ achievement levels and response styles. The NAPLAN Reading, Numeracy and Conventions of Language online tests are based on a multi-stage computer-adaptive test design. This means that most of the items presented to students are within their approximate zone of achievement. The tests therefore provide more efficient and precise estimates of students’ achievements than do the single-stage paper-based tests in which all students attempt the same set of items.

The proposed tailored test design is a multi-branching test consisting of interlocking item sets (testlets) covering Years 3, 5, 7 and 9 within each testing domain. For Reading, Numeracy and Spelling, each student works through three testlets, allocated by the system at two branching points. For Grammar and Punctuation, each student works though one testlet, allocated by the system based on their Reading score. The Writing test consists of a single prompt.

The number of items in the proposed tailored test design will be almost double that currently used in NAPLAN. This provides an opportunity to broaden the range of skills and content currently assessed.

Each testlet is comprised of approximately one-third of the items that make up a complete NAPLAN test. Figure 1: NAPLAN online multi-stage computer-adaptive test design – tailored test design

1.3. Work available for the panel of preferred providers

The purpose for establishing a panel of preferred providers is to identify, through a formal process, organisations and/or individuals with the required expertise and resources that ACARA may approach to request quotes for specific parcels of item development work related to National Assessment Program (NAP) activities.

Applicants may submit proposals for one or more test domains. Only applicants that are selected onto the preferred panel of providers of the applicable test domain will be able to quote for parcels of work falling within the scope of that test domain.

ACARA will request quotations for the development of packages of test items linked to the Australian Curriculum (http://www.australiancurriculum.edu.au/) in the domains of Reading, Numeracy, Writing and Conventions of Language (spelling, and grammar and punctuation) from the panel as work in this area is required. Once a quotation process has been undertaken for



http://www.australiancurriculum.edu.au/

development of an item package (including evaluation by ACARA), ACARA will raise an official purchase order in favour of the successful Contractor(s). Unless there is good reason to act otherwise (e.g. an evaluation is made that there is an unacceptable risk that the lowest quoting panel member is unable to deliver by the required date), the item packages will usually be awarded to the panel member presenting the lowest quote.

1.4. Role of a Preferred Provider

After formal acceptance of a purchase order from ACARA and initiation of a project, selected panel members are expected to work in close collaboration with ACARA at all stages of the project to ensure that ACARA’s assessment specifications, protocols, project deliverables and timelines are met.

2. Key Project Deliverables

After formal execution of a purchase order and initiation of a project the selected panel members (the Contractor) will provide the following key deliverables to ACARA for review and approval.

2.1 Deliverable 1 – Project Plan

Contractors will be required to provide a project plan after formal acceptance of a purchase order from ACARA. The level of detail required in the project plan will be determined in conjunction with ACARA and it will be based on the magnitude of the work to be undertaken. The project plan will need to be approved by ACARA and include the following:

• a description of item development methodology

• a risk management plan

• a quality assurance plan

• a description of auditing processes

• a timeline for deliverables and critical deadlines.

2.1.1 Item Development Methodology

The contractor will be required to provide a description of how the methodology for item development outlined in the original tender will be applied.

2.1.2 Risk Management Plan

The Contractor will be required to provide a Risk Management Plan after formal acceptance of a purchase order from ACARA. The risk management plan will outline how risks identified in the original tender, as well as any further risks that have been identified, will be managed in relation to the project. The Risk Management plan will need to be approved by ACARA. This plan must include how all aspects of security are addressed and include the names of all persons involved in the process and their respective roles.

2.1.3 Quality Assurance Plan

The Contractor will also be required to provide a Quality Assurance Plan after formal acceptance of a purchase order from ACARA. The quality assurance plan must be approved by ACARA and include descriptions of how the Quality Assurance and Quality Control processes outlined in the



original tender will be implemented and monitored to produce high-quality test materials by the Contractor throughout the project.

The Quality Assurance Plan must specify in detail all and any perceived risks that may impact on the quality of the contract deliverables, and must provide detailed advice on the strategies for risk management for each identified risk. It is not sufficient to provide information only about generic quality assurance systems and quality manuals.

2.1.4 Auditing Processes

The Contractors will provide a description of their auditing processes which outlines how they will ensure compliance with all test specifications.

2.1.5 Timelines

Prior to the commencement of a project, the Contractor will negotiate agreed timelines with ACARA.

2.2. Deliverable 2 – Project Management and Reporting

After execution of a purchase order and initiation of a project Contractors will be required to:

i. provide regular status reports to ACARA ii. be available for regular dialogue with ACARA personnel during each stage of item

development iii. provide agreed milestone reports to ACARA by the specified dates iv. provide risk alerts to ACARA immediately upon detection of issues considered high risk to

the project v. respond to feedback from ACARA following stakeholder reviews vi. modify items in response to feedback from ACARA vii. attend nominated review meetings viii. meet the project deadlines.

2.2.1 Meetings

The Contractor will be required to communicate with the ACARA Project Manager and ACARA Test Managers at key juncture points of the item development process. These meetings will be organised by ACARA and held at a time and location agreed with the Contractor(s).

A Contractor may be asked to attend a project close-out meeting with key members of ACARA’s project team at an agreed time after the delivery of the final test materials to discuss aspects of the project.

2.2.2 Progress Reports

Throughout a project, Contractor/s will regularly consult with the ACARA assessment team in relation to the day-to-day management of the project and measurement issues. Written progress reports are to be submitted at nominated times to ACARA outlining progress on the key deliverables and issues that have arisen. Exception reports are required in the event of unforeseen circumstances.

2.3 Deliverable 3 – Item Development



Item development will be contracted for one or more of the following domains. Detailed item specifications are outlined in the Item Development Guidelines in Appendix A.

2.3.1 Reading

• Reading testlets (stimuli and items)

• Reading stimuli (no items)

• Reading items (no stimuli)

Panel members will be requested to quote for whole testlets consisting of units of items and stimuli, or for just stimulus materials. Panel members may be asked to quote for one year group, a combination of year groups, or specified targeted testlets, e.g. challenging testlets (F) for all year groups. Contractors will develop an iterative matrix outlining the match between the stimuli being developed and the supplied specifications.

2.3.2 Numeracy

• Numeracy testlets

Panel members will be requested to quote for whole testlets consisting of sets of items. Panel members may be asked to quote for one year group, a combination of year groups, or a specified targeted testlet, e.g. challenging testlets for all year groups.

2.3.3 Conventions of Language

• Spelling testlets

• Grammar and Punctuation testlets

Panel members will be requested to quote for whole testlets consisting of sets of items. Panel members may be asked to quote for spelling or for grammar and punctuation; for one year group, a combination of year groups, or a specified targeted testlet, e.g. challenging testlets for all year groups.

2.3.4 Writing

• Writing prompts

For Writing, panel members will be requested to quote for packages of prompts targeted for year groups and/or genre.

2.3.5 Item Submission

The Contractor(s) will be required to submit and revise all stimulus materials, items and associated metadata using the online Item Authoring and Review System (IARS) provided by ACARA’s technical partner, Education Services Australia (ESA). The system is compliant with IMS Global (QTI 2.1 and APIP 1.0) standards. See Appendix B for the IARS user guides.

Contractors will also be required to provide source files for all graphics, stimulus material and/or prompts (including any Word, Illustrator, Photoshop, InDesign, or image files such as PNG or JPEG files) and audio files (MP3 files).

The design of items must comply with item guidelines and requirements, the system and display requirements of IARS and, in the case of paper-based items, the NAPLAN Style Guide (a copy of which will be provided to members of the panel of preferred providers).

Material that contains third party intellectual property can only be submitted if it strictly complies with the IP management requirements as detailed in the contract.



The item development phase will include opportunities for ACARA assessment officers and other stakeholders to review the items using pre-determined criteria (including quality, validity and fit to specifications) and provide feedback to item writers for item refinement. The contractor will be required to amend and/or replace test material in accordance with feedback provided during the review process.

All phases of the item-development process will be monitored by ACARA assessment officers for quality, risk and timeliness. (See page 105 of the IARS Item Authoring and Workflow user guide for a workflow diagram.)

All work completed by contractors for possible inclusion in tests must be kept highly secure during all phases of the work. All personnel involved in the work must be fully aware of the security requirements. This includes the electronic transmission of materials.

Administration, marking and data analysis of trial and final forms are out of scope for this panel.

3. Project Guidelines by Domain

Detailed guidelines and requirements for item development for Reading, Numeracy Conventions of Language and Writing are attached as appendices. These guidelines outline the specifications for the development of NAPLAN 2016 and the research trials for NAPLAN online. These guidelines and requirements are currently being revised and selected panel members will be provided with updated versions prior to being requested to quote for parcels of work. However, the attached guidelines and requirements should form the basis in support of an application to be considered for ACARA’s panel of preferred providers for National Assessment Program - Literacy and Numeracy item development (Reading, Numeracy, Writing and Conventions of Language).

The Contractor(s) will be required to ensure that item development conforms to the guidelines. Item quality will be evaluated against these criteria, and additional criteria supplied by ACARA.

4. Intellectual Property Management Contractors will be required to meet all obligations in relation to intellectual property set out in clause 14 of the Deed (Part B).

Contractors should also be aware that innovative stimulus items such as video and music clips may contain a number of separate copyrights owned or controlled by different parties. For example:

• Video – copyright exists in the screenplay, visual images and any music, recordings or third party footage that may be incorporated in the clip;

• Music – copyright exists in the music, lyrics and the sound recording of the music and lyrics.

It is the Contractor’s responsibility to identify all copyright owners for all third party materials and obtain the necessary copyright permissions from each of them in accordance with clause 14 of the Deed (Part B).

ACARA will not request multimedia stimuli for NAPLAN in the near future but, given technological changes and capacity building in schools, anticipates these materials will be requested in the future.



5. Required Contract Material

The following list of Materials is to be developed by the Contractor under the purchase order in which ACARA would retain intellectual property:

Test materials — Reading stimulus materials, Reading items, Numeracy items, Conventions of Language (spelling and grammar and punctuation) items, Writing prompts, stimulus and item metadata, and marking guides.

6. Secure Deletion of Data

Contractor(s) will be required to provide evidence that any information on item performance including but not limited to any data used in preparation for the delivered materials have been securely erased from all media including, but not limited to, networks, servers and computer hard drives. The timeline for this activity will be negotiated with the contractor.

7. Policies, Standards and Guidelines

Contractors should refer to the following documents:

Macquarie Dictionary, Fifth Edition, (2013), Macquarie Dictionary Publishers Pty Ltd, Sydney.

Style Manual, (2006), Commonwealth of Australia, John Wiley and Sons, Australia.

Australian Standards for Document Management (AS ISO 15489);

Commonwealth Fraud Control Guidelines (April 2011)

Data Standards Manual Student Background Characteristics (ACARA, 2012), as updated from time to time.

8. Time-frame

Note to Applicants: The timeframe will be confirmed as ACARA calls for quotations from the panel of preferred providers, and a final timeframe will be included in the purchase order.

Contractor(s) may be required to submit whole testlets in separate batches. Each batch must be representative of the final test composition, and contain a balance of item sub-strands, item types and difficulty across each of the four year levels.

A Contractor will be required to meet the timelines for the provision of items for Reading, Numeracy, Conventions of Language (spelling and/or grammar and punctuation) and Writing as specified in each Purchase Order issued by ACARA in accordance with the relevant Project Brief.

It is expected that Requests for Quote (RFQ) in support of Reading, Numeracy, Conventions of Language and Writing item development will commence from November 2015. Specific item development timelines and delivery dates will be defined with each preferred provider depending on the parcel of work agreed upon in the relevant Project Brief. However, it is expected that the delivery of test materials to ACARA will commence from December 2015.

Below are key testlet trialling dates in support of which ACARA will source item development from this panel:



Activity Testing Dates

NAPLAN online trial administration August – September 2016



APPENDIX A: ITEM DEVELOPMENT GUIDELINES

AVAILABLE ON REQUEST VIA THE CONTACT OFFICER

Item authoring and workflow

This guide is an overview of the functions provided to item authors through the current Item Authoring and Review System.

◘ Login and change password

◘ Create items, resources and stimuli

◘ Associate metadata with items, stimuli and resources

◘ Use the workflow and comment functions

Table of contents

Login ............................................................................................................................................................................................................................................................................. 6

Two Factor Authentication........................................................................................................................................................................................................................................ 7

Dashboard .................................................................................................................................................................................................................................................................... 8

Item Authoring System password rules ..................................................................................................................................................................................................................... 10

How to set a new password ................................................................................................................................................................................................................................ 10

Interactive gap match ................................................................................................................................................................................................................................................ 11

Scoring ................................................................................................................................................................................................................................................................ 11 Steps ................................................................................................................................................................................................................................................................... 11

Interactive graphic gap match ................................................................................................................................................................................................................................... 14

Scoring ................................................................................................................................................................................................................................................................ 14 Steps ................................................................................................................................................................................................................................................................... 14

Hotspot ...................................................................................................................................................................................................................................................................... 19

Scoring ................................................................................................................................................................................................................................................................ 19 Steps ................................................................................................................................................................................................................................................................... 20

Text entry ................................................................................................................................................................................................................................................................... 24

Scoring ................................................................................................................................................................................................................................................................ 24

Steps ................................................................................................................................................................................................................................................................... 24

Interactive match ....................................................................................................................................................................................................................................................... 27

Scoring ................................................................................................................................................................................................................................................................ 27

Steps ................................................................................................................................................................................................................................................................... 27

Multiple choice and multiple choices ........................................................................................................................................................................................................................ 32

Scoring ................................................................................................................................................................................................................................................................ 32

Steps: Multiple choice ......................................................................................................................................................................................................................................... 32

Steps: Multiple choices ....................................................................................................................................................................................................................................... 35

Interactive order ........................................................................................................................................................................................................................................................ 38

Scoring ................................................................................................................................................................................................................................................................ 38

Steps ................................................................................................................................................................................................................................................................... 38 © 2014 Education Services Australia Ltd, unless otherwise indicated Last updated: February 2015 2

Interactive graphic order ........................................................................................................................................................................................................................................... 41

Scoring ................................................................................................................................................................................................................................................................ 41 Steps ................................................................................................................................................................................................................................................................... 41

Extended text ............................................................................................................................................................................................................................................................. 46

Scoring ................................................................................................................................................................................................................................................................ 46

Steps ................................................................................................................................................................................................................................................................... 46

Hot text ...................................................................................................................................................................................................................................................................... 49

Scoring ................................................................................................................................................................................................................................................................ 49

Steps ................................................................................................................................................................................................................................................................... 49

Slider .......................................................................................................................................................................................................................................................................... 53

Scoring ................................................................................................................................................................................................................................................................ 53

Steps ................................................................................................................................................................................................................................................................... 53

Interactive position object ......................................................................................................................................................................................................................................... 57

Scoring ................................................................................................................................................................................................................................................................ 57

Steps ................................................................................................................................................................................................................................................................... 57

Select point ................................................................................................................................................................................................................................................................ 62

Scoring ................................................................................................................................................................................................................................................................ 62

Steps ................................................................................................................................................................................................................................................................... 62

Drawing ...................................................................................................................................................................................................................................................................... 66

Scoring ................................................................................................................................................................................................................................................................ 66

Steps ................................................................................................................................................................................................................................................................... 66

Interactive associate .................................................................................................................................................................................................................................................. 69

Scoring ................................................................................................................................................................................................................................................................ 69

Steps ................................................................................................................................................................................................................................................................... 69

Interactive graphic associate ..................................................................................................................................................................................................................................... 72

Scoring ................................................................................................................................................................................................................................................................ 72

Steps ................................................................................................................................................................................................................................................................... 72

Inline Choice .............................................................................................................................................................................................................................................................. 76

Scoring ................................................................................................................................................................................................................................................................ 76

© 2014 Education Services Australia Ltd, unless otherwise indicated Last updated: February 2015 3

Steps ................................................................................................................................................................................................................................................................... 76

Composite items ........................................................................................................................................................................................................................................................ 79

Scoring ................................................................................................................................................................................................................................................................ 79

Steps ................................................................................................................................................................................................................................................................... 79

Copying an item to a new module ............................................................................................................................................................................................................................. 84

Steps ................................................................................................................................................................................................................................................................... 84

Menu options available in item editor ...................................................................................................................................................................................................................... 85

Edit ...................................................................................................................................................................................................................................................................... 85

Insert ................................................................................................................................................................................................................................................................... 86

View .................................................................................................................................................................................................................................................................... 88

Format ................................................................................................................................................................................................................................................................ 89

Table ................................................................................................................................................................................................................................................................... 90

Other menu options ............................................................................................................................................................................................................................................ 91

Stimulus creation ....................................................................................................................................................................................................................................................... 93

How to create a new stimulus ............................................................................................................................................................................................................................. 93

How to add a stimulus to items ........................................................................................................................................................................................................................... 96

Resource creation ...................................................................................................................................................................................................................................................... 97

How to add a single resource ............................................................................................................................................................................................................................. 98

How to add bulk resources ................................................................................................................................................................................................................................. 99

Steps ................................................................................................................................................................................................................................................................... 99

Resources types ....................................................................................................................................................................................................................................................... 101

Support for MathML ................................................................................................................................................................................................................................................ 103

Steps ................................................................................................................................................................................................................................................................. 103

Workflow ................................................................................................................................................................................................................................................................. 104

Workflow states .................................................................................................................................................................................................................................................... 107 Draft (available to item authors) ....................................................................................................................................................................................................................... 107

Internal review (available to item authors) ........................................................................................................................................................................................................ 107

Management review (read only to item authors) .............................................................................................................................................................................................. 107

Expert review (read only to item authors) ......................................................................................................................................................................................................... 108


Adjudication (read only to item authors) ........................................................................................................................................................................................................... 108

Requires modification (available to item authors) ............................................................................................................................................................................................ 108

Modification review (available to item authors) ................................................................................................................................................................................................ 109 Rejected (read only to item authors) ................................................................................................................................................................................................................ 109

Accepted (read only to item authors) ................................................................................................................................................................................................................ 109

Comments ............................................................................................................................................................................................................................................................ 110

Steps ................................................................................................................................................................................................................................................................. 110

Versioning ............................................................................................................................................................................................................................................................ 110

Diagram showing versioning within the workflow ............................................................................................................................................................................................. 112 Advanced filter for Bulk Workflow Change tool (for Items) .................................................................................................................................................................................. 113

Appendix: item types and metadata for NAPLAN ................................................................................................................................................................................................... 114

Item types ......................................................................................................................................................................................................................................................... 114 Metadata ........................................................................................................................................................................................................................................................... 116

Terms ....................................................................................................................................................................................................................................................................... 126

Roles ........................................................................................................................................................................................................................................................................ 128


Login

Username Enter your username.

Password Enter your password. For newly created accounts, users receive an email at their registered email address which has a link to set password.

Log in Select Log in.

Forgot your password? Click here and enter your username or email address, then click Submit. A link to reset your password will be emailed to you.


Two Factor Authentication Two factor authentication provides an extra level of security. After you have entered your login credentials, you will receive a passcode on your mobile phone.

Submit Enter the passcode and click Submit.

Passcode Refer to your mobile phone to retrieve the passcode.

New passcode If you need to send a new passcode to your mobile phone, select Issue new passcode.


Dashboard

Main menu Select this icon to view a dropdown box displaying the following options: • Item authoring • My account • Log out.

Dashboard tiles These icons display the numbers of items and stimuli in the different workflow states.

Create an item This is a shortcut to creating a new item.

Create a stimulus This is a shortcut to creating a new stimulus.

Item authoring This section shows available menu items; these items will differ depending on your role or permissions in the system.


Recently viewed This section displays records most recently viewed by the logged-in user.

My Account Select My account to change your password.


Item Authoring System password rules • Minimum password length is eight characters. Passwords must have an upper case letter, a number and a symbol. • Passwords need to be changed every 60 days. • Previously set passwords cannot be reused (the system remembers the last three passwords). • Passwords cannot be changed within two days of the last change. • One login session is allowed at a time. • You are allowed four login attempts before the account will be locked. • Allow 30 minutes before attempting to log in again if the account becomes locked.

How to set a new password

Change password From the Manage dropdown menu, select Change password.

New password Enter a new password.

Confirm password Re-enter the new password.

Save password Select Save password.

Old password Enter the current password.


Interactive gap match An interactive gap match item asks the student to drag text or images from a source list into hotspots within text.

This item contains a number of hotspots that the student can fill from a set of choices. After filling the hotspots with their choices the student can review the content with the hotspots filled in, which provides context to their selections.

Scoring This item type is marked automatically. Score allocation is set by the item author at the time of authoring.

Steps

Items From the Item authoring menu select Items.

Add test item From the Manage dropdown box select Add Test Item.


Item stem Type the question in the item stem.

Orientation Above – drag items from above. Below – drag items from below.

Item type Select Interactive gap match.

Gaps Right click to view the available options. Gaps are added or removed by right mouse clicking and selecting Insert gap or Remove gap.

Discipline and module Select the Discipline and Module you want the item to belong to.

Name Enter a name for the item. The item’s identifier will be automatically generated.

Two scoring options are available: Correct/Incorrect The question is marked as either correct or incorrect and basic scoring applied. Per distractor Each answer or selection is marked and scored separately.


Note: If a word is added to the text column but is not mapped to a hotspot, it will still appear as an option for the student to drag.

Mapping Link the gap to the correct word or image in the text column by clicking in the box and selecting Gap 1, Gap 2 etc. A gap can be associated with more than one selection.

Match Max Here, you can define how many times a source option can be dragged to a gap. A blank (or ‘0’) entry means there is no limit to the number of times a source option can be used.

Text Enter the selections that students will drag into the text column fields.

Image You can add images instead of text. The image column opens the resource browser to allow image addition.

Preview You can preview the item by selecting Preview from the item header bar.

Scoring This list shows all source and gap associations, allowing them to be selected as a correct answer and to be given a score.

The student must drag and drop all the correct options to get full marks.


Interactive graphic gap match An interactive graphic gap match item is an image containing a set of gaps, defined as ‘hotspots’. The student answers the question by dragging images or text from a set of options into the hotspots.

After filling the hotspots with their choices the student can review the content with the hotspots filled in, which provides context to their selections.

Care should be taken when designing this item to ensure the hotspots on the image are a suitable size to receive the required answer options. If the student indicates the association by positioning an answer over a hotspot (e.g. drag and drop) the system will ‘snap’ that answer into position and force the text or image to match the dimensions of the hotspot.


Steps




Discipline and module Select the Discipline and the Module you want the item to belong to.

Item stem Enter the question in the item stem.

Item type Select Interactive graphic gap match.

Orientation Above – drag items from above. Below – drag items from below.

Prompt Select Add prompt to add further instructions about the item.




Response image Go to ‘Please select an image to start’ and click on the icon. Next, select an image from the available resources, or upload a new image.

Hotspots Once the image is loaded, hotspots can be created by clicking on the image where you would like the hotspot to appear.

Shape of hotspot Select the shape of the hotspot to be either a rectangle or a circle. Then set the dimensions for the source image. Add a label before selecting OK. The label will not be visible to the student.

Text This column shows the labels for each hotspot that has been created.

Match max This column defines the maximum number of choices the hotspot can accept.


Add ‘source’ images and/or text. The student will be able to drag these options into the hotspots. For the purpose of this example images have been loaded.

Image Select the image icon to load a resource from the resource browser, or create a new one.

Text Alternatively you can type text for the source selection in this field.

Mapping Link the destination to the source in the mapping column by clicking in each box and selecting the correct answer. • One source can be mapped to more

than one destination. • One destination can be mapped to more

than one source.

Match maximum This column defines the maximum number of times the source can be used as a selection.


Scoring In the Correct column, define the correct response. In the Score column, set a score for each destination and source pair. In this example the correct response will receive a one point score. The student will get the score for each pair they create. It is possible to define negative scores for incorrect pairs.

Save Select Save test item.



Hotspot A hotspot item asks the student to select one or more areas (or ‘hotspots’) on an image. These hotspots are visible to the student during the test and they are highlighted when the student selects them.

You would create a hotspot item if you wanted students to choose from options that were physically spaced from each other (for example, if you wanted students to select from a range of options on a map).

• Hotspots can be circles, rectangles or polygons. • There is no limit to the number of hotspots that can be created. • Hotspots can overlap. The hotspot highest up the list will be the one that is ‘on top’ and will be the one that is selected if the student clicks on the overlap.

This could be used to award more points for a more accurate answer.



Steps





Item stem Enter the question in the item stem field.

Item type Select Hotspot.



Prompt Select Add Prompt to add item prompt.


Response image Select the image icon to load an image from the available resources.

Correct answer Identify correct answers by checking the box beside them.

Maximum and minimum choices Enter the maximum and minimum number of choices a student must make before they are allowed to submit their response.

Delete You can remove a hotspot by selecting this icon.

Hotspots Select a hotspot shape and then click on the image where you would like the hotspot to be placed. Hotspots can be polygons, rectangles or circles. Click on the image and drag the cursor to create round or rectangular hotspots. To create a polygon hotspot, click on the image at multiple points to define the shape and size of the hotspot.

Label Label your hotspots. These labels will not be visible to candidates.

Save Test Item Select Save Test Item to save the item.

The position and dimensions of the hotspot can be altered by entering values directly into these fields.




Text entry A text entry item requires a student to provide a correct keyword. It is possible to enter many correct keywords.


Steps




Expected Length This number determines the width of the entry field.


Item type Select Text entry.

Match Type all possible correct answers in the response box. Note: Answers are case sensitive; therefore include all correct versions of an answer.

Name and identifier Enter a name for the item. The identifier will be automatically generated.

Input restriction This restricts the characters that can be typed by the student. • None – no restrictions • Integer – only whole

numbers • Decimals – numbers with

decimal places • Alphabet – only a to z

(upper and lower case) • Alphanumeric – a to z

(upper and lower case) and digits.


Four scoring options are available: Correct/Incorrect The question is marked as either correct or incorrect and basic scoring applied. Per distractor Each answer or selection is marked and scored separately. Numeric Range – Correct/Incorrect This option is available when the Input Restriction is either Integer or Decimal. The correct answer must fall within a specified range. Multiple ranges can be entered. The question is marked as either correct or incorrect and basic scoring applied. Numeric Range – Graded This option is available when the Input Restriction is either Integer or Decimal. Each range is marked and scored separately.


Preview You can preview the item type by selecting Preview from the item header bar.


Interactive match An interactive match item allows the student to match images or text correctly by dragging them into pairs, using checkboxes to select a match or drawing lines to create a match.


Steps






Orientation Select how you want the source choices to be presented for the Drag and Drop sub type. • Horizontal from above • Horizontal from below • Vertical from above • Vertical from below

Item type Select Interactive match.

Sub type Select from either Drag and drop, Checkbox or Draw lines. Prompt

Add instructions or information regarding the item.




Match maximum Defines how many times a source selection can be dragged to a destination category.

Source This is the selection that will be dragged by the student. Images or text can be used. For the purpose of this example images have been loaded.

Image Select the image icon to add an image for students to drag. Once the image has been added, you can set the width and height of the image.

Text Enter text options for students to drag. Actions

Selecting the delete button in the actions column deletes the source selection as an option.


..

Destinations The text and/or images you enter into the Destinations field will become the category headings when the student views the question. The students will then drag their answers into the different categories you have created.

Map The Map field allows you to map the correct answer/s to each category. Click a map field to choose from the answer options you created under ‘Source’.

Scoring All pairs are shown in this area. Select the Correct checkbox for each correct combination. Enter the score for each correct or incorrect answer pair.


Save Scroll down and select Save test item to save the item.


Checkboxes This example shows the item type displayed with checkboxes. Note: the wording of the item stem may need to be altered depending on which sub-type is selected.

Draw lines This example shows the item type displayed with checkboxes. Note: the wording of the item stem may need to be altered depending on which sub-type is selected.


Multiple choice and multiple choices • A multiple choice item allows the author to specify multiple answers, but only one can be correct. • A multiple choices item allows the author to specify multiple answers with several being correct. • The standard presentation for a multiple choice question is a list of radio buttons. • The standard presentation for a multiple choices question is a list of checkboxes.


Steps: Multiple choice




Response Insert answer options in the table provided and identify the correct answer by checking the box beside it. Text or images may be used as options.

Shuffle Select the checkbox to have the options shuffle randomly.



Item type Select Multiple choice.

Orientation Distractors (answer options) can be displayed vertically or horizontally.





Preview You can preview the item type by selecting Preview from the details form. This example shows the distractors displayed vertically.


Steps: Multiple choices





Response Type in the answer options or add images and check the boxes beside the correct answers.

Save Select Save test item

Shuffle Select the checkbox to have the options shuffle randomly.



Item type Select Multiple choices.

Orientation Distractors (answer options) can be displayed vertically or horizontally.





Interactive order An interactive order item requires a student to order a set of text or images.

Note: You can choose to display the answer options in a particular order or you can choose for them to be shuffled.


Steps






Item type Select Interactive order.

Orientation Select either horizontal or vertical orientation.


Order of correct answer Add text answers or images in the exact order they should be in to be marked correct.

Display order of answer options Enter the order in which the items will be displayed. The number entered fixes the answer in that position, with 1 being first. Note: Leaving the field blank causes the answer to be shuffled in the list.





Interactive graphic order An interactive graphic order item requires a student to drag numbers onto defined and visible hotspots on an image to indicate the correct order. The student must ensure the arrow is located inside the hotspot (which will turn green) before the number will ‘snap’ into position.

The creation of this item type is very similar to a hotspot item. The two differences are:

• instead of defining which hotspots are correct, the order of the correct hotspots must be defined • the location of the numbers to be dragged must be defined.


Steps





Item type Select Interactive graphic order.

Orientation Select the orientation, either above or below.



Response image Select an image from the available resources for the item or create a new one.



Edit hotspots You can change the hotspot’s dimensions and label. Select OK to save any changes or Delete to remove the hotspot from the image.

Hotspots Hotspots are added by clicking on the image in the correct location. A pop up box will appear and it is here that you can select rectangular or circular shapes. Enter dimensions of the hotspot and add the label. The label will not be visible to the student. Complete the process by selecting OK.

Image size Change the size of the image by entering the new width and height.


Maximum and maximum choices Enter the maximum and minimum number of choices a student may make before they may submit their response.


Actions The hotspot can be deleted by selecting the red icon.

Correct order Enter the correct order of the answer options.




Extended text An extended text item requires students to write an extended amount of text. This item can be used for answers that vary from a single sentence to a full essay.

Scoring This item type cannot be marked automatically and must be manually scored.

Steps





Item type Select Extended text.

Item stem Type the question in the item stem.



Line count Setting this number determines the displayed height of the entry field (in lines).

Word limit Setting this number determines how many words the response should be limited to. If left blank there is no limit.

Word limit action After setting a word limit, the action can be set. The choices are: Ignore, Advise and Restrict. These choices cause the following effects when a student enters more words than you have set as the limit: • Ignore – the system will

ignore the fact that they have gone over the limit.

• Advise – the system will advise them that they have gone over, but will allow them to continue.

• Restrict – the system will inform them of the limit and notify them that they cannot exceed that limit.




Hot text A hot text item requires the student to select parts of the text to highlight. The author specifies which parts of the text should be able to be highlighted and also which highlights are correct.

Scoring This item type is marked automatically. Score allocation is set by the Item author at the time of authoring.

Steps





Item type Select Hot text.

Item stem Enter the instruction in the item stem.

Prompt Click Add prompt to add additional instructions if required.


Spot visibility The selectable hot text areas can be made visible or hidden until cursor is over the spot. Note: this feature is not QTI compatible



Correct Check the correct box beside the correct words for the question. In this example it is the proper nouns.

Hot text In the Interaction body section, right click where you want to add hot text and select Insert text spot. Once a hot text spot has been added it will appear underneath the item stem.

Interaction body text In the Interaction body section, enter the text that the student will interact with.

Score Scoring can be set for correct and incorrect answers.

Answer The spots appear in the answer column ready for the answer to be added.





Slider A slider item requires students to choose a correct value by dragging a slider to the correct position.


Steps




Item type Select Slider.



Orientation Select a horizontal or vertical orientation of the slider bar.

Reverse orientation By default, the slider displays the lower bound on the left or the bottom, depending on orientation. If Reverse orientation is checked the lower bound appears on the right or top of the slider.




Upper bound Enter the highest value possible for the response.

Step Enter a number in the Step field to define the increments on the slider bar. The default value is 1. If you have selected 'Allow floating point numbers', you can enter decimal increments. If you enter a very small increment, the markers on the slider may appear to be a continuous line.

Allow floating point numbers If this box is checked, decimal numbers (e.g. 10.5) are allowed in student responses. If left unchecked responses must be whole numbers.

Lower bound Enter the lowest value possible for the response.

Show step labels Select this box to show the step labels on the slider. If there are too many steps, not all the step labels will be able to be rendered.


Answer Enter the correct answer.




Interactive position object An interactive position object item type requires a student to drag one image and position it on another (larger) image.

Multiple images can be dragged by defining hotspots. This item differs from the other hotspot item types by not displaying any indication to the student of where the hotspots are on the image. In addition, the selections do not ‘snap’ into place to reveal the presence of hotspots.


Each image/hotspot combination can be given different scores. If multiple hotspots overlap, the highest one on the list is the most important. This means you can author a precise area worth, for example, 10 points, and then a wider area worth 1 point.

Steps






Item type Select Interactive position object.

Response image Select an image from the available resources for the item or create a new one. This is the larger image that the smaller images will be dragged to.





Edit hotspots You can always change the dimensions and label of the hotspot Select OK to save any changes or Delete to remove the hotspot from the image.

Hotspots Hotspots are added by clicking on the image in the correct location. A pop up box will appear and it is here that you can select rectangular or circular shapes. Enter dimensions of the hotspot and add the label. The label will not be visible to the student. Complete the process by selecting OK.


Source images Add source images that the student will drag on to the larger images. Click on the image icon to load each image. Note: You can change the height and width of the image to fit the hotspot area.

Mapping Click on the mapping box to link the source image to the correct hotspot destination by selecting it from the dropdown box.


Minimum and maximum choices Enter minimum and maximum number of choices a student may make and the maximum number they are allowed to make with each source image.

Actions The hotspot can be deleted by selecting the Delete icon.

Scoring Select the Correct checkbox beside each correct answer pair and assign a score. You can have different scores for each answer. In this example the brain and heart earn one point and the stomach earns two.

Destination text This is the label that you gave the hotspot when you created it.


Preview You can preview the item type by selecting Preview from the item header bar. Note the lack of hotspots being displayed to the student.


Select point Like hotspot items, a select point item type is a graphic interaction. The student is required to select one or more points.

A select point interaction is different to the hotspot interaction in two ways:

• the hotspots are not visible to the student • the student clicks on the image to answer the question.


Steps






Response image Select an image from the available resources for the item or create a new one.

Item type Select Select point.





Edit hotspots You can always change the dimensions and label of the hotspot.

Select OK to save any changes or Delete to remove the hotspot from the image.

Hotspots Hotspots are added by clicking on the image in the correct location. A pop up box will appear and it is here that you can select rectangular or circular shapes.

Enter dimensions of the hotspot and add the label. The label will not be visible to the student.


Correct Identify the correct answer by checking the box beside it.

Score Set the score for the hotspot.


Minimum and maximum choices

Enter minimum and maximum number of choices a student may make before they submit their response.


Drawing The drawing item type allows the student to draw using basic drawing tools. The author can specify an image to use as the background for the student to draw on. The item author has no control over the tools available to the student.

Scoring This item type is not automatically marked, so there is no need to add marking information.

Steps






Item type Select Drawing.

Prompt Detailed information or instructions can be added to help the student complete the item successfully.

First click Add prompt. A dropdown screen will appear. More information about the dropdown screen is available on the next page.



Select image Select an image from the available resources for the item or create a new one.




Interactive associate

The interactive associate item type allows students to create pairs of items using drag and drop. The order of the pairs has no bearing on the response. The pairings can be images or text (or a combination).


Steps






Item type Select Interactive associate.

Minimum and maximum associations Set a minimum and a maximum number of associations that the student may make. If you set a maximum number larger than the minimum, an Add pair button will become available to the student.




Display order You can order the items that the student will drag and drop. If left blank the system will automatically shuffle the order.

Text and images Text and images can both be added. Type the text in the text column, or click on the image icon and load an image.

Match minimum Set a minimum and maximum number of times this item can be used. (The student will not be able to move on until the item has been used a number of times within this range.)

Mapping When you map items to each other, the pair will automatically appear in the Scoring section. In the Scoring section, you can specify whether the answer is correction and assign a score.



Interactive graphic associate

The interactive graphic associate item type uses a background image with hotspots defined on it. The student drags lines between the hotspots to associate the hotspots with each other. Use this item type if the hotspots are related to the background image. Otherwise, use the interactive associate item type (which does not have a background image).


Steps






Item type Select Interactive graphic associate.

Prompt More detailed information or instructions can be added to help the student complete the item successfully. Select Add prompt to add more information or instructions.

Response image Click the icon to select an image from the Item Authoring and Review System or upload a new image from your browser.




Match minimum Set a minimum number of times this item must be used. (The student will not be able to move on until the item has been used the set number of times.)

Hotspots Click on the image to create each hotspot. The shape and dimensions of the hotspots can be customised to fit into the image. Hints: • Don’t make the spots too big; large spots make it difficult for students to

draw/delete the connecting lines. • Click the middle of the connecting line to delete it. • The hotspots should not overlap each other as it is too difficult to differentiate

between the two. The system will always snap the line to only one of the spots.

Label Create a label for each hotspot. The label will not be visible to the student.

Mapping When you map items to each other, the pair will automatically appear in the Scoring section. In the Scoring section, you can specify whether the answer is correction and assign a score.

Match maximum Set a maximum number of times this item may be used. For the purpose of this example Botany Bay needs a maximum of one match as it is the beginning of the voyage. Bustard Bay needs a maximum of two matches as it must be linked to both Botany Bay and the next destination.

Minimum and maximum associations Set the minimum and maximum number of associations that the student may create. In general, the maximum will match the number of hotspots as this allows for the widest scope of answers.


Scoring • The order of the

mapping does not matter.

• Select the correct mappings and define the score for each mapping.



Inline Choice Please note: This item type is only available for use in the composite item type.

In an inline choice interaction within a composite item, students select an option from a drop-down list. The drop down list can be positioned ‘in line’ with a body of text.


Steps Follow the steps detailed in the next section of this guide for the creation of a composite item.

An inline choice interaction can then be added in the Item Interactions section.

Item Interactions Select the Create one now link.


Interaction type Select Inline Choice.

Name Enter a name for the interaction. The identifier will be automatically generated.

Answer Enter the options that will appear in the drop-down list.

Two scoring options are available: Correct/Incorrect The question is marked as either correct or incorrect and basic scoring applied. Per distractor Each answer or selection is marked and scored separately. Note: This choice of scoring options is only available if Multiple graded responses has been selected as the scoring mode for the composite item this interaction forms part of.


Once the inline choice interaction has been saved, it can be added to a body of text in the Item Body section.

Right-click in the location within the text where you want the drop-down list to appear and select Insert Interaction from the options.

Select the interaction you wish to insert and select OK.


Composite items A composite item is a test item that consists of two or more item interactions. For example, a composite item might consist of a multiple choice question followed by an extended text field. Students are required to complete all of the interactions within a composite item to be considered to have successfully answered the composite item.

Scoring Scoring can be assigned for the composite item as a whole, or the individual interactions can be scored and marked separately. Score allocation is set by the item author at the time of authoring. Scoring information for individual interaction types is covered in the descriptions of the different item types.

Steps


Add Composite Item From the Manage dropdown box select Add Composite Item.


Discipline and Module Select the Discipline and the Module you want the item to belong to.


Save Select Save Composite Item.

Scoring Mode Select the Scoring Mode you want to set for the composite item. Two Scoring Modes are available: Single Correct Response All interactions must be correctly answered for the item to be marked as correct. Multiple Graded Responses Scoring can be specified for each individual interaction.


Item Body and Item Interactions Select the Create one now link.

Edit Composite Item To edit the item details, select Edit Composite Item.


The process for creating an interaction is the same as described earlier for the various item types. Any combination of interactions can be used in a composite item.

When you have created one or more interactions, you can specify where each interaction should go within the item in the item body field. ‘Text entry’ and ‘inline choice’ interactions can appear in the middle of a line of text, whilst all other interaction types occupy separate paragraphs.

Item Body Enter the question text in this field.


Interactions can be organised within the item body field by right-clicking and choosing ‘Insert Interaction’, then selecting the desired interaction from the list as shown below.

Note: If interactions are not organised in this way, they will appear after the item body text in the order in which they are listed in the Item Interactions field.

Item Body

Right-click in Item Body field and select Insert Interaction from the dropdown menu.

Insert Interaction Select the interactions from the Interaction dropdown box.

Interactions can be removed or reordered using these icons.


Copying an item to a new module Copying an item will create a new item with all the fields prepopulated from the old item. Any field can be changed, including the discipline and module (depending on the user’s access to disciplines and modules).

The following changes will be applied:

• a new item identifier is regenerated • the new item name will have ‘-Copy’ appended • metadata will be copied • psychometric data will not be copied • the version will be reset to 0.01 • the workflow state will be set to ‘Draft’.

Steps Navigate to the item details page through Item Authoring > Items.


Menu options available in item editor Edit In the item editor, select Edit to use basic editing functions.


Insert Select Insert menu to insert resources into the item stem.

For interactive gap match items, gaps can be inserted by selecting Insert gap.


Search Click on the camera icon to search your local browser.

Resource Click on a resource. The details will populate in the field to the right. For the purpose of this example, the resource titled ‘nemo’ has been selected.

Insert resource Details can be altered here before clicking Insert resource.

OK Select OK to save the resource.


View Select View to choose from the various viewing options.


Format Select Format to use the formatting options.


Table

Select Table to add and edit a table.


Other menu options

Right mouse click to trigger a pop up menu; there are options to insert and work with gaps and tables.


Select the resource and HTML source icons to insert a resource and to view and edit the HTML source.


Stimulus creation A stimulus is text, a resource or a combination of both. A stimulus displays separately from the question in an expandable frame. Even if a stimulus is linked to multiple items there is only one copy in the system and any changes to the stimulus will appear in any linked items.

How to create a new stimulus

Stimuli From the Item authoring menu select Stimuli.

Add stimulus Select Add stimulus from the Manage drop-down menu.


Discipline and module Select Discipline and Module you want the item to belong to.

Stimulus content To create your stimulus content, select the Free HTML icon. A rich text editor will appear. You can use it to add text, images, videos and source code.

Save Select Save stimulus.


Preview You can elect to have a different version of the stimulus displayed when minimised. By default, the preview view is set to ‘Same as expanded stimulus’. You can change that by selecting Preview, deselecting Same as expanded stimulus and entering content using the rich text editor.

Description Enter an optional description of the stimulus.


Add text Text can be typed directly into the text box.

Source code Select this button to enter or copy HTML code into the text box.

Insert You can insert resources, URLs and special character via this drop-down menu.


How to add a stimulus to items Navigate to the item details page of the item you wish to add the stimulus to, and scroll down to the Stimulus section.

Edit Stimulus Select the Edit Stimulus icon.

Select Stimulus Expand the drop-down list to select the stimulus to be added to the item.

Edit Click this icon to edit the selected stimulus.

Save After selecting the stimulus to be added, select Save.


Resource creation A resource can be an image, audio, video, PDF or other file. Resources can be added to the item directly, in the item stem or the interaction area, depending on the item type.

Resources From the Item authoring menu select Resources. Add Resource

Activate the Manage dropdown box to add a single resource or bulk resources.


How to add a single resource First, click on Add resource from the Manage dropdown box. Then follow the steps below.

Description Enter an optional description of the resource. This will aid searching.

Resource type Select the resource type. For the purpose of this example ‘Image’ has been selected and a picture has been loaded.

Select file Select this button to browse for the file you wish to add as a resource.

Save Select Save Resource.

Discipline and module Select Discipline and Module you want the item to belong to. Name and identifier

Enter a name for the item. The identifier will be automatically generated.


How to add bulk resources When bulk-uploading resources, only one type of resource can be imported at a time. The system will import all files with the supported extensions as indicated in the following steps.

Steps

Module and discipline Select the Module and the Discipline. The identifiers will be automatically generated.

Tags Tags can be applied to all files bulk uploaded. Enter text in the field and select + to add tag.

Select files Click on Select files.

Clear imported on select Check this box to remove items from the list once they are imported.

Resources From the Item authoring menu, select Resources.

Add bulk resources Activate the Manage dropdown box and select Add bulk resources.


Unsupported file types If an unsupported file type is selected, the following error message is displayed listing the unsupported extensions.

Overwrite This message will appear if you try to upload a file with a filename that matches an existing resource in the system. You can either choose not to import the file, or you can overwrite the file already in the system by checking the box beside the resource before importing.

Select files Select this button to choose files to upload.

Check the box next to a file if you want to perform an action on it (such as importing it or clearing it from the list). New resource files will automatically be checked, ready to import. They can be manually unchecked by clicking on the checkbox.

Import Ensure that each file that you wish to import has a ticked checkbox. Then select Import. Once the import is complete, the tick in the checkbox will disappear and the status will change to Imported.


Resources types Resource Type File Extension Container Codec Dimensions

Image

jpg

Optimal

480 x 320

Recommended

png Supported

gif Supported

Video

mp4 MP4 H.264

Optimal

480 x 320h

Recommended

m4v MP4 H.264 Supported

f4v MP4 H.264 Supported

mov MP4 H.264 Supported

Audio

mp3 mpeg MP3 Recommended

aac MP4 AAC Supported

m4a MP4 AAC Supported

PDF pdf Recommended

Script txt Supports any standard text file format

Style txt Supports any standard text file


Resource Type File Extension Container Codec Dimensions

css format

HTML

txt

Supports any standard text file format

html

htm

HTML Snippet

Each resource is of a different type. If you select an unsupported file type or a file type that does not match the resources type (e.g. selecting a word document file for an image resource), then an error is displayed.


Support for MathML MathML support is available for all item types (including composite items).

MathML can be entered, by selecting the source code icon in the rich text editor, into the:

• item stem • item prompt • expanded stimulus field • preview stimulus field

MathML can also be entered directly into the distractor fields of various item types.

Steps

Source Code Select Source Code in the Prompt area to enter the MathML codes.

Enter the MathML code and select Ok.


Workflow The Item Authoring and Review System supports a workflow that allows the appointment of multiple reviewers during item development in order to manage the quality and approval process.

Workflows can be attached to an item or stimulus. Resources will be reviewed as part of the items they are associated with. Resources linked to multiple items will be reviewed in the context of each item.

Workflows can be changed using the workflow widget in the widget bar at the top of each item. Transitions will only appear if the user has the correct permissions assigned through their role.

Currently, there are nine states in the workflow. Items and stimuli can only be deleted from the ‘draft’ and ‘internal review’ states. Note that if an item or stimulus is deleted, it is removed from the system and cannot be recovered.


The following sections describe the details of each of the item authoring and review workflow states.

Workflow The workflow is located in the details form of each item.

Workflow states As soon as items and stimuli are created they are placed in the draft state.

Draft (available to item authors) During this state the items and stimuli are being developed by item authors.

An item or stimulus in this state:

• is not ready to be reviewed either internally by the service provider, or externally by ACARA • can be deleted • can have comments added and modified (with the appropriate permissions).

Transitions: an item or stimulus in this state can transition to internal review or management review depending on the policies of the service provider.

Internal review (available to item authors) The internal review state is an optional state that may be used by some service providers. This state indicates that the item author has finished authoring the item or stimulus and it is ready for the service provider to conduct an internal review.

An item and stimulus in this state:

• is still not ready to be reviewed by test managers or item reviewers • can be deleted • can have comments added and modified (with the appropriate permissions).

Transitions: an item or stimulus in this state can transition to management review or back to draft state.

Management review (read only to item authors) An item or stimulus in this state is ready to be reviewed by ACARA. Test managers conduct an initial review and add comments. Test managers decide if items and stimuli in this state are ready to be reviewed by experts or if they need further modification by service providers.

An item or stimulus in this state can have comments added and modified (with the appropriate permissions).

Transitions: an item or stimulus in this state can transition to:

• expert review – this is the normal process • adjudication – in some cases, an expert review can be skipped; this could be because the expert review has already been undertaken, because the item or

stimulus has since been modified, or for other reasons


• requires modification – if modification is required before expert review, the item or stimulus will be sent back to the service provider • accept – skip expert review and adjudication and go straight to accepted; normally used for items and stimuli that have been through revision • rejected – skip expert review and adjudication and go straight to rejected; normally used for items and stimuli that have been through revision.

Expert review (read only to item authors) During this state an item or stimulus will be reviewed by expert reviewers. The expert reviewers have a defined time to look at the item or stimulus and add their comments. This enforcement of time is done outside the system, and item or stimulus will not automatically change its state. Multiple expert reviewers will look at the item or stimulus during this state and all their comments will be visible to test managers. Item reviewers can only see their own comments.

Transitions: An item or stimulus in this state can only transition to the ‘Adjudication’ state. This is done by the test manager.

Adjudication (read only to item authors) During this state the comments by the expert reviewers entered during the expert review state will be reviewed by test managers. They will decide which comments should be actioned and which comments should be ignored. They may seek clarification from the expert reviewers.

After reviewing all the feedback the fate of the item will be decided, it can either be accepted, rejected or sent back for further modification by the service provider.

Transitions: An item or stimulus in this state can transition to:

• accept – if the item or stimulus meets the quality standards and requires no further modification • rejected – if there is no way the item or stimulus can be modified to meet the quality standards • requires modification – the item or stimulus requires further modification by the service provider • management review – the item or stimulus can be sent back to management review so that test managers can provide feedback to reviewers and then

send it back to review.

Requires modification (available to item authors) An item or stimulus in this state have been sent back from ACARA’s review process to be modified by the service provider. They will probably have comments from item reviewers and test managers that need to be actioned. This state is analogous to the draft state in that an item or stimulus can be sent to modification review or management review. The difference is that an item and stimulus in this state cannot be deleted.


Transitions: An item or stimulus will only transition to modification review or management review depending on the policies of the service provider.


Modification review (available to item authors) This state is analogous to the internal review state that comes after draft. An item or stimulus in this state indicates that the item author has finished modifying the item or stimulus and it is ready for the service provider to conduct an internal review. The item or stimulus is still not ready to be re-reviewed by ACARA test managers or item reviewers.


Transitions: An item or stimulus in this state can transition to ‘management review’ or back to the ‘requires modification’ state.

Rejected (read only to item authors) An item or stimulus in this state has been deemed as not meeting the quality standards and no amount of modification will improve the quality so that it can meet the quality standard.

Transitions: An item or stimulus in this state can be transitioned to management review if a mistake is made, or if it is later decided to use the item or stimulus (with or without modifications).

Accepted (read only to item authors) The item or stimulus has met the quality standards and has been approved for inclusion in the item bank and further use.

Transitions: An item or stimulus in this state can be transitioned management review if a mistake is made, or if it is later decided the item or stimulus needs modifications. Once the item or stimulus is in the management review stage, it can then be sent back for revision, or reviewed again.


Comments Comments will appear as a collapsible section on each test item, stimulus and resource. Item authors have access to unhidden comments on their module only. Item authors can add comments in their module, and only in draft, internal review, requires modification and modification review workflow states. The following functionality will be available:

Steps Comments

Select the Comments title and it will expand.

Post Click on Post to submit your comment.

Attach files You can attach files to comments by clicking on Select files.

Add comment Add comments with a subject, message and file. Adding a file is optional.

Version Click on Version to choose which version you wish to comment on.


Versioning Software versioning is the assignment of unique version names or numbers to unique states of software. Versions are made up of major and minor components. The major version is to the left of the decimal place, the minor to the right. For example 1.53 has a major version of 1 and a minor version of 53.

• An item, stimulus or resource starts at version 0.01 when it is first saved or imported. • Every time an item, stimulus or resource is saved the minor version number increases by 0.01. • Modifications to metadata increase the minor version number by 0.01 every time a metadata set is saved. • If the minor version reaches .99, the increment changes to 0.001. If it reaches .999, the increment changes to 0.0001. • When an item or stimulus transitions to the management review workflow state, the major version is increased by 1 and the minor reset to zero (e.g.

0.55 would become 1.00). • If a resource or stimulus that belongs to an item is updated, the version number of the item will not be updated.

Trigger Version change

New item

Imported item

Begins at 0.01

Item saved The version increases by an increment as described in the list above.

Item transitions

• from: draft, internal review, requires modification or modification review

• to: management review.

The version increases by an increment of 1.0


Diagram showing versioning within the workflow


Advanced filter for Bulk Workflow Change tool (for Items) The workflow state of multiple items can be changed simultaneously. From the Items list page, select the Bulk Workflow Change option from the Manage dropdown menu.

In the example below, all items from the ‘Authoring for Accessibility’ Module that have not been reviewed for PNP1 are selected for workflow change.


Appendix: item types and metadata for NAPLAN Item types

Term Definition

Inline choice An inline choice item asks students to select from a dropdown box or fill in the blanks from a prepared passage of text.

Interactive gap match An interactive gap match item asks students to drag text or images from a source list into gaps within the text.

Interactive graphic gap match An interactive graphic gap match item allows hotspots to be defined on an image. It asks students to drag a set of source images or text into the hotspots.

Hotspot A hotspot item asks students to select a point on an image. All selectable points are visible to the student.

Text entry A text entry item asks students to provide a correct keyword. It is possible for the item author to enter many correct keywords.

Interactive match An interactive match item asks students to match images or text with other images or text. The item allows for multiple matches, with each match having different possible answers.

Multiple choice and multiple choices

A multiple choice item asks students to choose one correct answer.

A multiple choices items asks students to choose one or more correct answers.

Interactive order An interactive order item asks students to put in order a set of text or images. The order can be horizontal or vertical.

Interactive graphic order An interactive graphic order item asks students to drag numbers onto defined and visible hotspots on an image.

Extended text An extended text item asks students to write text. Answers can vary from a single sentence to an essay.

Hot text A hot text item asks students to highlight parts of a text.


Term Definition

Slider A slider item asks students to choose a value by dragging a slider.

Interactive position object An interactive position object item asks students to drag one image into position on another (larger) image.

Multiple short answers A multiple short answers item asks students to provide multiple short answer responses given a common text.

Select point A select point item asks students to click on an image to select points. Unlike a hotspot item, a select point item does not display specific points for the student to choose from.

Drawing A drawing item asks the students to draw using basic drawing tools. The author can specify an image to use as the background for the student to draw on.

Interactive associate An interactive associate item asks students to create pairs of items using drag and drop. The order of the pairs has no bearing on the response. The pairings can be images or text (or a combination).

Interactive graphic associate An interactive graphic associate item uses a background image with hotspots defined on it. Students are asked to drag lines between the hotspots to associate the hotspots with each other.

Composite item A composite item is an item type containing various other item types within it as its interactions (sub-items). A candidate needs to successfully attempt all of these interactions to complete a composite item.

Interactions Sub-items within a composite item type.


Metadata The following tables describe the metadata fields for items. The attributes highlighted in blue do not need custom metadata as the information is contained in standard fields. All fields, unless otherwise specified, are optional.

Metadata fields are found on the item details page under the section heading for the set name.

Discipline Custom attribute name Custom attribute description Custom attribute type

Custom attribute options

Common Identifier An automatically generated unique identifier that is applied to every item, resource and stimulus

Use inbuilt identifier field – mandatory

Item name Short descriptive text Use inbuilt name field – mandatory

Contractor ID An automatically generated unique identifier that is applied to every test item service provider with a role of author

String Max length: 9

Contractor company name The name of the company that the service provider works for


Item type A description of the type of the stimulus, prompt and/or test item

Use inbuilt item type field – mandatory

Marking type The type of marking that will be applied to the item

List

Marking guide Detailed instruction to human markers Use inbuilt marking guide field

Marking guide attachments Authors to upload supporting documents for marking guide

Use inbuilt attachments on item




IPENDO ID Unique identifier generated by and used within the IP tool (IPENDO) that ACARA hosts

String

Test cycle – development Year of NAPLAN testing that the item is being developed for, e.g. 2015

Integer Min value: 2000 Max value: 2099

Batch An identifier manually assigned for a batch of test items


Program code Program code item will be used for List




Reading – stimulus text

Genre A class of endeavour that takes a particular form

List

Text type A description of the text is viewed as being. Examples include: poster, narrative, poem, webpage, report, email, procedure, letter, recipe, etc.


Word count The number of words a stimulus contains Integer Min value: 1 Max value: 99999

Y3 targeted testlet Given the current tailored NAPLAN test design, this would be [A/B/C/D/E/F] – for example, 3B or 3F

String Regex: 3[A-F]

Y3 text difficulty The level of difficulty that the stimulus text represents to a Year 3 student

















Required AC information TBD String Max length: 125

Links – appears in multiple year groups

Descriptive Y or N and is populated as part of an administration, therefore a stimulus text may have multiple instances associated to it for each administration

Boolean

Stimulus descriptive Descriptive and keywords String Max length: 125



Reading – Test Item Strand A unit that outlines the knowledge that a student needs to acquire. A strand provides the context within which students will develop a particular skill and concept

List

Content description The Australian Curriculum EL code as per the Australian Curriculum version 5.1. Example: ACELT1640

String Regex: [A-Z]{5}[0-9]{3,4}




Cognitive process The comprehension processes used by readers to construct meaning from the text

List

Key The correct answer for multiple choice test items

Use inbuilt correct answer field

Key string The correct response for constructed response items

Use inbuilt correct answer field

Constructed response answer The answer to the constructed test item Use inbuilt correct answer field

Item descriptor Details of what the item is assessing (free text)


Y3 Item difficulty The level of difficulty that the item represents to a Year 3 student








Dependent item details Descriptive details of any dependent test items





Stimulus ID The unique identifier associated to a specific reading stimulus text.

String Max Length: 9



Language Conventions

Strand A unit that outlines the knowledge that a student needs to acquire. A strand provides the context within which students will develop a particular skill and concept.

List

Content description The Australian Curriculum code as per the Australian Curriculum version 5.1. Example: ACELA1563

String Regex: [A-Z]{5}[0-9]{3,4}

Sub-strand The comprehension processes used by readers to construct meaning from the text.

List

Key The correct answer for multiple choice test items.

Use normal correct answer field

Key string The correct response for constructed response items


Constructed response answer The answer to the constructed test item. Use normal correct answer field

Item descriptor Details of what the item is assessing (free text)





Y3 item difficulty The level of difficulty that the item represents to a Year 3 student








Dependent item details Descriptive details of any dependent test items


MC option A The text and option reasoning for multiple choice option A


MC option B The text and option reasoning for multiple choice option B


MC option C The text and option reasoning for multiple choice option C


MC option D The text and option reasoning for multiple choice option D


MC option E The text and option reasoning for multiple choice option E





Numeracy Proficiency strand The actions in which students can engage when learning and using the content. While not all proficiency strands apply to every content description, they indicate the breadth of mathematical actions that teachers can emphasise.

List

Content description The Australian Curriculum codes for mathematics as per the Australian Curriculum version 5.1. Example: ACMNA014, ACMMG006, ACMSP011

String Regex: [A-Z]{5}[0-9]{4}

Key The correct answer for multiple choice test items


Key string The correct response for non-multiple choice items


MC option A The text and option reasoning for multiple choice option A. Examples: • 25 (key) • 6.4 (place value error) • Image (rotated in wrong direction)


MC option B The text and option reasoning for multiple choice option B



MC option C The text and option reasoning for multiple choice option C


MC option D The text and option reasoning for multiple choice option D


MC option E The text and option reasoning for multiple choice option E


Item descriptor Details of what the item is assessing String Max length: 250

Calculator The calculator demand of the test item List

Y3 item difficulty The level of difficulty that the item represents to a year 3 student








Input restriction The allowable input type for responses List


Discipline Custom Attribute Name Custom Attribute Description Custom Attribute Type

Custom Attribute Options

Writing Genre A class of endeavour that takes a particular form

List

Topic keyword Cue to topic String Max length: 125

Year level The year level/s for which the item has been developed, i.e., 3, 5, 7 and 9

String Regex: [3,5,7,9]


Terms The following table defines special terms used in the item authoring system.

Term Definition

ACARA Australian Curriculum, Assessment and Reporting Authority

Administration An instance of a delivery of a test. e.g., a test event.

Auth A plugin to provide log-on, password and session management to ensure each user is authenticated before interacting with the system

Custom attribute A custom attribute is a metadata field for items, resource and/or stimuli.

Custom attribute set A set of custom attributes that are grouped together

Disciplines An area of learning (e.g. numeracy, reading)

ESA Education Services Australia

Item An item is a test question or statement on which a student can submit a response. The item can contain one or more interactions with response data collected as a result of the interaction. An item can contain its own unique prompt. It can contain stimulus, share stimulus with other items and also contain resources that are unique or shared with other items.

Item bank A shared repository of reusable items, resources and tests

Jurisdiction The state/territory or the school sector (government, independent or Catholic)

Metadata Data that describes an item

Module The containers for content


Term Definition

Organisation A top-level ‘silo’ for organising users (e.g. a school system, zone or jurisdiction). In this project an organisation will be developed for ESA and ACARA

Organisation unit A division of an organisation that is usually a school or business unit

Psychometric data Metadata that measures the performance of an item

Resource Image, audio, animation, video or other file such as plain text added to an item to highlight a concept or question

Role The grouping of a set of access rights to entities that can be assigned to a user

Stem (Item stem) The stem is the beginning part of the item that presents the item as a problem to be solved, a question asked of the respondent, or an incomplete statement to be completed, as well as any other relevant information.

Stimulus Supplementary text, graphic, video and audio files or a combination thereof associated with an item to assist a student when responding to an item or item set

Displayed along with the item stem on the screen but can be expanded or compressed independent to the item stem as required

User A person who interacts with the system

Workflow A prescribed set of states applied to tests, items and resources to ensure quality when these components are accessed in the right sequence


Roles The following table defines the item authoring system roles.

Term Definition

Administrator Has the highest level of user administration. This role allows the user access to the various areas of the system and enables them to perform system permissions management. Can run and create reports.

Test manager Manages a discipline, including reviewing items. Can run and create reports.

Item author Authors items, resources, and stimuli in the system and add associated metadata for these. Can run and create reports.

Item reviewer Reviews items and adds comments. Can run and create reports.

Psychometrician Views items and comments but does not make any changes. Can run and create reports. Can add or modify metadata against all items. Can add psychometric metadata.

Basic user Base user permissions which allow user to login to the system, but give no access to perform any other actions. This role is used for users who will then have further access granted using Extra Roles.

Accessibility Reviewer An expert in specific accessibility areas, such as braille or motor assistive technology. This role reviews NAPLAN-trialled test items for their accessibility suitability for a defined set of personal needs and preference codes.


Item authoring: Importing and exporting QTI items and importing associated metadata

This guide describes the import and export functions.

◘ Import/export QTI items with stimuli

◘ Import item metadata via spreadsheet

◘ Import/export psychometric data

Table of contents

Import QTI Items ...................................................................................................................................................................................................................................................... 3

Steps ..................................................................................................................................................................................................................................................................... 3 Warnings and errors ............................................................................................................................................................................................................................................. 6

Export QTI Items ...................................................................................................................................................................................................................................................... 7

Export individual items .......................................................................................................................................................................................................................................... 7

Export bulk QTI items ........................................................................................................................................................................................................................................... 8

Import item metadata via spreadsheet ................................................................................................................................................................................................................... 11

Steps ................................................................................................................................................................................................................................................................... 12 How to populate the spreadsheet ....................................................................................................................................................................................................................... 13

Using zip files ...................................................................................................................................................................................................................................................... 14

Import psychometric data via spreadsheet ............................................................................................................................................................................................................ 20

Data sheet .......................................................................................................................................................................................................................................................... 22

Supported attributes sheet ................................................................................................................................................................................................................................. 22

Import psychometric data for composite items....................................................................................................................................................................................................... 29 Data sheet .......................................................................................................................................................................................................................................................... 29

Terms ..................................................................................................................................................................................................................................................................... 32

Roles ...................................................................................................................................................................................................................................................................... 33


Import QTI Items Individual or multiple items that conform to the Question and Test Interoperability (QTI) specifications may be imported from other authoring tools, item banks and assessment systems. This applies to standard and composite items, and their associated stimuli and resources.

Steps

Items From the main menu or the Dashboard, select Items.

Import QTI Items From the Manage dropdown menu, select Import QTI Items.


Discipline and Module Select the Discipline and Module for the items to be imported into.

Select QTI Packages Select this button to browse for the location of the files to be imported.

Di i li d


Note: When multiple QTI packages have been selected, they can be imported individually using the adjacent Import buttons, or as a group by selecting Import All.

Import / Import All Once the QTI packages are selected, click on Import or Import All buttons to import items into the system.


Warnings and errors The system will display the warnings and errors of each package after validation, as shown in the screen shot below. Warnings should be noted, but the item can still be imported if desired.

When errors are indicated, the item will not be imported. The errors must be addressed before the import can be completed.

In the example above, the warnings (yellow icons) are indicating that some of item values will not be imported as the IARS does not support them and they can be ignored if desired. The error message (red icon) however, will stop the import of the item until the issue has been fixed.


Export QTI Items

The system provides the ability to export individual or multiple items (including composite items), and associated stimuli and resources, as QTI compliant packages.

Export individual items

QTI Export From the item list, select the QTI Export icon to export the item from the system.

Download Select Download to save the item as a zip file.


Export bulk QTI items All items in a filtered or unfiltered list can be exported as a single package.

All items in a module can also be exported from the Modules page.

From the Item Authoring menu, select Modules.

Select the module to be exported.

Export QTI Items From the items list, select Export QTI Items to export items in the list.


Export QTI Items Select Export QTI Items. This will select all the items in that module to be bulk exported.


Select the item workflow states you wish to export by clicking in the check boxes and then select Export QTI Items.


Import item metadata via spreadsheet Metadata for an item can be imported separately from the item itself.

Item authors can import item metadata via a spreadsheet.

Items From the main menu or the Dashboard, select Items.

Import custom attributes From the Manage dropdown menu, select Import custom attributes.


Steps The system generates a dynamic metadata template specific to the selected discipline and module.

Email address Enter an email address. A summary of the importation will be sent to the address.

Spreadsheet template Once the Discipline and Module have been selected, download the spreadsheet template.

Import Select Import and email result. The metadata will be imported and a summary emailed to the specified email address.

Select file Select the populated spreadsheet file or a zip file containing the spreadsheet and images.

Cells containing string clear field When this checkbox is selected, the string in the text box below will clear the field if it is found in the spreadsheet cell.

Treat empty cells as no change When this checkbox is selected, empty cells in the spreadsheet will not overwrite the field in the system. When the checkbox is not selected, empty cells in the spreadsheet will clear any existing information in that field in the system.

Discipline and Module Select the Discipline and Module for the custom attributes to be added to.


How to populate the spreadsheet The metadata template that you downloaded in the previous steps contains two worksheets: Data and Supported attributes.

The following guidelines can be used when populating the worksheets.

Data sheet

The Data worksheet will have a column for each custom attribute available for that discipline.

Metadata can be added to normal and composite items as well as to individual interactions within composite items by specifying the item and/or interaction identifiers in the Identifier column.

Note: • The first or second line of the spreadsheet is the header row (a blank line at the top is allowed) • To fill in the values in the spreadsheet columns under custom attributes which are of type list, the user has to enter the identifier of the list item eg: ‘AS’

for auto-scoring • Blank cell values will clear the custom attribute field (unless the option ‘Treat empty cells as no change’ is selected) • Column order does not matter • If a field isn’t in the spreadsheet, the custom attribute will not be updated • If an individual row has a validation error, other rows will still be imported • If a wrong custom attributes identifier has been entered, other columns will still be imported

Item identifier Fill in the item or interaction identifier.

Custom attribute identifiers The ‘Custom attribute identifier’ columns are where you will find the attributes for the item.

Validation Validation will be performed on each field to make sure it matches the custom attribute type and also the restrictions on the field.

Items Each row below the headers represents one item.


Supported attributes sheet

The second worksheet of the spreadsheet contains more information on the custom attribute fields:

• Identifier • Caption • Type • Is Multiple • Supported list values

Using zip files When uploading a zip file, observe the following guidelines:

• The zip file should contain only one spreadsheet in the root directory of the file. • The spreadsheet must be in the format described above. • The zip file can contain image files in the same folder as the spreadsheet or in subfolders. • The spreadsheet should contain the relative paths for the image files in the column for the image metadata field. • The maximum file size is 200MB.


Practical example: Custom attribute sets – common (ESA UAT)

Item IDs: 001172 and 001173

1. Fill in the template with the required data.

2. Compress the spreadsheet and images into a zip file.


3. Select the zip file.


4. Select Import and email result and a message will show the process on the screen.

5. An import result summary will be emailed to the nominated email address.


Below is an example of an import result summary with errors.


6. Confirm data is imported to actual items.


Import psychometric data via spreadsheet Psychometric data will be imported in spreadsheet format. This option will be available from the Administration details screen. A template spreadsheet will be available from this page to show the columns that will need to be filled out. Psychometricians and administrators have full access to psychometric administration and data.

Import PD From the main menu navigate to Item authoring > Administrations. Import PD

Select an administration.

Import PD Select Import psychometric data.


As with the psychometric data import template, the Supported attributes sheet will explain the requirements of the fields in the Data sheet.

• A spreadsheet can only contain data for one administration, but multiple spreadsheets can be imported for a single administration (one at a time) • Data will be validated when it is imported against the requirements set on the custom attribute (min number, max number, max length, allowable values,

etc.)

Select Select the populated spreadsheet or a zip file.

Email Address Enter an email address, the import summary will be sent to this address.

Spreadsheet Template Download the template and populate the columns with the new items.

Import Click the import button and the import result summary will be emailed.


Data sheet

Custom Custom Custom Custom Custom

Question ID

Distractor 1 - PBCorrelation


Responsecode 7 - PBCorrelation profile-image multiselect

I0001255 distractor A data distractor B data 566 mouse.png exp,wb

Supported attributes sheet

Question ID System auto-generated identifier for items. This field is mandatory. For Composite Items, the identifier needs to be matched with the interaction identifier for these to be populated at interaction level.

Distractor x – Name Fill in multiple choice question distractors.

Response code x – Name Fill in the additional response codes.

Other PD fields Fill in the other psychometric data fields on the item.

Import image field Zip an image file with the same name populated in the image field and upload it with the spreadsheet in the same folder.


Practical example: ESA UAT

Steps

1. Prepare the psychometric data spreadsheet

2. Compress the spreadsheet and image folder into a zip file


3. Select the psychometric data zip file

4. Click “Import and email result” and a message will show the process on the screen 5. An import result summary will be emailed to the nominated email address 6. Navigate to the individual item and click psychometric icon


7. Select different administrations and click “show”


8. Review the psychometric data


Export psychometric data via reports Psychometric data will be available for export using Customisable reports. Each psychometric field will be available for selection in reports on Items. Please refer to the Item Authoring Search and Reporting Guide for detailed instructions on how to create a report.

Entity Select the “Administration Item”.

Filters Add new filters by selecting the relevant fields and values.

Report Items Select the custom attributes which need to be exported.


An example of the report:

Report format Select the report format for exporting the psychometric data.


Import psychometric data for composite items Each Interaction within a Composite Item can have its own Psychometric Data (PD). As for normal items, it’s possible to use the Psychometric Data Import tool. You simply need to reference the identifier for the interaction in the spreadsheet.

As with the psychometric data import template, the Supported attributes sheet will explain the requirements of the fields in the Data sheet.

• A spreadsheet can only contain data for one administration, but multiple spreadsheets can be imported for a single administration (one at a time) • Data will be validated when it is imported against the requirements set on the custom attribute (min number, max number, max length, allowable values,

etc.)

Data sheet

Custom Custom Custom Custom Custom

Question ID



Responsecode 7 - PBCorrelation profile-image multiselect

I0001255 distractor A data distractor B data 566 mouse.png exp,wb

Question ID System auto-generate identifier for items. This field is mandatory. For Composite Items, the identifier needs to be matched with the interaction identifier for these to be populated at interaction level.

Distractor x – Name Fill in multiple choice question distractors.

Response code x – Name Fill in the additional response codes.

Other PD fields Fill in the other psychometric data fields on the item.

Import image field Zip an image file with the same name populated in the image field and upload it with the spreadsheet in the same folder.


The Imported PD are shown at Composite Item Interaction level.

Click on Edit or View icon to access the PD details.


Interaction

Select the Interaction for which PD need to be added.

Upon importing of PD, this will be reflected in the Composite item menu bar at the top.

Save

Click Save after adding the PD details.


Terms The following table defines special terms used in this user guide.

Term Definition

Administration System administration features and functionality managed by this role; a single running of a test event

Custom attribute A custom field that can be linked to an object in CLS

Custom attribute set A set of custom attributes that are grouped together

Disciplines The CLS representation of domains

Item An item is a test question or statement on which a student can submit a response. The item can contain one or more interactions with response data collected as a result of the interaction. An item can contain its own unique prompt. It can contain a stimulus and share a stimulus with other items. It can also contain resources that are unique or shared with other items.

Metadata Data that describes an item

Module The containers for content

Psychometric data Metadata that measures the performance of an item

Role The grouping of a set of access rights to entities that can be assigned to a user

Composite item A composite item is an item type containing various other item types within it as its interactions (sub-items). A candidate needs to successfully attempt all of these interactions to complete a composite item.


Term Definition

Interactions Sub-items within a composite item type

Roles The following table defines the item authoring system roles.

Term Definition

Administrator Has the highest level of client administration, including access to the various areas of the system and the ability to perform system permissions management.

Test manager Manages a discipline, including reviewing items. Can run and create reports.

Item author Authors items in the system.

Item reviewer Review items and adds comments. Can run and create reports.

Psychometrician Views items and comments but does not make any changes. Can run and create reports. Can add or modify metadata against all items.

Basic user Has basic user permissions. Can log in to the system but has no access to perform any other actions. Users with this role are granted further access using ‘extra roles’.



Search and Report Guide

This guide demonstrates the search and report functions:

◘ Search

◘ Customisable reports

Contents Search ......................................................................................................................................................................................................................................................................... 3

Global Search ......................................................................................................................................................................................................................................................... 3

Items ...................................................................................................................................................................................................................................................................... 4

Resources and Stimuli ............................................................................................................................................................................................................................................ 6

Disciplines and Modules ........................................................................................................................................................................................................................................ 8

Tags, Administration (psychometric data) and Customisable reports .................................................................................................................................................................. 9

Users .................................................................................................................................................................................................................................................................... 10

Reports ................................................................................................................................................................................................................................................................. 11

Customisable reports ............................................................................................................................................................................................................................................... 12

Creating a new report .......................................................................................................................................................................................................................................... 14

Filtering ................................................................................................................................................................................................................................................................ 17

Filters ................................................................................................................................................................................................................................................................ 17

Filter groups ..................................................................................................................................................................................................................................................... 17

Operators ............................................................................................................................................................................................................................................................. 19

Value .................................................................................................................................................................................................................................................................... 19

Parameters ........................................................................................................................................................................................................................................................... 19

Security filtering ................................................................................................................................................................................................................................................... 19

Selecting columns ................................................................................................................................................................................................................................................ 20

Running a report .................................................................................................................................................................................................................................................. 21

Exporting reports ................................................................................................................................................................................................................................................. 22

Saving reports ...................................................................................................................................................................................................................................................... 23

Customisable reports ........................................................................................................................................................................................................................................... 24

Scheduling a report .............................................................................................................................................................................................................................................. 25

Terms ....................................................................................................................................................................................................................................................................... 26

System roles ............................................................................................................................................................................................................................................................. 28

© 2015 Education Services Australia Ltd, unless otherwise indicated Last updated: May 2015 2

Search

The search capability in the Item Authoring and Review system makes it possible to search for and find specific pieces of information quickly. You can search from many different sections within the system.

Global Search

The global search tool will retrieve results from the following areas: Items, Stimuli, Resources and Tags.

Global search is located in the header at the top of every page.


Items

Filtered items are also available via links elsewhere in the system. For example, on the Module details page there is a link to all the items in the Module. Other pages with filtered item links:

• Modules list and details pages • Resources list and details pages • Stimuli list and details pages • Tags list and details pages • Administrations list page (psychometric data)

Items can be searched by: • free text on item stem,

item name, item identifier

Items can be filtered by the following: • item type • module • tags • whether the user has

created, edited or reviewed the item

• the workflow state of the item

Advanced Select Advanced to open advanced filter options.


Advanced filters for item searching

The item list page has advanced search filtering capabilities, and remembers the last saved query.

Items can be filtered by standard item information and custom attributes

Filters

A filter consists of a property (a field), an operator and a value. The property is either a property belonging to the item – e.g. the identifier – or a custom attribute.

Filter groups

Filter groups allow multiple filters to be grouped using the AND and OR logical group operators. Using these nested groups, it is possible to build any set of filters. The AND operator will apply all filters in the group when searching. The OR operator will return results where the conditions of at least one filter in the group have been met.

For example, you may want to search for items with a version number less than 1, for which the Y3 or Y5 Item difficulty is above 50. This would be defined as:

Version < 1, AND

o Y3 item difficulty > 50, OR o Y5 item difficulty > 50

In the example above, the indented bullet points are a single group.


This is how it would look on the screen.

Note: The search criteria are saved until the user selects the Reset search option. The search can also be saved as a bookmark.

Groups Grouped filters are nested (indented).


Resources and Stimuli

Filtered resource lists are also available via:

• Modules list and details pages • Tags list and details pages • Item details pages

Resources can be searched by: • free text on name,

identifier and description

Stimuli can be searched by: • free text on name,

identifier and description

Stimuli can be filtered by the following: • module • tags • whether the user has

created, edited or reviewed the stimulus

• the workflow state of the stimulus

Resources can be filtered by the following: • resource type • module • tags • whether the user has

created, edited or reviewed the resource


Disciplines and Modules

Filtered module lists are also available via:

• Disciplines list and details pages • Tags list and details pages

Disciplines and modules can be searched by: • free text on name and

identifier


Tags, Administration (psychometric data) and Customisable reports

Filtered tag lists are also available via:

• Stimuli list and details pages • Resources list and details pages • Item details pages

Tags, Administrations and Reports can all be searched by name. Tags can also be searched by description.


Users

Users can be searched by: • free text on name,

email and username. Users can be browsed by ‘date created’. These fields can also be used in conjunction with other search criteria to limit search results. Users can be browsed by

group. Click in the search field to see a list of available groups.


Reports

Reports can be searched by: • free text on name.


Customisable reports

In the Item authoring and review system, customisable reports can be prepared on the following subjects1:

• items

• resources

• stimuli

• administrations

• users

• user activities

• items with stimuli

• interactions within items

• comments on items and stimuli

Users are able to:

• save reports for later use – create a portfolio of reports

• share reports – save a report for a single user, all users or an organisation

• schedule reports – schedule reports to run at nominated times, dates and intervals and receive the results via email

• export reports – export reports as Excel, CSV or PDF files

1 A user’s ability to generate reports will be determined by that user’s role within the Item Authoring system and the organisation they belong to.


The process flow of the reporting system is as follows:

Define Report

Run Report

Export Report

Save Report

List Reports Create New

Run Redefine

Export

Save

Save

Select


Creating a new report

To begin creating a report:

1. Select Item Authoring from the main menu dropdown list

2. Select Reports from the Item Authoring menu

3. Select Create a report from the Customisable reports page

On the Customisable reports page, select Create a report.

From the main menu dropdown list, select Item authoring.


The steps involved in building a report are:

• select the report ‘entity’ (the subject of the report) • apply any filters to narrow search results • select the data columns to be included in the report

The entity is the basis for the report. For example, a report with ‘Item’ selected as the entity will contain a list of items. Reports can be prepared on the following entities:

• items • stimuli • items with stimuli • item interactions • resources • tests • item comments • stimulus comments • items within administrations (an administration is an instance of a test) • users • user activities • user extra roles

Different user roles are able to report on different elements within the system as detailed in the table below.

Entity Test Other User Comments User role

Item Test Test item Item stimulus

Admin. item

Item w. interactions

Stimuli Resource User Log-on session

User docu-ment activity

User extra role

Item Stimuli

Item author

Item reviewer

Accessibility Reviewer

Test manager

Psychometrician

Administrator


User entities • Log-on session • User • User extra role • User document

activity

Test entities • Item • Item stimulus • Test • Test item • Administration

Item • Item with

interactions

Comment entities • Item comment • Stimuli comment

Other entities • Resource • Stimulus


Filtering

Using filters enables you to refine your report data.

Filters

A filter consists of a property (a field), an operator and a value. The property is either a property belonging to the entity – e.g. a user’s last name – or a custom attribute.

For example, the following properties can be used in filters when preparing a report on the ‘User’ entity.

• Organisation and Organisation unit – select from the dropdown menu or match on name. • Role name –match on the main role name. • Last logged in and date created – date fields related to the user’s account and activity. • Custom attributes - filter by metadata fields in the custom attribute sets.

Filter groups

Filter groups allow multiple filters to be grouped using the AND and OR logical group operators. Using these nested groups, it is possible to build any set of filters. The AND operator will apply all filters in the group when compiling the report. The OR operator will return results where the conditions of at least one filter in the group have been met.

For example, you may want a report for a specific module, where either the Y3 or Y5 Item difficulty is above 50. This would be defined as:

Module = <Desired Module>, AND

o Y3 item difficulty > 50, OR o Y5 item difficulty > 50

In the example above, the indented bullet points are a single group.


This is how it would look on the screen.

Control If this box is checked, other users will be allowed to modify the filter operator and value.

Groups Grouped filters are nested (indented).


Operators

Operators are the logical operations. Depending on the data type of the filter property or field, different operators are available. For example, ‘contains’ cannot operate on a date field but can be applied to a text field.

The following operators are available:

• =/!= (equals/does not equal) − looks for an exact match, available on all data types • >/< (greater than/less than) − available on text (alphabetical order) and numerical field values • contains/doesn’t contain – available for text and numerical field values • after/before – available only for date field values • starts with/ends with – available for text and numerical field values • is null/is not null – determines whether the field has any data, available on all data types

Value

The 'Value' field changes its input control depending on the data type. For example, for a date field such as ’Last modified on’ it would show a date picker; for a text field such as ‘Email’ it would show a text entry box. When the report is run, the value entered here will be compared with the relevant field value in all the entity records the user has access to.

Parameters

Some data types support parametric comparison values rather than explicit values. For example, it may be useful to run a report of users that logged in during the last month. Rather than selecting a ‘hard’ date which would need to be updated each time the report is run, it is possible to select a parametric value (relative value or range) such as ’Last month’. Parametric comparison values are only available for date type properties with the following parameters supported:

• Today • Yesterday • This Month • This Year • Last Month • Last Year

Security filtering

In addition to the filters specified by the user creating the report, the system applies further filtering based on the user’s role and access permissions. A user whose base role is ‘item author’, for example, will not be able to prepare reports on the ‘user’ entity. A user preparing a report on the ‘item’ entity will only be able to report on items in modules to which the user has access.


Selecting columns

The ‘Select column’ section is where the user specifies the entity fields that they want included in the report.

Select columns Tick the box beside the column name to include that field in the report.

Run Report This button will run the report and display an HTML version of it.

Set column order The display order of the columns can be specified.


Running a report

When the entity filters and columns have been chosen, select Run report to display the report data on the Item Authoring system. The report can then be modified, saved or exported.

Refine the report This returns user to previous screen where they can modify the columns selected for viewing in the report and change filters.

Save the report Once you are satisfied with the report, it can be saved for future use.

Export the report Reports can be exported in the following formats: • PDF – the report is generated as a PDF file where it is displayed in a table. • Microsoft Excel – the report is generated as an Excel file. The first row

contains the column headings. • CSV (comma separated values) – the report is generated as a text file. The

first line contains the column headings. Each column is delimited by a comma and the data surrounded by speech marks (‘).


Exporting reports

An example of a customisable report exported in PDF format.

An example of a customisable report exported in Excel format.

An example of a customisable report exported in CSV format.


Saving reports

Reports can be saved so they can be reused. It is the report definition with the query - i.e. the entity, filters and data columns - that is saved rather than the results. This means every time the report is run, the query will show data that is relevant to the user and their access level. One user, for example, may save a report on items in draft state, which, when run by another user with permissions for different modules, will return a completely different set of items.

Users can only update a report created by another user if that user selected the ‘Allow others to overwrite’ checkbox when saving the report. Otherwise they will need to save the report as a copy with a new name in order to change the report settings.

Name Appears on the report list screen.

Description Appears under the name of the report when it is run.

Scope This defines which users are able to see and run the report. The options are: • All users (default) • Me • Me and others

(‘others’ are individual users specified by the report author)

• My organisation • My organisation unit

Save as copy Only visible when the report has already been saved. This saves a new copy of the report. If not selected, the existing report will be updated.

Allow others to overwrite Allows other users who can see the report to overwrite the report definition. If this isn’t checked, they can only save as a copy.


Customisable reports

This screen lists the saved reports that the user has access to. This is based on the sharing permissions set when each report was saved.

Schedule a report Schedule a report to run at a specified time or interval.

Class type (Entity) The entity is the subject of the report as selected by the report author.

Edit report Open report in edit mode.

Created by This is the report author.

Created on This is the date the report was created.

Name Select a name to run that report.


Scheduling a report

Reports can be scheduled to run at specified times and at regular intervals. The completed report/s will then be sent to your email address or saved to the location you specified.

Enabled By checking this box, the job can be disabled without being deleted.

Custom report This is the name of the report to be scheduled.

Email address The report will be sent to the email address entered here.

File format Export the report in one of the following formats: • CSV • XLSX • PDF

Run from / Run to Reports will be run within this date range. If Run to is left blank, the reports will run indefinitely.

Schedule type Reports can be run at regular intervals or at scheduled times and days of the week.


Terms

The following table defines special terms used in the item authoring system.

Term Definition

Administration An instance of a delivery of a test. e.g., a test event.

Custom attribute A custom attribute is a metadata for items, resource and/or stimuli.

Custom attribute set A set of custom attributes that are grouped together.

Disciplines The IARS (Item Authoring and Review System) representation of domains.

Domain An area of learning. E.g. numeracy, reading, etc.

IARS Item Authoring and Review System – System that facilitates the creation and importing of items, resources and stimuli to be used in tests.

Item An item is a test question or statement on which a student can submit a response. The item can contain one or more interactions with response data collected as a result of the interaction. An item can contain its own unique prompt. It can contain stimulus, share stimulus with other items and also contain resources that are unique or shared with other items.

Metadata Data that describes an item.

Module The containers for items, stimuli and resources.

Organisation A top-level 'silo' for organising users: for example, a school system, zone or jurisdiction. In this project an organisation will be developed for ESA and ACARA.


Term Definition

Organisation unit A division of an organisation that is usually a school or business unit.

Psychometric data Metadata that measures the performance of an item.

Resource Image, audio, animation, video or other file such as plain text added to an item to highlight a concept or question.

Role The grouping of a set of access rights to entities that can be assigned to a user.

Stimulus Supplementary text, graphic, video and audio files or a combination thereof associated with an item to assist a student when responding to an item or item set. Is displayed along with the item stem on the screen but can be expanded or compressed independent to the item stem as required.

User A person who interacts with the system.

Workflow A prescribed set of states applied to tests, items and resources to ensure quality when these components are accessed in the right sequence.


System roles

The following table defines the item authoring system roles.

Term Definition

Administrator Has the highest level of user administration. This role allows the user access to the various areas of the system and enables them to perform system permissions management. Can run and create reports.

Test Manager Manages a discipline, including reviewing items. Can run and create reports.

Item Author Authors items, resources, and stimuli in the system and add associated metadata for these. Can run and create reports.

Item Reviewer Reviews items and add comments. Can run and create reports.

Psychometrician Views items and comments but does not make any changes. Can run and create reports. Can add or modify metadata against all items. Can add psychometric metadata.

Basic user Base user permissions which allow user to login to the system, but give no access to perform any other actions. This role is used for users who will then have further access granted using Extra Roles.



Documents

SPECIFICATIONS IN RELATION TO PANEL OF PREFERRED …€¦ · 2.1 Deliverable 1 – Project Plan 3 2.2. Deliverable 2 – Project Management and Reporting 4 ... original tender will