Copyright © 1995-2007 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark

Copyright © 1995-2007 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Perception Assessment Management System

AFPAJune 7th-8th, 2007

Slide 2 Copyright © 1995-2007 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Contents: Day 1 Structuring Assessments: Blocks Changing Look and Feel: Templates Publishing Assessments


Contents: Day 2 System Administration Results Manager Enterprise Reporter



Structuring Assessments: Blocks


The Anatomy of an Assessment


Assessment Editor Edit default control parameters Add additional question blocks Add jump blocks

Enable adaptive or multi-stage testing

Add reported topic outcomes Add/edit assessment outcomes



Changing Look and Feel: Templates


Assessment Delivery “Items” (questions,

choices, feedback, etc.) are stored within the question database (table)

Perception merges “items” with html using custom “templates” or “style sheets” and “format” files at the time assessment is taken

questions assessments

templateformat files

Repository

Inet

Participant

Participant

Participant

Participant

f

DeliveringAssessments


Assessment Delivery

Order files are applied to final output

Format Files Contains all default settings

Template File Contains just the items you wish

to change from the defaults Style Sheet

Works with the template, adding styles to the code contained in the template

Questions Contains html used in questions

Format Files

Template StyleSheet

AssessmentQuestions


Assessment Delivery The exact HTML produced is defined by:

Perception software settings.format and htmlcore.format local.format and local_login.format containing

any global customization Templates (with or without references to Style

sheets), selected using Assessment Manager, to modify the defaults for a particular assessment


HTML Produced by Perception<html><title> Assessment Name </title><body> assessment content’s “look and feel” is defined by merging items with html from format files, session templates, and style

sheet</body></html>


Final HTML Produced<html><title> Assessment Name </title><body>

< background graphics>< assessment header with CSS designation > < pre-section CSS designation>

<pre-question html tags> the question <post-question html tags>< pre-choice CSS designation> choice <post-choice CSS designation >< pre-choice CSS designation > choice <post-choice CSS designation >< pre-choice CSS designation > choice <post-choice CSS designation >

< post-section CSS designation>< text in buttons>< assessment footer with CSS designation ></body></html>


What are Format Files? Control how the QML code of the questions is

converted to a page of HTML ASCII text files with the extension .format Stored in \perception4\authoring\format folder

on the Perception authoring PCs, and the \perception4\server\format folder on the Perception Server PC

settings.format is the most important; use for reference purposes only

Settings in files local.format or local_login.format overwrite the settings in the main format files


What are Templates? Enable Perception authors to change how the

questions appear to participants Are ASCII text files with the extension .template A template file can contain no settings, or any

number of settings Stored in the \[repository]\templates folder

within each repository A number of template files are supplied with

Perception as examples You can use the Template Editor in Authoring

Manager or a text editor to alter template settings


Template Editor


Template changes All format file settings can continue to be used for templates using CSS with

the exception of the following: All START_xxx and END_xxx definitions in settings.format All definitions in the [FEEDBACK], [STYLE], [CHOICE] and [OPTION] sections All button definitions :

SUBMIT, RESET, CONTINUE, QUIT, HOME_BUTTON, BRANCH_BUTTON, PIP_HOME_BUTTON SEPARATOR OPEN_DOCUMENT_HEADER CLOSE_DOCUMENT_HEADER OPEN_FEEDBACK_HEADER CLOSE_FEEDBACK_HEADER OPEN_END_HEADER CLOSE_END_HEADER BLOCK_SCORE RUNNING_SCORE FINAL_SCORE ANSWERED_DEF NOTANSWERED Template definitions to be used with CSS should not include any font tags – font

face, size and color are to be controlled by the cascading style sheet.


What are Style Sheets? Define rules that control the styling of

HTML elements Allow you to control the rendering — e.g.

fonts, colors, leading, margins, typefaces, and other aspects of style — of a Web document without compromising its structure

Separate the style of a page from the code, or HTML, makes it easier to apply corporate standards to a web page


Using Style Sheets Style sheet support is turned on using the

variable USE_STYLESHEET=1 Style sheet used is defined by the setting

STYLESHEET=MyStyle.css Question by Question (QxQ) delivery does

not currently support CSS QxQ templates should include USE_QXQ=1

or USE_STYLESHEET=0 to disable the use of Style sheets



Publishing Assessments



Authoring Manager Security and Workflow Management


Managing Administrators

Manage Administrator in Enterprise Manager

Manage Administrator in Windows Authoring Manager


Personal Settings


Roles and Groups

Roles are for

Workflow


Authoring Rights


Controlling Access to Topic and Assessment Folders

Topic and assessment folder

rights menu


Topic Rights per Administrator


Assigning Administrators from the Navigation Pane


Topic Rights Set in Browser-based Authoring


Assessment Folder Rights per Administrator


Assessment Folder Rights Set in Browser-based Authoring


Topic and Folder Rights No Access - the selected administrators cannot do anything with

the selected root topics and any subtopics below them, and they will not appear to these administrators in the Authoring Manager window.

View/Use - the selected administrators can only view, tryout or copy the selected root topics and any subtopics below them

Edit/Use - the selected administrators can do everything with the selected root topics and any subtopics below them, except:

Delete the root topic Delete a subtopic below the root topic Delete a question below the root topic Delete an item resource associated with the root topic or a subtopic below

it Delete item content associated with the root topic or a subtopic below it Delete a topic outcome associated with the root topic or a subtopic below

it Cut any of the above

Full Access - the selected administrators can do everything with the selected root topics and any subtopics below them


Administration Rights


Reporting Rights


Results/Monitoring Rights



Enterprise Manager Overview


Enterprise Manager Main Menu


Enterprise Manager Main Menu

Enterprise Manager Main Menu Menu options will vary on the basis

of administrator privileges



System Administration


System Administration Main Menu


Creating Participants Create a single participant Create multiple participants Import participant list


Create Single Participant


Sample Participant


Create Multiple Participants

Assign multiple, new participants to a single group

Generate Participant IDs automatically


Import Participants

Import large numbers of participants from properly formatted data file and add them to a group single handedly.


Manage Participants


Manage Groups

Groups of participants Can be used to reflect your organization’s

structure E.g. geographic regions or functions (i.e.

sales, marketing, finance, HR, R&D, Administration)

Schedule groups to take an assessment Report on groups collectively

Compare performance of one group to another


Edit Groups

Create unlimited numbers of subgroups to reflect your organizational structure


Schedules Schedule individual participants to take an

assessment Schedule groups to take an assessment Schedule options

Limit attempts Override time limit in hours Limit number of days between retakes Access period Require monitoring


Schedule Individuals


Schedule Individual – Step 2

Select the Schedule of Choice• Web Delivery• Certified Test Center


Add Assessment

Add an Assessment to Individual Participants


Select Assessment


Schedule Assessment Set Schedule Properties

Name the schedule Limit the number

of attempts a participant can take the assessment

Override the time limit in minutes

Limit the number of days between retakes

OR Set the precise

access period for the specified assessments

Indicate whether the schedule requires monitoring or not


Review Schedule

Review participants assigned to the schedule

Indicate whether to announce the schedule via email


Test Centers Schedule individuals or groups to take

assessments at a certified test center Create new test centers


Schedule Test Center


Create Test Center Schedule


Add Assessment to Group


Schedule Details


Review Groups


Manage Schedules


Creating Administrators The Root Manager can setup administrative

users with different levels of permission and access to: Questions and Assessments Reports Participants and Groups other related content and features

The Administrator permissions that can be set in Enterprise Manager mirror those that can be set in Windows-based Authoring Manager


Creating Administrators – Personal & Group Permissions


Creating Administrators – Authoring Permissions


Authoring Rights in Enterprise Manager and in Authoring Manager

Authoring Rights in Windows-based Authoring Manager


Creating Administrators – Administration Permissions


Creating Administrators – Reporting Permissions


Creating Administrators – Results Permissions


Creating Administrators – Monitoring Permissions


Managing Administrators

Manage Administrator in Enterprise Manager

Manage Administrator in Windows Authoring Manager


E-mail Broadcast Email Broadcast

Send e-mails to individual or groups Send an announcement when scheduling participants

or groups for an assessment Send a reminder based on schedules and remaining

attempts Select session variables to include in message

E.g. unique username and password


Email Broadcast


Server Management

Maintenance and configuration tools used exclusively by the Root Manager (a.k.a. the System Administrator)



Results Manager


Results Manager Tools for managing

results data Access to each tool is

selectively assigned to administrators


Delete Results

Results Remover


Delete Selected Results

Sometimes the results of an assessment are invalid and need to be deleted so as not to skew the aggregate results


Score Editor

Score Editor Sometimes it’s necessary to manually

adjust the score for a question The correct answer may have been mis-keyed There may be more than one correct answer The question may have been ambiguous The question may have been unfair


Score Editor


Scoring Tool Evaluate Essay and Spoken

Response questions according to a rubric

E-mail performance-specific feedback to participants



Results Importer

Results Importer Allows you to use Perception’s Enterprise Reporter with

assessments that are not delivered by Perception Enter results manually or import from a properly

formatted data file Associated results with an existing Perception

assessment Or create a new assessment to associate the results with



Enterprise Reporter Overview


Reporting Evaluate progress and identify trends Understand assessment performance over time Share results with key stakeholders Eleven standard reports

Reports can be customized / Design your own report Filter data by group, participant, date, score band,

status, and special fields Administrator access to specific reports can be controlled

by security settings Open and well-documented database can be used with

3rd-party tools


Reporting Process1. Identify the assessment(s) or the participant on which to

run the report2. Edit report template to display necessary information3. Set filters on data set

a. Groupb. Datesc. Statusd. Special Fields

4. Generate report5. Distribute results

a. Electronicb. Hard-copyc. URL


Finding an Assessment

Simple Search

Advanced Search


Finding a Participant

Simple Search

Advanced Search


Assessment Overview Report Generate for a single assessment or a

selected group of assessments Graphical representation of

Assessment score distributions (histogram) Topic store distributions Score band distributions

Calculates standard deviation for assessment scores and topic scores


Assessment Overview Report


Export to ASCII Export results for a single assessment or a

selected group of assessments to an ASCII text file Files can then be imported into other tools

E.g. SPSS Includes

Question detail Topic analysis

Specify file parameters Include column headings Field delimiters Percentage format Date format


Export to ASCII Export data for a single assessment or multiple assessments

Use advanced search capabilities to identify the assessments for reporting


Export for Excel Export data on a single assessment to an

MS Excel spreadsheet Includes results data and question data

Enables further analysis and manipulation of results

Enables more control over format and layout of data


Export for Excel


Coaching Report Provides a detailed view of how an individual

participant performs on a single assessment Includes detailed information on individual

questions Type of question Outcomes chosen/Possible outcomes Answers given Actual score/Maximum score Feedback presented and much more

Includes topic analysis Excellent tool for feedback and remediation Can be linked to from the Grade Book and

Transcript reports


Coaching Report


Coaching Report – Topic Analysis


Score List Report Generate a list of scores for different

participants for a single assessment Include link to Coaching report for details

on an individual’s performance


Score List Report


Grade Book Report Generate a list of participants with their associated

assessment results for more than one assessment Define a list of assessments to include in report

This list can include all assessments used in a given course, for example.

Assign weights to assessments Useful for calculating cumulative scores where the scores

of certain assessments comprise a greater percentage of the cumulative score than those for other assessments

Include link to Coaching report for details on an individual’s performance

Slide 100Copyright © 1995-2007 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Grade Book Report


Transcript Report Generate a list of all assessments and their

associated results taken by an single participant

Include link to Coaching report for details on a single assessment


Transcript Report


Gap Report Used to compare two sets of results

Compare the performance of two groups Compare pre-course test results to post-course

test results

Displays differences graphically Displays assessment score histogram and

topic score histogram for each group


Gap Report


Survey Report Ideal for reporting on reaction assessments

E.g. Course evaluations, Level 1 evaluations

Graphically displays frequency of responses

Calculates mean score for each question


Survey Report


Question Statistics Derive useful statistics on individual

questions Analyze questions by assessment or by

individual topic Use information to improve quality of your

measurements


Question Statistics


Item Analysis Report Provides detailed psychometric statistics on

individual question P Value (difficulty) Item/Outcome discrimination Item/Outcome correlation

Facilitates quality assurance measures for soundness, fairness, and validity

Identify items that are too easy or too hard Identify items that do not discriminate well between high

performers and low performers Identify items that do may not be valid as their

correlation between a correct answer and a good score on the assessment is weak


Item Analysis Report


Item Analysis Report


Reporting Security

Specify which reports an administrator can run Specify if administrator can run reports on

results for all groups or just those to of which he or she is a member

Documents

Copyright © 1995-2007 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark