Questionmark | - Trustable Assessment Results · 2015-06-18 · About Questionmark. ... •Assessment Management System •Questionmark OnDemand (SaaS/cloud) •Questionmark Perception

Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Six Steps for Trustable Assessment Results: Ensuring Valid and Reliable Assessments with Questionmark

A look at how Questionmark’s tools enable valid, reliable and defensible testing and assessment

Brian McNamaraMarketing Director, Questionmark

Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Assessments – Examples and Applications

What makes assessment results “trustable”

Six steps for trustable results

Examples of supporting technology

Today’s webinar…

Slide 2


• Founded in 1988

• Offices in US, UK, Canada; Distribution partners in 27 countries

• Assessment technologies and portal solutions that enable organizations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and improved learning outcomes.

Background

• 1,000 current enterprise customers worldwide

• Corporate and Government (Learning/HR function); Higher Education; Awarding Bodies and Test Publishers

Customers

About Questionmark


• Assessment Management System

• Questionmark OnDemand (SaaS/cloud)

• Questionmark Perception (installable software)

• Multilingual

• Localized for dozens of different languages and cultures

• Global 24/7 Support

Products and Services

About Questionmark


White Paper:

Slide 6

www.questionmark.com/go/trustable-results-white-paper


Quizzes, Tests, Exams

•Questions and scoring algorithms

•Scores and feedback at question, topic and assessment level

Surveys

•Questions relate to a person’s opinions on a topic

•Conclusions drawn based on analyzing frequency of responses

•Examples include course evaluations, employee attitude surveys

Observational Assessments

•Someone assesses or rates a person’s performance, knowledge, skills, and abilities

Types of “assessments”

Slide 7


What can we learn from assessments?

Slide 8

Did they like it?

Did they learn it?

Can they do it?

Can we prove it?


Regulatory Compliance

• Ensure knowledge of laws, regulatory requirements, health & safety

• Demonstrate compliance; reduce risk

Qualification & Certification

• Reliable, valid tests to certify knowledge, skills

• Make sound, defensible personnel decisions

Sales Channel Verification

• Ensure competence and product knowledge

• Increase revenues and customer satisfaction

Technical Channel Verification

• Ensure competence and expertise of technical and service professionals

• Reduce costs and improve customer satisfaction

Measure & Enhance Learning

• Measure knowledge, skills and attitudes –before, during and after learning

• Correct misconceptions and increase knowledge retention

How our customers use assessment


Getting Trustable Results


Trustable assessment results matter

Slide 13


Recent Headlines


Make informed, defensible personnel decisions

Mitigate regulatory compliance risk

Aligned workforce competencies with organizational objectives

Target learning investments accurately

Onboard new employees effectively

Ensuring competency of sales force impacts revenue, customer satisfaction

Ensuring competency of technicians impacts costs, customer satisfaction

Why trustable results matter

Slide 15


• Assessments need to be • Valid• Reliable• Secure

• Sound practices for developing, delivering and reporting on assessments

• Technologies to support sound assessment practices

What makes results “trustable” ?

Slide 16


Valid and Reliable Assessments Reliable: dependable, repeatable, consistent

Valid: measures appropriate knowledge and skills

Figure 1:Reliable but not Valid

Figure 2:Not reliable, not Valid

Figure 3:Reliable and Valid


Planning assessment

Authoring items

Assembling assessment

Pilot and review

DeliveryAnalyze results

Six Steps to Trustable Assessments

Security


Assessment Management System

Collaborative, secure authoring of surveys,

quizzes, tests, and exams

Evaluate results and inform stakeholders

with timely, meaningful reports

Blended delivery to support a broad range of devices and security

needs


Planning assessment

Authoring items


Pilot and review


Examples of Supporting FeaturesIdentify, validate content with JTA;

security requirements

Involve SMEs in collaborative

authoring and review

Secure delivery; deliver to tablets and

smartphones

How are my tests doing? How are my employees doing?

Random, rules-based question selection

from item bank

Include “experimental” questions that don’t

impact scores


Planning assessment

Authoring items


Pilot and review

Delivery

Analyze results

Sound Assessment Practices +Assessment Management System


A closer look…Examples of features and functions supporting sound assessment practices

Slide 22

Planning assessment

Authoring items


Pilot and review



Organize items in an item bank with topic structure

Metatag items to associate with task knowledge, skill or ability

Use Job Task Analysis (JTA) surveys to help blueprint assessments

Authoring processes secure to protect against content theft

Manage translations to aid multilingual assessments

Planning assessment

Slide 23


Validate assessment content using Job Task Analysis

Ask Subject Matter Experts:

• How difficult is it to do this task?

• How important is it to do this task?

• How frequently do this task?


Multilingual Assessments

• Ability to manage translations within the application

• Import/export assessment content for localization by third party


Authoring tool Subject Matter Experts can use directly

20+ question types

Ability to include relevant stimulus including video, audio and equations

Item version history for legal defensibility

Easy collaboration for item reviewers to help make items more valid

Search questions to identify those that need update

Authoring items

Slide 26

Ability to include relevant stimulus including video, audio

and equations

Intuitive, browser-based application; SMEs can start using

with little or no orientation


Authoring tool SMEs can use collaboratively, securely

Set permissions for authors and reviewers


Item version history for defensibility

Slide 29

Audit trail of edits and option to “rollback”

Easily “track changes” made in collaborative

authoring or review


Random selection of questions from item bank

Random ordering of questions and choices

Set a pass score including topic pre-requisites

Set assessment time limit (and override if required)

Topic scoring and feedback

Assembling the assessment

Slide 30

Corrective feedback points participant to appropriate learning resources

“Topics” of questions answered in assessment

Score on topic


Assembling assessment:Random item selection

Slide 31


Easily deliver assessments for trial purposes

Capture participants comments on items to improve validity

Run useful reports on the results of trial assessments

Use experimental questions without impacting scores

Pilot and review

Slide 32


Consumer quality participant UI

Blended delivery, mobile support

Deliver observational assessments

Option to present different questions on a retake

Time limits; set time periods before a retake

Delto help measure performance

Launch and track assessments from other systems of record

Delivery

Slide 33


Deliver:Responsive Design, Blended Delivery

One Assessment…

…many devicesand browsers


Observational Assessment: Can they do it?

Observer “rates” the participant

Smartphones/tablets ideal for observational assessment


Results saved at the server “as you go” so IT failures don’t lose data

Hope for the best; prepare for the worst

Slide 36


Secure browser to reduce content theft and improve exam integrity

Delivery for high stakes

Slide 37

Windows Mac iPad

PrintingScreen captureTask switching

Control/system keysRight-click / context menu

optionsView source

Accidentally exiting exam

Prevents….


Option to require monitor to confirm participant identity

Delivery can be integrated with online proctoring solutions

Require Proctoring

Slide 38


Text sizing, contrast controls to enhance accessibility

Also support for…- Non-mouse navigation; alternate

pointing devices- Screen readers


Accurate, robust and tamper-proof results storage

Differential security on results so only the right people see them

Item analysis report to weed out poor items

Test analysis report to calculate reliability

Wide range of accurate reports to help professionals make good decisions

Results can be anonymous to help make employee surveys valid

“Big Data” potential to make better business decisions

Analyze results

Slide 40


What it measures: % answered correctly

How it is expressed: Number from 0.0 to 1.0

How to interpret: Too high = too easy Too low could mean…

Too difficult Confusing/ambiguous Miscoded

What it measures: Correlation between

item score and test score

How it is expressed: Number between

-1.0 and +1.0.

How to interpret: Generally…

< 0.2 low > 0.3 good

Reliable results: Basics of Item Analysis

Difficulty index (aka P “value”) Item correlation discrimination:

Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.Slide 42

Difficulty/P-Value

Correlation -Discrimination

Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.Slide 43


Built-in Reports Custom Reports and Dashboards

Microsoft Excel BI Tools

Big Data Applications: Results API

Slide 44

R e s u l t s W a r e h o u s e


Summary of key functions

Collaborative Authoring

•Secure

•Browser-based

•Available globally

Responsive Design, Blended Delivery

•Latest browsers

•Tablets/smartphones

•Accessibility

•Secure delivery for high stakes exams

Extensive reporting and analytics

•How are my participants doing?

•How are my tests doing?

•Results API

Multilingual

•Translation Management

•Screens/Controls in 34 languages

•Right-to-left languages

Integration with Enterprise Systems

•Supports industry standards

•APIs

•Connectors

http://www.viewcentral.com/

http://www.viewcentral.com/

http://www.sumtotalsystems.com/

http://www.sumtotalsystems.com/

http://www.trainingpartner.com/

http://www.trainingpartner.com/

http://www.meridianksi.com/

http://www.meridianksi.com/

http://www.proctoru.com/

http://www.proctoru.com/

https://www.questionmark.com/us/aicc_information.aspx

https://www.questionmark.com/us/aicc_information.aspx


White Paper:

Slide 46

www.questionmark.com/go/trustable-results-white-paper


Keep in Touch

Slide 47

blog.questionmark.com

plus.google.com/+QuestionmarkCorp/

twitter.com/questionmark

linkedin.com

youtube.com/questionmarkvideo

facebook.com

www.questionmark.com/newsletter

https://plus.google.com/+QuestionmarkCorp

http://www.questionmark.com/newsletter

Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Thank you for attending!

Brian McNamaraMarketing Director, Questionmark

Documents

Questionmark | - Trustable Assessment Results · 2015-06-18 · About Questionmark. ... •Assessment Management System •Questionmark OnDemand (SaaS/cloud) •Questionmark Perception