Delivering Assessments Safely and Securely-Letter

8/2/2019 Delivering Assessments Safely and Securely-Letter

1/24

QuestionmarkWhite Paper

Delivering AssessmentsSafely and Securely

This paper discusses the issues to consider in delivering assessments using computer or web-basedtechnologies. It is designed to help readers select delivery methods and technologies that are most in

keeping with a particular assessments purpose and nature. The paper explores the various meansand technologies available for deploying a wide range of assessment types safely and securely.

Authors: Eric ShepherdJohn Kleeman

Joan PhaupGreg Pope


2/24

3.29.200910:18:24 Copyright Questionmark 2003, 2009 Page 2 of 24

1. Introduction

The use of computer- and web-based applications to assess knowledge, skills, and attitudes is nowwidespread. Today, distinguishing between the various delivery and security requirements for eachstyle of assessment is becoming more important. Over-engineering low-stakes assessments can resultin unnecessary costs and wasted time. Under-engineering high-stakes assessments can undermine

peoples confidence, organizations processes, and undermine the face validity of the assessment.

This paper focuses primarily on assessments that utilize computer- and web-based applications ratherthan paper-based assessments. It describes various types of assessments and explains numerousdelivery options so that you can select appropriate methods for deploying each type of assessmentsafely, securely, and cost effectively. The paper also explains how Questionmark products can beused to ensure the safe and secure delivery of assessments.

One important consideration when selecting a delivery method is whether assessments are used to:a. Measure knowledge, skills, attitudes, and personality traits; orb. Promote learning and reduce forgetting.

Another way of choosing a delivery method relates to an assessments consequences. After someassessments people might be hired, fired, promoted, demoted, graduated, not graduated, releasedfrom custody, and/or authorized or certified to perform a particular job or role. The potentialoutcomes of an assessment could affect a participants propensity to cheat and consequently howmuch security must be applied.

Specific assessments are often referred to as quizzes, tests, exams, or surveys depending on whetherthe intent of the assessment is to measure or promote learning. We have created some distinctionsfor some terms in the table below that we will use throughout this paper:

Term Definition Measure or

Learn

Consequences

or stakesAssessment Any systematic method of obtaining evidence

from posing questions to draw inferences aboutthe knowledge, skills, attitudes and othercharacteristics of people for a specific purpose.

Exam A summative assessment used to measure astudents knowledge or skills for the purpose ofdocumenting their current level of knowledgeor skill.

Measure High

Quiz A formative assessment used to measure a

students knowledge or skills for the purpose ofproviding feedback to inform the student oftheir current level of knowledge or skill.

Promote

learning

Low

Survey A diagnostic assessment to measure theknowledge, skills, and/or attitudes of a groupfor the purpose of determining needs requiredto fulfill a defined purpose.

Measure Low


3/24


Test A diagnostic assessment to measure astudents knowledge or skills for the purpose ofinforming the student or their tutor on theircurrent level of knowledge or skill.

Measure topromotelearning

Medium

(See the Appendix for detailed examples of assessments types.)

In addition to this terminology we can classify assessments by using the following terms:

Name ofassessment

Uses

Formative These assessments are designed to help students turn information into knowledgeby asking questions that tend to increase remembering and reduce forgetting.

Summative These assessments are designed to measure knowledge, skills and attitudes byasking questions and measuring the responses.

Diagnostic These assessments are designed to provide a diagnosis and prescription to help

people reach their objectives.

The method of delivering an assessment depends largely upon the use and consequences of theresults, but also on the motivation of the participant to complete the assessment.

Type ofassessment

Consequences Motivationto completeassessment

Exams Consequences for passing or failing an exam might extend tobeing hired, fired, promoted, demoted, released from custody,

authorized or certified, graduating or not graduating.

High

Quiz There are few if any consequences for passing or failing a quiz.Quizzes are normally formative assessments used to promotelearning and reduce forgetting rather than forming a summative

judgment.

Medium

Survey There should never be direct consequences to a respondent i.e.the person answering a survey. However, as results aretabulated for the group, training courses, job aids, and otherinterventions may be planned. Sometimes the opinion mighthave consequences for others such as an instructor in the contextof a course evaluation or managers in the context of an employee

attitude survey.

LowThe challenge is to

get people torespond.

Test The consequences for passing or failing an individual test are nothigh. However, consistent failures might extend to being hired,fired, promoted, demoted, graduating or not graduating.

Medium


4/24


This paper does not attempt to describe all of the consequences, legal liabilities, validity, reliabilityand planning issues that are involved with writing an assessment that is suitable for its purpose.Many other publications detail the processes and procedures used to produce good and fairassessments. However, as you can see from the chart below all of these issues are related to thestakes of an assessment.

What kind of assessments are we talking about?


5/24


This paper details issues related to safely and securely delivering assessments without over- or under-engineering the delivery and deployment process. The following table details some of the issues thatyou must consider when using computer or web-based applications to deploy assessments:

Type of assessment Delivery Issues to consider

Exams o Face validity (trust of the results)o Motivation to cheat is higho Over exposure of contento Content is expensive or time consuming to produceo Content protectiono Authentication of candidateo Correct authorization (pre-requisites completed)o Time window to limit accesso Disallow repeated access to examo Consistency of deliveryo Secure player requiredo Answers saved regularly in case of a technical failure

Quiz o Content protectiono Environment similar to performance situationo Motivation to cheat is medium

Survey o Ballot riggingo Anonymity of respondent

Test o Face validity (trust of the results)o Motivation to cheat is higho Over exposure of contento Content protectiono Environment similar to performance situationo Time window to limit accesso Limit the number of times a person can have repeated access to test

o Consistency of deliveryo Secure players are recommendedo Answers saved regularly in case of a technical failure


6/24


2. Environments for delivering assessments

The delivery environment for an assessment is driven by the stakes of the assessment. The higherthe value of the assessment, then the more care will be taken to control the environment and thesupervision of the assessment. As assessments come attached with a wide variety of consequences,there are a wide variety of ways and places that can be used to deliver an assessment.

Delivery method

High-Stakes Assessments

Low-Stakes Assessments

Professional monitor(proctor/invigilator)

No monitoring

Dedicated workstation Professional center

Dedicated workstation Franchised center

Secured workstation Training center

Any computer controlled environment

Any computer in an uncontrolled environment

Low volume High volume

2.1 Testing Centers

Testing centers are controlled environments set up to provide groups of users with a venue where theconditions laid out by the organization that is running the assessment can be met. This means thatcandidates can rely on fair conditions under which to take the assessment.

High-stakes assessments must offer a consistent environment that is free from distractions togetherwith some form of monitoring by an invigilator or proctor. Low-stakes assessments do not requiresuch rigor. The greater the rigor that is expected, the more expensive it will be to deliver anassessment. The physical environment, technology, monitoring standards and authentication methodsfor delivering an assessment are stipulated by the sponsoring organization that owns the assessment.

Physical Environment: Testing centers should be enclosed rooms without through traffic. Theyshould be quiet and free from distractions, well lit and with a comfortable temperature throughoutthe year. Workstations should be separated from one another by acoustic partitions and the

equipment should offer similar operation, accessibility options and performance. Candidatesshould not be able to print or capture the assessment content and should be prevented fromaccessing resources, such as web pages, that might unfairly assist them while they take theassessment. Also: See tips in Sections 4 and 5 of this white paper.

Technology Environment: Assessments should be delivered on consistent and comparabletechnology. Screen sizes, processor speeds, and network connections do not have to be thelatest and greatest but they do need to be comparable. It is vital to anticipate and plan for


7/24


technical requirements for delivering bandwidth-intensive content such as video, which putsincreased demands on computer networks.

Monitoring: Testing centers can be monitored (or proctored/invigilated) in a number of ways.Candidates should be in an environment that permits constant visual monitoring. This can beachieved directly by a person in the room or via video surveillance equipment. Also: See tips inSections 4 and 5 of this white paper.

Authentication: It is important to confirm that the person taking the test or exam is truly theperson authorized to do so. This can be done by confirming a candidates identity via aphotographic ID provided by the government, academic institution, or, in some cases theiremployeror simply a signature.Also: See tips in Section 4 about screening candidates.

To aid the following discussion we have described the issues to consider and established the followinggrades to help you quickly understand the suitability of the delivery method for a particular style ofassessment.

Quick Reference Guide for Suitability

Suitability for style of assessment GradeVery suitable A

Suitable BSuitability will vary by circumstance C

Unsuitable DDefinitely not suitable F

2.2 Types of dedicated testing centers

Testing centers come in various guises. Some of the ways in which they can be defined are as

follows:

Professionally controlled centers: Professionally controlled centers provide a veryconsistent and highly controlled environment where a candidate could expect to receiveexactly the same treatment and receive the same experience from one testing center toanother. This consistency would start with the reception and continue through the assessmentprocess to the time that they leave the premises. These centers are used for very high-stakestests such as nursing and medical exams and are expensive to use and maintain. Thesecenters provide the consistency and rigor required for very high stakes exams such asgovernment regulated certifications exams. Non-regulatory exams are rarely delivered in thesecenters due to their high running costs.

Professionally controlled centersForm of assessment Grade

Very high-stakes exams A+High-stakes exams A

Quiz FSurvey FTest D


8/24


Franchised centers: Franchised centers are similar to professionally controlled centers butthey cannot enforce the rigor required for very-high-stakes assessments. However, thesecenters are less expensive and provide a great place to offer high-stakes tests in a reasonablyconsistent environment. These centers are far more prolific than professionally controlledcenters and deliver government regulated exams as well as non-government regulated exams.There are thousands of franchised centers around the world.

Franchised centers

Form of assessment GradeVery high-stakes exams B

High-stakes exams AQuiz F

Survey FTest C

Higher education centers: Many higher education institutions offer their testing facilities to

local companies and are sometimes part of a franchised assessment network. These centersare similar to professionally controlled centers and they enforce the rigor required for theirinstitutions. These centers are less expensive and provide a great place to offer high-stakestests in a reasonably consistent environment. These centers are ideal, not only for theinstitutions use but also for local companies that need to administer high-stakes assessmentsin conjunction with a local institution.

Higher Education centers

Form of assessment GradeVery high-stakes exams B

High-stakes exams A

Quiz DSurvey FTest C

Corporate centers: Increasingly large organizations are establishing their own testingcenters to provide a consistent, confidential, and cost-effective method to deliver high-stakesassessments. These centers might be testing potential employees, partners or existingemployees on safety or government regulation related issues. These centers are equipped in asimilar fashion to professionally controlled or franchised centers.

Corporate centers

Form of assessment GradeVery high -exams AHigh-stakes exams A

Quiz CSurvey CTest B


9/24


10/24


Remotely supervised in work place

Form of assessment GradeVery high-stakes exams C

High-stakes exams C+Quiz C

Survey C

Test B

Unsupervised at work place:Low- and medium-stakes assessments taken unsupervised ata participants desk or other work venue are becoming increasingly important as organizationsstrive to discover how effective their training is, determine who needs training, and providenew ways of promoting the learning process. Although this form ofassessment doesnt requirethe same monitoring, it does require assessment software to prevent people printing orcapturing confidential information.

Unsupervised in work place

Form of assessment Grade

Very high-stakes exams FHigh-stakes exams D

Quiz ASurvey ATest A

Supervised at home:Some distance learning institutions and small-scale certificationauthorities are now administering higher-stakes assessments at home. However, thecandidate must provide the organization with a choice of professionals (e.g. a doctor, lawyeror accountant) to supervise the assessment. Once again, the success of this kind of delivery

lies in the assessment software and the selection methods for the monitor.

Supervised at home

Form of assessment GradeVery high-stakes exams C

High-stakes exams AQuiz C

Survey CTest A

Unsupervised at home: Most commonly used for delivering low-stakes self-assessmentsaimed at promoting learning and reducing forgetting.

Unsupervised at home

Form of assessment GradeVery high-stakes exams F

High-stakes exams FQuiz A


11/24


Survey ATest B

Unsupervised in a public place (library): Most commonly used for delivering low-stakesself-assessments aimed at promoting learning and reducing forgetting.

Unsupervised at home

Form of assessment GradeVery high-stakes exams F

High-stakes exams FQuiz A

Survey ATest B


12/24


3. Creating the right environment

Delivering assessments securely is becoming easier as the quality of networks and assessmentsoftware improve. Secure assessment players such as QuestionmarkSecure, as well as monitoringsolutions, are pushing back the barriers of what types of assessments can be safely and securelydelivered either remotely, at the desk or in a training room. When close monitoring and a consistentenvironment are essential for geographically dispersed candidates, there are many excellent testingcenters to choose from.

Many organizations are now being empowered by assessment software to create and run high-stakesassessments within their own environments. As the quality of software, monitoring and networksolutions grow, this trend will certainly continue.

Case study

Not all organizations believe that it is essential to have exams proctored.

A U.S. software company perceived that it was more important to ensure thatcertification was available to everybody who wanted to get it than to insist that

all certification exams be proctored. Setting proctored assessments in specificlocations at specific times inevitably limits the opportunities for sitting theassessments.

Instead, following the successful completion of the software training courses,candidates can register using a credit card for the exam to become a certifiedprofessional in the use ofthis companys products. Candidates then take theappropriate assessments at home or wherever they wish. The assessments areauthored, delivered and reported on using QuestionmarkPerception.

The company argues that, should anybody dispute the validity of a certified

professional, they can arrange to serve as proctor and request that the personretake the test. As a result, the company has found it much easier to run itscertification program and is confident that the system is not abused.

Many organizations have varying approaches to delivering their assessments. Below are several casestudies that exemplify how Questionmark is used in varying contexts.

Case study

Healthcare Financial Management Association (HFMA)

The Healthcare Financial Management Association (HFMA) is a not-for-profit

organization for CFOs, comptrollers, accountants and other financialmanagement professionals at healthcare organizations. HFMA members can earnthe Certified Healthcare Financial Professional (CHFP) designation throughprofessional experience, education, and successful completion of two certificationexams: a general exam in healthcare finance practices and an exam in one offour -specialties: accounting and finance, patient and financial services, financialmanagement of physician practices, or managed care. Exams are administeredusing Questionmark Secure, which locks down candidates desktops and prohibits


13/24


task switching. Each examinee must enter a password, (good for only 24 hoursand allowing only one attempt). The proctora certified HFMA member who hassigned an affidavit agreeing to abide by stated proctoring policiesmust alsoenter a password. There are three versions of each exam, and questions poolsare updated every two years.

Case studyUniversity of South Alabama, College of Medicine

At the University of South Alabama (USA) in Mobile, the College of Medicineoffers courses in specialties such as neuroscience and microbiology to students intheir first two years of medical school. For many years, tests were administeredby way of paper and pencil. But the Board of Medical Examiners requires medicalstudents to take its certification exams online, and faculty wanted to helpstudents prepare for them using online tests. High-resolution images andgraphics have replaced the low-quality photocopies and slides that studentspreviously viewed together during tests. Now students can move at their ownpace, viewing clear images of cells, abnormalities, and processes that they must

identify. A portable computer lab utilizes rented laptops dedicated strictly toonline testing with the protection of Questionmark Secure.

Case study

San Francisco Bay Area Rapid Transit District (BART)

The San Francisco Bay Area Rapid Transit District (BART) rail system has longrequired job candidates to take tests before they are hired. For many years, testquestions were handwritten on 3 x 7 cards. BART now uses QuestionmarkPerception to screen potential hires, most of them applying for jobs inelectronics. Pre-employment tests pose multiple-choice questions about such

electrical concepts as Ohm's Law, circuits, currents, voltage, and otherknowledge. Because the candidates are often very tech-savvy, QuestionmarkSecure prevents instant-messaging, Web surfing and other tasks. The softwarelocks the desktop so that none of these applications can be used during the test.

Case study

Louisiana State Universitys Health Sciences Center

At Louisiana State Universitys Health Sciences Center, tests for pathologystudents incorporate images of specimen slides, tissues, and other similar visual

material. In the days of paper-based tests, Individual test booklets had to beduplicated and then shredded after each exam, a costly process. When the U.S.Medical Licensing Exam or USMLE began using computerized testing, LSUfollowed suit. LSUs medical school had already set the groundwork for usingcomputerized testing by requiring all students to own laptops that run the sameoperating system and use the same software. In addition to the static imagesthat have always been an important part of pathology testing, online questionsdelivered from a centralized server now also incorporate video clips and flash


14/24


animations. Every student gets the same exam, but questions are randomized toproduce individualized tests. Even the choices are randomized, making the testsmuch more secure and reducing the temptation to cheat.

Case study

Security for assessments tied to employee promotions

A California-based financial services firm tests loan trains its underwriters,account executives, account managers and quality control auditors in keepingwith Federal Deposit Insurance Corporation (FDIC) regulations and investorsguidelines. The company uses Questionmark Perception to create and administerassessments for required procedures in underwriting, fraud detection, regulatorymanagement, and funding, among others. Some of the companys assessments tare tied to promotions, and therefore require particularly tight security, so theyare delivered using Questionmark Secure which locks down computers andmakes it impossible for test takers to access the Internet, print the test or useother software programs during test-taking.

Creating the physical right environment: The ideal assessment environment would include thefollowing characteristics:

Comfortable, clean and smoke-freeAdequate lighting, ventilation, comfortable seating and work surfaces for the candidateduring the assessment sessionFree from external distractionsNo opportunity to use equipment such as printers, fax machines, copiers or phoneswhile taking an assessmentProvide a viewing window and/or video surveillance systemIf a monitor is present, they should have an unobstructed view of each candidatewithin the assessment room.IT equipment should be consistent and robust and comply with the local accessibilityrules and regulations.

Deploying software to deliver secure assessments:Although many issues can be addressedwithin a physical environment, new software features can also be used to ensure that assessmentsare delivered in a secure way. The following features are to be found in QuestionmarkPerception:

Authoring secure assessments: There are many safeguards that can be taken during theassessment authoring process to help protect the security of the assessment. One of the mostpopular is to shuffle the order of the choices. The questions can also be delivered in a random

order themselves. Both of these features help prevent cribbing when users are sitting in non-screened testing centers.

Encrypted communications: With so many assessments being delivered via the Internet or anintranet, it is important that communications between the browser and server are encrypted. Thismeans that if someone sniffs the networkthey would not be able to see the information sentfrom server to browser and vice versa. Perception deploys Secure Sockets Layer (SSL), a protocol


15/24


that allows the browser and web server to encrypt their communication. This means that anyoneintercepting the communication would not be able to read it.

Scheduling assessments: Using Perception enables users to set start times of assessments,the length of assessments and the number of times that assessments may be taken. It alsoenables user names to be specified and password protects the assessments.

Monitoring assessments: Perception also includes a monitoroption. A monitored assessmentcannot be started by the participant until a proctor or invigilator has logged on after thecandidate. The monitor can also be limited to a range of IP addresses to ensure that a certainphysical location is used to administer the assessment.

Secure browsers: One of the key recommendations is to lock downcomputers that are beingused in assessments to keep users from accessing other applications and websites while engagedwithin a medium- or high-stakes assessment. This lockdown is usually performed via a securebrowser, such as QuestionmarkSecure, and prevents candidates from printing, capturing screens,accidentally exiting the assessment viewing source, task switching, etc. The secure browser mustbe effectively authenticated by the server to prevent spoofing by technically competent cheats.

Having considered the alternatives for delivering assessments, it is now worth considering theassessment software, tools and environments that you might need to create, deploy and report onyour own secure assessments.


16/24


4. Tips to help prevent cheating and ensure assessment security

Ensuring the security of assessment results is crucial to ensuring the validity of scores. This sectionprovides some tips regarding how to prevent cheating and help ensure that assessment security ismaintained. These can be used in conjunction with the technologies outlined in section 6.

Screening tests

A small pre-screening can be administered to prevent people from taking an assessment for whichthey are not yet prepared. This could contain a few questions about the course they have taken. Ifthe participant cant answer a certain number ofthese questions correctly they will not be allowed tosee the remainder of the assessment. This can be achieved using jump blocks within Perception.

Candidate agreements

Candidate agreements or examination honor codes are codes of conduct that a participant mustagree to before they start an assessment. Candidate agreements generally are phrased in a personalmanner; the participant agrees by clicking on an OK or Yes button to the code of conduct for the

exam.

An example of a simple candidate agreement may be: I agree to answer the questions on thisassessment without obtaining assistance from another person or via electronic means. I agree to notshare my answers with anyone during or after the exam. I further agree to not memorize orotherwise steal the intellectual property contained in this exam. I accept that if any of theseconditions are violated my exam results will be set to a zero, I will not be able to retake the exam fora period of 10 years, and I may be charged with a crime under regional laws.

Topics that may be covered in a candidate agreement:The test vendor will have the option to terminate the assessment if suspicious behavior isdetectedThe candidate must abide by the rules of the test center, organization, or programThe candidate will not provide false ID or false papersThe candidate cannot take the test on behalf of someone elseThe candidate will not engage in cheating in any formThe candidate will not assisting others to cheatThe candidate will not assist others to cheat by disclosing information about the assessmentThe candidate will not using aid that are not allowedThe candidate will not solicit someone else to take the testThe candidate will not cause a disturbance in the testing centerThe candidate will not tampering with the test center in any wayThe candidate will not share information about the assessment content they saw (non-

disclosure agreement)

Limiting content exposure/leakage

In order to limit the amount of question content being shown to a participant at any given time,consider using question-by-question templates. These present questions one at a time to participantsso that exam content is not completely exposed on screen. Participants who may intend to take


17/24


pictures of the exam content or otherwise steal intellectual property will not be able to do so all atonce.

Screening participants who achieve perfect scores

Many organizations will automatically investigate participants who achieve perfect scores on anassessment, since obtaining perfects scores is generally a rare event. They may have a perfect scorebecause answer keys have been compromised or because they were aided in answering thequestions. The Questionmark Score List Report provides a fast and easy way to identify participantswho obtain 100% on their assessments. An organization can then conduct an investigation of theseparticipants to ensure that no suspicious behavior had occurred (e.g., interviewing the exam proctor).

Verifying expected IP addresses

If assessments are to be taken from a specific location, often the IP address of the computer in thatlocation will be known. Verifying that all participants who took an assessment did indeed take theassessment from the expected location(s) is a useful way to screen whether participants somehowtook an assessment from an unauthorized location. The Questionmark Score List Report provides the

IP address for all participants taking an assessment and is a useful tool for screening expected IPaddresses.

Reviewing time to finish information

The overall time it takes for a participant to complete an assessment can be a useful way to screenfor suspicious behavior. If a participant takes a very short period of time to complete theirassessment yet achieve a high score this could be an indication that they cheated in some way,

Using Trojan horse or stealth items

Trojan horse or stealth items can be used to help detect whether a participant has memorized theanswer key. Stealth items are inserted into an assessment and look just like the other questions, butthey are purposely keyed incorrectly. The actual correct answer is marked incorrectly and one of thedistracters is marked as the correct answer. These items are generally included as non-scored itemson the assessment. If a participant is simply memorizing content and keyed correct answers they willlikely choose alternatives that they have memorized. Participants with overall reasonable assessmentscores who got the stealth itemscorrectmight have memorized the answer key

Post information that cheater prevention tactics are used

An effective tool in preventing cheating is to inform participants that reviews are regularly conductedto identify cheaters. Participants should not know details about what sort of reviews are conducted

(for example the tips listed in this section) but should be told that cheater-detection tactics areregularly employed. This can help to deter the low-motivation cheaters.

Proper seating arrangements for participants

Implementing a seating plan where participants are adequately spaced with limited ability to seeanother participants screen/paper is an import strategy for enhancing test. The proctor should beaware of cheating techniques such as the flying V seating arrangement where the giver at the


18/24


point of the V feeds information to a number of receivers behind them. The givers and receivers cancommunicate in a number of ways, using sign language, dropping notes on the floor, etc.

Example of the flying V answer copying formation (Cizek, 1999):

This and other techniques for cheating are eloquently documented in Dr. Gregory Cizeks bookCheating on Tests: How to Do it, Detect it, and Prevent it,which is an excellent resource on thistopic.

Using unique make-up exams

Many organizations offer make-up exams for participants who were sick or had legitimate excuses fornot being able to take an assessment at the scheduled date and time. If an organization uses thesame exam that was administered at the scheduled date and time for their make-up exam, they openthemselves to risks of the exam form being compromised. Sometimes the make-up exams are notadministered in the same strict proctored environment as the scheduled exam, allowing participantsthe opportunity to cheat or steal content. Having another test form available specifically for make-upexams can lessen the risks of cheating and exposure for the actual large-scale exam.

Using more constructed response questions

Constructed response questions, like essay or short answer questions, provide less opportunity forparticipants to cheat because they require them to produce unique answers to questions. There is noanswer key to steal, and participants who copied other peoples constructed response answers areeasily identified via a side-by-side comparison of answers.


19/24


5. Tips to help ensure the security of intellectual property

Create and administer multiple test forms

Rather than having only one form of the assessment being administered, delivering multiple forms ofthe same exam can help limit item exposure. If one exam form is breached, the other exam formscan stay in circulation. This method also allows for the possibility of interspersing large-scaleintegrated beta test questions within the forms to collect psychometric information on newlydeveloped questions.

For low-stakes exams, Perception allows for the random selection of questions from topics for eachparticipant, lessening the chance that participants will see the same questions and thereby limitingitem exposure (depending on the size of the repository).

Restrict and control administration of beta test items

Beta testing questions is an important part of high-stakes assessment, ensuring the psychometric

quality of questions before they appear on actual assessments. However, it is vital that a wellconceptualized beta test model is in effect which limits the exposure of newly developed questions toparticipants. Also, the beta test questions must be administered in secure environments, in similarconditions to the actual exam. This prevents the exposure of new questions before they appear on anactual assessment.

Some rejected beta test questions could be considered to be part of exam prep materials, discussedlater in this section.

Update exam forms periodically

Letting exam forms become stale (not cycling out old question and cycling in new questions) canover-expose questions to participants, increasing the likelihood of IP theft. Periodically updating examforms (e.g., annually) can help limit the exposure of questions. An organization could consider

retiring old exam forms and turning them into exam prep materials that can be sold to participants.In this way, participants could periodically expect new practice questions. The sales of the exam prepmaterial or practice tests can help offset increased question development costs required to provideevergreen exam forms.

Produce exam prep materials

Organizations should consider making exam prep materials available to participants before anassessment. This will help reduce the demand for participants to try and obtain exam questions via

illegal means as they will have access to the type of questions that will be asked on the actualassessment.


20/24


6. QuestionmarkTechnologies for Deploying Assessments Safely and Securely

Questionmarkprovides technologies to help you author, deliver and report on assessments safely andsecurely. Questionmarkalso provides services to help you deploy its technologies successfully.However, Questionmarkdoes not provide testing centers (bricks and mortar) or monitoring services(invigilators or proctors).

The following chart details certain features that Questionmarktechnologies provide to help youdeploy your assessments safely and securely.

Type ofassessment

QuestionmarkPerception Features to use

Exams o QuestionmarkSecureo Keeps assessment content secureo Prevents printing, capturing content, task switching, etc.

o Secure Socketso Prevents network pirates intercepting content

o Monitorso To authenticate candidateo To monitor the candidate

o Monitor from an specific range of IP addresseso Ensures the correct room/computer is used

o Assessment only available during a limited time windowo Reduces opportunities for people to relay information about the

testo Shuffle choices

o Prevents candidates seeing which choice another candidateselected

o Randomized questions

o Candidates get different questions, which reduces the possibilitythat they will see the answers from another candidate

o Jumps can stop tests for poor candidateso Prevents exposing content to candidates who obviously wont

passo Restrict delivery to specific test centers

o Prevents participants from taking exams in unauthorizedlocations

o Define assessment access periods and time periods (e.g., number ofdays) before a re-attempto Prevents participants from taking an assessment more often

than they are supposed to

Quiz o QuestionmarkSecureo Keep assessment content secureo Prevents printing, capturing content, task switching, etc.

o Secure Socketso Prevents network pirates from intercepting content

o Assessment only available during a limited time windowo Reduces opportunities for people to relay information about the


21/24


test and short-circuit the cognitive processo Randomized questions

o Candidates get different questions to practice witho Provide repeated search and retrieval practice

Survey o Respondents details (name, IP address) can be erased to:o Maintain anonymityo Eliminate ballot rigging

Test o QuestionmarkSecureo Keeps assessment content secureo Prevents printing, capturing content, task switching, etc.

o Secure Socketso Prevent network pirates intercepting content

o Monitorso To authenticate candidateo To monitor the candidate

o Monitor from an specific range of IP addresses

o Ensures the correct room/computer is usedo Assessment only available during a limited time window

o Reduces opportunities for people to relay information about thetest

o Shuffle choiceso Prevents candidates seeing which choice another candidate

selectedo Randomized questions

o Candidates get different questions, which reduces the possibilitythat they will see the answers from another candidate

o Jumps can direct students to helpful contento Prevents exposure of content that they arent ready for


22/24


Appendix:

Detailed examples of assessment types

Surveys Main purpose of assessment is to gather information or opinion

Level 1 Surveys,Course evaluations

These are specific evaluations, given after a course, to get the students feedback onthe learning activity. Level 1 Surveys are also called course evaluations or smilesheets. Level 1 Surveys typically ask for the learners evaluation of the course content,course materials, the instructor, the information delivery, etc.

Needs Analysis Surveys These surveys are given to a group to explicitly determine what areas they want tolearn more about, or where their knowledge is weak, with the purpose of providinglearning offerings to meet those needs.

Job Task AnalysisSurveys

These surveys are given to a group of people doing a job to determine which tasksthey perform, how regularly they perform them and the significance of the task to

their role.Employee Attitude/Opinion Surveys

These surveys are used within an organization to collect input from employees abouttheir feelings on one or more topics.

Customer/PartnerSatisfaction Surveys

These surveys are used by an organization with other groups than their employees; itmight include customers, prospects, partners, and others outside the organization.Classic market research is an example of this category.

Information andOpinion Surveys

These surveys are used to gather information and opinions that do not fall into thecategories above. For example surveying conformance with maturity models or sixsigma.

360 (Level 3) Surveys These are surveys designed to measure whether behavior has changed on the job,usually delivered by a 360/180 survey to colleagues. Often data is aggregated toidentify the effectiveness of learning activities

360 degree survey(employee appraisals) Information gathered from someone's self-assessment and assessment by peers (andwhere relevant superiors and subordinates) to obtain input on how the person canimprove. Organized by HR as part of a performance appraisal

360 learner peerreview assessments

Information or judgments gathered from peers in a learning context to aid learningand provide feedback.

Formative

assessments

Used to strengthen memory recall by practice and to correct misconceptions

and to promote confidence in ones knowledge

Quizzes duringLearning Quizzes are typically used at the end or in intermediate points of learning activities.Usually do not store results (or if do so just to see if questions should be promoted or forother very low stakes use).

Practice tests Practice versions of higher stakes exams, made available or sold for practice purposes


23/24


Diagnosticassessments

Assesses knowledge, skills, behaviors, understandings and/or attitudes todetermine gaps and potentially provide a diagnosis and prescription of

learning and/or other activitiesPre Tests A diagnostic assessment before a specific learning activity. Used to create intrigue, to set a

benchmark for comparison with a post course test, as a pre-requisite or route to anappropriate learning activity, and to provide instructors and mentors information on thestudent's abilities.

Placement tests A diagnostic assessment used to measure place someone in the right course or learningactivity. Has a summative element but main purpose is to diagnose and prescribe. Used toroute to the correct learning activity.

Self-diagnostictools

Assessment which asks non-judgmental questions and gives feedback to the participant toprovide recommendations for products, services and/or learning activities. An examplemight be a financial survey which asks for information and recommends financial products.

Personality

assessments

Assessment which analyzes personality traits in order to predict behaviors includes Myers-

Briggs. Sometimes gives information directly to participant and sometimes requires anexpert to review and interpret.

Summativetests

Main purpose of assessment is to measure or certifyknowledge, skills and aptitudes (KSAs)

Post CourseTests

Post Course Tests (aka Post Tests) are typically given to identify the learner's KSAs and maycontribute to passing the course. Sometimes data from these tests are used to indicate howmuch a student or group of students have advanced their knowledge, skills and attitudes bycomparison to the pre-test.

Exams duringstudy

Assessments given during or at the end of a prolonged course of study which are used todefine the grade or passing of an extended learning activity, for instance university courses.Exams are normally available only to students who have registered with an organization and

studying with organization, often in a cohort. Most commonly academic, e.g. universityexams

Internal exams Exams open to a closed community, for instance employees or partners set by anorganization for its own purposes. Examples include regulatory compliance exams andpartner verification exams.

Certificationexams

Exams open to anyone who qualifies that are used to certify or score KSAs. Commonexamples are IT certifications (e.g. Microsoft, Cisco) or general academic entry exams (e.g.TOEFL, SAT) and IQ tests. Authority comes from an organization not a legal source, butpassing can be a passport to greater pay or rewards and so the stakes are often high.

Licensing exams Exams which must be passed to meet a legal requirement or a quasi-legal requirement, forexample driving tests, medical licensing exams. These are very high stakes and need to belegally defensible.

Pre-employmentscreening

Asks questions of applicants about KSAs and experience to gate applicants. Used todetermine whether applicants meet basic requirements and can be considered for the nextstage of a recruitment process. In some cases close to a survey.

Psychologicalassessments

Clinical psychological assessments which ask questions and then determine a psychologicalprofile. These are typically validated against a norm population.

Pre-employmenttests

Tests given in pre-employment to measure KSAs to see if the applicant is suitable forrecruitment.


24/24

About Questionmark:

Questionmark provides technologies and services that enable organizations to measure knowledge,skills and attitudes securely for certification, regulatory compliance and improved learning outcomes.Questionmark solutions enable reliable, valid and defensible assessments by empowering subjectmatter experts through collaborative authoring, accommodating participant needs with blended andmultilingual delivery and informing stakeholders through timely reporting and meaningful analytics.

The Questionmark Perception assessment management system, available as a licensed, hosted orsubscription-based solution, enables organizations to create, administer and report on surveys,quizzes, tests and exams. Complete details are available athttp://www.questionmark.com

Questionmark

535 Connecticut AvenueSuite 100Norwalk, CT 06854Tel: (800) 863-3950

(203) 425-2400Fax: (800) [email protected]

Questionmark

4th Floor, Hill HouseHighgate HillLondon N19 5NAUnited KingdomTel: +44 (0)20 7263 7575Fax: +44 (0)20 7263 [email protected]

Questionmark

Parc dActivits EconomiquesAvenue Lon Champagne, 2B 1480 Tubize (Saintes)BelgiumTel: +32 (0)2 398 02 01Fax: +32 (0)2 398 02 [email protected]

Legal note

This document is copyright Questionmark Corporation (Questionmark) 2003 and 2009.

Although Questionmarkhas used all reasonable care in writing this document, Questionmarkmakes no representations aboutthe suitability of the information contained in this and related documents for any purpose. The document may includetechnical inaccuracies or typographical errors, and changes may be periodically made to the document or to the softwarereferenced. This document is provided as is without warranty of any kind. See your Perceptionsupport contract for furtherinformation.

Company and product names are trademarks of their respective owners. Mention of these companies in this document doesnot imply any warranty by these companies or approval by them of this guide or its recommendations.
http://www.questionmark.com/http://www.questionmark.com/http://www.questionmark.com/mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]://www.questionmark.com/

Documents

Delivering Assessments Safely and Securely-Letter