Embed Size (px)
Citation preview
Nathan Thompson Terry Ausman
SIFT: Software for Investigating Fraud in Testing
2
SIFT
Cheating happens everywhere and it is important to minimize it if we want to keep validity
3
SIFT Most operational focus is on deterrence (rightfully so) Data forensics is less widespread Two things prevent more orgs from doing this
important work
1. Lack of summary literature & resourcesa. Until recently – Wollack & Maynes (2013) and Kingston & Clark
(2014) – you were on your own!
2. Lack of user-friendly software
4
The Need for SIFTNot the issue here:
-Whether cheating happens-Choice of forensics to find it
The issue: Lack of user-friendly software for data forensics, so more orgs can do it!
5
SIFT
Options to implement data forensics Hire consultants Write your own code Find old software (SCheck, Scrutiny, Integrity) New: SIFT Also new: R packages
6
Options for Data ForensicsIntra-Indi
vvidual
•Time/RTE (CBT only)•Response patterns•Score gains•Person fit
Inter-Indivi
dual
•Collusion Indices•Erasure (paper only, also Group level)
Group
•Roll-up of intra and inter•Other useful stats (pass rate, mean score…)
7
It’s a Hypothesis Test!
12/2/2014
If you aim at nothing, that’s exactly what
you’ll hit.
8
It’s a Hypothesis Test!
Independent variables Test centers/locations Countries Training programs Test forms Individuals Operational vs. Pretest items
9
It’s a Hypothesis Test!
Dependent variables Item response or test time Item statistics Test statistics (mean/SD, pass rate) Person statistics (intra-individual) Collusion indices
12/2/2014
10
SIFT Output
Intra-Individual Response Time Effort (RTE) Mean item/test time Response sets
Option proportion > x Operational vs. Pretest items Score gains (future) Person fit (future)
11
SIFT Output
Inter-Individual
Collusion indices
Response time similarity (future)
Started with older/simpler indices and working forward!
12
SIFT Output
Group Descriptive stats (mean, SD, pass rate…) Roll-up of collusion indices Roll-up if intra indices Time usage Item P Operational vs. Pretest scores
13
How do these look with data?
Data set 1: Real School district summative assessment 31 items N=1372 IRT parameters available (can do Omega)
14
How do these look with data?
Data set 2: Modified Took Set 1 and…
Created fake schools/teachers Implemented collusion for a few teachers Shortened item times for one teacher’s students
15
How do these look with data?
Data set 3 English assessment from Indonesia N = 16,666 6 districts 50 items
16
How do these look with data?
Data set 4 Math assessment from Botswana N = 2,185 10 districts, 336 schools 72 items
17
Summary
Use case SIFT allows, for example, a small certification
organization to obtain all of this output in only a few hours of work, and quickly investigate locations before a test is further compromised.
18
Summary Future of SIFT More indices to be added (post-2000)
Currently in MVP stage Add secondary analysis of indices (e.g., mean per
location) Group-level Z statistics (Sotaridona) Graphics!
Free version available for download assess.com
19
Summary
Future of SIFT Most important: getting integrated with our online
testing platform so that our clients do not need psychometric expertise
Easier to get the output (saves time/money) – many professionals do not have the time to export files and learn SIFT or R
Easier to interpret – will take a lot of good thought and UX design!
Summary – Q&A
