Streamlining assessment, feedback, and archival with auto-multiple-choice

WEB SIGMAA meetingJanuary 8, 2016

Matthew LeingangCourant Institute of Mathematical Sciences

New York University

My issues• I like weekly quizzes on

paper

• I hate data entry

• I commute by bus

•I am a digital packrat

•I have bad handwriting

Things I have tried

Home rolled scripts with PDF forms

Last yearPapers LMS

My Dream

Scantron ScanSnap

Dream Workflow #1Papers LMS

Dream Workflow #2Papers LMS

OMR Software

•SDAPS (python)•queXF (PHP)•G’n’T Eval (Ruby)•AMC (perl)

Why AMC•FOSS (perl, sqlite, LaTeX) •Good LaTeX package for creating

docs•features MCQ, FRQ, ID encoding,

numerical answers•GUI with many features•Unix Xwindows; Macports package

Examples

Simple Example\documentclass{article}\usepackage[insidebox]{automultiplechoice}

\begin{document}\begin{question}{taxicab} What is the smallest integer that can be represented as the sum of two cubes in two different ways? \begin{choices} \correctchoice{1729} \wrongchoice{1728} \wrongchoice{1001} \wrongchoice{4104} \wrongchoice{13832} \end{choices}\end{question}\end{document}

Output

Supported Variants•Check all that apply

(questionmult)•Horizontal vs. vertical layout

(choiceshoriz)•compatible with the multicols

environment•smart “None of the above” option

\begin{questionmultx}{SEssCalcET2 13-1-012} \scoring{b=2} Let $R = [0,1]\times[1,3]$.\par Evaluate $\iint_R ( x^3 - 9x^2 y^2)\,dA$. \AMCnumericChoices{-7.5}{ digits=3,decimals=2,sign=true, borderwidth=0pt, backgroundcol=lightgray, approx=5}\end{questionmultx}

Numerical Responses

Output

\begin{question}{SEssCalcET2 12-3-016} Let $E$ be the solid that is bounded on the outside by the sphere $x^2 + y^2 + z^2 = 16$ and inside by the cylinder $x^2 + y^2 = 4$. Find the volume of $E$. \AMCOpen{lines=3,dots=false}{% \wrongchoice[0]{0}\scoring{0}% \wrongchoice[1]{1}\scoring{1}% \wrongchoice[2]{2}\scoring{2}% \wrongchoice[3]{3}\scoring{3}% \correctchoice[4]{4}\scoring{4}}\end{question}

Free Response Qs

Output

\begin{minipage}[b]{0.3\textwidth} \AMCcode{NNumber}{8}%\end{minipage}\hspace*{1em}\begin{minipage}[b]{0.5\textwidth}\raggedright MATH-UA 123.007 Calculus III\\ [...] Code the eight digits of your N number to the left, and write your name below. \vspace{3ex} \hfill\namefield{\fbox{ \begin{minipage}{.9\linewidth} Name: \vspace*{1cm} \end{minipage} }}\hfill\vspace{0.1ex}\end{minipage}\end{document}

ID Blocks

Output

Problem Groups•Select k from a

group of n

•shuffle items in a group

•pipe dream: keep questions in a flat file or database

Workflows

Method #1 (quizzes)Prof AMCStudents LMS

LaTeX

PDFgive quiz

take quiz

PDF

Scores

Scan

write

Roster CSV

Gradebook CSV Gradebook CSV

Annotated PDF

Grade

AMC Workflow #1Papers LMS

Dream Workflow #1Papers LMS

Method #1 (quizzes)Prof AMCStudents LMS

LaTeX

PDFgive quiz

take quiz

PDF

Scores

Scan

write

Roster CSV

Gradebook CSV Gradebook CSV

Annotated PDF

Grade

Method #2 (exams) ( (exams)quizzes)

Prof AMCStudents LMS

take quizPDF

Scores

Scanwrite

Problem PDFsGrade

on iPadProblem PDFs

PDFExpert Grading

•PDF annotator•Syncs with cloud

storage apps•$10 but totally

worth it😕😎 😱😆😬

Emoji

Stamps

More stamps

Image Stamps

AMC Annotation

Report

Annotation

Annotation

Annotation

Annotation

Annotation

Cool trick: returning graded “papers”

•AMC annotation method marks MCQs and creates a new PDF

•Email feature sends them back•Sakai “dropbox” (not Dropbox) has

a DAV interface•Script to batch PUT PDFs into

Sakai!

What is DAV?• Distributed

Authoring and Versioning

• Treat website “directory” like a bona fide filesystem share.

• Extension of HTTP

AMC-sakai-return.pl• Command line, interfaces with AMC data

model and app config.• Attach files (e.g., solutions, summary

stats)• Encrypt so they can't alter the file• Dry run feature• Upload via curl library 

$ curl --netrc -T file.pdf https://newclasses.nyu.edu/dav/group-user/<site-id>/<username>/<fullname>_<examname>.pdf

Money Line

Another cool trick: saving files to Evernote

What is Evernote?•Instead of folders and files,

notebooks and notes (and tags)•notes can contain rich text and file

attachments•a note is only in one notebook but

can have arbitrarily many tags•Very searchable

Archiving Script

•Same command line interface•Create note with PDF, tagged by

course, term, student, etc.•Upload via Evernote SDK

Advantages• Paper when you need it, not when you don't

• No more office/book bag clutter

• Quick turnaround

• Reduces cheating by good eyesight

• Eliminates cheating by altering the graded sheet

• Regrades and post-exam conferences are easy

Disadvantages•Command line is not for everyone•Printing individual exams can take

time•Scanning multipage exams on a

ScanSnap takes time•Files can be big•Students who write in the margins!

Future directions

Next steps

•Scale up•Tighter integration with AMC GUI•Item analysis

Goodnight Seattle

CreditsFrancois Roche

Mark Atwood

Torben Hansen

CreditsPaul Van der Werf

Andi Narvaez

simasima.info unknown

CreditsWiki episteme

Jody Cloutier

Wonderlane

CreditsKevin Kerrick

Dan McKay

Adelle Charles

Another cool trick: saving files to Evernote

Seattle Municipal Archives

Credits

Ethan Kan

Clipart from OpenClipArt.org