A National Online Essay Evaluation Service: Valid, Reliable

A National Online Essay A National Online Essay Evaluation Service:Evaluation Service:Valid, Reliable, & Cost Valid, Reliable, & Cost Effective Writing AssessmentEffective Writing Assessment

Leslie C. Perelman <Leslie C. Perelman <[email protected]@mit.edu>>

Program in Writing and Humanistic StudiesProgram in Writing and Humanistic Studies

Massachusetts Institute of TechnologyMassachusetts Institute of Technology

iMOATiMOATEnhancing the best way of assessing Enhancing the best way of assessing writingwritingEnthusiastic reception from both Enthusiastic reception from both students & facultystudents & facultyCost effectiveCost effectiveReliable scoring and deliveryReliable scoring and delivery

MotivationsMotivations

Existing modes of placement Existing modes of placement inadequateinadequate

Standardized testsStandardized testsTimed impromptuTimed impromptuPortfolioPortfolio

Students write on computers not Students write on computers not paperpaper

The Common PracticeThe Common Practice

Student walks into large Student walks into large room and writes with room and writes with pen and paper on small pen and paper on small deskdeskStudent reads, at most, a Student reads, at most, a two or three paragraph two or three paragraph text for each essaytext for each essay

Student writes one or Student writes one or two essays on specific two essays on specific questionsquestions

Handwritten essays are Handwritten essays are scored (usually by two scored (usually by two readers)readers)

Score is sent on paper to Score is sent on paper to student and / or advisorstudent and / or advisor

Student works at homeStudent works at home

Student views readings of Student views readings of any length onlineany length onlineStudent accesses Student accesses questions onlinequestions onlineStudent writes one or Student writes one or two essays at home two essays at home within a specified period within a specified period using a computer.using a computer.

Essays are scored on Essays are scored on either printed copy or on either printed copy or on screen by readers in real screen by readers in real or virtual grading or virtual grading sessionssessionsStudent and advisor Student and advisor access both score and access both score and text of essays.text of essays.

iMOATiMOAT

A Better Way to Assess A Better Way to Assess WritingWriting

Developed by Experts in Writing Developed by Experts in Writing Assessment at Five Diverse Assessment at Five Diverse UniversitiesUniversities

With iMOATWith iMOATStudents WriteStudents Write

This WayThis Way Not This WayNot This Way

Intelligent ReadingsIntelligent Readings

Give Students Time Give Students Time to Think & Writeto Think & Write

Assess the Writing ProcessAssess the Writing Process

More Time (and savings) More Time (and savings) Allowing for Two EssaysAllowing for Two Essays

““if you want to give real weight to if you want to give real weight to [student] essays, you must secure two test [student] essays, you must secure two test essays from each student with some essays from each student with some separation in time as well as in topic, and separation in time as well as in topic, and the least possible separation in time as the least possible separation in time as well as in topic, and the least possible well as in topic, and the least possible separation in time is to have one test essay separation in time is to have one test essay written in the morning and the other in written in the morning and the other in afternoon of the same day.afternoon of the same day.””

---- Paul Paul DiederichDiederich of ETSof ETSThe founder of modern writing The founder of modern writing assessmentassessment

iMOATiMOAT FeaturesFeaturesAutomates all components of Automates all components of best practices in writing best practices in writing assessmentassessment

Focusing on writing processFocusing on writing processAllowing students modern toolsAllowing students modern toolsOnline holistic scoring sessions with Online holistic scoring sessions with real time quality control measuresreal time quality control measures

Digital Archives of EssaysDigital Archives of Essays

For program assessmentFor program assessment

For portfoliosFor portfolios

For institutional researchFor institutional research

Two Perpetual QuestionsTwo Perpetual Questions

What about cheating?What about cheating?

Will all students have access?Will all students have access?

Reponses & ResultsReponses & Results

Feedback from Feedback from StudentsStudents

““I am from a small town in Missouri. My I am from a small town in Missouri. My writing essay was easy for me because I writing essay was easy for me because I enjoy thinking about these things, such enjoy thinking about these things, such as philosophy and the future. I went as philosophy and the future. I went out to my cabin and I just sat around out to my cabin and I just sat around the campfire the whole time, just the campfire the whole time, just enjoying the silence and writing enjoying the silence and writing whatever came into my head. The next whatever came into my head. The next day I typed it up.day I typed it up.”” (DePaul)(DePaul)

Feedback from StudentsFeedback from Students

““Taking the online writing tests, I Taking the online writing tests, I found, gave me the opportunity to found, gave me the opportunity to leave the information when I needed leave the information when I needed time to think and come back to it time to think and come back to it when I was ready. The freedom also when I was ready. The freedom also allowed me to revise my work and allowed me to revise my work and not feel though my work was rushed not feel though my work was rushed or inhibited in any way.or inhibited in any way.”” (DePaul) (DePaul)

Benefits for Writing FacultyBenefits for Writing FacultyAllows essays and questions to reflect the Allows essays and questions to reflect the universityuniversity’’s and programs and program’’s specific s specific educational needs, priorities, and goalseducational needs, priorities, and goalsMaintains control of all aspects of Maintains control of all aspects of evaluation process within the university evaluation process within the university and writing programand writing programOnline grading sessions maintain Online grading sessions maintain ““staff staff consensus building consensus building ”” benefit while benefit while potentially reducing costspotentially reducing costsReflects and measures the writing process Reflects and measures the writing process used and valued in collegeused and valued in collegeEncourages sharing of Encourages sharing of ““best practicesbest practices””among schools using the systemamong schools using the system

ReliabilityReliability

Reliability of AssessmentReliability of Assessment

““Faculty involved in English Faculty involved in English placement strongly feel that the placement strongly feel that the Online represents much better the Online represents much better the kind of writing students will do in kind of writing students will do in our courses, and, so far, it is our courses, and, so far, it is working equally well for a wide working equally well for a wide range of students from honors to range of students from honors to very low developmental.very low developmental.”” (U. of (U. of Cincinnati)Cincinnati)

Reliability of Assessment at MIT:Reliability of Assessment at MIT:Fall 2002 Survey of Instructors Fall 2002 Survey of Instructors Teaching Introductory Writing ClassesTeaching Introductory Writing Classes

1st Semester Students in Introductory Writing

Classes N = 147

1% 12%

34%53%

1 2 3 4

1=Should not 1=Should not be in classbe in class

4=Definitely should4=Definitely shouldBe in classBe in class

Reliability of Assessment at MIT:Reliability of Assessment at MIT:Fall 2002 Survey of Faculty TeachingFall 2002 Survey of Faculty TeachingWritingWriting--Intensive ClassesIntensive Classes

1st Semester Students in Writing-Intensive Humanities Classes N=125

0%10%20%30%40%50%60%70%80%90%

100%

1 2 3 4Faculty Responses 1=Should not have been in class; 4 = appropriate writing

skills for class

Pass

Subject Req

Reliability of Assessment: Reduction in Reliability of Assessment: Reduction in Complaints and Appeals at MIT 1992Complaints and Appeals at MIT 1992--20032003

Appeals & Complaints by Students about Scores on MIT Placement Test

0

10

20

30

40

50

60

70

80

90

1992

1993

1995

1996

1997

1998

1999

2000

2001

2002

2003

Year

# of

Stu

dent

s

Architectural OverviewArchitectural Overview

Web Server MS Windows 2000 / IIS

MS ASP.NETC#

DatabaseMS SQL/2000

InternetInternet

Administrator

Student

Reader

iMOAT HighiMOAT High--Level WorkflowLevel Workflow

Create Class

Administrator Student Reader

Create Test

Upload Students

View Readings & Questions

Write & Submit Essay

[Write Comments]

Assign Readers

Finalize Scores

Publish Results

Reporting

Register for Exam

View Results [& Comments]

Read & Score Essays


Create Class


Admin Admin -- Create ClassCreate Class

Admin Admin -- Create ClassCreate Class

Admin Admin -- Create Test GroupCreate Test Group

Admin Admin -- Create Test GroupCreate Test Group


Create Class


Create Test

Admin Admin -- Create Test Create Test -- DefineDefine

Admin Admin -- Create Test Create Test –– Set DatesSet Dates

Admin Admin -- Create Test Create Test –– Enter ReadingsEnter Readings



Admin Admin -- Create Test Create Test –– Enter QuestionsEnter Questions

TestWrite HighTestWrite High--Level WorkflowLevel Workflow

Create Class


Create Test

Upload Students

Admin Admin -- Upload StudentsUpload Students





Create Class


Create Test

Upload Students Register for Exam

Student Student -- LoginLogin

Student Student -- RegistrationRegistration





Create Class


Create Test

Upload Students


Register for Exam

Student Student –– View ReadingsView Readings

Student Student –– View ReadingsView Readings

Student Student –– View AssignmentView Assignment

Student Student –– View AssignmentView Assignment


Create Class


Create Test

Upload Students



Register for Exam

Student Student –– Submit EssaySubmit Essay

Student Student –– Submit EssaySubmit Essay

Student Student -- Confirmation EmailConfirmation Email


Create Class


Create Test

Upload Students



Assign Readers

Register for Exam

Admin Admin –– Assign ReadersAssign Readers

Admin Admin –– Assign ReadersAssign Readers


Create Class


Create Test

Upload Students



Assign Readers

Register for Exam

Read & Score Essays

Reader Reader –– Get Essay to ReadGet Essay to Read

Reader Reader –– Get Essay to ReadGet Essay to Read

Reader Reader –– Read EssayRead Essay

Reader Reader –– Grade EssayGrade Essay

Reader Reader –– Second Reader, Second ReadSecond Reader, Second Read


Create Class


Create Test

Upload Students



Assign Readers

Finalize Scores

Register for Exam

Read & Score Essays

Admin Admin –– Resolve Split ScoreResolve Split Score



Admin Admin –– RealReal--time Reader Reportingtime Reader Reporting





Create Class


Create Test

Upload Students



Assign Readers

Finalize Scores

Publish Results

Register for Exam

Read & Score Essays

Admin Admin –– Define Result CategoriesDefine Result Categories



Admin Admin –– Set Test CutsSet Test Cuts

Admin Admin –– Send to ResultsSend to Results

Admin Admin –– Finalize and PublishFinalize and Publish


Create Class


Create Test

Upload Students



Assign Readers

Finalize Scores

Publish Results

Register for Exam

View Results

Read & Score Essays

Student Student –– View Results (not yet available)View Results (not yet available)

Student Student –– View ResultsView Results




Create Class


Create Test

Upload Students



Assign Readers

Finalize Scores

Publish Results

Reporting

Register for Exam

View Results

Read & Score Essays

Reporting Reporting –– Download ResultsDownload Results




Reporting Reporting –– Download Results (Excel)Download Results (Excel)


Create Class


Create Test

Upload Students



Assign Readers

Finalize Scores

Publish Results

Reporting

Register for Exam

View Results

Read & Score Essays

SavingsSavings

Greater Efficiency in Scheduling Greater Efficiency in Scheduling and Staffing Classesand Staffing Classes

Saving Money: Reliability & Timeliness Saving Money: Reliability & Timeliness of Assessment Reduce Number of of Assessment Reduce Number of Sections NeededSections Needed

Accuracy of Online Evaluation = Accuracy of Online Evaluation = Fewer SectionsFewer Sections

LSU (4,000 1LSU (4,000 1stst Year Students) is Year Students) is scheduling 4 fewer sections per year scheduling 4 fewer sections per year because of fewer students being because of fewer students being misplacedmisplaced

Save Money: Typed texts (even if they Save Money: Typed texts (even if they are longer) cost less to gradeare longer) cost less to grade

At MIT At MIT (@$25 / hr.)(@$25 / hr.)

the average the average handwritten text handwritten text costs $7.15 to be costs $7.15 to be scored by two scored by two readersreadersthe average typed the average typed text costs $5.40 text costs $5.40 to be scored by to be scored by two readerstwo readers

$0.00

$1.00

$2.00

$3.00

$4.00

$5.00

$6.00

$7.00

$8.00

Handwritten Typed

$7.15

$5.40

Save Money: Automatically Save Money: Automatically AccommodateAccommodateStudents with DisabilitiesStudents with Disabilities

Save Money: Save Money: Eliminate Data EntryEliminate Data Entry

Save Money: Eliminate Save Money: Eliminate PaperworkPaperwork

Save Money: Avoid Hotel Save Money: Avoid Hotel Expenses for ReadersExpenses for Readers

Save Money: Eliminate Save Money: Eliminate Travel ExpensesTravel Expenses

Save Money: Reduce Save Money: Reduce Expense of Feeding ReadersExpense of Feeding Readers

iMOATiMOAT

A more valid and reliable A more valid and reliable assessmentassessmentGreater faculty and student Greater faculty and student satisfactionsatisfactionSignificant reduction in costsSignificant reduction in costsProven and reliable platformProven and reliable platform

Documents

A National Online Essay Evaluation Service: Valid, Reliable