View
913
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
CSUF Pilot to Improve Student Writing with
ETS Criterion®, Spring 2010
As compiled by Institutional Research & Assessment and Planning, CSALT, Jennifer Ivie, Henry Delcore,
and ISWI Assessment Team
Sponsored by CSUF ISWI Pilot Program and CSALT Presentation Developed by Professor Kim Morin, December 13,2010
1Copyright © December 2010
Fresno State Student, Stephanie Canales, Discusses ETS Criterion
• http://www.youtube.com/watch?v=pP6feJotLVM
Copyright © December 2010 2
How This All Began
• I first sought a computer essay scoring program out of desperation!
3Copyright © December 2010
I Believed In Giving Writing Assignments To Students…
“Writing helps with - • Making personal
connections• Problem solving• Reasoning• Communication
skills.”
4Copyright © December 2010
But Then I Kept Asking…
“Where Is the Time to
Grade All of Those
Essays?”
5Copyright © December 2010
So I Discovered…
Computer Essay Scoring ProgramsI narrowed my search to three of the biggest programs available at the college level… and tried out ETS Criterion®!
6Copyright © December 2010
How It Works
• Students submit essays electronically• Computer program scores the students’
work based on sample essays previously scored by human raters
• “e-rater” scoring system (analysis of more general writing skills)• “Hybrid Feature Technology” combines syntactic variety and
discourse with analysis of semantic structure
7Copyright © December 2010
Why I Chose To Pilot ETS Criterion
• Combines Diagnostic with Holistic feedback • Flexibility
– Instructor can create a topic– Instructor can easily provide personalized feedback
• Simple Registration Process – Students are able to purchase Access cards through Bookstore
• No need for preliminary essay submissions• Reasonable Cost
– $13.00 per student per semester– Same fee regardless of number of classes
8Copyright © December 2010
Positive Aspects
• Easy to use
• Immediate feedback
• Consistency in scoring
• Errors are highlighted but not corrected
• First and last submission are available for comparison
9Copyright © December 2010
Positive Aspects
• Students often do a number of re-writes trying to improve their score
• Students tend to get frustrated with the computer - not the instructor
• Students discover that good writing means more than correcting grammatical errors!
10Copyright © December 2010
Take A Tour
ETS CRITERION Online Tour
http://www.ets.org/Media/Products/Criterion/tour2/critloader.html
11Copyright © December 2010
Campus-Wide InitiativeBy The Numbers...
Over the course of 2 semestersSpring 2010• 134 Instructors used Criterion® • 197 Classes • 5,073 students Fall 2010• 68 Instructors used Criterion® • 173 Classes • 3,756 students
12Copyright © December 2010
CSUF DemographicsDiverse student population with a significant number of underprepared and first generation students, including many involved in the agricultural and farming economies in the region.
13Copyright © December 2010
Assessments for Spring 2010..
– Analyzed trends from Automatically Generated Data within Criterion®
– Collected Student and Faculty Surveys (Analysis by Jennifer Ivie)
– Faculty Learning Community Focus Groups provided qualitative feedback
14Copyright © December 2010
Assessments for Spring 2010..
4. College Learning Assessment (CLA) comparison for students with & without Criterion® (April, 2010)
5. Faculty Scoring Panel compared 1st
and last papers submitted by students on Criterion®
15Copyright © December 2010
Assessments for Spring 2011..1. Studies with controlled variables –
same course, same instructor, randomly selected group uses Criterion®, control group does not
2. Continue collecting back end data from Criterion®
3. Faculty and student surveys, focus groups
4. CLA5. Other measures?
16Copyright © December 2010
Automatically-Generated Data• Students who made two submissions to ETS
Criterion had higher average holistic scores than students who only made one submission.
• Students with lower initial holistic scores were able to raise their scores, with subsequent submissions, to the same level as those whose initial holistic scores were higher.
17Copyright © December 2010
Collegiate Learning Assessment– A significant difference was found
between Criterion users and Criterion non-users on the CLA performance task.
– No significant differences were found between the two groups on any of the analytic writing task scores.
18Copyright © December 2010
Faculty Scoring Panel• Overall, student writing improved in classes
that used ETS Criterion.
• The greatest improvements from first to last submission were made in classes in which the instructors engaged students in substantial discussions of writing and how to use Criterion.
19Copyright © December 2010
Student Survey HighlightsStudents felt that Criterion
– …helped with writing mechanics.– ESL students were more likely than English
as a First Language students to respond that Criterion® helped them improve their writing.
– ESL students often preferred Criterion feedback to instructor feedback.
20Copyright © December 2010
ESL Student ResponseESL students submitted their documents more times than non-ESL students, and were more likely to agree with these statements: (Pearson r’s, all at p<.05 or less):
- I like the kind of feedback Criterion gives. - I spent more time improving my writing with Criterion. - I was able to apply Criterion’s feedback to other writing assignments. - I wish I could use Criterion for other classes.- I feel like the feedback from Criterion has made me a better writer.
21Copyright © December 2010
Overall Student Response - Positive
What most students liked about Criterion®: • Available on-line 24 hours a day • It checked grammar and spelling • It allowed them to correct mechanical and grammatical errors • It gave immediate feedback. • The website was easy to navigate. • The grading was fair
22Copyright © December 2010
Student Comments• I like the instant feedback. I also like that it goes
into depth about why you didn't get the max score or why you did well. It is also easy to use just copy and paste!
• I love the fact that I can revise my work for a better score. It is very helpful and I can see my growth as a writer even from this second essay.
• When I submit my paper, I look forward to seeing my score. Usually I score a 5 but I push myself to go and correct my errors until I score 6. I really like this program...
• I like the instant feedback. I also like that it goes into depth about why you didn't get the max score or why you did well. It is also easy to use just copy and paste!
• I love the fact that I can revise my work for a better score. It is very helpful and I can see my growth as a writer even from this second essay.
• When I submit my paper, I look forward to seeing my score. Usually I score a 5 but I push myself to go and correct my errors until I score 6. I really like this program...
23Copyright © December 2010
Student Sample
• First Essay - Went from Level 4 - 6
• 22 Attempts
Student Comments- Summaries of Challenges
•Some students felt –They were already strong writers, so they felt like they did not need the program.–The comments should be more specific–Frustrated that Criterion identified technical terms or citations as errors.
25Copyright © December 2010
Overall Faculty Response (n=66 out of 124)
•Creating assignments in Criterion® was easy.•Students found Criterion® useful.•Criterion® helped their students become better writers.•Grading was easier after papers had been submitted to Criterion®.•Students spent more time improving their writing assignments when using Criterion®.
26Copyright © December 2010
Faculty Response
However, many faculty also felt that:Criterion® didn’t do as much as they hoped it would.They would be able to use the program more effectively with more experience and training
In fact, over 100 faculty members continued with the pilot program for Round 2 in Fall 2010.
27Copyright © December 2010
Effective Uses of Criterion®
Data from Spring 2010 indicates the following practices as more effective:1. Multiple assignments 2. Multiple revisions over time3. Students correct “fixable errors” as identified by
Criterion®4. Early orientation for students so they practice
using the program and gain a better understanding of how it works
5. Students are given ample time between first and final submissions to make revisions
28Copyright © December 2010
Effective Uses of Criterion® (cont) • Instructor uses the tools in Criterion® to give
additional feedback. • Students are motivated to use the Criterion®
program as part of the revision process• Criterion® is used as an editing tool to help students
think critically about their writing• Students and faculty understand that the computer
program will make mistakes; it is one part of the editing process
• Scoring is set at the appropriate level (College- 1st year or College 2nd year)
29Copyright © December 2010
Less Effective Uses of Criterion®
oA single paper at the end of the semester
oSingle submissions – no revision o Instructor gives little or no training or
feedback regarding Criterion® oTechnical writing, business formats,
field specific terminologyoUsing ETS Criterion as "Extra Credit"
30Copyright © December 2010
ConclusionsWe are still evaluating the effectiveness of Criterion®, but from what we’ve seen so far, under certain circumstances it appears to be useful in improving student writing. The program appears to be most effective with multiple assignments that require multiple submissions over time. Faculty need to be made aware of key features of Criterion® and what they do, then carefully select which features to use. To use the program effectively, students must understand how to access and interpret feedback.
31Copyright © December 2010
Fresno State Student, Katy Michelle Lewis,
Discusses ETS Criterion
• http://www.youtube.com/watch?v=GTQLF7AujSQ
Copyright © December 2010 32
Fresno State Student, Nick Mullins, Discusses ETS Criterion
• http://www.youtube.com/watch?v=TRbYAVqC6tg
Copyright © December 2010 33
ISWI Is Broadly Endorsed & Supported By:
Writing Competency Subcommittee (subcommittee of Senate’s GE Committee)Professional Development Subcommittee (subcommittee of Senate’s Personnel Committee)Chair of the General Education CommitteeOffice of Undergraduate StudiesDivision of Graduate StudiesInstitutional Research & Assessment Planning Director leading faculty and ISWI Assessment TeamProvost’s Office as implemented by Associate Provost Ellen Junn through the Center for the Scholarly Advancement of Learning and Teaching (CSALT)
34Copyright © December 2010
Acknowledgements
Bill Covino, ProvostEllen Junn, Associate ProvostChristina Leimer, Director, IRAP ISWI Assessment Team:Kim Morin, Faculty, Theatre ArtsJohn Beynon, Faculty, English Henry Delcore, Faculty, AnthropologyBradley Hufft, Faculty, MusicJennifer Ivie, Faculty, PsychologyEileen Walsh, Faculty, HistoryAsao Inoue, Faculty, EnglishJohn Farrell, Faculty, HistoryDimitri Rogulkin, IRAP, StaffDiane Robinson, ISWI, CSALT, Staff
35Copyright © December 2010