Calculus Courses’ Assessment Data › fulltext › EJ1149356.pdfCalculus courses and the data they produce. The online learning environment collects a lot of timestamped data about

(2017).Calculuscourses’assessmentdata.JournalofLearningAnalytics,4(2),12–21.http://dx.doi.org/10.18608/jla.2017.42.3

ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0)

12

Calculus Courses’ Assessment Data

MattiPaunaDepartmentofMathematicsandStatistics,

UniversityofHelsinki,[email protected]

ABSTRACT. In this paper we describe computer-aided assessment methods used in onlineCalculus courses and the data they produce. The online learning environment collects a lot oftimestampeddataabouteveryactionastudentmakes.Assessmentdatacanbeharnessedintouseasafeedback,predictor,andrecommendationfacilityforstudentsandinstructors.Wealsodescribe late professor Mika Seppälä’s seminal work at the University of Helsinki to developonlinematerialsandtoolsforlearningmathematicssince2001.HealsoutilizedthesemethodsinCalculus teachingatFloridaStateUniversity.Theopenonlinecourse“SingleVariableCalculus”washeldinHelsinkiin2004.ThisintensiveworkevolvedintoacompleteonlineEnglishCalculuscurriculum starting from the Fall 2013 and soon recognized as an alternative route for takingtraditional university Calculus courses in Helsinki. Automatic assessment systems ofmathematical competencies, such as STACK andWeBWorK, can take a student’s answer as amathematical object, e.g. a function or an equation, and check whether it satisfies therequirementssetforacorrectansweraswellasgiveimmediateandmeaningfulfeedback.Thatisapowerfultoolespeciallyforformativeassessment:logdatashowsthatmanystudentsprefertostartwithquizzesandwhennecessary,consultlecturingmaterials.

Keywords:Assessmentdata,automaticassessment,peerassessment,Calculus

1 INTRODUCTION

Work presented here is based on the seminal efforts by late professor Mika Seppälä (XambóDeschamps,Bass,BolañosEvia,Seiler,&Seppälä2006)whodevelopedthecorematerialsandmethodsused in creating an online calculus curriculum at the University of Helsinki. Seppälä also developedonline content and tools used in classroom teaching at the Florida StateUniversity. Seppälä also didpioneeringworkondigitalrepresentationofmathematicalknowledge,mostnotably intheOpenMathproject, whose results lead to the MathML encoding standard of mathematical formulas and theirsemantics. MathML is not only used to encode mathematical information in web pages but also inapplications,suchasMicrosoftOffice.

At the University of Helsinki, we started on 2001 to develop new methods for online teaching ofcalculus,especiallyforproducingpresentationmaterialstailoredforthewebandtechniquestoassessstudents’ mathematical competencies online. Themain goal at the timewas to produce fully onlinecalculuscourseswhichmaterializedin2004astheonlineSingleVariableCalculuscourse.Alreadythen,the theory and examples were presented as ten-minute talk videos focusing on a single concept,



13

technique,application,etc.HomeworkpracticewasimplementedthenwithMapleTAonlinequizzes.In2009, we moved to an automatic assessment system called STACK whose benefit is that it is fullyintegrated into theMoodlevirtual learningenvironment (Caprotti,Ojalainen,Pauna&Seppälä2013).Onlineassessmentisdescribedinmoredetaillaterinthispaper.

Soonwesawtheneedforacompletetrackofcalculuscoursescorrespondingtothecurrenttraditionalsetof calculus courses. This leadus todevelopcoursesCalculus I,Calculus II, andAdvancedCalculus,whichtogethercoverthetraditionalundergraduatecalculuscurriculumthusprovidinganotherpathforstudentstostudycalculusonline.

2 ONLINE ASSESSMENT FOR CALCULUS COMPETENCIES

Whendevelopingonlineassessments, it isworthconsideringthetypesofmathematicalcompetenciesthatare tobeassessed, forwhich therearemany taxonomies.Wehavechosenasabasis themodeldevelopedbyPointonandSangwin(2013)andextendedbyRämö,Oinonen,andWikberg(2015)bythecategory“Informationtransfer”whichrequiresthestudentstorepresenttheobjectsandinformationinmanyformsduringworkingwithanassignment.Thecategoriesinthistaxonomyareasfollows:

1.Factualrecall2.Carryoutaroutinecalculationoralgorithm3.Classifysomemathematicalobject4.Interpretsituationoranswer5.Proof,show,justify(generalargument)6.Extendaconcept7.Constructexample/instance8.Criticizeafallacy9.InformationtransferCurrently,ourCalculuscoursescontaintwotypesofdigitallymediatedassessments,whichwediscussnext.

2.1 Automatic Assessment

Online automatic assessment systems such as STACK (Sangwin, 2013) andWebWorK (Gage, currentissue)provideversatilewaysofpracticingwithcalculusproblems.WithSTACK,onecanaskquestionswhoseansweristypedinasamathematicalformula.This is,thus,superiortomultiplechoicequizzes,forexample.Themaindifferenceisthatthestudenthastoproducetheanswerratherthanchoose(bywhicheverstrategy)fromgivenpossibleanswers.Thesystemcangenerateproblemsfromatemplatewhereparameterschangeeachtimetheproblemisdeployed. It isabletoanalyzeastudent’sanswerand mathematically evaluate whether it satisfies the conditions required for a correct answer.Furthermore,itcanrecognize,throughcertainconditions,orpatterns,iftheanswerispartiallycorrectandprovidefeedbackaccordingly.



14

InCalculusIandIIcourses,STACKproblemsareorganizedinpracticequizzeswhichstudentscantakeasmany times as they see necessary to master the topic. As the problems contain complete examplesolutions, students actually learn by taking quizzes aswell.We can see from the log data thatwhenstudying online, some students prefer to start workingwith the quizzes and consulting theorywhenneededandotherstudentsgothroughthepresentationalmaterialsbeforetakingquizzes.Quizzesarealsousedasdiagnostictests,givingstudentsanunderstandingofwheretheystandatthebeginningofacourse, and as practice tests for exams. We see that continuous and meaningful feedback isindispensableforlearninginanonlinemathematicscourse.

Thetypesofcompetenciesthatcanbecurrentlyassessedbyautomaticquizzesaretypicallyatthelowerlevels of the taxonomy, i.e. from factual recall to classify somemathematical object. However, taskssuchasconstructexample/instancecanbeaskedwiththissystem.Figure1displaysanexample.

Figure1:Anexampleofanautomaticquizproblemwithanopenanswer

2.1.1 Automatic Assessment Data Theunderlyinglearningenvironment(Moodle)storesdetailedtimestampeddatafromeveryuseractionandprovidesconvenientviewsforstudentsandinstructors.Theanswer(s)toeachproblemarestored,asseeninFigure2.

Figure2:Logdatasavedfromananswerattempttoaquizproblem



15

Thistimestampeddataenablesinstructorstoseecommonmisconceptionsandtoadjusttheirteachingaccordingly.Largescaleitemanalysiswouldshowparticularlyusefulproblemsforlearningandalsohelpinimprovingtheproblembank.

Thedatafromquizzes includesdataforeachstudent includingtheirtimetakenandtotalscore.Manystudents take a quiz several times, usually until they get a perfect score, which can be seen ascontributingtowardstotheattributeofpersistencyinastudent’sprofile.Otherstudentattributesthatcan be extracted using these data include how late before the deadline of the quiz they finish thequizzes(procrastination)and,ofcourse,howmanyattemptstheyneedtogetthefullscore(mastery).

Figure3:Logdatafromastudent’sattemptinaquiz

ThepartofagradebookfromasinglequizandsinglestudentshowninFigure3tellsthatthisstudenthastakenthequiz fivetimesuntil thehighestpossiblegrade(6.00) is reached.Thestudent’sname intheleftmostcolumniscovered.Thisquizcontainedfiveproblemswhosescoresaremarkedinthefiveright most columns. The last problem has turned out to be themost difficult and an instructor canexamineeachofthestudent’sanswersbyclickingthescoreforeachattempt.Weseethatthestudenthas worked with the quiz over a course of two days. The last attempt took over two hours (fourthcolumnfromtheleft),significantlymorethanothers.Fromthedatathatthesystemgatherswecannotfindanexplanationforthat.Theproblemsmayhavetakenalotoftimeforthestudenttosolveortheymighthavehadabreakfromonlinework.Inanycase,thisstudenthassolvedaltogether25problemsonacertaintopicinCalculus.



16

3 PEER ASSESSMENT

Whileautomaticquizzesmainly focuson learningprocedural skills, suchasdifferentiation techniques,not all calculus competencies can (at themoment) be assessed automatically. Students also need tolearnhow topresent theirworkedout solutionand justify themethodsused in the solutionprocess.ThisisemphasizedintheAdvancedCalculusCourse,wherehomeworkproblemsaskstudentstoprovestatements.Weusepeer-assessedworkshopstoaccomplishthiscentralpartoftheonlinestudyactivity.Students are given a set of problems weekly where they have to present stepwise calculations orderivations of results. After that, students get to assess each other’s solutions with the help of anexamplesolutionsandassessmentguidelines.Wesee,andcoursefeedbackfromstudentssupportthisclaim,thatstudentsoftenlearnmoredeeplybyassessingotherstudents’work.Theyhavetostudytheexample solutionandother students’ solutions carefully tobeable to assess them. Students are alsorequiredtogiveconstructiveandcorrectivefeedbackwhenasolutionneedsimprovement.

Peerworkshopsprovideaversatilewaytoaskmanytypesofmathematicalquestionsasthesolutionsare assessed by humans togetherwith assessment instructions. Also technical limitations of enteringsolutionsarediminishedbecausemanystudentssubmitaphotographoftheirwrittenpapers.However,it isnotworthaskingsimpleproceduraltasks,asautomaticassessmentcanhandlethoseand it isnotalwaysinstructiveforthepeertoassesssimpleoneexpressionanswers.Furthermore,studentsreportthat they learn from followingother students’ reasoningand the stepsof theirworkandaugmentingtheirworkforacorrectsolutionwhennecessary.

3.1 Peer Assessment Data

Frompeerassessmentwegettheusualtimestampeddata,suchaswhenstudentssubmitwork,whenstudentsassessothers,andthescorestheygive.Forinstance,itisinterestingtoseethatthescoresthestudentsgivetoasubmissionaresurprisinglysimilar.Thatisimportantforthestudentstofeelthattheassessment is fair even though it is not done by the instructor, who has designed the assessmentinstructionsandoverseestheprocess.

Students’ submissions aremostly digital images of their handwritingwithmathematical formulas liketheexampleshowninFigure4.



17

Figure4:Asolutiontoworkshopproblemsubmittedbyastudent

Using these imagesmakes it rather difficult to automatically extract information from the solutions.There are a small number of students who submit their work as pdf documents written by wordprocessorsorevenLaTeX.(Forcompatibilityreasonsbetweenthedifferentcomputersystemsstudentsuse,pdf istheonlyacceptabledocumenttype.)However,thefeedbackstudentsgiveis inputinatextareaintheassessmentsheet.AnexampleisshowninFigure5.

Figure5:Assessmentfeedbackandscoregivenbyapeer

This textual data opens many possibilities for textual analysis methods, for example whether thefeedbackispurelyfactualortechnical,doesitcontainnegative/positiveexpressions,doesitaddresstheotherstudentdirectlywithapronounortalkinthepassivemode,etc.Resultsoftheseanalysescanbecombined with the numerical data from scores or time taken to write the assessments. There is ofcourseaninherentunderlyingproblemofmeasuringthe“timetakenonatask”sinceforinformationweonlyhave the timestampof the startingof theprocess, timestampsof theactions inside the task (or



18

timestamps of actions outside the taskwhile student has not ended the task) and the timestamp offinishing the task. A student can very well take breaks, leave a task open for the next day, etc, andthereforethemeasurementoftimetakenonataskhassomenoise.

4 CALCULUS COURSE DATA

Combiningalloftheabovedatafromassessmentswithotherclickstreamdataenablesustofindstudypaths, i.e. sequences of actions taken by a student while working in an online Calculus course. Ourcoursesaremodelledwiththeparadigm,whosefocalpointsareworkshops,showninFigure6.

Figure6:Calculuscoursearchitecturesupportingindividualstudentlearningpaths

ThemodelshowninFigure6allowsastudenttofollowtheirpreferencesforwhichordertostudytheresourcesandtasks.Forinstance,aseeminglytraditionalwaywouldbetostudythetheory,thenlookatexamplesandapplications,takeaquiztochecklearning,andfinallysubmittheworkshopproblems.Onthe other hand,we see thatmany students first start taking quizzes, perhaps to checkwhether theyalreadymasterthesubject(e.g.frompreviousstudies)andconsultingwrittenmaterialswhenneeded.Alltheactivitiesareaimedatsupportingstudentstosolvethemoreinvolvedworkshopproblems.

Alltheseloggedactivitiesenableustobuildstudypaths,i.e.aclickstreamdataofanindividualstudentoftheiractionswhileworkingonanonlinecourse.Gradedactivitiesdefinecontrolpoints inthepath:weseewhatkindofmaterialsastudentusedbeforetakingaquizorworkshopandhowsuccessfultheywereatthetask.Thisgivesusamethodofassessingthequalityandrelevanceofmaterials.Butmostimportantly,combinedwithdataofthestudentprofilewecouldseewhetherthatpathwaseffectiveforthat type of student. By analysing all this data and extracting models using machine learning ortopologicalmethods,itwouldbepossibletobuildasystemthatgivesfeedbackandrecommendationstoastudentastowhichturnsmakeintheirstudypath.



19

5 CALCULUS COURSES AT THE UNIVERSITY OF HELSINKI

Finally,wedescribeteachingofonlinecalculuscoursesattheUniversityofHelsinki,astheauthorhasaccesstothatdata.OnlineCalculusisofferedasanalternativepathtothemoretraditionalandpopularlecturedcourses.AsthecoursesareofferedinEnglishtheyarepopularamongexchangestudentsande.g. students who have limited access to the campus. We run Calculus I, Calculus II, and AdvancedCalculuscoursesthattogethercorrespondtothetraditionalCalculuscurriculum.TherotationissothatCalculusIandAdvancedCalculusarerunintheFallandCalculusIIintheSpringallowingthestudenttotakethesecoursesinthreeconsecutivesemesters.Thefirstfullrotation(AdvancedCalculuswasaddednext Fall) started Fall 2012. As of end of Fall 2015 altogether 390 student registrations have been inthesecourses.110 studentshave taken themidtermexamofanyof thesecourses.82 studentshavepassedacourse.WelistsomedescriptivestatisticsofvarioustypesofscoresinthesecoursesinTable1:

Table1:DescriptivestatisticsofscoresfordifferenttypesofassessmentsinCalculusCourses N Min Max Mean SD

CourseGrade 82 0.00 5.00 3.32 1.34Exam1 82 7.00 24.00 19.52 4.12ExamsCTotal 80 18.00 47.00 34.60 7.67QuizCTotal 82 0.00 100.00 79.73 18.84WorkshopCTotal 82 9.36 96.03 65.12 18.56

ThemaximumforcoursegradeinHelsinkiisa5andastudentmustearn75percentofthetotalpointsto achieve it. For the lowest passing score, a 1, a studentmust earn 50 percent. Themidterm exam(Exam1)andthefinalexameachgivemaximumof24pointstotalling48acrossbothexams(ExamsCTotal),amaximumscorethatnoneofthe80studentsgot.Here,quizzesaretheautomaticassessmentpartofthecourseandworkshopsarethepeerassessmentpart.

ItisinterestingtoaskwhichtypeofassessmentactivitymightbemostbeneficialforlearningCalculus.Weunderstandthatnoconclusionsfromthisdatacannotbeyetmadeandmoredataisneeded.SomeinitialdatarelatingperformanceonquizzesandworkshopstoexamscoresarepicturedinFigure7.



20

Figure7:ScatterplotsforcorrelatingExamscoreswithQuizscores(left)andworkshopscores(right).

Correlation coefficients show that there is no correlationbetweenquiz andexam scores (r = .01,p =.95). This may be because a quiz can be retried until full points are achieved. There is also a non-significantrelationbetweenworkshopsandexams(r=.19,p=.10),butthelackofstatisticalsignificanceis likely due to the small sample size andwithmore data this relationmay be important. There is asignificantcorrelationbetweenquizandworkshopscores(r=.43,p<.001),likelybecausequizzeswereplannedtoprepareforsolvingworkshopproblems.

6 CONCLUSION

Inthispaper,wehavedescribedonlineCalculuscoursesandcomputer-aidedassessmentsusedinthem,namely automatic assessmentwith STACKquizzes andpeer reviewedassessment. These assessmentscanbealsothoughtofashomeworkorpractice. In the first type,studentsgetautomatic, immediate,andmeaningful feedback from the computer, and in the second type, students get to see authenticsolutionsfromotherstudentsandgiveandreceiveconstructivefeedbacktowardsabettersolutiontoamathproblem.

Wehavestartedan initialanalysisofvarioustypesofassessmentandcourseactivitydatathatcanbegatheredfromtheonlinelearningenvironment.Studyingthesedatashouldenableustofindbestwaysofusinge-assessmentstosupportlearninginonlinecourses.

REFERENCES

Caprotti,O.,Ojalainen, J., Pauna,M.,&Seppälä,M. (2013).WEPSPeer andAutomaticAssessment inOnline Math Courses. Electronic Proc. 21st Int. Conf. Technology in Collegiate Mathematics.Retrievedfromhttp://archives.math.utk.edu/ICTCM/i/21/S026.html

Gage, M. (2017). Methods of interoperability: Moodle and WeBWorK. Special Section: Shape ofEducational Data Proceedings. Journal of Learning Analytics, 4(2), 22–35.http://dx.doi.org/10.18608/jla.2017.42.4



21

Pointon,A.,&Sangwin,C.J.(2003).Ananalysisofundergraduatecorematerialinthelightofhand-heldcomputer algebra systems. International Journal of Mathematical Education in Science andTechnology,34(5),671–686.http://dx.doi.org/10.1080/0020739031000148930

Rämö, J., Oinonen, L., & Vikberg, T. (2015). Extreme Apprenticeship – Emphasising ConceptualUnderstandinginUndergraduateMathematics.InK.Krainer&N.Vondrová(Eds.),ProceedingsoftheNinthCongressoftheEuropeanSocietyforResearchinMathematicsEducation(pp.2242–2248).Prague:CharlesUniversityinPrague,FacultyofEducationandERME.

Sangwin,C.(2013).ComputerAidedAssessmentofMathematics.OxfordUniversityPress.Xambó Deschamps, S., Bass, H., Bolaños Evia, G., Seiler, R., & Seppälä, M. (2006) e-Learning

Mathematics.Proc.Int.Conf.ofMathematicians.Madrid,Spain:ICM.

Documents

Calculus Courses’ Assessment Data › fulltext › EJ1149356.pdfCalculus courses and the data they produce. The online learning environment collects a lot of timestamped data about