Transforming exams across Australia platform for e exams ... · Transforming exams across Australia: Processes and platform for e-exams in high stakes, supervised environments Final

TransformingexamsacrossAustralia:Processesandplatformfore-examsinhighstakes,supervisedenvironments

Finalreport 2019

Lead institution: MonashUniversity

Partner institutions: CentralQueenslandUniversity,EdithCowanUniversity,MacquarieUniversity,RMITUniversity,TheUniversityofQueensland,TheAustralianNationalUniversity,UniversityofNewSouthWalesatADFA,UniversityofSouthAustralia,UniversityofTasmania

Project leader: DrMathewHillier

Teammembers: AssociateProfessorMatthewBower,AssociateProfessorMichaelCowling,DrAndrewFluck,DrRuthGeer,DrScottGrant,DrBethHarris,MrKennethHowah,DrDavidMeacheam,MrDominicMcGrath,DrJeremyPagram,MrBruceWhite

Projectmanager: MsVilmaSimbag

SoftwareDeveloper: MrMartinColeman

TransformingExams.com

SupportfortheproductionofthisreporthasbeenprovidedbytheAustralianGovernmentDepartmentofEducation.TheviewsexpressedinthisreportdonotnecessarilyreflecttheviewsoftheAustralianGovernmentDepartmentofEducation.

WiththeexceptionoftheCommonwealthCoatofArms,andwhereotherwisenoted,allmaterialpresentedinthisdocumentisprovidedunderCreativeCommonsAttribution-ShareAlike4.0InternationalLicensecreativecommons/4.0/license.

ThedetailsoftherelevantlicenceconditionsareavailableontheCreativeCommonswebsite(accessibleusingthelinksprovided)asisthefulllegalcodefortheCreativeCommonsAttribution-ShareAlike4.0InternationalLicensecreativecommons/4.0/legalcode.

Requestsandenquiriesconcerningtheserightsshouldbeaddressedto:HigherEducationProgramManagementGovernanceQualityandAccessBranchHigherEducationGroupAustralianGovernmentDepartmentofEducationGPOBox9880LocationcodeC50MA7CANBERRAACT2601<[email protected]>

2019ISBN978-1-76051-756-4[PDF]ISBN978-1-76051-757-1[DOCX]ISBN978-1-76051-758-8[PRINT]

TransformingexamsacrossAustralia | iii

AcknowledgementsTheTransformingExamsAcrossAustraliaprojectteamwouldliketogivespecialthankstothestudentsandeducatorsfromourpartneruniversitieswhoparticipatedintriallinge-Examsintheircourses.EducatorswhohostedoneormoretrialssincetheSeedProject2014include:AlistairCampbell(ECU),AmyHubbell(UQ),AndrewFluck(UTAS),AndrewGilbert(UNSWADFA),AroshaWeerakoon(UQ),BruceWhite(UniSA),CharanjitKaur(Monash),DavidBooth(UQ),JeremyPagram(ECU),KatrinaWilliams(UQ),KennethHowah(CQU),LizSpringfield(UQ),LyndenGriggs(UTas),MalcolmJones(UQ),MaryTom(CQU),MatthewBower(MQU),MichaelCowling(CQU),MichaelStokes(UTas),MichaelTate(UTas),NathanielLyon(MonashCollege),RahatHossain(CQU),RebekahScotney(UQ),RobinFitzgerald(UQ),RuthGeer(UniSA),ScottGrant(Monash),ShaniTobias(Monash),TomDunning(UTas),WendyBalassa(UTas).Theirparticipationledtheprojectteamtowardssignificantfindingsandrecommendationstoimprovethee-Examtechnology,processesanddocumentation.WealsothankinitialteammembersandsupportersinvolvedearlyintheprojectincludingKimBlackmore(ANU),PaulNewhouse(ECU—deceased),DominicVerity(MQU),MarilynBaird(Monash),ShonaLeitch(RMIT),CarolineSteel(LaTrobe),GeoffreyCrisp(UC),GeraldCrowhurst(UQ),Sarah-RobertsThompson(UQ),HelenFarley(USQ).TheprojectteamwouldliketoacknowledgetheadditionalfinancialsupportprovidedbyMonashUniversityfrom(former)Pro-ViceChancellor,DarrellEvans,intheformofastrategicfundsallocation.Thisenabledtheacquisitionofmuch-neededspecialisttechnicalequipmentandinfrastructure.AspecialthanksisalsoduetoRobertFootefromtheNationalAccreditationAuthorityforTranslatorsandInterpretersforhissupportininitiatingafundedspin-offprojectinlanguagetranslatorcertificationtesting.Acknowledgementisdueforhighlyvaluableworkonthedevelopmentofe-ExamstechnologybyLubosRandek(late2015tomid-2016)andMartinColeman(early2016to2019);alsospecialthankstoourprojectmanagerVilmaSimbagforkeepingusallontrack.WewanttoexpressourgratitudetoDeanneGannaway,theprojectexternalevaluator,whohelpedusremainfocusedonworkingtowardstheproject’sdeliverables.

TransformingexamsacrossAustralia | ivT

Listofacronyms

ADFA AustralianDefenceForceAcademy

ANU TheAustralianNationalUniversity

AR AugmentedReality

ASCILITE AustralasianSocietyforComputersinLearninginTertiaryEducation

BYOD BringYourOwnDevice(alsoasinBYOlaptop)

CQU CentralQueenslandUniversity

DET DepartmentofEducationandTraining,AustralianGovernment

ECU EdithCowanUniversity

LMS Learningmanagementsystem

MCQ Multiple-choicequestion

Moodle ModularObjectOrientedDynamicLearningEnvironment(aLMS)

Monash MonashUniversity

MQU MacquarieUniversity

NAATI NationalAccreditationAuthorityforTranslatorsandInterpreters

NAPLAN NationalAssessmentProgram-LiteracyandNumeracy

OLT OfficeforLearningandTeaching

RMIT RMITUniversity

SES SocialEconomicStatus(asinlowSES)

TEQSA TertiaryEducationQualityandStandardsAgency

UniSA UniversityofSouthAustralia

UNSW UniversityofNewSouthWales

UQ TheUniversityofQueensland

UTas UniversityofTasmania

TransformingexamsacrossAustralia | vT

ExecutivesummaryProjectcontext

Aparadigmshiftinhighereducationisunderway.Wearemovingfromanalogue,disconnected,paper-centricassessmentpracticestowardstheinclusionofdigitallyconnected,relevantandauthenticapproachestolearningandteaching.Thisenablesustoaccreditgraduatesasreadyfortheworkandsocialworldofthe21stcentury.

Thismoveisnecessarybecausecurrentinvigilatedexaminationpracticesarestuckinthe20thcentury.Moderne-toolsofthetrade,intheformofsophisticatedsoftwareapplicationsandinformationresources,aredisallowedinfavourofanalogueisolationintheexamroom.Studentsaccustomedtoconnected,digitalwaysofworkingarestrugglingtodemonstratetheirskillsunderconditionsrarelyfacedintheprofessionalandvocationalsphere.Academicsarealsoheldbackfromdevelopingricher,more-authenticwaysofassessinglearninginlightofthecurrenttrade-offsbetweenpedagogicalefficacy,integrityandfeasibility.Althoughincreasingnumbersofdigitalassessmenttoolsarebeingoffered,manyneglectauthenticityinfavourofscalabilityorsecurity,orfocusonanarrowcontextofclassroomquizzesortestingcentres.Thesesystemsareoftenreliantonhavingperfectnetworkconnectivity.Thereiscurrentlynorobust,viablemethodtodoauthentice-assessmentthatwillworktoalignbothin-classprogressiveassessmentforlearningandhigherstakessummativeassessmentundertakeninlargescaleexamhalls.

Thisprojecttransformsexaminationsbyidentifyingthenecessaryunderlyingconditionsandcontextualfactorsforsuccessfulauthentice-assessmentpractice.Thereweretwocriticaldevelopmentsinthisproject.Firstly,therecognitionthatweb-browser-basedassessmentalonedoesnotpermitcandidatestodemonstrateskillsbasedonsophisticatedcomputersoftwareapplicationsfitforprofessionalwork.Secondly,theuseofacommonUSBbootingsystemprovidestotalinstitutionalcontrolofstudent-ownedcomputerswithoutinterferingwiththeirpersonaldataorsoftware.Thesecriticalideasledtheprojectteamtodesignarobuste-Examplatform,whichisalsoresilienttonetworkbreakdowns.Theprojectteamthenwentontoimplementandverify,acrossarangeofinstitutionalcontexts,aviabletechnologicalandproceduralapproachtoscaleauthentice-assessment.

Aims

Thisprojectaimedtoenablethedevelopmentanddeliveryofauthentice-assessmentinthesupervisedexamroomcontext,inamannerthatwasscalableandsustainable.Tofacilitatethemovefrompapertodigitalexams,theprojectsetouttodeliverapracticalsolutionintheformofane-Examtechnologyplatformsupportedbyguidesforstudents,teachersandadministrators.Evidenceastotheviabilityoftheapproachtoauthentice-Examswastobeprovidedby35research-ledtrialsacrossarangeofinstitutionalcontextsthatincludedstudent,teacherandadministratorexperiences.Thesefindingswillguideteachers,assessmentdevelopers,technologysupport,examinationsmanagersandinstitutionalleadershipinmattersofpolicy,practicalimplementationandpedagogicdesign.

Approach

Theprojectinvolvediterativephasesofdevelopmentalongaroadmap.Aseriesofe-examtrialswereconductedindifferentinstitutionalcontextswiththefindingsofeachroundfeedingforwardintothenextiteration.Eachinstitutionprogressedattheirownpace.

TransformingexamsacrossAustralia | viT

Start> >>> >>> >>> >>> >Future>GetReady Phase1 Phase2 Phase3 Phase4 Phase5

Institutionalapprovals,

researchethics,hardwareandinfrastructure

Paper-equivalentsmallscale

Post-papersmalltomedium

Mediumtolargescale

WhitelistedandloggedInternet

OpenbutfullyloggedInternet

Basicdocexamstobegin!

Expandingtheappandmedialandscape

Addingthepowerofan

LMSNetworkedBYODexam

NetworkmixedmodeBYODexam

Crawling Walking Running Jumping Flying!

Acrossthefouryearsoftheproject,10partnerinstitutionshadinput,35e-examswereconductedwithover3000students,teachers,andadministratorsparticipatinginexamsessions,insurveys,focusgroupinterviews,workshopsandaninternationalsymposium.

Projectdeliverables

Theprojectteamnotonlymetalldefineddeliverablesoftheprojectbutalsoprovidedadditionaloutputsthatcentredondisseminationandadoption.Theprojectteamfocuseduponactivitylikelytohavethegreatestimpactonthehighereducationsector,thisbeingdevelopingknowledgeaboutauthentice-assessmentpracticeandinprovidingthepracticalinfrastructuretoimplemente-examsinsupervisedsettings.Significantoutputsincluded:

• theTransformingExamsprojectwebsitehousesthemajorityofprojectoutputslistedbelowandservedasaportalforupdates,seehttp://transformingexams.com

• thee-Examstechnologyplatformconsistingofaclient,serverandadministrationcomponents.Thissoftwarewasupdatedastheprojectprogressedanddistributedtoprojectpartners.Demonstrationversionsareavailableontheprojectwebsite

• learningresourcesandguidesforstudents,teachers,technicaladministratorsandexaminationsmanagersweredevelopedandmadeavailableontheprojectwebsite

• anationalroadshowandsymposiumencompassedfivecapitalcitiesinwhichover300highereducationpersonnelparticipated

• disseminationoutputsincludedover60presentationsatconferencesandinstitutionalmeetings,withsessionrecordingsortheslidesmadeavailableviatheprojectwebsite.Thesesessionsadvisedthehighereducationcommunityoftherationaleandaroadmaptowardsauthentice-assessmentaswellastheresearchfindingsfrom35diverseresearch-lede-Examtrials

• atotalof15refereedpublicationswereproduced(withmoretocome)coveringresultsfrom35e-Examtrialsinwhich1700studentstypedtheirexam.Findingscoverede-Examsystemdesignanduse,students'perspectivesone-Examsandontheirwritingstrategies,possibledigitalassessmentfuturesandpotentialimpactsofauthentice-assessmentoncurriculumtransformation.

KeyFindings

Theprojectencompassedmultiplerealmsfromtheory,perceptionandpolicytopracticalmeasuressuchastechnologydevelopment,implementationanduse.Keyfindingsincluded:

• creationofaviable,robusttechnologyplatformfore-Exams.Itisflexible,supportingatransitionfrompaper-equivalentexamstorich,authenticdigitalassessment.Atechnologyplatformfore-Examsmustsupportpedagogicalalignmentatallstagesoflearningfromtheclassroomintotheexamroom.Therefore,itintegratesandleveragesexistingsystemssuchascommonwordprocessors,learningmanagement

TransformingexamsacrossAustralia | vii

systemsandsoftwarecommonlyusedinthediscipline.Aviablee-Examsystemmustworkintheroughspacesofclassroomsandeducationsettings.Thee-Examsystemwasfoundtobesecureandreliable,highlyresistanttonetworkoutagesandcanfunctionoffline.Thisallowsstudentstofocusoncompletingtheirexamandminimisesdisruptiontoexamtimetables

• studentsaccepte-Examsandfindthemsuitedtotheirlargelypreferredwaysofworking.Studentsreportthattypingispreferabletohandwritinginexams.However,somestudentsneedsupportinthetransitionwithasignificantminorityfacingchallengesinadjustmentofwritinghabitsandinsupplyingequipmentthatisuptostandard.Thetransitionroadmap,alongwithamixofinstitutionandstudenttechnology,willenabletheuptakeofmediumtohigherstakesdigitalassessmentacrosseducationalinstitutions

• logisticalsupportandattentiontodetailmattersinthehighstakes,time-pressuredexams.Whereorganisation,resources,processandcommunicationarelacking,achievinggoodoutcomeswithcomputerisedexamsismadesignificantlyharder.ExaminationsofficepersonnelareexpertsatlogisticssoleveragingthatexpertiseincollaborationwithotherkeyareassuchasITsupportisimportanttosuccess

• academicdevelopmentintermsofdigitalliteracyanddigitalpedagogywillbeimportanttoembeddingandscalingauthentice-assessmentacrossthecurriculum.Whilee-Examsareonepartofthepicture,ourcurrentanalogueexampracticesareoneofthe'blockers'tobroadersystemicchange.Theneedtoensurealignmentofpracticesinclassroomsandexamhallsmeansthatacademicteachingandlearningsupportstaffmustalsohavetheskillsandcapabilitiestodeployrelevante-examstechnologiesandtoolsintheirclassrooms

• policyisamechanismtosupportchange,asismanagerialsupport.Policiesonexaminationsthatsetouttoframee-Examsmustextendbeyondtraditionalconfines.Newsystemicconnectionsmustbemadewithbroaderareas,includingdevelopingaholisticdigitallearningstrategy,togetherwithaBringYourOwnDevice(BYOD)-focusedinfrastructuredevelopmentstrategy,ITsupport,pedagogicprofessionaldevelopmentandstudentsupporttoenableanequitabletransition

• universitiesdonotstandaloneintheassessmentspace.InAustralianschools,NAPLAN(nationalnumeracyandliteracytesting)andmatriculationexamsaretentativelymovingonline,OthercountriessuchasFinlandhavealreadycompletedthetransitionwithtestsandexamsinschoolforYears10–12,aswellasuniversityentranceexamsconductedentirelyoncomputers.Thee-Examplatformiswellplacedforuseinschool-basedassessmentsandmatriculationexaminations,aswellasinvocationalandprofessionaleducationsettings.

Futurework

Theprojectwasagreatsuccessindemonstratingwhatrobust,authenticdigitalassessmentcouldbelike.Butthefindingsshowourworkisnotyetcomplete.Furthertechnicalworkisrequiredtoimprovethee-Examtechnologyplatform,itsintegrationwithothersystemsanditsadministrativesupporttools.Furtherinvestigationandinvestmentintosystemicsupportmechanismsisrequiredtoscaledigitalassessmentathighereducationinstitutionsinareassuchasprocess,professionaldevelopment,policyandinfrastructure.

TransformingexamsacrossAustralia |viii

TableofContentsAcknowledgements...................................................................................................................iiiListofacronyms........................................................................................................................ivExecutivesummary....................................................................................................................vTablesandfigures.....................................................................................................................ixChapter1. Theproject...........................................................................................................1

1.1 Background....................................................................................................................11.2 Projectlogistics..............................................................................................................21.3 Projectmanagement.....................................................................................................31.4 Statusofprojectdeliverables........................................................................................3

Chapter2. Ane-Examplatformforauthenticassessment....................................................42.1 Rationaleforthedevelopmentofane-Examsolution.................................................42.2 Priore-Examssolutions.................................................................................................42.3 Historicaldevelopmentofarobuste-Examplatform...................................................52.4 Thelateste-Examplatform...........................................................................................7

Chapter3. e-Examtrialfindings...........................................................................................103.1 Researchdatacollection.............................................................................................103.2 Studentperspectives...................................................................................................113.3 Logisticsfindings..........................................................................................................123.4 Minicases....................................................................................................................14

Chapter4. Dissemination.....................................................................................................154.1 Websiteandsocialmedia............................................................................................154.2 Presentationsandpublications...................................................................................154.3 e-Examcasestudies....................................................................................................164.4 e-Examworkshops......................................................................................................164.5 e-Examroadshow........................................................................................................164.6 e-Examssymposium....................................................................................................17

Chapter5. Impact.................................................................................................................195.1 Changedpractices.......................................................................................................195.2 Othere-examprojects.................................................................................................195.3 PartnershipwithNAATI...............................................................................................195.4 Potentialjointventurepartnershipandcommercialisation.......................................195.5 Furtherresearchandjointprojectwork.....................................................................20

Chapter6. Projectevaluationandlessonslearnt................................................................216.1 Projectexternalevaluation.........................................................................................216.2 Policyadjustment........................................................................................................216.3 Logisticsandcommunicationswithinpartnerinstitutions..........................................22

Chapter7. Conclusion..........................................................................................................24References..............................................................................................................................25AppendixACertification.........................................................................................................27AppendixBProjectdeliverabledetails...................................................................................28AppendixCe-Examtrialdetails...............................................................................................30AppendixDUpdatedImpactMatrix.......................................................................................35AppendixEFormativemid-projectevaluationreport............................................................37AppendixFFinalexternalevaluationreport...........................................................................39

TransformingexamsacrossAustralia |ix

TablesandfiguresTablesTable1Statusofprojectdeliverables......................................................................................3Table2e-Examsystemfeaturesandaffordances...................................................................7Table3e-Examtrialsconductedacrossinstitutionsandyears..............................................10Table4e-Examsummarystatistics........................................................................................11FiguresFigure1Projectorganisation...................................................................................................2Figure2Tensionsandtrade-offsindevelopingorselectinge-Examssolutions......................5Figure3e-Examsystemdevelopmentandadoptionroadmap...............................................6Figure4e-ExamworkflowforrobustMoodleuse(phasethree)............................................8Figure5Photo:arobustonlinee-Examsessioninprogress..................................................10Figure6Aggregatedstudentfeedbackonthee-Examsystem..............................................11Figure7StudentsurveyresponsesfollowingrobustMoodleexams....................................11Figure8Handwritingneatnessandtypingprowess..............................................................12Figure9Powersupplyfore-Exams:layingcablesandbatterylife........................................13Figure10Websitevisitorsbycountry....................................................................................15Figure11Photo:e-Examsymposiumsessions24November2018.......................................17

TransformingexamsacrossAustralia |1

Chapter1. Theproject1.1 BackgroundThisproject‘TransformingexamsacrossAustralia:Processesandplatformfore-examsinhighstakes,supervisedenvironments’wasbornfromthegrowingdisconnectbetweenthewayhighstakestestingisconductedusingpen-on-paperexamsandstudents’everydayexperiencesofstudy,workandlife.Thecomplexityofproblemsfacedbyallindustriesandsocietydemandstheuseofsophisticatedproblem-solvingapproaches,typicallyemployingadvancedsoftwaretechnologies.Theuseofinformationandcommunicationtechnology(ICT)inhighereducationand,inparticular,invigilatedhigherstakesassessmentlagsbehindthecommonuseofICTbystudentsinotherareasoftheirstudy,workandsociallife.Thisisdespitehighereducationinstitutions’ambitionstoprovidestudentswithcoretechnologyliteraciesrequiredforsuccessinamodernworld.Thepaucityofauthenticassessmentattheinvigilatedmediumandhigherstakesendoftheassessmentspectrumisholdingbacktheabilityforhighereducationtoevaluatestudents'learninginlightofdesired21stcenturycapabilities(Fluck&Hillier,2016).TheAustralianhighereducationsectorneedsagreatlyexpandedcapabilityforhigherstakes'assessmentandpromotionofstudentlearning'ifourgraduatesaretoremaincompetitiveintheinformation-rich,technology-intensiveworldoftoday.

Thisprojectaimedtoexplorehowauthenticformsofassessmentcouldbeundertakenifsophisticated'e-toolsofthetrade'weremadeavailabletoassessmentdesignersandstudentsintheinvigilatedexamroomcontext.Theprojectteamsetaboutdevelopingandevaluatingacomprehensivetechnologyenvironmentforexamsthatwouldenableauthentic,complex,constructedassessment.Thee-ExamsprojectteamrecognisedthattherearecontextualchallengessurroundingtheimplementationofICTintheexamroom,particularlyintheformofasuitable,scalableandaffordableexamstechnologyplatform.Surroundingissueswerealsowithinthescopeofprojectdeliberationsincludedconsiderationoftheefficiencyofexammanagement,thelogisticsofcarryingoutcomputerisedexaminations,theavailabilityofsuitablespacesatinstitutions,theneedforcomputerequipmenttocaterforalargenumberofexamcandidatesandstudentacceptanceofe-exams.

Thepredecessorseedprojectidentifiedtheneedtofurtherenhanceasetofe-examsgoodpracticeguidelinesacrossmultipleinstitutionalsettings.Theseedprojectalsoprovideddirectionforthefurtherdevelopmentofarobuste-examsinfrastructurewithsector-widerelevanceandcross-disciplineapplicability.Theteamalsoarticulatedhowe-examscouldevolveintheAustralianhighereducationcontextoverthefollowing10years(Hillier&Fluck,2015).

Thisprojectsetoutto:

1. buildontheexistinge-examplatform(developedunderthepriorseedproject)intendedforuseinsupervised,'bringyourownlaptop'(BYOL)settingstoincludecomputer-markedquestionsandelectronicreticulationofstudentresponses

2. developguidelinestoassiststudents,educatorsandadministratorstoeffectivelyprepareandundertakee-exams,takingintoconsiderationthewholeassessmentworkflow.

TheprojecthasdeliveredarangeoffindingsonstudentpreferencesandageneraltoolsetforICT-enhancedinvigilatedmediumandhigherstakesassessmentusinganapproachthat


isdesignedtobeopen,modularandastechnologyneutralaspossible.Thee-Examdeliveryplatformdevelopedbytheprojectteamenablesinvigilatedhighstakestestingoncampusata largescale.Itmustbenotedthatthisprojectdidnotspecificallyaddressonline-onlyordistanceeducationcontexts.

1.2 ProjectlogisticsTheprojectteamwasorganisedintoclustersateachpartneruniversitywiththeleadinstitutionhosting'e-ExamHQ'.TheorganisationshowingtherelationshipsbetweeninstitutionsandgroupsisdepictedinFigure1.

Figure1ProjectorganisationOnreceptofthegranttheprojectleaderpreparedthegroundworkforimplementation.Theactivitiesincluded:

• recruitmentofstaffincludingtheprojectmanagerandleadtechnicaldevelopers• obtainingethicalclearancefromtheleadinstitutionMonashUniversity,which

facilitatedtheapprovalbyprojectpartnerinstitutions:CentralQueenslandUniversity(CQU),EdithCowanUniversity(ECU),MacquarieUniversity(MQU),TheUniversityofQueensland(UQ),UniversityofSouthAustralia(UniSA),andUniversityofTasmania(UTAS).TheUniversityofNewSouthWales–AustralianDefenceForceAcademyjoinedtheprojectin2017.Note:TheAustralianNationalUniversity(ANU)andRMITUniversitydidnotparticipateine-examtrials

• developingsubcontractagreementsbetweenMonashUniversityandindividualpartnerinstitutionsthatoutlinedresponsibilitiesanddeliverables.Wheregrantpaymentswereplanned,theseweremadeconditionalupondeliverablesbeingmetateachstage

• holdingapreparationworkshopintheopeningmonthsoftheproject.HostedatMonashUniversity,CaulfieldCampus,itwasattendedby21projectcollaboratorsandteammembers.Theaimwastoprovideaprojectoverviewanddetailedinstructionsontheimplementationofthee-examstrialsandahands-onsessionusingthee-Examsystem.Attendeesincludedacademics,learningdesignersandexamsofficersfrompartnerinstitutions

• developingapackageoffilestoruntrialsthatwasprovidedtoeachpartner.Thisincludedtemplatesforapreliminarymeetingchecklist,ethicsprotocoldraft,participantinformedconsentinformationandform,surveyandfocusgroupquestionnairesanddatacollectionspreadsheettemplates.Inthelead-uptoeachtrial,meetingswereheldwithcourseacademicsbyprojectteammembersateachinstitutiontoarrangecontextspecificactionplans

• collectingdataduringandfollowingtrialsthatwasreportedtotheprojectleaderviaprojectpartnerinstitutionrepresentativesusingthedatacollectionspreadsheettemplates.

Core partner institution(s). Project-funded sites. Lead + RA staff. Trial sites.

Associate partner institution(s) Self-funded. a) Trial sites: receive mentorship; return data. b) Technology exchange partners.

Lead institution. Funded. 'e-Exam HQ': Project leader > project manager, technical development staff. Main trial site.

Reference group DET

Other key stakeholders.

External Evaluator


1.3 ProjectmanagementThefollowingactivitiesenabledthenationalscaleprojecttobemanaged.

• Regularprojectmanagementteammeetings:TheprojectmanagementteamcomprisingtheprojectleaderandprojectmanagerheldregularweeklySkypemeetings.

• Projectteammeetings:Theteamhadfiveface-to-facemeetings:onein2016,twoin2017andtwoin2018.

• Regularemailcommunication:Teammembersandcollaboratorscommunicatedcloselyviaemail.Communicationwascentredonrunningofparticulare-examtrials,arrangingsitevisitsandpreparingjointpublicationsorpresentations.

• Useofprojectmanagementtools:Theprogresstowardsdeliverablesandoutcomeswastrackedusingtheonlinetool'Basecamp'.Thistoolallowedprojectmemberstocommunicatewitheachother,kepttrackoftheirtasks,timelinesandmilestones.Italsoservedasrepositoryforprojectdocumentsrelatingtoindividuale-examtrials.Ametaprojectwasestablishedwitheachpartnerinstitutionhavingitsownsubproject.TeamGanttsyncedwiththeBasecampaccountandprovidedanintegratedviewofalltasksandschedulesforthecoremanagementteam.CloudstorageservicessuchasDropbox,GoogledriveandAARNETCloudstorwereusedasrepositoriesforcommonprojectdocumentsandtodistributeupdatede-Examsystemfiles.

1.4 StatusofprojectdeliverablesFollowingonfromtheoriginallyplannedprojectdeliverablesitwasfoundthattheenvironmentandfocusofourworkevolvedovertheprojectperiod2015to2018.Alldeliverablesweremet,althoughsomeadjustmentsweremadetotheformdeliverablesasnewinformationcametohand.Thestatusofeachoftheoriginallyplannedprojectdeliverables,alongwiththeachievedprojectoutputs,isshowninTable1withfurtherexplanationofeachprovidedAppendixB.

Table1StatusofprojectdeliverablesOriginaldeliverable Deliveredoutpute-Examstechnicalinfrastructure Delivered.Arobustandmodulare-exams

platformresistanttonetworkoutagesGoodpracticeguides Delivered.Theseevolvedintoasetofuser

guides.Adraftsetofassessment/examspolicyrecommendationsforinstitutions

Alternativedelivered.Seesection6.1.foralternativefindingsanddiscussion

Asetofexamplee-exams Delivered.35trialsconductedacrossthreemodalitiesandmultipledisciplines

Researchfindingspublished Delivered.15refereedpublicationsand60presentations

Workshopmaterialstointroduceacademicstoe-exams

Delivered.Severalworkshops,nationalroadshowandaninternationale-ExamsSymposium

An'e-examsfoundation'drafttermsofreference

Alternativedelivered.Recommendationforjointventureorcommercialisation


Chapter2. Ane-Examplatformforauthenticassessment2.1 Rationaleforthedevelopmentofane-ExamsolutionAtthebeginningoftheprojectanumberofpremiseswereoutlinedthatformedarationaleforthedevelopmentofane-Examssolution.Thechiefamongthesewasthegrowingdisconnectbetweencurrentexaminationpracticesandtheworldofwork.Similarly,examswerebecomingfurtherdisconnectedfromtheremainderofstudents'learningexperiencesandtheirextensiveuseofcomputersintheirstudies.Therewasanoverwhelmingsensethatexaminationsneededtoenable'authentice-assessment'whilemaintainingassessmentintegrity.Therewasalsorecognitionthatane-Examapproachhadtobescalableandsustainable.Thestartingrationalewasoutlinedinthreebroadareaspresentedinthepointsbelow,withfurtherexplanationandaconceptmapprovidedinHillierandFluck(2013).Studentrequirements:

• studentsrarelyhand-writeassessmentresponses• studentsarefamiliarwiththeirowndevices• highstudentownershipoflaptops• student-owneddevicesarediverse• studentsdislikeinterferencewiththeirdevice.

Pedagogicalrequirements:• technologycanenhancetheauthenticityofassessment• equivalentsoftwareenvironmentforallstudents• provideopportunitiestopractice.

Institutionalrequirements:• scalabilityofthesolutiontoservelargestudentgroups• institutionalcontextsdifferthatdemandsaflexiblesolution• resilienceandsecurity• unauthoriseddataandcommunicationmustbeexcludedfromtheexam• workflowefficiency.

2.2 Priore-ExamssolutionsAtthebeginningoftheinnovationanddevelopmentoftheprojectamarketscanwasconductedthatlookedatexistingcomputerisedexamsolutionsandapproachesbeingused.Thesewerefoundtobewanting.Typically,solutionsfocusedonsecurityorefficiencybutoftenneglectedpedagogicalsophistication.ThistensionisshowninFigure2(left).Theexistingapproachestoconductingcomputerisedexamshadanumberoflimitations,suchthatoneormoreofthefollowingelementswereevident:

• Limitedpedagogicscope.Limitationsinthetypeofquestionsandtasksthatarepossible.Systemsoftenfocusedonasingletypeofassessmentormodality.Typicallyasanisolatedquizinvolvingselectedresponsequestionsoressayquestiontypes.

• Candidateswerenotpreventedfromaccessingunauthorisedmaterials.TheuseofstandardquiztoolsembeddedinalearningmanagementsystemmeansthatstudentshaveaccesstoothertoolswithintheLMS.

• Candidatenumberswerelimitedtothecapacityofon-campuscomputerlabs.Testsundertakeninteachinglabswithafinitecapacitycanonlycatertoasmallnumberof


seats,thisisinsufficientforthenormalhighvolumeofcandidates.Thelayoutoflabroomsisoftenunsuitableforhighstakesexamswithoutsomemodification.

• Proprietarysoftwareapplicationsandlimitedflexibility.Licensingconditionsmaylimitbroaderusewithoutadditionalcostsbeingincurred.Similarly,mostsystemsarepresentedasaseparateenvironmentthatisnotpartoftheregularlearningandteachingtoolset.Thiscreatesadiscontinuityofexperiencebetweenlearningandsummativetesting.This'blackbox'styleseparationalsolimitsthedatasharingtoafinaltaskgrade.Thus,itlimitsthepotentialforbigdatalearninganalyticstoincludefine-graineddetailwhenassessmenttasksareundertakeninseparatesystems.

• Invasivetechnology.Somesolutions,particularlytargetingstudentdevices,enforceundesirableconditionsonstudentssuchasrequiringinvasivelockdownsoftwaretobeinstalledintostudent'spersonaldevicesthatmaycreateconflictswithinsystems.

• Lackofholisticcontrol.Inparticular,solutionsthatutilisedremotesupervisionservicespresentedrisksofexamprotocolbreachesduetothelackofaholisticcontrolsphereandcommonlyutilisedinvasivetechnology(asperpreviouspoint).Ifoutsourcedtestingcentresorserviceswereused,thencontrolispassedtootherorganisationsorindividuals.Theseservicesweremainlyintendedforsmallnumbersofexternalordistanceeducationstudentsandwerefoundtobecostlyperstudent,negatingtheiruseatscale.However,ithasbeenfoundthatoverthedurationoftheprojecttheuseofsuchserviceshasbeenincreasing,withanever-largernumberofcoursesmovingtoonline-onlydelivery.Cautionisurgedintheraceforefficiencyandaccesswheretheintegrityoftheassessmentsystemmaybeplacedinjeopardy.

Itwasfeltnecessarytodevelopanewtypeoftechnologyplatformthatwouldovercomemanyoftheissuesoutlinedabove,whileenablingamuchgreaterdegreeofauthenticityinassessmenttaskswithinthesupervisedcontextofexamhallsandclassrooms.Giventheapparentgapinthemarket,theprojectteamhastakena'pedagogyfirst'approach,buthasmaintainedakeeneyeontheenablingcost-effectiveness,implementationandintegrity.Inmovingtowardsasolution,thewell-knownprojectmanagementaxiomof'good,quickandcheap–picktwo'wasapparentinimplementinge-Examsprojects.Thisisoftenshotintostarkreliefwherecorporatemanagerialprocessesareinstigatedtoselecta'solution'.Itwasobservedthatatensionbetweenthespeedoreaseofimplementation,costandpedagogicalaffordancesoftenledtothelatterlosingout.Thistensionasitappliestoe-ExamtechnologyisrevisitedinFigure2(right).

Figure2Tensionsandtrade-offsindevelopingorselectinge-Examssolutions

2.3 Historicaldevelopmentofarobuste-ExamplatformThee-Examplatformstartedlifein2007attheUniversityofTasmania.Initially,in-classe-testswereconductedbybootingfromCD-ROMandwritinganswerfilestoasecondaryUSBstick.Thismethodcentredonusingword-processingdocumentsasthequestionand


responsecompositionenvironment.Thiscontinuedin2008and2009forin-classe-Exams(SeeFluck,citedinLane,2009;Fluck,Pullen,&Harper,2009).In2010,thefirstLiveLinuxUSBstickwasusedtobootWindowsPCswhileCD-ROMscontinuedtobeusedonApplecomputers.In2011,e-ExamswereformalisedwithUniversitySenateapproval.Atthistime,aUSBstickversionwasdevelopedforusewithbothWindowsandApplecomputers.Senateapprovalinitiatedamoveoutofcampuscomputerlabsandintocentrallyadministeredexaminationsvenues.In2013,theOfficeofLearningandTeachingapprovedaseedprojectanddevelopmentbegunatTheUniversityofQueenslandonversionfiveofthee-Examsystem.Thisphaseaimedtostreamlineandautomateaspectsoftheuserexperience,addadditionalsoftwareapplicationsandexpandtherangeofcompatiblehardware.AprototypeofflineMoodleinstancewithinthee-Examsystemwasalsodevelopedin2014(althoughitwasnotreadyforuseinexams).FollowingthesuccessfulawardofanAustralianGovernmentDepartmentofEducationandTrainingInnovationandDevelopmentgrantin2015,upgradestothecoreoperatingsystemandcapabilityofthee-Examsystemwererealised,producingversionsixofthee-Examsystem.EarlyintheInnovationandDevelopmentProjecttheteamarticulatedaphasedroadmap(seeFigure3)forthedevelopmentofsystemcapabilitiesandthenatureofexaminationsthatcouldbeundertakenateachphase.Theroadmapsetoutapathwaytowardsauthentice-assessmentwiththeintentionthateachinstitutionwouldprogressattheirownpace.

Start> >>> >>> >>> >>> >Future>GetReady Phase1 Phase2 Phase3 Phase4 Phase5

Institutionalapprovals,

researchethics,hardwareandinfrastructure

Paper-equivalentsmallscale

Post-papersmalltomedium

Mediumtolargescale

WhitelistedandloggedInternet

OpenbutfullyloggedInternet

Basicdocexamstobegin!

Expandingtheappand

medialandscape

Addingthepowerofan

LMSNetworkedBYODexam

NetworkmixedmodeBYODexam

Crawling Walking Running Jumping Flying!Figure3e-Examsystemdevelopmentandadoptionroadmap

Aseriesoftrialswerethenimplementedthatcoveredphasesonetothree.Thesetrialsincludedusingwordprocessordocuments(inbothphaseoneandtwo),includingmulti-languagetranslationexams;usingaspreadsheetasaformtoenableautomatedmarking(seeHillier&Grant,2018);andanewclient-servermodeforMoodlequiz-basedexams(phasethree).Furtherupdatesincludedcomprehensivesystemlogging(hardwareandsoftwareprocesses)andthedevelopmentofprototypesessionrecordingofuseractions(screenvideorecording,webcam,screenimagesandtextcapture).A'robust'integrationofSafeExamBrowserwithintheLiveLinuxenvironmentwasalsodevelopedthatallowsanexamsessiontocontinueintheeventofnetworkoutages.Livetrialsofthe'robust'Moodleapproachtoe-examsweresuccessful(seeHillier,Grant,&Coleman2018).In2018,theprojectteam,incollaborationwithcolleaguesatBondUniversity,developedaprototypeAugmentedRealitye-Examtocollectspatialdatafromuseractions(Cowling,Hillier,&Birt,2018).BytheendoftheInnovationandDevelopmentProjectin2019theprojectteamwasbeginningnegotiationstotrialthe'robust'e-ExamsystemapproachinalternativelearningmanagementsystemssuchasCanvasandearlydiscussionsofjointventurearrangementswithEuropeanorganisationshadalsobegun.


2.4 Thelateste-ExamplatformAttheconclusionofthisprojectthee-ExamplatformhadreachedastateofmaturityhavingbeentestedinphasethreetrialswithMoodle-centricrobustonlineexams.Therearethreemaincomponentsavailableforthee-Examplatform:

• studentuserclient'e-ExamOS'(liveUSB)forofflineor'robust'onlinemode• servertohostMoodleLMSand'robust'plugins• administrationtool(alsoalivesystemUSB)tousewithlargeUSBhubs.

ThesystemasitstandshasanumberofsignificantfeaturesasoutlinedinTable2.

Table2e-ExamsystemfeaturesandaffordancesPertinentfeatures AffordancesA'wholecomputer'environment(operatingsystem,securebrowser,mediaplayersandarangeofsoftwareapplications)asanexamdeliveryenvironmentratherthanasingletestingapplication.

Vastlyexpandedpedagogicallandscapeoverthatofabrowserwindoworsingletestingapplication.Arangeof'e-toolsofthetrade'canbeusedtoconstructsophisticatedresponses.Examplesincludeofficesuite,spreadsheet,languagetranslationdictionaries,programmingkitsuchasScratchorPython,mathematicalapplicationssuchasRorGeoGebra,augmentedrealityapp.

Responsesviaafullwordprocessororconstructedviaspecialistapplications(mainlyhumanmarked)andresponsessecurelyreticulatedviaalearningmanagementsystem(LMS)quizwithcomputerisedmarking.

Catersforintroductiontoadvanceduses,frompaper-equivalenttopost-papermodalities.Softwarecomponentscanbeaddedorremovedtosuit.Computerisedresponsecollectionfacilitatesassessmentanalytics.

Robustagainstnetworkoutages—noorminimallivenetworkconnectivityisrequiredduringexam,evenforLMSquestions.

Notdependantonahighdegreeofnetworkreliabilitywhilemaintainingtheadvantagesofnetworkedcontentdeliveryandresponsesubmission.Canbeusedinisolatedareaswithpoorconnectivity.

Livealternativeoperatingsystemused—meansstudent-ownedequipmentcanbeusedtohosttheexamenvironmentbutisleftuntouched.

Anethicalapproachtoscalability.Thereisnoinvasivesoftwaretoinstall.Allcandidateshaveanequivalentsoftwaretoolsetwithsomeaccessibilityfeaturesbuiltin.

Modular,opensourcecodebaseandcommodity'offtheshelf'components.LeveragingLinuxsecuritywithextensiveloggingandsessionvideorecording.

Leveragingpopularandsustainableprojectsforbetterefficiency.Fully'known'totestadministrators(no'blackbox').Asecureenvironmentwithextensiveauditing.

OneversionworksonmostIntelbasedlaptops—Apple,Windows,Linux,thathaveaUSBport.CanbeusedincampuslabsandBYOlaptops.

Onesoftwareversioncanserveall.Streamlinesdevelopmentandmaintenance.Allexamcandidatesareequippedwiththesamesetofsoftwareapplications.

Userguidesareavailablefor:students(Preparationandpracticeguide),teachers(Preparingexammaterialsguide),examinationprocessadministration(Organisinge-Examsguide),technicaladministration(Administrationtoolguide,Alternativeadministrationguide,Hardwareguide,Laptoprequirementsguide),generaluserguides(Quickstartguides,Detailedstart-upguide).Thestudentclientportionoftheplatformanduserguidescanbeobtainedfromtheprojectwebsiteathttp://transformingexams.com.Theprocessforusingthee-ExamsysteminarobustonlineMoodledeployment(phasethree)isshowninFigure4


withfurtherdetailavailableinHillier,GrantandColeman(2018).Detailsoftheworkflowusedforofflinedeployment(asperphaseoneandtwo)isavailableinHillierandFluck(2017).

Figure4e-ExamworkflowforrobustMoodleuse(phasethree)

Exammodalitiesusedtodatewithintheproject(inphasesonetothree)include:1. Paper-equivalentworddocument-basedexam.Primarilythesewereconductedon

opt-inbasisandusedforearlystagetrialsateachinstitution(asperphaseone).Nonetworkinfrastructurewasrequiredtoruntheseexams.Studentsweregivenachoiceoftypingorhandwrittenresponses.Academicspreparedexamsusingawordprocessorandstudentsrespondedwithinacopyofthedocument.

2. Post-paperworddocumentcentricexam.Theserequiredallcandidatestotype(asperphasetwo)andweregenerallyundertakenfollowingseveralroundsofopt-intrialsatagiveninstitution.Post-paperexamsexploredassessmenttasksthatcouldnotbeperformedonpaper(e.g.multimediaprompts,spreadsheettasksorcomputerprogrammingtasks).Nonetworkinfrastructurewasrequiredtoruntheseexams.Academicspreparedexamquestionsusingawordprocessor,butalsoselectedandpreparedadditionalpromptsandresourcestoaccompanythequestions(e.g.video,audioclips,oradditionalsoftwareapplications).Studentsrespondedusingamixoftheexamworddocumentandbysubmittedfilesconstructedusingothersoftwaretools(e.g.aScratchprogrammingfile).

3. Client-serverbased'robust'onlineMoodleexam.TheseweresimilarinnaturetothephasetwoexamsbutwereconductedbasedonaMoodlequizviaSafeExamBrowser.TheseexamsrequirednetworkinfrastructuretobeinplaceintheformofaMoodleserverandWi-Ficonnectivity.Multimediawasalsoutilisedinsomeexams(e.g.languagelisteningtestsusingaudiofilesandearphones).AcademicspreparedexamquestionsusingtheMoodlequiztoolsandembeddedmediafilesasrequired.Selectsupportingsoftwareapplicationswereaddedasrequired(e.g.aChineselanguagetranslationdictionaryapplication).Ademonstrationaugmentedreality(AR)


e-ExamwasalsodevelopedwithanAndroidVirtualmachinecontaininganARapploadedintothee-Examsystem.TheARappcollectedspatialdatapertainingtouseractionsinrelationtomarkers.Thesedatawerethentransmittedtoaserverforanalysis.SeeCowling,HillierandBirt(2018)forfurtherinformation.

Additionalfeatureswereprototypedbuthaveyettobetrialledinliveevents.Theseinclude:4. Accesstoexternallywhitelistedinternetresourcesduringanexam,withaccess

logging.Thisrequiresahigherdegreeofnetworkavailabilityforthedurationoftheassessmenteventbecauseanyexternalresourceneedstobeavailablewhenthestudentcallsuponit.AdemonstrationexamfeaturingthistypeofquestionisprovidedontheprojectdemoMoodleserver.

5. Openonlineexamwithextensivesystemanduseractionlogging.Theremovalofrestrictionsoninternetresourcesandtheadditionofextensivesessionrecording.Userloggingcanincludevideoscreenrecording,screenimagecapture,webcamimagecapture,keystrokerecordingandcursormovementrecording.

Incomparingourprogresstothatofothere-examprojectsaroundtheworld(Fluck,2019;Fluck,Adebayo,&Abdulhamid,2017;Fluck&Hillier,2017),werealisewestillhavealottodo,yetitisalsoevidentthatthetechnicalstrategyandpedagogicalgoalswehavesetarealsoachievable.Wehighlightedtwosuchexamplesatoure-ExamSymposiumin2018.Thehome-grown'Abitti'LinuxUSB-basede-examapproachusedinFinlandseniorschoolsandmatriculationexamssaw47,000USBsdistributednationallyin2018.Thenetwork-booted'SEE'LinuxclientandMoodleserverapproachatauniversityinAustriareached50percentpenetrationforcourseexamsduring2018.Bothexamplesshowourideasarescalable.


Chapter3. e-Examtrialfindings3.1 ResearchdatacollectionAnethicsprotocolwasapprovedattheleadinstitution.Ethicsapprovalwasalsosoughtbyeachpartnerinstitutionmemberfromtheirlocalethicscommittee.Bothqualitativeandquantitativedatawerecollectedfromstudentusersusinganinformedconsentprocess.Datacollectionmethodsincludedpre–postsessionsurveys,focusgroups,observationandtechnicaldatalogging.Examsessionswereconductedinclassroomsandinexamhalls.

Figure5Photo:arobustonlinee-Examsessioninprogress

Stakeholdersurveysincluded:a) Pre-examsurvey:Doneatpracticesession—twoparts:Technicaldetailsofstudent

laptopequipmentandfirstimpressionsfollowinginitialuseoftheexamsystem.b) Post-examsurvey:Donefollowingarealexamevent.Experienceofane-examand

opinionsaboutaspectsofrespondingtoassessmenttasksandexams.Sometrialsincludedaseriesofexameventsprovidingasmalldataserieswithingroups.

c) Focusgroups:Donefollowingarealexamevent.Semi-structureddiscussionaroundselectedthemesorquestionsrelatingtocandidate'sexperienceofthee-exam.

d) Stakeholderandcommunitysurveys:Donefollowingaworkshoporpresentationtogainfeedbackonsessionrelevance,e-examknowledgeandinterestintheproject.

Atotalof31e-examtrialswereconductedacrosseightinstitutionsfrom2016to2018(seeTable3).Themostrecenttrialsutilisedthe'robust'networkresiliencefeaturesforonlineexamshostedviaaMoodlequizthatincludedwrittenandlisteningaudioresponsetasks(seeFigure5).RefertoAppendixCofthisreportfordetailsofthesetrials,includinghostinstitution,date,course,exammodality,questiontypesused,assessmentweighting,durationandthenumberoftypistsandhand-writersundertakingeachexam.

Table3e-ExamtrialsconductedacrossinstitutionsandyearsInstit'n CQU ECU Monash MQU UniSA UNSW UQ UTAS Total2016 3 2 52017 2 2 9 1 1 1 2 1 192018 1 6 1 2 1 11Total 3 2 18 1 2 1 4 4 35

Participationacrossthe35trialsconsistedofover1700studentstypingtheirexamswith1300electingtohand-writetheirexam(whereachoicewasprovided).Examsrangedfrom15minutesto180minutesindurationandrangedinweightingfrom5percentto50percentofthecoursegrade.ThesefiguresaresummarisedinTable4.


Table4e-ExamsummarystatisticsExams Typists Penonpaper Weight MinutesSum 1750 1309 ~ 4145Mean 40 37 32% 106Smallest 1 ~ 5% 15Largest 166 ~ 50% 180

3.2 StudentperspectivesStudentsurveyresultshaveshownpositivefeedbackfromstudentuserswithratingsof4+outof5acrossmostuseracceptancemeasures.Whenaskedinpost-examsurveysifstudentswouldrecommendthee-Examsystemtoothers,alargemajoritywereinagreementwith90percentsayingtheywoulduseitwithoutconcernand70percentpositivelyendorsingthee-examapproach(seeFigure6,left).

“Iwouldrecommendthee-Examsystemtoothers”

“Moodleworkedwellforexams”

Figure6Aggregatedstudentfeedbackonthee-Examsystem

TherecenttrialsalsoaskedforopinionsoftheuseoftheMoodleLMSasanexamenvironment—mostreportedapositiveexperience(seeFigure6,right).Studentswhoparticipatedinrecent'robust'Moodlee-examtrialsinsemesters1and22018providedresponsestoaseriesofsurveyswhereapracticesessionandthreeexameventswererunineachcourse.ThisseriesispresentedinFigure7withfurtherfindingsinHillier,GrantandColeman(2018).Additionalreportsofe-examtrialsincludeHillier(2015),HillierandGrant(2018),HillierandLyon(2018b)andPagram,Cooper,JinandCampbell(2018).

Figure7StudentsurveyresponsesfollowingrobustMoodleexams


Arangeofotherissueswerealsoputtostudentsincludingassessment-writingstrategies,habitsandstyles(seeHillier&Lyon,2018a).Whenitcametowritingincomputerisedexamsthetwoissuesmostfrequentlyraisedbystakeholderswerehandwritingneatnessandtypingprowess,withthelatterthoughttobeapotentialsourceofdisadvantagewhenadoptinge-exams.AsseeninFigure8,studentsthoughtthatmessyhandwritingwasthemoreprevalentproblemincomparisontothevastmajorityofstudentswhofelttheirtypingwasspeedwasgoodenoughforane-exam.

“Myhandwritingisneat”

“Mytypingisfastenoughforexams”

Figure8Handwritingneatnessandtypingprowess

3.3 LogisticsfindingsOurexperiencewithrunninge-examsessionsraisedanumberoflogisticalmattersthatneedattention.Feedbackfromstudentsandacademics,aswellaspracticalexperiencefromteammembers,providedindicationsofthevarietyofissuesfacedbyinstitutionsoperatinginverydifferentcontexts,includingin-class,centralexamhalls,multi-campusandexternalvenues.ManyoftheparticipatinginstitutionswererunningaBYOlaptope-Examforthefirsttimeandthisisreflectedtosomedegreeinthefeedbackandintheminicasesofthesetrials(seesection3.4).TheteamsatMonashandUTashadpriorexperienceofexamsandtheirfocuswasonmovingonwardtousepost-paperandnetworkedexammodalities.

Themostrecentphaseofworkduring2018demonstratedtherobustnatureofthetechnicalapproachusedinthisprojectforclient-servercomputerisedexams.Weidentifiedthatonlineexamsfaceasignificantriskoffailureifnetworkconnectivityisrelieduponforthedurationoftheexamsession(Hillier,Grant,&Coleman2018).Wehaveshownhowthee-Examplatformaddressesthisriskwiththeabilitytorecoverfollowingsystemcrashesandnetworkoutageswithoutlossofstudentdataandonlyminimaldisruptiontoworkingtimeintheexam.Thee-Examplatformallowsworktocontinueinanexamevenifthenetworkconnectivityispermanentlylostinasession.Thismeansthatexaminationsneednotbeinterruptedorrescheduledandtimetablesneednotbedisrupted.

Theapproachofusinga'BYOlaptopfirst'approachtoequipmentprovisionhasmeantadditionalchallengesthanwouldhavebeenthecaseifusewerelimitedtoinstitution-ownedcomputers.Theaddedcomplexityisduetothedifferingstart-upstepsrequiredacrossdifferentdevices.Thishasbeenrecognisedwitharecommendationstorunpracticesessionsforstudentsandtheneedtofurtherdevelopsupportresourcessuchasanonlineself-helpserviceforindividualusers.Short'Quickstart'guideshavebeendevelopedtoadviseusershowtostarttheirlaptopwiththee-Examsystem.Anadditional'Detailedstart-upguide'wasdevelopedwithin-depthinstructionsandtroubleshootingadviceforawiderangeofhardwaretypes.Theseareavailablefromtheprojectwebsite.


Arangeofguidelinesarerecommendedtominimiseproblemsatvenuesusedforpracticesessionsandexamevents:

Runpre-exampracticesessionstoensurestudentsandstaffareprepared.Checkeachstudenthasacompatiblelaptop,itsbatterylifeissufficientfortheplannedexamdurationandthatthestudentsarefamiliarwiththeprocedures.Keeprecordstoensureexameventscanbeadequatelyprovisionedwithsparelaptops,wiredmice,powersockets,enoughe-ExamUSBsticksandspareWi-Fidongles(whereonlineexamsaretoberun).

WhenBYOlaptopsareused,expecttosupplyapproximately10percentofinstitution-ownedcomputersintheearlystagesofe-examuse.Acomputerlabmaybesetasideforthispurposeoratrolleyoflaptopsmadeavailable.Allowingstudentstobookaloanlaptopisaviablestrategy,especiallyintheearlystagesofe-examintroductionorwherelargercohortsoflowSESstudentsarepresent.Ashiftofresponsibilitycanoccurascohortsbecomeaccustomedtoequipmentrequirements(i.e.placeresponsibilityonstudentstosourceanalternativeworkinglaptop).UTashasshownthatmovingtoapre-certificationprocess(viahelpdeskorself-check)andcompatibilitycertificateisalsopossiblewithmaturityofane-examprogram.

Providepowersocketstoreducerelianceonbatterypower—ideallyonesocketperstudent,butthisratiomaybereducedforshorterexamsandwherestudentshavebecomemoreaccustomedtotherequirementsofe-exampractice.Powerextensioncordsandmulti-outletpowerboardscansupplementsuppliesinspaceswhereinstalledsocketsmaybefewinnumber.Inexamhallswithrowsofdesksitisrecommendedtoreserveeachsecondrowforpowersupplycablestoavoidtrippinghazardswhilestillallowinginvigilatoraccesstoeachstudent(seeFigure9,left).Surveysof510studentparticipantsconductedduringprojecttrials(2016–2018)askedforanecdotalreportsregardingthebatterylifeoftheirlaptopfromafullcharge(SeeFigure9,right).Ameanof4.9hourswasfound;however,therewasawiderangefromzerotoeight(plus)hoursreportedwithastandarddeviationof2.2hours.Thedatawereabnormallydistributedandthislikelyreflectsthefactthatstudentscomealongwitholdandnewlaptopstotheexaminvaryingconditions.Ifweexcludenewcomputers(thespikeof8+hours)thenthemeanbatterylifeisfourhours.Thosereportinglowbatterylifeoftwohoursorlessaccountedfor16percentofthegroup.Itmustalsobenotedthatbatteryconsumptioncanbehigherwhenusingthee-ExamUSBsystemorWi-Fi.

Layingpowercablesinanexamhall Studentself-reportedlaptopbatterylifeFigure9Powersupplyfore-Exams:layingcablesandbatterylife

0

20

40

60

80

100

120

0 1 2 3 4 5 6 7 8+

Num

berofusers

Bawerylifeinhours

Self-reportedlaptopba]erylife


Roomsandseatingarrangements:• Utiliseflatfloorroomswithchairsandtables• Wheregrouptablesareusedhavestudentssitat90degreestoeachother• Possibility:addcardboardcrossscreensordividersontables• Possibility:userandomorderforquestionsanddistractorswhereavailable• Whererowsareused,seatingallfacingfront>>>>>aresatisfactoryoralternating

face-to-face,back-to-back<><><><>mayreducethevisibilityofother'sscreens• Avoid'flappychairs'duetothembeingunstableforlaptopsandprovideinsufficient

spaceforlaptopsandotherexammaterials• Avoidtightlypackedtieredroomsduetorestrictedaccesstoeachstudentandthe

potentialforseeingother'sscreens• Preferroomswithquietacoustics—carpetedfloorsarerecommended.Outof1033

respondents,51percentcouldheartyping,ofthose28percentfounditdistracting(12percentofthewholegroup)

• Considerspacetoallowtypistsandhand-writerstositseparately• Itisimportantthatroombookingsaretimelytoensureasuitablevenueissecured.

GoodWi-Fiorwirednetworkingisneededwhenusingnetworkede-exams.However,ifusingthe'robust'e-examsystem,networkconnectivityismostimportantatthestartoftheexameventandcertainlyconvenientattheendoftheeventforresponsesubmission.ThetrialsconductedatMonashUniversityfoundWi-FitobeaviableoptionforBYOlaptop-basedexams.TheWi-Ficompatibilityoflaptopswasfoundtobelowerthanforthee-Examsystemingeneral.AsetofUSBWi-Fidongles($12eachfromcomputerpartsretailers)canbeusedtoboostthestockofcompatiblelaptopsandarequicktodeployintheexamroom.

Multi-campusandexternalsitesneedextraprecision.Ensureadequatetimeisallowedforsendinge-ExamUSBstootherlocations,thatadditionalspareUSBsaresentandthatclearrulesandcommunicationchannelsareestablished.Trialsatmulti-campusinstitutionsoratoff-sitelocationsexperiencedproblemswhentheseelementswereneglected.

3.4 MinicasesAseriesofeightminicasesareavailablefromtheprojectwebsitethatrepresenttherangeofe-exammodalitiesthatwereundertaken.Thecasesinclude:

• CentralQueenslandUniversitysemester12017:Paper-equivalentICTmanagementexamconductedatmultiplecentrallycontrolledon-campusvenues.

• EdithCowanUniversitysemester12017:Paper-equivalentexamindesigneducationandpost-paperexaminprogrammingeducationconductedin-class,oncampus.

• MacquarieUniversitysemester12017:Paper-equivalentICTeducationexam.• MonashUniversitysemester12017:Paper-equivalentexamusingaspreadsheetas

adatacollectionformwith'airgap'separationofresponsecollectionandevaluation.• MonashUniversitysemester12018:Post-paper'robust'onlineMoodleexamsthat

wereresistanttonetworkoutages.Moodlequizincludingaudioprompts.• TheUniversityofQueenslandsemester12017:Paper-equivalentFrenchlanguage

translationexamconductedin-class,oncampus.• UniversityofSouthAustraliasemester12017:Paper-equivalentmathseducation

theoryexamconductedoncampus.• UniversityofTasmaniasemester12017:Post-paperworddocumentexamsinICT

educationconductedovermultiplecentrallycontrolledvenuesanddistancecentres.


Chapter4. DisseminationAsignificantfeatureofthisprojectwasitsstrongdisseminationstrategy.Theprojectleaderandteammembersprovidednumerouspresentationsandconductedseveralworkshops,bothlocallyandinternationallyinconferencesandotherprofessionalforums.ThesedisseminationactivitiescreatedawarenessandgeneratedinterestfromhighereducationorganisationsacrossAustraliaandothercountries.Themaindisseminationstrategiesareoutlinedinthefollowingsections.

4.1 WebsiteandsocialmediaAprojectwebsitewasestablishedathttp://transformingexams.comtoprovidepublicinformationandmostoftheprojectdeliverables.Itincludeslinkstoinformation,softwaredownloads,documentation,presentations,publicationsandnotificationsforevents.Overthelifeofthewebsiteitreceived16,793visitorsand37,478pageviewswererecordedatthehomepage.VisitorsfromAustraliamadeupapproximatelyonethirdoftheaudience.Thetop10countriesareshowninFigure10.Figure10Websitevisitorsbycountry

SocialmediasuchasLinkedIn,TwitterandTransformingAssessmentemaillistwereusedtopromoteprojecteventssuchaspresentations,theroadshowworkshopsande-Examsymposium.

4.2 PresentationsandpublicationsFormaloutputsofresearchandprojectprogressweredisseminatedvia:

• over60presentationsofferedatconferencesandaspartofprivatebriefingsessionsatindividualinstitutions

• atotalof15refereedpublicationsproducedfrom2015to2018.Furtherpublicationsarebeingpreparedbytheprojectteam

• additionalinformalposters,documentsanddiagramsweremadeavailable.Afulllistofdisseminationeventsandpublishedoutputsarelistedathttp://transformingexams.com/research.html

Theseoutputsservedtospreadnewsabouttheprojectanddisseminatefindingsfromtrialsastheprojectprogressed.Thisalsoprovidedopportunitiesfornetworking,gatheringofinterestprojectamongcolleaguesandelicitingfeedbacktotheprojectteamonprogress.

Thewidereachofthepresentationsgivenbyprojectmembersisdemonstratedinthelistofconferenceswherepresentationswereoffered.Theseinclude:ACMInternationalComputingEducationResearch(ICER)Conference2016,AustralianCouncilforComputersinEducation2016,AustralianGovernmentOfficeforLearningandTeachingConference2016,AustralianLearningAnalyticsSummerInstitute2016,AustralasianSocietyforComputersinLearninginTertiaryEducationconference2017and2018,Assessmentintegrityandtheroleofexaminations2017,AssessmentinHigherEducationconference2017,EDTechPosium(ACT)2017and2018,ExaminationsNetworkConference2017and2018,HigherEducationResearchandDevelopmentSocietyofAustralasia2015and2018,InternationalConferenceone-LearningandInnovativePedagogiesConference2019,InternationalComputingEducationResearch2016,InternationalFederationofInformationProcessingOpenConferenceforComputersinEducation(Austria)2018,InternationalFederationof

Figure10Websitevisitorsbycountry


InformationProcessingWorldConferenceonComputersinEducation(Ireland)2017,OpenandDistanceLearningAssociationofAustraliaBiannualconference2017,TertiaryEducationQualityStandardsAuthorityconference2018andMoodleMoot2018.

4.3 e-ExamcasestudiesAseriesofeightminicasestudiesweredevelopedtooutlinethecharacteristicsandoutcomesofselectede-examtrialsconductedatdifferentpartnerinstitutions.Eachone-pagecasestudyshowcasesane-examusingdifferentaformatincluding:offlineword-processingdocument,spreadsheetasaform,computerprogrammingandonlineLMSuse.Thecasescanbeobtainedfromtheprojectwebsite.

4.4 e-Examworkshopse-Examshands-onworkshopswereconductedforthehighereducationcommunityandforprojectteammembers.Thesesessionswere:

17February2017:‘e-Examprojectpartnersworkshop’,heldatMonashUniversity,Caulfieldcampus,Melbourne.Thisprivateworkshopprovidedanoverviewforprojectpartnerspriortorunninglivetrialsattheirsites.8September2017:‘Runningauthentice-examsworkshop’atExaminationsNetworkconferenceinHobart.Thispublicworkshopfocusedone-examadministrationandlogisticsforexamsofficersatAustralianuniversities.7December2017:‘Transformingexams—HowITworksforBYODe-Exams’,ASCILITEconference,Toowoomba.Thispublicworkshopprovidedanopportunityforsomehands-ontimewithdifferenttypesofassessmentsthatarepossiblewithe-Exams.Thesessiontargetedacademics,learningdesignersandadministrativestaff.

4.5 e-ExamroadshowAseriesofroadshowpresentationsandworkshopswereconductedin2018andwerehostedbyprojectpartnerinstitutionscoveringmostcapitalcitiesinAustralia.Sessionswerefreetotheeducationcommunity.Keyteachingandlearningleaders,academicsandsupportstaffwereinvitedfromlocalinstitutionswithinthehoststate.Invitationswereissuedtostaffinuniversities,vocationaleducationprovidersandstateeducationauthorities.

Eachroadshowconsistedoftwoone-hoursessions.Thefirstsessionexploredtherationaleandsurroundingissuesfore-Exams.Thesecondsessionexploredpracticalapplicationsofe-examsandincludedhands-ontime.Theprojectleaderledthesessionsandwasoftenjoinedbymembersofthelocalprojectteamatthehostinstitution.Anexampleprogramisbelow:

1hour:Part1:Rationaleforauthentice-Exams.Forleaders/decisionmakers,allwelcome• Anargumentforauthentice-Examsforemployability.• Abusinesscaseforauthentice-Exams.• Keyfindingsonstudentfeedback.• Latestdevelopmentson'robust'Moodleuse(phase3development).

30min.break.(cateredfood/drink).1hour:Part2:Exploringthedetailsofe-Exams.• Workshopthefindingsonstudentfeedback,logistics,examplesofassessments.• Workshopaselectionofproposedprojectresources(e.g.samplecasestudiesto

influencedecisionmakers).• Timeforpractical'hands-on'withour'robust'Moodlee-Examsystem.


Thelocations,datesandregistrationnumbersforroadshowsessionswere:Adelaide(31August:50+30online),Brisbane(24September:27),Canberra(31October:29),Perth(17September:40),andSydney(21September:23),Hobart(5October:20).TheAdelaidesessionprovidedonlineaccessvialivewebvideoconferencingforthefirstsession.TheHobartsessionwasalsobroadcastviavideoconferencetoLaunceston(Newnham),BurnieandCradleCoastcampusesofUniversityofTasmania.

4.6 e-ExamssymposiumThee-ExamSymposiumwasorganisedbytheprojectteamandheldon24November2018asafull-dayeventinMelbourneatMonashUniversity,Caulfieldcampus.KeyrepresentativesfromeachAustralianuniversity,stateeducationdepartments,medicalcollegesandothereducationorganisationswereinvited.Theninespeakersand86participantsfromAustralia,Austria,Finland,NewZealandandSwedenrepresented23universities,twoprivateproviders,fourprofessionalmedicalcolleges,fourstategovernmenteducationdepartments(Victoria,NewSouthWales,Tasmania,WesternAustralia)andtheTertiaryEducationQualityandStandardsAgency(TEQSA).

Aseriesofplenary-stylepresentations,discussionsectionsandapracticalhands-onsessioncoveredmultipleaspectsofimplementinge-exams(seeFigure11).Twointernationalguestspeakerspresentedontheimplementationofe-examsintheircontextusinghome-developedapproachesandtechnologythatfocusedonauthentice-assessment.ThisincludedcomputerisednationalmatriculationexamsinFinlandanduniversityexaminationsinAustria.Themajorityofprojectpartnersalsopresentedorattendedtheevent.

Figure11Photo:e-Examsymposiumsessions24November2018

Thesymposiumprogramisoutlinedbelow.Welcome:Arationaleforafocusonauthentice-examsandprogramoverview.Session1:TheAustrianexperiencewithe-Exams.GabrieleFrankl(Alpen-Adria

Universität,Klagenfurt,Austria)Session2:Thepedagogyofe-Exams—examplesandtransition.MathewHillier(Monash

University),MatthewBower(MacquarieUniversity)andAndrewFluck(UniversityofTasmania).

Session3:Qualitativeanalysis,studentconcerns:thegoodthebadandtheugly.DavidMeacheam(UniversityofNewSouthWales,Canberra/ADFA),AmyHubbell(TheUniversityofQueensland)andCristinaSavin(Student,MonashUniversity).

Session4:Anationalrolloutofe-examsforhighstakesMatriculation.ThomasVikberg(FinnishMatriculationExaminationBoard,Finland)


Session5:Trade-offs,benefitsandcosts.AndrewFluck(UniversityofTasmania)andMathewHillier(MonashUniversity)

Session6:e-Examssecurityandintegrity.MichaelCowlingandKennethHowah(CentralQueenslandUniversity)

Session7:Runninge-Examsoncampusandatadistance.AndrewFluck(UniversityofTasmania)andMathewHillier(MonashUniversity).

Session8:Handson!Technologyformovingfrompapertoauthentice-assessment.MathewHillier(MonashUniversity).

Session9:Ane-ExamsSociety?Anopendiscussiononwheretonextforthee-Examsproject.MathewHillier(MonashUniversity)andAndrewFluck(UniversityofTasmania)[Notrecorded].

Thesymposiumproceedingswerevideorecordedandplacedonlinewithopenaccess.Anoutlineofeachsession(1to8),slideset,linksandvideosareavailableat:http://transformingassessment.com/e-exam_symposium_2018.php


Chapter5. ImpactTheprojecthashadadirectimpactintermsofdevelopingknowledgeofe-examsimplementationandtheadoptionofprojectdeliverablesinasmallnumberofcases,butlargerimpactsmaybecomeevidentinthefuture.Itisstillearlydaysforthefullimpactoftheprojectworktobecomeapparent.Thecurrentstateofimpactasatthecloseoftheprojectisoutlinedbelow.AnannotatedIMPELframeworkispresentedinAppendixD.

5.1 ChangedpracticesTheprojecthashaddirectimpactintermsofknowledgeofparticipatingacademicsthatwereengagedintherunningofe-examtrialsattheirrespectiveinstitutions.Theoutcomesvariedaccordingtocontext.Someacademicscontinuetousee-Examapproachesdevelopedaspartoftheproject.TheseincludeattheUniversityofTasmaniaineducation,atMonashUniversityinlanguagesandattheUniversityofSouthAustraliainmathematicseducation.Inothercases,lessonslearntprovidedavaluablebasisforfutureconsiderationofe-examsadoptionincontext.Theseoutcomeswerereportedaspartofthee-Examssymposiumandviapublications.

5.2 Othere-examprojectsTheprojectteamwasconsultedbyotherorganisationsintheAustraliancommunityonmattersrelatingtoe-exams.Projectsorevaluationsrelatingtoe-examsatotheruniversitiesandorganisationswereindependentlyinitiated,someoftheseonlydiscoveredafterthefact.Oftenproponentsfoundourpublications,ourprojectwebsiteorweretoldaboutourworkviawordofmouth.Examplesinclude:UNSWSydneywhereseparateprojectsoccurredinlawandmedicine.AtUNSWCanberrawholaterjoinedtheprojectformallyandatUQinmedicine,whosoughtadvice.AtNAATIwithwhomaspin-offprojectwasdevelopedtointegrateamulti-languagecapability.AcommercialcomputerisedtestingvendorintheUnitedStateshassoughtconsultancyandadvicefromtheprojectleader.Additionally,aprofessionalaccountingbodyrecentlysoughtconsultancy.Wehavealsoreachedagreementtopilot'robust'e-ExamswiththeLMS(Canvas)during2019ofastategovernmenteducationdepartment.

5.3 PartnershipwithNAATIAsignificantpartnershipwasdevelopedwiththeNationalAccreditationAuthorityforTranslatorsandInterpreters(NAATI)inparallelwiththisproject.NAATIcommissionedworktowardsbuildingadditionalmulti-languagecapabilitiesintothee-Examsystemtobettercaterformulti-languagetranslationexaminations.PreliminaryroundsoflivetrialswereconductedatMonashUniversityin2017and2018coveringadefinedsetoflanguages.Atthetimeofwriting,workisongoingtofurtherenhancee-Examsystemcapabilities.Lessonslearntfromtheapplicationofe-ExamstechnologyinthespecificcontextofprofessionalaccreditationforlanguagetranslatorshasalreadyhadadirectimpactofhowNAATIisconsideringtheconductofcomputerisedexamsinthefuture.Alarge,high-levelimpactcouldbeexpectedintheareaoflanguagetranslatortestinginAustraliashouldtheoutcomesofthisworkbeadaptedbyNAATI.

5.4 PotentialjointventurepartnershipandcommercialisationSignificantinterestwasgarneredfromseveralorganisationsoutsideofAustraliawithaview


topartneringinajointventureorcommercialisationarrangement.Atthetimeofwriting,twomembersoftheprojectteamareindiscussionwithanorganisationfromtheEuropeanUniontopartnerinajointventurewithaviewtocommercialisingelementsofthetechnologyandapproachesdevelopedaspartofthisproject.

5.5 FurtherresearchandjointprojectworkSignificantinterestwasreceivedfromastate-leveleducationauthorityregardingextendingthe'robust'client-serveraspectoftheonlinee-examstechnologyforlocalusewiththeirLMS(Canvas).Asmallfeasibilitystudyisunderwayatthetimeofwritingwithplansforasmall-scaletrialinmid-2019.


Chapter6. ProjectevaluationandlessonslearntWhiletheprojectwasultimatelysuccessfulinachievingouraimofenablingauthentice-assessmentinexamhallsandclassrooms,someareasofworkweremorechallengingthanfirstanticipated.Formalexternalprojectevaluationandinternaldeliberationshighlightedsuccesses,challengesandlessonslearnt.

6.1 ProjectexternalevaluationTheprojectmanagementteamheldmeetingswiththeexternalevaluatorswhowerebriefedontheprogressoftheproject.Anarms-lengthfinalevaluationwasalsoconducted.Theexternalevaluationsassistedinthelearningandimprovementofprojectprocessesandoutcomes.

Twoexternalevaluationswereproduced:

I. Aformativemid-projectreportwasproducedin2017.Thiswasformulatedasasurveyofstakeholderviewsbytheexternalprojectevaluator.Thereportoutlinedpointsofsuccessandissuesfortheprojectteamtoaddressgoingforward.Dataweregatheredviasurveyingandtelephoneinterviewswithprojectstakeholders.AsummaryofthisformativeexternalevaluationisincludedasAppendixE.

II. Asummativefinalreportproducedattheendoftheprojectin2019.RefertoAppendixFforthefinalevaluationreport.

Astheprojectprogressedtwomainareasemergedthatweredifferenttoinitialexpectations.Thefirstofthesewasthenatureofchangeneededininstitutionalpoliciesandtheotherwasinthedetailsoftheunexpectedlogisticalchallengeswefaced.

6.2 PolicyadjustmentOneoftheproposeddeliverablesfromtheprojectwastoprovidepolicychangerecommendations.Thiswasbotheasyanddifficultatthesametime.

Firsttheeasypart!Followingareviewofseveralinstitutionspoliciesitwasfoundthatverylittledirectpolicyonexaminationsstoodinthewayofusingcomputersinexamsatmostuniversities.Typically,currentpoliciesorproceduresspecificallyrelatedtotheconductofexaminationshadphrasessuchas‘Noelectronicdevicesarepermittedintheexaminationvenues’.Minoradjustmentstopolicywordingcanclearthewayforcomputerstobeusedinexams.Forexample,rewordingsuchphrasesto:‘Nounauthorisedelectronicdevicesarepermittedintheexaminationvenues’or‘Onlyspecificallyauthorisedelectronicdevicesarepermittedintheexaminationvenues’.Itisthennecessarytoestablishaprocessforauthorisingelectronicdevices.OncesuchexamplewasdevelopedattheUniversityofTasmania.Astudentself-helppracticeandlaptop-testingprocessprovideda'compliancecertificate'attestingtothestudent'ssuccessatstartingandusingthee-ExamUSBsystemwiththelaptop.Thecompliancecertificateisthenshownbythestudentattheexamvenue.

Institutionsmaybeinclinedtoshowactivesupportfortheuseofcomputersinexaminationsthroughaformofofficialendorsementatseniorlevels.Examplesinclude:theUniversityofTasmaniasenatepassingaresolutiontosupporttheuseofe-Exams.AtTheUniversityofQueenslandamotionwaspassedbythecentralassessmentcommitteetosupportandplaceconditionsaroundthetriallingofe-Exams(sincelifted).However,otherinstitutionsinvolvedintheprojecthavenotseenitnecessarytoformallyendorseorplace


conditionsontrials.Inthesecases,theethicsprotocolandresearchsubcontractwasseenasadequatetoframeparticipationintheproject.

Difficultiesarise,however,increatingtheconditiontosupportofbroaderadoption.Moresystemicpolicyandmanagerialattitudeswerefoundtobesignificantinmovingonfromintroductionofe-Examsataninstitutiontobroader,centralisedsupport.Thepolicylandscapeisinterconnectedandinterdependent.Thiscansupportordetractfrome-Examsimplementationanduse.Contextualpolicies,proceduresandofficialendorsementisrequiredtoactivelyandsystemicallysupporttheuseofcomputersinexamsatinstitutions.Examplesincludedevelopinga'BYODstrategy'tocoverinfrastructure(power,Wi-Fiandnetworking),an'equipmentequityprogram','equipmentcertification',additionstoexamclassifications,additionstopermittedmaterialslists,newe-examtemplates,developmentofsupportresourcesandhelpdesksforstudentsaswellasprofessionaldevelopmentforacademicandprofessionalstaff.Muchofthisvaluablework,however,wasbeyondthescopeandresourcesofthecurrentprojecttoaddress.

Onapositivenoteforthefuture,ithasbecomeapparentthatsincethebeginningofthisprojectinstitutionshavebeguntodevelopbroader'digitallearning'strategiesthathaveencompassede-examsore-assessmentimplementation.AnexampleofthisisatMonashUniversitywherea'DigitalLearningStrategy'androadmapwasdevelopedthatincluded'Authentice-Assessment'asoneofthetargetitemsalongsidea'BYODstrategy'.

6.3 LogisticsandcommunicationswithinpartnerinstitutionsTheimplementationofe-examtrialsatpartnerinstitutionssawobstaclesencounteredthatgenerallyarosefromthedivergentobjectivesofindividualgroupswithinanorganisation,aswellaslongchainsofcommunicationwherekeymessagesbecamelostordiluted.Examplesofmanifestedproblemsthatoccurredattrialpartnersinclude(innoparticularorder):

Thecentralexaminationsofficehaditsownsurveyforstudents.Thiswasthecaseatonepartnerinstitutionthatimpactedacoupleoftrialsrunatthatsite.Inprevioustrialsatthesamesitetheissuedidnotarise;however,theexamsofficethemselvesdecidedtoevaluatee-examsandsotheyproducedtheirownstudentsurvey,whichdemonstratedhandwritingcandidatesadjacenttoe-Examcandidateswereunaffected.Alackoftwo-waycommunicationresultedinconfusionforstaffinexamroomsthatmeantthatoursurveywasnotprovidedtostudents.Thishighlightsaneedforcontinuedandclosecollaborationwithexamvenueoperatorsforeachtrialinstance.Itcannotbeassumedthatbecauseanarrangementwasinplacepreviouslythatitwouldcontinueinthefollowingsemester.Hadbettercommunicationoccurreditwouldhavebeenpossibletodevelopacombinedinstrumentthatwouldhaveservedtheneedsofbothparties.Fortunately,dataweresuccessfullycollectedduringthemajorityofthetrials.

Lowstudentsurveyresponseratesatsomevenues,oftenduetodelayedsurveyavailabilitytostudents.Insomecasessurveyswerenotadministeredtostudentsimmediatelyfollowingexameventsanddramaticallyreducedresponserateswereexperienced.Itwasrecommendedthatthesurveybehandedtoeachstudentatthetimetheycompletedtheassessment(i.e.intheexamroomitself).Thisneededtobearrangedwiththegrouprunningtheexamvenue(e.g.examinationsoffice)andforthemtocollaboratewiththeprocess,suchastoprovideanannouncementattheendoftheexamtocompletethesurveyformandallowamemberoftheprojectteamtobeon-site,ifpossible,todistributeandcollectsurveyforms.


Equipmentprovisionandbackup.BYODwasfoundtobeasuccessfulequipmentprovisionstrategybutitwasnotwithoutitsproblems.Usingself-certificationoflaptopsandpre-exampracticesessionsishighlyrecommendedtofacilitatepreparednessofstudentsforBYODe-exams.Intheearlyphasesofimplementinge-examsthereislikelytobeaheightenedneedtoprovidespareorbackuplaptops,withusersinneedbestidentifiedviapracticesessions.Hardwarecompatibilitywasfoundtobeanissuein10to20percentofcases.Similarly,provisionofelectricitysocketstoeachstudentisrecommended;however,theratiosofpowerandbackuplaptopsmayberelaxedasstudentsbecomeaccustomedtoneedingareliablelaptopforexams.Mostpracticalissuesencounteredinexamroomsduringtheearlystagesofe-Examusecanbegreatlyalleviatedbyprovidingbackupequipment,powerandenforcingpracticesessionattendance.

Provisionofsparelaptopsatremoteexamcentres.Someinstitutionsfounditdifficultduetoresourcesnotbeingallocatedorundesirabletoprovidebackupandon-sitesupportweremultiplevenuesorexternalexamcentreswereused.Atoneinstitution,ane-examwasofferedindistancemodeatanoff-campusexternalexamcentre(e.g.anotheruniversitybyagreementorsomeotherpremisesbyarrangement),however,noarrangementwasputinplacefortheprovisionofsparelaptops.Thiswasexacerbatedbecausenopre-exampracticeorsystemcheckswereconducted.Thisresultedinsomestudentsneedingtoreverttopaper-basedexamsonthedayoftheexam.Itisacknowledgedthatitisalogisticallydifficultissuetoovercome,however,itissomethingthatmustbeaddressedife-examsaretobewidelyavailable.

Expectationsabouttechnicalmaturity.Theneedforguidanceanddocumentationforalluserswashighlighted(students,teachersandexamvenuestaff).Duringtheprojectthee-Examsystemwasstillundergoingactivedevelopmentwhereiterativeimprovementstothesystemwerebeingmadefollowingeachtrial.Thesecircumstancesmeantthatthosetriallingthesystemforthefirsttimeweremorelikelytoencounterproblemsthathadnotyetbeenencounteredbyothers.Furthermore,localorinstitutionaltechnicalsupportvariedmeaningthatthosewhowereleadingtrialshadtobesubstantiallyself-reliant,beyondwhatwouldbeexpectedifthee-Examsystemweretobeaninstitutionallysupportedtool.Althoughallpartnerswereclearlymadeawareofthesecircumstances,issuestendedtoarisewheretechnologyskillsorlocalsupportwaslackingorwherestakeholderssoughttocomparearesearchprojecttothatofamatureproductbackedbyafullservicesupportcontract.Wherethesecircumstancesprevailedthistendedtoresultinadegreeofdisappointment.Supportiveelements,suchasongoingrefinementoftheuserdocumentation,wasundertakentomatchchangesinsoftwareandhardwarecapabilitiesbuttheiterativenatureofthedevelopment,limitedprojectresourcesandthetimingofindividualtrialsdidnotalwaysresultinalignment.Furtherworkisalsoneededtoaddressthewishesandneedsofusers.Thereisalsoawordofcautionintheselectionofearlytrialpartnerswhoneedahighdegreeofresilienceintacklingtheunknowninnewcontexts.Thegoodnewsisthatthatforthosewhowerepersistent,goodoutcomeswereachieved!


Chapter7. ConclusionThisprojecthadgreatsuccessindemonstratingwhatrobust,authenticdigitalassessmentcouldbelike.Butthefindingsshowourworkisnotyetcomplete—notechnologydevelopmentprojectisevercomplete!Furthertechnicalworkisrequiredtoimprovethee-Examtechnologyplatform,itsintegrationwithothersystemsanditsadministrativesupporttools.Whilewehavelaidthegroundworkandhavealreadyachievedmuchonthepedagogicalfront,moreremainstobedonetoenhancethescalabilityandefficiencyoftheapproach.Furtherinvestigationandinvestmentisrequiredtoensuresystemicsupportmechanismscanbedevelopedtoscaledigitalassessmentathighereducationinstitutions.Furtherworkisneededonrefiningprocesses,professionaldevelopmentcapabilities,studentsupport,alignmentofvariousinstitutionallearningandtechnologystrategiesandpolicy;aswellasenhancingtechnologyinfrastructuretocopewiththehigherdemandsoftime-limited,higherstakesauthentice-assessment.Finlandhasdemonstratedthepotentialofanationalapproach—thismayalsoberequiredinAustralia.Tappingintointernationalandcross-sectorcollaborationinvolvingschools,vocationaleducation,professionallearningandinternationalhighereducationsettingswillenablefurtherleveragingofthelessonslearntinthisproject,aswellaslearningfromother'ssuccessindeployingdigitalassessment.

Lookingtothefuture,thereismuchpotentialtoleveragedevelopmentsinfieldssuchaslearninganalytics,openknowledgesystems,mixedmediatechnologiesorusingblockchaintechnologytoenhancesecurityandauditcapabilitiesine-Exams.Whilethesearerelativelyunderexploitedareastheyholdgreatpromise,particularlyiftheintegrated,openarchitectureadvocatedinourapproachtothee-Examplatformcanbecomemorewidespread.However,Australianhighereducationisnowatacrossroadswithsignificantrisksforinstitutionsinhowtheytransitiontoagreateruseofdigitalassessment.Onesuchexampleistheuseofmanagerialisttechnologyselectionapproachesthattendtodiscountpedagogicalaffordancesforwhatappeartobequickandeasy-to-managesolutions.Similarly,adecisiontooutsourceassessmentsystemspresentsrealstrategicriskthatinstitutionscouldloseholdoftheassessmentprocesstoexternalinterests,ashasbeenthecaseinresearchpublishingandlearningcontent.Assessmentisoneofthejewelsinthecrownoflearningandteachingthatinstitutionsstillholdwithintheirgraspandisarguablythemostvaluableelementinthestudentlearninglandscape.Institutionswoulddowelltoconsiderverycarefullythestrategytheyadoptinmovingtodigitisingassessment.Ifthetruevalueandbenefitsofdigitalassessmentaretoberealised,institutionsmusttakebackownershipoftheireducationaltechnologyinfrastructureandreinvestinmorehome-growndevelopmentandinnovation,justastheydointhediscipline-basedresearchdomain.WeurgeAustralianhighereducationinstitutionstostakeaninvestedclaimindigitallearningandassessmentbecausethatwillgoalongwaytosecuringthefutureofeducationfortheirstudents,staffandwidersociety.

Theprojectgeneratedfindingsone-Examsystemdesignanduse,exammanagementandlogistics,students'perspectivesone-Examsandontheirresponsestrategies.Thesefindingsequipteachers,assessmentdevelopers,technologysupport,examinationsmanagersandinstitutionalleadershiptomakeinformeddecisionsinmattersofpolicy,practicalimplementationandtransformationalpedagogicdesign.Thiswillenableinstitutionstodeliverrich,relevant,assessmentfitforlearningandteachinginthe21stcentury.


ReferencesCowling,M.A.,Hillier,M.,&Birt,J.R.(2018,Nov).Integratingmixedrealityspatiallearning

analyticsintosecureelectronicexams.InM.Campbell,J.Willems,C.Adachi,D.Blake,I.Doherty,S.Krishnan,S.Macfarlane,L.Ngo,M.O’Donnell,S.Palmer,L.Riddell,I.Story,H.Suri,&J.Tai(Eds.),ASCILITE2018ConferenceProceedings—OpenOceans:Learningwithoutborders(pp.330-334).Geelong,ASCILITE.http://transformingexams.com/files/cowling-hillier-birt-ASCILITE-2018.pdf

Fluck,A.(2019).AninternationalreviewofeExamtechnologiesandimpact.Computers&Education,132,1–15.https://doi.org/10.1016/j.compedu.2018.12.008

Fluck,A.,&Hillier,M.(2016,Sep/Oct).InnovativeassessmentwitheExams.PaperpresentedatAustralianCouncilforComputersinEducationConference,Brisbane.http://conference.acce.edu.au/index.php/acce/acce2016/paper/view/34/27

Fluck,A.,&Hillier,M.(2017,July).eExams:Strengthindiversity.PaperpresentedattheWorldConferenceforComputersinEducation(IFIPTC3),Dublin,Ireland.http://transformingexams.com/files/fluck_hillier_2017_eExams_strength_diversity_IFIP_WCCE_paper_45.pdf

Fluck,A.,Adebayo,O.S.,&Abdulhamid,S.M.(2017).Securee-examinationsystemscompared:Casestudiesfromtwocountries.JournalofInformationTechnologyEducation:InnovationsinPractice,16,107–125.https://doi.org/10.28945/3705

Fluck,A.,Pullen,D.L.,&Harper,C.(2009).Casestudyofacomputerbasedexaminationsystem.AustralasianJournalofEducationalTechnology,25(4),509–523.https://ajet.org.au/index.php/AJET/article/view/1126/379

Hillier,M.(2015,Nov/Dec).Totypeorhandwrite:Student'sexperienceacrosssixe-Examtrials.PaperpresentedatAustralasianSocietyforComputersinLearninginTertiaryEducationconference,Perth.http://transformingexams.com/files/Hillier_2015_ascilite_fp.pdf

Hillier,M.,&Fluck,A(2013,December).Arguingagainfore-Examsforhighexaminations.Proceedingsofthe30thAustralasianSocietyforComputersinLearninginTertiaryEducation,(pp.385–397).MacquarieUniversitySydney,Australia.http://transformingexams.com/files/hillier_fluck_2013_ascilite_fullpaper.pdf

Hillier,M.,&Fluck,A.(2015,Nov/Dec).Apedagogicalendgameforexams:Alook10yearsintothefutureofhighstakesassessment.PaperpresentedatAustralasianSocietyforComputersinLearninginTertiaryEducationconference,Perth.http://transformingexams.com/files/hillier_fluck_2015_exam_futures.pdf

Hillier,M.,&Fluck,A.(2017,Dec).Transformingexams—HowITworksforBYODe-Exams.InH.Partridge,K.Davis,&J.Thomas.(Eds.),Me,Us,IT!ProceedingsASCILITE2017:34thInternationalConferenceonInnovation,PracticeandResearchintheUseofEducationalTechnologiesinTertiaryEducation(pp.100–105).Toowoomba.http://2017conference.ascilite.org/wp-content/uploads/2017/11/Concise-HILLIER.pdf

Hillier,M.,&Lyon,N.(2018a,June).Writinge-Examsinpre-universitycollege.PaperpresentedattheOpenConferenceonComputersinEducation(IFIPTC3),Linz,Austria.http://transformingexams.com/files/Hillier_Lyon_OCCE2018_writing_eexams_preuni_FP.pdf


Hillier,M.,&Lyon,N.(2018b,June).Studentexperienceswithabringyourownlaptope-Examsysteminpre-universitycollege.PaperpresentedattheOpenConferenceonComputersinEducation(IFIPTC3),Linz,Austria.http://transformingexams.com/files/Hillier_Lyon_OCCE2018_student_experiences_eexams_preuni_FP.pdf

Hillier,M.,&Grant,S.(2018,Nov).Do-it-yourselfe-Exams.InM.Campbell,J.Willems,C.Adachi,D.Blake,I.Doherty,S.Krishnan,S.Macfarlane,L.Ngo,M.O’Donnell,S.Palmer,L.Riddell,I.Story,H.Suri&J.Tai(Eds.),ASCILITE2018ConferenceProceedings—OpenOceans:Learningwithoutborders,Geelong(pp.121–130).http://transformingexams.com/files/hillier-grant-diy-eexams-ASCILITE-2018.pdf

Hillier,M.,Grant,S.,&Coleman,M.(2018,Nov).Towardsauthentice-Examsatscale:RobustnetworkedMoodle.InM.Campbell,J.Willems,C.Adachi,D.Blake,I.Doherty,S.Krishnan,S.Macfarlane,L.Ngo,M.O’Donnell,S.Palmer,L.Riddell,I.Story,H.Suri&J.Tai(Eds.),ASCILITE2018ConferenceProceedings—OpenOceans:Learningwithoutborders,Geelong(pp.131–141).http://transformingexams.com/files/hillier-grant-coleman-Towards-authentic-e-Exams-at-scale-ASCILITE-2018.pdf

Lane,B,(2009,November18).Laptopspassthebigexam.TheAustralian[MediaInterview].Retrievedfromhttps://www.theaustralian.com.au/higher-education/laptops-pass-the-big-exam/news-story/ea5759df6b9d6dbca751b5e4f3c4b4c9

Pagram,J.,Cooper,M.,Jin,H.,&Campbell,A.(2018).Talesfromtheexamroom:TriallinganE-Examsystemforcomputereducationanddesignandtechnologystudents.EducationSciences,8(4),article188https://doi.org/10.3390/educsci8040188

TransformingexamsacrossAustralia |

27

AppendixACertificationCertificationbyDeputyVice-Chancellor(orequivalent)IcertifythatallpartsofthefinalreportforthisOLTgrantprovideanaccurate

representationoftheimplementation,impactandfindingsoftheproject,andthatthe

reportisofpublishablequality.

Name:ProfessorSusanElliott,Positiontitle:DeputyVice-ChancellorandVice-President(Education)MonashUniversityDate:24May2018


28

AppendixBProjectdeliverabledetailsDiscussionpointscomparingtheoriginallyproposedprojectdeliverablesandtheactualprojectdeliverableoutputsareoutlinedinthetablebelow.

Originaldeliverable Deliveredoutpute-Examstechnicalinfrastructure:developedtofacilitateanelectronicworkflowincluding:Set-uptemplatesforacademicstocreateexams,set-uptoolsforstudents,examsubmissionservertoreceiveexamsfromacademics,production/retrievalinfrastructureformanagingthee-examsclientUSBs,electronicreticulationofresponsesvianetworks(possiblywireless),electronicdistributionofstudentresponsestomarkersinformatsconducivetoe-marking(includescollationofstudentresponsesintoclassgroupsandmergerofMoodledatabasestoenablecomputerisedmarking).Howeverthescopecurrentlyexcludesthefacilitiesforelectronicmarkingitself.

Delivered.Arobuste-Examplatformwasdevelopedandusedin35liveexamevents.ItsupportsBYOlaptoptoMoodleservere-examsthatareresistanttonetworkoutages,evenoverWi-Ficonnections.ComputerbasedmarkingutilisedMoodlequestionswhilee-submissionofcomplexconstructedresponsefileswereachievedbyusingafileattachmenttoaMoodlequizquestion.Theoriginallyintendedon-boardLMSmodelevolvedtouseaclient-servermodelduetotherelativeeaseofimplementingitasworkingsystem.Howeverlivenetworkingalsopresentedtechnicalchallengeswithrespecttonetworkoutagevulnerabilitiesduringanexamevent.Theseweresolvedbytheprojectteam'se-Examplatform.

Goodpracticeguides:forstudents,academics,examsoffice(adminprocedures,roomsetupandinvigilation)andITsupportpersonnel.Thisbuildsontheseedgrantdraftversionsofsomeoftheseguidesbutaddsfurtherguidesforacademicsandexamsofficefunctions.

Alternativedelivered.Theseevolvedintoasetofuserguidesforstudents,teachers,examinationsofficersandtechnicaladministration.Theseweremadeavailableviatheprojectwebsite.

Adraftsetofassessment/examspolicyrecommendationsforinstitutions:Thesewillbeintheformofasetofprinciplesforwritingpolicieswithaccompanyingexamplesratherthanspecificfullywrittenpoliciesasitisexpectedinstitutionswillwanttowritetheirown.

Alternativedelivered.Followinganalysisofasampleofassessmentpoliciesathighereducationinstitutionsitwasfoundthatverylittlechangewasrequiredtoexitingpolicies.Simplewordchangeswouldnormallysufficetoenablee-assessmentstooccur.Seesection6.1.foradiscussionandrecommendationsfore-Examstobesystematicallysupportedbycontextualenablingpolicies.


29

Originaldeliverable DeliveredoutputAsetofexamplee-examtemplatesandquestions:Thiswillleverageandadaptwheresuitablethe'TransformingAssessment'onlinee-assessmentexemplars(Crisp'sALTCfellowship,Hillier's2014OLTExtensiongrant).Thiswillbuildonexample'paper-equivalent'questionformatsdevelopedduringtheseedgrantandaddfurtherexamplesgatheredfrompartnerinstitutions.

Delivered.Therewere35trialsconductedacrossthreemodalities(paper-equivalent,post-paperandrobustonline)inarangeofdisciplinesusingworddocuments,spreadsheetsandonlineMoodlequizzes.Examsincludedcomplexconstructedresponsessuchascomputerprogrammingfilesandeditedlanguagetranslationpieces.AprototypeAugmentedRealitye-Examwasalsodeveloped(yettobetrialled).Asetofeightminicasesweredevelopedtohighlightthedifferente-examtrials.Furtherdetailsofeachexamtypeareavailablefrompublications.Theseresourcesareavailablefromtheprojectwebsite.

Researchfindings:Thepublishedresultsfromtrialscompletedatamultipleinstitutionsthatincludesanalysisofresultsofstudentexperience.

Delivered.Therewere15referredpublicationsand60presentationsoverthedurationoftheproject.Thesepublicationsincludedtheresultsofstudentexperiencefeedback.Afulllistisprovidedontheprojectwebsite'Research'section.Furtherpublicationsareforthcoming.

Workshopmaterialstointroduceacademicstoe-exams:tobeofferedatnationalT&LfocusedconferencessuchasHERDSAorASCILITEinyearstwoandthree.

Delivered.Workshopsdeliveredatconferencesin2017,inanationalRoadshowforeachcapitalcityduring2018,thee-ExamsSymposiumheldon24Nov2018.Workshopsarelistedontheprojectwebsite.RecordingsononeroadshowsessionandallsymposiumpresentationsareavailableviatheprojectwebsiteandonYouTube.

An'e-examsfoundation'drafttermsofreferenceandacallformembership:ThefoundationistoprovideforthelongtermtechnicalsupporttomaintainICTinfrastructureandprovidelicencestocommercialserviceproviders.ThebusinessandlegalmodelwilllikelybebasedonthesuccessfulMoodleopensourcearrangements,whichhasproducedasustainableoutcomeforthelongtermsupportofthatAustralianinnovation.

Alternativedelivered.Itbecameapparentthatthesectorwasnotinthefinancialormanagerialpositiontosupportaneworganisation.Aplenarystylediscussionwasheldwiththe86attendeesate-ExamSymposiumon24Nov2018.Thisincludedrepresentationfrom23universities.Whilstsomeinterestwasshown,onbalancetheprojectteamconcludedthatthemostlikelywayforwardwouldbeviajointventurewithaEuropeanuniversityorcommercialisation.Theprojectleadershipisnowpursuingtheseoptions.


30

AppendixCe-ExamtrialdetailsThetablebelowdisplaysallknowntrialsconductedusingthee-Examplatformtodate.TrialsconductedundertheInnovationandDevelopmentprojectincludesthosesince2015.Priorworkundertakenwithpredecessorversionsofthee-Examsystemarealsoincludedforcompleteness.Examcandidatecountsarelistedfortypistsandhand-writerswhereknown.

Instit'n Year S Course Mode QuestionTypes

Weight Time(mins)

Context TypedHandTotalStaff Grant

CQU 2018 2 EthicsandSocialIssues(IT)(COIT11223)TBC

Worddoc Shortanswer 50% 180 Outofclass,central,multicampus,finalexam

? ? ? MichaelCowling

OLTID

Monash2018 2 IntroductiontoChineselanguage(ATS1002-2002)

Moodlequiz,audiofiles

Shorttext,TF,MCQ,fillinblank.Pluslisteningtestshortresponse.

15% 60 In-class,midsemesterexam(alltype).10minlisteningtest+70minutewritingtest.

74 0 74 ScottGrant OLTID

Monash2018 2 Chineseonlinemedia(ATS3038)

Moodlequiz,DimsumandSyngChinesedictionarytools

Longtexttranslation,shorttext,matching,fillinblank

15% 60 In-class,midsemesterexam(alltype).In-class1hourtests.25%(12.5%x2)Weeks7&11.Final2hourexam45%

22 0 22 ScottGrant OLTID

Monash2018 2 LanguageTranslation(AGP5048)

Worddocandmultilingual.Spellingon.

Shorttext,longtext

30% 180 InClass-3hr.15minfinalexam+separate14NAATIstudents(total48).

40 8 48 ShaniTobias

OLTID

CQU ? EthicsandSocialIssues(IT)(COIT11223)

Worddoc Shortanswer 50% Outofclass,central,multicampus,finalexam

115 115 MichaelCowling

OLTID

UQ 2018 2 FrenchlanguageB(FREN3113)

Worddoc Longtext 30% 120 In-class,Schoolrun,finalexam,collaborativelearningspace

? ? 66 AmyHubbell

OLTID

UQ 2018 2 FrenchlanguageB(FREN3380)

Worddoc Longtext 30% 120 In-class,Schoolrun,finalexam,collaborativelearningspace

? ? 38 AmyHubbell

OLTID

UniSA 2018 1 MathematicsforSecondaryTeaching1(EDUC5184)

Worddoc ShortAnswertextresponsesApprox600wordseach

40% 120 Inclass 31 2 RuthGeer,BruceWhite

OLTID

Monash2018 1 IntroductiontoChineselanguage(ATS1001-2001)

Moodlequiz,audiofiles

Shorttext,TF,MCQ,fillinblank.Pluslisteningtestshortresponse.

16% 80 In-class,midsemesterexam(limitedto20typingslots).10minlisteningtest+70minutewritingtest.

14 NotSev110

ScottGrant OLTID


31


Weight Time(mins)


Monash2018 1 Chineseonlinemedia(ATS3037)

Moodlequiz,DimsumandSyngchinesedictionarytools

Longtexttranslation,shorttext,matching,fillinblank

15% 70 In-class,midsemesterexam

27 4 ScottGrant OLTID

Monash2018 1 LanguageTranslation(AGP5690).PracticeFriday25thMayfrom10to12noon+examTuesday5thJunefrom9amto1pm

Worddocandmultilingual

Shorttext,longtext

30% 180 InClass-finalexam

26 5 ShaniTobias

OLTID

UTAS 2018 1 EducationDigitalTechnologies(ESH380)

Worddoc. Shorttext,longtext,Scratchprogramming,softwareappraisal

47% 120 Final 125 20 145 AndrewFluck

OLTID

CQU 2017 2 EthicsandSocialIssues(IT)(COIT11223)

Worddoc,PDF,videofile

Shortanswer 50% 180 Outofclass,central,multicampus,finalexam

123 0 MichaelCowling

OLTID

ECU 2017 2 TeachingIntroductoryComputerProgramming(CSE4102)

WorddocandPythonIDE

Shorttext,longtext,programming

15% 120 In-class,computerlab

8 0 JeremyPagram

OLTID

ECU 2017 2 MaterialsDesignandTechnologyOHS(DTE4271)

Worddoc Shorttext,longtext,images

30% 90 In-class,computerlab

9 0 JeremyPagram

OLTID

Monash2017 2 IntroductoryChineselanguage(ATS1002)

Spreadsheetasform.

MCQ,T/F,fillinblank,shorttext

16% 70 Outofclass,mid-sem

16 57 ScottGrant OLTID

Monash2017 2 LanguageTranslation(APG5048)


Shorttext,longtext,editingtask


23 8 ShaniTobias

OLTID



Shorttext,longtext

30% 180 In-class,finalexam,somesatasNAATIqualificationexam

26 5 ShaniTobias

OLTID

CQU 2017 1 KnowledgeManagementPrinciples(IT)(COIT12205)

Worddoc Shortanswer 50% 180 Outofclass,central,examhall/centres,multicampus,finalexam

30 50 RahatHossain

OLTID

Monash2017 1 BusinessStatistics(EFT5900)

Worddoc MCQ,shorttext

5% 30 In-classtest 20 Notsurveyed

CharanjitKaur

OLTID


32


Weight Time(mins)




Shorttext,longtext

30% 180 In-class,finalexam,somesatasNAATIqualificationexam

31 17 ShaniTobias

OLTID

Monash2017 1 Globalisationx4classes(Monashcollege)

Worddoc. Shorttext,longtext

10% 70 In-classtest 40 42 NathanielLyon

OLTID

MqU 2017 1 ICTinEducation(EDUC261)

Worddoc MCQ,extendedessay

40% 80 Outofclass,Schoolrun,finalexam

17 181 MatthewBower

OLTID

UniSA 2017 1 MathematicsforSecondaryTeaching(EDUC5184)

Spreadsheet

Formulae,calculationcharts

40% 120 In-class,finalexam(twosittings)

9 15 RuthGeer,BruceWhite

OLTID

UNSWADFA

2017 1 Airpower(ZHSS2424)

Worddoc Shorttext,extendedessayquestions

35% 120 Outofclass,Schoolrun,finalexam

42 8 AndrewGilbert

OLTID

UQ 2017 1 Frenchlanguagetranslation(FREN3310)

Worddoc Longtext 10% 50 In-class 16 2 AmyHubbell

OLTID


Worddoc,media,Scratch


47% 120 Outofclass,central,multicampus,finalexam

88 3 AndrewFluck

OLTID

Monash2016 2 Geographyx2classes(Monashcollege)

Worddoc Singleessay 10% 70 In-class 25 13 NathanielLyon

OLTID



Shorttext,longtext

Zero 45 In-class,mockexam

18 0 ShaniTobias

OLTID

UTAS 2016 2 EducationFoundationsofTeaching(ESH102)

Mcqs,worddoc


40% 120 Outofclass,central,multicampus,finalexam

83 3 WendyBalassa

OLTID


Worddoc,media,Scratch,PDF,Windowssoftware


47% 120 Outofclass,centrallymanaged,multicampus,finalexam

110 14 AndrewFluck

OLTID


Worddoc,media,PDF,Scratch,Windowssoftware


44% 120 In-classtesteexam

110 7 AndrewFluck

OLTID

UQ 2015 1 PunishmentandSociety(CRIM2100)

Worddoc Longtext 20% 90 In-class,midsemesterexam(includes20%typedessaysectionand15%MCQusingbubblesheets).

4 33 RobinFitzgerald

UQTEL


33


Weight Time(mins)


UQ 2015 1 ResearchProcesses(DENT4092)

Worddoc Table,shorttext


19 66 AroshaWeerakoon

UQTEL

CQU 2014 2 SystemsAnalysis(COIT11226)

Worddoc Shortanswer 50% 180 Outofclass,central,multicampus,finalexam,opt-in

6 50 MichaelCowling

OLTSeed

UQ 2014 2 Zoology(BIOL2204).BiologicalSciences

Worddoc MCQ,shorttext,diagram


10 81 DavidBooth,

OLTSeed

UQ 2014 2 OccupationalTherapyforChildrenandYouth(OCTY7826)

Worddoc MCQ,casestudy,shorttext,longtext,table,listfill


3 24 LizSpringfield

OLTSeed

UQ 2014 1 PhysiotherapySpecialitiesNeurology(PHTY3140/7814)(multiplesmallgroupsof12)

Worddoc,videoprojection

Table,shorttext

15% 15 In-class,midsemesterexam(groupsof12.Written15%sectiondonepriorto20%OSCE)

25 108 KatrinaWilliams

OLTSeed

UQ 2014 1 AnimalandVeterinaryBiology(ANIM1060)

Worddoc,image

MCQ,shorttext,imageidentification

15% 45 In-class,finalexam

5 109 MalcolmJones

OLTSeed

UQ 2014 1 PunishmentandSociety(CRIM2100)

Worddoc Longtextessay

10% 70 In-class,midsemesterexam(timeincludes10%typedessaysectionand25%MCQonpaper).

17 50 RobinFitzgerald

OLTSeed

UQ 2014 1 AnimalHealthTechnology(VETS2001)(twogroups:Internal,external)

Worddoc,image

MCQ,T/F,shorttext,imageidentification,diagramlabelling

15% 90 In-class,midsemester(internalandexternalgroupsdone2weeksapart).

11 78 RebekahScotney

OLTSeed

UTAS 2014 1 Education:DigitalTechnologies(ESH380)

Worddoc,media,PDF,Windowssoftware

Shorttext,longtext,

44% 120 Outofclass,multicampus,centrallymanaged,finalexam,multi-site

166 16 AndrewFluck

OLTSeed

UTAS 2013 1 Education:InformationandCommunicationTechnology(ESH380)

Worddoc,video,GIMP,PDF,Windowssoftware

Shorttext,longtext,


123 25 AndrewFluck

UTASTDG


Worddoc,PDF,Windowssoftware

Shorttext,longtext,


48 5 AndrewFluck

UTASTDG


34


Weight Time(mins)



Worddoc,PDF,Windowssoftware,animatedPPTwithaudio

Shorttext,longtext,


88 1 AndrewFluck

UTASTDG

UTAS 2011 2 Law:ConstitutionalLaw(LAW205)

Worddoc Shorttext,longtext,

50% 120 Outofclass,singlecampus,centrallymanaged,finalexam

69 179 MichaelStokes

UTASTDG

UTAS 2011 2 Law:PropertyLawB(LAW324)



26 ? LyndenGriggs

UTASTDG

UTAS 2011 2 Law:InternationalLaw(LAW631)



1 ? MichaelTate

UTASTDG

UTAS 2010 2 Arts:History-AtlanticWorlds1450-1807(HTA249/349)



3 ? TomDunning

UTASTDG

UTAS 2010 2 Education:ModesofcurriculumInquiryA(ICTandDesign&Technology)[EPC353]

Worddoc,video,PDF,Windowssoftware,PPTwithaudio

Shorttext,longtext,


69 4 AndrewFluck

UTASTDG


35

AppendixDUpdatedImpactMatrixTheIMPELmatrix(shownonthefollowingpage)providesanestimateofanticipatedchangesoutlinedintheprojectproposalalongwithtagstoindicatethestatusofeachproposedimpactpoint.Theseare:achieved,likely,possible,alternative(Theseareshownincapitallettersinthetablebelow).Itshouldbenotedthatduetochangesintheprojectfocusortheenvironmentsomeimpactpointsarenowconsideredunlikely,butinallsuchcasesweforeseeanalternativeoutcomebeingpossible.Themajorareaswherealternativesarepossibleinclude:

a)Theproposede-ExamsFoundationwasthoughtunsuitedtocurrentsectorconditions(financialandmanagerial)howeveranalternativeoutcomemaybeajointventureorcommercialisationarrangement.

b)Theadoptionofprojectdevelopede-examtechnologies.GiventheapparentstrategiesandprocessesofAustralianuniversitymanagement,theadoptionofe-ExamsatAustralianinstitutionsismorelikelytobeintheformofcommerciallyavailablesolutionsratherthan'homegrowninnovations'.Unfortunatelythecommercialmarkethasyettobeabletoofferasolutionwiththeextensiveauthentice-assessmentcapabilitiesofferedbythee-Examplatformdevelopedinthisproject.ThisisincontrasttoeducationorganisationsinregionssuchasEuropewhereanumberofhomegrowne-examapproachesincorporatingauthentice-assessmentarebeingadoptedwithgreatsuccess(Seee-ExamSymposiumsessions1and4forexamplesinFinlandandAustria).


36

AnnotatedIMPELMatrix

AnticipatedchangesatProjectcompletion 6Monthspost 12monthspost 24monthspost

1. TeamMembers

ACHIEVEDTeammembersknowledgeableaboutusinge-examstocreatepost-paperexaminationsintheircourses.

LIKELYResearchandpublicationsone-examsfindingsfromresearchproject.Planningnextresearchphase.Furtherinvitestocollaboratefrominstitutions.Continuedpresentationsgiven.

POSSIBLEApplicationunderwayforsubsequentgrant,fellowship(e.g.developlearninganalyticstoolsandtechniquestoleveragee-examsdata),orteachingawardapplicationsforteammembers.

POSSIBLE,ALTERNATIVEResearchandpracticeintoexamscontinues.Continueduseincoursesbyprojectteammembers.

2. Immediatestudents

ACHIEVEDSeveralgroupsofstudentsundertheinstructionofteammembersexperiencedmultiplee-exams.

POSSIBLE,ALTERNATIVELargernumbersofstudentsexperiencinge-exams

3. Spreadingtheword

ACHIEVEDWebsite,practiceguides,technicalinfrastructurereleased.Policyrecommendations.Resultsfromtrials.Softwaredistribution.

LIKELYDatapackages/archivessharedonlineforotherresearcherstoaccess.Websiteliveson.Publications,presentationscontinue.

LIKELYPublications,presentationsbeingcited.

4. Narrowopportuneadoption

ACHIEVEDSeveralcoursesemployingthee-exams.'Associate'partnerinstitutionscontinuetrials

POSSIBLEInstitutionsmakingcontacttotrialthee-examsystemonaself-fundedbasis.

5. Narrowsystematicadoption

ALTERNATIVEe-examsFoundationinitiated.Supportpartnersintereststart.Majorstakeholdersocieties,HERDSA,ASCILITE,CADADacknowledgingvalueoftheproject.

POSSIBLE,ALTERNATIVEOngoingdatacollection.Passiveonlinesurveyforreportinguse.Askinguserswhatimpactithadxmonthsafter.Corepartnerinstitutions'officially'supportinge-exams.

POSSIBLE,ALTERNATIVEFoundationunderwaytocontinuethetechnicaldevelopmentwithcommercialtechnicalsupport.Firstpartnersjoinedandpayingcontributionstothetechnicaldevelopment.

6. Broadopportuneadoption

LIKELY,ALTERNATIVESeveralinstitutionsemployingthee-exams.

LIKELY,ALTERNATIVEConsultancyservice(byprojectteammembers)assistinginstitutionstostarte-examsandcustomisetoneed.

POSSIBLE,ALTERNATIVEFirstcommercialproviderssigneduptofoundationandprovidingservicestothehighereducationsector.

7. Broadsystematicadoption

LIKELY,ALTERNATIVEPolicy'shape'recommendations.

LIKELYPolicychangeconsideredbyinstitutionsbeyondprojectcore.Someinstitutions'officially'supportinge-exams.

POSSIBLEConsultancyservicesofferingpedagogical/technicale-examsprocesssupport.

POSSIBLEPoliciesadoptedallowinge-examsinmultipleinstitutions.Moreinstitutions'officially'supportinge-exams.


37

AppendixEFormativemid-projectevaluationreportThee-ExamsystemItwastheopinionofallprojectparticipantsthattheprojecthasbeenhighlysuccessfulindevelopingasuitablee-examplatformforBYODsupervisedassessments.TheplatformhasbeenfoundtoworksuccessfullyonbothWindowsPCandAppleMacintoshlaptops.Someparticipantsnotedthatstudentsareincreasinglybuyingintotheidea,withfewerstudentsoptingforhandwrittensubmissions.Theguidelineshavebeenseentobeuseful,describedas"excellentdocumentationforthetechnicalaspectsofUSBproduction,answer-scriptcollectionandrecycling."Furtherthattheprojecthasbeenusedasimpetuousforarangeofconceptualandtechnologicaldevelopments:“Ourparticipationinthetrialprecipitatedunexpecteddemandsfromacademicsfore-assessments.Wehaveservicedthosedemandsbyrunningverylowteche-assessmentsthathavetaughtusalotaboutwhatneedstobeseentoinbackground,non-technicalarrangements.”

ChallengesandopportunitiesAllparticipantsviewedtheopportunitiesaffordedbythepotentialofe-examsasoutweighingthehindrances.Wherechallengeswerenoted,theywereidentifiedasareasthatcouldbeenhanced,ratherthanasrationalefordiscontinuinganyfurthertrials.Participantscitedchallengeswithinternalapprovalsandauthorisationsashindranceswiththeproject.Anexampleofonesuchchallengewastheexperienceinanearlierpilotprojectconductedin2014thatsawtoofewUSBsbeingprovided,resultinginstudentcomplaints.Thisexperiencedidprovideapositiveoutcomeinthat,asaresult,“betterbackupproceduresforlackofUSBorfailure”havebeendevelopedandimplementedinlatertrialsconductedaspartofthisproject.Intermsofthesoftware,thelimitedgraphicscapabilitywasidentifiedasbeingproblematicforsomeprogrammingexams.Alsoseentobeproblematicwastheinabilitytolimitaccesstoparticularwebsites.Thethesaurusfunctionwithinthee-examsystemwasnotedasbeingparticularusefulintranslation,andalsoacknowledgedtobeahithertounexploredresource.ThepotentialtoexpanddictionariestoincludebothEnglishandFrenchwasviewedaspotentiallybeneficial.

ProjectteamOverall,allparticipantswereverypleasedwiththeprogressoftheprojectandwereallpleasedwiththeserviceandresponsivenessoftheprojectteam,findingtheirengagementwiththeteamtobehighlyproductive.“Technicalsupportranseamlesslyinthepracticeexamsandtheactualtrials.Vilmawasveryorganisedandkeptusuptodateandontimewithvarioustasksthroughoutthetrials.Mathewisextremelyknowledgeableabouttheopportunitiesinthee-examsystemandhisbackgroundinpreparingtranslationexamswasusefulforme.”Thehelpfulnessandpositiveresponsivenesswascitedasthereasonbehindthesuccessfultrialsandimplementation.Exchangeswerenotedtoassistindeveloping“ourunderstandingofnotjustthetechnicalelementsofe-examsbutalsotheorganisationalelements.”

ConclusionsandRecommendationsTheprojectparticipantsat7ofthe9partnerinstitutionsrespondingtothesurveyindicatethattheyaresatisfiedwiththeprojectprogress.Theyareenthusiasticparticipants,seekingtocontinuetheirengagementforthesuccessfuloutcomesandexperiencesfortheir


38

students.Reviewingengagementdatainlightofthesurveyresponses,itisclearthattheprojectisprogressingasanticipated,andhasawiderreachandlevelofengagementthanpreviouslyanticipated.Workingwithconservativeinstitutionalpoliciesandacademicandtutoringstaffnotyetconvincedofthebenefitsofe-examtechnologieswillneedtobeconsideredinthefuture.Itisalsoworthconsideringtheuseofotherstoragetechnologiesasthistechnologycontinuestoevolveasanover-dependenceonUSBscouldbepotentiallylimitinginthefuture.

DrDeanneGannawaySFHEAAustralianLearningandTeachingFellow,Academiclead:Professionallearning.InstituteforTeachingandLearningInnovation(ITaLI),[email protected]


39

AppendixFFinalexternalevaluationreportEXTERNALEVALUATIONREPORT:TRANSFORMINGEXAMSACROSSAUSTRALIAThisfinalexternalevaluationreportprovidesanoverviewoftheoutcomesfromtheTransformingExamsAcrossAustraliaprojectconductedbetween2016and2019andledbyDrMathewHillier.

1. EXTERNALEVALUATIONTEAM

TheexternalevaluationwasconductedbyDeanneGannawayandKarenSheppard.

DeanneGannawayBAHDipEd(Wits),MEd(USQ).PhD(Flinders)

DeanneGannawayisaseniorlecturerintheInstituteforTeachingandLearningInnovationattheUniversityofQueensland.Deanne’sresearchinterestsfocusonthedevelopmentofeducationalevaluationandenhancementofhighereducationcurriculum.Deanneisfrequentlycalledontoofferadviceonappropriateevaluationapproachesandmethodsforlargescaleteachingandlearningprojectsandcurriculum-basedactivitiesandhasbeencommissionedastheexternalevaluatoronanumberoflargescaleALTC/OLT-fundedprojectsandfellowshipprograms.

KarenSheppardBA(UWA),Dip.Ed.(WACAE),MEd(ProfessionalEducationandTraining)(Deakin)

KarenSheppardisalearningdesignerintheInstituteforTeachingandLearningInnovationattheUniversityofQueensland.

DeanneandKarenhavebeenactivelyinvolvedinanumberofnationalteachingandlearningfundedprojects,includingnationallyfundedevaluationofthedisseminationstrategiesusedintheAustralianLearningandTeachingCouncil(ALTC)grantsschemes:AReviewoftheDisseminationStrategiesusedbyProjectsFundedbytheALTCGrantsScheme.Thisprojectbuiltontheresearchcompletedin2004forthenationally-focused,DESTfundedresearchprojectondisseminationstrategiesforinnovationprojectoutcomeswhichdirectlyinfluencedthegrantschemesandprocessesdevelopedbytheALTC.

ThisfinalreporthasbeendraftedbyDrDeanneGannaway.

2. THEEVALUATIONAPPROACHThisexternalevaluationadoptedtheKeyEvaluationChecklistapproach(Scriven,2007)todevelopananalyticalframework.Thischecklisthasbeenusedtoevaluateprograms,plansandpoliciesandallowsforamulti-layeredmixedmethodmodeofinvestigation.(Stufflebeam,2001;Davidson,2005;Scriven,2007;Martz,2010).TheformativeevaluationstrategydirectedbythechecklistapproachpromotedbyScrivenfacilitatesprojectteamreflection,allowingforagileandresponsiveprojectprogressratherthansolelyasummativemeasurementofprojectimpactorattainment.

InkeepingwiththeScrivenapproach,theevaluationaimstoascertainthesuccessoftheprojectinrelationtoitsvision,mission,goals,deliverablesandplan,specifically,measurementandidentificationof:

• achievementofprojectgoals,objectivesandintendedoutcomes;• whetherneedsofstakeholderssuchasstudents,staff,projectparticipantsand


40

fundingbodyarebeingmet;• relevance,appropriatenessandqualityofthechoseninvestigationandproject

managementstrategiesusedintheproject;• goodpracticeandareasforimprovementacrosstheprojectlife;and• effectivenessofdisseminationandtake-upamongstakeholders.

Theformativeevaluationapproachadoptedrequiredacloseworkingrelationshipbetweentheprojectteamandtheexternalevaluationteaminordertomonitorandadviseontheevaluation,dissemination,resourcingandinvestigationstrategiesadoptedbytheprojectteam.Thisapproachenabledtheidentificationofgoodpracticeandareasforimprovementacrossthelifeoftheprojectand,mostimportantly,enablestheprojectteamtorespondtotheareasidentified.

EVALUATIONQUESTIONSANDACTIVITIES

Thefollowingdatacollectionstrategieswereusedtocollectandanalysetherequireddataagainsttherelevantevaluationquestions:

1. Wereprojectgoals,objectivesandintendedoutcomesachieved?

a. ReviewofprojectactivitiesconductedProjectdeliverablesarereviewedatregularintervalsacrossthelifeoftheproject.Theevaluationteamisnotacontentexpertandthereforenotinapositiontoevaluatetheaccuracyorrelevanceoftheresourcesdeveloped.Theexternalevaluationteamactedascriticalfriends,providingformativeinputacrossthedevelopmentprocesses.

b. EmailsurveyofprojectpartnersAnemailsurveywasconducted,targetingprojectparticipantswhoarealsopotentialadoptersofprojectoutcomes.Inacknowledgementofpossiblesurveyfatigue,ratherthananonlinesurvey,aquickemail-basedsurveywassenttoparticipants.AcopyofthequestionsisavailableasAppendix1.Thesurveywassentto15individualsresultingin7responses,providinginputfrom7ofthe9partnerinstitutions.Thetimingofthesurveymeantthatmanyofthepotentialparticipantswerealreadyonleaveforthesummerbreak.

c. InterviewswithprojectteammembersTheprojectleader,projectmanagerandtheexternalevaluatorhaveconductedregular“check-in”meetingstomonitorprojectprogressionandtoreviewprojectevaluationactivities.

2. Howrelevantandappropriatewerethechosenprojectplans,activitiesandstrategiesforachievingtheprojectaims?

a. MonitoringprojectprogressProgressreportswerereviewedandfeedbackgiven.Projectplanswerediscussedduringtheevaluationmeetingsandprojecttimelinesreviewed.

b. ReviewofevaluationdatacollectedEvaluationactivitiesconductedbytheprojectteamwerereviewedbytheevaluationteamtoensuretheveracityofprojectfindingsandconclusions.

3. Wereanticipatedprojectoutcomesmet?


41

a. ReviewofmaterialsdevelopedWeb-basedmaterialsandotherresourcesdevelopedasdeliverablesfromtheprojectwerereviewedandfeedbackprovidedinmeetingswiththeprojectcoreteam.

b. ObservationofprojectactivitiesKarenSheppardattendedandparticipatedinsomeofthenationalsymposia,collectingobservationaldataandcommentaryfromotherparticipants.

EvaluationOutcomes

1) Towhatextentdidtheprojectmeetand/orexceeditsobjectives?

Theoriginalprojectaimswere:1. Buildingane-examplatformtobeusedinsupervised,'bringyourowndevice'

(BYOD)settingsthatincludescomputermarkedquestionsandelectronicreticulationofstudentresponses.

2. Developingguidelinestoassiststudents,educatorsandadministratorstoeffectivelyprepareandundertakee-examstakingintoconsiderationthewholeassessmentworkflow.

Atotalof35trialswereconductedacrosseightinstitutionsfrom2016to2018withparticipationfromover1700students.Studentswereabletooptintothetrials.Projectparticipantsregularlycommentedthat,intheiropinion,theprojecthasbeenhighlysuccessfulindevelopingasuitablee-examplatformforBYODsupervisedassessments.TheplatformhasbeenfoundtoworksuccessfullyonbothWindowsPCandAppleMacintoshlaptops.Someparticipantsnotedthatstudentsareincreasinglybuyingintotheidea,withfewerstudentsoptingforhandwrittensubmissions.

Theguidelineshavebeenseentobeuseful,describedas“excellentdocumentationforthetechnicalaspectsofUSBproduction,answer-scriptcollectionandrecycling.”

Possiblymoreimportantly,theprojecthasbeenusedasimpetuousforarangeofconceptualandtechnologicaldevelopments:“Ourparticipationinthetrialprecipitatedunexpecteddemandsfromacademicsfore-assessments.Wehaveservicedthosedemandsbyrunningverylow-teche-assessmentsthathavetaughtusalotaboutwhatneedstobeseentoinbackground,non-technicalarrangements.”

2) Howrelevantandappropriatewerethechosenplans,activitiesandstrategiesforachievingtheprojectaims?

Overall,participantsindicatedthattheywerepleasedwiththeprogressoftheprojectandwereallpleasedwiththeserviceandresponsivenessoftheprojectteam,findingtheirengagementwiththeteamtobehighlyproductive.Projectpartnersunanimouslyagreedthattheprojectleaderandprojectmanagerwerehighlyeffective,accessibleandknowledgeable,facilitatingchangesinassessmentpolicyandpracticeandopeningupinstitutionalconversationsthatpotentiallyleadtoassessmenttransformationinthefuture.

“Technicalsupportranseamlesslyinthepracticeexamsandtheactualtrials.Vilmawasveryorganisedandkeptusuptodateandontimewithvarioustasksthroughoutthetrials.Mathewisextremelyknowledgeableabouttheopportunitiesinthee-examsystemandhisbackgroundinpreparingtranslationexamswasusefulforme.”

Thehelpfulnessandpositiveresponsivenessoftheprojectteamwasfrequentlycitedasthe


42

reasonbehindthesuccessfultrialsandimplementation.Exchangeswerenotedtoassistindeveloping“ourunderstandingofnotjustthetechnicalelementsofe-examsbutalsotheorganisationalelements.”

Theprojectparticipantsat7ofthe9partnerinstitutionsrespondingtothesurveyindicatethattheyaresatisfiedwiththeprojectprogress.Theyareenthusiasticparticipants,seekingtocontinuetheirengagementforthesuccessfuloutcomesandexperiencesfortheirstudents.Therewerenonegativeresponsestoanyofthequestions,butsomesuggestionsforfutureimprovementswereoffered.

Reviewingengagementdatainlightofthesurveyresponses,itisclearthattheprojecthashadawiderreachandlevelofengagementthanpreviouslyanticipated.

Participantscitedchallengeswithinternalapprovalsandauthorisationsashindranceswiththeproject.Anexampleofonesuchchallengewastheexperienceinanearlierpilotprojectconductedin2014thatsawtoofewUSBsbeingprovidedbytheinstitution’scentralexamsoffice,resultinginstudentcomplaints.Thisexperiencedidprovideapositiveoutcomeinthat,asaresult,“betterbackupproceduresforUSBlackorfailure”havebeendevelopedandimplementedatthisinstitutionwhichinlatertrialsconductedaspartofthisproject.Otherexamplesincludeinstitutionalpolicydecisionscappingtheweightingofassessmentallowedtobeincludedinsuchtrials,limitingengagement.Otherhindrancesidentifiedincludeinfrastructurelimitations,suchasshortagesinthenumbersofpowerpointsavailable;thepotentialforsomestudents’computerstonotbecompatible.Thefactthatthecurrentsystemdoesn'tworkonallmachines(particularlymoremodernmachines)andinvolvesthetime-consumingandtechnicallydifficultuseofUSBsticks,isaproblem.IamlookingforwardtofutureversionsofthesystemthattieinwithMoodleasamoreefficientwayofmanagingexams.ThechangingnatureofstoragetechnologyandtheneedtomovebeyondUSBinthefuturewasnotedasapotentialfuturedirection.Anotherpotentialhindrancewasthenegativeresponseevidentinteachingteammembersoutsideoftheprojectpartners.Somepartnersdescribedthechallengesofconvincingtutorsandacademicsnotfamiliarwiththesoftware,includinghavingtoprintstudentworksothattheycouldbemarkedasapaper-basedartefact.Thischallengesuggeststhatthereisstillmoreworktobedoneintermsofrefiningthemessagingandinformationprovidedtostakeholdersandpotentialusers.

Intermsofthesoftware,thelimitedgraphicscapabilitywasidentifiedintheinterimreportasbeingproblematicforsomeprogrammingexams.Alsoseentobeproblematicwastheinabilitytolimitaccesstoparticularwebsites.Thethesaurusfunctionwithinthee-examsystemwasnotedasbeingparticularusefulintranslation,andalsoacknowledgedtobeahithertounexploredresource.ThepotentialtoexpanddictionariestoincludebothEnglishandFrenchwasviewedaspotentiallybeneficial.

Participantsnotedtheneedtomonitortheevolutionofstoragetechnologiesinthefutureasanover-dependenceonUSBscouldbepotentiallylimitinginthefuture.

3) Howeffectivearethechosendisseminationstrategies?

TheTransformingexamsacrossAustraliawebsite,roadshowworkshops,conferencepresentations,journalpublicationsandthee-Examsymposiumallhaveservedtoraiseawarenessoftheprojectdeliverablesandoutcomes.However,workingwithconservativeinstitutionalpoliciesandacademicandtutoringstaffnotyetconvincedofthebenefitsofe-


43

examtechnologiesisafundamentalhindrancetotransformationofpracticeandfurtherspreadofinnovation.Participantsinthedisseminationactivitieshavetendedtobeearlyadoptersorthosealreadyconvincedthatthisisaviableoption.Thisprojectdidnothaveenterpriseculturalchangeasafundamentalaim,sotheawarenessraisingdisseminationstrategiesadoptedareentirelyappropriate.TheadoptionofthesolutionbyagenciesoutsideoftheAustralianuniversitysectorbodeswellforongoingcollaborationsandsustainabilitybeyondthelifeoftheproject.

CONCLUSIONSANDRECOMMENDATIONS

Insum,thisprojecthasmetandexceededitsobjectives.Thee-examsolutionusesallMoodlequestiontypes,andismovingtowardsmoreauthenticassessmentplusprovidingaccesstoarangeofsoftwaretools.AkeybenefitofthesolutionistheuseoftheBYODoverwifiorwirednetworks.Further,thecomprehensivenetworkoutageprotectionmeansthatstudentresponsesarefullycachedandautosavedtolocalstorageevenifthereisnonetwork.

Whiletheprojecthasdevelopedandtestedahardwareandsoftwaresolution,themostimportantcontributionoftheprojecthasbeentheinitiationofanationaldiscussion.Theprojecthasignitedactivitiesthathavealreadyseenfurtherinstitution-wideinitiatives(seeCMM17June,2019,forexample).Theprojectalsoleavesanimportantlegacyintheformofthehttp://transformingexams.comwebsitethatisattractingongoinginternationalandnationalengagement.

Thehigh-risk,highcostnatureofimplementingthetechnologyisstillalimitingfactoroffurtheradoption,despiteincreaseddesireonthepartofinstitutionalbodiestomoveintothisspace.Themovetowardsmorewidescaleadoptionrequiresawhole-of-institutioncommitment,thatincludesbuy-infromacademics,isembeddedinpolicy,istrustedbystudentsandhassubstantialresourcingallocatedtoensurethatthetechnologyisembeddedinamannerthatisatanenterpriselevel.Whiletheseaspectswereoutsideofthescopeofthisproject,thisprojecthasshiftedthesectortowardssuchafuture,bydevelopingaworkingtechnologyandanimplementationframework.Thereismuchmoreworktobedoneonasector-widelevelbeforewecanseewideradoption.Thisprojecthasleftbehindafertileground,readyforthatwork.DrDeanneGannawaySFHEAAustralianLearningandTeachingFellow,Academiclead:ProfessionallearningInstituteforTeachingandLearningInnovation(ITaLI),[email protected],E.(2005).Evaluationmethodologybasics:Thenutsandboltsofsoundevaluation:SagePublications,Inc.Martz,W.(2010).Validatinganevaluationchecklistusingamixedmethoddesign.EvaluationandProgramPlanning,33(3),215-222.doi:https://doi.org/10.1016/j.evalprogplan.2009.10.005Scriven,M.(2007).Thelogicandmethodologyofchecklists.Stufflebeam,D.L.(2001).EvaluationChecklists:PracticalToolsforGuidingandJudgingEvaluations.AmericanJournalofEvaluation,22(1),71-79


44

FinalEvaluationreport:APPENDIX1From:DeanneGannaway[mailto:[email protected]]Sent:Tuesday,5December20177:10PMCc:DeanneGannaway<[email protected]>Subject:Externalevaluationofe-ExamsprojectThee-ExamsprojectfundedbytheOLTandledbyDrMatHillierisdueforcompletioninDecember2018.Todate22testcaseshavebeenconductedacrossAustralia,acrossmultipleuniversitiesandover1400studentshaveengagedwiththetool.Thesoftwareandassociatedprocessesarenearingfinalisation,providinganopportunemomentintimetotakestockwithcollaboratorsandpartners,sothatanyfeedbackcanbeincorporatedintothefinalstagesoftheproject.ArequirementofOLTfundedprojectsisthattheprojectisexternallyevaluated.IhavebeenworkingwithDrHillierinthiscapacityacrossthelifeofthisprojectandamcurrentlydraftingtheinterimexternalevaluationreport.Iamcontactingyouasyouhavebeenidentifiedaseitherauser,collaboratororpartnerinthisproject.Theexternalevaluationofthisprojectwouldgreatlybenefitfromyourinputregardingthefollowing:

1. Yourviewsonwhetherthematerialsdevelopedachievetheprojectaims2. Opportunitiesandhindrancesyouhaveexperiencedorenvisageinimplementinge-

Examsintoyourassessmentpractice3. Yourexperienceswithengagementwiththeprojectteamtoensureoptimaltrial

experiencesAshortresponsetooneorbothoftheaboveviareplyemailwouldbemosthelpful.Youarealsowelcometoprovidecommentaryonanyotheraspectoftheprojectdeliverablesorprocesses.Yourresponseswillremainconfidentialandyouwillnotidentifiedinthereportunlessyouspecificallyrequesttobesoidentified.TheinterimexternalevaluationreportispubliclycirculatedonlytotheprojectleaderandtotheOLT.Icanforwardacopyoftheexternalevaluationtoyouifyouwouldliketoseeafinalcopy,withDrHillier'spermission.Ifyouwouldbewillingtorespond,couldyoupleasedosobythe15thofDecember?Pleasedonothesitatetocontactmewithanyquestionsorqueries.IlookforwardtohearingfromyouRegardsDeanne

Documents

Transforming exams across Australia platform for e exams ... · Transforming exams across Australia: Processes and platform for e-exams in high stakes, supervised environments Final