EvaluationReportonDukeUniversity’sCourseraSpecializationsonBusinessAnalyticsandJavaProgrammingKimManturuk,ProgramEvaluator,CenterforInstructionalTechnology
Page 2 of 15
ExecutiveSummaryIn2015,DukeUniversityrapidlydevelopedandlaunchedSpecializationsinBusinessAnalyticsandJavaProgramming.WhiletheshorttimelineforcreatingtheseSpecializationscreatedseveralchallenges,thecourseshavebeenverysuccessfulintermsofattractingenrollments.Overall,theSpecializationshavebeensuccessfulatthefollowing:
• Providingcareer-oriented,high-demandskillstrainingforpeopleworkingin,orseekingtoenter,professionalfields
• GeneratingrevenueforDukeandforthecourseinstructors• RaisingDuke’svisibilityandreputationasoneoftheleadingeliteAmerican
universitiesofferingopen,onlineeducation• Providingthecatalystforseveralnewmethodstoassesscoursequalitythat
willbeusedinfutureprojectsincludingjust-in-timemodulefeedbacksurveysandin-housebetatesting.
ThereweresomeareasinwhichtheSpecializationswerenotassuccessfulasenvisionedasfollows:
• TheJavaProgrammingSpecializationfellshortofthestatedgoalofattractingwomenandpeopleofcolortothecoursesandthefieldofcomputerscience;additionaleffortsareunderwaytoaccomplishthisgoal
• TheBusinessAnalyticsSpecializationdidnotinitiallyincludesufficientqualityassuranceprocesssorevisionshadtobeundertakenafterthecourseslaunchedtocorrecterrors
• CourserastaffandpotentialindustrypartnersdidnotprovidethetechnicalsupportexpectedbyCITandthefacultymembers;revenueexpectationswerealsonotcommunicatedeffectively
Lessonslearnedfromthesetwoprojectsincludethebenefitsofhavingmoreflexibletimelinesandtheneedtosetrealisticexpectationsatthestartofaproject.Infuturesimilarprojects,effectivequalityassuranceandcommunicationprocesseswillbecriticaltosuccess.Overall,werecommendthatanysimilarprojectundertakeninthefutureexplicitlyspecify,ineithertheprojectagreementortheworkplan,howthefollowingtaskswillbecarriedout:
• ObtaindetailsinwritingofanysupportorresourcesthatwillbeprovidedbycompaniesorpartnersoutsideofDukeUniversity
• Forprojectsinvolvingmultiplefacultymembers,planinadvanceforhowrevenuewillbeadjustediftimecommitmentschangewithinthegroup
• EstablisheffectivequalityassuranceprocessbeforelaunchingcoursesanddesignateaQAleadperson
Page 3 of 15
ProjectOverviewStartinginAugust2015,OnlineDukeandtheDukeCenterforInstructionalTechnologycollaboratedwithsixinstructorstodeveloptwoCourseraSpecializations.Fourinstructorsworkedtogethertocreatea5-course(4+capstone)SpecializationonJavaProgramming;twoinstructorscreatedaSpecializationonBusinessAnalyticsinthesameformat.Thesetwoprojectswereuniqueformanyreasonsincluding:
• Thecourseteamshadjust4weekstocreateeachofthe5courses,amuchfastertimelineforcoursecreationthanonanypriorproject
• TheprojectleadershadtocoordinatemuchlargercourseteamsthanDukehasusedinpriorprojects
• Thetechnologiesrequiredtocreatethecourseactivitiesandassessmentswerenewtotheprojectteamsonewskillshadtobedevelopedveryrapidly
• Just-in-timecoursefeedbackwasprovidedtocourseteamsthroughtheuseofveryshortfeedbacksurveysembeddedineachcoursemodule
• Courseteamswereawarethatthesecourseswouldlikelybeveryvisibleandgeneratesignificantrevenuefortheuniversity
ThecapstonecoursefortheJavaProgrammingSpecializationwascompletedinmid-January2016andtheBusinessAnalyticscourseteamfinishedproductionontheircapstonecourseinMarch2016.Sincethattime,wehavecollectedsurveydatafromlearnersinbothSpecializationsandtrackedlearneractivitypatterns.WealsoconductedinterviewswithmostofthecourseteammembersonbothSpecializationstounderstandthestaffandinstructorexperience.Thisreportsummarizesthatinformation.NotethatwehavealsouseddatafromseveralsourcestohelpbothcourseteamsidentifyelementsintheirSpecializationsthatwouldbenefitfromrevisions.Becausethoserevisionsarein-progress(JavaProgramming)orhavejustrecentlybeencompleted(BusinessAnalytics),anevaluationoftherevisionprocessandchangeswillnotbecoveredinthisreportbutwillbeforthcoming.Anupdateonrevisionsisprovidedattheendofthisreport
IndustryPartnershipsAfifthuniqueaspecttotheseSpecializationsthatonlypartiallymaterializedwasthattheyinvolvedseveralpartnershipsthatbothcontributedvaluableresourcestotheprojectyetdidnotalwaysdeliverasexpected.TheBusinessAnalyticsteampartneredwithAirbnbtoprovidelearnerswithdataforassignments.ThesameteamalsopartneredwithTerraDataandTableautoprovidestudentssoftwareaccess.Thesepartnershipsdidprovidelearnerswithvaluableresources,butCITprojectteamshadtodomoreworkthanexpectedtograntstudentsaccesstotheseresources.Infutureprojects,teamleadsshouldstartthisprocessassoonaspossibleandexplorewhetherDuke’sOITofficecanprovidesupport.
Page 4 of 15
TheJavaProgrammingSpecializationwasinitiallygoingtobeofferedinpartnershipwithGoogle,butachangeinprioritiesduetointernalpersonnelchangesledGoogletosignificantlyreducetheroleitwouldplayinthedevelopmentandmarketingofthecourse.TheydecidednottoaffiliatetheGooglenamewiththecourseandlimitedtheirinvolvementtoprovidingsomeshortvideointerviewswithGoogleengineers.AllthefacultymemberswhoparticipatedinthisprojectwereveryfrustratedthatGooglelargelywithdrewfromtheproject;theplannedpartnershipwithGooglewasasignificantfeaturethatdrewthefacultymemberstoparticipateintheprojectbecausetheyanticipatedbeingabletoaccessexpertiseandresourcesfromGoogle.ThefacultymembersalsoanticipatedthatthepartnershipwithGooglewouldincreaseenrollmentsandthusimpactrevenuefromtheSpecialization.HenceGoogle’swithdrawfromtheprojectwasdisappointingforseveralreasons.Whilefuturesimilaroccurrencesarelargelyunavoidable,itisimportanttonotethenegativeimpactitwouldhaveonanyfutureprojectteamifanimportantindustrypartnershipweretofallthrough.
LearnerDemographicsLearnersenrolledintheSpecializationstypicallyfittheprofileoflearnersinothercourses.Table1belowshowssomedemographicmeasuresdividedbycourse.Mostlearnersalreadyhadacollegedegreeandwereemployed.BothSpecializationsenrolledfewerwomenthanmen,andfewerwomenthantheCourseraaverageof39%.TheJavaProgrammingcourseinstructorshadagoalofattractingwomanandpeopleofcolortothecourse,soadditionalmarketingeffortstoattractthosegroupsarerecommended.Table1:SelectedDemographicMeasures
BusinessAnalytics
Course1 Course2 Course3 Course4
Course5(capstone)
NorthAmerica 43% 30% 44% 42% 44%Women 30% 28% 28% 29% 25%Collegedegree 86% 84% 91% 88% 92%Employed 76% 77% 81% 74% 71%JavaProgramming
Course1 Course2 Course3 Course4
Course5(capstone)
NorthAmerica 36% 33% 32% 33% 38%Women 29% 20% 20% 17% 24%Collegedegree 63% 75% 78% 77% 88%Employed 70% 70% 70% 73% 79%
Page 5 of 15
LearnerActivityDataBothSpecializationswere,asanticipated,highlyvisiblelikelyduetothefactthattheyfocusonteachinghigh-demand,career-orientedskillsthataretypicallynotpartofacollegeeducation.CoursetwointheBusinessAnalyticsSpecializationhas,asofthiswriting,attractedalmostahalfmillioncoursevisitorsandenrolledover100,000people.Thenumberofcoursevisitorsandenrollmentsforeachcourseareshowninthechartsbelow.Chart1:VisitorsandActiveLearners–BusinessAnalytics
Chart2:VisitorsandActiveLearners–JavaProgramming
98,660
426,135
98,813 103,672
19,49244,624
126,266
33,592 26,653458
Course1 Course2 Course3 Course4 Course5(capstone)
VisitorsActivelearners
147,178
106,398
66,528 68,786
16,147
52,202
30,28314,722 12,414
366
Course1 Course2 Course3 Course4 Course5(capstone)
VisitorsActivelearners
Page 6 of 15
Charts3and4belowshowthenumberofcoursepaymentsandapprovedfinancialaidapplications1foreachcourse.Alllearnershavetheoptiontoapplyforfinancialaidbywriting3briefessaysdescribingtheirfinancialneed,whytheywanttoenrollinacourse,andhowtheywillmaintainacademicintegrity.LearnerscanapplyforfinancialaidfromeithertheindividualcoursepagesorfromtheSpecializationhomepage,buttheymustapplyindividuallyforeachcourseintheSpecialization;itisnotpossibletocompleteoneapplicationandreceivefinancialaidforallfiveclasses.Asshownbelow,coursepaymentsconsistentlyoutpacefinancialaidapplicationsintheBusinessAnalyticscourses.Notethatthenumbersinthechartsbelowdonotrepresentindividuallearners;theyarecoursepayments,originatingfrombothindividualcoursepagesandfromtheSpecializationpage.Chart3:PaymentsandFinancialAid–BusinessAnalytics
IntheJavaProgrammingchart(below),thenumberstotherightofthelinearepaymentsandthenumberstotheleftarefinancialaidapplication.IncontrastwithBusinessAnalytics,thefirstcourseintheJavaProgrammingSpecializationhasreceivedmorefinancialaidenrollmentsthanpaidenrollments.Subsequentcoursesintheserieshavemorepaidenrollments.
1Courseraallowslearnerstoapplyforfinancialaidbyansweringafewquestions.Ifapproved,theyreceiveafeewaiverforonecourse.ThisprocessisentirelyinternaltoCourseraandisnotassociatedinanywaywithfederalfinancialaidoranyschool-basedaidprograms.
9,544
7,072
5,3814,823
3,262
5,071
3,824
1,876 1,681
216
Course1 Course2 Course3 Course4 Course5(capstone)
PaymentsFinancialaid
Page 7 of 15
Chart4:PaymentsandFinancialAid–JavaProgramming
Thenexttwochartsshowtheconversionpercentwithineachclassfromcoursevisitorstoactivelearners,andfromactivelearnerstopaidenrollments.Notethatweusetheterm“conversion”loosely;weknowthatnotallpaidlearnersstartasactivelearnersandcompleteanactivitypriortopurchasinganenrollment.However,displayingthedatainthismannerishelpfulforillustratingwhatportionofactivelearnersarepaidlearners.Asillustratedpreviously,thesecondcourseintheBusinessAnalyticsserieshasattractedthehighestnumberofcoursevisitorsandenrollmentsofanycourseineitherSpecialization.However,ithasthelowestactive-to-paidconversionrateat6%.Thisindicatesthat94%ofpeoplewhoareactiveinthecoursehavenotpaid.Sinceweknowthatthenumberoffinancialaidlearnersisroughlyhalfthenumberofpaidlearners,wecaninferthataround91%ofactivelearnersareauditingthecourse.
5,718
2,326
1,711 1,555 1,432
5,940
1,884877
632183
Course1 Course2 Course3 Course4 Course5(capstone)
PaymentsFinancialaid
Page 8 of 15
Chart5:ConversionPercents–BusinessAnalytics
AcrossallcoursesintheJavaProgrammingSpecialization,theactive-to-paidconversionrateisbetween8%and13%.Thereisadeclineinthepercentofcoursevisitorswhobecomeactiveinacoursewitheachsubsequentcourseacrosstheseries.Thiscouldbetheresultofthecoursesbuildingoneachothersoitwouldbedifficultforsomeonetovisitacourseinthemiddleoftheseriesandbecomeactiveiftheyhadnotalreadydonethepriorcourses.ThisislesstrueintheBusinessAnalyticsseries.Chart6:ConversionPercents–JavaProgramming
18%
16%
6%
21%
26%
34%
30%
45%
Course4
Course3
Course2
Course1
VisitorconversiontoactivelearnerActivelearnerconversiontopaid
13%
12%
8%
11%
18%
22%
28%
35%
Course4
Course3
Course2
Course1
Visitorconversiontoactivelearner
Activelearnerconversiontopaid
Page 9 of 15
LearnerFeedbackSurveysAlllearnerswhoenrollinanyofthecoursesineitherSpecializationareemailedasurveyatthebeginningofthecourse.Alllearnerswhosuccessfullycompleteacourseareemailedafollow-upsurveysolicitingtheirfeedback.Thereisanexceptiontothis;pre-coursesurveyswerenotdoneintheJavaProgrammingcoursesbecauseGoogleresearchersweredistributingtheirownsurveysatthebeginningofeachcourseinthatseries.Wereceivedapprovaltoresumesurveysinthosecourseslastmonthsothosedatawillbeavailableinfollow-upstothisreport.Inadditiontothelearnerfeedbacksurveys,wealsocollecteddataonthelearnerexperiencefromtheOnlineCourseBuilderswhowereactiveineachofthecoursesandthushadinsightintowhatlearnersfounddifficultorchallenging.Thereareseverallimitationstothelearnerfeedbacksurveysthatshouldberecognized.First,theyarenotgeneralizabletoalllearners.Onlypeoplewhosuccessfullycompleteacourseareemailedthepost-coursesurveysotheresultsshouldbegeneralizedonlytothatsub-groupofenrolledlearners.Second,evenwithinthegroupofcoursecompleters,theresponseratesarelowandpeopleself-selectwhetherornottotakethesurvey.Itisnotknownhowselectionbiasimpactsthedata,butitlikelydoes.Inspiteofthoselimitations,thedataareveryusefulforindicatingtrendsandpatterns.Whenthedatafromonecourselooksdifferentfromtherest,itindicatesthereislikelyasignificantdifferenceinthatcourse.AnexampleofthisisshowninTable2below.Thistableshowsthemeanoverallcourseratingscoresgiventoeachcourse.Learnerswereasked,oncetheyfinishedtheentirecourse,toassignascorefrom1(verypoor)to7(excellent).AllthecoursesinthetwoSpecializationshaveaveragescoresbetween5.7and6.62withtwoexceptions:BusinessAnalyticscoursetwohasalowerscore(5),andJavaProgrammingcoursefivehasahigherscore(6.6).Basedonthis,weworkedwiththecoursedevelopmentteamandtheinstructorsofthesecondBusinessAnalyticscoursetoidentifywhytheratingwaslowerandtargetrevisionstoaddressproblems.Thoserevisionsarecurrentlybeingcompleted.Table2:OverallCourseRatings(Max.7)
Course1 Course2 Course3 Course4
Course5(capstone)
BusinessAnalytics 5.9 5 6.1 5.8 5.7JavaProgramming 5.7 6.0 6.2 6.1 6.6WeusedtheoverallcourseratingsasastartingpointtoidentifywhylearnersgavetheJavaProgrammingcapstonecoursehigher-than-averageratings.Severalstudentscommentedthattheyfoundthecapstoneprojecttobeagoodlearningexperiencebecauseitusedmanydifferentskillstheyhadlearnedthroughouttheseries.Theyalsopraisedthe“real-world”natureoftheprojectandthefactthattheskillstheyusedwerethingstheywouldputtofutureuse.Forexample,astudentnamedCathywrotethefollowingcommentaboutherexperienceafterthecapstone:
Page 10 of 15
Thiswholespecializationwasfantastic!IwasalinguisticsmajorasanundergradandtriedtolearnJava,butIreallystruggledwithit.ThisfallIamenteringgradschoolforamastersprograminHumanLanguageTechnology.Iamaimingtowardacareerincomputationallinguisticsandnaturallanguageprocessingbutlearningtoprogramhasbeenahugehurdle.NowthatIhavecompletedtheSpecializationIhavesomuchmoreconfidenceinmyprogrammingskillsandIfeelreadytoapplythemingradschool.
Otherstudentscommentedthat,“Thiswasanawesomeproject.ItwasgreatgettingtoimplementeverythingIlearnedoverthecertificateprogram”,and“Thiscapstoneprojectisanicewaytoreinforcethelearningofthematerialinthecourse”.Asanalternativemeasure,wealsocomparedthecoursescoresassignedbylearnersthroughtheCourseraplatform.Learnerscanclickonstarstoassignacoursearatingbetween1and5stars(notethatthescaleusedbyCourseraisa5-pointscalewhileDuke’slearnersurveysreportedinTable2usea7-pointscale).Duke’soverallcourseaverageis4.52andCoursera’soverallaverageis4.63.TheBusinessAnalyticsSpecializationcoursesreceivedratingsof4.5,4.2,4.6,4.7,and4.7.Thisisconsistentwiththepatternseeninthelearnerfeedbacksurveys.TheJavaProgrammingscoresare4.4,4.5,4.7,4.6,and4.7,alsokeepingwiththepatternseeninthelearnerfeedbackandcomparabletoothercoursesofferedontheplatform.Thesehighcourseratesarealsoreflectedinthestudentcommentsprovidedinthefeedbacksurveys.Forexample,alearnerinthefirstcourseintheJavaProgramingSpecializationwrote:
Iwouldliketoexpressmygratitudetothecourseinstructors,Susan,Owen,AndrewandRobertfortheiroutstandingaffordstoteachusprogramming.I'velearntmuchmoreaboutprogrammingandmoreimportantlynowIfeelmyselfconfidenttosolvingprogrammingproblem.TheinstructorsprovideahighqualityofeducationofcomputerscienceinveryusefulmannerandmademefeelI'mreallydoingprogramming.
Similarly,astudentinthethirdBusinessAnalyticscourseofferedpraiseforthewayallthecourseelementsworkedtogether:
ThankstoProf.Jana.Thewholelearningexperienceisexcellent.Thevideolectures,practicequizandgradedquizallworkedtogethertodeliveranexcellentcourse.Itisbyfarthebestinitsclass-beingtechniqueandtooloriented.Prof.Jana'slectureontheneuroscienceofvisualperceptionandattentionisthebestfoundationIhadonvisualizationandcommunication.
Page 11 of 15
AsecondsetofmeasuresweusedtoevaluatethelearnerexperienceintheSpecializationsarequestionsaskingpeopletoself-assesstheirownlearningandtheextenttowhichtheyfeltthecoursewasengaging.Charts7and8belowindicatethepercentoflearnersineachcoursewhosaidtheyagreedwiththefollowingtwostatements:
• FortheamountoftimeIinvestedinthiscourse,I’mhappywithwhatIlearned
• ThecoursematerialswerepresentedinanengagingmannerThedatafromtheBusinessAnalyticsSpecializationconfirmsthepatternindicatedbytheoverallcourseratingscores.Farfewerlearnersinthesecondcoursearehappywithwhattheylearnedorfindthecoursematerialsengaging.IntheJavaProgrammingSpecialization,thepercentoflearnersindicatingeachoutcomeissimilaracrossallcourses.Overall,thelearnersareextremelysatisfiedwiththecourses.Chart7:LearnerSurveys–BusinessAnalytics
93%
60%
95% 90%100%
91%
52%
97%89% 86%
Course1 Course2 Course3 Course4 Course5(capstone)
HappywithwhatIlearnedCoursewasengaging
Page 12 of 15
Chart8:LearnerSurveys–JavaProgramming
StaffandFacultyExperienceWeconductedinterviewswiththeprojectleads,coursesupportstaff,andinstructorstounderstandthestaffandfacultyexperience.Wealsowantedtoidentifyprocessesorexperiencesthatcouldbechangedtoimprovethecoursedevelopmentprocessinthefuture.Duringthecourseofthoseinterviews,severalthemesemergedthatwererepeatedinthemajorityofconversations.Thefollowingsectiondescribesthosethemes.
SettingExpectationsEarlyItisimportanttosetexpectationsforthecoursedevelopmentprocessandtomakeallexpectationsexplicit,notimplicit.Everypersonwespoketocitedatleastonetime,andsometimesmanytimes,duringdevelopmentwhenpeopleonacourseteamhaddifferentexpectationsforhowsomethingwouldbedoneanditcausedprojectdelays,frustration,andpeoplerepeatingwork.Specifictopicsthatshouldbeaddressedup-frontinclude:
• Specificdivisionoftaskswithintheprojectteam,includinginstructors• Howtohandleerrorsandmistakes–whotocontactandhowtheyrespond• Realisticenrollmentnumbersandroyaltyamounts• Expectationsforfrequencyandformatofcommunication• Aprocessforrevisitingandrevisingtaskassignmentsasneeded
Severalpeoplesuggestedhavingalongerkick-offmeetingtoreviewexpectations.Similarly,onepersonsuggestedhavingavideokick-offmeetingfirstandthenwaitingaweekandhavingafullteamkick-off.Thisletstheinstructorsseethe
88% 92% 95% 93% 94%87% 89% 93% 93% 94%
Course1 Course2 Course3 Course4 Course5(capstone)
HappywithwhatIlearnedCoursewasengaging
Page 13 of 15
typesofvideoavailableandcometotheteamkick-offreadytomakedecisions.Anotherpersonsuggestedhavingawork-orientedkick-offmeetingandaseparatekick-offsocialeventtobuildteamcohesion.
NavigatingShortTimelinesAsdiscussedatthebeginningofthisreport,oneofthebiggestchallengeswithboththeseSpecializationswasthattheywerecreatedonaveryshorttimeline–about4weeksforeachcourse.Almosteveryoneweinterviewedmentionedthisasachallenge,andseveralinstructorssaidtheywouldnotdoaprojectlikethisagainunlessthetimelinewasmuchlonger.Weidentifiedseveralreasonswhythetimelinewasproblematicandchangesthatcouldhelpalleviatetheburdencreatedbythetimeline.First,thetimelinewasproblematicbecausecoursebuildingisnotlinear.Theprocessbecomesfasteraspeoplegainexperience;theJavainstructors,forexample,feltthatbuildingcoursefourwentmuchfasterthanbuildingcourseone.Thesettimeline,however,allowed4weeksforeachcourse.Second,therearesometasksduringcoursedevelopmentthatrequireasignificantamountofinactivetime,suchaswaitingforatechnologytooltobebuiltorapproved.Alineartimelineplacestheseeventssequentiallyinsteadofsimultaneously.Finally,otherdemandsonfacultymembers’timearenotconstant.Severalinstructorssaidthattheyhadtimesduringtheprojectwhentheycouldhavedonemorethantheyhadscheduled,andtimeswhentheywereunabletodoanyworkontheprojectatall.Theobviousrecommendationistoallowforalongertimelineonfutureprojects.Fourweeksisnotenoughtimetobuildacourseundermostcircumstances.Beyondthat,someotherpossiblechangesare:
• Identifyaback-uppersonforeachmajortasksothatifthingsfallbehind,thereisapersontohelp
• Beginallinactivetasks(seekingapprovals,gainingcopyrightclearance,workingwithvendors)atthestartoftheproject
• Overlapcoursedevelopmentwhenpossible,suchashavinginstructorsplanallthecoursesinaseriessotheyhavetheoptiontomakelargeamountsofvideocontentformultiplecourseswhentheyfindtimeavailable
QualityAssuranceThetwoSpecializationteamstookdifferentapproachestoqualityassurance.TheJavaProgrammingteamaskedanOnlineCourseBuilderwithastrongtechnicalbackgroundandcodingexperiencetotakeallofthecoursesasalearner,assumingthatifsheexperienceddifficultylearninganyofthematerial,itwouldposeproblemsformostpeople.Thisapproachhelpedthemidentifyseveralplaceswheretheyneededtomakechangestobothvideosandassessments.Therewaswidespreadagreementthat,whiletimeconsuming,thisapproachtoqualityassurancewasveryeffectiveandallowedthemtomakechangesbeforeacourseopenedorshortlythereafter.
Page 14 of 15
TheBusinessAnalyticsteamreliedmoreheavilyonthemodulesurveysthatbeganafteracourselaunchedforfeedback.Theyalsolookedatthingslearnerswereflagginginthecourseandfixederrorswhentheywerepresent.Thiswasaneffectiveapproachforcorrectingerrors,butdidnotallowthemtheoptiontofixthembeforethecourseopened.Itwasalsonotalwaysclearwhowasresponsibleforlookingatfeedbackandflagsandmakingchanges;sometimestheprojectstafffeltthattheinstructorsneededtomakechangessincetheywerethecontentexpertsbutsometimestheerrorswereissuesbestfixedbypeoplemorefamiliarwiththeplatform.Bothprojectteamsagreedthatbetatestingofcourseswasveryimportant.Giventheshorttimeline,betatestingdidnotalwayshappenbeforeacourseopened.Whenitdidnothappen,therewerenoticeablymorelearner-reportederrors.
SupportStaffAfull-timededicatedTeachingAssistant(TA)whowashiredbythefacultymembersassistedintheJavaProgrammingcoursesbyansweringlearnerquestionsandparticipatinginthediscussionforums.About5-10hoursaweekofhertimeisspentrespondingtolearnerquestions.Shealsoassistswithdevelopingandimprovingcontent,creatingassessments,anddesigningA/Bteststoexperimentwithnewmaterials.Shealsohelpedthecoursementors2byansweringtheirquestionsandprovidingguidanceonhowtheyshouldrespondtovariousissues;shetypicallyrespondedthesamedayaquestionwasposted.ItislikelythatthereasonpeoplevolunteeredtobementorsisbecausetheTAwasactiveinthecoursefromthebeginningandlearnerssawthediscussionforumsasawaytogetinvolved.ItisclearthathavingadedicatedTAwhoisacontentexperthasgreatlyenhancedthelearnerexperienceintheJavaProgrammingcoursesandbeeninstrumentalinmakingthediscussionforumsatoolforactiveandengagingdiscussions.Incontrast,theBusinessAnalyticsSpecializationsdidnothaveadedicatedTAandtheinstructorswerenotactiveinthediscussionforums.Thisresultedinlearnerpostswithquestionsnotreceivingquickresponseswhenthequestionswerecontent-related(CITstaffmostlyrespondtonon-contentquestionssuchashowtoaccessaresource).Overtime,somelearnershavevolunteeredtobementors,butthey have not received a lot of direction on what they should do. One volunteer mentor posted in the dedicated mentor forum:
I'mhavingtroublekeepingupalreadywiththerequestsforhelpintheforums,especiallysinceI'vebeenawayfromthismaterialfornearlyayearandwillneedtimetore-masterit.Arethereothermentorsforthiscoursewhocanhelp?
2Courseracoursementorsarelearnerswhosuccessfullycompletedthecourseandthenvolunteeredtomentorothersthroughdiscussionforumparticipation.
Page 15 of 15
Ittook17daysforsomeonetorespondtothisquestion,andtheeventualresponsecamefromanothermentor.SeveraloftheweeklydiscussionforumsintheBusinessAnalyticscoursesdonothaveanyactivediscussionsfor1-2monthsatatime.ThiswasespeciallyproblematicintheMySQLclassbecauselearnersfoundittobeatechnicallydifficultcourseandthereforehadalotofquestions.Thecourseteamrespondedbyhiring2part-timeTAstospend7-10hoursaweekrespondingtoquestionsintheforums.Thecourseteamreportsthatthishasmakeasignificantpositiveimpactonlearnersatisfactionandreducedthenumberofquestionsbeingaskedbecauselearnerscansearchforexistingthreadsthatanswertheirquestion.Wheneverpossible,atleastapart-timeTAwithcontentexpertiseshouldbehiredforfuturecourses.
RevisionsBasedonapreponderanceofevidence,theBusinessAnalyticsteamtargetedcoursetwoforrevisions.Weanalyzedalltheavailabledataandprovidedthecourseprojectteamwithareportrecommendingchangesandidentifyingtargetsbywhichtomeasuresuccess.Thecoursewasre-launchedwithrevisionsasofSeptember1.Atthistime,wehaveonlycollected17post-coursesurveyssoitistooearlytoassesswhethertherevisionshavebeensuccessful.However,basedonreadingstudentcommentsinthemodulesurveys,itappearsthattherevisionshaverectifiedmostifnotalltheproblems.