1|P a g e

ASSESSMENT REPORT ACADEMIC YEAR 2018 – 2019

REPORT DUE DATE: 10/26/2019

Who should submit the report? – All majors, minors (including interdisciplinary minors), graduate and non-degree granting certificate programs of the College of Arts and Sciences. Programs can combine assessment reports for a major and a minor program into one aggregate report as long as the mission statements, program learning outcome(s) evaluated, methodology applied to each, and the results are clearly delineated. Note: Dear Colleagues: In an effort to produce a more streamlined and less repetitive assessment report

format, we are piloting this modified template for the present annual assessment cycle. We are requesting

an assessment report that would not exceed eight pages of text. Supporting materials may be appended. We

will be soliciting your feedback on the report as we attempt to make it more user-friendly.

Some useful contacts:

1. Prof. Alexandra Amati, FDCD, Arts – adamati@usfca.edu

2. Prof. John Lendvay, FDCD, Sciences – lendvay@usfca.edu

3. Prof. Mark Meritt, FDCD, Humanities – meritt@usfca.edu

4. Prof. Michael Jonas, FDCD, Social Sciences – mrjonas@usfca.edu

5. Prof. Suparna Chakraborty, AD Academic Effectiveness – schakraborty2@usfca.edu

6. Ms. Corie Schwabenland, Academic Data & Assessment Specialist- ceschwabenland@usfca.edu

Academic Effectiveness Annual Assessment Resource Page:

https://myusf.usfca.edu/arts-sciences/faculty-resources/academic-effectiveness/assessment

Email to submit the report: assessment_cas@usfca.edu

Important: Please write the name of your program or department in the subject line.

For example: FineArts_Major (if you decide to submit a separate report for major and

minor); FineArts_Aggregate (when submitting an aggregate report)

<Master of Science in Data Science >

2|P a g e

I. LOGISTICS & PROGRAM LEARNING OUTCOMES

1. Please indicate the name and email of the program contact person to whom feedback should be

sent (usually Chair, Program Director, or Faculty Assessment Coordinator).

Nicholas Ross (ncross@usfca.edu)

Stephen Devlin (smdevlin@usfca.edu)

2. Were any changes made to the program mission statement since the last assessment cycle in

October 2018? Kindly state “Yes” or “No.” Please provide the current mission statement below.

If you are submitting an aggregate report, please provide the current mission statements of both

the major and the minor program.

No.

3. Were any changes made to the program learning outcomes (PLOs) since the last assessment cycle

in October 2018? Kindly state “Yes” or “No.” Please provide the current PLOs below. If you are

submitting an aggregate report, please provide the current PLOs for both the major and the minor

programs.

Note: Major revisions in the program learning outcomes need to go through the College

Curriculum Committee (contact: Professor Joshua Gamson, gamson@usfca.edu). Minor editorial

changes are not required to go through the College Curriculum Committee.

Yes, our PLOs were changed in December of 2018 and ratified by a vote of the faculty to the following:

1. Possess a theoretical understanding of classical statistical models (e.g., generalized linear models,

linear time series models, etc.), as well as the ability to apply those models effectively.

2. Possess a theoretical understanding of machine learning techniques (e.g., random forests, neutral

networks, naive Bayes, k-means, etc.), as well as the ability to apply those techniques effectively.

3. Effectively use modern programming languages (e.g., R, Python, SQL, etc.) and technologies (AWS,

Hive, Spark, Hadoop, etc.) to scrape, clean, organize, query, summarize, visualize, and model large

volumes and varieties of data.

3|P a g e

4. Prepare for careers as data scientists by solving real-world, data-driven, business problems with

other data scientists, and understand the social, ethical, legal, and policy issues that increasingly

challenge and confront data scientists.

5. Develop professional communication skills (e.g., presentations, interviews, email etiquette, etc.),

and begin integrating with the Bay Area data science community.

4. Which particular Program Learning Outcome(s) did you assess for the academic year 2018-2019?

TwoPLOswereassessedthisyear:

(1) Program Learning Outcome 4. Be prepared for careers as data scientists by solving real-world

data-driven business problems with other data scientists.

(2) Program Learning Outcome 5. Develop professional communication skills (e.g., presentations,

interviews, email etiquette, etc.), and begin integrating with the Bay Area data science

community.

II. METHODOLOGY

5. Describe the methodology that you used to assess the PLO(s).

For example, “the department used questions that were inputted in the final examination

pertaining directly to the <said PLO>. An independent group of faculty (not teaching the course)

then evaluated the responses to the questions and gave the students a grade for responses to those

questions.”

Important Note – WSCUC advises us to use “direct methods” which relate to a direct evaluation of

a student work product. “Indirect methods” like exit interviews or student surveys can be used only

as additional l complements to a direct method.

For any program with fewer than 10 students: If you currently have fewer than 10 students in your

program (rendering your statistical analysis biased due to too few data points), it is fine to describe

a multi-year data collection strategy here. It would be important to remember that every 3 years,

we would expect you to have enough data to conduct a meaningful analysis.

Important: Please attach, at the end of this report, a copy of the rubric used for assessment.

Threedatasciencefacultycontributedrepresentativequestionsfromoneofmoreexamorquiz.

4|P a g e

Thetablebelowcontainsthelistofcoursesthatarebeingassessedalongthesedimensions.Adescriptionoftheassessmentplanfordifferentgroupsofcoursescanalsobefoundinthedescriptionsbelow.

Traditional Course Assessment: 610:CommunicationsforAnalytics603:ProductAnalyticsWechoosetoassessthefinalpresentationsforbothProductAnalyticsandCommunicationsforAnalytics.RubricsforthefinalpresentationcanbefoundinAdditionalMaterial#9and#10.Notethatproductanalyticsisagroupproject-basedcourses(studentsworkinteamsbetween5-10peopletobuildaweb-appover7weeks)whileCommunicationsforAnalyticsisamoretraditionalcourse.ThedistributionofFinalpresentationgradesforCommunicationsforAnalyticscanbefoundbelow.Notethatthepresentationwasoutof22points.

Wedefinemasteryas22,incompletemasteryas21,partialmasteryas19or20anddidnotmasteratlessthan18.Theunderlyingdimensionsofthisassessmentarealsoshowninthetablebelow.Notethatdifferentcriteriawereoutofdifferentamountofpoints.

0%

5%

10%

15%

20%

25%

30%

35%

40%

<=18 19 20 21 22

PercentofStudentsScores

5|P a g e

Fromtheabovewecanseethatthemostcommonfailuresarealongverbalcommunication.Forproductanalyticswelookedatthefinal,in-coursepresentationandthiswasassessedagainsttherubricintheadditionalmaterials.Thebox-and-whiskerchartbelowcontainsthedistributionoffinalgrades.Thehighwas63andthelowwas43withmoststudentsscoringbetweenhigh40sandhigh50s.

6|P a g e

Breakingdownthetotalbyeachsectionyieldsthefollowingtable.Notethatsomeofthecriteriaareoutof20andsomeareoutof10,whichiswhytheheadingsarelabeledassuch. Presentation

SkillsPresentation Content

Product Demonstration

Product Polish

Perfect 2 2 3 318+/8+ 4 2 2 715+/6+ 3 5 7 3Other 4 4 1 0Ascanbeseenintheabovethe“ProductDemonstration”areawaswherestudentsdidnotdemonstratethehighestlevelofmastery. Direct Job Assessment: 640:SeminarSeriesI641:SeminarSeriesII642:SeminarSeriesIII643:SeminarSeriesIV644:SeminarSeriesV10HoursofrequiredinterviewtrainingWedirectlyassessthesebylookingattheratebywhichstudentsgetjobs.MSDS program outcomes are strong. Given the industry focus of the program, the primary metric for evaluation is employment and almost all students receive offers before or shortly after completion of the program. See Table below. (Note that for 2019, we are only 3 months out from graduation, but have already obtained 95% with offers and have a current median salary of $120,000. Detailed data below.

Median starting salaries remain consistent across demographics, with women and international students earning salaries equivalent to those of all graduates. Female and international graduates in 2018 earned a median base salary of $120,000. Graduates with no prior work experience earned a median base salary of $115,000. The return on investment of this program remains very high as graduates who were working before entering the program increased their median base salary by an average of $50,000 after graduation.Since 2012, 52% of graduates were hired with the title of data scientist, 14% as data analysts, and 8% as machine learning or data engineers. The top companies that hire from our alumni pool include Amazon (9), Apple (5), Facebook (9), Google (6), Pinterest (4), Trulia (4), Uber (5), and Walmart (4). Graduates continue to advance in their careers as 17% of graduates now hold the title of senior, manager, director or lead.

7|P a g e

As part of the practicum class, faculty mentors work closely with students to ensure maximal success on the job market. This includes several rounds of editing resumes and vetting them for appropriateness for technical job applications. Since acquiring a job is process with much randomness, faculty also track via spreadsheet the number of weekly job applications to ensure that the volume is sufficient. Students who have struggled on the job market often fail to apply to enough positions.

In the Appendix of this document (Additional Material #1, #2 and #3), you can find material showing specific examples of how these things are tracked, how communication occurs and

Direct Job Assessment & Behavioral Assessment: 605:PracticumI625:PracticumII627:PracticumIII632:PracticumIVThepracticumistherequiredinternshipprogramassociatedwiththeMastersofDataScience.Studentsworkinavarietyofdifferentindustriesandundermultipleworkingconditions.Duetothevariabilityinthework(aswellasNDAsandotheremploymentagreements),direct,on-the-job,performanceassessmentisdifficulttoconsistentlyget.Wehaveamulti-prongedstrategyonpracticumassessment:

1. Regularcheck-inswitheachcompany(seeexampleemailinAdditionalMaterials#6).Periodicemailsaresenttothecompanymentorstoverifythatthingsareprogressinginaccordancewiththeirexpectations.

2. Facultymentorcheck-ins.Thepracticumdirectorspeakswiththefacultymentorsonaregularbasistoverifythatstudentsareperforming.

3. WeeklyPracticumupdateform(seeAdditionalMaterial#7fortheform).Studentsarerequiredtodoaweeklypracticumupdateformwhichcontainsinformationabouttheworkthattheyhavedone.

8|P a g e

III. RESULTS & MAJOR FINDINGS

6. What are the major takeaways from your assessment exercise?

This section is for you to highlight the results of the exercise. Pertinent information here would

include:

a. how well students mastered the outcome at the level they were intended to,

b. any trends noticed over the past few assessment cycles, and

c. the levels at which students mastered the outcome based on the rubric used.

To address this, among many other options, one option is to use a table showing the

distribution, for example:

Level Percentage of Students

Complete Mastery of the outcome 8.7%

Mastered the outcome in most parts 20.3%

Mastered some parts of the outcome 66%

Did not master the outcome at the level

intended

5%

For these PLO’s are major assessment is job placement and making sure that students get

jobs within 3 months of graduation in the field of data science. Based on that metric, over

90% of students have demonstrated mastery of the material.

Level Percent of Students

Complete Mastery of the outcome >90%

Not complete Masters of the outcome <10%

9|P a g e

IV. CLOSING THE LOOP

7. Based on your results, what changes/modifications are you planning in order to achieve the

desired level of mastery in the assessed learning outcome? This section could also address more

long-term planning that your department/program is considering and does not require that any

changes need to be implemented in the next academic year itself.

Short term: Product analytics will add more details around how to do a better product

demonstration, as this was obviously a weak point for the students.

Longer term: Continue monitoring and investing in job outcomes and placement to

make sure that we continue on this path.

We may also want to talk about having better gradations around our placement metrics.

For example, tracking number of jobs before graduation, etc. in order to identify

potential slippage in outcomes.

10|P a g e

8. What were the most important suggestions/feedback from the FDCD on your last assessment

report (for academic year 2016-2017, submitted in October 2017)? How did you incorporate or

address the suggestion(s) in this report?

N/A

ADDITIONAL MATERIALS

(Any rubrics used for assessment, relevant tables, charts and figures should be included

here)

Additional Material #1: Sample student Job spreadsheet Thisisasampleofoneofthestudent’sjobspreadsheetthattheyarerequiredtododuringtheirtimeintheprogram.Itcontainsinformationonwhatjobstheyhaveappliedto,whentheyappliedandwhatresponsetheyreceived.Allstudentsarerequiredtofillthisoutuntiltheygraduate.

11|P a g e

Additional material #2: Job Tracking Spreadsheet Wehavetwostagesofmonitoringjoboutcomes.Thefirststageinvolvestakinginformationfromeachstudent’sspreadsheetandcollatingitintoasingledocument.Ascreenshotofthisdocumentisshownbelow.

12|P a g e

Eachrowrepresentsastudentandeachcolumnrepresentsthetotalnumberofjobsthattheyhaveappliedtouptillthedatelisted.Wetrackthisinformationforeverystudent,makingsurethattheyareapplyingtojobswhichareappropriateandprogressingaccordingly.Additional Materials #3: Sample Job Email Theemailscreenshotbelowisanexampleoftheemailsenttopracticummentorsforeachstudenteachweek.Practicummentorship(fromthefacultytothestudent)includesassistingthestudentsintheirjobsearch.Theemailbelowissenteachweekandgiveseasyaccesstostudent’sjobperformance.

13|P a g e

Additional Material #4: List of Program Learning Outcomes

1. Possess a theoretical understanding of classical statistical models (e.g., generalized linear models,

linear time series models, etc.), as well as the ability to apply those models effectively.

2. Possess a theoretical understanding of machine learning techniques (e.g., random forests, neutral

networks, naive Bayes, k-means, etc.), as well as the ability to apply those techniques effectively.

3. Effectively use modern programming languages (e.g., R, Python, SQL, etc.) and technologies (AWS,

Hive, Spark, Hadoop, etc.) to scrape, clean, organize, query, summarize, visualize, and model large

volumes and varieties of data.

4. Prepare for careers as data scientists by solving real-world, data-driven, business problems with

other data scientists, and understand the social, ethical, legal, and policy issues that increasingly

challenge and confront data scientists.

5. Develop professional communication skills (e.g., presentations, interviews, email etiquette, etc.),

and begin integrating with the Bay Area data science community.

14|P a g e

Additional Material #5: PLO Course mapping BelowyoucanfindthemappingofcoursestoPLOs.

15|P a g e

Additional Material #6: Check in email example

Additional Material #7: Weekly check-in form

16|P a g e

Additional Material #8: List of 2019 Outcomes 95%withanofferwithin3monthsofgraduating.Salaries: Medianbasesalary:$120kMedianbasesalary,internationalstudents:$125kMedianbasesalary,women:$124kJob Titles: MostCommonJobTitle:DataScientist(63%)SecondCommonJobTitle:DataAnalyst(10%)ThirdCommonJobTitle:DataEngineer(6%)FourthCommonJobTitle:MachineLearningEngineer(4%)Job titles include: AppliedScientistComputerVisionEngineerConsultantDataAnalystDataAnalyticsEngineerDataEngineerDataScienceArchitectDataScientistDecisionScientistDeepLearningEngineerMachineLearningDataScientistMachineLearningEngineerResearchAnalyticsScientistSoftwareDevelopmentEngineerCompanies Aktana ManifoldInc ZimmermanAdvertisingAmazon Metromile ZumeIncAmazonWebServices NakedPoppy AppleContractorApple O’ReillyMedia VerizonMediaBeamSolutions Oracle BirdRides PicnicHealth Booking.com PocketGems Checkr,Inc RHL Clari RavelLaw ConeHealth RemedyPartners DRIVIN Scribd DataGrand Sentry(sentry.io)

17|P a g e

Dictionary.com Shutterfly EXLService SiriusXM Earnest SmithsDigitalForge Emailage StanfordUniversity FLYR ThirdLove Fair Trulia GoGuardian Twitch HappyMoney Scribd Hoodline UltimateSoftware Ibotta VerizonMedia Iherb Vian.ai KomodoHealth Vian.ai Lendingclub VisaInc LexisNexisGroup VisaInc LinkedIn WalmartLabs LogisticsManagementInstitute(LMI)

Wish

MOBE Wish ZSAssociates UltimateSoftware

18|P a g e

Additional Material #9: Business Communications Final Presentation Rubric

19|P a g e

Additional Material #10: Grading Rubric for Final Presentation Product Analytics