8th Sakai Conference

4-7 December 2007Newport Beach

Programmatic Assessment Using Programmatic Assessment Using Goal-Aware Activities in SakaiGoal-Aware Activities in Sakai

Daniel Tyger (SVSU) & Noah Botimer (UM)

2

Assessment Project SiteAssessment Project Site• Mid-MI Regional State Institution• Saginaw Valley State University (since 1963)• ~9500 Students• Blackboard (BASIC) since 2000• Began piloting Sakai in SP ’06• Around 50 / 50 adjunct / full-time faculty

3

Project GoalsProject Goals• Programmatic Data Sought

– NCATE – accreditation-related, granular + summative data

– Programmatic change indicators over time

• “Digital,” “Outcomes-based,” “portfolio” all used during the search for the perfect software that does everything OOTB for an indefinite period of time.

• Reporting data and digital student products both desired. (This is possible)

• Must be easy for faculty (and techs) and ‘natural’ for students.– “Matrices” were failing us here…

4

Primary Target AudiencesPrimary Target Audiences

• 5 College of Education Programs– Reading MAT– Elementary Education (Undergraduate)– Elementary Education MAT (Graduate)– Early Childhood Education– Special Education

• Emotionally-Impaired• Cognitively-Impaired• Learning-Disabled

• Sociology Department (recent project)

5

PrerequisitesPrerequisites

• Program faculty reach consensus:– Program goals (if not pre-determined by

accrediting agency or national authority)– Benchmark assignments / activities that meet

program goals– Earmark specific courses where these activities

would be delivered – Rating scale and interpretation / application of it

» Show assignment summary example» Show rating scales / definitions

6

Program WorksitesProgram Worksites• House template activities for the program’s courses• All program instructors enrolled in site• House program goals / goal set(s)• Program rating scale source• Editable by program coordinator (only!)• Attach authoritative instructions, rubric to each

activity• Reports and activity summaries in Resources• Requires established deadline dates before

“injection” and reporting

» Program worksite» Show goal set example

7

Program Coordinator RoleProgram Coordinator Role

• Consolidate proposed program changes for ITS by deadline– Changes in an activity’s linked goals– Activity name changes– Additional activities

• Maintain program worksite semester-semester– Accurate, generic activity instructions for each activity– Accurate attached rubric(s) for each activity

• Communicate to program faculty their responsibilities

8

Faculty RoleFaculty Role

• 2 major responsibilities– Have students submit assignment(s)– Rate student performance against linked goals

• Training Issues?– Material online, email, phone– Workshops

• Faculty impressions: “Easy” “No big deal”

» Show training materials site

9

Student RoleStudent Role

• Normal workflow and expectations– Makes sense to students– Something they do anyway

• Complete and submit assignments, as directed by faculty

• Non-punitive (not graded, per se)

• Student acclimates to concept of identifying benchmark assignments / learning outcomes

10

Sakai Tools UtilizedSakai Tools Utilized

• Goal Management– Provides goal linking and rating for activities

• Assignments– Used where a student product is submitted,

evaluated against a rubric, and rated on each linked goal. Course must be published.

• Data Points– Used when student achievement toward a goal is

rated, but a submitted student product is not necessary.

11

Data Model / StrategyData Model / Strategy

• Mapped program enables automation for consistency and ease of use

• Comprehensive solution for outlined assessment scenarios

• Builds “Assessment Management” on top of Goal Management

» Show Data Model

12

Program Tracking SetupProgram Tracking Setup

• Associate program with program worksite ID• Associate assessments with a program• Associate activities to assessments• Extract activity template references from

program site activities• Associate activities to course sites / sections

(“Deployment Context[s]”)

» NCATE.adp shots

13

Deployment a.k.a. “Injection”Deployment a.k.a. “Injection”

14

Deployment a.k.a. “Injection” Deployment a.k.a. “Injection”

• Injection query (all activities sites)

• Web services + helper scripts (Python)– Create all program course sites that don’t

already exist, roster them– Associate course sites with program worksites– Verify needed tools (Assignments / Data Points)– Copy template activities into each site– Copy linked goals to each activity

15

ReportingReporting• Near semester close…

– ITS runs audit of all program sites for existing activity / rating counts

– Forward to program coordinator• Contact delinquent faculty to rate their activities

• Granular Data– Every rating on every goal on every activity in a program

can be collected across adjunct and full-time faculty• Summative automated reports – next step• BIRT Designer / Engine?

» Show audit tool and example» Show report example

16

Schedule of Tasks / DeadlinesSchedule of Tasks / Deadlines

• Agreement needed here

• Accountability, accuracy, punctuality needed

17

Longitudinal ConcernsLongitudinal Concerns

• Archive snapshot of what happens each deployment period for future reference– Activities change over time– Linked goals may change– New activities added / removed over time– Comparisons / longitudinal studies complicated

18

ProblemsProblems

• Communication & Coordination• Faculty are very busy when programmatic

changes are due• Consistent interpretation of rating scale and

application of that scale across many instructors

• Interpreting what NCATE actually wants?• Faculty removing or altering activities after

injection

19

FutureFuture

• “Assessment Management” (?) service as a contrib tool (Q1 2008)

• Program Mapper web interface

• Course Management integration

• BIRT integration for stock and ad-hoc assessment reports