Institutional Research and the Institutional Research and the Campus Commitment to a Campus Commitment to a

Culture of AssessmentCulture of Assessment

AIRPO ConferenceAIRPO ConferenceJune 20, 2006June 20, 2006

James A. Anderson, Ph.D.James A. Anderson, Ph.D.Vice President for Student SuccessVice President for Student Success

Vice Provost for Institutional Assessment and DiversityVice Provost for Institutional Assessment and DiversityUniversity at AlbanyUniversity at Albany

JAANDERSON@UAMAIL.ALBANY.EDUJAANDERSON@UAMAIL.ALBANY.EDU

Integrative Assessment Integrative Assessment ReportingReporting

Institutional Goals or ObjectivesInstitutional Goals or Objectives

College/Division/Accreditation Goals and College/Division/Accreditation Goals and ObjectivesObjectives

Department/Program Department/Program Goals/Objectives/OutcomesGoals/Objectives/Outcomes

Course-Bases Objectives/OutcomesCourse-Bases Objectives/Outcomes

Integrated Assessment Process Integrated Assessment Process Means IntegratingMeans Integrating

Institutional EffectivenessInstitutional Effectiveness

Program ReviewProgram Review

Outcomes AssessmentOutcomes Assessment

Institutional ResearchInstitutional Research

Enrollment ManagementEnrollment Management

Survey ResearchSurvey Research

Student-Centered Learning Student-Centered Learning EnvironmentEnvironment

Refers to a student success effort which is Refers to a student success effort which is characterized by:characterized by:

– Clearly defined and measurable outcomes in the Clearly defined and measurable outcomes in the areas of teaching, learning, retention, and student areas of teaching, learning, retention, and student social-psychological developmentsocial-psychological development

– Front-loaded activities which engage the student, as Front-loaded activities which engage the student, as soon as possible, to the academic and social soon as possible, to the academic and social expectations of the institutionexpectations of the institution

– An ongoing assessment effort associated with student An ongoing assessment effort associated with student and program improvementand program improvement

– Faculty and staff who are well-trained to deliver the Faculty and staff who are well-trained to deliver the appropriate servicesappropriate services

Profiling the Student Support ClienteleProfiling the Student Support Clientele

1.1. What institutionally based information on student needs and What institutionally based information on student needs and characteristics is utilized by student support personnel?characteristics is utilized by student support personnel?

2.2. What specific information about student subpopulations What specific information about student subpopulations (women, students of color, non traditional students, etc.) is (women, students of color, non traditional students, etc.) is utilized by student support personnel? utilized by student support personnel?

3.3. What information about student culture, Faculty concerns What information about student culture, Faculty concerns and campus climate is utilized by student support and campus climate is utilized by student support personnel?personnel?

4.4. Describe a profile of the students who utilize studentDescribe a profile of the students who utilize studentsupport services the most/least?support services the most/least?

5.5. Who benefits the most/least? Who benefits the most/least?

Generate Longitudinal Database to Monitor Generate Longitudinal Database to Monitor Key Indicators of Student SuccessKey Indicators of Student Success

ExamplesExamples

Overall course completion ratiosOverall course completion ratiosSuccessful completion of gatekeeper coursesSuccessful completion of gatekeeper coursesSuccessful completion of developmental educational Successful completion of developmental educational coursescoursesSemester-to-semester retentionSemester-to-semester retentionYear-to-Year retentionYear-to-Year retentionDegree and certificate attainment ratesDegree and certificate attainment rates

From: “Achieving the Dream: Community Colleges Count” Lumina FoundationFrom: “Achieving the Dream: Community Colleges Count” Lumina Foundation & Pell Institute& Pell Institute

Retention ScorecardRetention Scorecard

Indicators of Institutional EffectivenessIndicators of Institutional Effectiveness

Indicators of Program EffectivenessIndicators of Program Effectiveness

Indicators of Student Academic/Social Indicators of Student Academic/Social IntegrationIntegration

Indicators of Student Success/AchievementIndicators of Student Success/Achievement

Indicators of Student LearningIndicators of Student Learning

Indicators of Impact of/on DiversityIndicators of Impact of/on Diversity

Questions That May Help You Recast Questions That May Help You Recast Your Services or Delivery SystemsYour Services or Delivery Systems

What are the thinking tasks, intellectual experiences, What are the thinking tasks, intellectual experiences, and/or cocurricula experiences that need to be and/or cocurricula experiences that need to be designed relative to the preparation level and designed relative to the preparation level and diversity of the students at your institution?diversity of the students at your institution?

What structures need to evolve to assure that What structures need to evolve to assure that students have the opportunity to enhance their students have the opportunity to enhance their academic self-concept and understand their role in academic self-concept and understand their role in the culture of learning at your institution?the culture of learning at your institution?

What data sets do I need to access to generate the What data sets do I need to access to generate the appropriate questions and answers about student appropriate questions and answers about student engagement and student learning?engagement and student learning?

Academic successAcademic success Transition to First Year of Transition to First Year of CollegeCollege

Assessment and Self-Assessment and Self-DiscoveryDiscovery

|| || ||

Developmental/Developmental/

general curriculum general curriculum coursescourses

Intrusive advisingIntrusive advising Learning stylesLearning styles

|| || ||

Academic supportsAcademic supports College course analysis and College course analysis and time managementtime management

Non-cognitive factorsNon-cognitive factors

|| || ||

Study skill Study skill assessmentassessment

Information and computer Information and computer literacyliteracy

Career explorationCareer exploration

|| || ||

Test-taking skillsTest-taking skills Co-curricular experiencesCo-curricular experiences Active learning strategiesActive learning strategies

||

Active learning Active learning strategiesstrategies

Adapting to CultureAdapting to Culture

of Collegeof College

Integrated Model of Undergraduate Education

ACADEMIC OUTCOMESACADEMIC OUTCOMES

Grades (General)Grades (General)

Grades (By Course Type)Grades (By Course Type)

Retention (Semester to Semester) Retention (Semester to Semester)

11stst to 2 to 2ndnd Year Year

Graduation RatesGraduation Rates

Time to Degree (Calendar vs. Seat Time)Time to Degree (Calendar vs. Seat Time)

Changes in Academic InterestsChanges in Academic Interests

Transfers to Four-Year InstitutionsTransfers to Four-Year Institutions

STUDENT OUTCOMESSTUDENT OUTCOMES

Academic Adjustments/IntegrationsAcademic Adjustments/Integrations

Social Adjustments/IntegrationSocial Adjustments/Integration

Attitudes Towards LearningAttitudes Towards Learning

Expectations for SuccessExpectations for Success

Academic Self-EsteemAcademic Self-Esteem

Utilization of Academic SupportsUtilization of Academic Supports

Non-Cognitive FactorsNon-Cognitive Factors

Questions to Ask YourselfQuestions to Ask Yourself

Which outcome and assessment methods Which outcome and assessment methods will help me understand what it is that I will help me understand what it is that I am doing that is leading to the outcome?am doing that is leading to the outcome?

Which outcome and assessment methods Which outcome and assessment methods help me understand why I am doing what I help me understand why I am doing what I am doing?am doing?

Will this kind of evidence help me make Will this kind of evidence help me make the decisions I need to make? the decisions I need to make?

Components of An Assessment PlanComponents of An Assessment Plan

MissionMissionObjectivesObjectivesOutcomesOutcomesEvaluation MethodsEvaluation Methods– By OutcomesBy OutcomesImplementation of AssessmentImplementation of Assessment– Who is Responsible for What?Who is Responsible for What?– TimelineTimelineResultsResultsDecisions and RecommendationsDecisions and Recommendations

Clark College Clark College (Allied Health Program)(Allied Health Program)

Category ACategory A– SATSAT– HS TranscriptHS Transcript– Personal InterviewsPersonal Interviews– RecommendationsRecommendations

Category BCategory B– California Study TestCalifornia Study Test– Coop Eng. TestCoop Eng. Test– Watson-Glaser Critical ThinkingWatson-Glaser Critical Thinking– Appraisal Appraisal

Category CCategory C– Basic Information SheetBasic Information Sheet

(Personal Information & Personal/Social Adjustment)(Personal Information & Personal/Social Adjustment)

Program OutcomesProgram Outcomes

Students in your program will develop a positive Students in your program will develop a positive attitudes towards learning and increase their attitudes towards learning and increase their commitment and responsibility for their own commitment and responsibility for their own learninglearning– Pre-post gains or comparison with matched pair Pre-post gains or comparison with matched pair

cohort on dimensions like academic skills, critical cohort on dimensions like academic skills, critical thinking, values toward learning, etc.thinking, values toward learning, etc.

– Develop a plan of studyDevelop a plan of study– Develop a learning portfolioDevelop a learning portfolio– Learning environment preferences Survey (Perry Learning environment preferences Survey (Perry

Scheme)Scheme)

Reasons for Learning Style Assessments Reasons for Learning Style Assessments that Incorporate Diversitythat Incorporate Diversity

Self-assessment feedbackSelf-assessment feedback

Cohort comparisonsCohort comparisons

Cluster analysis of behaviorsCluster analysis of behaviors

Development of effective cooperative Development of effective cooperative clustersclusters

Matching of learning styles/teaching stylesMatching of learning styles/teaching styles

Correlation with other dimensionsCorrelation with other dimensions

Identification of critical dimensionsIdentification of critical dimensions

Meaningful Use of DataMeaningful Use of Datafrom Peggy Maki, Ph.D.from Peggy Maki, Ph.D.

Collect data from different sources to make a Collect data from different sources to make a meaningful point (meaningful point (for example, program samples for example, program samples and other samples of student work).and other samples of student work).Collect data you believe will be useful to Collect data you believe will be useful to answering the important questions you have answering the important questions you have raised.raised.Collect data that will help you make decisions for Collect data that will help you make decisions for continuous improvement.continuous improvement.Organize reports around issues, not solely data.Organize reports around issues, not solely data.Interpret your data so that it informs program Interpret your data so that it informs program improvement, budgeting, planning, decision-improvement, budgeting, planning, decision-making, or policies. making, or policies.

Some Methods That Provide Some Methods That Provide Indirect EvidenceIndirect Evidence

from Peggy Maki, Ph.D.from Peggy Maki, Ph.D.

Alumni, Employer, Student SurveysAlumni, Employer, Student Surveys

Focus groupsFocus groups

Exit Interviews with GraduatesExit Interviews with Graduates

Graduate Follow-up StudiesGraduate Follow-up Studies

Percentage of students who go on to graduate Percentage of students who go on to graduate schoolschool

Retention and Transfer StudiesRetention and Transfer Studies

Job Placement StatisticsJob Placement Statistics

Indirect EvidenceIndirect Evidence Cont.Cont.

Courses selected or elected by studentsCourses selected or elected by studentsFaculty/Student ratiosFaculty/Student ratiosPercentage of students who study abroadPercentage of students who study abroadEnrollment trendsEnrollment trendsPercentage of students who graduate Percentage of students who graduate within five-six yearswithin five-six yearsDiversity of student bodyDiversity of student bodyCAS StandardsCAS Standards

Some Methods That Provide Some Methods That Provide Direct EvidenceDirect Evidence

Student work samplesStudent work samplesCollections of student work (e.g. Collections of student work (e.g. Portfolios)Portfolios)Capstone projectsCapstone projectsCourse-embedded assessmentCourse-embedded assessmentObservations of student behaviorObservations of student behaviorInternal juried review of student projectsInternal juried review of student projectsEvaluations of performanceEvaluations of performance

Direct EvidenceDirect Evidence Cont.Cont.from Peggy Maki, Ph.D.from Peggy Maki, Ph.D.

External juried review of student projectsExternal juried review of student projectsExternally reviewed internshipExternally reviewed internshipPerformance on a case study/problemPerformance on a case study/problemPerformance on problem and analysis (Student Performance on problem and analysis (Student explains how he or she solved a problem)explains how he or she solved a problem)Performance on national licensure examinationsPerformance on national licensure examinationsLocally developed testsLocally developed testsStandardized testsStandardized testsPre-and post-testsPre-and post-testsEssay tests blind scored across unitsEssay tests blind scored across units

Building an Assessment WebsiteBuilding an Assessment Website

Easy to navigateEasy to navigate

DefinitionsDefinitions

Principles of Student LearningPrinciples of Student Learning

Resources and Tool KitsResources and Tool Kits

Projects and PortfoliosProjects and Portfolios

Presentations and PapersPresentations and Papers

Contact InformationContact Information

Community College Assessment Community College Assessment ChallengesChallenges

Academic programsAcademic programsAdministrative servicesAdministrative servicesStudent development programsStudent development programsIntegrate/connect proposed and ongoing assessment Integrate/connect proposed and ongoing assessment efforts across units and levelsefforts across units and levelsDeveloping a culture of evidenceDeveloping a culture of evidenceEmphasize link between MACRO-LENS and MICRO-Emphasize link between MACRO-LENS and MICRO-LENS of assessmentLENS of assessmentHelping faculty see link between institutional Helping faculty see link between institutional effectiveness, aggregate data gathering, and outcomes–effectiveness, aggregate data gathering, and outcomes–based assessmentbased assessment

Community College Assessment Community College Assessment ChallengesChallenges

Questions pertaining to the creation and Questions pertaining to the creation and improvement of an “Effective Academic improvement of an “Effective Academic Learning Environment” cannot be Learning Environment” cannot be answered byanswered by– Admissions/enrollment dataAdmissions/enrollment data– Persistence/attrition dataPersistence/attrition data– Survey dataSurvey data– Demographic comparisonsDemographic comparisons

Community College Assessment Community College Assessment ChallengesChallenges

– Global teaching evaluationsGlobal teaching evaluations– Department comparisonsDepartment comparisons– Teaching loads and credit hoursTeaching loads and credit hours– Entering first year student statisticsEntering first year student statistics– Transfer ratesTransfer rates

How Can We Support Faculty Work and How Can We Support Faculty Work and Impact Academic and Student Development?Impact Academic and Student Development?

Lead the discussion that illuminates critical Lead the discussion that illuminates critical questions/concerns/issuesquestions/concerns/issuesProvide initial sources of data and documentation directly Provide initial sources of data and documentation directly to departments and programsto departments and programsKeep the focus on measurable objectives and outcomesKeep the focus on measurable objectives and outcomesNeutralize political concernsNeutralize political concernsDo all necessary programmingDo all necessary programmingFacilitate the development and implementation of an Facilitate the development and implementation of an assessment web siteassessment web site““Champion” the development of assessment plansChampion” the development of assessment plans

Partner in assessment with your other providers of the Partner in assessment with your other providers of the serviceservice