The Missing Link:

Development of

Programmatic Outcomes

Christopher Meseke,Ph.D.Park University

What is Assessment? “As a whole, assessment is a framework for

focusing faculty attention on student learning and for provoking meaningful discussions of program objectives, curricular organization, pedagogy, and student development” (Allen, 2004).

Assessment is the quality control of the educational process

Assessment is -- first and foremost --

about student learning.

Assessment Levels Student Classroom Course Program College Division Institution

Common Reactions to Assessment Initiatives

• Ignoring it• Bribing someone else to do it• Complaining about it • Losing sleep over it• Sitting down and writing it

Levels of assessment: quality control and assurance Institutional/Programmatic National/State Licensure Exams Certain academic programs (Nursing,

Engineering, Physical Therapy, Social Work, etc) Accreditation Council for Graduate Medical

Education (ACGME) Higher Learning Commission of the North

Central Association of Colleges and Schools

Assessment is important to the accreditation process The Higher Learning Commission of the

North Central Association of Colleges and Schools (HLC-NCA) requires it.

Assessment of student learning provides evidence at multiple levels: course, program, and institutional.

Assessment of student learning includes multiple direct and indirect measures of student learning.

Assessment results inform improvements in curriculum, pedagogy, instructional resources, and student services.

Interpretation of the HLC-NCA Statement on Assessment on Student Learning, 2003. It is the faculty members who must

take ownership of the assessment process. Their buy-in, ownership, and implementation is directly related to both the mission of the institution and attitudes, knowledge, and skills required for the student to successfully complete the program requirements.

Goals of Program Assessment Planning

Measure student learning, not teaching, curricular content, processes or resources

Measure things that are important to us Involve faculty in development of the

assessment program, processes and instruments

Use multiple measures to produce valid and reliable data (triangulation)

Make it manageable and cost affordable

Goals of Program Assessment Planning

Map assessment data back to the curriculum for improvement

Produce annual reports of assessment outcomes that show the data, interpretation of data, and improvement plans where indicated

Same assessment procedures for all campuses and sister programs

Modified Hatfield Assessment Model

Meta-Competencies/Institutional Goals

objective

objective

objective

Competency

Learning Outcome

objective

CompetencyCompetency

indicators

Degree to which Competency is achieved

Learning events

Programmatic Competencies A measurable, complex behavioral

statement may be written for each Key Component (meta-competency) and competency.

Statements reflect what we think we can actually measure in an educational environment considering time and resources.

Program Assessment Learning Competencies Develop a set of Learning Goals (Meta-

competencies) which represent the attributes of the graduate

General Education (Competencies) Discipline (Competencies)

Course (Outcomes)

KNOWLEDGECOMPREHENSION

APPLICATIONANALYSISSYNTHESISEVALUATION

CiteCountDefineDraw

IdentifyList

NamePointQuoteRead

ReciteRecordRepeatSelectState

TabulateTell

TraceUnderline

AssociateClassifyCompareComputeContrast

DifferentiateDiscuss

DistinguishEstimateExplainExpress

ExtrapolateInterpolate

LocatePredictReportRestateReview

TellTranslate

ApplyCalculateClassify

DemonstrateDetermineDramatize

EmployExamineIllustrateInterpretLocate

OperateOrder

PracticeReport

RestructureScheduleSketchSolve

TranslateUse

Write

AnalyzeAppraiseCalculate

CategorizeClassifyCompareDebate

DiagramDifferentiateDistinguish

ExamineExperiment

IdentifyInspect

InventoryQuestionSeparate

SummarizeTest

ArrangeAssemble

CollectComposeConstruct

CreateDesign

FormulateIntegrateManageOrganize

PlanPrepare

PrescribeProduceProposeSpecify

SynthesizeWrite

AppraiseAssessChoose

CompareCriticize

DetermineEstimateEvaluate

GradeJudge

MeasureRankRate

RecommendReviseScoreSelect

StandardizeTest

Validate

Lower Order Bloom’s verbs

KNOWLEDGECOMPREHENSION

APPLICATIONANALYSISSYNTHESISEVALUATION

CiteCountDefineDraw

IdentifyList

NamePointQuoteRead

ReciteRecordRepeatSelectState

TabulateTell

TraceUnderline

AssociateClassifyCompareComputeContrast

DifferentiateDiscuss

DistinguishEstimateExplainExpress

ExtrapolateInterpolate

LocatePredictReportRestateReview

TellTranslate

ApplyCalculateClassify

DemonstrateDetermineDramatize

EmployExamineIllustrateInterpretLocate

OperateOrder

PracticeReport

RestructureScheduleSketchSolve

TranslateUse

Write

AnalyzeAppraiseCalculate

CategorizeClassifyCompareDebate

DiagramDifferentiateDistinguish

ExamineExperiment

IdentifyInspect

InventoryQuestionSeparate

SummarizeTest

ArrangeAssemble

CollectComposeConstruct

CreateDesign

FormulateIntegrateManageOrganize

PlanPrepare

PrescribeProduceProposeSpecify

SynthesizeWrite

AppraiseAssessChoose

CompareCriticize

DetermineEstimateEvaluate

GradeJudge

MeasureRankRate

RecommendReviseScoreSelect

StandardizeTest

Validate

Higher Order Bloom’s verbs/Upper division

Course / Program outcomes

An Example of Departmental/Programmatic Competencies Students will be able to:

Demonstrate biological knowledge appropriate for the course level.

Demonstrate a working knowledge of the scientific method.

Demonstrate the ability to communicate scientific concepts and findings, both in oral and written format.

Apply interdisciplinary knowledge to the biological sciences.

Demonstrate an awareness of ethical issues in the life sciences.

Mapping Competencies to Courses

Program Competencies

Course1

x

x

x

Course2

x

x

x

x

Course3

x

x

x

Course4

x

x

x

x

x

x

x

x

Course5

Mapping Outcomes to Courses

Program Competencies

Course1

x

x

x

Course2

x

x

x

x

Course3

Course4

x

x

x

x

x

x

x

x

Mapping Outcomes to CoursesCourse

5

Triangulation of Assessment Strategies: Direct Measures (Pick 2)

Licensure Scores GRE/GMAT/MCAT/other Departmental Exit Exams Portfolio Core Assessment Capstone Experience Others

Triangulation of Assessment Strategies: Indirect Measures (Pick 1)

Advisory Boards Senior Survey Employer/Professional

School Survey Focus Groups/Interviews Other

Three-Year Assessment Plan All departments/programs should be

assessed on three-year cycles Not all department/program competencies

should/can be assessed Focus on 2-3 competencies at most

Establish meaningful criteria At end of three-year cycle examine data for

meaningful trends Only after the three-year cycle are changes

made, only if needed

An example of a three-year assessment cycle

Three-Year Schedule of Program Assessment Plans Submissions to the Assessment Committee    2013-2014 2014-2015 2015-2016

Program G/UG      Accounting UG X    Adult Education G  X  Athletic Training UG    XBiology UG  X  Business Administration G   XBusiness Core (CPC) UG    XChemistry UG X     Communication Arts UG    X  Communication and Leadership G     X Computer Information Systems UG  X    Criminal Justice UG    X  Economics UG      X

Early Childhood Education (Educational Studies) G/UG  X    Educational Leadership G   X  Elementary Education UG      XEngineering UG  X    English UG   X   Finance UG     X Fine Arts UG  X    Geography UG   X   Graphic Design UG     X Healthcare Leadership G X     Healthcare Management UG   X   History UG     X Human Resource Management UG X     Information and Computer Science UG   X   Interior Design UG     X International Business UG  X    International Center for Music G/UG   X   Language and Literacy G     X Legal Studies UG  X    Liberal Education UG   X   Liberal Studies UG     X Logistics UG X    Management UG    X  Management Information Systems UG     XMarketing UG  X    Mathematics UG   X   Middle and Secondary Education UG     X Modern Languages UG  X    Nursing UG   X   Political Science UG     X Psychology UG  X    Public Administration (BPA) UG   X   Public Affairs (MPA) G     X Social Psychology UG X    Social Work UG    X  Sociology UG     X Teacher Leadership G  X    

Summary Choose an Assessment Model Identify (develop) programmatic competencies Write program level competencies based upon

the Institutional meta-competencies. Establish key performance indications

(KPOs)based upon the competencies Identify 3 measures for each competency (2

direct, 1 indirect) and set success criteria (may be different across various university units)

Develop data gathering and reporting mechanisms and templates

Develop close-the-loop mechanisms

Big Mistakes in Assessment• Assuming that it will go away

• Trying to do too much, too soon

• Expecting to get it right the first time

• Not considering implementation issues when creating plans

• Borrowing plans and methods without acculturation

Big Mistakes in Assessment

• Demanding statistical research standards

• Doing it for accreditation instead of improvement

• Confusing assessment with student learning

• Making assessment the responsibility of one individual

• Assuming collecting data is doing assessment

Thank you

A special thanks to Dr. Susan Hatfield, for her gracious input.

Questions?