OKAIRP Fall Conference

NSU @ Broken Arrow

November 4, 2005

Designing Methods for Determining Educational Gain: Involving Faculty in Assessment

Mark L. Giese

Northeastern State University

Tahlequah, Oklahoma

Measuring Program Dispositions as a Model

Knowledge

Skills

Dispositions

What are dispositions?

What is the big deal about dispositions?

Dispositions are:Concepts we inherently value as a

profession.Esoteric to the discipline.

What are dispositions in law?

Ethical

Passion for service

Accessible

What are dispositions in Education?

In Health and Kinesiology, dispositions may include:

Hard workTeamworkOur students or clients can improve with

practice

Who is involved?

Use all persons with a stake:

Faculty

Administrators

Let faculty make mistakes

Why measure?

Looks at the Affective Learning Domain.

In the past, typical areas of measurement have only included knowledge and skills.

Accrediting agencies (learned societies) require measurement in this domain.

Steps in developing a measurement tool for your unit

PlanningReview the literature and ask your friends

what they are doing.Decide on which dispositions to measure

(committee).Decide if you intend to use this exercise as

an exit competency or if you are looking for a change from entry to exit.

Determine an a priori standard.

Steps in developing a measurement tool for your unit

ImplementationDevise a measurement tool (rubric)Collect your dataAnalyze your dataDetermine if an intervention is necessaryREMEASURE!

Examples

Example of a college rubric for measuring dispositions in pre-service teacher education candidates in the COE at NSU.

Examples

Example of a Sociology department rubric for measuring dispositions:

Attempt to measure “Social Distance”

Importance

Measuring dispositions is important because it gives us another piece of information to determine our effectiveness and quality.

Examples of Presentations on Social Distance

“Social Distance as a Disposition of Teacher Candidates.” Presented at AACTE, Chicago, Illinois, February 8, 2004.

Examples of Presentations on Social Distance

“Attitudes Towards Social Distance of Pre-Service Teacher Education Students.” Presented at Higher Education Research Day, Edmond, Oklahoma, November 2003.

Examples of Presentations on Social Distance

“Attitudes Towards Social Distance of Pre-Service Teacher Education Students.” Presented at the Oklahoma Association of Colleges for Teacher Education Conference, Norman, Oklahoma, November 9, 2000.

Two Types of Measurement

One-time (barrier)

Demonstrate program effectiveness

(gain)

Measurement Goals

One Time: Graduation requirement etc.

Using a pre/post test design in an attempt to determine program gain. May be done at entry and exit of any major.

Does your data fit the model?

Parametric (t, ANOVA)

Non-parametric

Determining Program Gain

Measure at pretest.Measure at post test.Determine the difference.Use hypothesis testing to determine if a

significant difference exists.

How ANCOVA works

The variable under study is the dependent variable and the covariate is the “other” data point. Both must be known for each subject (matched).

How ANCOVA works

The influence of the covariate is removed using a straight-forward linear regression method.

The ANCOVA produces an adjusted post test score for each subject and an adjusted mean as well.

ANCOVA Example

Three Methods of teaching French:Pretest……..instruction……Post testMeasures of intelligence availableRemove the effect of intelligence so the

French method may be the cause of the change from pre/post score.

ANCOVA

Group 1Pre

Post

. .

. .

. .

. .

. .

. .

. .

. .

X X

Group 2Pre

Post

. .

. .

. .

. .

. .

. .

. .

. .

X X

Group 3Pre

Post

. .

. .

. .

. .

. .

. .

. .

. .

X X

Assumptions of ANOVA

Homogenity of variance

Equal distribution

Random sampling

Additive effects

Extra Assumptions of ANCOVA

IV does not influence the covariate variable

Equal regression slopes

Linearity

r of .60 (Shavelson, 1996)

Review

Ask faculty if they need/desire to measure dispositions.

Assist with choosing an instrument and measurement model.

Assist in determining gain and the magnitude of the effect.

Questions

??QUESTIONS??