Upload
hiero
View
53
Download
0
Embed Size (px)
DESCRIPTION
Larry D. Gruppen, Ph.D. University of Michigan. From Concepts to Data:. Conceptualization,. Operationalization, and. Measurement. in Educational Research. Objectives. Identify key research design issues Wrestle with the complexities of educational measurement - PowerPoint PPT Presentation
Citation preview
Larry D. Gruppen, Ph.D.University of Michigan
From Concepts to Data:
Conceptualization, Operationalization, and
in Educational Research
Measurement
Objectives
• Identify key research design issues
• Wrestle with the complexities of educational measurement
• Explain the concepts of reliability and validity in educational measurement
• Apply criteria for measurement quality when conducting educational research
Agenda
• A brief nod to design• From theory to measurement• Criteria for measurement quality
– Reliability– Validity
• Application: analyze an article
Guiding Principles forScientific Research in Education
1. Question: pose significant question that can be investigated empirically
2. Theory: link research to relevant theory
3. Methods: use methods that permit direct investigation of the question
4. Reasoning: provide coherent, explicit chain of reasoning
5. Replicate and generalize across studies
6. Disclose research to encourage professional scrutiny and critique
Study design
• Study design consists of:– Your measurement method(s)– The participants and how they are assigned– The intervention– The sequence and timing of measurements
and interventions
Comparison Group
• Pre-post design - compare intervention group to itself
• Non-equivalent control group design - compare intervention group to an existing group
• Randomized control group design - compare to equivalent controls
Overview of Study Designs
• Symbols– Each line represents a group.– x = Intervention (e.g. treatment)
– O1, O2, O3…= Observation (measurement) at Time 1, Time 2, Time 3, etc.
– R = Random assignment
Non-Experimental Designs
x O1
One-Group Posttest
x O1
Quasi-Experimental Designs
x O1
O1
Posttest-Only Control Group
O1 x O2
One-Group Pretest-Posttest
O1 x O2
O1 O2
Control Group Pretest-Posttest
Experimental Designs
Posttest Only Randomized Control Group
R x O1
R O1
R O1 x O2
R O1 O2
Randomized Control Group Pretest-Posttest
Theory
Constructs
Operational Definition
Measurement
From Theory to Measurement
Measurement
• Measurement: assignment of numbers to objects or events according to rules
• Quality: reliability and validity
The Challenge of Educational Measurement
• Almost all of the constructs we are interested in are buried inside the individual
• Measurement depends on transforming these internal states, events, capabilities, etc. into something observable
• Making them observable may alter the thing we are measuring
Examples of Measurement Methods
• Tests (knowledge, performance): defined response, constructed response, simulations
• Questionnaires (attitudes, beliefs, preferences): rating scales, checklists, open-ended responses
• Observations (performance, skills): tasks (varying degrees of authenticity), problems, real-world behaviors, records (documents)
Reliability
• Dependability (consistency or stability) of measurement
• A necessary condition for validity
Types of Reliability
• Stability (produces the same results with repeated measurements over time):– Test-retest – Correlation between scores at 2 times
• Equivalence/Internal Consistency (produces same results with parallel items on alternate forms):– Alternate forms; split-half; Kuder-Richardson; Chronbach’s alpha – Correlation between scores on different forms; Calculate
coefficient alpha (a)• Consistency (produces the same results with different observers or
raters):– Inter-rater agreement – Correlation between scores from different raters; kappa
coefficient
Validity
• Refers to the accuracy of inferences based on data obtained from measurement
• Technically, measures aren’t valid, inferences are
• No such thing as validity in the abstract: the key issue is ‘valid’ for what inference
• Want to reduce systematic, non-random error• Unreliability lowers correlations, reducing validity
claims
Conventional View of Validity
• Face validity: logical link between items and purpose—makes sense on the surface
• Content validity: items cover the range of meaning included in the construct or domain. Expert judgment
• Criterion validity: relationship between performance on one measurement and performance on another (or actual behavior) Concurrent and Predictive Correlation coefficients
• Construct validity: directly connect measurement with theory. Allows interpretation of empirical evidence in terms of theoretical relationships. Based on weight of evidence. Convergent and discriminant evidence. Multitrait-MultiMethod Analysis (MTMM)
Unified View of Construct Validity(Messick S, Amer Psych, 1995)
• Validity is not a property of an instrument but rather of the meaning of the scores. Must be considered holistically.
• 6 Aspects of Construct Validity Evidence– Content—content relevance & representativeness– Substantive—theoretical rationale for observed consistencies in
test responses– Structural—fidelity of scoring structure to structure of construct
domain– Generalizability—generalization to the population and across
populations– External—convergent and discriminant evidence– Consequential—intended and unintended consequences of score
interpretation; social consequence of assessment (fairness, justice)
Finding Measurement Instruments
• Scan the engineering education literature (obviously)• Email engineering ed researchers (use the network)• Examine literature for instruments used in prior studies• General education/social science instrument databases
– Buros Institute of Mental Measurements (Mental Measurement Yearbook, Tests in Print) http://buros.unl.edu/buros/jsp/search.jsp
– ERIC databases http://www.eric.ed.gov/– Educational Testing Service Test Collection http://www.
ets.org/testcoll/index.html• Construct your own (last resort!)
– Get some expert consultation (test writing, survey design, questionnaire construction, etc.)
Example
• In your groups, analyze the Steif & Dantzler statics concept inventory article. Look for:– Theoretical framework– Constructs used in the study– How constructs were operationalized– Measurement process
• Attention to reliability and validity
References
• Campbell DT, Stanley JC. Experimental and quasi-experimental designs for research. Chicago: Rand McNally; 1969.
• Cook, T.D. and Campbell, D.T. (1979). Quasi-Experimentation: Design and Analysis for Field Settings. Rand McNally, Chicago, Illinois.
• Messick S. Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist. 1995;50:741-749.
• Messick S. Validity. In: Linn RL, ed. Educational measurement. 3rd ed. New York: American Council on Education & Macmillan; 1989:13-103.