The measurement model: what does it mean and what you can do with it? Presented by Michael Nering, Ph. D.

  • Published on
    14-Dec-2015

  • View
    212

  • Download
    0

Embed Size (px)

Transcript

<ul><li>Slide 1</li></ul> <p> Slide 2 The measurement model: what does it mean and what you can do with it? Presented by Michael Nering, Ph. D. Slide 3 Goals of this session Present commonly used measurement models IRT Show how these models form the backbone of any large scale assessment program Equating Scaling To discuss the meaning of the measurement models Ability estimation Item characteristics Slide 4 Now, really . This session is an introduction to the world of psychometrics My goal is for you to understand that psychometrics: Is not a black box Is really just a set of procedures Slide 5 A little about me Yes, Im a psychometrican B.A. in psychology at Kent State Ph. D. in psychology at Univ of Minn Working at Measured Progress since 1999 Research areas of interest: IRT, equating, scaling, person fit, adaptive testing Slide 6 Slide 7 Why Psychology? Psychometricians typically come from: Educational measurement programs Psychometric programs I/O programs Ultimately, we are all after the pursuit of understanding people by way of quantification Slide 8 Psychometrics Defined Psychological Measurement Psycho metrics The business of measuring psychological things Slide 9 What are psychological things? Any latent trait Any characteristic that is not directly observable Examples: Depression, bi-polar, personality disorder Math, reading, writing, science abilities We dont care lets use Slide 10 Counterparts to Psychometrics Econometrics Measurement of economic things Sociometrics Measurement of social things Slide 11 All metrics are ultimately a blend of things Psycho- metrics PsychStatsMath Slide 12 Quantification in Psychology Deep roots that came originally from philosophy Philosophy in the 1500s branched into several disciplines because of the need to quantify certain things to better understand human beings Slide 13 Philosophys Many Branches This desire to better understand humans lead to two primary areas of study Physiology 1543 Belgian physiologists practices the dissection of cadavers Psychology 1524 Marco Marulik publishes The Psychology of Human Thought Slide 14 Yes, I did just use the word cadaver but trust me its okay Slide 15 The last 100 years of psychometrics Classical test theory and Spearmans 1904 contribution True score theory Reliability theory, p-values, point biserial coefficients Item response theory Slide 16 Lets talk about IRT When I say measurement model I really do mean some sort of IRT model Lots of historical developments Lord &amp; Novick text of 1968 Many advantages over CTT Slide 17 So, what is IRT? A family of mathematical models that describe the interaction between examinees and test items Examinee performance can be predicted in terms of the underlying trait Provides a means for estimating scores for people and characteristics of items Common framework for describing people and items Slide 18 The ogive Natural occurring form that describes something about people Used throughout science, engineering, and the social sciences Also, used in architecture, carpentry, engineering, photograph, art, and so forth Slide 19 The ogive Slide 20 Slide 21 A little jargon The item characteristic curve (ICC) Also called: Item response function Trace line Etc. Stochastic: 1) involving a random variable, or 2) involving chance or probability Slide 22 The ICC Does this one little function really do everything? Scale items &amp; people onto a common metric? Help in standard setting? Foundation of equating? Some meaning in terms of student ability? Slide 23 Does this one little function really do everything? Lets talk more about the ICC Slide 24 The ICC Any line in a Cartesian system can be defined by a formula The simplest formula for the ogive is the logistic function: Slide 25 The ICC Where is the item parameter, and is the person parameter The function represents the probability of responding correctly to item i given the ability of person j. Slide 26 is the inflection point Item i i =0.125 Slide 27 We can now use the item parameter to calculate p Lets assume we have a student with =1.0, and we have our = 0.125 Then we can simply plug in the numbers into our formula Slide 28 Using the item parameters to calculate p p = 0.705 i =1.00 Slide 29 Wait a minute What do you mean a student with an ability of 1.0?? Does an ability of 0.0 mean that a student has NO ability? What if my student has a reading ability estimate of -1.2? What in the world does that mean???? Slide 30 The ability scale Ability is on an arbitrary scale that just so happens to be centered around 0.0 We use arbitrary all the time: Fahrenheit Celsius Decibels DJIA Slide 31 Scaled Scores Although ability estimates are centered around zero reported scores are not However, scaled scores are typically a linear transformation of ability estimates Example of a linear transformation: (Ability x Slope) + Intercept Slide 32 The need for scaled scores the kids will have negative ability estimates Slide 33 Scaled Scores Slide 34 Use of scaled scores Student/parent level report School/district report Cross year comparisons Performance level categorization Slide 35 Theres a lot here Scaled scores are surface level information Behind the scenes: we use fancy formulas to depict interaction between students and test items theres a probabilistic relationship between students and test items Slide 36 Unfortunately, life can get a lot worse Items vary from one another in a variety of ways: Difficulty Discrimination Guessing Item type (MC vs. CR) Slide 37 Items can vary in terms of difficulty Ability of a student Easier item Harder item Slide 38 Items can vary in terms of discrimination Discrimination is reflected by the pitch in the ICC Thus, we allow the ICCs to vary in terms of their slope Slide 39 Good item discrimination 2 close ability levels Noticeable difference in p Slide 40 Poor item discrimination smaller difference Same 2 ability levels Slide 41 Guessing This item is asymptotically approaching 0.25 Slide 42 Polytomous Items Slide 43 Im sure by now you might be having a couple of thoughts How can I get up, open the door, and walk out without anybody noticing? Im stuck in a psychometric prison help me! Slide 44 But, trust me Im really trying to make a simple point Slide 45 Items and people Interact in a variety of ways We can use IRT to show that there exists a nice little s-shaped curve that shows this interaction As ability increases the probability of a correct response increases Slide 46 Advantages of IRT Because of the stochastic nature of IRT there are many statistical principles we can take advantage of A test is a sum of its parts Slide 47 The test characteristic curve A test is made up of many items The TCC can be used to summarize across all of our items The TCC is simply the summation of ICCs along our ability continuum For any ability level we can use the TCC to estimate the overall test score for an examinee Slide 48 A bunch of ICCs are on a test Slide 49 The test characteristic curve Slide 50 From an observed test score (i.e., a students total test score) we can estimate ability The TCC is used in standard setting to establish performance levels The TCC can also be used to equate tests from one year to the next The test characteristic curve Slide 51 Estimating Ability Total score = 3 Ability0.175 Slide 52 Standard Setting Advanced Prof. Basic Below Failing Basic Slide 53 Equating Year 1 TCC Slide 54 Equating -3-2 0 1 2 3 Year 2 TCC &amp; Scale Slide 55 Equating Our 2 nd scale goes away and our TCC are closer together Slide 56 Equating Remaining differences due to non-common items Slide 57 Equating The adjustment to the TCC can be done a variety of different ways Lets take a look a one commonly used method of equating, namely the Mean Shift method Slide 58 Mean shift method of equating Slide 59 These items are common between the two years Slide 60 Mean shift method of equating Slide 61 Mean shift method of equating Slide 62 Mean shift method of equating The difference between 1 and 2 is our scaling constant This is used to make an adjustment to all the items administered in Year 2, so that they are then on the same scale as Year 1 Slide 63 Example = 0.20 2 = -0.10 We need to add 0.30 ( 1- 2:.2+.1=.3) in order for the equating items to have the same mean This 0.30 difference is due to an arbitrary scaling difference and NOT due to any differences in ability Slide 64 Mean shift method of equating 0.30 is then added to all the item difficulty values Slide 65 Mean shift method of equating By shifting all our item difficulties to last years scale we are ultimately putting this years TCC onto last years scale Slide 66 Equating The example we just saw was merely one example of an equating methods There are several methods (Kolen &amp; Brennan) that are available Slide 67 What have we learned? IRT: used to model interaction between items and people Item characteristics: item vary in terms of difficulty, discrimination, guessing, etc. Equating: used to relate test from one year to the next Scaling: used to represent student ability Slide 68 The Assessment Cycle Administration ICCs &amp; TCCs Equating Ability estimates &amp; scaling Reporting Slide 69 So, how is this all done? Slide 70 Psychometricians often play the role of the magical wizard of assessments Slide 71 But, really This session has served as your training in psychometric methods For career opportunities please send along a copy of your vita to: Measured Progress Attn: psychometric Department 171 Watson Road Dover, Nh 03820 </p>