BPS Test User Conference Keynote paper - · PDF fileapproach that we require major ... our approach to measurement and assessment of individuals is unlikely to occur without an

Slide 1

Paul Barrettemail: [email protected]

http://www.liv.ac.uk/~pbarrett/paulhome.htm

Affiliations: The State Hospital, CarstairsDept. of Clinical Psychology, Univ. Of Liverpool

5th June, 1998


http://www.liv.ac.uk/~pbarrett/paulhome.htm

Affiliations: The State Hospital, CarstairsDept. of Clinical Psychology, Univ. Of Liverpool

5th June, 1998

BPS Test User Conference 1998

The purpose of this paper is primarily to :• Introduce many “occupational” test users and trainers to the kind of thinking and

philosophical considerations now having to be examined in the area of individualdifferences psychology by theoreticians and researchers like myself. Why? Because asGoldberg, Kline, Eysenck, Blinkhorn, and myself have indicated, the field is stagnatingboth in terms of theory and investigation. As these others others have variously indicated- we are suffering from a surfeit of advanced methodologies and patchwork “fixes” topoor theory - but little clear evidence of a systematic rationale in much of what isproduced - other than for “applied” utilitarian purposes (which is not a problem at all,except when you wish to “do better”).

• Inform the test user about issues, such as what constitutes “scientific investigation” vs theapplication of quantitative methodologies. Although perhaps a surprise to many,psychology is barely “the scientific study of behaviour”, and most psychometric variablesdo not satisfy the first pre-requisite for a quantitative science - empirical evidence that avariable possesses a quantitative structure.

• Let the test user see the kind of research being undertaken that might impact on “test-user” practice in several years’ time.

This paper does not set out to “rubbish” psychology, psychometrics, or occupational testuse. Rather, it is concerned with answering a blunt question ...“having got this far - nowwhat”. This requires a careful, considered, evaluation of the current methods, theory, andimplicit philosophies being used by all in the area, then, drawing upon knowledge andthinking from other domains, I have set out the problems as I see them, and my proposedframework for some solutions (both theory and empirical research).

Slide 2

What do you think is missing from thecurrent generation of tests that might

explain why they seem to predict behaviourso poorly?

-is it--is it-More of the same is what we need?

-or--or-Is there something so flawed with our whole

approach that we require majorphilosophical surgery?

The QuestionsThe QuestionsThe Questions

Paul Barrett: BPS Test User Conference 1998 2

Well, these are the questions as I see them. Every member of this audience can probably thinkof a few specific reasons as to why our measures are not better at predicting behaviour. I hopein the next slides to have summarised these issues for you.

I have been much engrossed over the past year, considering whether improved more of “thesame” is what is required to achieve better results (whether for selection, training, or career-development), or whether the foundational methods, theory, and tests emanating from thelikes of Spearman, Thurstone, Guildford, Cattell, and Eysenck, have in fact yieldedeverything that could ever be asked of them, but are near to running out of “theoretical andempirical steam”.

The reference to “major philosophical surgery” is there to stress that a fundamental change inour approach to measurement and assessment of individuals is unlikely to occur without anequally fundamental change in our philosophy that underlies this practice. This philosophy isvery much concerned with how we view the “scientific or otherwise” status of psychologicalmeasurement and assessment. But here, science has a very specific meaning, as does itsmeasurement. In addition, the tendency toward “eliminative materialism” also needsconsideration. This is the hypothesis that all mental events are identical to certain brain eventsand that all explanations currently given in psychological terms will eventually be replaced byexplanations given in neurochemical and neurophysiological terms.

Slide 3


Within which information domains might we seek the answer?

Within which information Within which information domains might we seek the answer?domains might we seek the answer?

u The Person (P: psychological attributes)u The Situation (S: environment attributes)u The Behaviour (outcomes of P x S)

Do we seek some or all of our answers in the more detailed examination of person attributes,the situations, or the behavioural outcomes?

Much of current psychometrics has focussed entirely upon the psychological attributes of anindividual. However, it was Cattell who always stressed the importance of the situation,including situational parameters as modifiers within his dynamic calculus. It is also to benoted that one of the research projects in the new SHL/UMIST Research Centre in Workand Organisational Psychology (http://www.umist.ac.uk/~webshl/) is concerned withinvestigating the development of a taxonomy of situations (Prof. Ivan Robertson and MilitzaCallinan).

Behavioural (outcome) classification is already an implicit part of many personalityquestionnaires (in that many tests asks individuals about the situations they prefer, or thebehaviours in which they prefer to engage). As with many “competency” questionnaires, thecompetencies are in fact behavioural outcomes classified as “competencies”. They do notmeasure dispositions, but observed “outcomes” of whatever cognitive processes are at work inan individual in order that the behaviour can be generated, then classified.

Slide 4


Which Methodologies (and their underlying Philosophies)

must we consider?

Which Methodologies (and Which Methodologies (and their underlying Philosophies) their underlying Philosophies)

must we consider?must we consider?

u Quantitative - psychometricsu Qualitative - judgementsu Non-linear Neural Nets/chaos dynamics

Here we contrast the standard quantitative, generally statistical methodologies, vs a qualitative(essentially subjective judgement approach) vs the as yet rarely used non-linear dynamicalsystems methodologies as found within neural net applications and decision processes.

I would like at this point to introduce an important distinction between measurement andassessment. This distinction arose in a recent conversation with Sean Hammond([email protected]) in connection with measurement issues. Sean has carefullydistinguished between the measurement process (which is concerned with measuringmagnitudes on psychometrically defined variables) and the assessment process (which isconcerned with formulating a psychological description or profile of an individual, usingvarious sources of information, including psychometric test scores). I (and he) contend thatthese are two quite different problems, which although related, require a fundamentallydifferent approach and perspective for each. As will be seen below, I conclude that thisconfusion in purpose has perhaps afflicted both the BPS Level B standards committee, as wellas many test publishers involved in training test users.

I discard the “qualitative approach” from hereon as it is largely irrelevant to the issues at hand(except when considered as the preferred methodology within a post-modern philosophicalframework)

Slide 5


Key features/problems of Psychometrics .1Key features/problems of Psychometrics .1Key features/problems of Psychometrics .1

u Objective MeasurementÊThe scoring procedures are objectiveÊ Actuarial use of the scores is objectiveÊ Interpretation of scores is subjective

u Descriptive Taxonomies of Behaviour

One of the “knee-jerk” responses to psychometric measurement is the assertion that it isobjective. Well, one has to consider very closely where the objectivity is demonstrable. Thisis largely confined to the scoring and actuarial use of the test scores. The “test-scoreinterpretation skills” that are so much part of Level B training are entirely subjective.That is, unless empirical evidence exists to support each and every statement made as part ofthe interpretation of a set of test scores, then such interpretation is subjective. This“subjectivity” can be bolstered by statistical “inter-rater reliability” studies - but presentsalmost insurmountable logistical and conceptual problems.

This does not mean that the interpretation is wrong, or that such training is undesirable.Rather, it suggests that test publishers must make clear that the objectivity of measurement isconfined to scoring and actuarial profiling. Test interpretation is then “trained” according tothe test publisher’s preference - which in some cases may be entirely subjective. However, thedifficulty for test users and publishers alike is that psychometric test scores are relativelylimited items of information about any individual. Unless one decides upon a rigid actuarialapproach, then all that is left is some form of reasonable subjective interpretation andinferential process - akin to clinical judgement and formulation.

However, in the light of Ziskin at al, one must question test users’ capacity to interpretativelymanipulate so many variables at once - mentally. Surprisingly, this result suggests that acomputer-based narrative report should be of more help to an interpreter than just test scores.

Slide 6


u Reliance on Self-ReportÊ Assumes equal insight/introspectionÊ Applicant vs non-applicant systematic biasÊ Dispositions vs preferencesÊ The “Repressor” problem and Anxiety

u Quantitative Methodologyu Limited Explanatory Power

Key features/problems of Psychometrics .2Key features/problems of Psychometrics .2Key features/problems of Psychometrics .2

This reliance on “self-report” is probably the area that most test users find problematic(in that systematic distortion is a known feature of applicant vs non-applicant respondents onpersonality and interest questionnaires).

Further, whether someone expressing preferences for certain behaviours or indicatingtheir attitude toward those behaviours permits a conclusion that this preference or attitudinalposture is indicative of a psychological “disposition” or “trait”, is questionable. Jackson andMaraun (1996a, b) have argued convincingly that it is most unlikely, given some typicalquestionnaire items.

The “repressor” problem was first identified by Weinberger (1979). Those scoring“low” on trait anxiety can be divided into those who are truly “low anxious” (as defined byphysiological indices in situations designed to evoke arousal indications), and those whoscore as “low anxiety”, but in fact show physiological arousal equal to or greater than “hightrait anxiety “ individuals. In psychometric terms, repressors can be identified as low-scoring anxiety with high scoring social desirability. The evidence is rather compelling andreplicable.

Jackson, J.S.H., Maraun, M. (1996a) The conceptual validity of empirical scale construction:the case of the Sensation Seeking Scale. Personality and Individual Differences, 21, 1, 103-110Jackson, J.S.H., Maraun, M. (1996b) Whereof one cannot speak, thereof one must remainsilent. Personality and Individual Differences, 21, 1, 115-118Zuckerman, M. (1996) "Conceptual Clarification" or confusion in "The study of SensationSeeking" by Jackson and Maraun. Personality and Individual Differences, 21, 1, 111-114

Slide 7


Conventional Solutions .1Conventional Solutions .1Conventional Solutions .1

u Augment the predictive/explanatorylimitation with “clinical” interpretation of testscores and “expert” judgements.u Reduce subjectivity by actuarial profilingu Adjust self-reports for bias, acquiescence,and distortionu Measure situations - develop taxonomies

The first point is concerned with a temptation to augment the inherent limitation of apsychometric test score with “clinical interpretation”. That is, to begin a process of candidate“formulation”, complete with “expert judgements” and calls to “experience”. This is a naturalresponse to the fact that a test score conveys little of actual meaning, other than that providedby the responses to the items themselves. The problem though with extrapolating judgementsfrom item responses and test scores is the subjectivity implicit in such a process, as well as theevidence against this practice discussed by Ziskin et al concerning the flaws in clinicaljudgement.

The use of actuarial profiling may well work - assuming we move beyond merequestionnaire score profiling and improve our profiling and templating methodologies. This isalready the focus of some of my own research.Hofstee,W., Ten Berge, J., and Hendriks, A.A. (in press) How to Score Questionnaires.Personality and Individual Differences. ([email protected]) have recentlyexamined the response bias/acquiescence problem in detail, using a novel procedure toeliminate a general “social desirability” response factor from questionnaire data, theniteratively scoring the data using continuous principal component weights for each item, withiterations ceasing when no more items are rejected (through low weights).

With regard to “situations”, the paper by Mischel and Shoda (1995) is of interest, but,I’m not convinced that situations can be classified beyond mere generalities.

Mischel, W. & Shoda, Y. (1995) A cognitive-affective system theory of personality:Reconceptualizing Situations, dispositions, dynamics, and invariance in personalitystructure.Psychological Review, 102(2) 246-268

Slide 8


Conventional Solutions .2Conventional Solutions .2Conventional Solutions .2

u Create more measures of psychologicalattributesu Develop more sophisticated taxonomicmodels of psychometrically measuredvariablesu Develop more sophisticated test theoriesu Begin to measure “performance” and makeobservations of actual behaviours

Creating more measures seems the obvious way to go .. The OPQ, EPP, and NEO are primeexamples of many-facets questionnaires. At the item level, Lewis Goldberg and associates atOregon have probably gone furthest than most, with the International Personality ItemPool. This is a “Scientific Collaboratory for the Development of Advanced Measures ofPersonality and Other Individual Differences” that contains 1,252 questionnaire items used byGoldberg and his associates in a comprehensive mapping of personality space, using all majorquestionnaires as targets. Further, these items are free to use for research purposes. The web-site is: http://ipip.ori.org/ipip/ with the online foundational paper:

"A Broad-Bandwidth, Public-Domain, Personality Inventory Measuring the Lower-LevelFacets of Several Five-Factor Models," by Lewis R. Goldberg. August, 1997. Athttp://ipip.ori.org/ipip/text.htm#Text

With regard to the development of more sophisticated taxonomies of personality andintelligence, the recent exposition of the Horn-Cattell and Carroll Gf-Gc model ofpsychometric intelligence by McGrew and Flanagan (1998) is an example of howpsychometric models themselves have moved forward in terms of sophistication andcomplexity. Given a 3-level hierarchy, with 10 second order intelligence factors, includingmemory, and the demise of the classic Verbal-Performance split of established tests like theWAIS-R, there is no doubt that continuing such taxonomic modelling does have its benefits.

McGrew, K.S., Flanagan, D.P (1998) The Intelligence Test Desk Reference (ITDR). Allyn andBacon. ISBN: 0-205-19857-0

Slide 9


BUT..

Nearly all psychometric variables areconstructed from Observations,Inductive Classifications, and

Measurement of

Outcomes

However, even allowing for the current success and what promises to be some non-trivialimprovements in various aspects of psychometrics and test use over the next 5 years, I stillmaintain that essentially we have reached the limit of this entire technology and methodology.Essentially, we are mapping, describing, classifiying, and correlating outcome behaviours.Although we speak of traits, dispositions, and internalised psychological variables, our onlyjustification seems to be an assertion that someone who says they prefer, express an attitudetoward, experience a feeling of, or engage in - behaviours, actually possesses a “disposition”or “tendency” to engage in such behaviours.

We actually have no real idea about the precise generating mechanisms or processes involvedin generating such behaviours - only speculation and wishful thinking! Models couched inpsychophysiological terms have largely failed to provide replicable evidence of causal process(see Barrett, 1997) - when trying to explain psychometric personality and intelligence“traits” (with Robinson’s arousal model (1996) an exception).

Barrett, P.T. (1997) Process Models in Individual Differences Research. In Verma, V. andCooper, C. (eds.) Processes in Individual Differences. Routledge.

Robinson, D.L. (1996) Brain, Mind, and Behavior: A New Perspective on Human Nature.Praeger Press. ISBN: 0-275-95468-4.

Slide 10


So.. Is there a problem with this?

Consider the MEANING of:

u Intelligenceu Openness to Experienceu Conscientiousnessu Authoritarianism

Consider the classic definition of intelligence - as “what intelligence test measure”. Further,think how we define each of the other psychological attributes - it is from the observedbehaviours. What we cannot talk about are the processes that take place within an individualthat cause them to behave in the manner say of a high scoring individual onConscientiousness. Why should an individual be conscientious, and another not? Whatmediates these outcome behaviours? What model underlies the causal process of“Conscientiousness” - and are the variables in this model amenable to change by say trainingor instruction?

Kyllonen (1996) has also made this point in his chapter on working memory capacity andconventional psychometric measurement of intelligence. For example, p.53… “We know thereis a general factor, we even have learned how to measure it efficiently. We simply do notknow what it is in any psychological sense.”

Kyllonen, P. (1996) Is Working Memory Capacity Spearman’s g? In I. Dennis and P.Tapsfield (eds.) Human Abilities: their Nature and Measurement. Lawrence Erlbaum ISBN:0-8058-1800-6.

Slide 11


Is there another problem as well?

Consider the MEASUREMENT of:

u Intelligenceu Openness to Experienceu Conscientiousnessu Authoritarianism

and..

Psychometric measurement of psychological variables is, barring use of the Raschmeasurement model, empirically justifiable only as ordinal level measurement. That is, noneof our measures have a meaningful zero, and we conveniently assume they are equal interval.Classical Test Theory is predicated upon the assumption that item responses and test scorespossess equal-interval, additive, properties, and proceeds with statistical generalisationsaccordingly. Basically, we proceed with much of our measurement on a series of unprovenassumptions, and some knowledge that in most cases, linear (equal-interval additive)functions will approximate non-linear (ordinal) functions. The fact that most of the time weobtain usable, replicable, results from many of our psychometric measures does suggest thatthe assumptions made are tenable, if not optimal.

It was Guttman and the Israeli school of psychometricians who first showed that one couldwork with a purely ordinal or qualitative approach to psychological theory and measurement.This development has been continued here in the UK with David Canter’s group at Liverpooland with Sean Hammond at Surrey University (and Broadmoor Hospital- as chief scientist) …[email protected]

Slide 12


Michell, Science, and Measurement

u The Scientific HypothesisÊ a variable of interest possesses quantitative structure

Ê ordinality Ê additivity (concatenation)Ê a unit “object” of measurement and a standard series

Ê Direct, Extensive, Fundamental Measurement(1 variable)Ê Indirect, Derived Measurement (2 or morefundamental variables)Ê Implicit (conjoint) Measurement (2 or more“ordinal or categorical” variables)

What do we know about the practice of Science? /1What do we know about the practice of Science? /1What do we know about the practice of Science? /1

Extensive measurement is concerned with the discovery of ordering and concatenationrelations on the objects that directly reflect the quantitative structure of the variable involved.

Examples of such extensively measured variables are: length, weight, duration (time),electrical resistance.

Within the physical sciences, indirect measures of say velocity, acceleration, force, work, etc.are composed of extensive measures (weight, time, length).

Examples of conjoint measurement within psychology are rare - one implementation is theStankov and Cregan (1993) article that examines the hypothesis that intelligence (as proposedto be measured by a Letter Series task) could be considered a quantitative variable, measuredconjointly by working memory capacity and motivation, thereby satisfying the 9 constraintsgiven in Appendix 1 of the written version of the conference paper . Measures using theRasch measurement model also satisfy the constraints of conjoint measurement.

Stankov, L. and Cregan, A. (1993) Quantitative and qualitative properties of an intelligencetest: series completion. Learning and Individual Differences, 5, 2, 137-169.

Slide 13



Michell, Science, and Measurement

u The Instrumental HypothesisÊ developing more accurate measures ofmagnitudes for the quantitative variable

But - what about the UNIT of Measurement of latentvariables in implicit (conjoint) measurement - wasCampbell right when he said that psychology can never bea science unless its measurement of variables isfundamental (in the specific sense of the term)?

From Michell (1997).. “If science is taken realistically (i.e. as the attempt to understand theways of working of natural systems), and its successes leave us no reasonable alternative, thena major task for the philosophy of science is to specify the kind of place the world must be, inits most general features, for it to be possible that some scientific theories are true (where, bytrue is meant absolutely true i.e. things being just as stated in those theories). Applying this toquantitative science (exemplifed paradigmatically in physics), the task is to specify thecharacter which quantitative attributes must have if they are both measurable and interrelatedcontinuously”

Essentially, this requires us to accept that some scientific theories are realistically or FPP (forpractical purposes) “true” - which then allows examination of the conditions of the worldwherein such “truths” could ever exist, and the properties of measurement that are madewithin such a world. Although one might question whether there are any absolute truths, onemust also consider the accuracy to which classical physics makes its “approximatemeasurement”, and the extent to which modern physics makes accurate measurement (frompredictions) at the quantum level of particle/wave interaction.

Acceptance of this proposition also requires us to accept that the use of the methodology ofscience must therefore be the optimal procedure for the discovery of new “truths” concerningnatural systems. Of course, we must also ask the question of whether psychology is asuitable area of study as a science, or whether it is mostly (apart from cognitive andcomputational neuroscience, psychophysics, and perception) suited to qualitative and non-computational study.

Slide 14


The Unit of Measurement

The Barrett and Kline or (!) Kline and Barrett “position”

Although we might aim for conjoint measurement, in orderto test the first scientific hypothesis that our variable of interest (thelatent variable) is quantitative, is there not something missing here… i.e. what precisely is our unit of measurement?We might be making scientific measurement, but of what?

There is a delicate issue here between the status of our measurementand the nature of our causal theory.* Paul Kline’s forthcoming new book on “The New Psychometrics” discusses this issue inrelation to fundamental measurement and psychometrics in general.

Within the physical sciences, measurement can be either extensive or indirect. Bothmeasurement types satisfy well-defined axiomatic principles. The principles for indirectmeasurement were shown to be quantitatively testable for "ordinal" variables, such thathypothesised additive (in addition to the ordinality constraint) relations between twocombined variables could be adduced (the conjoint measurement axioms from Luce, Krantz,Suppes, and Tversky). However, within the physical sciences, indirect measures of sayvelocity, acceleration, force, work, etc. are composed of extensive measures (weight, time,length) - where an extensive measure can be loosely defined as one where there is a more orless direct isomorphism between the numerical quantities of a scale, and the property ofconcatenation of an object being measured. It is this "extensive" - strongly isomorphic -nature of such measures that would seem to us to be the "other" requirement for investigationof phenomena causation. Thus, within psychology in general, failure to adopt or at leastrecognise fundamental measurement axioms is the first fatal flaw, the second, in our opinion,(IF one is trying to elucidate causal processes for phenomena) is to sidestep the problem ofthe "unit of measurement".

Therefore, if the above holds, Rasch measurement of course conforms to the first pre-requisite of scientific investigation, but looks a little queasy on the second - in that one of itsvariables - the "facility/ability” measure, has no fundamental (extensive?) unit ofmeasurement (the "latent trait" sidestep). This issue is very much tied up with the obsession inpsychology of seeking to measure “things”, without any regard as to the status of themeasurement within any kind of testable causal theory or explanatory process. So, in TST wecount behaviours, score them, adopt an equal-interval scale, and make pronouncements aboutthem, without ever once considering what the unit of measurement is, and how we might testthat this unit is in any way meaningful with regard to our understanding of the construct weclaim to be measuring (instead of just its utility value).

Slide 15



Haig, (+ a little bit of Barrett!), PhenomenaDetection, and the process of science.

u Phenomena Detectionu Measurement, deduction and abduction(Haig) of relations allied to...u Causal, explanatory process theory

But, as Einstein has noted, many of his equations predictedphenomena that had not been observed or even looked for. ForEinstein (and many theoretical physicists), it is the mathematicaldescriptions of the world that are the primary genesis forphenomena identification.

Brian Haig ([email protected]) is currently writing an exposition of this approachto investigative processes within psychology. I have augmented and extended (probablyruinously) some of his early musings on this topic!

However, we must take into account the Penrose argument (also espoused by Kuhn,Andrich,Wright, and others) …A popular view of how science works is that the scientist,looking to explain a series of observations, finds a mathematical relationship that accounts forthe data. However, these individuals argue that the mathematical relationship is thereality, and the data merely an expression of it. Einstein conceived of his special andgeneral relativity equations before data were available to verify them; when data becameavailable, his calculations were found to be exact. This questions the very basis and meaningof “phenomena identification”.

Slide 16


So, how does Psychometric measurement compare with Scientific measurement?So, how does So, how does PsychometricPsychometric measurement measurement compare with compare with ScientificScientific measurement? measurement?

u Few, if any, meaningful units of measurementu Except for Rasch analysis, or Luce and Tukey’sadditive conjoint analysis, we are using variables thathave no empirically tested quantitative structure.

u The entire focus is on phenomena identificationu No serious attempt is made to examine causalprocess

Slide 17


BUT.. Is Individual Differences Psychology actuallyamenable to Scientific Investigation?

u Emergent Properties of Complex Systems

uPenrose and the Non-computability hypothesis

u Maraun, rules, grammar, and meaning

Ê Non-linear dynamical systems/chaos theory

Ê Deterministic but Non-computable -Cellular Automata

Ê The autonomy of constitutive rule systems

Emergence … a hypothesised property of complex computational systems. As part of thebuilding of computational systems to mimic human attributes, these systems would somehowproduce the “emergence” of complex attributes such as emotion, creativity, andconsciousness.

Penrose, Godel, and non-computability. A simplified form of the Gödel theorem ..“Mathematical insight cannot be coded in the form of some computation that we know to becorrect” (Penrose, 1997 p.112). Non-computability refers to the fact that certain systems canbe shown to be deterministic (in that the rules determining their existence are finite andcompletely defined), but that the evolution of system states is non-computable (nocomputational-mathematical solution available). These forms of systems have been calledcellular automata (after Stanislaw Ulam) and investigated more completely bymathematicians and cognitive scientists such as Wolfram and Hameroff. The system rules canbe implemented on a computer, but there is no known computational decision procedure toevaluate the final outcome. The book by Casti (1994) is an excellent overview of the field andits application to evolutionary emergent systems.

Maraun’s contribution, relying heavily upon the Wittgenstein view of meaning, shows thatfrom a different perspective, the Gödelian paradox exists also within psychometrics. If weclassify outcomes, can we determine from the empirical data (items) alone the correctmeaning of what is being measured? For Maraun (via Wittgenstein), meaning is not able to bedetermined from the rules that govern the operations carried out on empirical item data.Further, the existence of “rules” cannot be tested with empirical data, as the very test itselfimplies the rule’s existence.(continues on next slide)..

Slide 18


What do you think is missing from thecurrent generation of tests that might

explain why they seem to predict behaviourso poorly?

-is it--is it-More of the same is what we need?

-or--or-Is there something so flawed with our whole

approach that we require majorphilosophical surgery?

Back to The QuestionsBack to The QuestionsBack to The QuestionsSo..

“For example, no empirical finding can refute or or support a measurement claim. Forexample, the claim that ‘these are measurements of IQ’ cannot be shown to be correct orincorrect on the basis of the actual numbers recorded, nor the correlation of these numberswith other sets of numbers (e.g. measurements of school performance etc.). On the contrary,rules are constitutive for empirical evidence: these empirical findings are not about IQ at allunless they already are based upon numbers that have meaning as measurements of IQ.” ..(Jackson and Maraun (1996), p. 115)This logic, as he shows in the Jackson and Maraun papers mentioned earlier, questions thewisdom of the issue of construct validity of Meehl and Cronbach. Further, the entire testconstruction process within classical test theory is put in doubt.

Essentially, Maraun’s position is that unless one follows rules (standards of correctness) forthe measurement of a construct (e.g. Michell’s quantitative structure “rules”), then theassignation of meaning to what is being measured can never be resolved. Further, these rulesare considered autonomous and non-discoverable - such as the axioms within fundamentalmeasurement.

Casti, J.L. (1994) Complexification: Explaining a Paradoxical World Through the Science ofSurprise.. Harper-Collins. ISBN: 0-06-016888-9.

Jackson, J.S.H., Maraun, M. (1996b) Whereof one cannot speak, thereof one must remainsilent. Personality and Individual Differences, 21, 1, 115-118

Slide 19


My Responses to “what is missing”My Responses to “what is missing”My Responses to “what is missing”

u Fundamental measurement of variablesu Meaningful units of measurementu Causal theory and explanatory coherence

My observations of “flawed” thinking My observations of “flawed” thinking My observations of “flawed” thinking

u The “fractal” fallacyu The fixation on static taxonomy and traitsu No attention paid to the knowledge comingfrom computational & cognitive neuroscience,and molecular genetics.

The “fractal” fallacy: that there are a finite set of behaviours and outcomes that can bemeasured in more and more detail. Fractal computational geometries tell us that as weuncover a level of complexity, it in turn requires us to uncover its own complexity, and so on.The “stopping point” is a tradeoff between utility of prediction and cost of using more andmore variables.

Some Neuroscience: Stress and Hippocampal cell regeneration.For years, it has been assumed that once animals or people reach adulthood, they may losebrain cells but can never grow new ones. But, Gould and her colleagues have found thatmonkeys are constantly making new brain cells in the hippocampus, an area of the brainassociated with memory functions. Moreover, they found that this cell production wasdiminished when the animals are put under extreme stress.They used two chemical tracers-one that bound into cells that were dividing, the other that labelled only mature cells.Gould, E., Tanapat, P., McEwen, B., Flugge, G. and Fuchs, E. (1998) Proliferation of granulecell precursors in the dentate gyrus of adult monkeys is diminished by stress. Proceedings ofthe National Academy of Sciences, 95, 6, 3168-3171.

(cont. on next slide)

Slide 20


So, what does my research program now look like? So, what does my research program now look like?

u The PracticalÊ Computational profiling technology

u PsychometricsÊ Rasch scaling of all variables and tests

u Cognitive Process- PsychometricsÊ Bits-per-second information processingÊ Pattern Recognition tasksÊ Working Memory and memory in generalÊ Speed of Rule AcquisitionÊ Inspection Time

Some Neuroscience: A new Neurobiological Theory of ExtraversionExtraversion, defined by two constructs (interpersonal engagement, and impulsivity), ispostulated as being based upon incentive motivation. The neural pathways and regional brainstructures (frontal, amygdala, and hippocampus), and the major transmitter network(dopamine) have been identified from animal models. This is a revision of Gray’s work - butmuch more than that in reality. It is this paper/monograph which brings home the difficulty ofusing traits as causal explanations. An approach that uses process to explain behavioural“traits” barely needs traits at all, but rather can begin to explain behavioural outcomes assimple functions of basic processes, without ever needing to use the word “trait”. However,this is a secondary issue concerning the use of taxonomies vs the use of process models.

Depue, R.A. and Collins, P.F. (1998- in press) Neurobiology of the structure of personality:dopamine facilitation of incentive motivation, and extraversion. Behavioural and BrainSciences - http://www.cogsci.soton.ac.uk/bbs/Archive/bbs.depue.htm

Slide 21


And...? And...?

u Biophysical

Ê Oscillatory vs Component-Process EEGÊ Nerve pathway transmission characteristicsas one of a class of foundational properties ofcognitive processingÊ Neuromagnetic source-localised MEEG

u Computational ModellingÊ Nerve Conduction transmission, givenmembrane properties, myelin density, axonalfibre width, synaptic variables, andmicrotubule/actin deformation of the synapse

Slide 22


From: Eysenck (1997)

From:

Eysenck, H.J. (1997) Can personality study ever be objective? The role of experiment in discoveringthe structure of personality.. In Cooper, C. and Varma, V. (Eds.). Processes in Individual Differences.Routledge. ISBN: 0-415-14119-2.

Slide 23

Self Reportof Behaviours

BehavioursPerfomance

Competencies

NeurosciencePhysiology

Biochemistry

Molecular GeneticsParticle Physics

PersonalityStudies

IntelligenceStudies

Astrology, Graphology, Relativism

strong causal linkages

weakcausal linkage


My perspective

Slide 24


homepagehttp://www.liv.ac.uk/~pbarrett/paulhome.htmpresentations pagehttp://www.liv.ac.uk/~pbarrett/present.htm

Affiliations: Chief ScientistThe State HospitalCarstairsLanarkScotland ML11 8RP

Fax: 01555-840024

Also: Senior Research FellowDept. of Clinical Psychology, Univ. Of Liverpool


homepagehttp://www.liv.ac.uk/~pbarrett/paulhome.htmpresentations pagehttp://www.liv.ac.uk/~pbarrett/present.htm

Affiliations: Chief ScientistThe State HospitalCarstairsLanarkScotland ML11 8RP

Fax: 01555-840024

Also: Senior Research FellowDept. of Clinical Psychology, Univ. Of Liverpool

Documents

BPS Test User Conference Keynote paper - · PDF fileapproach that we require major ... our approach to measurement and assessment of individuals is unlikely to occur without an