Measurement Properties of the Neurobehavioral Cognitive StatusExamination (Cognistat) in Traumatic Brain Injury Rehabilitation

Nicholas A. DoningerKettering Memorial Hospital and Wright State University

School of Medicine

Dawn M. EhdeUniversity of Washington School of Medicine

Rita K. BodeRehabilitation Institute of Chicago and Northwestern University

Kristin Knight and Charles H. BombardierUniversity of Washington School of Medicine

Allen W. HeinemannRehabilitation Institute of Chicago and Northwestern University

Objective: Evaluate measurement properties of the Neurobehavioral Cognitive Status Examination(Cognistat) using Rasch analysis. Design: Calibration of item responses from 120 individuals admittedto a rehabilitation medicine service for traumatic brain injury (TBI) and 296 community-dwelling adultswith TBI. Results: Three strata of performance were differentiated despite a skewed distribution towardhigh performance among the community sample. Elimination of easier items created a better targetedinstrument (i.e., generated more spread among individuals) without a significant increase in error.Memory and verbal reasoning were the most difficult domains for each sample; however, analysesindicated significant measurement error. Conclusions: As a screening instrument, the Cognistat reliablyclassifies multiple levels of cognitive status in both acute and postacute TBI settings; however, thismeasure is unsuitable for generating a profile of neurocognitive strengths and weaknesses.

Keywords: neuropsychological tests, brain injury, psychometrics, rating scale analysis

Neuropsychological assessment serves an important role in theacute rehabilitation of persons with traumatic brain injury (TBI) bydescribing cognitive abilities and providing information pertinentto treatment and disposition planning. The administration of fullneuropsychological batteries in the inpatient setting has becomeincreasingly difficult and impractical with current health care mar-ket trends toward decreasing lengths of stay. Shortened lengths ofstay and the inability of many patients with TBI to participate infull evaluations during acute rehabilitation have generated a need

for sensitive, brief, and easily administered cognitive screeningassessments. Ideally, these instruments would be sensitive to thepresence of cognitive dysfunction, identify cognitive domains thatwarrant further neuropsychological evaluation (Lezak, Howieson,& Loring, 2004), and have the potential to track neuropsycholog-ical change over time. For a screening battery to be clinicallyuseful in the acute rehabilitation setting, the measure should yieldinformation that is useful to the treating professionals, patient, andfamily in planning for treatment and discharge.

The Neurobehavioral Cognitive Status Examination (Cognistat;Kiernan, Mueller, Langston, & Van Dyke, 1987) is a brief screen-ing instrument designed to quantify a range of cognitive functions(Northern California Neurobehavioral Group, 1988). The cogni-tive domains measured by the Cognistat include orientation, basicattention, language, constructional ability, verbal memory, mentalcalculations, and reasoning. The language domain consists of threesubsections—comprehension, sentence repetition, and object nam-ing. Verbal reasoning is assessed with similarities and judgmentsubtests. The examiner also rates and describes the patient’s levelof consciousness.

The Cognistat uses a screen and metric approach whereby aninitial item serves as a general screen for each domain, exclud-ing orientation and memory. If the screening item is passed, theunderlying ability presumed to mediate performance is consid-ered normal, a maximum score is assigned, and no furthertesting is conducted. If the patient fails the screening item,additional metric items are administered to establish perfor-mance level. The score from each subtest is plotted on a profile

Nicholas A. Doninger, Wallace-Kettering Neuroscience Institute, Ket-tering Memorial Hospital, Dayton, Ohio, and Department of Psychiatry,Wright State University School of Medicine; Dawn M. Ehde, KristinKnight, and Charles H. Bombardier, Department of Rehabilitation Medi-cine, University of Washington School of Medicine; Rita K. Bode andAllen W. Heinemann, Rehabilitation Institute of Chicago, Chicago, Illi-nois, and Feinberg School of Medicine, Northwestern University.

Funding was provided by the National Center for Injury Prevention andControl and the Disabilities Prevention Program, National Center forEnvironmental Health, of the Centers for Disease Control (Grants R49/CCR and 512721). We express our appreciation to the clinical and admin-istrative staff at Ohio State University’s Traumatic Brain Injury Network,Miami Valley Hospital, Wright State University’s Substance Abuse Re-sources and Disability Issues (SARDI) Program, and the RehabilitationInstitute of Chicago.

Correspondence concerning this article should be addressed to Nich-olas A. Doninger, PhD, Wallace-Kettering Neuroscience Institute, 3533Southern Boulevard, Suite 5200, Kettering, OH 45429. E-mail:nicholas.doninger@kmcnetwork.org

Rehabilitation Psychology2006, Vol. 51, No. 4, 281–288

Copyright 2006 by the American Psychological Association0090-5550/06/$12.00 DOI: 10.1037/0090-5550.51.4.281

281

This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

that provides a domain-specific representation of the patient’scognitive status. Performance is rated as average (no impair-ment), mildly impaired, moderately impaired, or severely im-paired. In addition to brevity (10 –20 min) and ease of admin-istration, the Cognistat assesses independent areas of cognitivefunctioning rather than providing a summation score, which isthe typical result of most cognitive status screening examina-tions (Schwamm, Van Dyke, Kiernan, Merrin, & Mueller,1987).

Previous studies have established the clinical utility of theCognistat as a measure of cognitive status in a variety of medicalpopulations, including neurological populations (Cammermeyer &Prendergast, 1997), neurosurgical patients with documented brainlesions (Cammermeyer & Evans, 1988; Northern California Neu-robehavioral Group, 1988; Schwamm et al., 1987), stroke patients(Osmon, Smet, Winegarden, & Gandhavadi, 1992), geriatric indi-viduals (Fields, Fulop, Sachs, Strain, & Fillit, 1992; Fladby et al.,1999), and people with psychiatric disorders (Logue, Tupler,D’Amico, & Schmitt, 1993; Mitrushina, Abara, & Blumenfeld,1994). In addition, the Cognistat is significantly more sensitive tothe effects of neurological impairment and normal aging than twowidely used cognitive status examinations—the Cognitive Capac-ity Screening Examination and the Mini-Mental Status Examina-tion (Drane et al., 2003; Schwamm et al, 1987).

The psychometric utility of the Cognistat in persons with TBI iscomparatively less understood. Research has underscored the Cog-nistat’s sensitivity to cognitive impairment across a range of injuryseverity (Blostein, Jones, Buechler, & Vandongen, 1997) andrehabilitation settings and documented significant relations be-tween performance during initial inpatient rehabilitation for TBIand clinical outcomes 1 year postinjury (Poon, Zhu, Ng, & Wong,2005). Additional research has reported relations between Cogni-stat subtests and paired neuropsychological tests assessing relatedcognitive domains (Nabors, Millis, & Rosenthal, 1997; Wallace,Caroselli, Scheibel, & High, 2000). Despite evidence of concurrentvalidity, individual Cognistat subtests were found to classify indi-viduals as impaired versus unimpaired at a much lower rate rela-tive to neuropsychological tests, which draws into question theability of the Cognistat to delineate domain-specific cognitivefunctioning (Marcotte, van Gorp, Hinkin, & Osato, 1997; Wallaceet al., 2000).

Rating scale analysis (RSA) was used to evaluate the psycho-metric properties of the Cognistat in a sample of 186 adults livingin the community following TBI (Doninger, Bode, Heinemann, &Ambrose, 2000). A ceiling effect was observed, with most com-munity-dwelling participants performing very well, which sug-gests that the Cognistat may be a more useful measure in sampleswith considerably greater cognitive impairment. To our knowl-edge, RSA has not been used to examine the utility of the Cogni-stat in persons with acute and potentially more severe cognitiveimpairment.

The purpose of this study is to (a) calibrate item responsesapplying various RSA measurement strategies to investigate theCognistat’s ability to distinguish meaningful levels of cognitiveimpairment in an inpatient sample of individuals with acute TBI ascompared with an outpatient sample of individuals with TBI and(b) investigate the appropriateness of the Cognistat for character-izing different types of cognitive impairments.

Method

Participants

This study uses Cognistat data collected from two independent samples.The first sample consisted of 134 patients admitted to the inpatient reha-bilitation medicine service at Harborview Medical Center between January1996 and January 1999 for treatment of TBI. Participants were drawn fromconsecutive admissions. On the basis of initial Glasgow Coma Scale (GCS;Teasdale & Jennett, 1974) scores and head computed tomography findings,30.6% of brain injuries were categorized as mild (GCS � 13), 24.6% werecategorized as moderate (GCS � 9–12), and the remaining 44.8% werecategorized as severe (GCS � 8). Participants (n � 14) were excluded ifthey did not speak English, were younger than 18 years of age, or expe-rienced severe psychosis, as clinically diagnosed by the staff psychologistor team physiatrist.

Of the 120 remaining eligible participants, 98 were men (82%), with anaverage age of 37.5 years (SD � 12.6, range � 18 to 77 years). Themajority (82%) were Caucasian (n � 98); the next most populous groupwas African American (n � 11), at 9%. Eighty-one were unmarried (67%),37 were either married or cohabiting (31%), and the marital status of 2individuals was unknown. An overwhelming majority (93%) had a perma-nent residence (n � 111), with only 5 homeless participants (4%). Theliving situation of 4 individuals was unknown. Ninety participants wereemployed at the time of injury (75%), 26 were unemployed (22%), and theemployment status of 4 participants was unknown. Only 22 participants(18%) had an education beyond high school, 50 (42%) had a high schooleducation, and 47 (39%) had not completed high school.

Measurement properties of the Cognistat were previously reported for asample of 186 community-residing adults who sustained a TBI (Doningeret al., 2000) and served as participants in a multicenter study conductedbetween January 1997 and May 2000 that examined characteristics ofcommunity-based case management programs and treatment outcomes(Heinemann, Corrigan, & Moore, 2004). These participants are included aspart of a larger current sample (n � 296), which, coupled with the inpatientrehabilitation sample, was designed to enhance and broaden the potentialrange of cognitive functioning.

Inclusion criteria included a brain injury of traumatic origin, current ageof 18 to 65 years, English speaking, no clinical reports of active oruntreated psychosis, and persistent posttraumatic amnesia greater than 24hr, verified through available medical records indicating at least moderateinjury severity (Bond, 1990). The community sample was populated pre-dominantly with men (n � 230) and averaged 36 years of age (SD � 9.9).The majority (69%) were Caucasian (n � 205); the next most populousgroup was African American (n � 73), at 25%. Most participants wereunmarried (n � 240; 83%), whereas 51 were either married or cohabiting(17%); the marital status of 5 participants was unknown. Sixty-four wereemployed or attending school (22%), 213 were unemployed (72%), and theemployment status of 19 was unknown. Eighty-nine participants (31%) hadan education beyond high school, 123 (42%) had a high school education,and 76 (26%) had not completed high school. A description of the demo-graphic characteristics for each sample is presented in Table 1.

Measures

The Cognistat (NCSE; Kiernan et al., 1987; Northern California Neu-robehavioral Group, 1988) is a brief screening instrument designed toquantify a range of cognitive functions, including orientation, attention,language (Naming, Commands, and Repetition subtests), constructionalpraxis, memory, calculations, and verbal reasoning (Judgment and Simi-larities subtests). Level of consciousness is also assessed with a qualitativerating (alert, lethargic, fluctuating) but not scored. Performance on eachsubtest is summarized as a profile with age-corrected cutoffs that isintended to provide a concise, differentiated representation of cognitive

282 DONINGER ET AL.

This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

status. Scores from each subtest fall into the normal range or one of threeimpairment levels—mild, moderate, or severe. Previous research examin-ing the validity of screen items has yielded false negative rates rangingfrom 0% to 15%, with the highest rates observed for the Calculations(11%), Attention (13%), and Similarities (15%) subtests (Marcotte et al.,1997; Oehlert et al., 1997). Participants in the current study were admin-istered the entire Cognistat.

Procedure

At Harborview Medical Center, participants completed a routine batteryof neuropsychological tests, including the Cognistat, during their acuteinpatient rehabilitation stay. Those who were alert and able to participatein a testing battery were tested when feasible on the basis of their treatmentschedule and psychometrist availability. The Cognistat was administered tocommunity-dwelling individuals with TBI at either the initiation of casemanagement services or study commencement.

All testing was performed in a standardized manner by trained psychom-etrists. The tests were administered, on average, 29 days (Mdn � 16 days)after injury (range � 4 to 244 days, SD � 29) in the acute inpatient sampleand 2,496 days (Mdn � 1,079 days) after injury (range � 31 to 15,939days, SD � 3,168) in the community-dwelling sample. For both samples,responses for each domain of cognitive functioning were scored in accor-dance with rating scale structures that varied across the Cognistat’s 10subtests. One item, orientation to person, had no variability and thereforewas deleted from the item set prior to calibrations. All procedures for thisstudy were approved by the human subjects institutional review boards atthe respective centers.

Analysis Plan

RSA of the data was performed with a Windows 95/98 program calledWINSTEPS (Linacre & Wright, 1999). This technique provides a means ofcalibrating item responses to construct a scale on which linear measuresunderlying the observations are defined. The calibrations performed in thecurrent study parallel those reported in a previous study with an outpatientsample of adults with TBI (Doninger et al., 2000).

Each of the calibrations produces estimates of person ability and itemdifficulty along a shared continuum as well as estimates of item and person fitto the measurement model. Responses to Cognistat items were originallyrecorded on an ordinal scale; however, these raw data only permit rankordering of scores. Equal-interval scales are necessary to accurately measureindividual attributes or make quantitative comparisons between individuals.Rasch analysis (Rasch, 1980; Wright & Masters, 1982) is a measurementprocedure that can be used to evaluate instrument reliability and estimateinterval-level measures from ordinal data using a logit transformation.

The reliability of the estimated measures was determined by examina-tion of the person and item separation indexes and their related separationreliability. Separation refers to the number of distinct strata that can beidentified in the distribution of persons and tasks. High separation indicatesthat measures vary because people actually possess or tasks actuallycharacterize different amounts of the attribute, whereas low separationindicates either a homogeneous sample in terms of ability or the presenceof substantial measurement error that undermines confidence in stating thatpeople differ in regard to a measured attribute. On the basis of theseparation index, RSA yields related separation reliabilities that are inter-preted similarly to Cronbach’s alpha. A value equivalent to or exceeding.80 (corresponding to a separation index of 2.0) indicates a desirable levelof reliability in that three strata of performance (high, middle, low) can bedistinguished (Fisher, 1992).

RSA determines construct validity by examining the hierarchy ofitem difficulty and the coherence, or fit, of items with the latentconstruct in a manner analogous to a chi-square analysis; that is,observed responses are compared with those expected on the basis ofthe overall level of the construct, via specified criteria. Item infitindicates the extent to which persons respond to items as expectedaccording to their estimated level of the attribute (i.e., cognitive status).Item misfit provides an indication that an item may be measuring aconstruct other than the one measured by the remaining items or isproviding information that is redundant with the remaining items.Acceptable cutoffs to index item misfit vary. A stringent criterionidentifies infit mean squares between 0.80 and 1.20 as fitting; however,a more lenient criterion of values between 0.70 and 1.30 was identifiedas fitting on the basis of prior studies examining the measurementproperties of instruments with samples of individuals with TBI (Bode,Heinemann, & Semik, 2000; Doninger et al., 2003).

Ideally, persons possessing more of the measured attribute would likelyperform difficult tasks, whereas persons possessing less of the attributewould perform less difficult tasks. The Rasch model assumes that theprobability of a given patient passing an item or task is a logistic functionof the relative distance between item difficulty and the ability of therespondent, which are expressed as logit scores—that is, the natural loga-rithm of the probability of a correct response.

Results

Version 1: Original 51-Item Cognistat

Results from the series of calibrations are summarized in Table2. Calibration of the entire item set with data from the inpatientrehabilitation sample (see row 1, Table 2) was conducted to derivea single measure from the Cognistat. Three strata of performancewere distinguished, as evidenced by a person separation index of2.18 (reliability � .83). The item hierarchy showed that the easiestitems measured basic language functions, whereas the hardestitems assessed orientation and memory. Although the item sepa-ration index was acceptable (3.71, corresponding to a reliability of.93), there was more than a 2-logit difference (2.18) betweenrespondents’ ability and item difficulty, indicating that the instru-ment was far too easy for this sample. Five items assessing

Table 1Description of Inpatient Rehabilitation and Community-DwellingSamples

VariableInpatient

rehabilitation Community

Men/women 98/22 230/66Age 37.5 (12.6) 36 (9.9)Ethnicity (%)

Caucasian 82 69African American 9 25

Marital statusUnmarried 81 (67%) 240 (83%)Married/cohabiting 37 (31%) 51 (17%)Unknown 2 5

Employment statusEmployed 90 (75%) 64 (22%)Unemployed 26 (22%) 213 (72%)Unknown 4 19

Education� 12 years 22 (18%) 89 (31%)High school 50 (42%) 123 (42%)� 12 years 47 (39%) 76 (26%)

Note. For the inpatient rehabilitation sample, n � 120. For the commu-nity sample, n � 296. Values are sample sizes except for age, for whichvalues are means, with standard deviations in parentheses.

283MEASUREMENT PROPERTIES OF THE COGNISTAT

This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

constructional ability, repetition, memory, and orientation demon-strated misfit (mean square infit � 1.30), suggesting that theseitems may tap extraneous domains. Infit values for a repetition anda judgment item indicated that these items overfit the construct ofinterest (i.e., cognitive status), contributing little new informationbeyond that provided by the remaining items.

Calibration of the entire set of items with data from thecommunity-dwelling sample (see row 2, Table 2) distinguishedtwo strata of performance, as evidenced by a person separationindex of 1.69 (reliability � .74). Because the distribution ofmeasures was so skewed, with skewness statistics for eachdomain score greater than two times the standard error (Tabach-nick & Fidell, 1996), the mean square error term associatedwith the person separation measure (root-mean-square error �9.72) served as a more appropriate measure for defining strataof performance (Wright & Masters, 2002). Multiplying themean square error term by 3 and serially subtracting the result-ant product from the maximum value of 110 logits on a trans-formed, equal-interval person measure distribution definedthree strata of performance. The item hierarchy showed that theeasiest items assessed basic language functions, whereas thehardest items assessed memory, verbal reasoning, and construc-tions. Although the item separation index was acceptable (3.73,corresponding to a reliability of .93), there was nearly a 3-logitdifference (2.80) between average person ability and item dif-ficulty levels, indicating that the instrument was also too easyfor this sample. An orientation item and a memory item mis-fitted the construct of cognitive status, whereas a single reason-ing item provided little new information.

Despite the fact that the distribution of the community-dwellingsample was skewed, the hierarchy of item difficulty was similar tothat found for the acute inpatient rehabilitation sample (see Figure1). This is illustrated in Figure 1 in the proximity of points to theidentity (middle) line representing identical item calibrations foreach sample. Notable discrepancies were evident on items exam-ining orientation to time, which were among the most difficultitems for the acute rehabilitation sample and slightly below aver-

age in difficulty for the community-dwelling sample, suggestingthat these items might have measured different constructs betweenthe two samples.

Version 2: Shorter Cognistat Instruments

With a second set of calibrations, we attempted to improve theinstrument’s ability to distinguish levels of cognitive performanceby eliminating the easiest items. We created shorter instrumentsfor each sample separately by deleting items with a difficulty levelof less than 0.0 logits. Calibration of the resultant 25-item instru-ment for the inpatient rehabilitation sample (row 3, Table 2)yielded a slight decrease in person separation, to 2.05 (reliability �.81), whereas item separation increased to 5.02 (reliability � .96).Similar to the initial calibration, orientation, memory, and reason-ing items demonstrated misfit. A repetition item also exhibitedslight misfit. Despite the negligible decrease in person separation,three distinct strata of performance remained identifiable. Itemsconstituting the comprehension subtest were deleted entirely; how-ever, the difference between the average person ability level andthe item difficulty level was reduced to 0.91 logits, thus improvingthe targeting of items to this sample.

Deleting easier items resulted in a 28-item instrument for thecommunity-dwelling sample, with 22 of the items overlappingwith the shorter instrument created with the inpatient sample.Calibration of this modified instrument (row 4, Table 2) yieldeda slight decrease in person separation (1.60; reliability � .72),whereas item separation increased to 3.91 (corresponding to areliability of .94). Similar to the calibration with the full in-strument, a memory item and a reasoning item demonstratedmisfit. Deleting the easiest items reduced the difference be-tween the average person ability level and item difficulty levelto 1.53 logits, thus improving the targeting of items to thissample. Applying the mean square error term associated withthe person ability measure (11.11) also allowed three strata ofcognitive status to be defined.

Table 2Summary of Neurobehavioral Cognitive Status Examination Calibrations

Calibrationno.

No. ofitems Sample

Misfitting items(mean square infit)

Person separation(reliability)

Item separation(reliability)

Mistargeting(logits)

1 51 Acute rehabilitation Construction (1.31, 1.34)Repetition (0.67, 1.43)Memory—green (1.41)Judgment (0.67)Orientation—date (1.64)

2.18 (.83) 3.71 (.93) 2.18

2 51 Community sample Memory—green (1.63)Orientation—date (1.34)Reasoning (0.67)

1.69 (.74) 3.73 (.93) 2.80

3 25 Acute rehabilitation Memory—green (1.33)Orientation—date (1.60)Judgment (0.66)Repetition (1.32)

2.05 (.81) 5.02 (.96) .91

4 28 Community sample Memory—green (1.56)Reasoning (0.65)

1.60 (.72) 3.91 (.94) 1.53

5 10 Acute rehabilitation 0.93 (.47) 4.69 (.96) 1.356 10 Community sample Naming (0.62)

Comprehension (0.68)0.54 (.22) 4.98 (.96) 1.66

284 DONINGER ET AL.

This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

Version 3: Cognistat Domains

Given the original intention to provide a differentiated profileof cognitive status, the 10 domain scores were treated as itemsand calibrated separately for each sample. In accord with nor-mative data provided in the manual, raw scores for each domainwere converted to a rating scale as follows: average (1), mildlyimpaired (2), moderately impaired (3), or severely impaired (4).Memory and verbal reasoning were identified as the mostdifficult domains for persons following TBI, whereas naming,orientation, and comprehension were among the easiest do-mains. Although the separation indexes indicated that itemscould be differentiated into easy, average, and hard strata forboth samples, person separation reliabilities for the rehabilita-tion sample (.47) and community-dwelling sample (.22) indi-cated only slightly more variation in measures than what couldbe attributed to measurement error.

Calibrating the 10 domain ratings for the acute rehabilitation sam-ple (row 5, Table 2) yielded a person separation index of 0.93(reliability � .47). Application of the mean square error term (0.62)for the person ability level measure indicated that the domains, treatedas separate items, did not reliably differentiate more than those indi-viduals scoring in the normal range on all subtests from those who didnot. The item separation index for this analysis was 4.69 and was

associated with an item reliability value of .96, which suggests that theordering of domains reflects real differences in difficulty level ratherthan measurement error. None of the items demonstrated misfit. Thedifference between the average person ability level and item difficultylevel with this 10-domain item set increased relative to the 25-itemset, to 1.35 logits.

A similar calibration conducted with data from the community-dwelling sample (row 6, Table 2) yielded a person separation of0.54 (reliability � .22), indicating that essentially all the variationin measured ability among persons could be the result of measure-ment error. Application of the person measure mean square errorterm (0.66) indicated that the domains, treated as items, do notreliably differentiate more than those individuals who scored in thenormal range on all subtests from those who did not. The itemseparation index for this analysis was 4.98 and was associated withan item reliability value of .96, which suggests that the ordering ofdomains reflects real differences in difficulty level rather thanmeasurement error. The difference between the average personability level and item difficulty level with this 10-domain item setincreased slightly relative to the 28-item set, to 1.66 logits. Thenaming and comprehension domains contributed little to the over-all construct of cognitive status measured by the other items, asindicated by infit statistics of 0.62 and 0.68, respectively.

Figure 1. Scatterplot of item difficulties for acute and community samples. The difficulty of the scale is expressedin logits, with the center of the scale set at zero. The diagonal line represents item calibrations that would be identicalfor acute rehabilitation and community-dwelling individuals. Integers indicate the Neurobehavioral Cognitive StatusExamination item number: Items 1–6 are the Orientation subtest, Items 7–14 are the Attention subtest, Items 15–20are the Comprehension subtest, Items 21–26 are the Repetition subtest, Items 27–34 are the Naming subtest, Items35–37 are the Constructional Ability subtest, Items 38–41 are the Memory subtest, Items 42–45 are the Calculationssubtest, Items 46–49 are the Reasoning subtest, and Items 50–52 are the Judgment subtest.

285MEASUREMENT PROPERTIES OF THE COGNISTAT

This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

Discussion

RSA of the Cognistat was conducted with data from separatesamples of inpatient rehabilitation and community-dwelling indi-viduals with TBI. Several calibrations using various strategieswere performed to construct a single measure targeted for eachgroup. The series of calibrations revealed inconsistencies in theobserved measurement properties between the two samples.

For the inpatient rehabilitation sample, the observed person anditem separation indexes suggest that the full instrument actuallycharacterizes different amounts of a particular attribute (i.e., cog-nitive status). Elimination of the easiest items created a bettertargeted 25-item instrument, as evidenced by a reduction in thedifference between the average person ability level and item dif-ficulty level with no significant increase in error. This abbreviatedmeasure remained capable of distinguishing three distinct strata ofperformance. By comparison, the calibration of separate domainswas only able to reliably differentiate individuals who scored inthe normal range on all subtests from those who did not.

The series of calibrations conducted with data from the com-munity sample yielded consistently poor separation indexes andassociated reliabilities. Calibration of the entire item set distin-guished two strata of performance; however, the distribution ofmeasures was skewed. Applying the mean square error term, anestimated quantity that, when subtracted from a logit measure,gives the least distance required before a difference becomesmeaningful, identified three distinct levels of performance. Al-though RSA does not presume that measures are normally distrib-uted, a mismatch between the ability level of individuals in asample and the difficulty level of items undermines the ability toreliably distinguish different levels of performance. The restrictedvariability and skewed distribution of raw scores toward higherlevels of performance revealed by RSA indicated that this measurewas far too simple for a postacute, high-functioning sample ofadults with TBI.

Despite the fact that the distribution of the community-dwellingsample was skewed, the hierarchy of item difficulty was similar tothat found for the acute inpatient rehabilitation sample. Initialcalibration of the entire instrument for both samples showed thatthe easiest items assessed basic language skills, whereas the hard-est items assessed memory, verbal reasoning, and constructionalability. Both samples also evidenced consistently high item sepa-ration indexes and associated reliabilities, regardless of the strat-egy attempted, which indicates that the items themselves actuallycharacterize different amounts of the measured attribute (i.e., cog-nitive status); however, in larger samples, estimates of measure-ment error are likely to be smaller, which in RSA yields higherseparation values.

Person separation indexes and associated reliabilities were gen-erally adequate for the inpatient rehabilitation sample, with theexclusion of the calibration of domain items, whereas consistentlylow reliabilities were found with the community-dwelling sample.Low person reliability is obtained if items are applied to a rela-tively homogeneous sample in terms of cognitive status, as sug-gested by ceiling effects, which were observed with the commu-nity-dwelling sample. The low person separation reliabilitiesassociated with the calibration of separate domains mainly reflects

the fact that a small number of items are generally inadequate todifferentiate large samples into different levels of cognitive status.

The final set of calibrations for each sample yielded similarfindings, identifying memory and verbal reasoning as the mostdifficult domains, consistent with clinical findings regarding per-sistent neurocognitive deficits following TBI (Arciniegas, Held, &Wagner, 2002; Damasio & Anderson, 1993; Levin, 1989; Tate,Fenelon, Manning, & Hunter, 1991; Van Zomeren & Van Den-burg, 1985). Other facets of executive function, including cogni-tive processing speed, set shifting, and response inhibition, whichare commonly affected by TBI, were not reflected in the orderingof domain difficulty revealed by the calibrations, because thesecognitive abilities are not assessed by the Cognistat. Althoughcomprehensive neuropsychological profiles cannot be garneredfrom the administration of a screening instrument, this resultsuggests that the likelihood that a specific neurocognitive domainwill be affected remains stable across different stages of injury.Prior research has demonstrated the usefulness of data on theordering of domain difficulty in identifying unexpected patterns ofperformance among community-dwelling individuals with TBI(Doninger et al., 2000).

In summary, the Cognistat identified three strata of performancein both acute inpatient rehabilitation and community-dwellingsamples of adults with TBI, despite the presence of negativelyskewed distributions. Although the data analyzed in this studywere collected between 5 and 9 years ago, improvements inneurotrauma care and intensity and amount of rehabilitation thatmight have ensued since data collection would be unlikely to affectthe current results given that most participants performed well onthis measure. Even in the inpatient rehabilitation sample, themeasure was found to be relatively easy for many patients. Al-though we were able to improve measurement by omitting easy ormisfitting items, the measure may still be relatively insensitive todetecting specific cognitive impairments that may occur after braininjury. The strategy of deleting easier items eliminated the Com-prehension subtest for the inpatient rehabilitation sample.

Although the calibration by domains may provide useful infor-mation in terms of assisting in identifying unusual patterns ofperformance, the person separation indexes from each sampleindicated that the majority of the observed variance in personmeasures was the result of measurement error. Previous investi-gations with geriatric and substance-abusing populations havecautioned against the use of Cognistat subtests to accurately de-lineate specific areas of neuropsychological impairment (Fals-Stewart, 1997; Marcotte et al., 1997). The Cognistat may beequally unsuitable for inpatient as well as outpatient TBI settingswhen the goal is to reliably identify specific cognitive impairmentsand efficiently generate a differentiated profile of neurocognitivestrengths and impairments. This result may, in part, reflect heter-ogeneity with respect to several related variables, includingelapsed time between injury and Cognistat administration, receiptof rehabilitation services, and general pattern of recovery. Theclinical utility of this instrument in identifying more specific areasof cognitive impairment may instead be revealed with futureresearch restricted to more homogeneous subgroups (e.g., severeinjury only, short delay between injury and administration).

The Cognistat was designed in an attempt to accomplish thelaudable and practical goal of quantifying a range of cognitivefunctions in a brief screening instrument. Despite a number of

286 DONINGER ET AL.

This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

studies supporting the utility of the Cognistat with neurologicpatients other than those with TBI (Cammermeyer & Evans, 1988;Cammermeyer & Prendergast, 1997; Fields et al., 1992; Fladby etal., 1999; Northern California Neurobehavioral Group, 1988; Os-mon et al., 1992; Schwamm et al., 1987), it appears that theCognistat fails to accomplish its purported objective of providinga differentiated representation of cognitive status among personswith TBI across a range of severity and settings.

As a primary screening device, the Cognistat is able to reliablydifferentiate multiple strata of performance in both acute andpostacute TBI settings; however, we caution clinicians againstusing this measure as an efficient means of generating a profile ofcognitive strengths and impairments that may influence rehabili-tation and reintegration into the community. Future researchshould focus on developing brief, efficient, and repeatable neuro-psychological instruments that are sensitive to the subtle cognitivedeficits (e.g., memory, reasoning) that are characteristic of indi-viduals with TBI, which influence the design of continuing inter-vention strategies. RSA could serve as an informative method forevaluating the effectiveness of matching instruments with an in-dividual’s ability level, which may be expected to vary with timesince injury and receipt of rehabilitation services. In essence, thismeasurement technique can profitably be used in developing eco-nomical measures that are sensitive to variations in the breadth andmagnitude of residual cognitive deficits across different stages ofrecovery.

References

Arciniegas, D. B., Held, K., & Wagner, P. (2002). Cognitive impairmentfollowing traumatic brain injury. Current Treatment Options in Neurol-ogy, 4, 43–57.

Blostein, P. A., Jones, S. J., Buechler, C. M., & Vandongen, S. (1997).Cognitive screening in mild traumatic brain injuries: Analysis of theNeurobehavioral Cognitive Status Examination when utilized duringinitial trauma hospitalization. Journal of Neurotrauma, 14, 171–177.

Bode, R. K., Heinemann, A. W., & Semik, P. (2000). Measurementproperties of the Galveston Orientation and Amnesia Test (GOAT) andimprovement patterns during inpatient rehabilitation. Journal of HeadTrauma Rehabilitation, 15, 637–655.

Bond, M. R. (1990). Standardized methods of assessing and predictingoutcome. In M. Rosenthal, M. R. Bond, E. R. Griffith, & J. D. Miller(Eds.), Rehabilitation of the adult and child with traumatic brain injury(2nd ed., pp. 59–74). Philadelphia: Davis.

Cammermeyer, M., & Evans, J. E. (1988). A brief neurobehavioral examuseful for early detection of postoperative complications in neurosurgi-cal patients. Journal of Neuroscience Nursing, 20, 314–323.

Cammermeyer, M., & Prendergast, V. (1997). Profiles of cognitive func-tioning in subjects with neurological disorders. Journal of NeuroscienceNursing, 29, 163–169.

Damasio, A. R., & Anderson, S. W. (1993). The frontal lobes. In K.Heilman & E. Valenstein (Eds.), Clinical neuropsychology (3rd ed., pp.409–460). New York: Oxford University Press.

Doninger, N. A., Bode, R. K., Heinemann, A. W., & Ambrose, C. (2000).Rating scale analysis of the Neurobehavioral Cognitive Status Exami-nation. Journal of Head Trauma Rehabilitation, 15, 683–695.

Doninger, N. A., Heinemann, A. W., Bode, R. K., Sokol, K., Corrigan, J.,& Moore, D. (2003). Predicting community integration following trau-matic brain injury with perceived health and cognitive status measures.Rehabilitation Psychology, 48, 67–76.

Drane, D. L., Yuspeh, R. L., Huthwaite, J. S., Klingler, L. K., Foster, L. M.,

Mrazik, M., & Axelrod, B. N. (2003). Healthy older adult performanceon a modified version of the Cognistat (NCSE): Demographic issues andpreliminary normative data. Journal of Clinical and Experimental Neu-ropsychology, 25, 133–144.

Fals-Stewart, W. (1997). Detection of neuropsychological impairmentamong substance-abusing patients: Accuracy of the NeurobehavioralCognitive Screening Examination. Experimental and Clinical Psycho-pharmacology, 5, 269–276.

Fields, S. D., Fulop, G., Sachs, C. J., Strain, J., & Fillit, H. (1992).Usefulness of the Neurobehavioral Cognitive Status Examination in thehospitalized elderly. International Psychogeriatrics, 4, 93–102.

Fisher, W. P. (1992). Reliability statistics. Rasch Measurement Transac-tions, 6, 238.

Fladby, T., Schuster, M., Gronli, O., Sjoholm, H., Loseth, S., & Sexton, H.(1999). Organic brain disease in psychogeriatric patients: Impact ofsymptoms and screening methods on the diagnostic process. Journal ofGeriatric Psychiatry and Neurology, 12, 16–20.

Heinemann, A. W., Corrigan, J. D., & Moore, D. (2004). Case managementfor traumatic brain injury survivors with alcohol problems. Rehabilita-tion Psychology, 49, 156–166.

Kiernan, R. J., Mueller, J., Langston, J. W., & Van Dyke, C. (1987). TheNeurobehavioral Cognitive Status Examination: A brief but differenti-ated approach to cognitive assessment. Annals of Internal Medicine, 107,481–485.

Levin, H. S. (1989). Memory deficit after closed head injury. Journal ofClinical and Experimental Neuropsychology, 12, 129–153.

Lezak, M. D., Howieson, D. B., & Loring, D. W. (2004). Neuropsycho-logical assessment (4th ed.). New York: Oxford University Press.

Linacre, J. M., & Wright, B. D. (1999). WINSTEPS for PC compatibles.Chicago: Mesa Press.

Logue, P. E., Tupler, L. A., D’Amico, C., & Schmitt, F. A. (1993). TheNeurobehavioral Cognitive Status Examination: Psychometric proper-ties in use with psychiatric inpatients. Journal of Clinical Psychology,49, 80–89.

Marcotte, T. D., van Gorp, W., Hinkin, C. H., & Osato, S. (1997).Concurrent validity of the Neurobehavioral Cognitive Status Examsubtests. Journal of Clinical and Experimental Neuropsychology, 19,386–395.

Mitrushina, M., Abara, J., & Blumenfeld, A. (1994). Aspects of validityand reliability of the Neurobehavioral Cognitive Status Examination(NCSE) in assessment of psychiatric patients. Journal of PsychiatricResearch, 28, 85–95.

Nabors, N. A., Millis, S. R., & Rosenthal, M. (1997). Use of the Neurobe-havioral Cognitive Status Examination (Cognistat) in traumatic braininjury. Journal of Head Trauma Rehabilitation, 12, 79–84.

Northern California Neurobehavioral Group. (1988). Manual for the Neu-robehavioral Cognitive Status Examination. Fairfax, CA: Author.

Oehlert, M. E., Hass, S. D., Freeman, M. R., Williams, M. D., Ryan, J. J.,& Sumerall, S. W. (1997). The Neurobehavioral Cognitive Status Ex-amination: Accuracy of the “screen-metric” approach in a clinical sam-ple. Journal of Clinical Psychology, 53, 733–737.

Osmon, D. C., Smet, I. C., Winegarden, B., & Gandhavadi, B. (1992).Neurobehavioral Cognitive Status Examination: Its use with unilateralstroke patients in a rehabilitation setting. Archives of Physical Medicine& Rehabilitation, 73, 414–418.

Poon, W. S., Zhu, X. L., Ng, S. C., & Wong, G. K. (2005). Predicting oneyear clinical outcome in traumatic brain injury (TBI) at the beginning ofrehabilitation. Acta Neurochirurgica Supplement, 93, 207–208.

Rasch, G. (1980). Probabilistic models for some intelligence and attain-ment tests. Chicago: University of Chicago Press.

Schwamm, L. H., Van Dyke, C., Kiernan, R. J., Merrin, E. L., & Mueller,J. (1987). The Neurobehavioral Cognitive Status Examination: Compar-ison with the Cognitive Capacity Screening Examination and the Mini-

287MEASUREMENT PROPERTIES OF THE COGNISTAT

This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

Mental State Examination in a neurosurgical population. Annals ofInternal Medicine, 107, 486–491.

Tabachnick, B. G., & Fidell, L. S. (1996). Using multivariate statistics (3rded.). New York: Harper Collins.

Tate, R. L., Fenelon, B., Manning, M. L., & Hunter, M. (1991). Patterns ofneuropsychological impairment after severe blunt head injury. Journalof Nervous and Mental Disease, 179, 117–126.

Teasdale, G., & Jennett, B. (1974). Assessment of coma and impairedconsciousness. Lancet, 2, 81–84.

Van Zomeren, A. H., & Van Denburg, W. (1985). Residual complaints ofpatients two years after severe head injury. Journal of Neurology,Neurosurgery, and Psychiatry, 48, 21–28.

Wallace, J. J., Caroselli, J. S., Scheibel, R. S., & High, W. M. (2000).Predictive validity of the Neurobehavioral Cognitive Status Examination(NCSE) in a post-acute rehabilitation setting. Brain Injury, 14, 63–69.

Wright, B. D., & Masters, G. (1982). Rating scale analysis: Rasch mea-surement. Chicago: Mesa Press.

Wright, B. D., & Masters, G. (2002). Number of person or item strata.Rasch Measurement Transactions, 16, 888.

Received June 10, 2005Revision received October 31, 2005

Accepted November 21, 2005 �

MAIL TO:Name

Address

City State Zip

APA Member #

Check enclosed (make payable to APA)

Charge my: VISA MasterCard AmericanExpress

Cardholder Name

etaD .pxE.oN draC

Signature (Required for Charge)

BILLING ADDRESS:Street

City State Zip

Daytime Phone

E-mail

ORDER FORMStart my 2007 subscription to RehabilitationPsychology! ISSN: 0090-5550

___ $55.00, APA MEMBER/AFFILIATE __________ $80.00, INDIVIDUAL NONMEMBER __________ $295.00, INSTITUTION ___________

In DC add 5.75% / In MD add 5% sales tax __________

TOTAL AMOUNT ENCLOSED $ _______

Subscription orders must be prepaid. (Subscriptions are ona calendar year basis only.) Allow 4-6 weeks for delivery ofthe first issue. Call for international subscription rates.

SEND THIS ORDER FORM TO:American Psychological AssociationSubscriptions750 First Street, NEWashington, DC 20002-4242

Or call 800-374-2721, fax 202-336-5568.TDD/TTY 202-336-6123.For subscription information, e-mail:subscriptions@apa.org REPA17

288 DONINGER ET AL.

This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.