Missing Data Analysis: Making It Work in the Real World · Annu. Rev. Psychol. 2009.60:549-576. Downloaded from arjournals.annualreviews.org by Pennsylvania State University on 08/18/10

ANRV364-PS60-21 ARI 27 October 2008 16:22

Missing Data Analysis:Making It Workin the Real WorldJohn W. GrahamDepartment of Biobehavioral Health and the Prevention Research Center, The PennsylvaniaState University, University Park, Pennsylvania 16802; email: [email protected]

Annu. Rev. Psychol. 2009. 60:549–576

First published online as a Review in Advance onJuly 24, 2008

The Annual Review of Psychology is online atpsych.annualreviews.org

This article’s doi:10.1146/annurev.psych.58.110405.085530

Copyright c© 2009 by Annual Reviews.All rights reserved

0066-4308/09/0110-0549$20.00

Key Words

multiple imputation, maximum likelihood, attrition, nonignorablemissingness, planned missingness

AbstractThis review presents a practical summary of the missing data literature,including a sketch of missing data theory and descriptions of normal-model multiple imputation (MI) and maximum likelihood methods.Practical missing data analysis issues are discussed, most notably theinclusion of auxiliary variables for improving power and reducing bias.Solutions are given for missing data challenges such as handling longitu-dinal, categorical, and clustered data with normal-model MI; includinginteractions in the missing data model; and handling large numbers ofvariables. The discussion of attrition and nonignorable missingness em-phasizes the need for longitudinal diagnostics and for reducing the un-certainty about the missing data mechanism under attrition. Strategiessuggested for reducing attrition bias include using auxiliary variables,collecting follow-up data on a sample of those initially missing, andcollecting data on intent to drop out. Suggestions are given for movingforward with research on missing data and attrition.

549

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


Contents

INTRODUCTIONAND OVERVIEW . . . . . . . . . . . . . . . . 550Goals of This Review . . . . . . . . . . . . . . 551What’s to Come . . . . . . . . . . . . . . . . . . . 551

MISSING DATA THEORY . . . . . . . . . . 552Causes or Mechanisms

of Missingness . . . . . . . . . . . . . . . . . . 552Old Analyses: A Brief Summary . . . . . 553“Modern” Missing Data

Analysis Methods . . . . . . . . . . . . . . . 555Dispelling Myths About MAR

Missing Data Methods . . . . . . . . . . 559PRACTICAL ISSUES: MAKING

MISSING DATA ANALYSISWORK IN THEREAL WORLD . . . . . . . . . . . . . . . . . . . 560Inclusive versus Restrictive Variable

Inclusion Strategies(MI versus FIML). . . . . . . . . . . . . . . 560

Small Sample Sizes . . . . . . . . . . . . . . . . . 560Rounding. . . . . . . . . . . . . . . . . . . . . . . . . . 561Number of Imputations

in MI. . . . . . . . . . . . . . . . . . . . . . . . . . . 561Making EM (and MI) Perform

Better (i.e., Faster) . . . . . . . . . . . . . . 561Including Interactions in the Missing

Data Model. . . . . . . . . . . . . . . . . . . . . 562

Longitudinal Data and SpecialLongitudinal MissingData Models . . . . . . . . . . . . . . . . . . . . 562

Categorical Missing Dataand Normal-Model MI. . . . . . . . . . 562

Normal-Model MIwith Clustered Data . . . . . . . . . . . . . 564

Large Numbers of Variables . . . . . . . . 564Practicalities of Measurement:

Planned Missing Data Designs . . 565ATTRITION AND MNAR

MISSINGNESS . . . . . . . . . . . . . . . . . . . 566Some Clarifications About Missing

Data Mechanisms . . . . . . . . . . . . . . . 567Measuring the Biasing Effects

of Attrition . . . . . . . . . . . . . . . . . . . . . 568Missing Data Diagnostics . . . . . . . . . . 569Nonignorable (MNAR) Methods . . . 570Strategies for Reducing the Biasing

Effects of Attrition . . . . . . . . . . . . . . 570Suggestions for Reporting About

Attrition in an Empirical Study . . 571Suggestions for Conduct

and Reporting of SimulationStudies on Attrition . . . . . . . . . . . . . 571

More Research is Needed . . . . . . . . . . 572SUMMARY AND CONCLUSIONS. . 573

INTRODUCTIONAND OVERVIEW

Missing data have challenged researchers sincethe beginnings of field research. The chal-lenge has been particularly acute for longitudi-nal research, that is, research involving multiplewaves of measurement on the same individu-als. The main issue is that the analytic pro-cedures researchers use, many of which weredeveloped early in the twentieth century, weredesigned to have complete data. Until relativelyrecently, there was simply no mechanism forhandling the responses that were sometimesmissing within a particular survey, or whole sur-

veys that were missing for some waves of a mul-tiwave measurement project.

Problems brought about by missing data be-gan to be addressed in an important way start-ing in 1987, although a few highly influentialarticles did appear before then (e.g., Dempsteret al. 1977, Heckman 1979, Rubin 1976). Whathappened in 1987 was nothing short of a rev-olution in thinking about analysis of missingdata. The revolution began with two majorbooks that were published that year. Little &Rubin (1987) published their classic book, Sta-tistical Analysis with Missing Data (the secondedition was published in 2002). Also, Rubin

550 Graham

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


(1987) published his book, Multiple Imputationfor Nonresponse in Surveys. These two books,coupled with the advent of powerful personalcomputing, would lay the groundwork for miss-ing data software to be developed over the next20 years and beyond. Also published in 1987were two articles describing the first truly acces-sible method for dealing with missing data usingexisting structural equation modeling (SEM)software (Allison 1987, Muthen et al. 1987). Fi-nally, Tanner & Wong (1987) published theirarticle on data augmentation, which would be-come a cornerstone of the multiple imputation(MI) software that would be developed a decadelater.

Goals of This Review

A major goal of this review is to present ideasand strategies that will make missing data anal-yses useful to researchers. My aim here is toencourage researchers to use the missing dataprocedures that are already known to be goodones. Efforts toward this goal involve summa-rizing the major research in missing data anal-ysis over the past several years. However, muchof the reluctance to adopt these procedures isrelated to the myths and misconceptions thatcontinue to abound about the impact of missingdata with and without using these procedures.Thus, a goal of this review is to clear up manyof the myths and misconceptions surroundingmissing data and analysis with missing data.

Work is required to become a practiceduser of the acceptable (i.e., MI and maximum-likelihood, or ML) procedures. But that workwould be a lot less onerous if one had con-fidence that learning these procedures wouldtruly make one’s work better and that criticismssurrounding missing data would be materiallyreduced.

Researchers should use MI and ML proce-dures (see Schafer & Graham 2002). They aregood procedures that are based on strong statis-tical traditions. They can certainly be improvedon, but by how much? I would argue that usingMI and ML procedures gets us at least 90% ofthe way to the hypothetical ideal from where

we were 25 years ago. Newer procedures willcontinually fine-tune the existing MI and MLprocedures, but the main missing data solutionsare already available and should be used now.

Above all, my goal is that this review willbe of practical value. I hope that my words willfacilitate the use of MI and ML missing datamethods. This is not intended to be a thoroughreview of all work and methods relating to miss-ing data. I have focused on what I believe to bemost useful.

What’s to Come

In the following sections, I discuss three majormissing data topics: missing data theory, anal-ysis in practice, and attrition and missingnessthat is not missing at random. In the first ma-jor section, I lay out the main tenets of whatI refer to as “missing data theory.” One centralfocus in this section is the causes or mechanismsof missingness. In this section, I discuss what Irefer to as the “old” methods for dealing withmissing data, but as much as possible, my dis-cussion is limited to methods that remain use-ful at least in some circumstances. This sectionbriefly presents the methods I fully endorse: MIand ML.

In the second major section, I focus on thepractical side of performing missing data anal-yses. Over the years, I have faced all of theseproblems as a data analyst; these are real solu-tions. Sometimes the solutions are a bit ad hoc.Better solutions may become available in thefuture, but the solutions I present are knownto have minimal harmful impact on statisticalinference, and they will keep you doing analy-sis, which is the most important thing. In thissection, I also touch on the developing area ofplanned missingness designs, an area that opensup new design possibilities for researchers whoare already making use of the recommended MIand ML missing data procedures. Contrary tothe old adage that the best solution to miss-ing data is not to have them, there are timeswhen building missing data into the overallmeasurement design is the best use of limitedresources.

www.annualreviews.org • Missing Data Analysis 551

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


In the final major section, I describe the areaof attrition and missingness that is not missingat random. This kind of missingness has provento be a major obstacle, especially in longitudi-nal and intervention research. A good bit of theproblem in this area stems from the fact that theframework for thinking about these issues wasdeveloped and solidified well before the miss-ing data revolution. In this section, I proposea different framework for thinking about attri-tion and make several suggestions (pleas?) as tohow researchers might proceed in this area.

MISSING DATA THEORY

Causes or Mechanisms of Missingness

Statisticians talk about missingness mecha-nisms. But what they mean by that term differsfrom what social and behavioral scientists thinkof as mechanisms. When I (trained as an ex-perimental social psychologist) use that word, Ithink of causal mechanisms. What is the reasonthe data are missing? Statisticians, on the otherhand, often are thinking more along the lines ofa description of the missingness. For example,it is not uncommon to talk about a vector R foreach variable, which takes on the value “1” ifthe variable has data for that case, and “0” if thevalue is missing for that case. This leads natu-rally to descriptions of the missing data, that is,patterns of missingness. For example, supposethat one has three variables (X, Y1, and Y2), andsuppose that X is never missing but Y1 is miss-ing for some individuals, and Y2 is missing for afew more. Or, thinking about it the other way,suppose one has data for all three variables forsome number of cases, but partial data (X andY1) for some number of cases and partial data (Xonly) for some other number of cases. The pat-terns of missingness for a hypothetical N = 100cases might look like those shown in Table 1.Also, as shown in Table 1, it not uncommonfor a small number of cases to be present at onewave, missing at a later wave, and then give dataat a still later wave.

When people talk about the mechanismsof missingness, three terms come up: miss-

Table 1 Hypothetical patterns of missingness

Variable

X Y1 Y2 N1 1 1 651 1 0 201 0 0 101 0 1 5

1 = value present; 0 = value missing.

ing completely at random (MCAR), missingat random (MAR), and missing not at random(MNAR). Although statisticians prefer not touse the word “cause,” they do often use thewords “due to” or “depends on” in this context.

With MAR, the missingness (i.e., whetherthe data are missing or not) may depend onobserved data, but not on unobserved data(Schafer & Graham 202).

MCAR is a special case of MAR in whichmissingness does not depend on the observeddata either (Schafer & Graham 2002).

With MNAR, missingness does depend onunobserved data.

More about MAR, MCAR, and MNAR. Al-though these three important terms do havespecific statistical definitions, their practicalmeaning is often elusive. MCAR is perhaps theeasiest to understand. If the cases for which thedata are missing can be thought of as a randomsample of all the cases, then the missingness isMCAR. This means that everything one mightwant to know about the data set as a whole canbe estimated from any of the missing data pat-terns, including the pattern in which data existfor all variables, that is, for complete cases (e.g.,see the top row in Table 1).

Another aspect of MCAR that is particularlyeasy to understand for psychologists is that theword “random” in MCAR means what psychol-ogists generally think of when they use the term.The word “random” in MAR, however, meanssomething rather different from what psychol-ogists typically think of as random. In fact,the randomness in MAR missingness meansthat once one has conditioned on (e.g., con-trolled for) all the data one has, any remaining

552 Graham

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


missingness is completely random (i.e., it doesnot depend on some unobserved variable). Be-cause of this, I often say that a more preciseterm for missing at random would be condi-tionally missing at random. However, if such aterm were in common use, its acronym (CMAR)would often be confused with MCAR. Thus, myfeeling about this is that psychologists shouldcontinue to refer to MCAR and MAR but sim-ply understand that the latter term refers to con-ditionally missing at random.

Another distinction that is often used withmissingness is the distinction between ignor-able and nonignorable missingness. Withoutgoing into detail here, suffice it to say thatignorable missingness applies to MCAR andMAR, whereas nonignorable missingness is of-ten used synonymously with MNAR.

An important wrinkle with these terms, es-pecially MAR and MNAR (or ignorable andnonignorable), is that they do not apply just tothe data. Rather they apply jointly to the dataand to the analysis that is being used. For exam-ple, suppose one develops a smoking preventionintervention and has a treatment group and acontrol group (represented by a dummy vari-able called Program). Suppose that one mea-sures smoking status at time 2, one year afterimplementation of the prevention intervention(Smoking2). Finally, suppose that some peoplehave missing data for Smoking2, and that miss-ingness on Smoking2 depends on smoking sta-tus measured at time 1, just before the programimplementation (Smoking1). If one includesSmoking1 in one of the acceptable missing dataprocedures (MI or ML), then the missingnesson Smoking2 is conditioned on Smoking1 andis thus MAR (note that even with complete caseanalysis, the regression analysis of Program pre-dicting Smoking2 is MAR as long as Smoking1

is also included in the model; e.g., see Graham& Donaldson 1993). However, if the re-searcher tested a model in which the Programalone predicted Smoking2, then the missingnesswould become MNAR because the researcherfailed to condition on Smoking1, the cause ofmissingness.

Many researchers have suggested modify-ing the names of the missing data mechanismsin order to have labels that are a bit closer toregular language usage. However, missing datatheorists believe that these mechanism namesshould remain as is. I agree. I now believe wewould be doing psychologists a disservice ifwe encouraged them to abandon these terms,which are so well entrenched in the statisticsliterature. Rather, we should continue to usethese terms (MCAR, MAR, and MNAR), butalways define them very carefully using regularlanguage.

Consequences of MCAR, MAR, andMNAR. The main consequence of MCARmissingness is loss of statistical power. Thegood thing about MCAR is that analyses yieldunbiased parameter estimates (i.e., estimatesthat are close to population values). MARmissingness (i.e., when the cause of missing-ness is taken into account) also yields unbiasedparameter estimates. The reason MNARmissingness is considered a problem is that ityields biased parameter estimates (discussed atlength below).

Old Analyses: A Brief Summary

This summary is not intended to be a thoroughexamination of the “old” approaches for dealingwith missing data. Rather, in order to be of mostpractical value, the discussion below focuses onthe old approaches that can still be useful, atleast under some circumstances.

Yardsticks for evaluating methods. I havejudged the various methods (old and new) bythree means. First, the method should yield un-biased parameter estimates over a wide rangeof parameters. That is, the parameter estimateshould be close to the population value forthat parameter. Some of the methods I wouldjudge to be unacceptable (e.g., mean substitu-tion) may yield a mean for a particular variablethat is close to the true parameter value (e.g.,under MCAR), but other parameters using this


Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


method can be seriously biased. Second, thereshould be a method for assessing the degree ofuncertainty about parameter estimates. That is,one should be able to obtain reasonable esti-mates of the standard error or confidence inter-vals. Third, once bias and standard errors havebeen dealt with, the method should have goodstatistical power.

Complete cases analysis (AKA, listwisedeletion). This approach can be very usefuleven today. One concern about listwise deletionis that it may yield biased parameter estimates(e.g., see Wothke 2000). For example, groupswith complete data, especially in a longitudi-nal study, often are quite different from thosethat have missing data. Nevertheless, the dif-ference in those two groups often is embodiedcompletely in the pretest variables (for whicheveryone has data). Thus, as long as those vari-ables can reasonably be included in the model ascovariates, the bias is often minimal, even withlistwise deletion, especially for multiple regres-sion models (e.g., see Graham & Donaldson1993).

However, there will always be some loss ofpower with listwise deletion because of the un-used partial data. And in some instances, thisloss of power can be huge, making this methodan undesirable option. Still, if the loss of casesdue to missing data is small (e.g., less than about5%), biases and loss of power are both likelyto be inconsequential. I, personally, would stilluse one of the missing data approaches evenwith just 5% missing cases, and I encourage you

POSITIVE DEFINITE

One good way to think of a matrix that is not positive definiteis that the matrix contains less information than implied by thenumber of variables in the matrix. For example, if a matrix con-tained the correlations of three variables (A, B, and C) and theirsum, then there would be just three variables worth of informa-tion, even though it contained four variables. Because the sum isperfectly predicted by the three variables, it adds no new infor-mation, and the matrix would not be positive definite.

to get used to doing the same. However, if aresearcher chose to stay with listwise deletionunder these special circumstances, I believe itwould be unreasonable for a critic to argue thatit was a bad idea to do so. It is also importantthat standard errors based on listwise deletionare meaningful.

Pairwise deletion. Pairwise deletion is usuallyused in conjunction with a correlation matrix.Each correlation is estimated based on the caseshaving data for both variables. The issue withpairwise deletion is that different correlations(and variance estimates) are based on differ-ent subsets of cases. Because of this, it is possi-ble that parameter estimates based on pairwisedeletion will be biased. However, in my experi-ence, these biases tend to be small in empiricaldata. On the other hand, because different cor-relations are based on different subsets of cases,there is no guarantee that the matrix will bepositive definite (see sidebar Positive Definite).Nonpositive definite matrices cannot be usedfor most multivariate statistical analyses. A big-ger concern with pairwise deletion is that thereis no basis for estimating standard errors.

Because of all these problems, I cannot rec-ommend pairwise deletion as a general solu-tion. However, I do still use pairwise deletionin one specific instance. When I am conduct-ing preliminary exploratory factor analysis witha large number of variables, and publicationof the factor analysis results, per se, is not mygoal, I sometimes find it useful to conduct thisanalysis with pairwise deletion. As a prelimi-nary analysis for conducting missing data analy-sis, I sometimes examine the preliminary eigen-values from principal components analysis (seesidebar Eigenvalue). If the last eigenvalue ispositive, then the matrix is positive definite.Many failures of the expectation-maximization(better known as EM) algorithm (and MI) aredue to the correlation matrix not being positivedefinite.

Other “old” methods. Other old methods in-clude mean substitution, which I do not recom-mend. In the “Modern” Missing Data Analysis

554 Graham

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


Methods section, I describe a method based onthe EM algorithm that is much preferred overmean substitution (see the sections on GoodUses of the EM Algorithm and Imputing a Sin-gle Data Set from EM Parameters). A secondmethod that has been described in the litera-ture involves using a missingness dummy vari-able in addition to the specially coded miss-ing value. This approach has been discreditedand should not be used (e.g., see Allison 2002).Finally, regression-based single imputation hasbeen employed in the past. Although the con-cept is a sound one and is the basis for manyof the modern procedures, this method is notrecommended in general.

“Modern” Missing DataAnalysis Methods

The “modern” missing data procedures I con-sider here are (a) the EM algorithm, (b) multipleimputation under the normal model, and (c) MLmethods, often referred to as full-informationmaximum likelihood (FIML) methods.

EM algorithm. It is actually a misnomer to re-fer to this as “the” EM algorithm because thereare different EM algorithms for different ap-plications. Each version of the EM algorithmreads in the raw data and reads out a differentproduct, depending on the application. I de-scribe here the EM algorithm that reads in theraw data, with missing values, and reads out anML variance-covariance matrix and vector ofmeans. Definitive technical treatments of vari-ous EM algorithms are given in Little & Rubin(1987, 2002) and Schafer (1997). Graham andcolleagues provide less-technical descriptionsof the workings of the EM algorithm for co-variance matrices (Graham & Donaldson 1993;Graham et al. 1994, 1996, 1997, 2003).

In brief, the EM algorithm is an iterativeprocedure that produces maximum likelihoodestimates. For the E-step at one iteration, casesare read in, one by one. If a value is present,the sums, sums of squares, and sums of cross-products are incremented. If the value is miss-ing, the current best guess for that value is usedinstead. The best guess is based on regression-

EIGENVALUE

Eigenvalues are part of the decomposition of a correlation ma-trix during factor analysis or principal components analysis. Eacheigenvalue represents the variance of the linear combination ofitems making up that factor. In principal components, the totalvariance for the correlation matrix is the number of items in thematrix. If the matrix is positive definite, the last eigenvalue willbe positive, that is, it will have variance. However, if the matrixis not positive definite, one or more of the eigenvalues will be 0,implying that those factors have no variance, or that they add nonew information over and above the other factors.

based single imputation with all other variablesin the model used as predictors. For the sums,the best guess value is used as is. For sumsof squares and sums of cross-products, if justone value is missing, then the quantity is in-cremented directly. However, if both values aremissing, then the quantity is incremented, anda correction factor is added. This correctionis conceptually equivalent to adding a randomresidual error term in MI (described below).

In the M-step of the same iteration, the pa-rameters (variances, covariances, and means)are estimated (calculated) based on the currentvalues of the sums, sums of squares, and sumsof cross-products. Based on the covariance ma-trix at this iteration, new regression equationsare calculated for each variable predicted byall others. These regression equations are thenused to update the best guess for missing valuesduring the E-step of the next iteration. Thistwo-step process continues until the elementsof the covariance matrix stop changing. Whenthe changes from iteration to iteration are sosmall that they are judged to be trivial, EM issaid to have converged.

Being ML, the parameter estimates (means,variances, and covariances) from the EM algo-rithm are excellent. However, the EM algo-rithm does not provide standard errors as anautomatic part of the process. One could ob-tain an estimate of these standard errors usingbootstrap procedures (e.g., see Graham et al.1997). Although bootstrap procedures (Efron


Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


1982) are often criticized, they can be quite use-ful in this context as a means of dealing withnonnormal data distributions.

Good uses of the EM algorithm. Althoughthe EM algorithm provides excellent parame-ter estimates, the lack of convenient standarderrors means that EM is not particularly goodfor hypothesis testing. On the other hand, sev-eral important analyses, often preliminary anal-yses, don’t use standard errors anyway, so theEM estimates are very useful. First, it is of-ten desirable to report means, standard devi-ations, and sometimes a correlation matrix inone’s paper. I would argue that the best esti-mates for these quantities are the ML estimatesprovided by EM. Second, data quality analy-ses, for example, coefficient alpha analyses, be-cause they typically do not involve standard er-rors, can easily be based on the EM covariancematrix (e.g., see Enders 2003; Graham et al.2002, 2003). The EM covariance matrix is alsoan excellent basis for exploratory factor analysiswith missing data. This is especially easy withthe SAS/STAT® software program (SAS Insti-tute); one simply includes the relevant variablesin Proc MI, asking for the EM matrix to be out-put. That matrix may then be used as input forProc Factor using the “type = cov” option.

Although direct analysis of the EM covari-ance matrix can be useful, a more widely usefulEM tool is to impute a single data set from EMparameters (with random error). This proce-dure has been described in detail in Grahamet al. (2003). This single imputed data set isknown to yield good parameter estimates, closeto the population average. But more impor-tantly, because it is a complete data set, it may beread in using virtually any software, includingSPSS. Once read into the software, coefficientalpha and exploratory factor analyses may becarried out in the usual way. One caution is thatthis data set should not be used for hypothe-sis testing. Standard errors based on this dataset, say from a multiple regression analysis, willbe too small, sometimes to a substantial extent.Hypothesis testing should be carried out withMI or one of the FIML procedures. Note that

the procedure in SPSS for writing out a singleimputed data set based on the EM algorithm isnot recommended unless random error resid-uals are added after the fact to each imputedvalue; the current implementation of SPSS, upto version 16 at least, writes data out withoutadding error (e.g., see von Hippel 2004). Thisis known to produce important biases in the dataset (Graham et al. 1996).

Implementations of the EM algorithm.Good implementations of the EM algorithmfor covariance matrices are widely available.SAS Proc MI estimates the EM covariance ma-trix as a by-product of its MI analysis (SASInstitute). Schafer’s (1997) NORM program, astand-alone Microsoft Windows program, alsoestimates the EM covariance matrix as a step inthe MI process. Graham et al. (2003) have de-scribed utilities for making use of that covari-ance matrix. Graham & Hofer (1992) have cre-ated a stand-alone DOS-based EM algorithm,EMCOV, which can be useful in simulations.

Multiple imputation under the normalmodel. I describe in this section MI under thenormal model as it is implemented in Schafer’s(1997) NORM program. MI as implementedin SAS Proc MI is also based on Schafer’s(1997) algorithms, and thus is the same kindof program as NORM. Detailed, step-by-stepinstructions for running NORM are availablein Graham et al. (2003; also see Graham &Hofer 2000, Schafer 1999, Schafer & Olsen1998). Note that Schafer’s NORM program isalso available as part of the Splus missing datalibrary (http://www.insightful.com/).

The key to any MI program is to restore theerror variance lost from regression-based singleimputation. Imputed values from single impu-tation always lie right on the regression line.But real data always deviate from the regres-sion line by some amount. In order to restorethis lost variance, the first part of imputation isto add random error variance (random normalerror in this case). The second part of restor-ing lost variance relates to the fact that eachimputed value is based on a single regression

556 Graham

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


equation, because the regression equation, andthe underlying covariance matrix, is based on asingle draw from the population of interest.

In order to adjust the lost error completely,one should obtain multiple random draws fromthe population and impute multiple times, eachwith a different random draw from the popu-lation. Of course, this is almost never possible;researchers have just a single sample. One op-tion might be to simulate random draws fromthe population by using bootstrap procedures(Efron 1982). Another approach is to simulaterandom draws from the population using dataaugmentation (DA; Tanner & Wong 1987).

The key to Schafer’s NORM program is DA.NORM first runs EM to obtain starting val-ues for DA. DA can be thought of as a kindof stochastic (probabilistic) version of EM. It,too, is an iterative, two-step process. There isan imputation step during which DA simulatesthe missing data based on the current parame-ter estimates, and a posterior step during whichDA simulates the parameters given the current(imputed) data. DA is a member of the Markov-Chain Monte Carlo family of algorithms. It isMarkov-like in the sense that all of the infor-mation from one step of DA is contained in theprevious step. Because of this, the parameter es-timates and imputed values from two adjacentsteps of DA are more similar than one wouldexpect from two random draws from the popu-lation. However, after, say, 50 steps of DA, theparameter estimates and imputed values fromthe initial step and those 50 steps removed aremuch more like two random draws from thepopulation. The trick is to determine how manysteps are required before the two imputed datasets are sufficiently similar to two random drawsfrom the population. Detailed guidance for thisprocess is given in Graham et al. (2003). In gen-eral, the number of iterations it takes EM toconverge is an excellent estimate of the num-ber of steps there should be between imputeddata sets from DA (this rule applies best to theNORM program; different MI programs havedifferent convergence criteria, and this rule maybe slightly different with those MI programs).In addition, diagnostics are available in NORM

and Proc MI to verify that the number of DAsteps selected was good enough.

Implementations of MI under the normalmodel. Implementations of MI under the nor-mal model are also widely available. Schafer’s(1997) NORM software is a free program(see http://methodology.psu.edu/ for the freedownload). SAS Proc MI (especially version 9,but to a large extent version 8.2; SAS Insti-tute) provides essentially the same features asNORM. For analyses conducted in SAS, ProcMI is best. Other implementations of MI arenot guaranteed to be as robust as are those basedon DA or other Markov-Chain Monte Carloroutines, although such programs may be use-ful under specific circumstances. For example,Amelia II (see Honaker et al. 2007, King et al.2001) and IVEware (Raghunathan 2004) aretwo MI programs that merit a look. See Horton& Kleinman (2007) for a recent review of MIsoftware.

Special MI software for categorical, longi-tudinal/cluster, and semi-continuous data.In this category are Schafer’s (1997) CAT pro-gram (for categorical data) and MIX program(for mixed continuous and categorical prob-lems). Both of these are available (along withNORM) as special commands in the latest ver-sion of Splus. Although CAT can certainly beused to handle imputation with categorical data,it presents the user with some limitations. Mostimportantly, the default in CAT involves whatamounts to the main effects and all possible in-teractions. Thus, even with a few variables, thedefault model can involve a huge number of pa-rameters. For example, with just five input vari-ables, CAT estimates parameters for 31 vari-ables (five main effects, ten 2-way interactions,ten 3-way interactions, five 4-way interactions,and one 5-way interaction).

Also included in this category is the PANprogram (for special panel and cluster-data de-signs, see Schafer 2001, Schafer & Yucel 2002).PAN was created for the situation in which avariable, Posatt (beliefs about the positive socialconsequences of alcohol use), was measured in


Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


grades 5, 6, 7, 9, and 10 of a longitudinal study,but was omitted for all subjects in grade 8. Othervariables (e.g., alcohol use), however, were mea-sured in all six grades. Because no subject haddata for Posatt measured in eighth grade, MIunder the normal model could not be used toimpute that variable. However, because PANalso takes into account growth (change) overtime, Posatt 8 can be imputed with PAN. PANis also very good for imputing clustered data(e.g., students within schools) where the num-ber of clusters is large. Although the potentialfor the PAN program is huge, its availabilityremains limited.

Olsen & Schafer (2001) have described mul-tiple imputation for semi-continuous data, es-pecially in the growth modeling context. Semi-continuous data come from variables that havemany responses at one value (e.g., 0) and aremore or less normally distributed for valuesgreater than 0.

Imputing a single data set from EMparameters. An often useful alternative to an-alyzing the EM covariance matrix directly is toimpute a single data set based on EM param-eters (+random error, an option available inSchafer’s NORM program; see Graham et al.2003 for details). With data sets imputed usingdata augmentation, parameter estimates can beanywhere in legitimate parameter space. How-ever, when the single imputation (+error) isbased on the EM covariance matrix, all param-eter estimates are near the center of the param-eter space. For this reason, if one analyzes justone imputed data set, it should be this one. Thisdata set is very useful for analyses that do not re-quire hypothesis testing, such as coefficient al-pha analysis and exploratory factor analysis (seeGraham et al. 2003 for additional details).

FIML methods. FIML methods deal with themissing data, do parameter estimation, and es-timate standard errors all in a single step. Thismeans that the regular, complete-cases algo-rithms must be completely rewritten to han-dle missing data. Because this task is somewhatdaunting, software written with the FIML miss-

ing data feature is limited. At present, the fea-ture is most common in SEM software (in al-phabetical order, Amos: Arbuckle & Wothke1999; LISREL: Joreskog & Sorbom 1996, alsosee du Toit & du Toit 2001; Mplus: Muthen,& Muthen 2007; and Mx: Neale et al. 1999).Although each of these programs was writtenspecifically for SEM applications, they can beused for virtually any analysis that falls withinthe general linear model, most notably multipleregression. For a review of FIML SEM meth-ods, see Enders (2001a).

Other FIML (or largely FIML) software forlatent class analysis includes Proc LTA (e.g.,Lanza et al. 2005; also see http://methodology.psu.edu) and Mplus (Muthen & Muthen 2007).

Other “older” methods. One other methoddeserves special mention in this context. Al-though SEM analysis with missing data is cur-rently handled almost exclusively by SEM/FIML methods (see previous section), an oldermethod involving the multiple group capabili-ties of SEM programs is very useful for someapplications. This approach was described ini-tially by Allison (1987) and Muthen et al. (1987).Among other things, this method has provento be extremely useful with simulations involv-ing missing data (e.g., see Graham et al. 2001,2006).

This method also continues to be useful formeasurement designs described as “acceleratedlongitudinal” or “cohort sequential” (seeDuncan & Duncan 1994; Duncan et al. 1994,1996; McArdle 1994; McArdle & Hamagami1991, 1992). With these designs, one collectsdata for two or more sets of participants ofdifferent ages over, say, three consecutive years.For example, one group is 10, 11, and 12 yearsold over the three years of a study, and anothergroup is 11, 12, and 13 years old over the samethree study years. Because no participantshave data for both ages 10 and 13, regularFIML-based SEM software and normal-modelMI cannot be used in this context. However,the multiple-group SEM approach may beused to test a growth model covering growthover all four ages.

558 Graham

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


Dispelling Myths About MAR MissingData Methods

Myths abound regarding missing data and anal-ysis with missing data. Many of these mythsoriginated with thinking that was developedwell before the missing data revolution. Parts ofthat earlier thinking, of course, remain an im-portant element of modern psychological sci-ence. But the parts relating to missing data needto be revised. I address three of the most com-mon myths in this section. Other myths aredealt with in the sections that follow.

Imputation is making up the data. It is truethat imputation is the process of plugging inplausible values where none exist. But the pointof this process is not to obtain the individualvalues themselves. Rather, the point is to plugin these values (multiple times) in order to pre-serve important characteristics of the data set asa whole. By “preserve,” I mean that parameterestimates should be unbiased. That is, the es-timated mean, for example, should be close tothe true population value for the mean; the es-timated variance should be close to the truepopulation value for the variance. In this re-view, I talk mainly about multiple imputationunder the normal model. Normal-model MI“preserves” means, variances, covariances, cor-relations, and linear regression coefficients.

You are unfairly helping yourself by imput-ing (AKA, it is okay to impute the indepen-dent variable, but not the dependent vari-able). There are several versions of this myth.In the past, some researchers were convincedthat imputation procedures such as normal-model MI were fine for imputing missing datathat might occur within the set of independentvariables (IVs) (and covariates) of a study. How-ever, these researchers were very reluctant toinclude the dependent variable (DV) in the MImodel when it, too, included missing values.They felt that it was somehow unfair to imputethe DV.

The truth is that all variables in the analy-sis model must be included in the imputation

model. The fear is that including the DV in theimputation model might lead to bias in estimat-ing the important relationships (e.g., the regres-sion coefficient of a program variable predictingthe DV). However, the opposite actually hap-pens. When the DV is included in the model, allrelevant parameter estimates are unbiased, butexcluding the DV from the imputation modelfor the IVs and covariates can be shown to pro-duce biased estimates. The problem with leav-ing the DV out of the imputation model is this:When any variable is omitted from the model,imputation is carried out under the assumptionthat the correlation is r = 0 between the omit-ted variable and variables included in the impu-tation model. Thus, when the DV is omitted,the correlations between it and the IVs (andcovariates) included in the model are all sup-pressed (i.e., biased) toward 0.

MAR methods don’t work if the MAR as-sumption does not hold (AKA, completecases are preferred if MAR does not hold).With some procedures, such as multiple linearregression, it is assumed that the data are mul-tivariate normal. Violation of this assumptionis known to affect the results (most notably thestandard errors of the regression coefficients).So with multiple regression, if the normalityassumption has been violated, one should usea different procedure. This logic makes goodsense with multiple regression analysis, but itdoes not apply to analysis with missing data be-cause with multiple regression, when the nor-mality assumption is violated, other commonprocedures work better. But with missing data,when the MAR assumption has been violated,the violation affects the old procedures (e.g.,listwise deletion) as well, and typically this vio-lation has greater effect on the old procedures.In short, MI and ML methods are always atleast as good as the old procedures (e.g., list-wise deletion, except in artificial, unrealistic cir-cumstances), and MI/ML methods are typicallybetter than old methods, and often very muchbetter.

An important difference between MI/MLmethods and complete cases analysis is that


Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


auxiliary variables (see next section) may be usedwith MI/ML in order to reduce the impact ofMNAR missingness. However, there is no goodway of incorporating auxiliary variables into acomplete cases model unless they can reason-ably be incorporated (e.g., as covariates) intothe model of substantive interest.

PRACTICAL ISSUES: MAKINGMISSING DATA ANALYSIS WORKIN THE REAL WORLD

The suggestions given here are designed tomake missing data analyses useful in real-worlddata situations. Some of the suggestions givenhere are necessarily brief. Many other practicalsuggestions are given in Graham et al. (2003)and elsewhere.

Inclusive versus Restrictive VariableInclusion Strategies (MI versus FIML)

In some ways, this is the most important lessonthat can be learned when doing missing dataanalysis in the real world. Collins et al. (2001)discussed the differences between “inclusive”and “restrictive” variable inclusion strategies inmissing data analysis. An inclusive strategy isone in which auxiliary variables are included inthe model. An auxiliary variable is a variable thatis not part of the model of substantive interest,but is highly correlated with the variables in thesubstantive model. Collins et al. (2001) showedthat including auxiliary variables in the missingdata model can be very helpful in two impor-tant ways. It can reduce estimation bias due toMNAR missingness, and it can partially restorelost power due to missingness.

Collins et al. (2001) note that the potentialauxiliary variable benefit is the same for MI andFIML analyses but that the typical use of MIis different from typical use of FIML. For MIanalyses, including auxiliary variables in the im-putation model has long been practiced and isvery easy to accomplish: Simply add the vari-ables to the imputation model. Furthermore,once the auxiliary variables have been includedin the imputation model, subsequent analyses

involving the imputed data benefit from theauxiliary variables, whether or not those vari-ables appear in the analysis of substantive inter-est (this latter benefit also applies to analysisof the EM covariance matrix). On the otherhand, FIML analyses have typically includedonly the variables that are part of the model ofsubstantive interest. Thus, researchers who useFIML models have found it difficult to incor-porate auxiliary variables in a reasonable way.Fortunately, reasonable approaches for includ-ing auxiliary variables into SEM/FIML mod-els now exist (e.g., see Graham 2003; also seethe recently introduced feature in Mplus foreasing the process of including auxiliary vari-ables). Note that although these methods workwell for SEM/FIML models, no correspondingstrategies are available at present for incorpo-rating auxiliary variables into latent class FIMLmodels.

Small Sample Sizes

Graham & Schafer (1999) showed that MI per-forms very well in small samples (as low asN = 50), even with very large multiple regres-sion models (as large as 18 predictors) and evenwith as much as 50% missing data in the DV.The biggest issue with such small samples is notthe missingness, per se, but rather that one sim-ply does not have much data to begin with andmissingness depletes one’s data even further. MIwas shown to perform very well under these cir-cumstances; the analyses based on MI data wereas good as the same analyses performed on com-plete data.

The simulations performed by Graham &Schafer (1999) also showed that normal-modelMI with nonnormal data works well, as well asanalysis with the same data with no missing val-ues. Although analysis of imputed data works aswell as analysis with complete datasets, noth-ing in the imputation process, per se, fixes thenonnormal data. Thus, in order to correct theproblems with standard errors often found withnonnormal data, analysis procedures must beused that give correct standard errors (e.g., thecorrection given for SEM by Satorra & Bentler

560 Graham

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


1994). Enders (2001b) has drawn similar con-clusions regarding the use of the FIML missingdata feature for SEM programs.

Rounding

Rounding should be kept to a minimum. MI wasdesigned to restore the lost variability found insingle imputation, and the MI strategy was de-signed to yield the correct variability. Roundingis tantamount to adding more variability to theimputed values. The added variability is ran-dom, to be sure, but there is definitely more ofit with rounding than without. This additionalvariance is evident in coefficient alpha analyseswith rounded and unrounded imputed valuesin a single data set imputed from EM parame-ters. Coefficient alpha is always a point or twolower (showing more random error variance)with rounding than without (also, see the Cat-egorical Missing Data and Normal-Model MIsection below for a discussion regarding round-ing for categorical variables).

Number of Imputations in MI

Missing data theorists have often claimed thatgood inferences can be made with the num-ber of imputed data sets (m) as few as m =3 to 5. They have argued that the relative ef-ficiency of estimation is very high under thesecircumstances, compared to an infinite numberof imputations. However, Graham et al. (2007)have recently shown that the effects of m onstatistical power for detecting a small effect size(ρ = 0.10) can be strikingly different fromwhat is observed for relative efficiency. Theyshowed that if statistical power is the main con-sideration, the number of imputations typicallymust be much higher than previously thought.For example, with 50% missing information,Graham et al. (2007) showed that MI withm = 5 has a 13% power falloff compared tothe equivalent FIML analysis; with 30% miss-ing information and m = 5, there was a 7%power falloff compared to FIML. Graham et al.(2007) recommend that at least m = 40 impu-tations are needed with 50% missing informa-

tion to guarantee less than a 1% power falloffcompared to the comparable FIML analysis.

Making EM (and MI) PerformBetter (i.e., Faster)

Factors that affect the speed of MI are the sameas those that affect the speed of EM, so I focushere on the latter. EM involves matrix manip-ulations to a large extent, so sample size hasrelatively little effect. However, the number ofvariables (k) affects EM tremendously. Con-sider that EM estimates [k (k+1)/2 + k] param-eters [k variances, (k(k−1)/2) covariances, andk means]. That means that with 25, 50, 100,150, and 200 variables, EM must estimate 350,1325, 5150, 11,475, and 20,300 parameters, re-spectively. Note that as the number of variablesgets large, the number of estimated parame-ters in EM gets huge. Although there is leewayhere, I generally try to keep the total numberof variables under 100 even with large samplesizes of N = 1000 or more. With smaller sam-ple sizes, a smaller number of variables shouldbe used.

Also affecting the speed of EM and MI isthe amount of missing information (similar to,but not the same as, the amount of missingdata). More missing information means EMconverges more slowly. Finally, the distribu-tions of the variables can affect speed of conver-gence. With highly skewed data, EM generallyconverges much more slowly. For this reason, itis often a good idea to transform the data (e.g.,with a log transformation) prior to imputation.The imputed values can be back-transformed(e.g., using the antilog) after imputation, ifnecessary.

If EM is very slow to converge, for exam-ple, if it takes more than about 200 iterations,the speed of convergence can generally be im-proved. If EM converges in 200 iterations, thenone should ask for 200 steps of data augmenta-tion between each imputed data set. With m =40 imputed data sets, one would need to run40 × 200 = 8000 steps of DA. If EM con-verged in 1000 iterations, one would need torun 40 × 1000 = 40,000 steps of DA. The


Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


additional time can be substantial, especially ifthe time between iterations is large.

Including Interactions in the MissingData Model

An issue that comes up frequently in missingdata analysis has to do with omitting certainvariables from the missing data model. Thisissue is sometimes referred to as being surethat the imputation model is at least as generalas the analysis model. A clear example is a testof the effect of an interaction (e.g., the product)of two variables on some third variable. Becausethe product is a nonlinear combination of thetwo variables, it is not part of the regular linearimputation model. The problem with exclud-ing such variables from the imputation modelis that all imputation is done under the assump-tion that the correlation is r = 0 between theomitted variable and all other variables in theimputation model. In this case, the correlationbetween the interaction term and the DVs ofinterest will be suppressed toward 0. The solu-tion is to anticipate any interaction terms andinclude the relevant product terms in the im-putation model.

Another way to conceive of an interactionis to think of one of the variables as a group-ing variable (e.g., gender). The interaction inthis case means that the correlation betweentwo variables is different for males and females.In the typical imputation model, one imputesunder the model that all correlations are thesame for females and males. A good way to im-pute under a model that allows these correla-tions to be different is to impute separately formales and females. The advantage of this ap-proach is that all interactions involving gendercan be tested during analysis even if a specific in-teraction was not anticipated beforehand. Thisapproach works very well in program effectsanalyses. If the program and control groups areimputed separately, then it is possible to testany interaction involving the program dummyvariable after the fact. One drawback to imput-ing separately within groups is that it cuts thesample size at least in half. This may be accept-

able with a large sample. But if the sample istoo small for this strategy, then including a fewcarefully selected product terms may be the bestoption.

Longitudinal Data and SpecialLongitudinal Missing Data Models

Missing data models have been created for han-dling special longitudinal data sets (e.g., thePAN program; Schafer 2001). Some people be-lieve that programs such as PAN must be usedto impute longitudinal data, for example, inconnection with growth curve modeling (see“Modern” Missing Data Analysis Methods sec-tion above). However, this is not the case. Itis easiest to see this by examining the variousways in which growth curve analyses can beperformed. Special hierarchical linear model-ing programs (e.g., HLM; Raudenbush & Bryk2002) can be used for this purpose. However,standard SEM programs can also be used (e.g.,see Willett & Sayer 1994). When analyses areconducted with these models under identicalconditions (e.g., assuming homogeneity of er-ror variances over time), the results of these twoprocedures are identical.

The key for the present review is thata variance-covariance matrix and vector ofmeans provide all that is needed for perform-ing growth modeling in SEM. Thus, any miss-ing data procedure that preserves (i.e., estimateswithout bias) variances, covariances, and meansis acceptable. This is exactly what results fromthe EM algorithm, and, asymptotically, withnormal-model MI. In summary, MI under thenormal model, or essentially equivalent SEMmodels with a FIML missing data feature, maysafely be used in conjunction with longitudinaldata.

Categorical Missing Dataand Normal-Model MI

Although some researchers believe that missingcategorical data requires special missing dataprocedures for categorical data, this is not truein general. The proportion of people giving the

562 Graham

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


“1” response for a two-level categorical variablecoded “1” and “0” is the same as the mean forthat variable. Thus, the important characteris-tics of this variable are preserved, even usingnormal-model MI. If the binary variable (e.g.,gender) is to be used as a covariate in a regres-sion analysis, then the imputed values shouldbe used, as is, without rounding (see Roundingsection above). If the binary variable must beused in analysis as a binary variable, then eachimputed value should be rounded to the near-est observed value (0 or 1). There are variationson this rounding procedure (e.g., see Bernaardset al. 2007), but the simple rounding is knownto perform very well in empirical data.

With normal-model MI (this also applies toSEM analysis with FIML), categorical variableswith two levels may be used directly. However,categorical variables with more than two levelsmust first be dummy coded. If there are p levelsin the categorical variable, then p − 1 dummyvariables must be created to represent the cate-gorical variable. For example, with a categoricalvariable with four levels, this dummy coding iscompleted as shown in Table 2.

If the original categorical variable has nomissing data, then creating these dummy vari-ables and using them in the missing data analysisis all that must be done. However, if the origi-nal categorical variable does have missing data,then imputation under the normal model maynot work perfectly, and an ad hoc fix must beused. The problem is that dummy coding hasprecise meaning when all dummy-code valuesfor a particular person are 0 or if there is exactlyone 1 for the person. If there is a missing valuefor the original categorical variable, then all ofthe dummy variables will also be missing and it

Table 2 Example of dummy coding withfour-level categorical variable

Dummy variable

Original category D1 D2 D31 1 0 02 0 1 03 0 0 14 0 0 0

is possible that a missing value for two (or more)of the dummy variables could be imputed as a 1after rounding. If any people have 1 values formore than one of these dummy variables, thenthe meaning of the dummy variables is changed.

If the number of “illegal” imputed values inthis situation is small compared with the overallsample size, then one could simply leave them.However, a clever, ad hoc fix for the problemhas been suggested by Paul Allison (2002). Toemploy Allison’s fix, it is important to imputewithout rounding. Whenever there is an illegalpattern of imputed values (more than a single1 for the dummy variables), the value 1 is as-signed to the dummy variable with the highestimputed value, and 0 is assigned to all others inthe dummy variable set. The results of this fixwill be excellent under most circumstances.

Estimating proportions and frequencieswith normal-model MI. Although normal-model MI does a good job of preserving manyimportant characteristics of the data set as awhole, it is important to note that it does notpreserve proportions and frequencies, except inthe special case of a variable with just two levels(e.g., yes and no coded as 1 and 0), in whichcase the proportion of people giving the 1 re-sponse is the same as the mean and is thus pre-served. However, consider the question, “Howmany cigarettes did you smoke yesterday?”(0 = none, 1 = 1–5, 2 = 6 or more). Re-searchers may be interested in knowing the pro-portion of people who have smoked cigarettes.This is the same as the proportion of peoplewho did not respond 0, or one minus the pro-portion who gave the 0 response. Although themean of this three-level smoking variable willbe correct with normal-model MI, the pro-portion of people with the 0 response is notguaranteed to be correct unless the three-levelsmoking variable happens to be normally dis-tributed (which is unlikely in most popula-tions). This problem can be corrected simplyby performing a separate EM analysis with thetwo-level version of this smoking variable (e.g.,0 versus other). The EM mean provides thecorrect proportion. Correct frequencies for all


Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


response categories, if needed, may be obtainedin this case by recasting the three-level categor-ical variable as two dummy variables.

Normal-Model MIwith Clustered Data

The term “clustered data” refers to the situa-tion in which cases are clustered in naturallyoccurring groups, for example, students withinschools. In this situation, the members of eachcluster are often more similar to one anotherin some important ways than they are to mem-bers of other clusters. This type of multilevelstructure requires special methods of analysis(e.g., Raudenbush & Bryk 2002; also see Murray1998). These multilevel analysis models allowvariable means to be different across the dif-ferent clusters (random intercepts model), andsometimes they allow the covariances to be dif-ferent in the different clusters (random slopesand intercepts model). If the analysis of choiceis a random slopes and intercepts model, then,just as described above, the imputation modelshould involve imputing separately within eachcluster. However, if the analysis of choice in-volves only random intercepts, then a some-what easier option is available. In this situation,the cluster membership variable can be dummycoded. That is, for p clusters, p − 1 dummyvariables would be specified (see Table 2 for asimple dummy variable example).

With the dummy coding strategy, the p −1 dummy variables are included in the imputa-tion model. As long as the number of clusters isrelatively small compared with the sample size,this dummy coding strategy works well. I haveseen this strategy work well with as many as35 dummy variables, although a smaller num-ber is desirable. Remember that the number ofdummy variables takes away from the numberof substantive variables that reasonably can beused in the imputation model.

When the number of clusters is too high towork with normal-model MI, several optionsare available. A specialty MI model (such asPAN; Schafer 2001) can be employed, but thatstrategy can sometimes be costly in terms of

learning new procedures. In addition, the per-formance of PAN has not been adequately eval-uated at this time. Alternatively, the number ofclusters can be reduced in a reasonable way. Forexample, if it is known that certain clusters havesimilar means, then these clusters could be com-bined (i.e., they would have the same dummyvariable) prior to imputation (note that this kindof combining of clusters must be done withinexperimental groups). For every combinationof this sort that can be reasonably formed, thenumber of dummy variables is reduced by one.I have even seen the strategy of performing ak-means cluster analysis on the key study vari-ables using school averages as input. This typeof analysis would help identify the clusters ofclusters for which means on key variables aresimilar.

One factor to take into consideration whenemploying the dummy variable approach tohandling cluster data is that sometimes vari-ables, especially binary variables, that have verylow counts (e.g., marijuana use among fifthgraders) will be constants within one or moreof the clusters. The data should be examinedfor this kind of problem prior to attempting thedummy variable approach. A good way to start isto perform a principal components analysis onthe variables to be included in the imputationmodel, along with the dummy variables. If thelast eigenvalue from this analysis is positive, thedummy coding strategy will most likely work.

Large Numbers of Variables

This problem represents perhaps the biggestchallenge for missing data analyses in largefield studies, especially longitudinal field stud-ies. Consider that most constructs are measuredwith multiple items. Five constructs with fouritems per construct translates into 20 individualvariables. Five waves of measurement produce100 variables. If more constructs are included inthe analysis, it is difficult to keep the total downto the k = 100 that I have recommended. It iseven more difficult to keep the variables in theimputation model to a reasonable number if onehas cluster data and is employing the dummy

564 Graham

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


variable strategy (see previous section). Below, Idescribe briefly two strategies that have workedwell in practice.

Imputing whole scales. If the analysis in-volves latent variable analysis (e.g., SEM) suchthat the individual variables must be part ofthe analysis, then imputing whole scales is notpossible. However, analyses often involve thewhole scales anyway, so imputing at that level isan excellent compromise. As long as study par-ticipants have data either for all or for none ofthe scale items, then this strategy is easy. Theproblem with using this strategy is how to dealwith the partial data on the scale. Schafer &Graham (2002) suggested that forming a scalescore based on partial data can cause problemsin some situations, but may be fine in others.In my experience, forming a scale score basedon partial data will be acceptable (a) if a rel-atively high proportion of variables are usedto form the scale score (and never fewer thanhalf of the variables), and (b) when the variablesare consistent with the domain sampling model(Nunnally 1967), and (c) when the variableshave relatively high coefficient alpha. Variablescan be considered consistent with the domainsampling model when the item-total correla-tions (or factor loadings) for the variables withinthe scale are all similar. Conceptually, it must bereasonable that dropping one variable from thescale has essentially the same meaning as drop-ping any other variables from the scale. Other-wise, one must either discard any partial data orimpute at the individual item level for that scale.Imputing at the scale level for even some of thestudy scales will often help with the problem ofhaving too many variables.

Think FIML. Another strategy that workswell in practice is to start with the variablesthat would be included in the analysis modelof substantive interest. If the FIML approachesare used, then these are the only variables thatwould be included using the typical practice.In addition, one should carefully select the fewauxiliary variables that will have the most ben-eficial impact on the model. Adding auxiliary

variables that are correlated r = 0.50 or bet-ter with the variables of interest will generallyhelp the analysis to have less bias and morepower. However, adding auxiliary variables withlower correlations will typically have little in-cremental benefit, especially when these addi-tional variables are correlated with the auxiliaryvariables already being used.

Good candidates for auxiliary variables arethe same variables used in the analytic model,but measured at different waves. For example,in a test of a program’s effect on smoking at wave4, with smoking at wave 1 as a covariate, smok-ing at waves 2 and 3 is not being used in theanalysis model but should be rather highly cor-related with smoking at wave 4. These two vari-ables would make excellent auxiliary variables.

A related strategy is to impute variables thatare relatively highly correlated with one an-other. Suppose, for example, that one would liketo conduct program effect analyses on a largenumber of DVs, say 30. But including 30 DVs,30 wave-1 covariates, and the corresponding60 variables from waves 2 and 3 (as auxiliaryvariables) would be too many, especially con-sidering that there are perhaps 10 backgroundvariables as covariates and perhaps 20 dummyvariables representing cluster membership. Allthese hypothetical variables total 150 variablesin the imputation model.

In this instance, a principal componentsanalysis on the 30 DVs could identify threeor four sets of items that are relatively highlycorrelated. This approach does not need to beprecise; it is simply used as a device for identi-fying variables that are more correlated. Thesegroupings of variables would then be imputedtogether. By conducting imputation analysesand analyses in these groupings, nearly all ofthe benefits of the auxiliary variables would begained at a minimum cost of time for imputa-tion and analysis.

Practicalities of Measurement:Planned Missing Data Designs

In this section, I outline the developing areaof planned missingness designs. If you are


Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


following my advice, you are already (or soonwill be) using the recommended MI and MLmissing data procedures. Thus, it is time toexplore new possibilities with respect to datacollection.

3-Form design. Measurement is the corner-stone of science. Without measurement, thereis no science. When researchers design the mea-surement component of their study, they areuniversally faced with the dilemma of wantingto ask more questions than their study partic-ipants are willing to answer, given what theyare being paid. Invariably, researchers are facedwith the choice of asking fewer questions orpaying their participants more. The 3-form de-sign (Graham et al. 2006), which has been inuse since the early 1980s (e.g., see Grahamet al. 1984), gives researchers a third choice.In its generic form, the 3-form design allowsresearchers to increase by 33% the number ofquestions for which data are collected withoutchanging the number of questions asked of eachrespondent. The trick is to divide all questionsasked into four items sets. The X set, whichcontains questions most central to the studyoutcomes, is asked of everyone. But the A, B,and C sets of items are rotated, such that oneset is omitted from each of the three forms.Table 3 describes the basic idea of the design(Raghunathan & Grizzle 1995 suggested a sim-ilar design involving all possible two-set com-binations of five items sets).

The benefit of the 3-form design is that onegathers data for 33% more questions than canbe asked of any one participant. Also, impor-tantly, at least one-third of the participants pro-vide data for every pair of questions. That is, allcorrelations are estimable. This feature is not

Table 3 3-form design

Respondent received item set?

Form X A B C1 Yes Yes Yes No2 Yes Yes No Yes3 Yes No Yes Yes

shared by most other measurement designs ofthis general sort (generically described as matrixsampling). The drawback to the 3-form designis that some correlations, because they are basedon only one-third of the sample, are tested withlower power. However, as Graham et al. (2006)show, virtually all of the possible drawbacks areunder the researcher’s control and can generallybe avoided.

Two-method measurement. Graham et al.(2006) also described a planned missingness de-sign called two-method measurement (also seeAllison & Hauser 1991, who describe a re-lated design). The two-method measurementdesign stems from the need to obtain good,valid measures of the main DV. Researchersin many domains face a dilemma: (a) collectdata from everyone with a relatively inexpen-sive, but questionably valid measure (e.g., aself-administered, self-report measure of recentphysical activity), or (b) collect data from a smallproportion of study participants using a morevalid, but much more expensive, measure (e.g.,using an expensive accelerometer). The two-method measurement design allows the collec-tion of both kinds of data: complete data forthe less expensive measure, and partial data (ona random sample of participants) for the expen-sive measure. SEM models are then tested inwhich the two kinds of data are both used as in-dicators of a latent variable of the construct ofinterest (e.g., recent physical activity). If certainassumptions are met (and they are commonlymet), then this SEM approach allows for the testof the main study hypotheses with more statis-tical power than is possible with the expensivemeasure alone and with more construct validitythan is possible with the inexpensive measurealone. For details on this design, see Grahamet al. (2006).

ATTRITION AND MNARMISSINGNESS

The methods described up to this point areclear. I believe that the directions we have

566 Graham

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


headed, and the MI and ML methods that havebeen developed, are the right way to go. Myadvice continues to be to learn and use thesemethods. I am certain that the major softwarewriters will continue to help in this regard, andin a very few years, the software solutions willbe abundant (e.g., see these recent missing datadevelopments: the feature in Mplus for easingthe process of including auxiliary variables; theassociation between SPSS and Amos software;and the inclusion of MI into SAS/STAT® soft-ware). But as we move forward into the arenaof attrition and MNAR missingness, the watersget a bit murky. The following section describesmore of a work in progress. Nevertheless, I be-lieve that we are making strides, and the way tomove forward is becoming clearer.

It has been said that attrition in longitu-dinal studies is virtually ubiquitous. Althoughthat may be true in large part, also ubiqui-tous is the fear researchers and critics expressthat attrition is a threat to the internal validityof the study and the readiness with which re-searchers are willing to discount study resultswhen more than a few percent of the partici-pants have dropped out along the way. To makematters worse, even the missing data expertsoften make statements that, if taken out of con-text, seem to verify that the fears about attri-tion were well founded. To be fair, the concernsabout attrition began well before the missingdata revolution, and in the absence of the cur-rent knowledge base about missing data, per-haps taking the conservative approach was theright thing to do.

But now we have this knowledge base, andit is time to take another, careful look at at-trition. It is important to acknowledge that theconclusions in some studies will be adversely af-fected by attrition and MNAR missingness, andI am not suggesting that we pretend that isn’tthe case. But I do believe that the effects of at-trition on study conclusions in a general senseare not nearly as severe as commonly feared.In this section, I describe the beginnings of aframework for measuring the extent to whichattrition has biasing effects, and I present evi-

dence that the biasing effects of attrition can betolerably low, even with what is normally con-sidered substantial attrition. Furthermore, I citeresearch showing that if the recommended safe-guards are put in place, the effects of attritioncan be further diminished.

With MAR missingness, missing scores atone wave can be predicted from the scores atprevious waves. This is true even though theprevious scores might be markedly different forstayers and leavers. With MNAR missingness,the missing scores cannot be predicted basedon the previous scores; the model that appliesto complete cases for describing how scoreschange over time does not apply to those whohave missing data. The practical problem withMNAR missingness is that the missing scorescould be anywhere (high or low). And becauseof this uncertainty, it is possible that clear judg-ments cannot be made about the study conclu-sions. The strategies I present in this sectionserve to reduce that uncertainty. Some of thesestrategies can be employed after the fact (es-pecially for longitudinal studies), but many ofthe strategies must be planned in advance formaximum effectiveness.

Some Clarifications About MissingData Mechanisms

The major three missingness mechanisms areMCAR, MAR, and MNAR. These three kindsof missingness should not be thought of as mu-tually exclusive categories of missingness, de-spite the fact that they are often misperceivedas such. In particular, MCAR, pure MAR, andpure MNAR really never exist because the pureform of any of these requires almost universallyuntenable assumptions. The best way to thinkof all missing data is as a continuum betweenMAR and MNAR. Because all missingness isMNAR (i.e., not purely MAR), then whetherit is MNAR or not should never be the issue.Rather than focusing on whether the MI/MLassumptions are violated, we should answer thequestion of whether the violation is big enoughto matter to any practical extent.


Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


Measuring the Biasing Effectsof Attrition

In order for researchers to move toward a miss-ing data approach that focuses on the likelyimpact of MNAR missingness, they need toolsfor measuring the effects of missingness (attri-tion) on estimation bias. Collins et al. (2001)made effective use of the standardized bias(presented as a percent of the standard er-ror) for this purpose: Standardized Bias =100 × (average parameter estimate − popu-lation value)/SE, where SE = the standard er-ror of the estimate, or the standard deviationof the sampling distribution for the parameterestimate in question. Collins et al. (2001) ar-gued that standardized bias greater than 40%(i.e., more than 40% of a standard error) rep-resented estimation bias that was of practicalsignificance. Armed with this tool for judgingwhether MNAR bias was of practical signif-icance, Collins et al. (2001) showed that theregression coefficient (X predicting Y), wherethe cause of missingness was Z, was biased toa practical degree only when there was 50%missing on Y and rZY = 0.90. With 50%missing on Y and rZY = 0.40, 25% missingon Y and rZY = 0.90, and 25% missing onY and rZY = 0.40, the bias was judged notto be of practical significance when the causeof missingness was omitted from the model(i.e., for MNAR missingness). (Note that al-though Collins et al. 2001 found that the re-gression coefficient was largely unbiased, themean of Y did show a practical level of bias inall four of their missingness and rZY scenariosdescribed above. This may be an issue in somestudies.)

The findings of the Collins et al. (2001)study are important for a variety of reasons:(a) they demonstrated the usefulness of stan-dardized bias as a way of measuring the practi-cal effects of bias due to attrition, and (b) theyshowed that MNAR missingness alone is of-ten not sufficient to affect the internal valid-ity of an experimental study to any practicalextent.

Suppression and inflation bias. Two kinds ofattrition bias are inflation and suppression bias.Inflation bias makes a truly ineffective program,or experimental manipulation, appear to be ef-fective. Suppression bias, on the other hand,makes a truly effective program look less effec-tive or an ineffective program look as if it hada harmful effect. When evaluation researcherswrite about attrition bias, they usually are talk-ing about inflation bias, which is a major con-cern because it calls into question the internalvalidity of a study. Suppression bias is muchless important if it occurs along with a signifi-cant program effect in the desired direction andthus does not undermine the internal validity ofthe study. On the other hand, suppression biascan be a significant factor if it keeps the trulyeffective nature of a program from being ob-served. The possibility of suppression bias is anespecially important factor during the planning(and proposal) stages of a project. The chancesare reduced that a project will be funded if thepower to detect true effects is likely to be di-minished unduly because of suppression bias.

Important quantities in describing miss-ingness. In any discussion of missing dataand attrition, three quantities are prominentlyfeatured: (a) the amount of missingness (i.e.,percent missing or percent attrition), (b) rZY, thecorrelation between the cause of missingness,Z, and the model variable containing missing-ness, Y, and (c) rZR, the correlation between thecause of missingness, Z, and missingness itself,R. This last quantity, rZR, is often manipulatedas MAR linear by allowing the probability ofa missing Y to be dependent on the quartilesof Z. In Collins et al. (2001), for example, theprobabilities of missing on Y were 0.20, 0.40,0.60, and 0.80 for the first, second, third, andfourth quartiles of Z, respectively. The magni-tude of this correlation, rZR, which depends onthe range of these probabilities, can be thoughtof as the strength of the lever for missingness.That is, a wide range means greater impact onmissingness (higher rZR), and a narrow rangemeans less impact (lower rZR). Collins et al.

568 Graham

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


(2001) used a rather strong lever for missingnessin their simulations. Their lever for 50% miss-ingness produces rZR = 0.447. A weaker leverfor missingness would involve setting the fourprobabilities for Y missing to different val-ues, for example, 0.35, 0.45, 0.55, and 0.65 forthe four quartiles of Z. This lever also pro-duces 50% missingness on Y, but corresponds torZR = 0.224.

Missing Data Diagnostics

Researchers often want to examine the differ-ence between stayers and leavers on the pretestvariables of a longitudinal study. Knowledgewill be gained from this practice, but not asmuch as researchers might think. Because themissingness is not MCAR, any differences ob-served on pretest variables should not be un-expected. And the information gained cannotindicate the degree to which the missingnessis MNAR. Perhaps the best value of this kindof analysis is to identify the pretest variablesthat most strongly predict missingness later inthe study; these variables can be included in themissing data model (e.g., see Heckman 1979,Leigh et al. 1993).

The diagnostics of Hedeker & Gibbons.However, making use of longitudinal data andexamining the missingness and the patterns ofchange over time on the main DV can be veryenlightening. Hedeker & Gibbons (1997) pro-vided an excellent example of this kind of di-agnostic. In the empirical study they used toillustrate their analytic technique (a patternmixture model), they plotted the main DVover the four main measurement points infour groups: (a) drug group, data for week 6;(b) placebo group, data for week 6; (c) druggroup, data missing for week 6; (d ) placebogroup, data missing for week 6 (where week 6was the final measure in the study).

These plots (Hedeker & Gibbons 1997,p. 72) show clearly that the changes over time inall four groups were nearly linear. Among thosewith data for the last time point, the participantsin the drug group were clearly doing better thanthose in the placebo group. Among those who

did not have data for the last measure (week 6),the people in the drug condition appeared tobe doing even better, and those in the placebocondition appeared to be doing even worse.

Although it is highly speculative to extrap-olate from a single pretest to the last posttest,extrapolation to the last posttest makes muchmore sense when one is working from a clearlyestablished longitudinal trend. That is, there ismuch less uncertainty about what the missingscores might be on the final measure. In theHedeker & Gibbons (1997) study, for exam-ple, it can be safely assumed that those withoutdata for the final wave continued along the same(or similar) trajectory that could be observedthrough three of the four time points.

Figure 1 displays the same kind of datafrom the Adolescent Alcohol Prevention Trial(AAPT; Hansen & Graham 1991). The figureillustrates the same four plots, this time for theprogram group (Norm) and comparison group(No Norm) for those who did have data forthe final follow-up measure at eleventh gradeand for those who did not. It is evident thatjust as in the study described by Hedeker &Gibbons (1997), the plots look reasonablysmooth, and it would be a reasonable to assume

1.2

1.4

1.6

1.8

2

2.2

2.4

2.6

smk7 smk8 smk9 smk10 smk11

Norm No Norm Norm No Norm

Figure 1Smoking levels over time for those with and without data for the last wave ofmeasurement. Students in the Norm program group (square markers) reportedlower levels of cigarette smoking than did students in the Comparison group(circle markers) for those who had data for the last wave of measurement(eleventh grade; white markers) as well as for those who had missing data for thelast wave of measurement (black markers).


Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


that the missing smoking scores for eleventhgrade followed this same or a similar trajectory.

All of these plotted points make it more dif-ficult to imagine that the missing points arehugely different from where one would guessthey would be based on the observed trajecto-ries. On the other hand, it would make littlesense to try to predict the missing eleventh-grade data based on the seventh-grade pretestdata alone. Based only on those pretest scores,the missing eleventh-grade scores could indeedbe anywhere.

The data described by Hedeker & Gibbons(1997) and the data presented here are very wellbehaved. However, not all plots of the main DVover time will be this smooth. In a longitudinalstudy, for example, the changes in the DV overtime may not conform to a simple, well-definedcurve (linear or curvilinear). Under those con-ditions, predicting the missing final data pointwould be much more difficult.

Nonignorable (MNAR) Methods

Nonignorable methods (e.g., see Demirtas2005; Demirtas & Schafer 2003; Hedeker &Gibbons 1997; Little 1993, 1994, 1995) maybe very useful. However, it is not necessarilytrue that any particular method will be betterthan MAR methods (e.g., normal-model MI orML) for any particular empirical study. It is wellknown that methods for handling nonignorabledata require the analyst to make assumptionsabout the model of missingness. If this model isincorrect, the MNAR model may perform evenless well than standard MAR methods (e.g., seeDemirtas & Schafer 2003).

On the other hand, MNAR methods such aspattern mixture models may, as argued above,be excellent tools for describing the missingnessin longitudinal fashion, thereby increasing one’sconfidence in many instances about the true na-ture of the missingness. As suggested by Little(1993, 1994, 1995), this type of model can be agood way to perform sensitivity analyses aboutthe model structure. If the same general studyconclusions are made over a wide variety of pos-sible missing data models, then one has greater

confidence in those study conclusions. In ad-dition, the models suggested by Hedeker &Gibbons (1997) may prove to be especially use-ful when the longitudinal patterns of the mainDV are as smooth as they described.

Strategies for Reducing the BiasingEffects of Attrition

Use auxiliary variables. Probably the singlebest strategy for reducing bias (and increasingstatistical power lost owing to missing data) isto include good auxiliary variables in the miss-ing data model (see Collins et al. 2001). AsLittle (1995) put it, “one should collect covari-ates that are useful for predicting missing val-ues.” It is important that these variables needonly be good for predicting the missing values;they need not be related to missingness, per se.Good candidate variables for auxiliary variablesare measures of the main DV that happen not tobe in the analysis model. However, if the analy-sis already involves all measures of the DV (e.g.,with latent growth modeling analyses), then theincremental benefit of other potential auxiliaryvariables is likely to be small.

Collins et al. (2001) showed that includingan auxiliary variable with rZY = 0.40 reducedrelatively little bias in any of the parameters ex-amined. However, including an auxiliary vari-able with rZY = 0.90 had a major impact on bias.Later simulations have suggested that the ben-efit from auxiliary variables begins to be notice-able at about rZY = 0.50 or 0.60. Furthermore,it appears that one or two auxiliary variableswith rZY = 0.60 are better than 20 auxiliaryvariables whose correlations with Y are all lessthan rZY = 0.40. This is true because such vari-ables are often intercorrelated, and the incre-mental benefit of adding them to the model isvery small.

Longitudinal missing data diagnostics. Theexcellent strategy described by Hedeker &Gibbons (1997) is discussed above. If longitu-dinal data are available, this strategy is a goodway to describe the missingness patterns. Not

570 Graham

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


all patterns will be good. The best longitudinalpatterns are those that reduce the uncertaintyabout what the missing scores might be at thelast waves of measurement.

Measuring intent to drop out. Schafer &Graham (2002) suggested that a potentiallygood way of reducing attrition bias is to mea-sure participants’ intent to drop out of the study.Some people say they will drop out and do dropout, whereas others say they will drop out anddo not. Those who do not drop out provide agood basis for imputing the scores of those whodo. Demirtas & Schafer (2003) suggested thatthis approach might be one good way of deal-ing with MNAR missingness. Leon et al. (2007)performed a simulation that suggested that thisapproach can be useful.

Collecting follow-up data from thoseinitially missing. Perhaps the best way ofdealing with MNAR missingness is to followup and measure a random sample of thoseinitially missing from the measurement ses-sion (e.g., see Glynn et al. 1993, Graham &Donaldson 1993). This strategy is not easy,and it is impossible in some research settings(e.g., where study participants have died). How-ever, even if some studies are conducted thatinclude these follow-up measures from a ran-dom sample of those initially missing, it couldshed enormous light on the issues surroundingMNAR missingness. With a few well-placedstudies of this sort, we would be an excel-lent position to establish the true bias fromusing MAR methods and a variety of MNARmethods.

One wrinkle with this approach is that al-though collecting data from a random sample ofthose initially missing can be very difficult, col-lecting data from a nonrandom sample of thoseinitially missing is much easier. Although in-ferences from this nonrandom sample are gen-erally weaker than inferences possible from arandom sample, data such as these may be ofmuch value (e.g., see Glynn et al. 1993).

Suggestions for Reporting AboutAttrition in an Empirical Study

1. Avoid generic, possibly misleading, state-ments about the degree to which attritionplagues longitudinal research.

2. Be precise about the amount of attrition;avoid vague terms that connote a miss-ing data problem. Use precise percentageof dropout from treatment and controlgroups if that is relevant.

3. Missingness on the main DV can becaused by (a) the program itself, (b) theDV itself, (c) the program × DV inter-action, or (d ) any combination of thesefactors. Perform analyses that lay out asclearly as possible which version of attri-tion is most likely in the particular study(e.g., see Hedeker & Gibbons 1997).

4. Based on longitudinal diagnostics, assessthe degree of estimation bias, for ex-ample, using standardized bias (e.g., seeCollins et al. 2001) for this configurationand percent of attrition, and determinethe kind of bias (suppression or inflation).

5. Draw study conclusions in light of thesefacts.

Suggestions for Conductand Reporting of SimulationStudies on Attrition

1. Avoid generic, possibly misleading, state-ments about the degree to which attritionplagues longitudinal research. Limit thenumber of assertions about the possibleproblems associated with attrition. Thoseof us who do this kind of simulation studymust shoulder the responsibility of be-ing precise in how we talk about thesetopics—what we say can be very influen-tial. Be careful to give proper citations forany statement about the degree to whichattrition is known to be a problem or not.Try to focus on the constructiveness oftaking proper steps to minimize any bias-ing effects of attrition.

2. Be precise about the amount of attri-tion in the simulation study. Provide a


Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


sufficient number of citations to demon-strate that this amount of attrition is ofplausible relevance in the substantive areato which the simulation study applies.

3. Be precise about the configuration of at-trition simulated in the study. If the con-figuration of attrition is simulated in amanner different from that used in othersimulation studies, then provide a de-scription of the procedure in plain termsand in comparison with the approachesof other simulation researchers (e.g., seethe difference in style between Collinset al. 2001 and Leon et al. 2007). Dif-ferent approaches are all valuable; read-ers with varying degrees of technical skilljust need to know how they relate to oneanother.

Be precise about the strength of attritionused in the simulation. For example, Collinset al. (2001) specified increasing missingnessprobabilities for the four quartiles of the vari-able Z (0.20 for Q1, 0.40 for Q2, 0.60 for Q3,and 0.80 for Q4). As it turns out, this was verystrong attrition compared to what it could havebeen (e.g., 0.35 for Q1, 0.45 for Q2, 0.55 forQ3, and 0.65 for Q4). This is important becausethe strength used produced bias in the Collinset al. (2001) study that would present practi-cal problems, whereas the latter strength wouldproduce a level of bias that Collins and cowork-ers would have judged to be acceptably low,even with 50% missingness on Y, and rZY =0.90 in both cases. Also, present a sufficientnumber of citations from the empirical liter-ature to demonstrate that the strength of ef-fect used in the simulation actually occurs inempirical research to an extent that makes thestudy useful. As noted above, the strength ofattrition used in the Collins et al. (2001) studywas greater than is typically seen in empiricalresearch.

More Research is Needed

Collins et al. (2001) did a nice job of describ-ing the standardized bias concept, which they

used as one of the primary yardsticks of prac-tical importance of missing data bias. In theirstudy, they used 40% bias (parameter estimateis four-tenths of a standard error different fromthe population value) as the cutoff for MNARbias that would be of practical concern. Any-thing greater than 40% would be considered tobe of practical concern. This implies that any-thing 40% or less would be considered to be ofno practical concern.

Collins et al. (2001) went out on a limbwith an estimate of a cutoff for practical effectof MNAR bias, and this study is an excellentstarting place for this kind of research. I amreminded of the early days of SEM research,when researchers were struggling to find in-dices of practical model fit and to find cut-offs for such indices above which a model’s fitmight be judged as “good.” Bentler & Bonett(1980) were the first to provide any kind of cut-off (0.90 for their nonnormed fit index). SEMresearchers were eager to employ this 0.90 cut-off, but with considerable experience with thisfit index, eventually began to realize that per-haps 0.95 was a better cutoff.

I suggest that researchers involved withwork where attrition is a factor (both empiricaland simulation studies) begin to develop expe-rience with the standardized bias concept usedby Collins et al. (2001). But after years of expe-rience, will we still believe that 40% bias is thebest cutoff? It is easy to show that a standard-ized bias of 40% corresponds to a change in thet-value of 0.4. Can we tolerate such a change?Other issues surround the use of standardizedbias. For example, larger sample sizes producemore standardized bias. Future research shouldaddress the possibility that different cut pointsare needed for different sample sizes.

Other indices of the practical impact ofattrition bias. In the SEM literature, re-searchers now enjoy a plethora of indices ofpractical fit. There are even three or four funda-mentally different approaches to these indicesof practical fit. We need more such approachesto the practical effects of attrition. I encouragethe development of such indices.

572 Graham

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


Collecting data on a random sample of thoseinitially missing. Many authors have recom-mended collecting data on a random sample ofthose initially missing. However, most of thishas involved simulation work and not actualdata collection. Carefully conducted empiricalstudies along the lines suggested by Glynn et al.(1993) and Graham & Donaldson (1993) todetermine the actual extent of MNAR biaseswould be valuable, not just to the individual em-pirical study, but also to the study of attritionin general. If this type of study is conductedproperly, it will give us much-needed informa-tion about the effect of MNAR processes onestimation bias. It is possible that the kinds ofstudies for which MNAR biases are greatest areprecisely the studies for which collection of ad-ditional data on the initial dropouts is most dif-ficult. Nevertheless, even a few such studies willbe a great benefit.

Empirical studies testing benefit of intentto drop out questions. The simulation studyconducted by Leon et al. (2007) suggests thatthis strategy is promising. However, empiricalstudies are needed that make use of these kindsof procedures. Best, perhaps, would be a fewcarefully conducted studies that examined thecombination of this approach with the approachof collecting data on a random sample of thoseinitially missing.

SUMMARY AND CONCLUSIONS

1. Several excellent, useful, and accessibleprograms exist for performing analy-

sis with missing data with multiple im-putation under the normal model andmaximum-likelihood (or FIML) meth-ods. Use them! My wish is that 10 yearsfrom now, everyone will be making useof these procedures as a matter of course.Having these methods serve as our basicplatform will raise the quality of every-one’s research.

2. Gain experience with MNAR missingdata (e.g., from attrition), especially withthe measures of the practical effect ofMNAR data. Use the indices that exist,and evaluate them. Come up with yourown levels of what constitutes acceptablelevels of estimation bias. Where possi-ble, publish articles describing a new ap-proach to evaluating the practical impactof MNAR missingness on study conclu-sions.

3. Try to move away from the fear of missingdata and attrition. Situations will occur inwhich missing data and attrition will af-fect your research conclusions in an unde-sirable way. But don’t fear that eventual-ity. Embrace the knowledge that you willbe more confident in your research con-clusions, either way. Don’t see this possi-ble situation as a reason not to understandmissing data issues. Focus instead on theidea that your new knowledge means thatwhen your research conclusions are de-sirable, you needn’t have the fear that yougot away with something. Rather, you cango ahead with the cautious optimism thatyour study really did work.

DISCLOSURE STATEMENT

The author is not aware of any biases that might be perceived as affecting the objectivity of thisreview.

LITERATURE CITED

Allison PD. 1987. Estimation of linear models with incomplete data. In Sociological Methodology 1987, ed. CClogg, pp. 71–103. San Francisco, CA: Jossey-Bass

Allison PD. 2002. Missing Data. Thousand Oaks, CA: Sage


Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


Allison PD, Hauser RM. 1991. Reducing bias in estimates of linear models by remeasurement of a randomsubsample. Soc. Method Res. 19:466–92

Arbuckle JL, Wothke W. 1999. Amos 4.0 User’s Guide. Chicago: SmallwatersBentler PM, Bonett DG. 1980. Significance tests and goodness of fit in the analysis of covariance structures.

Psychol. Bull. 88:588–606Bernaards CA, Belin TR, Schafer JL. 2007. Robustness of a multivariate normal approximation for imputation

of incomplete binary data. Stat. Med. 26:1368–82Collins LM, Schafer JL, Kam CM. 2001. A comparison of inclusive and restrictive strategies in modern missing

data procedures. Psychol. Methods 6:330–51Demirtas H. 2005. Multiple imputation under Bayesianly smoothed pattern-mixture models for nonignorable

drop-out. Stat. Med. 24:2345–63Demirtas H, Schafer JL. 2003. On the performance of random-coefficient pattern-mixture models for non-

ignorable dropout. Stat. Med. 21:1–23Dempster AP, Laird NM, Rubin DB. 1977. Maximum likelihood from incomplete data via the EM algorithm

(with discussion). J. R. Stat. Soc. B39:1–38Duncan SC, Duncan TE. 1994. Modeling incomplete longitudinal substance use data using latent variable

growth curve methodology. Multivar. Behav. Res. 29:313–38Duncan SC, Duncan TE, Hops H. 1996. Analysis of longitudinal data within accelerated longitudinal designs.

Psychol. Methods 1:236–48Duncan TE, Duncan SC, Hops H. 1994. The effect of family cohesiveness and peer encouragement on the

development of adolescent alcohol use: a cohort-sequential approach to the analysis of longitudinal data.J. Stud. Alcohol 55:588–99

du Toit M, du Toit S. 2001. Interactive LISREL: User’s Guide. Lincolnwood, IL: Sci. Software Intl.Efron B. 1982. The Jackknife, the Bootstrap, and Other Resampling Plans. Philadelphia, PA: Soc. Industrial Appl.

Math.Enders CK. 2001a. A primer on maximum likelihood algorithms available for use with missing data. Struct.

Equ. Model. 8:128–41Enders CK. 2001b. The impact of nonnormality on full information maximum-likelihood estimation for

structural equation models with missing data. Psychol. Methods 6:352–70Enders CK. 2003. Using the expectation maximization algorithm to estimate coefficient alpha for scales with

item-level missing data. Psychol. Methods 8:322–37Glynn RJ, Laird NM, Rubin DB. 1993. Multiple imputation in mixture models for nonignorable nonresponse

with followups. J. Am. Stat. Assoc. 88:984–93Graham JW. 2003. Adding missing-data relevant variables to FIML-based structural equation models. Struct.

Equ. Model. 10:80–100Graham JW, Cumsille PE, Elek-Fisk E. 2003. Methods for handling missing data. In Research Methods in

Psychology, ed. JA Schinka, WF Velicer, pp. 87–114. Volume 2 of Handbook of Psychology, ed. IB Weiner.New York: Wiley

Graham JW, Donaldson SI. 1993. Evaluating interventions with differential attrition: the importance ofnonresponse mechanisms and use of followup data. J. Appl. Psychol. 78:119–28

Graham JW, Flay BR, Johnson CA, Hansen WB, Grossman LM, Sobel JL. 1984. Reliability of self-reportmeasures of drug use in prevention research: evaluation of the Project SMART questionnaire via thetest-retest reliability matrix. J. Drug Educ. 14:175–93

Graham JW, Hofer SM. 1992. EMCOV Users Guide. Univ. S. Calif. Unpubl. documentationGraham JW, Hofer SM. 2000. Multiple imputation in multivariate research. In Modeling Longitudinal and

Multiple-Group Data: Practical Issues, Applied Approaches, and Specific Examples, ed. TD Little, KU Schnabel,J Baumert, 1:201–18. Hillsdale, NJ: Erlbaum

Graham JW, Hofer SM, Donaldson SI, MacKinnon DP, Schafer JL. 1997. Analysis with missing data inprevention research. In The Science of Prevention: Methodological Advances from Alcohol and Substance AbuseResearch, ed. K Bryant, M Windle, S West, 1:325–66. Washington, DC: Am. Psychol. Assoc.

Graham JW, Hofer SM, MacKinnon DP. 1996. Maximizing the usefulness of data obtained with plannedmissing value patterns: an application of maximum likelihood procedures. Multivar. Behav. Res. 31:197–218

574 Graham

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


Graham JW, Hofer SM, Piccinin AM. 1994. Analysis with missing data in drug prevention research. In Advancesin Data Analysis for Prevention Intervention Research, National Institute on Drug Abuse Research Monograph,ed. LM Collins, L Seitz, 142:13–63. Washington, DC: Natl. Inst. Drug Abuse

Graham JW, Olchowski AE, Gilreath TD. 2007. How many imputations are really needed? Some practicalclarifications of multiple imputation theory. Prev. Sci. 8:206–13

Graham JW, Roberts MM, Tatterson JW, Johnston SE. 2002. Data quality in evaluation of an alcohol-relatedharm prevention program. Evaluation Rev. 26:147–89

Graham JW, Schafer JL. 1999. On the performance of multiple imputation for multivariate data with smallsample size. In Statistical Strategies for Small Sample Research, ed. R Hoyle, 1:1–29. Thousand Oaks, CA:Sage

Graham JW, Taylor BJ, Cumsille PE. 2001. Planned missing data designs in analysis of change. In New Methodsfor the Analysis of Change, ed. LM Collins, A Sayer, 1:335–53. Washington, DC: Am. Psychol. Assoc.

Graham JW, Taylor BJ, Olchowski AE, Cumsille PE. 2006. Planned missing data designs in psychologicalresearch. Psychol. Methods 11:323–43

Hansen WB, Graham JW. 1991. Preventing alcohol, marijuana, and cigarette use among adolescents: peerpressure resistance training versus establishing conservative norms. Prev. Med. 20:414–30

Heckman JJ. 1979. Sample selection bias as a specification error. Econometrica 47:153–61Hedeker D, Gibbons RD. 1997. Application of random-effects pattern-mixture models for missing data in

longitudinal studies. Psychol. Methods 2:64–78Honaker J, King G, Blackwell M. 2007. Amelia II: A Program for Missing Data. Unpubl. users guide. Cambridge,

MA: Harvard Univ. http://gking.harvard.edu/amelia/Horton NJ, Kleinman KP. 2007. Much ado about nothing: a comparison of missing data methods and software

to fit incomplete data regression models. Am. Stat. 61:79–90Joreskog KG, Sorbom D. 1996. LISREL 8 User’s Reference Guide. Chicago: Sci. SoftwareKing G, Honaker J, Joseph A, Scheve K. 2001. Analyzing incomplete political science data: an alternative

algorithm for multiple imputation. Am. Polit. Sci. Rev. 95:49–69Lanza ST, Collins LM, Schafer JL, Flaherty BP. 2005. Using data augmentation to obtain standard errors

and conduct hypothesis tests in latent class and latent transition analysis. Psychol. Methods 10:84–100Leigh JP, Ward MM, Fries JF. 1993. Reducing attrition bias with an instrumental variable in a regression

model: results from a panel of rheumatoid arthritis patients. Stat. Med. 12:1005–18Leon AC, Demirtas H, Hedeker D. 2007. Bias reduction with an adjustment for participants’ intent to drop

out of a randomized controlled clinical trial. Clin. Trials 4:540–47Little RJA. 1993. Pattern-mixture models for multivariate incomplete data. J. Am. Stat. Assoc. 88:125–34Little RJA. 1994. A class of pattern-mixture models for normal incomplete data. Biometrika 81:471–83Little RJA. 1995. Modeling the drop-out mechanism in repeated-measures studies. J. Am. Stat. Assoc. 90:1112–

21Little RJA, Rubin DB. 1987. Statistical Analysis with Missing Data. New York: WileyLittle RJA, Rubin DB. 2002. Statistical Analysis with Missing Data. New York: Wiley. 2nd ed.McArdle JJ. 1994. Structural factor analysis experiments with incomplete data. Multivar. Behav. Res. 29(4):

409–54McArdle JJ, Hamagami F. 1991. Modeling incomplete longitudinal and cross-sectional data using latent

growth structural models. Exp. Aging Res. 18:145–66McArdle JJ, Hamagami F. 1992. Modeling incomplete longitudinal data using latent growth structural equation

models. In Best Methods for the Analysis of Change, ed. L Collins, JL Horn, 1:276–304. Washington, DC:Am. Psychol. Assoc.

Murray DM. 1998. Design and Analysis of Group-Randomized Trials. New York: Oxford Univ. PressMuthen B, Kaplan D, Hollis M. 1987. On structural equation modeling with data that are not missing

completely at random. Psychometrika 52:431–62Muthen LK, Muthen BO. 2007. Mplus User’s Guide. Los Angeles, CA: Muthen & Muthen. 4th ed.Neale MC, Boker SM, Xie G, Maes HH. 1999. Mx: Statistical Modeling. Richmond: Virginia Commonwealth

Univ. Dept. Psychiatry. 5th ed.Nunnally JC. 1967. Psychometric Theory. New York: McGraw-Hill


Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


Olsen MK, Schafer JL. 2001. A two-part random-effects model for semicontinuous longitudinal data. J. Am.Stat. Assoc. 96:730–45

Raghunathan TE. 2004. What do we do with missing data? Some options for analysis of incomplete data.Annu. Rev. Public Health 25:99–117

Raghunathan TE, Grizzle J. 1995. A split questionnaire survey design. J. Am. Stat. Assoc. 90:54–63Raudenbush SW, Bryk AS. 2002. Hierarchical Linear Models. Thousand Oaks, CA: Sage. 2nd ed.Rubin DB. 1976. Inference and missing data. Biometrika 63:581–92Rubin DB. 1987. Multiple Imputation for Nonresponse in Surveys. New York: WileySAS Institute. 2000–2004. SAS 9.1.3 Help and Documentation. Cary, NC: SAS Inst.Satorra A, Bentler PM. 1994. Corrections to test statistics and standard errors in covariance structure analysis.

In Latent Variables Analysis: Applications for Developmental Research, ed. A von Eye, CC Clogg, 1:399–419.Thousand Oaks, CA: Sage

Schafer JL. 1997. Analysis of Incomplete Multivariate Data. New York: Chapman & HallSchafer JL. 1999. Multiple imputation: a primer. Stat. Methods Med. Res. 8:3–15Schafer JL. 2001. Multiple imputation with PAN. In New Methods for the Analysis of Change, ed. LM Collins,

AG Sayer, 1:357–77. Washington, DC: Am. Psychol. Assoc.Schafer JL, Graham JW. 2002. Missing data: our view of the state of the art. Psychol. Methods 7:147–77Schafer JL, Olsen MK. 1998. Multiple imputation for multivariate missing data problems: a data analyst’s

perspective. Multivar. Behav. Res. 33:545–71Schafer JL, Yucel RM. 2002. Computational strategies for multivariate linear mixed-effects models with

missing values. J. Comput. Graph. Stat. 11:437–57Tanner MA, Wong WH. 1987. The calculation of posterior distributions by data augmentation (with discus-

sion). J. Am. Stat. Assoc. 82:528–50von Hippel PT. 2004. Biases in SPSS 12.0 Missing Value Analysis. Am. Stat. 58:160–64Willett JB, Sayer AG. 1994. Using covariance structure analysis to detect correlates and predictors of individual

change over time. Psychol. Bull. 116(2):363–81Wothke W. 2000. Longitudinal and multigroup modeling with missing data. In Modeling Longitudinal and

Multiple-Group Data: Practical Issues, Applied Approaches, and Specific Examples, ed. TD Little, KU Schnabel,J Baumert, 1:219–40. Hillsdale, NJ: Erlbaum

576 Graham

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.

AR364-FM ARI 11 November 2008 15:42

Annual Review ofPsychology

Volume 60, 2009Contents

Prefatory

Emotion Theory and Research: Highlights, Unanswered Questions,and Emerging IssuesCarroll E. Izard � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 1

Concepts and Categories

Concepts and Categories: A Cognitive Neuropsychological PerspectiveBradford Z. Mahon and Alfonso Caramazza � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �27

Judgment and Decision Making

Mindful Judgment and Decision MakingElke U. Weber and Eric J. Johnson � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �53

Comparative Psychology

Comparative Social CognitionNathan J. Emery and Nicola S. Clayton � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �87

Development: Learning, Cognition, and Perception

Learning from Others: Children’s Construction of ConceptsSusan A. Gelman � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 115

Early and Middle Childhood

Social Withdrawal in ChildhoodKenneth H. Rubin, Robert J. Coplan, and Julie C. Bowker � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 141

Adulthood and Aging

The Adaptive Brain: Aging and Neurocognitive ScaffoldingDenise C. Park and Patricia Reuter-Lorenz � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 173

Substance Abuse Disorders

A Tale of Two Systems: Co-Occurring Mental Health and SubstanceAbuse Disorders Treatment for AdolescentsElizabeth H. Hawkins � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 197

vii

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


Therapy for Specific Problems

Therapy for Specific Problems: Youth Tobacco CessationSusan J. Curry, Robin J. Mermelstein, and Amy K. Sporer � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 229

Adult Clinical Neuropsychology

Neuropsychological Assessment of DementiaDavid P. Salmon and Mark W. Bondi � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 257

Child Clinical Neuropsychology

Relations Among Speech, Language, and Reading DisordersBruce F. Pennington and Dorothy V.M. Bishop � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 283

Attitude Structure

Political Ideology: Its Structure, Functions, and Elective AffinitiesJohn T. Jost, Christopher M. Federico, and Jaime L. Napier � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 307

Intergroup relations, stigma, stereotyping, prejudice, discrimination

Prejudice Reduction: What Works? A Review and Assessmentof Research and PracticeElizabeth Levy Paluck and Donald P. Green � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 339

Cultural Influences

Personality: The Universal and the Culturally SpecificSteven J. Heine and Emma E. Buchtel � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 369

Community Psychology

Community Psychology: Individuals and Interventions in CommunityContextEdison J. Trickett � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 395

Leadership

Leadership: Current Theories, Research, and Future DirectionsBruce J. Avolio, Fred O. Walumbwa, and Todd J. Weber � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 421

Training and Development

Benefits of Training and Development for Individuals and Teams,Organizations, and SocietyHerman Aguinis and Kurt Kraiger � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 451

Marketing and Consumer Behavior

Conceptual ConsumptionDan Ariely and Michael I. Norton � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 475

viii Contents

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.


Psychobiological Mechanisms

Health Psychology: Developing Biologically Plausible Models Linkingthe Social World and Physical HealthGregory E. Miller, Edith Chen, and Steve Cole � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 501

Health and Social Systems

The Case for Cultural Competency in Psychotherapeutic InterventionsStanley Sue, Nolan Zane, Gordon C. Nagayama Hall, and Lauren K. Berger � � � � � � � � � � 525

Research Methodology

Missing Data Analysis: Making It Work in the Real WorldJohn W. Graham � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 549

Psychometrics: Analysis of Latent Variables and Hypothetical Constructs

Latent Variable Modeling of Differences and Changes withLongitudinal DataJohn J. McArdle � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 577

Evaluation

The Renaissance of Field Experimentation in Evaluating InterventionsWilliam R. Shadish and Thomas D. Cook � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 607

Timely Topics

Adolescent Romantic RelationshipsW. Andrew Collins, Deborah P. Welsh, and Wyndol Furman � � � � � � � � � � � � � � � � � � � � � � � � � � � � 631

Imitation, Empathy, and Mirror NeuronsMarco Iacoboni � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 653

Predicting Workplace Aggression and ViolenceJulian Barling, Kathryne E. Dupre, and E. Kevin Kelloway � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 671

The Social Brain: Neural Basis of Social KnowledgeRalph Adolphs � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 693

Workplace Victimization: Aggression from the Target’s PerspectiveKarl Aquino and Stefan Thau � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 717

Indexes

Cumulative Index of Contributing Authors, Volumes 50–60 � � � � � � � � � � � � � � � � � � � � � � � � � � � 743

Cumulative Index of Chapter Titles, Volumes 50–60 � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 748

Errata

An online log of corrections to Annual Review of Psychology articles may be found athttp://psych.annualreviews.org/errata.shtml

Contents ix

Ann

u. R

ev. P

sych

ol. 2

009.

60:5

49-5

76. D

ownl

oade

d fr

om a

rjou

rnal

s.an

nual

revi

ews.

org

by P

enns

ylva

nia

Stat

e U

nive

rsity

on

08/1

8/10

. For

per

sona

l use

onl

y.

Documents

Missing Data Analysis: Making It Work in the Real World · Annu. Rev. Psychol. 2009.60:549-576. Downloaded from arjournals.annualreviews.org by Pennsylvania State University on 08/18/10