Multilevel Factor Analytic Models

8/13/2019 Multilevel Factor Analytic Models

1/21


2/21

238 L. Diya, B. Li, K. Van den Heede, W. Sermeus and E. Lesaffre

are used as proxies of these constructs; for instance a single adverse event, say mortality, can

be used to represent patient safety. However, univariate analyses fall short in addressing the

multi-dimensional facets of most healthcare measures (Diya et al., 2011); hence multivariate

responses can be used. The relationship between a multivariate response vector and covariates

of interest can then be established by using a multivariate analysis. A special kind of multivariate

analysis which is gaining ground in the field of health outcomes and nursing research is factoranalysis (FA) (Lake, 2002; Bruyneelet al., 2009; Gajewskiet al., 2010). FA is used to quantify

the relationship between a multivariate response vector and a latent construct (or its domains),

i.e. in an FA model the multivariate response vector is regressed on latent variables and covari-

ates. Examples of the use of FA can be found in nursing research (Lake, 2002), bioinformatics

(Hochreiteret al., 2006), psychology (Norris and Lecavalier, 2010) and educational research

(Naicker, 2010).

In this paper the healthcare measure of interest is patient safety. Patient safety is defined as

the absence of the potential for, or occurrence of, health care associated injuries to patients that occursby avoiding medical errors as well as by taking actions to prevent errors from causing injury

(Agency for Healthcare Research and Quality, 2004). Since patient safety is a latent construct,

nurse-reported adverse events obtained from the Belgian chapter of the Europe Nurses Forecast-

ing Survey will be used as its proxy. Adverse events are injuries caused by medical management

rather than by the underlying disease conditions (Van den Heede, 2008). In the survey several

adverse events are recorded from nurses (nested within nursing units within hospitals), i.e. not

only do we have multivariate responses but also the data are multilevel in structure. Neglecting

the multilevel structure of the data in the analysis can be problematic since the fundamental

independence assumption underlying many commonly used statistical techniques, including FA

techniques, is violated (Goldstein and McDonald, 1988). When the degree of dependence is sub-

stantial, violation of the independence assumption biases parameter estimates (and standarderrors) and affects the power of statistical tests (Bliese and Hanges, 2004; Kenny and Judd,

1986). Using wrongly single-level FA may lead to the atomistic fallacy, which is incorrectly

assuming that the relationship between variables observed at the observational level also holds

at the cluster level. Aggregating the data at the cluster level does lead to independent observa-

tions but at the risk of committing the ecological fallacy, which is incorrectly assuming that the

relationship between variables at the cluster level also holds at the observational level (Robinson,

1950). So, to minimize the chances of these fallacies and to attain valid inferences for multilevel

data, models which take the multilevel structure of the data into account should be considered

(Longford and Muthen, 1992; Ansari and Jedidi, 2000; Goldstein and Browne, 2005; Grilli and

Rampichini, 2007). Another consequence of neglecting the multilevel structure of the data inFA is that an invalid factor structure can be proposed and the interpretation of the inferences

in FA is level dependent.

It might be useful to check that there is a multilevel structure in the data before embarking

on the more complex multilevel FA modelling. Univariate techniques such as the intraclass

correlation coefficient ICC or the F-discrepancy measure that was proposed by Yan and

Sedransky (2007), in a Bayesian context, can be used to verify the multilevel structure quickly.

ICC is the ratio of the variation between the clusters (level 2 units) and the total variation of

the observed or latent (for categorical observed responses) response. ICC-values range from

0 to 1 with larger values indicating greater variability between clusters compared with within-

cluster variability. As a rule of thumb, when a response has ICC>0:10 then we can argue that

there is enough level 2 variability to warrant a multilevel analysis (Dyer et al., 2005; Gajewski

et al., 2010). The F-discrepancy measure that was proposed by Yan and Sedransky (2007)


3/21

Multilevel Factor Analytic Models 239

uses analysis-of-variance (ANOVA) concepts in a Bayesian context to diagnose the need to

account for a hierarchical structure. This tool may be more interesting in multilevel data sets

like the Europe Nurse Forecasting Survey data with more than two levels of analysis. However,

in the nurse survey data interest is on a multivariate response; hence the F-discrepancy mea-

sure neglects the dependence between responses. We propose here to extend this measure to the

multivariate setting by making use of multivariate analysis-of-variance (MANOVA) concepts.The objective of this paper is to fit a multilevel FA model to establish the relationship between

patient safety and nurse-reported adverse events from the Belgian nurse survey data, and to

develop Bayesian diagnostic techniques for detecting hierarchical structures in multivariate

data sets.

The remainder of this paper is organized as follows. Section 2 describes the Europe Nurse

Forecasting Survey data set, with particular emphasis on the Belgian chapter. A brief review

of FA is presented in Section 3 and in Section 4 model checking is presented. In Section 5 the

application of the Bayesian diagnostic techniques on the Belgian nurse survey data is presented.

The paper ends with a discussion in Section 6. Details pertaining to a simulation study to com-

pare the MANOVA and the ANOVA discrepancy measures, and details of the modelling fittingfor FA models (MPLUS and JAGS codes) are provided in the on-line supplemental document

and available from

http://www.blackwellpublishing.com/rss

2. The Europe Nurse Forecasting Survey data set

The Europe Nurse Forecasting project aims at introducing innovative forecasting methods by

addressing not only volumes but also quality of staff and its effect on patient care. The project

involves researchers from 12 European countries, the USA, China, South Africa and Botswana.

Data collection was only done in 12 European countries, namely Belgium, Finland, Germany,Greece, Ireland, Poland, Spain, Sweden, Switzerland, the Netherlands, the UK and Norway.

Most of the data were collected in 2009 and 2010.

In the project two multicountry surveys were conducted, namely the nurse survey and patient

satisfaction survey. Hospital discharge data were also collected. The various data sets were

targeted at investigating how nursing qualifications, demographics, workload, wellbeing and

practice environment can affect productivity, patient safety and patient outcomes.

This research focuses on the Belgian chapter of the nurse survey data. It contains information

about the nature of a nurses job (e.g. nursephysician relationships), quality and safety issues,

the nurses most recent shift and demographics. The nurse survey seeks to establish the perception

of nurses with regard to their work conditions and their relationship with either patients or otherhealthcare professionals. In particular the focus is on relating nurse-reported adverse events to

the characteristics of the nurse, nursing unit and hospital.

The main question that we looked at here was How often would you say each of the fol-

lowing incidents occurs involving you or your patients?. For this question six incidents were

considered:

(a) patients received the wrong medication, time or dose,

(b) pressure ulcers after admission,

(c) patient falls with injury,

(d) urinary tract infections,(e) bloodstream infections and

(f) pneumonia.


4/21


The main question (for each incident type) had seven potential answers measured on a

Likert scale, i.e. 0, never, 1, a few times a year or less, 2, once a month or less, 3, a few

times a month, 4, once a week, 5, a few times a week, and 6, every day. In this paper new

responses were generated by discretizing the Likert-scaled variables because of software con-

siderations. (It is currently not feasible to implement the proposed latent variable or data

augmentation approach for categorical responses with more than two categories in JAGS.)A nurse was judged to have indicated that his or her nursing unit had a substantial num-

ber of events if he or she selected the values greater than 2. This led to binary responses,

namely

(a) wrong medication, time or dose (wrong),

(b) pressure ulcers (pressure),

(c) patient falls with injury (falls),

(d) urinary tract infections (uti),

(e) bloodstream infections (infections) and

(f) pneumonia.

The data that are considered here are multilevel in structure, i.e. nurses are nested within

nursing units which are in turn nested within hospitals. The data consisted of 3151 nurses

from 269 nursing units in 66 hospitals. 23 and 43 hospitals respectively were and were not

technologically advanced and four and 62 hospitals respectively were and were not univer-

sity hospitals. Of the 269 nursing units, 112 were surgical units, 125 were medical units and

32 were mixed surgical and medical units. Of the nurses who responded to the nurse survey,

1333 worked in surgical units, 1471 worked in the medical units and 347 worked in mixed

surgicalmedical units. 2167 of the nurses reported Flemish as their language of communica-

tion whereas 984 reported French. The covariates of interest in this paper were measured at

the nurse level, nursing unit level and hospital level. At the hospital level the covariates werethe number of beds (beds), whether the hospital is technologically advanced (tech = 1; other-

wise tech = 0) and whether the hospital was a university hospital (university = 1; otherwise

university = 0). At the nursing unit level the covariate of interest was the unit specification (1,

surgical, 2, medical, and 3, mixed). At the nurse level the covariates considered were gender

(male = 1 if gender is male; 0 otherwise), age, time in years as a registered nurse (expn), time

in years at the current hospital (exph), language (language = 1 if Flemish; 0 otherwise) and

whether a nurse was satisfied with his or her career choice (satisfied = 1 if a nurse was satisfied;

0 otherwise).

In this paper we shall focus on the nurse and the nursing unit level. The hospital level (third

level) is dropped because empirical evidence did not support its inclusion, i.e. there was lowvariability at the hospital level; see Section 5.2.2. Thus the role of hospitals is explored in a

non-hierarchical way.

3. Factor analysis models

3.1. Single-level factor analysis

We first consider a single-level FA model for continuous responses. Let Yj= .Yj1, . . . ,Yjp/be a

p-dimensional continuous response vector forthejth individual (j=1 , . . . ,N),Fj= .Fj1, . . . ,Fjk/

be a k-dimensional (kp) vector of latent constructs and be apkmatrix of factor loadings;

then the single-level FA model can be written as

Yj=j+Fj+j, .1/


5/21


where j= .j1, . . . ,jp/ is the mean vector, the common factors are multivariate normally

distributed with mean 0 and covariance matrix .FjNk.0,// and the unique factors are

multivariate normally distributed with mean 0 and covariance matrix (jNp.0,/). Note

that the mean of thesth (s=1 , . . . ,p) responseYjs can be modelled as a function of covariates,

i.e. js=Xjs. Factor loadings (elements of) are parameters which relate the unobserved

factors to the observed responses, i.e., the larger the value of a factor loading, the more thecorresponding response is said to load on the corresponding factor. Factor loadings are allowed

to vary across survey items but not across individuals. Assuming orthogonality of all common

and unique factors, the marginal covariance matrix is given by var.Yj/=U=+, i.e.

the observed covariances matrix can be decomposed into a component that is attributable to

the underlying factors and measurement error.

Two special cases of FA are

(a) ifFjis known then the FA model can be viewed as a multivariate regression model with

the factor loadings playing the role of regression coefficients and

(b) if the factor loadings are known then the FA model is equivalent to a two-level random-

slope model.

There are two modes of FA, namely exploratory FA (EFA) and confirmatory FA (CFA).

EFA is used when there is no a priori information about the factor structure, i.e. choosing

the number of factors, k, and establishing the variables that are related to a given factor

domain. In that case all factor loadings are free parameters. In both the frequentist and

the Bayesian contexts, EFA can be done by fitting several models with varying numbers of

factors and using goodness-of-fit indices (discrepancy measures) to select the best model (see

Section 4.2). In a Bayesian context, Lopes and West (2004) advocated the use of the reversible

jump Markov chain Monte Carlo (MCMC) approach. The ultimate objective of EFA is to

come up with the best factor structure for a given data set. In contrast, CFA is used tovalidate a priori hypothesized factor structures. In the literature, the most common CFA

structure is the simple factor structure, i.e. the factor loadings are allowed to load on one

and only one latent construct and all the other factor loadings (cross-loadings) are set equal to

0.

The above FA model caters for continuous responses assumed to be multivariate normally

distributed. For categorical or binary responses, we could assume that they are discretized

versions of latent Gaussian variables, i.e. consider the binary response vector given by Yj; then

Yjs = I.Yjs>0/ where Y

js, s= 1 , . . . ,p, is a latent Gaussian response. For identifiability, the

variance of the error term ("js) for the sth response, ss, is set to 1 such that all the diago-

nal elements of are 1. The matrix is called the polychoric or tetrachoric correlation matrixas it is the correlation matrix of the underlying latent Gaussian responses.

Additional constraints are required for both categorical and continuous response FA models

to be identifiable. For instance, for the single-level FA in equation (1) there arep.p+1/=2 unique

elements in but as many aspk+k.k+1/=2+p.p+1/=2 elements in+, implying that

there are more parameters to estimate than availableinformation. In this paper we shall constrain

the measurement error for different responses to be independent (off-diagonal elements of

equal to 0) and the common factors to be orthogonal. In CFA, an additional constraint that is

employed is to adopt a simple factor structure, i.e. a variable loads on only one factor, whereas

in an EFA this constraint is relaxed.

FA models can be fitted by using maximum likelihood estimation or (robust) weighted least

squares estimation; see Longford (1993). From equation (1), the total marginal likelihood of

the data is


6/21


L.,,,|Y/=

.2/pN=2|U|

N=2 exp

1

2

Nj=1

.Yj/1U .Yj/

f.Fj/dFj, .2/

where=X. For binary or categorical data the (latent) tetrachoric or polychoric correlation

matrix can be used. Computational strategies for obtaining maximum likelihood estimates have

been discussed in Mardia et al. (1979) and Lawley and Maxwell (1971). Another estimationtechnique which can be used is weighted least squares.

Alternatively, one could estimate the parameters of FA models by using the Bayesian

approach. Estimation is done via sampling using MCMC techniques. In the Bayesian approach

the likelihood is combined with prior information to obtain the posterior distribution of the

parameters of interest, i.e.

p.,,,|Y/L.,,,|Y/p.,,,/, .3/

where p.,,,/ is the prior distribution. For an FA on binary responses, we used in this

paper the latent variable technique called the data augmentation approach (Albert and Chib,

1993; Azevedo etal., 2011), where a latent Gaussianresponse vector is assumed that is discretized

to yield the binary responses. The complete marginal data likelihood function for the binary

single-level FA model is given by

L.,,,|Y/

Nj=1

ps=1

.Yjs;js; 1/.Y

js|Yjs/f.Fj/f.Y

/dFjdY, .4/

where

.Yjs|Yjs/= I.Yjs>0/I.Yjs=1/+ I.Yjs0/

I.Yjs=0/

is the conditional likelihood..Yjs;js; 1/ represents the value of the normal probability density,

with meanjs and standard deviation SD= 1, at the point Yjs with

js being the conditional

mean, i.e. js = js + sFj, where s is the sth row of. Note that the distribution of the

multivariate response vector can now be presented as a product of univariate distributions

because responses are conditional independent (conditional on the common factor).

The EFA models in this paper used the frequentist approach and were implemented by using

MPLUS (Muthen and Muthen, 2011). A frequentist approach was necessary since methods

for Bayesian covariance selection are scarce and computationally intense. CFA models can be

easily fitted by using both the frequentist and the Bayesian approaches (using link functions) in

MPLUS. However, these CFA models are restricted in term of flexibility. Hence, the CFA modelsin this paper were fitted by using the Bayesian approach, i.e. we used the data augmentation

approach called the latent variable approach in JAGS (Plummer, 2003). In particular, JAGS

was run from R (R Development Core Team, 2012) by using the R2jags (Su and Yajima, 2012)

package. The R packageCODAwas used to assess MCMC convergence (Plummeret al., 2006).

In this paper we restricted the convergence diagnosis to the trace plots and the Gelman, Brooks

and Rubin diagnostic R. The quick mixing of MCMC chains and values ofR close to 1 (say

R


7/21


in a CFA the cross-loadings can be given priors concentrated around 0 instead of fixing them

to 0.

The data augmentation approach has interesting features because it replaces the binary

response with a latent continuous response; therefore statistical techniques which are used for

continuous responses can be applied to the latent continuous response. In addition, it allows

quite general link functions by replacing the classical link functions (e.g. normal distributions(probit link)) by quite general forms of distributions (e.g. skewed normal distributions).

3.2. Multilevel factor analysis

When the data have a multilevel structure, multilevel FA approaches are needed. This implies

that the multivariate continuous response vector for an individual has attributes emanating

from the observational and the cluster levels. Let Yij= .Yij1, . . . ,Yijp/ be a p-dimensional

continuous response vector for the jth individual (j= 1 , . . . ,ni) in cluster i (i= 1 , . . . ,m),

F.1/ij = .F

.1/ij1 , . . . ,F

.1/ijk1

/be ak1-dimensional (k1p) vector of latent constructs at level 1,F.2/i =

.F.2/

i1, . . . ,F

.2/

ik2/ be a k

2-dimensional (k

2p) vector of latent constructs at level 2,.1/ be a pk

1matrix of factor loadings at level 1 and.2/ be apk2matrix of factor loadings at level 2; then

the multilevel FA model is given by

Yij=ij+.1/F.1/ij +

.1/ij +

.2/F.2/i +

.2/i , .5/

where ij is the mean vector. The vector of unique factors or measurement errors at the

observational level is multivariate normally distributed with mean 0 and variance .1/ (.1/ij

Np.0,.1//), the vector of unique factors or measurement errors at the cluster level is multi-

variate normally distributed with mean 0 and variance .2/ (.2/i Np.0,

.2//), the vector of

common factors at the observational level is multivariate normally distributed with mean 0

and variance .1/

(F

.1/

ij Nk1.0,.1/

/) and the vector of common factors at the cluster levelis multivariate normally distributed with mean 0 and variance .2/(F

.2/i Nk1.0,

.2//).

Assuming orthogonality of all common and unique factors, the marginal covariance matrix

consists of a within and between component, var.Yij/=M=W +B. The within-covariance

matrix is given by W =.1/

.1/

.1/+.1/ and the between-covariance matrix is given by

B =.2/

.2/

.2/+.2/. Note that both the within and the between covariances have a com-

ponent coming from the common factors (factor scores) and unique factors (measurement

errors or random effects). Multilevel FA can be estimated by using the frequentist approach

but this requires multi-dimensional integration to obtain a marginal likelihood which will then

be optimized to obtain parameters of interest. The Bayesian approach will bypass the multi-

dimensionalintegration by usingsampling-based algorithms(Ansari and Jedidi, 2000; Goldsteinand Browne, 2005).

4. Model checking

After the model has been fitted it is necessary to monitor the quality of the statistical model. This

implies that we are interested in detecting whether there are any systematic differences between

the model and the observed data. In a frequentist context this entails checking whether certain

functions of the data, test statistics, follow the distribution that is implied by the model. In a

Bayesian context, posterior predictive checks (PPCs) are used. PPCs are obtained by generating

replicated data sets from the posterior predictive distribution of the fitted model (Gelmanet al.,

2004). These data sets are then compared with the observed data set with respect to the feature of

interest by using discrepancy measures. We first discuss how to check for the multilevel structure


8/21


in Section 4.1, followed by model assessment (e.g. normality of both the common and the unique

factors, and the orthogonality of the common and unique factors) in Section 4.2.

4.1. Detecting the multilevel structure

The ICC-values can be used to assess informally whether there is substantial dependence to

warrant the use of multilevel models. In contrast, Yan and Sedransky (2007) proposed a Bayesian

approach for checking a multilevel structure in the data by making use of anF-statistic inspired

by classical ANOVA concepts. When there are two or more responses the approach of Yan and

Sedransky (2007) neglects to take the dependence between responses into account, thus leading

to a reduction in power to detect hierarchical structures. We propose the use of diagnostic checks

based on MANOVA.

To test for the independence assumption (no hierarchical structure) the relative sizes of the

between (or total) and within sums of squares and cross-products are considered. In a frequentist

context, the following test statistics are considered:

(a) the LawleyHotelling trace,LH = tr.BW1/;

(b) Pillais trace,PT = tr.BT1/;

(c) Wilks , WL =|W|=|T|.

Here the total, between and within sums of squares and cross-products are given by T=

mi=1

nij=1 .Yij

Y/.YijY/ ,B=mi=1ni.Yi

Y/.YiY/ andW=mi=1

nij=1.Yij

Yi/.Yij

Yi/ respectively.

Assuming multivariate normality of the responses, all test statistics are asymptotically equiv-

alent. When comparing two clusters all the test statistics reduce to a statistic which follows a

Hotelling T2 sampling distribution. Wilks WL is the most frequently used MANOVA test

statistic as it is related to the likelihood ratio criterion and its exact sampling distribution

can be derived for some special cases; for example see chapter 6 of Johnson and Wichern(2002).

In this paper we propose to use these test statistics as discrepancy measures in a Bayesian

context. The posterior predictiveP- (PPP-) values for the three multivariate discrepancies are

computed by taking the proportion of times that the predicted discrepancy exceeds the observed

discrepancy. If only the data structure is misspecified and not other model assumptions then

our approach does indeed diagnose the hierarchical structure. In cases where other model

assumptions do not hold then the discrepancy measures proposed might indicate a statistically

important difference even when the between-cluster variability is low. In such cases our

approach acts as a global goodness-of-fit check. Caution must be exercised when using PPCs

or PPP-values since the data are used twice, i.e. in fitting the model and in model checking,hence leading to some degree of bias (Bayarri and Castellanos, 2004). This can be avoided by

using a learning and a validation data set, where the learning data set is used to estimate the

model and a validation data set is used to assess the model fit. We employed this approach in

this paper.

The approach proposed can be applied only to continuous multivariate responses. It cannot

be directly applied to categorical multivariate responses, i.e. when using link functions, but it

can be applied to the latent continuous response vector when the data augmentation approach

is used.

4.2. Model assessment

After having checked the multilevel structure in the data, we can either select the single-level


9/21


or multilevel FA depending on the evidence that is presented. The next step entails model

assessment, for instance checking the normality assumption of both the common and the unique

factors, and validating the orthogonality assumption. In this paper we shall adhere to global

model checks using PPCs in a Bayesian context and variants of the 2 likelihood ratio statistic

in the frequentist context.

Recall that the general goal of an FA model is to test the hypothesis that the observed covari-ance matrix is equal to the model-based covariance matrix, i.e.

S=./, .6/

where S represents the covariance matrix of the observed response vector (a latent continu-

ous response vector for categorical data) and ./ represents the model-based covariance

matrix.

In a frequentist context, a multitude of goodness-of-fit indices have been proposed. In this

paper we shall look at three indices which are functions of the 2 likelihood ratio statistic, i.e.

the2 likelihood ratio statistic, the comparative fit index CFI (Bentler, 1990) and the Tucker

Lewis index TLI (Tucker and Lewis, 1973) and two indices which are based on how well agiven model approximates the true model, i.e. the root mean square of approximation RMSEA

(Browne and Cudeck, 1992; Steiger and Lind, 1980) and the standardized root-mean-square

residual SRMSR. These indices check whether the model-based covariance matrix is close to

the covariance matrix of the observed response vector. If the model fits the data well then

(a) the2 likelihood ratio statistic should be small,

(b) both CFI and TLI should have a value 1 and

(c) RMSEA and SRMSR should be close to 0.

As a rule of thumb, a model with CFI- and TLI-values of at least 0.90 are deemed acceptable

and values of at least 0.95 are indicative of a good fit. Alternatively, we could use RMSEA andSRMSR for which values less than 0.08 indicate an acceptable model fit and values less than

0.05 are indicative of a good fit (Hu and Bentler, 1999). Multiple fit indices are used because

each fit index has its own limitations and there is no agreed-on method for evaluating whether

the lack of fit of a model is substantively important. These indices are used for both single and

multilevel FA models.

In a Bayesian context, model assessment can focus on the likelihood and the priors. PPCs

are used to check how well the model fits the data; however, the discrepancy measures are now

different from those used in diagnosing the neglected hierarchical structure. From these PPCs,

PPP-values can be computed. In this paper the2 discrepancy measure (Gelman et al., 1996)

was used. The deviance information criterion DIC (Spiegelhalter et al., 2002) can be used tocompare models on the basis of different likelihood and prior specifications. The model having

the smallest DIC-value is then a good candidate; see Spiegelhalter et al.(2002). However, this

measure can be misleading as latent variables, e.g. factor scores and randomeffects, are treated as

parameters. This has led to numerous discussions in the literature with no conclusive suggestions

(Celeux et al., 2006); hence we shall use the mean-square prediction error MSPE instead. MSPE

measures how well the model fits the data and is given by

MSPE.Y/=E

mi=1

n1j=1

.Yij Yij/2

,

where Yij

is the posterior predicted response for nurse jin clusteri. Note that all model assess-

ments are carried out on a validation data set (independent of the learning data set that is used

to fit the model).


10/21


5. Application to Europe Nurse Forecasting Survey data

In this section, we want to establish the relationship between the latent construct patient safety

and six nurse-reported adverse events in the Belgian chapter of the Europe Nurse Forecasting

Survey. Hereby we wish to control for nurse, nursing unit and hospital characteristics, i.e. nurses

are seen as informants of not only their own patient safety record but also informants of patient

safety culture in their nursing unit. The response of interest is a multivariate vector of nurse-

reported adverse events, i.e.Yij=.Yij1, . . . ,Yij6/whereYij1 denotes wrong medication, or dosage

or time,Yij2 denotes pressure ulcers,Yij3patient falls,Yij4 urinary tract infections,Yij5blood-

stream infections andYij6 pneumonia.

Since no a-priori-specified factor structure is proposed for the adverse events in this paper

and patient safety, there is a need to explore the factor structure. So, to avoid double usage of

the data, the data were equally divided into the learning and validation data sets by using simple

random sampling (within a cluster). The factor structure was explored by using the learning

data set and the validation data set was used in confirming the factor structure. The learning

data set was then reused in the model assessment for the CFA model.

EFA was implemented by using a frequentist approach by making use of MPLUS since

the Bayesian approach is computationally intense. The Bayesian module of MPLUS is not

sufficiently flexible to allow for the computations of the discrepancy measures that are proposed

in this paper; hence CFA models were fitted in JAGS by usingR2jags.

5.1. Descriptive statistics

The descriptive statistics in this section pertain to the entire (learning plus validation) Belgian

nurse survey data. The learning and validation data sets had similar characteristics (as they are

random samples from the entire data set).

Table 1 shows the percentage of nurse-reported adverse events, the number of missingresponses and ICC. From Table 1, there are a few missing values for all the responses. The most

common nurse-reported adverse event is urinary tract infection, followed by wrong medication,

or time or dose and the least reported is patient fall accidents. The ICC-values are all greater

than 0.1, indicating the possible need for a multilevel FA model.

The Spearman correlation coefficient matrix of the percentages in a nursing unit of nurse-

reported adverse events shows that there were moderate-to-high correlations between the

nurse-reported adverse events. The largest correlation was between bloodstream infections and

pneumonia and the smallest correlation was between wrong medication, or time or dose and

pneumonia:

Table 1. Belgian nurse survey data:percentage of nurse-reportedadverseeventsand ICC

Type of adverse event %adverse event Number missing ICC

Wrong medication, dose or time 20.55 70 0.11Pressure ulcers 11.60 98 0.13Falls 9.08 89 0.19Urinary tract infection 21.54 110 0.21Infection 9.66 160 0.18Pneumonia 11.40 151 0.27

Calculated for the non-missing responses.


11/21


University hospitals were more likely to have frail patients compared with the non-university

hospitals. Thus this might lead to a higher incidence of nurse-reported adverse events. University

hospitals had higher numbers of nurse-reported adverse events than non-university hospitals

except for wrong medication, or dose or time and patient fall accidents.

5.2. Factor analysis

In this section we present the results from FA. All EFA models (single level and multilevel) were

implemented on the learning data set by using a frequentist approach in MPLUS. The use of

a frequentist approach here was necessitated by the fact that methods for Bayesian covariance

selection approaches are scarce in the literature, no software is available and these methods are

computationally intense. Since we are assuming latent Gaussian responses and the responses are

binary a probit link was assumed and parameter estimates were obtained by using the maximum

likelihood approach.

The Bayesian approach was used to fit all CFA models (single level and multilevel). In

particular the latent variable approach was used, which enabled the easy computation ofMANOVA discrepancy measures which are instrumental in detecting hierarchical structures

in the data.

5.2.1. Single-level factor analysis

5.2.1.1. Single-level exploratory factor analysis. Here we explored the factor structure, i.e.

the number of factors and the outcomes on which the chosen factors load, by using EFA in

MPLUS. It is customary in EFA that one starts by aiming for a simple factor structure and

on doing a CFA modify the structure accordingly. Fig. 1(a) shows a scree plot of the single-

level EFA. The scree plot indicates that two factors were adequate to account for much of thevariability in the six responses.

The two-factor solution was further supported by the results shown in Table 2, which gives

measures of fit for various factor structures. The measures of fit are all within acceptable limits;

see Section 4.2. Models with more than three factors were not identifiable as the number of

parameters to estimate exceeded the available information. We also looked at the factor loadings

to assess their classical statistical significance. On the basis of these results and on Fig. 1(a)

and Table 2 we concluded that a two-factor solution was a reasonably good single-level factor

structure for the Belgian nurse survey data.

Recall that, in Section 2, the responses were generated by discretizing the Likert-scaled vari-

ables by using 2 as a cut-off point. We carried out a sensitivity analysis by choosing other cut-off

points, i.e. 1, 3 and 4, and redoing the EFA. The proposed factor structures for all these cut-off

points were similar to the factor structure that is presented here. We also explored the factor


12/21


1 2 3 4 5 6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Number of factors

Variance

(Eigenvalue)

1 2 3 4 5 6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Number of factors

Variance

(Eigenvalue)

(a)

(b)

Fig. 1. Belgian nurse survey datascree plots for (a) the single-level and (b) the multilevel EFA: ,within; , between


13/21


Table 2. Belgian nurse survey data: measures of fit for single-level EFA

Number of 2 p-value CFI TLI RMSEA SRMRfactors

1


14/21


Table 3. Belgian nurse survey data:PPP-values for discrepancy measuresto diagnose hierarchical structures

Discrepancy PPP-value

LH 0.03PT 0.03WL 0.97

F1 0.14F2 0.58F3 0.12F4 0.12F5 0.66F6 0.29

include in the mean structure of the multilevel FA model. A covariate was selected for inclusion

in the FA model if it was significant at the 5% level for at least two adverse events. From this

exercise the following variables were considered:

(a) age,

(b) gender,

(c) satisfied,

(d) unit type (medical or mixed, with surgical being the default),

(e) university hospital and

(f) beds.

For computational reasons the continuous covariates were centred and standardized (SD=1).

5.2.2.1. Multilevel exploratory factor analysis. Using MPLUS a frequentist two-level EFA

was fitted to the learning data set. Here, the observational level consists of nurses who are

nested in the nursing unit (cluster level). Fig. 1(b) shows the scree plot for the multilevel factor

analysis. The scree plot indicates that two factors are needed at each level. The within-factor

structure remained the same as that suggested by the single-level EFA but, for the between-

factor structure, the factor indicators wrong medication, or time or dose and pressure ulcers

loaded on the first between-nursing unit factor whereas the remainder of the factor indicators

loaded on the second between-nursing unit factor, as shown in Fig. 2.

The two-factor solution was further supported by the measures of fit for the two-factor solu-tion at both levels. The 2 goodness-of-fitp-value was 0.5887, RMSEA (and the 95% confidence

interval) is given by 0.000 (0.000; 0.026), both CFI and TFI had a value of 1 and the within

and between SRMR are 0.018 and 0.057 respectively. Hence all the goodness-of-fit indices were

within acceptable limits. These goodness-of-fit statistics were better than for any other factor

solution.

5.2.2.2. Multilevel confirmatory factor analysis. To validate the factor structure pro-

posed (from multilevel EFA) we fitted a two-level (nurse and nursing unit levels) CFA model,

controlling for nurse, nursing unit and hospital characteristics, by using the data augmentation

approach in JAGS. We assumed similar priors to those for the single-level FA in Section 5.2.1.

With regard to the variance components for the random effects we used parameter expansion,


15/21


Fig. 2. Belgian nurse survey data: multilevel factor structure

i.e. we had a gamma distribution with scale and shape parameters equal to 0.5 as the prior for

the precision and a normal distribution with mean 0 and variance 106 for the parameter ex-

pansion term. The variances of the random effects were obtained through dividing the squared

parameter expansion term by the precision.

We considered three chains, each having 300000 MCMC iterations (with a burn-in of 50000

iterations). We thinned the chains by selecting one in 10 iterations. Hence, values in Table 4 are

based on 25000 sampled values. (This is so for all the models in this section, including models

used for sensitivity analysis.) The Gelman, Brooks and Rubins diagnostics for all parameters

were less than 1.1 and the MCMC chains mixed well, hence indicating no convergence problems.

Table 4 shows the posterior means (with SDs in parentheses) and the 95% credible intervals forthe factor loadings and the variance components of the second-level unique factors (random

effects).

Older nurses reported fewer adverse events than younger nurses, except for wrong medication,

or time or dose. Male nurses reported more incidences of wrong medication, or time or dose

than their female counterparts. Nurses who were satisfied with their career choice reported

fewer patient fall accidents and incidences of wrong medication, or time or dose. Flemish-

speaking nurses reported more incidences of urinary tract infections, bloodstream infections

and pneumonia than French-speaking nurses. There were no statistically important differences

between the reported adverse events in the surgical, medical and mixed units.

The factor loadings that are presented here were standardized and can be interpreted as

the correlation between the factor domain and the factor indicators (responses), i.e. the factor

loadings when the total variability (per response) is fixed to 1. All the posterior means of


16/21


Table 4. Belgian nurse survey data: results of the multilevel CFA model

Parameter Posterior 95% crediblemean (SD) interval

Within-level factor loadings.1/1,1

0.407 (0.053) (0.300; 0.508)

.1/1,2

0.947 (0.025) (0.889; 0.986)

.1/1,3 0.557 (0.054) (0.449; 0.659)

.1/2,4

0.828 (0.035) (0.756; 0.892)

.1/2,5 0.853 (0.037) (0.777; 0.921)

.1/2,6

0.804 (0.040) (0.722; 0.876)

Between-level factor loadings

.2/1,1

0.182 (0.098) (0.014; 0.375)

.2/1,2 0.224 (0.100) (0.023; 0.403).2/2,3

0.405 (0.102) (0.183; 0.574)

.2/2,4

0.226 (0.072) (0.082; 0.363)

.2/2,5

0.323 (0.067) (0.190; 0.450)

.2/2,6

0.381 (0.090) (0.201; 0.546)

Residual variances

21

0.117 (0.063) (0.015; 0.256)

22 1.117 (2.035) (0.012; 8.066)

23 0.226 (0.191) (0.008; 0.715)

24

1.652 (1.059) (0.453; 4.531)

25 0.199 (0.226) (0.008; 0.824)26 0.919 (0.724) (0.045; 2.786)

Fit indices

2 0.349 (0.477) (0.000; 1.000)MSPE 17206.897 (262.432) (17030.075; 17724.202)

.v/k,s is thekth standardized factor loading for variable s at level v.

2s is

the variance component for variable s. s: 1 denotes the wrong medication,or time or dose; 2 denotes pressure ulcers; 3 denotes patient falls; 4 denotesurinary tract infection; 5 denotes bloodstream infection; 6 denotes pneumoniaresponse.

the first-level factor loadings were greater than 0.4, implying that there was moderate-to-high

correlation between the factor domains and the factor indicators. The nursing unit level factor

loadings were between 0.18 and 0.41. Though these factor loadings were low to moderate they

were still statistically important. The 2 goodness-of-fit Bayesianp-value was 0.50, suggesting

that there was no major discrepancy between the data and the model. The model MSPE was

17206.897.

A sensitivity analysis was done by

(a) altering the common factor distribution to have a Studentt-distribution with 4 degrees

of freedom,

(b) altering the common factor distribution to have at-distribution with unspecified degrees

of freedom and


17/21


(c) fixing the first factor loading instead of the factor variance to 1.

The three models resulting from these alterations had larger MSPE-values than the initial

multilevel FA model, i.e. the MSPE-values were 17209.677, 17207.970 and 17207.180. We also

assessed whether there was substantial variability at the hospital level after accounting for the

nursing unit level. All the MANOVA discrepancy measures had values that were close to 0.5,hence indicating that there was not that much dependence between nurses in a hospital after

accounting for the nursing unit. This was used as evidence for not extending the FA model to

three levels (hospital level, nursing unit level and nurse level). Hence the effect of hospital was

explored in only a non-hierarchical way.

The within-factor structure was the same as that attained under the single-level FA. The

nurse-reported adverse events wrong medication, or time or dose, pressure ulcers and patient

fall accidents loaded on one factor whereas the remaining nurse-reported adverse events loaded

on another factor. This implied that nurses who reported that they encountered many incidences

of wrong medication, or time or dose were also more likely to report high numbers of pressure

ulcers and patient fall accidents (adhering to protocol and safety guidelines) whereas nurses

who reported that they encountered many incidences of urinary tract infections were also more

likely to have reported many incidences of bloodstream infections and pneumonia (hygiene or

hand hygiene). These three adverse events can also be considered as indicators of adherence to

safety guidelines, for instance, safe handling of perfusion lines (which if not done properly can

be a source of infection). However, these three adverse events are not as directly related to the

nursing care process as the adverse events which load on the first factor. We should note that the

nurses here were informants of not only their own experience but also that of the nursing unit

at large. The between-factor structure was reflective of what happens at the nursing unit level.

Wrong medication, or time or dose and pressure ulcers loaded on the first nursing unit factor,

and the remainder of the adverse events loaded on the second nursing unit factor. Clearly the

first factor is an indicator of whether the nursing unit administers medication correctly, in time

and is attentive to the physical need of immobile patients. The second factor is a mixture of,

for example, whether the right equipment is accorded to the patient and the general hygiene

condition of the nursing unit.

6. Discussion

In this paper we have illustrated how to address complex research questions by using FA. FA

models are used to relate the latent constructs to the responses of interest, e.g. adverse events.

These models are becoming popular in many fields, especially in health outcomes and nursingresearch where they are used to inform policy decisions. The data that are collected in health

outcomes and nursing research are often huge and for this reason, coupled with limitations

in available software, health outcome and nurse researchers tend to fit single-level FA models

based on simplistic assumptions, e.g. the independence assumption. However, most data in this

area of research are multilevel in structure; hence neglecting the data structure in the statistical

analysis can lead to invalid inferences which might lead to wrong policy recommendations.

Before embarking on complex multilevel FA modelling which is computationally intense there

should be an assessment of whether there is substantial dependence between subjects in the

same cluster. If the dependence between subjects in the same cluster is low the inference that is

obtained from the single-level FA is more likely to be valid, thus leading to simple conclusions,

i.e. we aim for parsimony, not only in the covariates that are included in these models but also in

the level of analysis. However, if there is substantial between-cluster variability, inferences from


18/21


the single-level FA will be invalid because the level of analysis has a bearing on the interpretation

of the factor structure and the inferences coming from such a structure.

We proposed the use of the MANOVA discrepancy measures to address formally whether

there is substantial dependence (between-cluster variability) in our multivariate multilevel struc-

tured data set. These measures are handy as they have more power than the ANOVA discrepancy

measures and intraclass correlation coefficients to diagnose the hierarchical structures. It is cru-cial that the factor structure is correctly specified when diagnosing the hierarchical structure

as misspecification of the factor structure implies that the discrepancy measure proposed will

then become sensitive not only to the hierarchical structure but also to the misspecified fac-

tor structure. Though it can be argued that all potential models, i.e. single-level and multilevel

models, can be fitted and then compared, for large data sets that are encountered in health

outcomes, nursing and epidemiological research (e.g. healthcare registers or healthcare admin-

istrative databases) this entails a huge computational burden. Instead a single-level FA model

can be fitted and from this model MANOVA discrepancies used to assess whether there is a need

to do multilevel FA. The MANOVA discrepancy measures that are presented in this paper are

not limited to FA models only but can also be used when other types of multivariate model areunder consideration. The MANOVA concepts can also be used in a frequentist context where

MANOVA test statistics can be used. A potential area for further research is the development

of diagnostic measures to check omitted non-hierarchical multilevel structures, for instance in

multiple-membership and cross-classified models.

One can easily implement simple FA models, using either the frequentist approach or the

Bayesian approach, in MPLUS. However, with regard to the Bayesian approach, the MPLUS

Bayesian module is quite limited, i.e. MPLUS cannot fit multilevel FA models with more than

two levels and it is not feasible to implement the latent variable approach or to obtain dis-

crepancy measures. In this paper the EFA employed the frequentist approach as the Bayesian

approach (Lopes and West, 2004) is computationally intensive. From a practical point of viewthe EFA is easy to implement in a frequentist context compared with the Bayesian context. Even

researchers with limited methodological comprehension can implement EFA in a frequentist

context. The Bayesian approach was adopted in fitting CFA models because it offers flexibil-

ity. It is fairly easy to deviate from common assumptions when using the Bayesian approach

compared with the frequentist approach; for example cross-loadings can be given priors con-

centrated around 0 instead of fixing them to 0. The Bayesian approach in this paper, the latent

variable approach, offers additional advantages over using link functions (e.g. logit and probit

links) in that distributional assumptions can be relaxed; for example a truncated skewed normal

distribution can be used and various discrepancy measures for continuous responses can be

used for discrete data.The PPP-values from the 2 discrepancy measures for both the single-level and the multilevel

FA suggested that the two models were good fits for the data at hand. It is clear that this

discrepancy measure failed to capture the misspecification of the data structure. Relying on

global goodness-of-fit measures without looking at specific aspects of the model can lead to

the wrong inference with regard to the compatibility of the model to the data. Thus, various

measures should be considered. PPCs use the data twice, i.e. in fitting the model and in checking

model adequacy, hence leading to some degree of bias. To eliminate this bias cross-validation

PPCs (Marshall andSpiegelhalter, 2003)canbe used; however, they are computationally intense.

When a model is correctly specified, the PPP-values should have a uniform distribution. This

characteristic of the PPP-values has been discussed elsewhere (Hjortet al., 2006; Steinbakk and

Storvik, 2009). Hjort et al. (2006) proposed recalibrating the PPCs, resulting in recalibrated

PPP-values which have a uniform distribution under the correct model specification. However,


19/21


the PPP-values in this paper are not recalibrated as computing the recalibrated PPP-values is

extremely computer intensive, requiring a double-simulation approach. So the data were split

into a learning data set and a validation data set, where the model fitting is done by using the

learning data set and model assessment by using the validation data set. Another potential

approach which is less computationally demanding is to compute the calibrated PPP-values by

using integrated nested Laplace approximations (Rue et al., 2009). This is an area which theauthors are currently exploring.

The factor structure for the single-level FA and the nurse level of the multilevel FA is in line

with expectations. Indeed, medication errors, pressure ulcers and patient fall accidents form

a logical group of adverse events. Belgian hospitals traditionally focus on these three adverse

events when setting up actions to improve patient safety in which nurses are involved. After all,

actions to prevent these adverse events are closely related to the nursing process (for example,

turning patients to prevent pressure ulcers, most medication is administered by nurses and

patient fall accidents can be prevented if patients are monitored and wear the correct type of

shoes). Urinary tract infections, bloodstream infections and pneumonia are all infections that

are potentially related to nursing care (for example, urinary tract infections can be prevented bypatients not having in-dwelling catheters for longer than needed). These can also be explained

by the severity of illness of the patients as well as by medical interventions (e.g. sterile insertion

central venous catheters). At the nursing unit level the variable patient fall accidents is now

grouped with the infections. The first two adverse events clearly reflect the nursing process at

the nursing unit level. The second nursing unit factor consists of patient fall accidents and

infections. This factor might be indicative of collective efforts taken at the nursing unit to

prevent injury (falls and infections) to patients, i.e. making sure that the environment that a

patient is exposed to is free from conditions which might increase the risk of infections or

patient fall accidents (e.g. clearing spillages from the floor or dropped food, clearing medical

equipment from nursing units and picking up dropped linen). However, the authors advocatefurther research to comprehend the grouping of factor indicators at the nursing unit level and

also to explore whether this structure is unique to Belgium or whether it can be established

in other European countries. As a sensitivity analysis the responses were discretized by using

different cut-off points. The results that are reported here were robust to changes in the cut-off

points.

Acknowledgements

The research leading to these results has received funding from the European Unions seventh

framework programme (FP7/2007-2013) under grant agreement 223468. The first and last

authors were partially supported by the Interuniversity Attraction Poles program P6/03, Belgian

State Federal Office for Scientific, Technical and Cultural Affairs.

References

Agency for Healthcare Research and Quality (2004) AHRQs patient safety initiative: building foundations,reducing risk.Interim Report 04-Rg005.Agency for Healthcare Research and Quality, Rockville.

Albert, J. H. and Chib, S. (1993) Bayesian analysis binary and polychotomous response data. J. Am. Statist. Ass.,88, 669679.

Ansari, A. and Jedidi, K. (2000) Bayesian factor analysis for multilevel binary observations. Psychometrika, 65,475496.

Azevedo, C. L., Bolfarine, H. and Andrade, D. F. (2011) Bayesian inference for a skew-normal IRT model under

the centred parameterization.Computnl Statist. Data Anal., 55, 353365.Bartholomew, D., Knott, M. and Moustaki, I. (2011) Latent Variable Models and Factor Analysis: a Unified

Approach, 3rd edn, sect. 2.10, pp. 3037. Chichester: Wiley.


20/21


Bayarri, M. J. and Castellanos, M. E. (2004) Bayesian checking of the second levels of hierarchical models. Statist.Sci., 22, 322343.

Bentler, P. M. (1990) Comparative fit indices in structural models.Psychol. Bull., 107, 238246.Bliese, P. D. and Hanges, P. J. (2004) Being both too liberal and too conservative: the perils of treating grouped

data as though they were independent. Organiznl Res. Meth., 7, 400417.Browne, M. W. and Cudeck, R. (1992) Alternative ways of assessing model fit. Sociol. Meth. Res., 21, 230258.Bruyneel, L., van Den Heede, K., Diya, L., Aiken, L. andSermeus, W. (2009) Predictive validity of theinternational

hospital outcomes study questionnaire: an RN4CAST pilot study. J. Nursng Schol., 41, 202210.Celeux, G., Forbes, F., Robert, C. and Titterington, D. (2006) Deviance information criteria for missing data

models.Am. Statistn, 1, 651674.Diya, L., Van den Heede, K., Sermeus, W. and Lesaffre, E. (2011) The relationship between in-hospital mortality,

readmission into the intensive care nursing unit and/or operatingtheater and nurse staffing levels. J. Adv. Nursng,68, 10731081.

Dyer, N. G., Hanges, P. J. and Hall, R. J. (2005) Applying multilevel confirmatory factor analysis techniques tothe study of leadership.Lead. Q., 16, 149167.

Gajewski, B. J., Boyle, D. K., Miller, P. A., Oberhelman, F. and Dunton, N. (2010) A multilevel confirmatoryfactor analysis of the practice environment scale: a case study. Nursng Res., 59, 147153.

Gelman, A., Carlin, J. B., Stern, H. S. and Rubin, D. (2004)Bayesian Data Analysis. London: Chapman and Hall.Gelman, A., Meng, X. and Stern, H. (1996) Posterior predictive assessment of model fitness via realized discrep-

ancies.Statist. Sin., 6, 733807.

Goldstein, H. and Browne, W. J. (2005) Multilevel factor analysis models for continuous and discrete data.In Contemporary Psychometrics: a Festschrift for Roderick P. McDonald (eds A. Maydeu-Olivares and J. J.McArdle), pp. 453475. London: Erlbaum.

Goldstein, H. and McDonald, R. P. (1988) A general model for the analysis of multilevel data. Psychometrika,53, 455467.

Grilli, L. and Rampichini, C. (2007) Multilevel factor analysis for ordinal variables. Struct. Equn Modlng, 14,125.

Hjort, N. L., Dahl, F. A. and Steinbakk, G. H. (2006) Post-processing posterior predictive p values.J. Am. Statist.Ass., 101, 11571174.

Hochreiter, S., Clevert, D.-A. and Obermayer, K. (2006) A new summarization method for affymetrix probe leveldata.Bioinformatics, 22, 943949.

Hu, L. and Bentler, P. (1999) Cutoff criteria for fit indexes in covariance structure analysis: conventional criteriaversus new alternatives.Struct. Equn Modlng, 6, 155.

Johnson, R. A. and Wichern, D. W. (2002)Applied Multivariate Statistical Analysis. Upper Saddle River: Pearson

Education International.Kenny, D. A. and Judd, C. M. (1986) Consequences of violating the independence assumption in analysis of

variance. Psychol. Bull., 99, 422431.Lake, E. T. (2002) Development of the practice environment scale of the nursing work index. Res. Nursng Hlth,25, 176188.

Lawley, D. N. and Maxwell, A. E. (1971)Factor Analysis as a Statistical Method, 2nd edn. London: Butterworth.Longford, N. T. (1993)Random Coefficient Models. New York: Oxford University Press.Longford, N. T. and Muthen, B. O. (1992) Bayesian posterior predictive checks for complex models. Psychomet-

rika, 57, 581597.Lopes, H. F. and West, M. (2004) Bayesian model assessment in factor analysis. Statist. Sin., 14, 4167.Mardia, K. V., Bibby, J. M. and Kent, J. T. (1979) Multivariate Analysis. New York: Academic Press.Marshall, E. C. and Spiegelhalter, D. J. (2003) Approximate cross-validatory predictive checks in disease mapping

models.Statist. Modllng, 22, 16491660.

Muthen, L. K. and Muthen, B. O. (2011)Mplus Users Guide, 6th edn. Los Angeles: Muthen and Muthen.Naicker, V. (2010) Educators pedagogy influencing the effective use of computers for teaching purposes in class-rooms: lessons learned from secondary schools in South Africa. Educ. Res. Rev., 5, 674689.

Norris, M. and Lecavalier, L. (2010) Evaluating the use of exploratory factor analysis in developmental disabilitypsychological research.J. Autsm Develpmntl Disordrs, 40, 820.

Plummer, M. (2003) JAGS: a program for analysis of Bayesian graphical models using Gibbs sampling. In Proc.3rd Int. Wrkshp Distributed Statistical Computing, Vienna, Mar. 20th22nd (eds K. Hornik, F. Leisch andA. Zeileis).

Plummer, M., Best, N., Cowles, K. and Vines, K. (2006) Coda: convergence diagnosis and output analysis formcmc.R News, 6, 711.

R Development Core Team (2012)R: a Language and Environment for Statistical Computing. Vienna: R Founda-tion for Statistical Computing.

Robinson, W. S. (1950) Ecological correlations and the behavior of individuals. Am. Sociol. Rev., 15, 351357.Rue, H., Martino, S. and Chopin, N. (2009) Approximate Bayesian inference for latent Gaussian models by using

integrated nested Laplace approximations (with discussion). J. R. Statist. Soc.B, 71, 319392.Spiegelhalter, D. J., Best, N. G., Carlin, B. P. and van der Linde, A. (2002) Bayesian measures of model complexity

and fit (with discussion).J. R. Statist. Soc.B, 64, 583639.


21/21


Steiger, J. H. and Lind, J. C. (1980) Statistically based tests for the number of common factors. A. Spring Meet.Psychometric Society, Iowa City.

Steinbakk, G. H. and Storvik, G. O. (2009) Posterior predictive p-values in Bayesian hierarchical models.Scand.J. Statist., 36, 320336.

Su, Y.-S. and Yajima, M. (2012) R2jags: a package for running JAGS from R.R Package Version 0.03-06.Tucker, C. and Lewis, C. (1973) A reliability coefficient for maximum likelihood factor analysis. Psychometrika,38, 110.

Van den Heede, K. (2008) Nurse-staffing levels and patient safety in acute hospitals: analyzing administrativedata at the nursing unit level. PhD Thesis. Centre for Health Services and Nursing Research, Department ofPublic Health, Katholieke Universiteit Leuven.

Yan, G. and Sedransky, J. (2007) Bayesian diagnostic techniques for detecting hierarchical structure.Baysn Anal.,2, 735760.

Supporting informationAdditional supporting information may be found in the on-line version of this article:

Supplementary Material: Multilevel factor analytic models for assessing the relationship between nurse reportedadverse events and patient safety.

Documents

Multilevel Factor Analytic Models