RESEARCH REPORT

A Confirmatory Factor Analysis ofthe User InformationSatisfaction Instrument

William J. Doll . The University of Toledo2801 W. Bancroft StreetToledo, Ohio 43606

T. S. Raghunathan Department oflnformation Systems& Operations ManagementThe University of Toledo2801 W. Bancroft StreetToledo. Ohio 43606

Jeen-Su Lim The University of Toledo2801 W. Bancroft StreetToledo. Ohio 43606

Yash P. Gupta The University of Colorado atDenverDenver. Colorado 80217

The structure and dimensionality ofthe user information satisfaction (UIS)construct is an important theoretical issue that has received considerable at-tention. Building upon the work of Bailey and Pearson (1983). Ives et al.(1983) conduct an exploratory factor analysis and recommend a 13-item in-strument (two indicators per item) for measuring user information satisfac-tion. Ives et al. also contend that UIS is comprised of three component mea-sures (information product, EDP staff and services, and user knowledge orinvolvement). In a replication using exploratory techniques. Baroudi and Or-likowski (1988) confirm the three factor structure and support the diagnosticutility ofthe three factor model. Other researchers have suggested a need forcaution in using the UIS instrument as a single measure of user satisfaction;they contend that the instrument's three components measure quite differentdimensions whose antecedents and consequences should be studied sepa-rately.

The acceptance of UIS as a standardized instrument requires confirmationthat it explains and measures the user information satisfaction construct andits components. Based on a sample of 224 respondents, this research usesconfirmatory factor analysis (LISREL) to test alternative models of underly-ing factor structure and assess the reliability and validity of factors and items.

1047.7047/95A)602AI 177/$01.25Copyright O 1995. Institute for Operations Research and the Managemeni Sciences

I n f o r m a t i o n Sys t ems Resea rch 6 : 2 177

Doll • Raghunathan • Lim • Gupta

The results provide support for a revised UIS model with four first-order fac-tors and one second-order (higher-order) factor. To cross-validate these re-sults, the authors reexamine two data sets, including the original Baroudi andOrlikowski data, to assess the revised UIS model. The results show that therevised model provides better model-data fit in all three data sets. Thus, theevidence supports the use of: (1) the 13-item instrument as a measure of anoverall UIS; and (2) four component factors for explaining the UIS construct.Confirmatory factor analysis—UKr informalion talisfaction—Us«r salisfactiun—Validily—ReliabiIit>—Higher-order factor models

For nearly two decades, the user satisfaction construct has occupied a central rolein behavioral research in information systems (Melone 1990).' User satisfaction

is an attitudinal or affective response; i.e.. the sum of one's feelings and attitudestowards a variety of factors (components) related to the delivery of information prod-ucts and services (Bailey and Pearson 1983, Ives et al. 1983). The structure and di-mensionality of user satisfaction are an important theoretical issues that have re-ceived considerable attention (Zmud 1978, Larcker and Lessig 1980. Swanson 1982,Doll and Torkzadeh 1988). These issues have not been fully resolved. Most of thisliterature focuses on explaining what user satisfaction is by identifying its compo-nents, but the discussion usually suggests that user satisfaction is a single construct.

The Ives, Olson and Baroudi (IOB) instrument is the most widefy used instrumentfor measuring general user information satisfaction (UIS). The IOB instrument im-plies a single second-order user information satisfaction (UIS) construct comprisedof three component factors (i.e., knowledge or involvement, information product,and EDP staff and services). Galletta and Lederer (1989) question the dimensionalityof the UIS instrument, suggesting that the three factors are comprised of heteroge-neous items that do not measure a single construct. Despite this criticism, substantiveresearch studies using the UIS instrument typically assume that user informationsatisfaction is a single construct.

Several researchers (e.g., Ives and Olson 1984, Straub 1989, DeLone and McLean1992) have stressed the importance of developing standardized instruments for mea-suring user satisfaction. Despite the importance of user information satisfaction as ameasure of system success, no rigorous follow-up (i.e., cross-validation) studies ofmodel-data fit have been conducted to confirm the structure and validity of the in-strument. Confirmatory factor analysis is needed to provide a more rigorous andsystematic test of alternative factor structures than is possible within the frameworkof exploratory factor analysis.

Research MethodsConfirmatory factor analysis involves the specification and estimation of one or

more putative models of factor structure, each of which proposes a set of latent vari-ables (factors) to account for covariances among a set of observed variables. LISRELVII (Joreskog and Sorbom 1989) is used to describe alternative models and to test thefit of each hypothesized model against the sample data. Model specification is accom-plished by fixing or constraining elements in three matrices that are analogous to the

' This paper was shortened to conform to requirements of a research report. A complete version of thispaper is available upon request from Dr. William J. Doll.

Information Systems Research 6 : 2

A Confirmatory Factor Analysis ofthe User Information Satisfaction Instrument

HODEL 1 MODEL Z HODEL 3

DIE FIUT-ORDEIt FACTOR i FIHSr-ORDER FACTOKS(UNCORHtLATEDI

3 FIHST-ORDER FACTORS(COIUEUTED)

MODEL 4

1 riRST-OROER FACTORSORDER FACTOR

I l ls - - USE* IIFOMATION SATISFACTION

(CORREUTEO)

X,:

"" fSE 5 1 ! * ' " * " " - - iMFOiWArioii PRODUCT-• at ITAFF t " KNOHLEOaE

I -- INVOLVEMENTProcasaing of raquaata for changaa to axiatlng ayataTim raquirsd tor rww ayatana davalopMntRalatlonahlp wlUi EDP aCaftKttltuda of EDP ataffcotaunlcatlon with EDP atafCRallBblllty of output inforaatlonRaJavoncy of output lnfor«atian

Accucacy of output lnforHtionPraclalon of output inforsatlonCoBplatanaaa of output Information[>*^a* ot EDP training provided totiBar'a undarataiullnci of ayatamatJasr'a fssllng of participation

FIGURE 1.

factor pattern matrix, factor correlation matrix, and communalities from a commonfactor analysis. Higher-order factor models require the specification of an additionalmatrix containing loadings ofthe first-order factors on the higher-order factors.

In this study, the analysis proceeds in three steps. First, based on logic, theory andprevious studies, plausible alternative models of underlying data structure are pro-posed. Using several goodness-of-fit indexes, confirmatory factor analysis is used tocompare data-model fit and examine evidence for a higher-order construct. Onemodel is selected as best representing the underlying factor structure in the sampledata. Second, confirmatory factor analysis is used to assess the reliability and validityofthe factors and items in the selected model. Third, to cross-validate these results,goodness-of-fit indexes for the plausible alternative models are compared using twodata sets reported in previous research on the IOB instrument.

The A Iternative ModelsThe need to examine alternative models (see Figure 1) of underiying data structure

is suggested by two apparent inconsistencies between the instrument developmentprocess and resultant recommendations for measuring UIS (Ives et al. 1983, Baroudiand Orlikowski 1988). First, factors were rotated orthogonally to derive three first-order factors yet these independent factors were combined into a single overall mea-sure of UIS. Second, they imply a single second-order factor without statistical evi-dence. Exploratory factor analysis, which can only identify first-order factors, wasused, yet a global indicator of user information satisfaction was proposed (BaroudiandOriikowski 1988).

Model 1 hypothesizes one first-order factor (UIS) accounting for all the commonvariance among the 13 items. Theory as well as substantive research studies using

June 1995 179

Doll • Raghunathan • Lim * Gupta

user satisfaction instruments typically assume that user satisfaction is a single first-order construct. This assumption is implicit in the typical practice of scaling the sat-isfaction construct by adding individual items to obtain a total score. Baroudi andOrlikowski {1988) scale UIS by using such a total score, implying that one first-orderfactor is a plausible model of underlying data structure.

Model 2 hypothesizes that the 13 items form into three uncorrelated or orthogonalfirst-order factors (EDP Statf and Services, Information Product. Knowledge or In-volvement). The Ives et al. use of varimax (orthogonal) rotation should have resultedin three uncorrelated factors: thus. Model 2 is considered a plausible alternativemodel of underlying data structure. Examining this model also provides a test of thenecessity of incorporating correlated factors by enabling a comparison ofthe increasein fit between uncorrelated and correlated models.

Model 3 hypothesizes that the three first-order factors are correlated with eachother. The 13 Items used in the IOB short-form instrument were a subset of 39 itemssuggested by Bailey and Pearson (1983) as measures of a single user satisfaction con-struct; thus, subscaies based on these items are likely to be correlated. Furthermore,the Ives et al. original study used corrected-item total correlations and correlationswith an overall criterion (a four-item scale emphasizing satisfaction with the "dataprocessing group") to eliminate items. This elimination method resulted in 13 itemsthat had substantial common variance. The factor scores from a varimax rotation areorthogonal, but the subscales are not necessarily orthogonal (uncorrelated). If theitems have a large amount of common variance, scales based on these items may becorrelated. This model was not explicitly proposed by Ives et al., yet it is plausiblebecause of common variance among the 13 items.

Model 3H (H denotes higher order construct) hypothesizes three first-order factorsand one second-order factor (UIS). This model was not explicitly proposed by IOB,yet it is plausible because of common variance. Their instrument development effortused exploratory factor analysis which does not reveal second-order constructs; thus,they could not provide statistical evidence of a second-order construct. The commonvariance shared by the three first-order factors may be statistically "caused" by asingle second-order factor (Tanaka and Huba 1984).

Model 4 hypothesizes that the thirteen items form into four correlated first-orderfactors (EDP Staff, EDP Services. Information Product. Knowledge or Involvement).The EDP Staff and Services constructs represent phenomena of a distinctly differentnature. The EDP St9*'"items focus on user relationships with the information systemsgroup using adjective pairs such as belligerent vs. cooperative (Attitude of EDP Staff),harmonious vs. dissonant (Relationship with the EDP Staff)̂ productive vs. destruc-tive (Communication with EDP Staff). The EDP Services items focus solely on deliv-ery time for products or services using adjective pairs such as fast vs. slow (ProcessingRequests for Changes to Existing Systems) and acceptable vs. unacceptable (TimeRequired for New Systems Development).

The EDP Staff and Services constructs have different determinants and conse-quences (Senn 1989). Delivery time for products or services is a function of projectmanagement or technical skills while good user relationships with the informationsystems group are thought to require interpersonal and communication skills. Deliv-ery delays have direct consequences for the cost of systems development and mainte-nance while poor user relationships are generally thought to have a more indirectlinkage to costs (e.g., through user resistance to change). Since these two constructs

180 Information Systems Research 6 : 2

A Confirmatory Factor Analysis of the User Information Satisfaction Instrument

reflect such different phenomena and are assumed to have different determinants aswell as consequences, it is plausible to test whether they should be treated as separateconstructs.

Model 4H (H denotes higher order construct) hypothesizes four first-order factorsand a single second-order factor (UIS). The rationale for examining the four factorsis the same as that described above for Model 4. Since substantive research studiesusing the IOB instrument typically treat UIS as a single construct, implying theoreti-cal consensus on this i^ue, we considered it appropriate to test for the existence of asingle second-order construct.

Criteria for Comparing Model-data FitBecause no one statistic is universally accepted as an index of mode! adequacy, our

interpretation of results emphasizes substantive issues, practical considerations, andseveral measures of fit. In this study, absolute indexes of goodness-of-fit such as chi-square, goodness-of-fit index (GFI), adjusted goodness-of-fit index (AGFI). and rootmean square residual (RMSR) are used to evaluate individual models. Relative orincremental fit indexes reflecting the improvement in fit of one model over an al-ternative (i.e., ratio of chi-square to degrees of freedom, normed fit index (NFI), andtarget coefficient are used to compare models.

Although Ihe chi-square statistic is a global test of model's ability to reproduce thesample variance/covariance matrix, its significance levels are sensitive to sample sizeand departures for multivariate normality; thus, the chi-square statistic must be in-terpreted with caution in most applications (Joreskog and Sorbom 1989). The GFIand AGFI are measures ofthe relative amount of variance and covariance impliedby the data set that are jointly accounted for by the model. The AGFI differs from theGFI by adjusting for the degrees of freedom in the model. GFI and AGFI range from0 to I with higher values indicating better fit (Joreskog and Sorbom 1989). Manyresearchers interpret GFI or AGFI scores in the 0.80 to 0.89 range as representingreasonable fit; scores of 0.90 or higher are considered evidence of good fit. The RMSR(Joreskog and Sorbom 1984) reflects the average residual obtained by taking thedifference between the model-generated and sample variance/covariance matrices.Smaller values are associated with better fitting models with scores below 0.05 con-sidered as evidence of good fit (Byrne 1989).

The ratio of chi-square to the degrees of freedom provides information on the rel-ative efficiency of competing models in accounting for the data. Researchers haverecommended using ratios as low as 2 or as high as 5 to indicate a reasonable fit(Marsh and Hocevar 1985). The NH (Bentler and Bonett 1980) assesses the fit of amodel relative to the fit of a null model by scaling the chi-square value from 0 to 1with larger values indicating better models. Well-fitting models generally yieldnormed fit indexes of at least 0.90, i.e., only a relatively small amount of varianceremains unexplained by the model (Harvey et al. 1985).

The target coefficient index (the ratio ofthe chi-square ofthe first-order model tothe chi-square ofthe higher-order model) is used to provide evidence ofthe existenceof a higher order construct (Marsh and Hocevar 1985). It reflects the extent to whichthe higher-order factor model accounts for covariation among the first-order factors.The target coefficient can be interpreted as the percent of variation in the first-orderfactors that can be explained by the second-order construct.

June 1995 181

Doll • Raghunathan • Lim • Gupta

Evaluating Validity and ReliabilityIn confirmatory studies, the concepts of reliability and validity are approached

somewhat differently than they are in exploratory studies. In confirmatory factoranalysis, factor loadings can be viewed as regression coefficients in the regression ofobserved variables on latent variables. On the first-order level of measurementmodels, the standard factor loadings of observed variables (items) on latent variables(factors) are estimates ofthe validity ofthe observed variables. For second or higherlevels, the standard structural coefficient of factors on higher-order constructs areestimates ofthe validity ofthe factors. The larger the factor loading or coefficients—as compared with their standard errors and expressed by the corresponding t values—the stronger is the evidence that the measured variables or factors represent the un-deriying constructs (Bollen 1989). There is no universally accepted cut-off value forfactor loadings. In this research, factor loadings or standard structural coefficientsabove 0.7 will be considered good measures of their latent construct.

Confirmatory factor analysis enables us to estimate the reliability of individualitems as well as factors and the overall instrument. On the first-order level of mea-surement models, the proportion of variance (R-square) in the observed variablesthat is accounted for by the latent variables influencing them can be used to estimatethe reliability ofthe observed variables (items) with R-square values above 0.49 con-sidered evidence of acceptable reliability. For second or higher levels, the proportionof variance (R-square) in the latent variables (factors) that is accounted for by thehigher-order construct influencing them can be used to estimate the reliability ofthe latent factors (Bollen 1989). The total coefficient of determination for observedvariables is an estimate ofthe reliability ofthe overall instrument.

The Confirmatory SampleThe confirmatory sample. like the sample Baroudi and Orlikowski used for explor-

atory factor analysis ofthe IOB instrument, was gathered in a variety of industries.Research assistants contacted firms and requested their participation in the survey.Questionnaires were completed by full time employees from 30 firms during workinghours. The sample consists of 224 respondents distributed by industry as follows:manufacturing. 70; education. 42; retail/wholesale, 30; government agencies. 12;banking and finance. 22; insurance. 11; healthcare. 31; and "other," 6. Respondentswere asked to identify their position within the organization; they responded as fol-lows: 93 managers. 69 professional employees without supervisory responsibility, and62 operating personnel. The sample is select, but the large number of firms and thevariety of industries supports the generalizability ofthe results.

ResultsThe goodness-of-fit indexes for the alternative models (see Figure 1) and the null

model are summarized in Table la. The primary purpose of the null model is toestablish the zero-point for the NFI. As expected, the null model provides a poor fitto the data as evidenced by a ratio of chi-square to degrees of freedom of 17.72.Models 1 and 2 have substantially better fit reiative to the null mode! for all indexes ofgoodness-of-fit. By the criteria described above for comparing model-data fit, neitherModel 1 nor Model 2 demonstrate a reasonable fit with the sample data.

Model 3 and 3H compare favorably with Model 1 and Model 2 on all indexes ofgoodness-of-fit. In the case of three correlated first-order factors, a second-order

182 Information Systems Research 6 : 2

A Confirmatory Factor Analysis of the User Informatioa Satisfaction Instrument

12

3

3H

4

4H

TABLE laGoodness-of-fit Indexes for Alternative Models (n =

Model

Null modelOne First-order factor3 First-order factors

(Uncorrelated)3 First-order factors

(Correlated)3 First-order factors1 Second-order factor4 First-order factors

(Correlated)4 First-order factors1 Second-order factor

Chi-square (df) Chi-square/df NH

1382.22(78)358.18(65)373.61 (62)

173.70(62)

173.70(62)

123.97(59)

126.16(61)

17.725.516.03

2.80

2.80

2.10

2.07

0.7410.730

0.874

0.874

0.910

0.908

224)

G¥\

0.3290.7730.788

0.882

0.882

0.917

0.916

AGFI

0.2170.6820.689

0.827

0.827

0.873

0.875

RMSR

0.3820.0820.285

0.059

0.059

0.051

0.052

Note: T coefficient for 4H using 4 as target model is 0.983.

model with three first-order factors (i.e., 3H) has the same degrees of freedom andchi-square; thus. Models 3 and 3H have identical goodness-of-fit coefficients.

Model 3H hypothesizes three first-order factors and a single second-order constructthat can be interpreted as UIS. The ratio of chi-square to degrees of freedom indicatesa reasonable fit but the NFI is below the 0.90 score required for a "well-fitting" model.Both GFI (0.882) and AGFI (0.827) are within the range indicating reasonable fit(0.80-0.89), but below the 0.90 score required for evidence of good fit. The root meansquare residual (0.059) is slightJy above the 0.05 score considered evidence of good fit.In this case or any three factor model, the T coefficient of 1.0 has no meaning becausethe goodness-of-fit indicators are identical and the degrees of freedom are the same.

Model 4 (four correlated first-order factors) provides a significantly better fit thanModel 3 or 3H. The change in chi-square of 49.73 for three degrees of freedom is

TABLE lbBauroudi and Orliskowski Data: Ooodness-of-fu lnde.xes for Alternative Models (n - 354)

0.12

3

.IH

4

4H

Model

Null model1 First-order factor3 First-order factors

(Uncorrelated)3 First-order factors

(Correlated)3 First-order factors1 second-order

factor4 First-order factors

(Correlated)4 First-order factors1 Second-order

factor

Chi-square (dO

2569.96 (78)799.57(65)514.11(62)

272.26(62)

272.26(62)

151.63(59)

160.27(61)

Chi-square/df

32.9412.308.29

4.39

4.39

2.57

2.63

Nn

0.6890.800

0.894

0.894

0.941

0.938

G H

0.3130.6860.817

0.892

0.892

0.940

0.935

AGH

0.1980.5610.731

0.842

0.842

0.907

0.903

RMSR

0.3960.1090.247

0.073

0.073

0.051

0,052

Note: 1 coefficient for 4H using 4 as target model is 0.946.

June 1995 183

Doll • Raghunathan • Lim • Gupta

0.12

3

3H

4

4H

TABLE lcT.S. Raghunathan Data: Goodness-of-Eit Indexes for Alternative Modets (n

Model

Null model1 First-order factor3 First-order factors

(Uncorrelated)3 First-order factors

(Correlated)3 First-order factorsI Second-order

factor4 First-order factors

(Correlated)4 First-order factors1 Second-order

factor

Chi-square (df

1661.05(78)597.04 (65)349.82 (62)

214.32(62)

214.32(62)

151.01 (59)

167.21(61)

1 Chi-square/df

21.309.195.64

3.46

3.46

2.56

2.74

NFI

_

0.6410.789

0.871

0.871

0.909

0,899

GFI

0.3260.6850.812

0.874

0.874

0.906

0.895

-233)

AGH

0.2130.5580.724

0.815

0.815

0.855

0.846

RMSR

0.3850.1100.267

0.087

0.087

0.066

0.075

l^ote: T coefficient for 4H using 4 as target model is 0.903.

significant at/j< 0.001. The NFI is above 0.90, indicatingaweil-fitting model. BothGFI and AGFI are substantially better than model 3H's scores with the GFI scoreabove the 0.90 required for evidence of good fit. The root mean square residual(0.051) is very close to the 0.05 score required for evidence of good fit.

Model 4H's scores for all indicators of goodness-of-fit are not significantly dififerentfrom Model 4"s scores. This suggests that Models 4 and 4H are satisfactory and com-peting representations ofthe underlying structure ofthe instrument. Model 4 freesthe correlations among the four factors but does not hypothesize a higher-order con-struct. Model 4H hypothesizes the existence of a single second-order construct (UIS)that accounts for the variation in the four first-order factors. The T coefficient wasused to test for the existence of this second-order UIS construct. Using Model 4 as thetarget model, the target coefficient is the ratio of chi-square of Model 4 to chi-squareof Model 4H. In this case, a target coefficient of 0.983 provides good evidence of ahigher-order UIS construct. Ninety-eight percent ofthe variation in the four first-order factors in Model 4 is explained by Model 4H's user information satisfactionconstruct.

Ofthe alternative models tested, the four factor models (4 and 4H) were judged todemonstrate the best fit with the sample data. Since theory in this field suggests theexistence of a single overall user information satisfaction construct. Model 4H is ofgreater theoretical interest than Model 4. In this study, there is good evidence (Tcoefficient equals 0.983) of a single second-order construct. Both Model 4 and Model4H enable us to examine the validity and reliability of individual items. Estimates ofitem validity and reliability are not sensitive to the addition of a second-order factor,thus, conclusions concerning the validity and reliability of tbe thirteen items wouldbe the same regardless which model was selected. Mode! 4H has the additional ad-vantage of providing estimates ofthe validity and reliability ofthe latent factors (EDPStaff, EDP Services, Information Product, Knowledge or Involvement). For thesereasons., the researchers recommend Model 4H and proceed with the analysis ofthevalidity and reliability of factors and items assuming this second-order model.

184 Information Systems Research 6 : 2

A Confirmatory Factor Analysis of the User Information Satisfaction Instrument

Validity and ReliabilityUsing confirmatory factor analysis, the LISREL's maximum likelihood estimates

of Model 4H's standardized parameter estimates are presented in the "present data"column of Table 2 for both latent variables and observed variables. For the observedvariables. Table 2 presents factor loadings, their corresponding (values, and R-squarevalues. With / values above 2.0 being considered significant, factor loadings can beinterpreted as indicators of validity for the thirteen items. All 13 items have signifi-cant factor loadings (/ values above 6.9) on their corresponding factors. Two items(X8 and X12) have factor loadings substantially below the 0.7 required for items thatare considered good measures of their latent factor. X7 has a factor loading (0.697)only slightly below the required 0.70. The proportion ofthe variances, or R-square,in the observed variables that is accounted for by its corresponding latent variable isused as an indicator of each item's common factor reliability. R-square values rangefrom 0.213 to 0.780. Ten items have acceptable reliability (above 0.49 R-square) withone (X7) only slightly below and two (X8 and X12) substantially below the 0.49 R-square value required for items with good reliability. The total coefficient of determi-nation for the first-order model is 0.996, indicating excellent overall reliability ofthe13 items combined.

For the latent variables. Table 2 presents the standard structural coefficient, theircorresponding t values, and R-square values. Standard structural coefficients can beinterpreted as indicators of validity ofthe latent factors as components ofthe userinformation satisfaction construct. With / values above 2.0 being considered signifi-cant, all factors have large (greater than 0.75) and significant (/ values above 8.7)structural coefficients, indicating good construct validity. R-square values for each ofthe four latent factors range from 0.57 to 0.78, indicating acceptable reliability for allfactors. The total coefficient of determination for the structural equation ofthe sec-ond-order factor is 0.889. indicating acceptable overall reliability of the four first-order factors combined.

To verify that the reliability problems found above were not caused by the respec-ification ofthe model (e.g., using Model 4H instead of 3H to evaluate item and latentfactor validity and reliability), the researchers examined item reliability and validityfor Model 3H. Model 3H had four items (XI, X2, X8 and XI2) with factor loadingssubstantially below the 0.7 required for items that are considered good measures oftheir respective latent factor. The same four items also had R-square values below0.49. indicating reliability problems with these items. In Model 3H, all three standardstructural coefficients had acceptable validity and reliability, indicating the first-orderfactors were good measures of the second-order UIS construct. Respecifying themodel from a three factor model (3H) to a four factor model (4H) reduced the num-ber of items that were not good measures (factor loading below 0.7) of their latentvariable. This provided confirmation that resF)ecifying the modei to provide separate"EDP Staff" and "EDP Services" factors improved validity and reliability.

Cross ValidationThe researchers sought to cross validate these results using two previously pub-

lished data sets collected by Baroudi and Orlikowski (1988) and Raghunathan andKing (1988). Cross validation provides assurance that the results obtained are notsample specific and enhances confidence that these results can be generalized to thereferent population.

June 1995 185

Doll • Raghunathan • Lim • Gupta

TABLE 2Standardized Parameter Estimates and l-valuesfor Model 4H

(4 First-order Factors. I Second-order Factor)

Item

XIX2X3X4X5X6X7X8X9XIOXIIX12X13

Statistics of Observed Variables of Three [>ata Sets

Present Data

Factor

Baroudi Data

FactorLoading R-Square Loading R-Square

0.708* 0.5010.765(8.47) 0.5850.806* 0.6500.764(12.10) 0.5840.883(13.92) 0.7800.828* 0.6860.697(11.00) 0.4860.469(6.93) 0.2130.796(12.99) 0.6340.779(12.65) 0.6070.736* 0.5420.592(7.68) 0.3500.745(9.21) 0.555

0.757*0.762(10.54)0.858*0.769(17.03)0.917(21.18)0.822*0.618(12.37)0.882(19.96)0.882(19.98)0.774(16.55)0.726*0.556(8.83)0.752(10.86)

0.5730.5810.7360.5910.8410.6760.3820.7780.7780.5990.5270.3090.566

Statistics of Latent Variables of Three Data Sets

Present Data Baroudi Data

StandardStructure

StandardStructure

Factor Coefficient R-Square Coefficient

EDP Staff 0.840(8.75)EDPServices 0.815(10.39)Information

Product 0.756(9.93)Knowledge and

Involvement 0.833(9.33)

0.706 0.786(10.78)0.664 0.783(13.00)

0.572 0.720(11.82)

0.780 0.77! (10.45)

R-Square

0.6180.613

0.518

0.594

Raghunathan Data

FactorLoading

0.715*0.784(7.72)0.862*0.780(14.61)0.952(18.73)0.838*0.604 (9.77)0.876(16.23)0.854(15.67)0.632(10.34)0.637*0.689(7.64)0.746(7.86)

R-Square

0.5110.6150.7430.6080.9060.7020.3650.7670.7290.3990.4060.4750.557

Raghunathan Data

StandardStructure

Coefficient

0.752(7.57)0.674(8.92)

0.783(10.12)

0.706 (7.02)

R-Square

0.5660.454

0.613

0.498

Notes: * indicates a parameter fixed at I.O in the original solution./-values for item factor loadings and factor structural coetlicients are indicated in parentheses.R-square indicates reliability of the items.

The Baroudi and Orlikowski (1988) data set consists of 354 respondents selectedfrom a variety of industries. Using a correlation matrix supplied by Baroudi, theresearchers computed goodness-of-fit indexes for the alternative models shown inFigure 1. The results, summarized in Table lb, show a pattern similar to our data (seeTable la) with the four factor models (4 and 4H) achieving better fit with the sampledata than the alternative models. As with the previous data set, an examination ofstandardized parameter estimates revealed that Model 4H improves both item andfactor reliabilities over Model 3H. Using Model 4 as the target model, the target co-efficient of 0.946 provides good evidence of a higher-order UIS construct in the Bar-oudi and Orlikowski data.

The Raghunathan and King (1988) data set consists of 233 respondents selected

186 Information Systems Research 6 : 2

A Confirmatory Factor Analysis ofthe User Information Satisfaction Instrument

from 200 organizations representing a variety of industries. Using a correlation matrixsupplied by Raghunathan, the researchers computed goodness-of-fit indexes for thealternative models shown in Figure 1. The results, summarized in Table lc, again showa pattern similar to our data (see Table la) with the four factor models (4 and 4H)achieving better fit with the sample data than the alternative models. Using Model 4 asthe target model, the target coefficient of 0.903 provides good evidence of a higher-order UIS construct. Ninety percent ofthe variation in the four first-order factors inModel 4 is explained by Model 4H"s user information satisfaction construct.

This cross validation indicates that Model 4H {the four factor model with a singlesecond order construct) fits the other published data sets better than the three factormodel (Model 3H). Even using this better fitting model (i.e., 4H) and despite anoverall fit that was considered acceptable, the short-form ofthe IOB instrument con-tinues to have individual items that have inadequate validity and reliability in allthree data sets (see Table 2). Items X7 and X12 have validity and reliability problemsin all three data sets. Perhaps indicating sample specific variance, X8 had inadequatevalidity and reliability in our data set and X10 and X11 had inadequate validity andreliability in the Raghunathan and King data set.

Discussion and ConclusionsThe results enhance our understanding of the nature and dimensionality of the

UIS construct. Ives et al. (1983) imply that user information satisfaction is a singlehigher-order construct. This single second-order model was plausible, but neither theefforts of Ives et al. in developing the short-form nor a replication conducted by Bar-oudi and Orlikowski (1988) provided statistical evidence of a higher-order UIS con-struct. With target coefficients above 0.90 in three independent samples, this researchprovides strong empirical evidence that UIS is a single second-order construct. Gal-letta and Lederer's (1989) concern that the UIS instrument is comprised of heteroge-neous items that do not measure a single construct does not appear to be wellfounded.

The results enhance our understanding ofthe strengths and weaknesses ofthe 13-item short-form instrument and its subscales. AH items comprising the proposed"EDP Staff' and "EDP Services" subscales have adequate validity and reliability inall three data sets, suggesting that researchers can use these subscales with confidence.Both the 'information Product" and the "Knowledge and Involvement" subscaleshave at least one item with inadequate validity and reliability in all three data sets.These results suggest the need for additional research to develop replacement itemsfor these subscales.

Additional research is necessary to assess whether EDP staff and EDP services havedifferent determinants and consequences. Which construct (i.e., EDP staff or EDP ser-vices) is most strongly linked to possible consequences such as performance or usage?How strongly are each of these constructs linked to possible antecedents such as sys-tems characteristics or user involvement? What are the key moderators of these links?

This study completes one exploratory-confirmatory research cycle by more rigor-ously validating the short-form ofthe three factor user information satisfaction in-strument developed by Ives et al. (1983). The results support a modified model con-sisting of four first-order factors and one second-order construct interpreted as UIS.Thus, the results support the thirteen-item instrument as a measure of an overall UIS.With subscale modifications to improve reliability and validity, these results support

June 1995 187

Doll • Raghunathan • Urn • Gupta

the use of four component factors (EDP Staff. EDP Services, Information Product,and Knowledge or Involvement) for diagnostic purposes and as an explanation of theUIS construct.*

Acknowledgement. The authors would like to thank Jack Baroudi of New York Uni-versity for making available the correlation matrix from his earlier study of the UserInformation Satisfaction Instrument.

* E. Burton Swanson, Associate Editor. This paper was received on March 3, 1994, and has been withthe authors 3 months for 1 revision.

ReferencesBailey. J. E. and S. W. Pearson, "Development of a Tool for Measuring and Analyzing Computer User

Satisfaction," Management Science. 29. 5 (May 1983). 530-545.Baroudi. J. J. and W. J. Orlikowski. "A Short-Form Measure of User Information Satisfaction: A Psycho-

metric Evaluation and Notes on Use," Journal of Management Information Sy.sU'ms. 4, 4 (Spring1988), 44-59.

Bentier. P. M. and D. G. Bonett. "Significance Tests and Goodness-of-fit in the Analysis of CovarianceStructure," Psychohgicai Bulletin. 88, 3 (1980), 588-606.

Bollen. IC. A.. Structural Equaiions with Latent Variables. Wiley & Sons, New York, 1989.Byrne, B. M.,A Primer of LISREL: Basic Applications and Programming for Confirmatory Factor Analytic

Modeh. Springer-Verlag. New York, 1989.DeLone, W, and E. McLean, "Information Systems Success: The Quest for the Dependent Variable,"

Information Systems Reaearch. 3. I (March 1992). 60-95.Doll. W. J. and G. Tork?adeh. "The Measurement of End-User Computing Satisfaction." MIS Quarterly.

12, 2 (June 1988). 259-274.Galletta, D. and A. Lederer, "Some Cautions on the Measurement of User Information Satisfaction."

Decision Sciences, 20, 3 (1989). 419-438.Harvey. R. J.. R. Billings, and K. J. Nilan. "Conlinnatory Factor Analysis of the Job Diagnostic Survey:

Good News and Bad News," Journal of Applied PsyK-hology. 70. 3 (1985), 461 -468.Ives, B. and M. Olson. "User Involvement and MIS Success; A Review of Research," Management Science.

30, 5 (May 1984), 586-603., and J. J. Baroudi, "The Measure of User Information Satisfaction," Communica-

tions of ACM. 26, 10 (October 1983). 785-793.Joreskog. K.G. and D. Sorbom, L/S'/i£'L VI: Analysis of Linear Structural Relationships by the Method of

Maximum Likelihood. National Educational Resources, Chicago. IL. 1984.and , LISREL VII User's Guide. Scientific Software. Inc.. Mooresville, IN. 1989.

Larcker, D. F. and V. P. Lessig. "Perceived Usefulness of Information: A Psychometric Examination."Decision Sciences. 11,1 (1980). 121-134.

Marsh, H. W. and D, Hocevar, "Application of Confirmatory Factor Analysis to the Study of Self-concept:First- and Higher-order Factor Models and Their Invariance across Groups." P.sychologicat Bulletin.97. 3 (1985). 562-582.

Melone, N. P.. "A Theoretical Assessment of the User-satisfaction Construct in Information System Ke-search,." Management Science. 36. I (January 1990), 76-91.

Raghunathan. T. S. and W. R. King, "The Impact of Information Systems Planning on the Organization,"Omega. 16,2(1988), 85-93.

Senn. J. A., Analysis and Design of Information Systems. McGraw-Hill. New York. 1989.Straub, D. W.. "Validating Instrument in MIS Research," MIS Quarterly 13. 2 (1989). 147-169.Swanson, E. B., "Measuring User Attitudes in MIS Research: A Review," Omega. 10. 2 (1982), 157-165,Tanaka, J. S. andG. J. Huba. "Confirmatory Hierarchical Factor Analysis of Psychological Distress Mea-

sures." Journal of Personality and Social Psychology. 46, 3 (1984), 621 -635.Zmud. R. W.. "Concepts, Theories and Techniques—An Empirical Investigation of the Dimensionality

oftheConceptoflnformation."ZJ«-monScience, 9(1978), 187-195.

188 Information Systems Research 6 : 2