Applied Multivariate Analysis - Vaasan yliopistolipas.uwasa.fi/~sjp/Teaching/mva/lectures/c7.pdf · Applied Multivariate Analysis ... These background factors serve from statistical

Applied Multivariate Analysis

Seppo Pynnonen

Department of Mathematics and Statistics, University of Vaasa, Finland

Spring 2017

Seppo Pynnonen Applied Multivariate Analysis

Structural Equation Models (SEM)

SEM

General steps in Structural Equation Modeling are:

1 Specification

2 Identification

3 Estimation

4 Testing

5 Modification



Model Specification

1 Structural Equation Models (SEM)

Model Specification

Measurement Models

Model Identification

Model Estimation

Model Assessing and Testing

Model Modification

Stability of the model

Direct, Indirect, and Total Effect

Modification Indices



Model Specification

SEM Background

Model specification involves using all available relevant theory,research, and information to construct the theoretical model.

In structural equation modeling this results to specifying therelationships between the relevant variables describing thephenomenon in interest.



Model Specification

Example 1

For a study in an industrial sales forcea the researcher wanted to answer

questions as: ”Is the relationship between performance and job

satisfaction myth or reality?” Does performance influence satisfaction, or

does satisfaction influence performance?”

�� job sat

�� perform-

�?

?

aBagozzi, R.P. (1980). Performance and satisfaction in an industrial sales force: An examination of their

antecedents and simultaneity. Journal of Marketing, 44, 65–77.

Remark 1

In diagrams the (latent) theoretical variables are presented as circles or

ovals and the observed variables in rectangles.



Model Specification

The researcher further believes that background factors, likeachievement motivation, task specific self esteem, andverbal intelligence possibly affect on job satisfaction as well asperformance.

These background factors serve from statistical modeling point of view ascovariates.

Their impact should be eliminated.

Otherwise the mutual relationship between job satisfaction andperformance may be just a manifestation of these background factors.

The elimination can be simply done by including the covariates into the

model.



Model Specification

Thus, the preliminary sketch of the model is the following:

��

job sat

perform

6?

? ?

��

achv mot

self est

verb int

XXXXXXXXXz

ZZZZZZZZZ~

��

��:

XXXXXXXXXz

��>

��

��:

'

&

-

-

'&-

-'&-

-



Model Specification

Remark 2

In SEM graphs, like above, the one-headed arrows indicate causal

relations and the two-headed arched arrows indicate mutual

dependencies (correlation).

Inclusion of unimportant factors or exclusion of important factors

will produce implied models that are misspecified.

Exclusion of important factors may result in biased parameter

estimates.

Inclusion of unimportant factors may result in loss of estimation

accuracy.

A misspecified model cannot reproduce adequately the observed

covariances, and hence will not fit the data.



Measurement Models


Model Specification

Measurement Models


Model Estimation


Model Modification






Measurement Models

If some of the variables in the model are latent (not directlyobservable), a measurement model must be specified.

This results to constructing observable variables that are supposedto measure the latent variables and defining relationships betweenthese variables and latent variables.



Measurement Models

Example 2 (Continued)

Above, e.g. the achievement motivation was measured with two indices,each of which were constructed as a sum of several other variables.

The measurement model is constructed for the explanatory (exogenous)latent variables and the explained (endogenous) latent variablesseparately.

Thus there are two measurement models, one for the exogenous variables

and one for the endogenous variables.



Measurement Models

Measurement model for the exogenous factors:

��

achv mot

self est

verb int

achm1

achm2

XXXXXy

��9

δ1

δ2

-

-

sestm1

sestm2

XXXXXy

��9

δ3

δ4

-

-

intlm1

intlm2

XXXXXy

��9

δ5

δ6

-

-

!

�

� !

�

�

$

%

�

�

The measurement model for the endogenous variables is constructed in

the same fashion.





Model Specification

Measurement Models


Model Estimation


Model Modification







In SEM the estimated parameters are functions of the samplecovariances.

That is, the system is a set of equations, where the parameters arethe unknowns.

If the unknowns can be solved uniquely, it is said that that themodel is identified.

Otherwise it is under-identified.




Example 3

In principle the question is of analogous to the situation where we havevariables x and y , which satisfy the equation x + y = 10.

With this information we cannot solve uniquely the values of x and y .

There are infinitely many solutions. If we know that x = 2 (say), then

there is a unique solution with y = 8.




Traditionally there are three levels of identification:

(1) Underidentified: If one or more parameters cannot solveduniquely on the basis of the covariance matrix S (moreunknown parameters than equations).

(2) Just-identified: There is just enough information in thecovariance matrix S to solve the parameters of the model(equally number of equations as unknown parameters).

(3) Over-identified: More than one way to estimate the unknownparameters (more equations than unknowns).

It is said that the model is identified if it is either just-identified orover-identified.




Remark 3

In an over-identified case one can statistically test whether the data

is supporting the extra restriction in the model.

Checking whether a model is identified is unfortunately not astraightforward task.

Usually, however, the computer programs can indicate whichparameters are not identified (not water proof).Count rule: A necessary condition is that there are at mostp(p + 1)/2 parameters to be estimated, where p is the numberof observed variables (dimension of the covariance matrix).




In SEM latent variables do not have natural scale.

Typical practices to fix the scales are either

by fixing one of the indicator loading of each latent variable toa constant (typically equal to one),

or defining the variances of latent variables equal to one(standardized latent variables).




Example 4 (Job satisfaction continued)

δ1

δ2

δ3

δ4

δ5

achm1

achm2

sestm1

sestm2

intlm

��

achv mot

self est

verb int

��

job sat

perform

ζ1

ζ2

jobsatm1

jobsatm1

performm

ε1

ε2

ε3

-

-

-

-

-

@I

�

@I

�

�

@@@R

BBBBBBBBBN

��

@@@R

��

��

6

?

��

AK

��3

QQs

-

�

�

�

�

�

1

q

�

��

R�

��

R

1

1

1

1

1

Note: Even though following one of the standard practices of fixing coefficient and the necessarycondition is satisfied, in this case the above model is still not yet identified! More restrictions needed.




Correlation matrix for Example 4 (N = 122):

performm jobsatm1 jobsatm2 achvm1 achvm2 sestm1 sestm2 intlmperfomm 1.000jobsatm1 .418 1.000jobsatm2 .394 .627 1.000achm1 .129 .202 .266 1.000achm2 .189 .284 .208 .365 1.000sesm1 .544 .281 .324 .201 .161 1.000sesm2 .507 .225 .314 .172 .174 .546 1.000intlm -.357 -.156 -.038 -.199 -.277 -.294 -.174 1.000Std dev 2.09 3.43 2.81 1.95 2.06 2.16 2.06 3.65




The identification problem in the above example stems from the fact thatverbal intelligence and performance are measure by one indicators.

One possibility to solve the problem is that intelligence and performanceare considered as observed variables, not latent ones.

Another solution is that if we know the reliability of these measure, we

can use it to identify the model.




We solve the above identification issues by defining Perform factor as an

observed variable performm.

Second we assume that the reliability of the intelligence measure is 85%,

so that the reliability is 15% and hence, as the variance estimate of the

intlm is 2.652 = 13.225 we estimate

var[δ5] = .15× 13.225 = 1.998375 ≈ 2.00, we fix the error variance to

this number.

After fixing these and the unit loadings in the measurement models, we

have 23 free parameters tio be estimated.

There are 8× (8 + 1)/2 = 36 elements in the covariance matrix. Thus,

the necessary conditions of identification is satisfy.




Given the above necessary condition, generally non-recursive models

(models with no loops, i.e. feed-back) are identified.

In a general case we rely on Brown (2006, p 202) ”Because latent variable

software programs are capable of evaluating whether a given model is identified, it is

often most practical to simply try to estimate the solution and let the computer

determine the models identification status.”1

There are three testable hypotheses: Performance and Job Satisfactionare correlated because

H1 : they have Acihev, Selest, and Verbintl as common causes

H2 : Performance influences Job Satisfaction

H2a : Job Satisfaction influences Performance

H3 : Job Satisfaction and Performance influence each other reciprocally

1Brown, T.A. (2006). Confirmatory Factor Analysis for Applied Research. New

York: Guilford Press.



Model Estimation


Model Specification

Measurement Models


Model Estimation


Model Modification






Model Estimation

The observed correlation matrix is S and the model implied(theoretical) correlation matrix is Σ, which is a function of themodel parameters.

The goal in the estimation is to find such parameter values thatthat the theoretical covariance matrix Σ is as close as possible tothe empirical covariance matrix S.



Model Estimation

Testing for hypothesis H1 in the Job satisfaction example, we run the

program by specifying the path model with performm <--> Jobsat in

which <--> denotes for covariation (correlation) between the factors.

Estimation yields covariance equaling 1.063 with p-value .0260

significance.

Standardized estimate (correlation) equals .19175 with p-value .0197

(about the same as non-standardized, as it should be).

The bottom line is that the relationship seems not to be spurious as it is

significant after controlling for Achievement Motivation, Task

Specific Self Esteem, and Verbal Intelligence.



Model Estimation

Testing for H2 the path coefficient is .51652 (p-val .0136) and for

hypothesis H2a the coefficient is .24556 (p-val .0167).

Both are significant, so neither hypothesis can be rejected.

Testing for H3 amounts to estimating a non-recursive model withfeedback relation

Performance →← Job satisfaction.

This causes an additional difficulty for confirming identification.



Model Estimation

Let us see what SAS says by checking the program log.

From the log you find

WARNING: Standard errors and t values might not be accurate with the use of the

Moore-Penrose inverse.

This indicates that SAS has found difficulties. This indicates typically

that the model is not identified (as it indeed is the case).

More restrictions are needed.

The model becomes identified if each of the covariate factor points to

only one endogenous factor.

By statistical non-significance (or low significance), we drop thelinks Achiev --> Perform, Selfes --> Jobsat, and Verbint

--> Perform.



Model Estimation

Estimating the model yields path coefficients estimates Performance

--> Job Satisfaction: .64923 (p-val < .0001) and Job

Satisfaction --> Performance: −.27643 (p-val .1658).

Thus, only Performance --> Job Satisfaction is significant. Thus,

the empirical results indeed support the one way causation.





Model Specification

Measurement Models


Model Estimation


Model Modification







After the model is estimated the next task is to asses how well themodel fits the data.

I.e., how well the theoretical model is supported by the data.

(a) Global type omnibus tests of the goodness of fit of the entiremodel (in SEM, one of the most popular is the the χ2

goodness of fit test, ”the chi-square test”).

(b) Individual parameter tests:

(i) statistical significance of individual parameter estimates (”thet-ratios”, see SAS results of Example 4),

(ii) restrictions, e.g., equality of some parameters, etc.




Some goodness of fit measures

Generally there exist several goodness-of-fit measures.

Here we look at some of the most popular used in particular in(international) business literature.

Most popular measures (in managerial research) seem to bechis-square statistic (χ2), root means square of approximation(RMSEA), comparative fit index (CFI), and non-normed fit index(NNFI).




Chi-square test

The chi-square (χ2) is a goodness-of-fit test statistics of thespecified structural equation model (the null hypothesis) againstalternative on unconstrained saturated model.

In technical terms this means that the population covariancematrix (Σ) satisfies the constraints implied by the structural model

H0 : Σ = Σ(θ) (1)

against the alternative

H1 : Σ 6= Σ(θ) (2)

where Σ(θ) means that the covariance matrix is a function of theparameters θ of the structural equation model.




Chi-square test

Under the null hypothesis the chis-square test statistic has χ2

distribution with degrees of freedom which is the difference ofelements in the covariance matrix and the number of estimatedparameters.

In the Job satisfaction – Performance example, estimating the final

model after dropping Job Satisfaction -- > Performance, gives

χ2 = 14.2 with df = 15 (= 36− 21) and p-value p = .5112, which

implies that the model is consistent with the data.

For other variants and derivatives of the chi-square, see SAS/STATmanual and references therein (see also Kenny’s web-page referredto at this course web-site).




Root means square error of approximation (RMSEA)

Unlike here, typically the chi-square test rejects the model.

As a consequence it is commonly considered as a too strict rule tojudge the usefulness of the model.

An alternative idea is to asses how good an approximation of thereality the proposed model is. (”A good model is ’reasonably’consistent with the data”)

RMSEA is aimed to measure this discrepancy.

Rule of thumb thresholds are: 0.01, 0.05, and 0.08 to indicateexcellent, good, and mediocre fit, respectively.

However, others have suggested 0.10 as the cutoff for poor fittingmodels.




RMSEA

In the Job satisfaction – Performance example RMSEA = 0.000, thus

indicating ’excellent’ fit.

Typically a model is considered fitting ’close’ if RMSEA is .05 orsmaller.

Many software event print a test of this ’close fit’ with a p-value toindicate the significance how much the observed RMSEA isexceeding the .05-value.

In our example ’Probability of Close Fit’ is .7681 indicating (for an

obvious reason) that the observed RMSEA is not statistically significantly

exceeding the close fit threshold.

Also (1− α)% confidence intervals are given for RMSEA. Bydefault α = .10 in many SEM software.

In our example the upper bound with α = .10 is .0817.




CFI and NNFI (or TLI)

Another popular measure is Comparative Fit Index (CFI). Therange is from 0 to 1.0.

Typically a value >= .90 is considered to indicate satisfactorygoodness-of-fit.

Job Satisfaction – Performance: CFI = 1.0000 indicating ”no departure

from close fit”.

Non-normed fit index (NNFI) also called Tuckey-Lewis Index (TLI)is closely related to CFI. NNFI can even be larger than one, inwhich case many software set it 1.

In our example, NNFI = 1.0034 ≈ 1.



Model Modification


Model Specification

Measurement Models


Model Estimation


Model Modification






Model Modification

If the fit of the model is not good enough (which is the usualcase), the model must be modified and subsequently evaluate thefit of the modified model.

In modern SEM-packages, like SAS, there are powerful tools forevaluating how to modify the model (”modification indices”).

These modification indices suggest which ones of the imposedrestriction most strongly disagree with the empirical data.





Model Specification

Measurement Models


Model Estimation


Model Modification







In a non-recursive model with feedback, the total effect is in fact aresult of an infinite loop of partial effect.

Consider the following simple case

��η1 ��

��η2

-�

β21

β12

Keeping other things constant, a one unit change in η1 causes achange of β21in η2, which cause a β12β21 change in η1.




Thus, a one unit change in η1 causes a change in η2, which due tothe link causes a change back in η1 of magnitude β21β12.

This again causes an additional change in η2 of magnitudeβ21(β12β21).

The total effect of unit change in η1 on η2 after one loop is

β21 + β221β12. (3)

The limit is∞∑j=1

βj21βj−112 = β21

∞∑k=0

(β12β21)k . (4)




If |β1β2| < 1 then (4) is a converging geometric series with endresult

β21

∞∑k=0

(β12β21)k =β21

1− β12β21. (5)

In the same manner, a unit change in η2 causes in η1 a totalchange of

total effect η2 → η1 =β12

1− β12β21. (6)

Again in the same manner, the total effect of η1 or η2 on itself canbe calculated to be

total effect ηi → ηi =β12β21

1− β12β21, (7)

i = 1, 2.

If these exist, we say that the system is stable.Seppo Pynnonen Applied Multivariate Analysis



SAS produces a stability index (which is the largest eigenvalue ofthe matrix of β-coefficients).

The index should fall between −1 and 1 in order the model to bestable.

If the index is (on absolute value) ≥ 1, it implies that the model iswrong or the sample is too small for reliable estimates.




Example: Peer Influence on Ambition

Example 5

Peer influences on ambition. Source: Duncan, O.D., A.O. Haller, and A. Portes(1968). Peer influences on aspirateions: A reinterpretation. American Journal ofSociology, 74, 119–137.

rpas: respondent’s parental aspirationrint: respondent’s intelligencerses: respondent’s socioeconomic statusbses: best friend’s socioeconomic statusbint: best friend’s intelligencebpas: best friend’s parental aspiration

roas: respondent’s occupational aspirationreas: respondent’s educational aspirationboas: best friend’s occupational aspirationbeas: best friend’s educational aspiration

rambition: respondent’s ambition (η1)

bambition: best friend’s ambition (η2).









Correlation matrix and standard deviations (N = 329):

===================================================================================

rint rpas rses roas reas bint bpas bses boas beas

-----------------------------------------------------------------------------------

rint 1.0000

rpas 0.1839 1.0000

rses 0.2220 0.0489 1.0000

roas 0.4105 0.2137 0.3240 1.0000

reas 0.4043 0.2742 0.4047 0.6247 1.0000

Best friend

bint 0.3355 0.0782 0.2302 0.2995 0.2863 1.0000

bpas 0.1021 0.1147 0.0931 0.0760 0.0702 0.2087 1.0000

bses 0.1861 0.0186 0.2707 0.2930 0.2407 0.2950 -0.0438 1.0000

boas 0.2598 0.0839 0.2786 0.4216 0.3275 0.5007 0.1988 0.3607 1.0000

beas 0.2903 0.1124 0.3054 0.3269 0.3669 0.5191 0.2784 0.4105 0.6404 1.0000

------------------------------------------------------------------------------------

SD 1.095 1.071 1.030 .978 1.001 1.036 1.021 1.002 1.055 1.022

===================================================================================





Estimation results (SAS CALIS):

Absolute Index Fit Function 0.0814

Chi-Square 26.6972

Chi-Square DF 15

Pr > Chi-Square 0.0313

RMSEA Estimate 0.0488

RMSEA Lower 90% Confidence Limit 0.0145

RMSEA Upper 90% Confidence Limit 0.0783

Bentler Comparative Fit Index 0.9859

Bentler-Bonett NFI 0.9694

Stability Coefficient of Reciprocal Causation = 0.2082863973

Total and Indirect Effects Converge

Stability Coefficient < 1





==========================================================

Path Parameter Estimate Standard

Error t Value Pr > |t|

----------------------------------------------------------

Measurement model

----------------------------------------------------------

Ramb ===> roas 1.00000

Ramb ===> reas _Parm01 1.08767 0.09226 11.7894 <.0001

Bamb ===> boas 1.00000

Bamb ===> beas _Parm02 1.04194 0.07876 13.2287 <.0001

roas <==> roas _Parm03 0.39421 0.04900 8.0458 <.0001

reas <==> reas _Parm04 0.33682 0.05220 6.4519 <.0001

boas <==> boas _Parm05 0.45033 0.05140 8.7606 <.0001

beas <==> beas _Parm06 0.32504 0.04797 6.7759 <.0001

Ramb <==> Ramb _Parm07 0.26876 0.04422 6.0778 <.0001

Bamb <==> Ramb _Parm08 -0.02169 0.04909 -0.4418 0.6586

Bamb <==> Bamb _Parm09 0.25383 0.04285 5.9233 <.0001

=========================================================





==========================================================

Path Parameter Estimate Standard

Error t Value Pr > |t|

-------------------------------------------------------

Structural

-------------------------------------------------------

rpas ===> Ramb _Parm10 0.14722 0.03542 4.1560 <.0001

rint ===> Ramb _Parm11 0.22297 0.03928 5.6763 <.0001

rses ===> Ramb _Parm12 0.20738 0.04197 4.9415 <.0001

bses ===> Ramb _Parm13 0.07012 0.04852 1.4453 0.1484

bpas ===> Bamb _Parm14 0.15326 0.03766 4.0696 <.0001

bint ===> Bamb _Parm15 0.33046 0.04432 7.4562 <.0001

bses ===> Bamb _Parm16 0.22404 0.04389 5.1042 <.0001

rses ===> Bamb _Parm17 0.05893 0.04929 1.1956 0.2318

-------------------------------------------------------

Reciprocal causation

-------------------------------------------------------

Bamb ===> Ramb _Parm18 0.18370 0.09482 1.9374 0.0527

Bamb <=== Ramb _Parm19 0.23617 0.12001 1.9680 0.0491

=========================================================





In terms of χ2 the model does not fit terribly well with p-value 0.031.RMSEA (0.049), CFI (.986), and NNFI (.969) indicate good fit, however.

The model can be improved, not by releasing additional parameters, butimposing additional restrictions.

Testing cov[zeta1, zeta2] = ψ12 = 0, gives

χ2 = 26.893− 26.697 = 0.186

with one degree of freedom, and a test of β12 = β21, given ψ12 = 0, yields

χ2 = 26.899− 26.893 = 0.006,

again with one degree of freedom. Obviously these hypotheses cannot berejected.

The overall goodness-of-fit, given ψ21 = 0 and β12 = β21 = 0, is

χ2 = 26.89 with 17 degrees of freedom and p-value 0.060 (borderline

accepted).





To proceed the analysis, from the path coefficient estimates it is seenthat the corresponding estimates of the respondent and the best friendare close to each other.

We next test whether the model is completely symmetric between therespondent and the best friend.

This is done by equating the estimates of the corresponding bathcoefficients of the respondent and the best friend.

Furthermore, the corresponding variances are equated.





With these restriction the model looks the following





The overall chi-square for this model is 33.44 with 25 degrees of freedomand a p-value of 0.121, RMSEA = .032, CFI = .990, and NNFI = .982.

Thus, this model is more parsimonious and has a better fit than the other

models.





Model Specification

Measurement Models


Model Estimation


Model Modification







Considering the general SEM system: y = Λyη + ε, x = Λxξ + δ,η = Bη+ Γξ+ζ, the direct, indirect, and total effect are as follows

ξ → η η → ηDirect Γ BIndirect (I − B)−1Γ− Γ (I − B)−1 − I − BTotal (I − B)−1Γ (I − B)−1 − I

ξ → y η → yDirect 0 Λy

Indirect Λy (I − B)−1Γ Λy (I − B)−1 − Λy

Total Λy (I − B)−1Γ Λy (I − B)−1

(see computer output)





Model Specification

Measurement Models


Model Estimation


Model Modification







Modification index is a measure is a measure of predicted changein the goodness-of-fit statistic, χ2, if a single fixed parameterrestriction is relaxed and the model is re-estimated.

Thus, if a fitted model is not satisfactory, modification indices canbe used to evaluate which relaxed restrictions would most improvethe model.

Typically modification indices are utilized in particular whenestimating and specifying the measurement models. We’ll returnto this later.





In our example the largest modification indices are the following:

Path Relations:

LM stat p-val Par Change

boas <--- roas 10.14 .0015 .152

roas <--- boas 6.26 .0123 .120

Error Covariances:

M.I. p-val Par Change

e1 <--> e4 7.76 .0003 -.086

e1 <--> e3 13.21 .0053 .117





Largest gain in χ2 would be obtained if the covariance between the errorterms ε1 and ε3 were relaxed.

Re-estimating the model with this modification produces a χ2 = 19.9with 24 degrees of freedom and p-value of 0.704.

Thus the true drop in χ2 is 13.5, which is a little larger than thepredicted.

However, because the original model indicates a satisfactory fit there is

no need to adopt the change.




Remark 4

Relaxing parameters should not be driven purely by data. Additional freeparameters should have sound substance based interpretation.

A data driven model improvement leads easily to a capitalization of

change, not true dependencies.


Documents

Applied Multivariate Analysis - Vaasan yliopistolipas.uwasa.fi/~sjp/Teaching/mva/lectures/c7.pdf · Applied Multivariate Analysis ... These background factors serve from statistical