Joshua Naranjo - wmich.edu · Joshua Naranjo Questionnaire validity. Joshua Naranjo Questionnaire validity. Factor Analysis: Explained variance Recall that S = P 15 i=1 iv iv i 0˘=P

Questionnaire validity

Joshua Naranjo

Stat 5630Western Michigan University

Joshua Naranjo Questionnaire validity

Questionnaire theory

Recall: Study context

research aims

design

measurement/questionnaire (satisfaction, ease of use,religiosity)

analysis

Statistical concepts in questionnaire theory

reliability and validity

intraclass correlation coefficient

Cronbach’s alpha

factor analysis



Reliability and validity of measurements

Ex: ”Association between intelligence, communication skills, andeffective leadership”

Validity – does an instrument measure what it claims to measure?Reliability – does it work consistently?



Types of validity:

Construct validity – is it valid in theory?

Face validity – does it measure what it aims to measure?

Convergent validity – does the survey compare well withanother measure of the same thing?

Discriminant validity – can it tell different groups apart?

(Ex: intelligence, communication skills, leadership)



Types of Reliability:

Internal reliability – do respondents give consistent answersover similar questions?

Test-retest reliability – do respondents give consistent answersover time?

Inter-rater reliability – do raters give consistent answers forthe same item?


Intraclass correlation coefficient

Ex: “How much agreement is there between physicians reading CTscans for disease progression?”

CT Scan Physician 1 Physician 21 7 52 6 9: : :: : :n 2 2

Q: Pearson correlation?

A: Which student is X and which is Y ?



Ex: “How much agreement is there between physicians reading CTscans for disease progression?”

CT Scan Physician 1 Physician 21 7 52 6 9: : :: : :n 2 2

Q: Pearson correlation?A: Which student is X and which is Y ?



ICC measures agreement between exchangeable variables (i.e. noinherent ordering).Used for test-retest reliability, and inter-rater reliability.

Let

x =1

2n

∑(xi1 + xi2)

s2 =1

2n

{∑(xi1 − x)2 +

∑(xi2 − x)2

}Then

ICC =1

ns2

∑(xi1 − x)(xi2 − x)



For groups of three,

x =1

2n

∑(xi1 + xi2 + xi3)

s2 =1

2n

{∑(xi1 − x)2 +

∑(xi2 − x)2 +

∑(xi3 − x)2

}Then

ICC =1

3ns2

∑{(xi1 − x)(xi2 − x) + (xi1 − x)(xi3 − x)

+(xi2 − x)(xi3 − x)}


Cronbach alpha

Internal Consistency: Hearing aid survey

Suppose we want to measure how well the hearing aid “listens inconversation”

Q: Do the questions “measure the same thing”?


Cronbach’s alpha:

Suppose that T = x1 + · · ·+ xk is the composite score for a factoror construct. Then

α =k

k − 1

(1−

∑var(xi )

var(T )

)


Cronbach alpha

α = 1010−1

(1− 19.022

40.6933

)= 0.59172


It may be easier to understand as follows:

α =k

k − 1

(1−

∑var(xi )

var(T )

)=

k

k − 1

(var(T )−

∑var(xi )

var(T )

)

=k

k − 1

(∑i 6=j cov(xi , xj)

var(T )

)So α is an average covariance between the xi s.


Cronbach alpha

But this is still just a computing formula. In CronbachPsychometrika, 1951, Cronbach proposes the statistic as a

“....mean of all split-half coefficients resulting from differentsplittings of a test... therefore an estimate of the correlationbetween two random samples of items...”


Examples

Published example:


Factor Analysis


Factor Analysis


Factor Analysis


Factor Analysis

Let Yi be the response to the ith question, i = 1, . . . , 15. The ideabehind factor analysis is that the 15 responses actually depend ononly two or three common underlying unobservable factors, i.e.

Yi = µi + ai f1 + bi f2 + ε

History of factor analysis

Charles Spearman (1904) two-factor theory of intelligence:general and specific

Thurstone 7-factor: numerical, verbal comprehension, wordfluency, perceptual speed, memory, inductive reasoning,spatial ability

Sternberg 3-factor: analytical (problem-solving), creative (newideas), and practical (everyday logic)


Factor Analysis

Back to article, let

Y1

Y2

:Y15

=

µ1µ2:µ15

+

l11 l12l21 l22: :

l15,1 l15,2

( f1f2

)+

ε1ε2:ε15

or

Y = µ+ LF + ε

where µ and L are constants, F and ε have mean 0

V (F) =

(1 00 1

), V (ε) = Ψ =

ψ1 0 . . . 00 ψ2 . . . 0...

......

0 0 . . . ψ15


so thatV (Y) = LL′ + Ψ

Estimation:

S =15∑i=1

λivivi′ ∼=

2∑i=1

λivivi′ =

(√λ1v1

√λ2v2

)( √λ1v1′

√λ2v2

′

)= L̂L̂′

where λ1 > λ2 > · · · > λ15. The estimated coefficients of theunderlying factors are derived from the eigenvalues andeigenvectors of S.


Factor Analysis: Rotation

Let A be any 2× 2 orthogonal matrix, i.e. AA′ = I. Then

Y = µ+ LF + ε = µ+ LAA′F + ε = µ+ L∗F∗ + ε

so that there is an ambiguity to the factor analysis model. But thiscan be exploited, might as well choose A′F∗ = (f ∗1 f ∗2 )′ so that thefactors are more interpretable. In practice, this means we want torotate so that each of the 15 questions will have high coefficient inonly one factor.



Factor Analysis: Explained variance

Recall that S =∑15

i=1 λivivi′ ∼=

∑2i=1 λivivi

′

trace(S) = s21 + s22 + · · · s215= λ1 + λ2 + · · ·+ λ15

The proportion of total variation explained by the ith factor is

λiλ1 + λ2 + · · ·+ λ15



Documents

Joshua Naranjo - wmich.edu · Joshua Naranjo Questionnaire validity. Joshua Naranjo Questionnaire validity. Factor Analysis: Explained variance Recall that S = P 15 i=1 iv iv i 0˘=P