Chapter5 Validity and Reliability

7/30/2019 Chapter5 Validity and Reliability

1/26

RELIABILITY AND VALIDITY

LOUIS COHEN, LAWRENCE MANION

& KEITH MORRISON


2/26

STRUCTURE OF THE CHAPTER

Defining validity Validity in quantitative research

Validity in qualitative research

Types of validity Triangulation

Ensuring validity

Reliability Reliability in quantitative research

Reliability in qualitative research


3/26

STRUCTURE OF THE CHAPTER

Validity and reliability in interviews

Validity and reliability in experiments

Validity and reliability in questionnaires

Validity and reliability in observations

Validity and reliability in tests

Validity and reliability in life histories


4/26

VALIDITY IN QUANTITATIVE AND

QUALITATIVE RESEARCH

Validity in quantitative research often

concerns: objectivity, generalizability,

replicability, predictability, controllability,homothetic statements.

Validity in qualitative research often

concerns: honesty, richness, authenticity,

depth, scope, subjectivity, strength of feeling,

catching uniqueness, idiographic statements.


5/26

Concurrent Consequential

Construct

Content Criterion-related

Convergent & discriminant

Cross-cultural Cultural validity

Descriptive

Catalytic

Ecological Evaluative

External

Face Internal

Interpretive

Jury Predictive

Systemic

Theoretical

TYPES OF VALIDITY


6/26

VALIDITY IN QUANTITATIVE RESEARCH

Concurrent

Construct

Content

Criterion-related

Convergent & discriminant

Cross-cultural

Evaluative

External

Face

Internal

Jury

Predictive

Theoretical


7/26

DIRECTION

OF CAUSALITY

MATURATION

TESTING

THREATS TOVALIDITY AND

RELIABILITY

TYPE 1 AND

TYPE 2

ERRORS

INSTRUMENT-

ATION

OPERATIONAL-

IZATION

REACTIVITY

HISTORY

EXPERIMENTAL

MORTALITY

CONTAMIN-

ATION


8/26

ESTABLISHING VALIDITY IN

QUALITATIVE RESEARCH Prolonged engagement inthe field

Persistent observation

Triangulation

Leaving an audit trail

Respondent validation

Weighting the evidence (giving priority)

Checking for representativeness

Checking for researcher effects Making contrast/comparisons

Theoretical sampling

Checking the meaning of outliers

Using extreme cases


9/26

ESTABLISHING VALIDITY IN

QUALITATIVE RESEARCH Ruling out spurious relations

Replicating a finding

Referential adequacy

Following up surprises

Structural relationships

Peer debriefing

Rich and thick description

Looking for possible sources of invalidity Assessing rival explanations

Negative case analysis

Confirmatory data analysis

Effect sizes


10/26


11/26

THREATS TO EXTERNAL VALIDITY

IN QUANTITATIVE RESEARCH

Failure to describe independent variables explicitly

Lack of representativeness of available and target

populations

Hawthorne effect

Inadequate operationalizing of dependent variables

Sensitization/reactivity to experimental/research conditions

Interaction effects of extraneous factors and experimental/

research treatments

Invalidity or unreliability of instruments

Ecological validity

Multiple treatment validity


12/26

THE HAWTHORNE EFFECT

Between 1927 and 1932 researchers carried out

experiments at the Western Electric Companys

Hawthorne plant.

Purposes: To examine the effects of changes of

working conditions on output of workers Sample: Six women, chosen as average workers

Method: Women worked in a test room. Output

measured under different conditions (e.g. no change

change to method of payment introduce tworest periods introduce six rest periods

changes in lighting conditions, early clocking-off,

five-day working week return to initial conditions

Duration: 15 weeks


13/26

THE HAWTHORNE EFFECT

Results: Output rose steadily during testperiod and after the test period.

Conclusion: Output did not seem to depend

on test conditions. Increased output seemedto be due to the fact that the people had been

involved in the experiment itself, i.e. the act of

research had affected the results. The

results were a research of the research itself. Implications: The act of being involved in

research itself affects the results.


14/26

THREATS TO EXTERNAL VALIDITY

IN QUALITATIVE RESEARCH Selection effects Setting effects

History effects

Construct effects


15/26

ENSURING VALIDITY AT THE

DESIGN STAGE

Choose an appropriate time scale;

Ensure adequate resources for the research

Select appropriate methodology Select appropriate instruments

Use an appropriate sample

Ensure reliability Select appropriate foci

Avoid having biased researcher(s)


16/26


DATA COLLECTION STAGE

Reduce the Hawthorne effect

Minimize reactivity

Avoid drop-out rates amongst respondents

Take steps to avoid non-return of questionnaires Avoid too long or too short an interval between pre-tests

and post-tests

Ensure inter-rater reliability

Match control and experimental groups Ensure standardized procedures for gathering data

Build on the motivations of respondents

Tailor instruments to situational factors

Address researchercharacteristics


17/26


DATA ANALYSIS STAGE

Use respondent validation; Avoid subjective interpretation of data

Reduce the halo effect

Use appropriate statistical treatments

Recognize extraneous factors which may affect data

Avoid poor coding of qualitative data

Avoid making inferences/generalizations beyond the data

Avoid equating correlations and causes Avoid selective use of data

Avoid unfair aggregation of data

Avoid degrading the data;

Avoid Type I and/or Type II errors


18/26


DATA REPORTING STAGE

Avoid using data selectively and unrepresentatively

Indicate the context and parameters of the

research

Present the data without misrepresenting the

message

Make claims which are sustainable by the data

Avoid inaccurate or wrong reporting of data Ensure that the research questions are answered

Release research results neither too soon nor too

late


19/26

RELIABILITY IN QUANTITATIVE

AND QUALITATIVE RESEARCH

Reliability in quantitative research:

consistency (stability), accuracy,predictability, equivalence, replicability,concurrence, descriptive and causalpotential.

Reliability in qualitative research:

accuracy, fairness, dependability,comprehensiveness, respondent validation,checkability, empathy, uniqueness,explanatory and descriptive potential,confirmability.


20/26

Reliability as stability:

Consistency over time and samples;

Reliability as equivalence:

Equivalent forms of same instrument;

Inter-rater reliability;

Reliability as internal consistency:

Split half reliability (e.g. for test items)

TYPES OF RELIABILITY IN

QUANTITATIVE RESEARCH


21/26

TRIANGULATION

Methodologies

Instruments

Researchers

Time Location

Theories

Samples Participants

Data


22/26

SPLIT-HALF RELIABILITY

(Spearman-Brown)Reliability =

r = the actual correlation between the two halvesof the instrument (e.g. 0.85);

Reliability = = = 0.919

r

r

12

85.01

)85.0(2

185

70.1


23/26

RELIABILITY IN QUALITATIVE

RESEARCH

Credibility

Neutrality

Confirmability Dependability

Consistency

Applicability Trustworthiness

Transferability


24/26

RELIABILITY AND REPLICATIONIN QUALITATIVE RESEARCH

Repeat: The status position of the researcher

The choice of informants/respondents

The social situations and conditions

The analytic constructs used

The methods of data collection and analysis

Address: Stability of observations

Parallel forms

Inter-rater reliability

Respondent validation


25/26

IMPROVING RELIABILITY

Minimise external sources of variation; Standardise conditions under which

measurement occurs;

Improve researcher consistency;

Broaden the sample of measurementquestions by:

a) adding similar questions to theinstrument;

b) increasing the number of researchers(triangulation);

c) increasing the number of occasions inan observational study.

Exclude extreme responses (outliers).


26/26

RELIABILITY AND VALIDITY AT

ALL STAGES Design and methodology

Sampling

Instrumentation

Timing

Data collection

Data analysis

Data reporting

Documents

Chapter5 Validity and Reliability