V ALIDITY IN Q UALITATIVE R ESEARCH. V ALIDITY How accurate are the conclusions you make based on your data analysis? A matter of degree Non-reification

VALIDITY IN QUALITATIVE RESEARCH

VALIDITY

How accurate are the conclusions you make based on your data analysis?

A matter of degree

Non-reification – validity is not a static entity

VALIDITY IN QUANTITATIVE RESEARCH

Content validity: How well does the measure cover the domain of the subject it is testing?

Criterion-related validity:Predictive validity: How well does the

measure predict appropriate success and failure?

Concurrent validity: How well does the measure predict appropriate success and failure in agreement with other (typically performance-based) similar measures?


Construct validity – Is the measure a valid measure of the construct?

Face validity: Does the measure meet the expectations for a valid test?


Threats to internal validity: Rival hypotheses, confounding variables, and alternative explanations (is the treatment in this case responsible for the observed changes?)


Threat Ways to ReduceSampling error and chance error Use inferential statistics.

History: Events that occur prior to the post-test might cause the effect

Use control group that is also subject to the events under question.

Instrumentation •Consistency in instrument•Quality of instrument (validity/ reliability)

Testing: Use of two or more testings with the same (or closely related) instrument – early testing experience may influence later results.

Use control group – should exhibit the effects of testing minus the treatment.

Regression: Occurs when extreme groups are selected for treatment without a control group.

Use control group.


Threat Ways to ReduceMortality: Individuals leave an experimental group prior to completion of the study.

Use control group with similar mortality.

Maturation: Growth or change, unrelated to the treatment, that might affect the measured effect occurs prior to completion of the study

Use control group.

Instrument decay: Changes in the way an instrument is used to measure the effect during the study (e.g. re-interpretation of scoring categories)

Instrument solidification prior to use Also consider ceiling and floor

effects.

Selection: Subjects are selected according to a factor that causes the same effect as the treatment.

Random assignment to groups


Threats to external validity: Restrictive conditions and explanations (To whom and what circumstances can the results be generalized?)

Six areas of research design that may restrict generalizibility:1. Subjects2. Situation3. Treatment4. Observation or measure5. Basis for sensing changes6. Procedure


Threat Ways to ReduceObtrusiveness and reactivity: Hawthorne effect Hypothesis guessing Guinea pigs Desire for treatment Rivalry between experimental and

control group (random assignment) Novelty effect Demoralization (control group

does not get special help) Diffusion (treatment is

communicated outside the context of the study)

Emotional bonding

Use unobtrusive procedures as much as possible


Threat Ways to ReduceResearcher expectancy: Those giving the treatment are aware of its potential effects

Keep researchers blind to which subjects are experimental or control

Multiple treatment interaction: Residual effects of one treatment influence a later treatment

Rotate the treatment into first position

Testing-treatment interaction: Treatment effect is affected by pre-testing

Post-test only designSolomon four-group design

Selection-treatment interaction: Treatment affects who is selected (e.g., people who are in particular need of the treatment)

Use of volunteers for treatment and control groups (weakens external validity)

Mortality-treatment interaction: Subjects drop out in reaction to the treatment

Use control group with similar mortality.

RELIABILITY IN QUANTITATIVE RESEARCH

Reliability: Does the measure yield consistent results? Does it measure consistently? Inter-observer/ inter-rater reliability Internal consistency: Are all items measuring

appropriate and similar things? Equivalence: Are different forms of the measure

equivalent? Stability: Is the behavior stable over time?


Researcher as instrument: Are you seeing/hearing what is really there? Are you seeing/hearing what is important to you? Are you presenting findings that accurately

reflect the reality of the studied context?

How will you know when You’ve collected enough data?


Coherence Comprehensiveness Authenticity Plausibility Trustworthiness Reflexivity Particularity Utility Accuracy Transferability


Maxwell (1992) Descriptive validity Interpretive validity Theoretical validity Generalizability Evaluative validity


Strategies: Methodological fidelity Data saturation Triangulation Discrepant (negative) case analysis Multi-method approaches Researcher-as-instrument Memoing Audit trail Member-checking

Documents

V ALIDITY IN Q UALITATIVE R ESEARCH. V ALIDITY How accurate are the conclusions you make based on your data analysis? A matter of degree Non-reification