1

Reliability and Validity

Quality of Data

2

Are we testing what we think we’re testing?

3

Quantitative Data

Reliability Validity

Face External Internal

4

Reliability

Implies that the same data would have been collected each time over repeated tests/ observations.

Would a particular technique (or survey question) yield the same result each time? “Did you go to church last week?” vs. “How

many times have you been to church in your life?”

Reliability does not ensure accuracy. Taken from Babbie, E.

5

Reliability

Problem if interpret questions differently

Poorly worded questions Inconsistent coding: coding errors as

with open-ended questions Lack of definition of key terms

6

Reliability

Poorly worded: Does the library have adequate facilities and equipment for physically disabled students

Better: Can patrons in wheelchairs retrieve books from the browsing collection?

7

Reliability (indicators)

Pretest Repeat question(s) Test/retest Split half and Parallel Interscore or scorer

8

Validity

A term to describe a measure that accurately reflects the concept it is intended to measure.

Which is a more “valid” indicator of intelligence- an IQ score, or number of hours spent studying?

Ultimate validity cannot be proven, but can be supported by face, internal, and external measures.

Babbie, E.

9

Types of Validity

Face validity: The quality of an indicator/ question/ test that makes it a reasonable measure of a variable. Church attendance is an indication of

religiosity. Number of grievances filed is an indicator

of worker morale

10

Internal Validity

Approximate truth about inferences regarding causal relationships

Typically applied to studies using inferential statistics (i.e. quantitative measures) than descriptive or observation studies.

Especially useful for studies assessing affects of programs

Only applicable to the study in question- not generalizable. Why not?

Key question: Whether observed changes can be attributed to your program (the cause) and NOT other possible impacts/ causes.

11

Internal Validity

Trochim, W.

12

Internal Validity

History or specific eventsHistory or specific events: raises the issue that some variable other than the independent variable accounted for the change in the dependent variable. E.G.: the length of time between conducting the pretest and posttest may have a detrimental effect.

MaturationMaturation: the change results from biological or psychological processes, which occurred over time, and not from the treatment itself. Maturation becomes more a concern the longer the period between the pretest and posttest

PretestingPretesting: may affect the dependent variable. Pretesting may alert participants or educate them about the topic under investigation. Therefore if subjects are administered a posttest, their performance may reflect a marked improvement

Measuring instruments or observational techniquesMeasuring instruments or observational techniques: These—not the treatment—may account for the change in the dependent variable. Further, the validity of study findings may have been influenced by the fact that the evaluators as observers, raters, graders, interviewers, and coders gained experience, became tired, obtained a more complete understanding of the project, or eased their expectations of test subjects

13

Internal Validity (continued)

A nonrandom assignment of subjectsnonrandom assignment of subjects to groups may signify that the groups were dissimilar from the beginning. Therefore any change might be attributed to the differential selection of subjects, rather than the actual treatment.

Statistical regressionStatistical regression refers to the tendency for extreme scores to regress or move toward the common mean of subsequent measures. The assignment of subjects to a particular test group on the basis of extreme views may affect study findings.

14

Internal Validity (continued)

MortalityMortality refers to the possibility that some subjects may have dropped out of the study after completion of the pretest but before the administration of the posttest. In such instances, every effort should be made to identify any common patterns or characteristics to ensure that any difference between a group’s pretest and posttest scores cannot be attributed to the loss of subjects.

InteractionInteraction refers to the fact that more than one of the previous threats might be in play. This is especially likely in those cases where subjects were not randomly assigned to groups and the evaluation was based on existing, intact groups.

15

External Validity

The approximate truth of generalizations drawn from a study.

The degree to which conclusions drawn from your study sample would hold true to other persons in other places at other times

Trochim, W.

16

External Validity

Trochim, W.

17

External Validity

Example: institutions of higher education in Massachusetts: control, highest degree offered, and some characteristics of library (staff number, budget, and volume number)

Return rate? Do respondents differ from non-

respondents as a group?

18

Validity

Content validity (for achievement test): How well does the test sample what the students learned? How well does a standardized test cover what was taught in the information literacy program?

19

Validity (continued)

Criterion-related (predictive) (attitude test to predict performance in a library skills program): Who well does the test predict achievement for college freshmen?

Criterion-related (diagnostic): How well does the test diagnose current problems with library use?

20

Validity (continued)

Construct validity: How well does the test measure comprehension of library use? Does a test on the use of an OPAC really measure effective and efficient use rather than one’s ability to read test items?

21

Qualitative Study Equivalent

Credibility Dependability Confirmability Transferrability

22

Qualitative Reliability

Researcher is the “instrument”- how to test for reliability? Provide details of method, and abundance of evidence Provide evidence of qualifications as observer Make assumptions (and possible biases) clear State research questions clearly Use early stages of study to generate focus Observe for an adequate period of time, across a full

range of activities Collect data from multiple sources Save data for reanalysis

23

Qualitative Validity

Depends upon reliablity. Like reliability, asserted by documenting steps Triangulation- data from different sources/ methods Full documentation of data- “chain of evidence” Logical connections between data and conclusions Conscious and deliberate inclusion of data that might not

support thesis Preparedness to entertain alternatives Self-reflection, acknowledgement of own biases Review of preliminary reports by objective observers Awareness of limitations

Gorman and Clayton

24

Qualitative Study: Increasing Reliability and Validty

Inquiry affected by

Results in Account for by:

To lead to For findings that are:

During After

Factor patternings

Non-interpretability

Prolonged engagement

Persistent observation

Peer Debriefing

Triangulation

Member checks

Establish structural corroboration (coherence)

Credibility Plausible

Situational Uniqueness

Non-comparability

Collect thick descriptive data

Do theoretical/ positive sampling

Develop thick description

Transferability Context relevant

-Gorman and Clayton

25

Qualitative Study: Increasing Reliability and Validity cont’d

Inquiry affected by

Results in Account for by:

To lead to For findings that are:

During After

Instrumental changes

Instability Use overlap methods

Use stepwise replication

Leave audit trail

Do dependability audit (process)

Dependability Stable

Investigator Predilections

Bias Do triangulation

Practice reflexivity (audit trail)

Do confirmability audit (product)

Confirmability Investigator-free

-Gorman and Clayton

26

Example

For a sweeping study

When conduct it?

For how long?

How deal with reliability and validity?

Course evaluation

27

References

Babbie, E. (2005). The basics of social research. Belmont, CA: Wadsworth Publishing.

Gorman, G.E. & Clayton, P. (2005). Qualitative research for the information professional: A practical handbook. London: Facet Publishing.

Trochim, W. M. K. (2006). Research methods knowledge base. Retrieved July 8, 2008 from http://www.socialresearchmethods.net/kb/intval.php

http://www.socialresearchmethods.net/kb/external.php