Managerial EpidemiologyPart III:

Measuring Health Outcomes

Ty Borders, Ph.D.

Assistant Professor

Department of Health Services Research & Management

School of Medicine

Learning objectives

• Describe and interpret validity

• Describe and interpret reliability

• Calculate sensitivity and specificity

• Identify potential health status and satisfaction measures

• Interpret a disease management guideline

Validity and Reliability

• Validity– the degree to which you are measuring what you

intend to measure

• Reliability– Consistency of a measure– Often expressed as a correlation coefficient

Types of validity

• True validity is based on some kind of gold standard, but usually there is not a gold standard

• Content validity (face validity) – the degree to which a measure represents a concept– determined by a panel of experts

• Construct validity

– compare results of measure to other phenomenon

Types of validity

• Criterion validity

– Compare a measure to other measures– Sensitivity

• Proportion of those who test positive for a trait and have a trait

– Specificity• Proportion of those who test negative for a trait and do

not have a trait

Sensitivity and specificity

Disease

Measure results Present Absent Total

Positive True + False + a + b

Negative False - True - c + d

Totals a + c b + d a+b+c+d

sensitivity = a / (a+c)specificity = b / (b+d)

Example: Cage questionnaire

Alcoholism

Score Present Absent Total

3-4 positives 130 11 141

0-2 positives 164 516 680

Totals 294 527 821

sensitivity = 130/294 = 44%specificity = 516/527 = 98%

Assessing Health Status - What is health?

• A simple definition– The presence or absence of disease

• WHO definition – “complete physical, mental, and social

well-being, and not merely the absence of disease or injury”

What are disease and illness?

• Disease– Professionally defined

– Precise

– Used for treatment

• Illness– Lay definition

– Individual’s reaction to biological state

– Influenced by culture

Assessing health status

• 5 outcomes of disease (5 Ds)

– Death (if unexpected or untimely)

– Disease (a set of symptoms, signs, and laboratory abnormalities)

– Discomfort (pain, nausea, itching, etc.)

– Disability (impaired ability to go about usual activities)

– Dissatisfaction (emotional reaction to disease)

A broader definition of health:Health-Related Quality of Life

• Health-related quality of life (HRQL) encompasses those aspects of life that are dominated or significantly influenced by personal health or activities performed to maintain health.

Core concepts and domains of HRQL

• Health perceptions – General health perceptions – Satisfaction with health

• Functional status– Social function– Psychological function– Physical function

Core concepts and domains of HRQL (cont.)

• Impairment– Symptoms/subjective complaints– Signs (observable evidence of abnormality)– Diagnoses (clinical judgments)

• Death and duration of life

• Opportunity– Resilience; social or cultural disadvantage

HRQL measures

• Disease-specific– Arthritis Impact Measurement Scale

• Generic– Sickness Impact Profile– Quality of Well-Being Index– SF - 36– SF-12

• Shorter version of the SF-36

• Measures physical and mental/emotional health

SF - 12

• Validity– Comparison of scores among people with

different diseases– Comparisons with other measures

• Test-retest reliability– Physical component score 0.864 to 0.890– Mental component score 0.760 to 0.774

Health-Related Quality of Life (SF-12)Adjusted for Predisposing, Enabling, and Need Factors

43.0552.7

41.27

54.9

0

20

40

60

80

100

Mean PhysicalScore

Mean MentalScore

Hispanics Non-Hispanic Whites

Measuring quality

• Technical care (Donabedian)

– Application of science, technology of medicine (and other health sciences) to the management of personal health

• Interpersonal care – Management of the social and

psychological interaction between client and practitioner

Dimensions of satisfaction

• Overall satisfaction with health care

• Overall satisfaction with provider

• Satisfaction with accessibility– Ability to see personal doctor– Ability to get appointment with specialist– Appointment waiting time– Costs of care– How often get help when phone doctor’s office

Dimensions of satisfaction (cont.)

• Satisfaction with quality– Interpersonal manner of physician– Communication– Office waiting times– Facility characteristics– Time spent with doctor– Technical competence

Level of measurement

• Clinic or provider level– Medical Outcomes Study (MOS) survey– Measures satisfaction with a particular encounter

• Health plan or population level– Consumer Assessment of Health Plans Survey– Developed by Agency for Healthcare Quality and

Research (formerly AHCPR)– Frequently used to assess health plan performance

Phys. function

Psych. function

Social function

Plan, institution, systemOrganized team

Several practitioners

Individual practitioner

Patient PersonIndividual Case load Individual Population

Comparison of satisfaction measuresInstrument Dimensions Data collection

PSQ-III Patient satisfaction 50 items, mailed

with medical care

CSQ-8 General satisfaction 8 items, self-admin.

SERVQUAL Reliability, empathy, 22 items, self-admin.

responsiveness,

assurance

VSQ Outpatient satis. 7 items, self-admin.

following visit

CAHPS Access, coms, choice 47 items, multiple

continuity, overall modes of collection

satisfaction, plan satis.

CAHPS• Single items, 0 (worst) to 10 (best)

– Global rating of health plan– Global rating of quality of care– Global rating of personal doctor– Global rating of specialist

• Composites– Getting care you need (4 items, never to always)– Getting care without long wait– Doctors communicate well with patients– Doctors spend enough time with patients– Reasonable paperwork, handling of payments

On a scale from 0 to 10 with 10 being the best…

how would you rate the health care you received

0 20 40

10

8

6

4

2

0

Rat

ing

Percentage

Global Satisfaction

Threats to the reliability and validity of surveys

• Validity– Choose non-validated measures– Choose wrong measures

• Reliability– Unrepresentative sample– Inadequate sample size

– Poor response rate

Administration

• In-person

• Telephone

• Mail survey

• Social desirability– Tendency to offer answers that are consistent

with values the respondent believes to be held by interviewer/person conducting survey

– Confidentiality issue

Response rates

• # completed surveys / # distributed

• Non-reponse error can occur if the response rate is not 100%

• As a rule of thumb, do not use data if the response rate is below 60%

• Increasing the sample size does not reduce the non-response error

Sampling and sample size

• When conducting surveys, more efficient to sample a subset of the population

• A sample approximates the population

• Sample size depends on several factors

• For more information, refer to Aday’s, Designing and Conducting Health Surveys