HSAG Performance Improvement Training HSAG Performance Improvement Training Statistical TestingStatistical Testing

Presented by Donald Grostic, MS

Health Services Advisory Group, Inc.

January 11, 2011

Agenda

I. Background Why do statistical testing? When is statistical testing used? Which PIP Activities and Evaluation Elements does statistical

testing apply?II. What statistical test do I use?

Chi-square test of inference How is the numerator and denominator for the rate of home-

delivered meals within 3 days used in a Chi-square test? Website recommendation for Chi-square test How do I interpret the results of the Chi-square test?

III. Question and answer session

Background

• Why do statistical Testing?– The CMS protocols require that statistical

testing be used to prove that any improvement in rates is real.

– Without statistical testing, a PIP would not meet the CMS protocols.

Background (continued)

• When is statistical testing used?– After collecting annual baseline data, completing a

causal barrier analysis, implemented interventions and collecting an annual first remeasurement.

– At this point, statistical testing should be completed to compare the annual baseline rate to the annual first remeasurement rate.

– Statistical testing would need to be performed again between the annual remeasurement 1 rate and the annual remeasurement 2 rate after a causal barrier analysis is completed.

Background (continued)

• Which PIP Activities and Evaluation Elements include statistical testing?– Activity VIII Evaluation Element 7: Identifies

statistical differences between initial measurement and remeasurement.

– Activity IX Evaluation Element 4: There is statistical evidence that observed improvement is true improvement.

– Activity X Evaluation Element 1: Repeated measurements over comparable time periods demonstrate sustained improvement, or that a decline in improvement is not statistically significant.

Chi-square Test

• What statistical test should I use?– For rates that are calculated using a numerator and

denominator, the Chi-square test, Fisher’s exact test or the Z test for proportions should be used used.

– For example if the home-delivered meals within 3 days measure for the baseline year had 20 clients receiving meals within 3 days out of 42 total clients, the rate would be 20/42 = 0.47x100 = 47.6 %.

– If in the annual first remeasurement had 60 clients receiving meals within 3 days out of 78 total clients, the rate would be 60/78 = 0.76.9x100 = 76.9%.

Chi-square Test (continued)

Hypothetical data showing Chi-square test Time Frame (X)HD Meals 3 Days (Y) Baseline R1 TotalYes (numerator) 20 60 80No 22 18 40Total (denominator) 42 78 120

• How is the numerator and denominator for the rate of home-delivered meals within 3 days used in a Chi-square test?

Chi-square Test (continued)

• Website recommendation for a Chi-square test.– http://faculty.vassar.edu/lowry/tab2x2.html

X

0 1 Totals

Expected Cell Frequencies per Null Hypothesis

1 20

60

80

28

52 Y

0 22

18

40

14

26

Totals 42

78

120

Time Frame (X)HD Meals 3 Days (Y) Baseline R1 TotalYes (numerator) 20 60 80No 22 18 40Total (denominator) 42 78 120

Note: Only enter numbers with arrows, the remaining numbers are calculated by the website.

Chi-square Test (continued)

• Chi-square value = 10.55 • P-value = 0.001162

Chi-Square Phi Yates Pearson +0.3

9.27

10.55

P 0.002329

0.001162

Chi-square Test (continued)

• How do I interpret the results of the statistical test?– If the p-value is less than 0.05, then the change in

rates is statistically significant.– Since the p-value is equal to 0.001162, which is

less than 0.05, there is a statistically significant improvement in the rate of home-delivered meals within 3 days between the baseline rate (47.6%) and first remeasurement rate (76.9%).

Questions

?