Overview of the Decision Guide: A Public Reporting Resource for CVEs

Patrick S. Romano, MD MPHProfessor of Medicine and Pediatrics

University of California, DavisJune 23, 2010

Available from AHRQ

Authors:

Patrick S. Romano, MD MPHDominique Ritley, MPHDavid Chin, PhD student

With the help of many CVE representatives

Decision Guide Checklist

Goals: To provide collaboratives with frameworks and tools for

selecting measures of quality and resource use (QI 101) To highlight key considerations in selecting measures of

quality and resource use, based on a collaborative’s evolutionary stage

Guide divides answers to 26 questions into five sections: Introduction to Data Introduction to Measures of Quality Introduction to Resource Use/Efficiency Measures Selecting Quality and Resource Use Measures Interpreting Quality and Resource Use Measures

We focus today on just 6 of those questions3

Design of public reporting programs starts with a candid self-assessment (Q25)

Resources•Financial resources •Analytic capabilities

Goals•P4P•Public reporting•Performance improvement

Environment•Available data•Potential partners•Stakeholder engagement

Quality measurement and reporting are important issues across many industries

In search of a balanced set of quality measures (Q20): Iowa’s inspection of Wright County Egg

In search of a balanced set of quality measures (Q20):USDA “grader” inspection of shell egg plant

What did the USDA miss?

What else did the USDA miss?

IOM Domains of QualityEffectiveness Providing services based on scientific knowledge (avoiding overuse

of inappropriate care, underuse of appropriate care)

Patient Centeredness Care that is respectful of and responsive to patient preferences,

needs, and values

Timeliness Reducing wait times and sometimes harmful delays

Safety Avoiding injuries to patients from care that is intended to help

Efficiency Avoiding waste of equipment, supplies, ideas, and energy

Equity Care does not vary in quality because of personal characteristics

In search of a balanced set of quality measures (Q20):Institute of Medicine, 2010

What types of measures should be collected and reported (HOW)?

Look inside the structure…

Look for the outcomes…

Classifying types of measuresDonabedian, 2003

Structure: conditions under which care is provided Material resources (facilities, equipment) Human resources (ratios, qualifications, experience) Organizational characteristics (size, volume, IT system)

Process: activities that constitute health care (adherence to guidelines) Screening and diagnosis Treatment and rehabilitation Education and prevention

Outcome: changes attributable to health care Mortality, morbidity (complications, readmissions) Knowledge, attitudes, and behaviors Patient experiences/satisfaction

Framework for selecting measures (Q20)IOM

DomainsStructure Process Outcome

Effective Cardiac nurse staffing, nursing skill mix (RN/total)

Use of ACE inhibitor or ARB for patients with systolic HF

30-day readmissions (or mortality) for heart failure

Patient Centered

Use of survey data to track patient-centered care

How often did you get an appointment as soon as you thought you needed?

Overall rating of care

Timely Physician organization policy on scheduling urgent appointments

Received beta blocker at discharge and for 6 months after AMI

Potentially avoidable hospitalizations for angina (without procedure)

Safe Computerized physician order entry with medication error detection

Use of prophylaxis for venous thromboembolism in appropriate patients

Postoperative deep vein thrombosis or pulmonary embolism

Efficient Availability of rapid antigen testing for sore throat

Inappropriate use of antibiotics for sore throat

Dollars per episode of sore throat

Equitable Availability of adequate interpreting services

Use of interpreting services when appropriate

Disparity in any other outcome according to primary language

What data sources should we use for quality measurement (Q8/Q9)?

Oakley, E. et al. Pediatrics 2006;117:658-664

Video recording to identify errors in pediatric trauma resuscitation:

Mean of 5.9 errors per resuscitation, 93% agreement between reviewers.

Mean of 2.2 errors in each seriously injured child, with 20% capture on medical records.

Gold standard = Direct observation

Data sources for quality measurement (Q8/Q9): Review documents and collect specimens

Hillandale Farms FDA inspection results

Data sources for quality measurement (Q8/Q9):Review documents and collect specimens

Observe/record encounters (real or simulated) - $$$$ Ask patients (CAHPS surveys)

Satisfaction and experiences Morbidity (complications, functional status, quality-of-life) Knowledge, attitudes, and behaviors

Ask health care providers Rate others’ reputation (US News) Describe material and human resources (Leapfrog survey) Describe safety-related practices (Leapfrog survey)

Review claims/administrative data sets Mortality, morbidity (deaths, complications, readmissions) Adherence to guidelines (HEDIS, PQRI)

Review/abstract medical records (including registries) Mortality, morbidity (deaths, complications, readmissions) Adherence to guidelines (HEDIS, PQRI)

Tremendous growth in NQF-endorsed physician measures (Q8)

National Voluntary Consensus Standards: Ambulatory Care 101 measures across 10 priority areas: asthma and respiratory

illness; bone and joint conditions; diabetes; heart disease; hypertension; medication management; mental health; obesity; prenatal care; and prevention (including screening)

7 instruments for patient experience 26 measures of specialty clinician care: bone and joint conditions,

eye care, geriatrics, emergency care, skin care (melanoma) “Additional Performance Measures 2008”

67 measures for cancer care, infectious disease, perioperative care, and licensed independent practitioners

“Using Clinically Enriched Administrative Data” 70 measures across most original priority areas plus child health,

chronic kidney disease, gastroesophageal reflux, gynecology, hepatitis, HIV/AIDS, and migraine

Hospital quality measures are now available “off the shelf” (Q9)

CMS Medicare’s HospitalCompare The Joint Commission’s QualityCheck Commonwealth’s “Why not the best?” Leapfrog’s voluntary survey on CPOE, ICU

staffing, evidence-based hospital referral, and NQF Safe Practices implementation

Specialized state/regional programs (HAIs, AHRQ Quality Indicators, myhealthfinder.com, registries)

Other (HealthGrades, USNews, etc.)

Hybrid data (Q2) Bring together administrative (electronic claims) and medical

record data to build on the strengths of each while compensating for weaknesses: Increase the number of data elements for outcome

ascertainment or risk adjustment. Reduce the number of records that must be reviewed manually. Reduce the time required to review each record.

At the physician level, use claims to identify patients with a relevant diagnosis or problem, and use medical records to identify specific clinical findings or treatments.

At the hospital level, combine ICD-9-CM coded administrative data with laboratory or other clinical data to enhance the performance of risk-adjustment models and to reduce bias in estimates of hospital performance. FL, MN, VA pilot projects Regulatory requirements in CA and PA (Michael Pine’s work)

How to select measures for reporting?National Quality Forum criteria (Q22)

Importance: leverage point for improving quality variation in quality of care or suboptimal performance (overall)

Scientific acceptability: well-defined and precisely specified… reliable valid, accurately representing the concept being evaluated discriminating between real differences in provider performance adaptable to patient preferences and variety of settings adequate and specified risk-adjustment strategy

Usability: can be used for decision making and implementing change differences should be meaningful practically and clinically

Feasibility: benefit should be evaluated against burden confidentiality concerns should be addressed audit strategy should be available

Who needs composite scores? (Q10)

Trends in scores over time in LA County

“A” grade was associated with 5.7% increase in revenue.“B” grade was associated with 0.7% increase in revenue“C” grade was associated with 1% decrease in revenue.Two studies showed 20% and 13% decreases in hospitalization for food-borne illness in Los Angeles County.

Source: CHOICES 2005; 20(2):97-102 (American Agricultural Economics Association)

Why composite measures for CVEs? (Q10) (aka summary measures, roll-up measures)

AHRQ: “condensing multiple quality measures into a single piece of information”: Reduces cognitive burden for consumers, providing clearer

“signal” and reducing the danger of “cognitive shortcuts” Enhances reliability or ability to discriminate between

higher-quality and lower-quality providers Fits well conceptually with pay-for-performance programs,

which explicitly translate multiple quality measures to dollars, allowing providers to prioritize their own efforts

But remember two potential concerns: Difficulty achieving consensus on composite construction

and scoring, perhaps due to lack of professional consensus. Loss of important information if the composite combines

unrelated metrics in a manner that washes out meaningful differences on individual indicators.

Two conceptual approaches (Q10)

Psychometric or reflective perspective - an underlying, unmeasured factor (“quality”) is the cause of what we observe; the observed data reflect this unmeasured factor Requires a correlation among the measures included in the

composite, because different measures can only reflect the same latent factor (i.e., quality) if they are correlated with each other.

Clinometric or formative perspective – focus on guiding decision-making to optimize welfare instead of measuring an unobserved, latent factor Use clinical judgment rather than empirical analysis to

select component measures Formed from or defined by specific indicators, so no

correlation among component measures is required

Recommended approach for creating a composite score (Q10)

Identify the purpose… and delineate the quality construct…

Select the individual measures and/or sub-composite measures to be combined… (may require standardization)

Ensure that the weighting and scoring of the components supports the goal that is articulated for the measure.

Combine the component scores, using a specified scoring method…

Testing for reliability and validity

Restaurateurs’ reaction to the NYC composite score Purpose of composite is invalid:

“There is no evidence that letter grading increases the identification of risk factors for foodborne illnesses”

“…sophomoric, and punitive and demeaning to restaurateurs, as if they are schoolchildren who must be graded“

Composite is poorly constructed: “How can you possibly justify including non-food safety related items? A leaky

faucet, a (missing) sign, a light bulb not covered, an uncovered waste receptacle… mislead the public when it sees a B or C in the window into thinking that the food here is not safe, when the difference between an A or B grade may have nothing to do with food safety.”

Composite is unreliable: “…inconsistency from one inspector to another"

Composite is invalid due to umeasured risk: '‘Most of their buildings (in LA) are not 200 years old, and most of them are not

next to empty lots with hundreds of rats. It would be nice if the city would clean up those lots.''

Composite will have unintended consequences (gaming): “…a scarlet letter that will keep people from eating out” “…encourage bribery and corruption… I remember when payoffs were so

commonplace that the FBI had to come in and arrest the inspectors.” “…could turn back the clock on New York as the food capital of the world.”

Source: New York Times, multiple articles, February-August 2010

New York City’s “dirtiest establishment” (now closed): Was it rats or roaches?

Scoring Method Definition Example Adopter

All-or-none The percentage of patients for whom all indicators triggered by that patient are met.

“Appropriate Care Measure” for 4 conditions (heart attack, heart failure, pneumonia, and SCIP).

PHCQA Progress and Performance Report of Hospital Quality

70% Standard All-or-none with less strict criteria (e.g., 70% not 100%).

None to our knowledge

Overall Percentage (Opportunity weighting)

Percentage of all care events that were properly delivered, where each opportunity to “do the right thing” counts equally.

149 hypertensive patients triggered 26 hypertension indicators 828 times. Required care was given 576 times yielding 69.9% (576/828).

CMS P4P Premier Hospital Quality Incentive Demonstration

Indicator Average (Equal indicator/ event weighting)

Scores are averaged across all indicators to represent the mean adherence rate.

Hospital quality of care for acute myocardial infarction, congestive heart failure and pneumonia.

Hospital Quality Alliance (HQA)

Patient Average (Equal patient weighting)

The percentage of indicators successfully met is computed for each patient, and then averaged at the patient level.

None to our knowledge

Expert Opinion (Evidence-based)

Indicators are weighted based on evidence of impact on population health and/or effort required to achieve.

General Medical Services contract pays physicians more for achieving performance targets that require more time and other resources.

UK National Health Service

Scoring composite measures (Q10)

Combining quality and resource use measures to highlight high-value care (Q26)

Conclusion: Use available tools from AHRQ!

CVE Learning Networkhttp://www.cvelearningnetwork.org/default.asp

AHRQ Talking Qualityhttps://www.talkingquality.ahrq.gov/default.aspx

AHRQ Health Care Report Card Compendiumhttps://www.talkingquality.ahrq.gov/content/reportcard/search.aspx

AHRQ’s National Quality Measures Clearinghousehttp://www.qualitymeasures.ahrq.gov/

My Own Network, powered by AHRQhttp://www.monahrq.ahrq.gov/

RWJF’s Aligning Forces for Qualityhttp://www.forces4quality.org/welcome

AHRQ Decision Guide on Selecting Quality and Resource Use Measures

Addresses 26 questions community leaders and stakeholders frequently ask about quality and resource use measurement

Community quality collaborative leaders informed development

Access on-line at: http://www.ahrq.gov/qual/perfmeasguide or to order hard copies free of charge: send an email to

AHRQPubs@ahrq.hhs.gov specify number of copies include AHRQ Pub. No. 09(10)-0073