Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
Bonnie:
Tips for Debugging Clinical Quality Language
Electronic Clinical Quality Measures
James Bradley
MITRE, Senior Software Engineer
March 2018
The MITRE Corporation operates the Centers for Medicare & Medicaid Services (CMS) Alliance to
Modernize Healthcare (CAMH), a federally funded research and development center (FFRDC)
dedicated to strengthening the nation’s health care system. MITRE operates CAMH in partnership
with CMS and the Department of Health and Human Services.
| 2 |
Agenda
▪ Assumptions
▪ Bonnie Overview
▪ Debugging CQL eCQMs in Bonnie
▪ Common Points of Confusion in CQL Logic
▪ Resources
▪ Other Related Presentations at HIMSS 2018
▪ Questions
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 3 |
Assumptions
▪ Audience is familiar with Clinical Quality Language (CQL) and electronic clinical quality measures (eCQM)
▪ CQL packages being debugged are produced and exported using the Measure Authoring Tool (MAT)
▪ Audience is using Bonnie to debug CQL eCQMs
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 4 |
Bonnie Overview
What is Bonnie?
▪ Bonnie is an electronic Clinical Quality Measure (eCQM) testing tool that allows users to:
▪ Load eCQMs from the Health Quality Measure Format (HQMF) exported from
the Measure Authoring Tool (MAT)
▪ Build synthetic patients using Quality Data Model (QDM) data elements
defined as part of the measure definition
▪ Test new and updated eCQMs using synthetic patients
▪ Explore the behavior and complexity characteristics of eCQMs
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 5 |
Bonnie Overview
Bonnie Screenshot
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 6 |
Bonnie Overview
Bonnie Screenshot (continued)
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 7 |
Debugging CQL eCQMs in Bonnie
Common Debugging Issues
▪ Overlapping Value Sets
▪ Out of Date Codes
▪ Alias Not Referencing Expected Value
▪ Timing Issues:
– “starts after”
– “day of”
▪ Points of Confusion:
– “exists”
– Multi-source queries
– Comparing Direct Reference Codes
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 8 |
Debugging CQL eCQMs in Bonnie
Problem: Overlapping Value Sets
▪ Problem: A “retrieve” is unexpectedly calculating as True.
▪ Debug Step: Check to see if the associated value set has overlaps that may be affecting calculations.
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 9 |
Debugging CQL eCQMs in Bonnie
Debugging: Overlapping Value Sets
▪ Hover over the “retrieve” and see which data element highlights.
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 10 |
Debugging CQL eCQMs in Bonnie
Debugging: Overlapping Value Sets (continued)
▪ Go to the Measure Details page, scroll to the bottom, and confirm there are overlapping codes.
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 11 |
Debugging CQL eCQMs in Bonnie
Problem: Out of Date Codes
▪ Problem: A “retrieve” is unexpectedly calculating as False.
▪ Debug Step: Check to see if there is a warning on the top of the Patient Builder page.
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 12 |
Debugging CQL eCQMs in Bonnie
Debugging: Out of Date Codes
▪ To address an outdated code, either:
– Select a new code from the code selection list and delete the old code.
– Drag a new data element onto the patient.
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 13 |
Debugging CQL eCQMs in Bonnie
Problem: Alias Not Referencing Expected Value
▪ Problem: An alias is referencing a value different than the one the measure developer expects. This would manifest in the calculation not behaving as expected.
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 14 |
Debugging CQL eCQMs in Bonnie
Debugging: Alias Not Referencing Expected Value
▪ Debug Step: Hover over the alias and see which element in the patient history highlights
▪ “Inpatient encounter” lit up, not “Emergency Department Visit”
▪ Review the logic of “Encounter with ED”
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 15 |
Debugging CQL eCQMs in Bonnie
Problem: Timing Issues
▪ Pinpoint where the issue is occurring using Bonnie’s logic highlighting.
▪ Double-check the documentation for expected behavior.
An example: “starts after”:
−The intent was to check if PCI started after the start of the Hospitalization.
− “starts after” checks if something starts after the end of a given interval.
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 16 |
Debugging CQL eCQMs in Bonnie
Problem: Timing Issues (continued)
▪ “day of”:
– “day of” causes the logic to pay attention to the date, rather than the time
– ‘ICM’ can start anywhere between 12am 1/15/12 and 11:59pm 1/16/12
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 17 |
Common Points of Confusion in CQL Logic
▪ “exists”
• Not intuitive for how it evaluates based on certain parameters
▪ “exists({})” -> false
▪ “exists({item})” -> true
▪ “exists(null)” -> false
▪ “exists({null}) “ -> true
• As long as the parameter is not an empty list or ‘null’, “exists” will evaluate to
‘true’.
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 18 |
Common Points of Confusion in CQL Logic (continued)
▪ Multi-source queries
– For example:
– If any of E, P, M, or D returns an empty set, this definition will not be evaluated.
– This is due to the Cartesian product of the sources. A Cartesian product with an empty set will always be the empty set.
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 19 |
Common Points of Confusion in CQL Logic (continued)
▪ Comparing Direct Reference Codes
▪ Use the Equivalent (“~”) operator instead of the Equal (“=”) operator
• For example:
performed.result ~ “Positive measurement finding (finding)”
▪ Equivalence (“~”) of codes is defined based on the code and system elements only
▪ “=” is too strict and will compare all elements of a direct reference code
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 20 |
Resources
▪ Main Bonnie site
– https://bonnie.healthit.gov/
▪ JIRA for tracking Bonnie-related issues
– https://oncprojectracking.healthit.gov/support/projects/BONNIE
▪ JIRA for tracking CQL-related issues
– https://oncprojectracking.healthit.gov/support/projects/CQLIT
▪ CQL Formatting and Usage Wiki
– https://github.com/esacinc/CQL-Formatting-and-Usage-Wiki/wiki/Questions-and-Answers
▪ Development team contact
▪ Open Source on GitHub
– https://github.com/projecttacoma/bonnie
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 21 |
Other Related Presentations at HIMSS 2018
March 7, 2018
10:00 AM – 10:30 PM PT
Cypress: New Features, Clinical
Quality Language Support, and
Burden Reduction
March 7, 2018
2:00 PM – 3:30 PM PT
eCQM Strategy: The Future of
eCQMs
March 8, 2018
10:30 AM – 12:00 PM PT
eCQM Strategy: The Future of
eCQMs
Sessions will be held in the MITRE Government Relations Room (Zeno 4608)
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 22 |
Follow up
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
To download a copy of this presentation, visit:
https://health.mitre.org/himss18
Data Interoperability to Reduce Clinician Burden
Follow us on social media: @MITREhealth
| 23 |
Questions?
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 24 |
Backup Slides
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 25 |
Common Points of Confusion in the Common CQL Library
▪ Hospitalization Function
– If an ED Encounter occurs within an hour of an Inpatient Encounter, “Hospitalization” returns the start of the ED Encounter and the end of the Inpatient Encounter.
– Events that occur within the gap between the ED Encounter and the Inpatient Encounter are within the Hospitalization period.
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.
| 26 |
Common Points of Confusion in the Common CQL Library (continued)
▪ CalendarAgeInYearsAt vs. CalendarAgeInYears
– ‘CalendarAgeInYearsAt(BirthDateTime DateTime, AsOf DateTime)’
▪ Returns the number of years passed between the first and second DateTime.
– ‘CalendarAgeInYears(BirthDateTime DateTime)’
▪ Returns the number of years passed between the passed in DateTime and Now.
▪ Function returns “CalendarAgeInYearsAt(BirthDateTime, Today())”
Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.