Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Supercharge Your Improvement Efforts with Predictive Analytics
Chris DeRienzo, MD, MPP, FAAPChief Quality Officer &
Neonatologist, Mission Health
Andrew O. Johnson, PhD
Manager, Data Science Clinical & Business
Analytics, Mission Health
Session #29
Learning Objectives• Explain the value proposition for building a robust internal data science team into an
integrated continuous improvement analytics strategy.
• Identify the required elements of a “Data Science Starter Kit” that can upgrade your organization’s analytics capabilities from reporting to predicting.
• Illustrate the role data scientists can play within a health system through the development of an all-cause, 30-day inpatient readmission model.
We are dead in the water without a culture of continuous improvement, grounded in analytics, that permeates everything we do, and all that we are.
The U.S. health system
ranks last among 11
industrialized countries on
measures of access, equity, quality, efficiency,
and healthy lives.
Wasteful spending in the health
system has been calculated at up to $1.2 trillion.
70% of hospital strategic initiatives
fail.
Only 32% of healthcare IT projects
meet their objectives,
while 24% totally fail, and 44% have difficulties
in meeting their goals.
About Mission Health
• Mission Health is westernNorth Carolina’s onlynot-for-profit, independent community healthcare system.
• Mission’s BIG(GER) Aim is to get every person to their desired outcome, first without harm, also without waste, and always with an exceptional experience for each person, family, and team member.
• Employing over 13,000 dedicated professionals, the system is comprised of seven hospitals including tertiary, critical access, and inpatient rehabilitation, 750 employed/aligned providers, and one of the largest Medicare Shared Savings ACOs in the nation.
Western North Carolina 18-County AreaPopulation (2016): 882,581Percent over 65: 22%
The Need at Mission HealthMajor process improvement was facilitated through the centralized performance improvement team—a highly skilled team of process engineering professionals.• Created internal dependency on the centralized team for complex projects.• Access to data only through IT and Informatics.• Difficult to provide real-time operational data or management roll-up data.
The Need:1. Provide health systemwide data in a way that allows broad access for analysis as well
as operational uses ranging from daily unit management to board-level discussion.
2. Create a culture of continuous improvement by outfitting key clinical and operational roles with skills, tools, and a sense of ownership for improving their own processes.
3. Create the capability to leverage both operational and clinical predictive analytics.
Where does your organization spend most of its time right now?
1) A2) B3) C4) D5) E6) F7) G8) Unsure or
not applicable
Poll Question #1
Standard Reports
Ad-hoc Reports
Query Drill-down
Alerts & Triggers
Statistical Analysis
Forecasting
Predictive What-if Analysis
Optimization
What happened?
How many? How often? Where?
What action is needed?
What exactly is the problem?
What is the best course of action?
How can I influence the future?
What are likely future outcomes?
Why did this occur?
A
B
C
D
E
F
G
Analytics
The Turning Point
Process transformation after“low hanging fruit.”• Early wins always lead
to more difficult and complex follow-up projects.
• Requires higher level of leadership, particularly from physicians.
Clinical program development.• Simply not possible
without strong physician leadership.
• Requires sensitive project facilitation.
Better together.• Clinical and
operational leaders and data scientists can accomplish more together than they can separately.
Results Successfully kick-started the journey “up the curve” in clinical analytics.
• Significant reductions in sepsis and stroke mortality, length of stay for bowel surgery and renal patients, and population screening for breast and colorectal cancer through workflows built into 50+ care process models.
Moved further up the curve toward predictive operational analytics.
• Large scale unit and flow simulations for building projects, regional transport optimization, and census prediction.
Launched integrated team to drive to meaningful clinical predictive analytics (e.g., readmissions predictor).
How We Did It
Began with the BIG(GER) Aim and asked “what do we need to best deliver on this promise in a population health world?”
Created a vision and support for data-driven continuous improvement grounded in analytics across Mission Health.
Built our analytics team with in-house data science expertise.
Cultivated a strong organizational motivation, capability, and process to move to predictive analytics operationally and clinically (initially focused on readmissions).
Moving From Reporting to Prediction
Things you must have to be successful: • Senior leaders with vision, budgetary impact,
and desire for change.• A trusted data infrastructure.• Well-defined problems with predictive
solutions.• Skilled and empowered data scientists.• The rest of the analytics team.• Processes to develop and sustain data
science projects.
Value Proposition for an In-house Data Science Team
Decide which analytic avenues to investigate.
1
2
3
4
5
6
Multiply returns on prior investments in data assets unique to your organization.
Rely on trusted in-house statistical advisors aligned with your goals.
Enjoy having allies when evaluating analytics
vendors’ products.
Tune prediction models to the “home-field advantage.”
Benefit from intense scrutiny over the validity
of data in your enterprise data warehouse (EDW).
Forecast ahead 18 months…Where do you want your organization spending most of its time?
1) A2) B3) C4) D5) E6) F7) G8) Unsure or
not applicable
Poll Question #2
Standard Reports
Ad-hoc Reports
Query Drill-down
Alerts & Triggers
Statistical Analysis
Forecasting
Predictive What-if Analysis
Optimization
What happened?
How many? How often? Where?
What action is needed?
What exactly is the problem?
What is the best course of action?
How can I influence the future?
What are likely future outcomes?
Why did this occur?
A
B
C
D
E
F
G
Analytics
Required: A Trusted Data Infrastructure
“76% of data scientists view data preparation as the least enjoyable part of their work.”
If you don’t have a reliable data infrastructure or EDW,you are wasting your time trying to operationalize data science.
Source: Press, G. (2016). Cleaning big data: Most time-consuming, least enjoyable data science task, survey says. Forbes Magazine. Retrieved from https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says
Required: Well-Defined Problems with Predictive Solutions
There are still so many beautiful things to be said in C-Major.
~ Sergey Prokofiev
There is still much good music that can be written in C-Major.
~ Arnold Schoenberg
““
Start with the essential quantities that the organization cares about.
Use existing data mart or reporting tables when available.
Trust your data scientists to know when to abandon
(or refuse to intake) a project.
Proceed incrementally.
Try to beat the performance of an existing model, or show
up sooner than it can.
Scopes creep when people get excited!
Twelve Key Questions to Intake a Predictive Project
12
3
4
5
67
8
9
10
11
12
1. What is the business problem?
2. What needs to be predicted?
3. How much improvement is needed?
4. Who is the targeted population?
5. How early in the process do you need it?
6. How often do you need updated predictions?
7. What time of day you need the predictions?
8. Have there been other projects that focused on this?
9. What will you do with the predictions?
10. Are there predictions being made now?
11. Are there any models to benchmark against?
12. Are there any minimum required levels of performance?
Required: Skilled Data Scientists…Where Is Everyone?
“Of the approximately 6,000 data scientists in the U.S., only 180 are estimated to work in the hospital and health care field. Given that there are nearly 6,000 hospitals and just 400 academic medical centers in the U.S.,
that’s stretching the available labor force a bit thin.”
Source: Huesch, M. D., & Mosher, T. J. (2017). Using it or losing it? The case for data scientists inside health care. NEJM Catalyst. Retrieved from http://catalyst.nejm.org/case-data-scientists-inside-health-care/
“Specialization is for insects.”– Robert A. Heinlein
“In terms of species diversity, total biomass, range of habitat, adaptations to adverse conditions, and intra-species pro-social behavior, insects are the most successful animals on Earth.”
– Every biology textbook ever
The Ability to Learn and Adapt Is a Key Requirement
On the Proper Care of Data Scientists
Top 5 rules for my data science team:
1Always ask questions. 2Try to be
comfortable with failure and uncertainty. 3Do not use
methods you do not understand. 4Before going
down a technical rabbit hole, ask
“What’s the value to Mission?” 5Always be nice
to the data architects.
Require them to continue learning and to set aside work time to do this.
A Tale of Two Job Offers
… The other issue is that I was uneasy about working with the physicians I’d met, as I felt their project vision and expectations exceeded their appreciation of the organizational, technical, and personnel requirements involved. When I asked them what they wanted from the person in the data scientist position, I heard: a statistical analyst, a project manager, a health economist, a strategy manager, and small amount of EMR build consulting. I can’t do all of that myself, …
Andy,…to be honest, we do expect all of those things from our data scientists. …
A Tale of Two Job Offers
… Can you give me any info on how I would be evaluated, and what sort of expectations you have for this position?
Hi Andy,Good to hear from you. Here are answers to your questions:
… The expectation would simply be to add value to our highly collaborative analytics program.
Required: “The Rest of the Team”
• Identify data sources.• Collect new data if
needed.• Merge, join, and
augment as appropriate.
• Understand structure of the data set - what does the data mean?
• Identify errors, outliers, case-deficient groups, etc.
• Understand statistical properties of the data set.
• Methodology assessment.
• Train model.• Test model.
• Literature review.• Identify competing
models.
• Automate.• Integrate into
application.• Validate.• Train.
DescriptiveAnalytics
PredictiveAnalytics
DataDiscoveryResearch ImplementationProblem
Definition
• Who is the customer?• What problem are we
trying to solve?• Is this the right
problem?• What is the value
added?
Knowledge Engineers
DataArchitects
BIDevelopers
QA andTraining
Integration with Analytics TeamReadmission Predictor v1Data Scientist:• Prototype input data frames.• Create prediction model.• Validate model performance.• Create model task/timing/output structure.
Knowledge Engineer:• Evaluate initial project proposal.• Set scope, deliverables, and feasibility.• Steer stakeholder validation of existing data
elements (LACE/Readmission Explorer).• Monitor progress to milestones.
Data Architect:• Productionize subject area mart (SAM) models
for input data.• Acquire additional fields from other sources.• Validate DS-created input data.• Build or modify application using model output.
BI Developer:• Develop best practices for in-app visualization.• Build or modify application using model output.
Training and QA Analyst: • QA check involved apps.• Maintain release schedule.• Communicate new releases.• Train end users.
Readmission Prediction v1 at MissionExisting tool: LACE
Problem: LACE is good, but not great, and we have unique patient populations and unusual data assets for building our own model.
Goal: To construct and automate the calculation of a risk model for 30-day, all-cause inpatient readmission.
Requirements:• Performance must beat LACE in our patient population.• Be available before 8 a.m. day after discharge.• First version must use fields currently existing in EDW.
A Relatively Simple ImplementationEDW
source mart
External source data
Internalsource data
EDW SAM
Predictive model input SAM
Predictive model output
SAM
R on DS server
R model
Staging server
R output to forecast model
database
Mission Analytics Portal
Visualizationapplication
Appreciate the importance of a skilled data science team.
Key Takeaways
Create the capability to
leverage both operational and
clinical predictive analytics
Create a culture of continuous improvement by outfitting key clinical and operational roles
with skills, tools, and a sense of ownership for
improving their own processes.
Provide systemwide data in a way that allows
broad access for analysis as well as
operational uses ranging from daily unit
management to board-level discussion.