Physician Prognostic Accuracy for In-Hospital Mortality in Percutaneous Coronary Intervention

© 2003 By Default!

A Free sample background from www.powerpointbackgrounds.com

Slide 1

Physician Prognostic Physician Prognostic Accuracy for In-Hospital Accuracy for In-Hospital

Mortality in Percutaneous Mortality in Percutaneous Coronary InterventionCoronary Intervention

Michael E. Matheny, MDMichael E. Matheny, MD

Medical Informatics FellowMedical Informatics FellowDecision Systems GroupDecision Systems Group

Brigham & Women’s HospitalBrigham & Women’s HospitalBoston, MABoston, MA

© 2003 By Default!


Slide 2

Specific AimsSpecific Aims

Primary HypothesisPrimary Hypothesis

– Accuracy of subjective physician estimations of Accuracy of subjective physician estimations of in-hospital mortality will be similar or improved in-hospital mortality will be similar or improved when compared with accepted objective risk when compared with accepted objective risk assessment methods for percutaneous coronary assessment methods for percutaneous coronary intervention (PCI)intervention (PCI)

© 2003 By Default!


Slide 3

Specific AimsSpecific Aims

Secondary HypothesesSecondary Hypotheses

– Accuracy of subjective physician estimations of in-Accuracy of subjective physician estimations of in-hospital major adverse cardiac events (MACE) will be hospital major adverse cardiac events (MACE) will be similar or improved when compared with accepted similar or improved when compared with accepted objective risk assessment methods for PCIobjective risk assessment methods for PCI

– Qualitative collection of risk factors could identify Qualitative collection of risk factors could identify additional important risk factors currently not included in additional important risk factors currently not included in the objective risk modelsthe objective risk models

– Incorporating subjective physician estimates into an Incorporating subjective physician estimates into an objective risk model will outperform either separatelyobjective risk model will outperform either separately

© 2003 By Default!


Slide 4

BackgroundBackground

DiscriminationDiscrimination– Ability to predict an outcome on a population levelAbility to predict an outcome on a population level– Area under the Receiver Operating Characteristic Curve Area under the Receiver Operating Characteristic Curve

(AUC)(AUC)

© 2003 By Default!


Slide 5


CalibrationCalibration– Ability to predict an outcome on a case/small group levelAbility to predict an outcome on a case/small group level– Hosmer-Lemeshow Goodness-of-Fit Test (HL-GF)Hosmer-Lemeshow Goodness-of-Fit Test (HL-GF)– Brier ScoreBrier Score

© 2003 By Default!


Slide 6


Subjective vs APACHE II Medical ICU Subjective vs APACHE II Medical ICU Mortality Mortality 11

– Discrimination: Discrimination: Objective BetterObjective Better– Calibration: Calibration: Subjective BetterSubjective Better– Forecasting Improves with TrainingForecasting Improves with Training

Subjective vs APACHE II Medical ICU Subjective vs APACHE II Medical ICU Mortality Mortality 22

– Discrimination: Discrimination: Subjective BetterSubjective Better– Calibration: Calibration: No DifferenceNo Difference

© 2003 By Default!


Slide 7


Subjective vs LR Acute CHF 90 day and 1 Subjective vs LR Acute CHF 90 day and 1 year Mortality year Mortality 3 43 4

– Discrimination: Discrimination: No DifferenceNo Difference– Calibration:Calibration: No DifferenceNo Difference– All estimations poorAll estimations poor

Subjective vs SNAP Neonatal ICU Mortality Subjective vs SNAP Neonatal ICU Mortality 55

– Discrimination:Discrimination: No DifferenceNo Difference– Calibration:Calibration: No DifferenceNo Difference

© 2003 By Default!


Slide 8


Subjective + PRISM III Pediatric ICU Subjective + PRISM III Pediatric ICU Mortality Mortality 66

– Discrimination: Discrimination: No DifferenceNo Difference– Calibration:Calibration: No DifferenceNo Difference– Combined modelCombined model

• Discrimination:Discrimination: Improved from eitherImproved from either• Calibration:Calibration: Improved from eitherImproved from either

© 2003 By Default!


Slide 9


Subjective vs LR Model of Post-Op mortality Subjective vs LR Model of Post-Op mortality for Open Heart Surgeries for Open Heart Surgeries 77

– Discrimination:Discrimination: No DifferenceNo Difference– Calibration:Calibration: No DifferenceNo Difference– Combined modelCombined model

• Discrimination:Discrimination: No DifferenceNo Difference• Calibration:Calibration: No DifferenceNo Difference

– Subjective assessments were more calibrated at Subjective assessments were more calibrated at the extremes of probabilitythe extremes of probability

© 2003 By Default!


Slide 10


Subjective Physician AssessmentsSubjective Physician Assessments

– Multiple Forms of Bias Multiple Forms of Bias 88

• Ego BiasEgo Bias• RegretRegret• Ignoring Negative EvidenceIgnoring Negative Evidence• FramingFraming

© 2003 By Default!


Slide 11


No work has been done evaluating No work has been done evaluating subjective physician estimates for in-hospital subjective physician estimates for in-hospital mortality in percutaneous coronary mortality in percutaneous coronary interventions.interventions.

© 2003 By Default!


Slide 12

Logistic Regression ModelsLogistic Regression Models• NationalNational

– American College of Cardiology American College of Cardiology 99

– 50123 pts 1998 - 200050123 pts 1998 - 2000• RegionalRegional

– Northern New England Northern New England 1010

– 15331 pts 1994 - 199615331 pts 1994 - 1996• LocalLocal

– Brigham & Women’s Hospital Brigham & Women’s Hospital 1111

– 2804 pts 1997 - 19992804 pts 1997 - 1999

BackgroundBackground PCI Objective Risk Model Gold StandardsPCI Objective Risk Model Gold Standards

© 2003 By Default!


Slide 13

Recent Evaluation of Models on Local Recent Evaluation of Models on Local Institution Data Institution Data 1212

– Discrimination (AUC)Discrimination (AUC)• ACC 0.90ACC 0.90• NNE 0.89NNE 0.89• BWH 0.89BWH 0.89

– Calibration (HL-GF)Calibration (HL-GF)• ACC <0.001ACC <0.001• NNE <0.001NNE <0.001• BWH <0.001BWH <0.001

BackgroundBackgroundPilot DataPilot Data

© 2003 By Default!


Slide 14


Objective Assessment ModelsObjective Assessment Models

– Multiple Forms of BiasMultiple Forms of Bias• Population/Demographic BiasPopulation/Demographic Bias

– Regional VariancesRegional Variances• Selection BiasSelection Bias

– Population referral biasPopulation referral bias• Temporal BiasTemporal Bias

– Medical Care StandardsMedical Care Standards– Data DocumentationData Documentation

• Data NoiseData Noise– Heterogeneous Data StandardsHeterogeneous Data Standards– Variation in Data Element CollectionVariation in Data Element Collection– Data Entry Quality VariationsData Entry Quality Variations

© 2003 By Default!


Slide 15


Incomplete model information?Incomplete model information?

Best Described Risk FactorsBest Described Risk Factors

AgeAge CHFCHFSexSex Unstable AnginaUnstable AnginaEjection FractionEjection Fraction Renal FailureRenal FailureRecent MIRecent MI Hx (CAD, DM, COPD, HTN)Hx (CAD, DM, COPD, HTN)Hemodynamic StabilityHemodynamic Stability Prior CABG or PCIPrior CABG or PCIIntra-Aortic Balloon PumpIntra-Aortic Balloon Pump

© 2003 By Default!


Slide 16

StudyStudyDesignDesign

Prospective Cohort StudyProspective Cohort Study

© 2003 By Default!


Slide 17

StudyStudyPopulationPopulation

LocationLocation– Brigham & Women’s Interventional Cardiology Brigham & Women’s Interventional Cardiology

SuitesSuites

Inclusion CriteriaInclusion Criteria– All Patients presenting for pre-operative All Patients presenting for pre-operative

evaluation for PCIevaluation for PCI

Exclusion CriteriaExclusion Criteria– Procedural Team declines to participate in Procedural Team declines to participate in

surveysurvey

© 2003 By Default!


Slide 18

StudyStudyData CollectionData Collection

Paper SurveyPaper Survey– Administration refused to allow survey to be part Administration refused to allow survey to be part

of medical recordof medical record

Subjective mortality assessment (0-100%) Subjective mortality assessment (0-100%) before and after procedurebefore and after procedure– AttendingsAttendings– FellowsFellows– Scrub NurseScrub Nurse

Qualitative additional risk factors from Qualitative additional risk factors from AttendingsAttendings

© 2003 By Default!


Slide 19

ExposuresExposures

Percutaneous Coronary Transluminal Percutaneous Coronary Transluminal Angiography with or without Coronary Angiography with or without Coronary StentingStenting

© 2003 By Default!


Slide 20

Covariates/ConfoundersCovariates/Confounders

AgeAge CHFCHFSexSex Unstable AnginaUnstable AnginaEjection FractionEjection Fraction Renal FailureRenal FailureRecent MIRecent MI Hx (CAD, DM, COPD, HTN)Hx (CAD, DM, COPD, HTN)Hemodynamic StabilityHemodynamic Stability Prior CABG or PCIPrior CABG or PCIIntra-Aortic Balloon PumpIntra-Aortic Balloon Pump

© 2003 By Default!


Slide 21

OutcomesOutcomes

In-Hospital DeathIn-Hospital Death In-Hospital MACEIn-Hospital MACE

© 2003 By Default!


Slide 22

Analysis PlanAnalysis Plan

Measure Discrimination & Calibration on local data Measure Discrimination & Calibration on local data for:for:– Objective MACE & Mortality ModelsObjective MACE & Mortality Models

• NationalNational• RegionalRegional• LocalLocal

– Subjective MACE & Mortality “Models”Subjective MACE & Mortality “Models”

Pair-wise Comparison of Objective and Subjective Pair-wise Comparison of Objective and Subjective models for statistical differencesmodels for statistical differences

Develop LR model with subjective data as a Develop LR model with subjective data as a covariate, and perform pair-wise comparisons with covariate, and perform pair-wise comparisons with objective and subjective models to determine if new objective and subjective models to determine if new model shows improvementmodel shows improvement

© 2003 By Default!


Slide 23

Analysis PlanAnalysis Plan

Local Institution DataLocal Institution Data

– ~1% Death Rate~1% Death Rate– ~5% MACE Rate~5% MACE Rate– ~200 Cases / month~200 Cases / month

© 2003 By Default!


Slide 24

Analysis PlanAnalysis PlanRough GuessRough Guess

Sample Size CalcSample Size Calc– Binomial Fisher’s ExactBinomial Fisher’s Exact– ΑΑ = 0.05 = 0.05– Power = 0.80Power = 0.80

– Effect Size & Estimated SampleEffect Size & Estimated Sample• MortalityMortality

– 1% to 1.5% = 81501% to 1.5% = 8150– 1% to 2% = 25141% to 2% = 2514

• MACEMACE– 5% to 7.5% = 15505% to 7.5% = 1550– 5% to 10% = 4735% to 10% = 473

© 2003 By Default!


Slide 25

Analysis PlanAnalysis PlanRecruitmentRecruitment

Multi-CenterMulti-Center– Exploring recruitment possibilities from Beth-Exploring recruitment possibilities from Beth-

Israel and Massachusetts General Cath LabsIsrael and Massachusetts General Cath Labs No Termination DateNo Termination Date

– Implemented as Quality Control method in BWH Implemented as Quality Control method in BWH Cath LabCath Lab

© 2003 By Default!


Slide 26

LimitationsLimitations

Sample SizeSample Size

Paper SurveyPaper Survey

Study Population ComplianceStudy Population Compliance

Selection BiasSelection Bias

© 2003 By Default!


Slide 27

Time TableTime Table

IRB ApprovalIRB Approval– CompletedCompleted

Physician Survey TemplatePhysician Survey Template– August 2005August 2005

IC Lab Tech Data Collection TrainingIC Lab Tech Data Collection Training– September 2005September 2005

Data CollectionData Collection– September 2005 - OpenSeptember 2005 - Open

© 2003 By Default!


Slide 28

AcknowledgementsAcknowledgements

Co-AuthorsCo-Authors– Nipun Arora, MD Nipun Arora, MD – Lucila Ohno-Machado, MD, PhDLucila Ohno-Machado, MD, PhD– Frederic S. Resnic, MD, MSFrederic S. Resnic, MD, MS

FundingFunding– NLM 1-T15-LM-07092NLM 1-T15-LM-07092

© 2003 By Default!


Slide 29

[email protected]@dsg.harvard.edu

Michael Matheny, MD Michael Matheny, MD Brigham & Women’s HospitalBrigham & Women’s Hospital

Thorn 309Thorn 30975 Francis Street75 Francis Street

Boston, MA 02115Boston, MA 02115

The EndThe End

Documents

Physician Prognostic Accuracy for In-Hospital Mortality in Percutaneous Coronary Intervention