Automatic Strategies for Decision Support in Telephone Triage

Framework and Testbed in Smalltalk

Carlos E. Ferroceferro@ciudad.com.arDirector: Dan Rozenfarb

Agenda

Introduction Software Application: ExpertCare Overview of the Framework:

Concept representation Session, automation and simulation Strategies

Some statistics Examples and results

Telephone Triage

Phone call from a patient

Initial data gathered Questions and

answers Presumptive

diagnosis Ambulance dispatch

or treatment indications

ExpertCare

Initial identification data on a standard form

ExpertCare

Selection of all initial symptoms reported by the caller

ExpertCare

New questions are suggested.

Answers are recorded.

Diagnoses are re-evaluated.

Question in plain Spanish

List of scored diagnosesList of symptoms

Session information

ExpertCare

As new information is gathered,

some diagnoses

are separated according

to their score

ExpertCare composition

Modules of ExpertCare system

Inference Engine

Interrogator

Ontology

Knowledge base:Symptoms, Syndromeswith attributes

Interrogatory Rules

Scorer

ExpertCare ontology

Symptoms (e.g. fever, headache, dyspnea)

Syndromes (e.g. appendicitis, osteomyelitis, asthma, schizofrenia)

Systems (e.g. circulatory, digestive) Severities (red, yellow, green) Frequencies (high, medium, low)

ExpertCare syndrome definition

Syndrome definitions are logical expressions in terms of symptoms.

Examples:Definition of Appendicitis :“right iliac fossa pain” AND “abdominal pain” AND NOT “appendix operation”Definition of Massive Obesity :“intense weight increase” OR “intense body fat increase”

ExpertCare size in numbers

Rules 3209Symptoms 2383Syndromes 673Other 157

Rules account for 50% of size, but 80% of complexity and 90% of costs.They also hinder software evolution.

Target

Our main metrics is the amount of questions:

Red (Emergency): 3 or 4 questions Yellow (Urgency): 4 or 5 questions Green: around 6, but may reach 12

Solution approach

Automated strategy Dynamic interrogatory Navigation and gathering of information

from the knowledge base Adaptation to session status

Framework for session and strategies Virtual lab as testbed

Concept representation

Logical expressions for definitions

Expression

value: aContextsatisfiersacceptVisitor: anExpressionVisitor

Constant

value: aContextsatisfiersacceptVisitor: anExpressionVisitorbe: aBoolean

falsetruevalue: aBoolean

Variable

value: aContextsatisfiersacceptVisitor: anExpressionVisitorname: aString

named: aString

Conjunction

value: aContextsatisfiersacceptVisitor: anExpressionVisitoroperator

Comparison

value: aContextsatisfiersacceptVisitor: anExpressionVisitor

Disjunction

value: aContextsatisfiersacceptVisitor: anExpressionVisitoroperator

BinaryExpression

value: aContextsatisfiersacceptVisitor: anExpressionVisitorleft: anExpressionright: anExpressionoperator

of: anExpression and: otherExpression

Negation

value: aContextsatisfiersacceptVisitor: anExpressionVisitorexpression: anExpressionoperator

of: anExpression

Session, Diagnoses and other

Automation AnswerProvider simulates a

patient/caller Strategy guides the interrogatory,

suggesting #nextQuestionFor: aCallSession

SUnit tests run through all syndromes using different strategies

StatisticalCollector gathers and caches information from the knowledge base

Grouping and statistics

Grouping and statisticsSeverity red - Syndromes by system

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Severity green - Syndromes by system

0 10 20 30 40 50 60

Severity yellow - Syndromes by system

0 5 10 15 20 25 30

Grouping and statistics

Step-by-step example This is a typical

red syndrome. According to the

definition, AnswerProvider can choose among 8 pairs of symptoms (2x4).

Each one is called a subsyndrome

Diabetic Ketoacidosis

System: MetabolicFrequency: LowSeverity: RedDefinition: (Diabetes OR

History of diabetes) AND (Unconsciousness OR Confusion OR Ketonic breath OR Dyspnea)

Step-by-step example

Choosing clues: Diabetes Systems pregnancy and metabolic Confusion Associated to 9 different systems Let’s choose Diabetes as a clue, and try to

establish the presence of Confusion, in order to make a Diabetic Ketoacidosis diagnosis.

Diabetic Ketoacidosis_2Definition: Diabetes AND Confusion

Example - Step 1

Choosing symptoms to ask:We should try to discern the system from among these two

Diabetes

772 syndromes51 systems

2 systems 18 syndromes

positive evidence

6 syndromes2 systems

Example - Step 1

Choosing symptoms to ask:Using information from the knowledge base and some abductive reasoning, we have 9 candidates left.We choose symptom pregnancy, in order to confirm or discard system pregnancy.

System pregnancy6 syndromes31 symptoms

System metabolic13 syndromes48 symptoms

System pregnancy1 syndrome1 symptom

Diabetes

System metabolic5 syndromes8 symptoms

Example - Step 2

Now we “know” that only one system has chances left.

DiabetesNot pregnancy

772 syndromes51 systems

1 system5 syndromes

positive evidence

Example - Step 2

Now we try to discern severity, first trying to decide whether it is red or yellow.Using information from the knowledge base and some abductive reasoning, we have 8 symptom candidates left.Here is where we need some tool for comparing or choosing among them. For instance, we could ask for symptom dyspnea.

System pregnancy6 syndromes31 symptoms

System metabolic13 syndromes48 symptoms

System pregnancy0 syndrome0 symptom

DiabetesNot pregnancy

System metabolic5 syndromes8 symptoms

1 syndrome

3 syndromes

9 syndromes

1 syndrome

2 syndromes

2 syndromes

Example - Step 3

The new information did not reduce syndromes

DiabetesNot pregnancyNot dyspnea

772 syndromes51 systems

1 system5 syndromes

positive evidence

Example - Step 3

We still try to discern severity, because “not dyspnea” only rejected some branches of some syndromes, but did not reduce the total number.Now we have 7 symptom candidates left. This way, we could use up to 7 more questions to “hit” the symptom that the simulated patient has and make a diagnosis.

System metabolic13 syndromes48 symptoms

DiabetesNot pregnancyNot dyspnea

System metabolic5 syndromes8 symptoms

1 syndrome

3 syndromes

9 syndromes

1 syndrome

2 syndromes

2 syndromes

Strategies One family of first attempts, using none or

little information:

SequentialStrategy, RandomStrategy, MoreSatisfiersStrategy, LessSatisfiersStrategy, MiddleSatisfiersStrategy, MoreCriticSeparationStrategy

Second family, attempting to guess the system by different indicators:

GuessSystemByFrequencyStrategy, MoreCorrelationStrategy, LessNegationStrategy, GuessSystemStrategy, GuessSystemUsingPairsStrategy, LessNegationPairStrategy

Results of preliminary strategies

StrategyRed

Average MedianDiag Error

%Sev Error

%

Sequential 739.04 745 19.00 5.58

Random 746.58 732.5 18.05 5.46

LessSatisfiers (*) 726.28 932 18.84 0.00

MiddleSatisfiers 579.28 794 18.05 2.44

MoreSatisfiers 85.29 52 14.23 2.51

MoreCriticSeparation 174.44 35 18.29 7.13

GuessSystemByFrequency 362.37 219 18.18 1.46

GuessSystemBySeverity 92.12 134 75.77 66.15

GuessSystemUsingPairs (**) 9.27 2 40.48 23.81

Strategies - Support

We coined the notion of support

Intuitively, it is a numeric representation of the degree of likelihood of a given set of syndromes in the current session.

Calculation is straigthforward. Syndromes with full diagnoses add a large positive

value. Syndromes with disproved diagnoses add a large

negative value. For the rest, symptoms confirmed add positive value

and symptoms negated add negative value. Finally we perform a normalization.

SupportSeparationStrategy

The third family of strategies is based on support.

Most promising results 15 different strategies Hierarchy 7 levels deep Every level evolving from the

previous one SUCCESS according to target

Results of support strategies

StrategyRed

Average Median Diag Error % Sev Error %

SupportSeparation 6.09 3 15.32 5.58

SupportSeparationWithImplication 4.62 2 18.29 7.13

SupportSeparationImplicationTracking 3.98 2 15.32 5.58

SupportSeparationImplicationTrackingClosing 2.95 1 16.86 7.13

SupportMainSyndrome 3.96 2 15.32 5.58

SupportMainSyndromeScoring 5.41 3 15.20 5.70

SupportOnlyPositive 3.34 2 15.20 5.11

SupportOnlyPositiveClosing 2.40 1 16.86 6.77

SupportOnlyPositiveCandidatesClosing 1.85 1 16.50 6.41

SupportOnlyPositiveClosingDifSev 2.12 1 20.55 6.29

SupportOnlyPositiveStrictClosing 2.13 1 16.86 6.77

SupportLessMissingScore 0.22 0 57.84 39.55

SupportMoreCoincidences 1.65 1 16.50 6.41

SupportMoreCoincidencesPassThru_1 1.65 1 16.50 6.41

SupportMoreCoincidencesPassThru_2 1.65 1 16.50 6.41

SupportMoreCoincidencesPassThru_3 1.96 1 15.32 5.23

SupportMoreCoincidencesPassThru_4 2.12 1 15.20 5.10

Results of support strategies

StrategyYellow

Average Median Diag Error % Sev Error %SupportSeparation 11.75 6 9.61 3.91SupportSeparationWithImplication 8.53 6 16.76 9.72SupportSeparationImplicationTracking 7.31 5 10.33 4.16SupportSeparationImplicationTrackingClosing 5.94 4 16.25 10.39SupportMainSyndrome 7.18 5 10.95 4.47SupportMainSyndromeScoring 7.29 6 10.80 4.88SupportOnlyPositive 5.11 4 10.69 4.27SupportOnlyPositiveClosing 4.24 3 12.08 5.91SupportOnlyPositiveCandidatesClosing 3.15 2 11.21 5.19SupportOnlyPositiveClosingDifSev 4.65 3 12.90 6.17SupportOnlyPositiveStrictClosing 3.61 2 11.52 5.40SupportLessMissingScore 0.48 0 55.89 35.63SupportMoreCoincidences 2.63 2 11.47 5.45SupportMoreCoincidencesPassThru_1 2.63 2 11.47 5.45SupportMoreCoincidencesPassThru_2 2.62 2 11.47 5.45SupportMoreCoincidencesPassThru_3 3.04 2 11.10 5.09SupportMoreCoincidencesPassThru_4 3.07 2 10.00 4.99

Results of support strategies

StrategyGreen

Average MedianDiag Error

% Sev Error %SupportSeparation 25.43 8 18.50 9.97SupportSeparationWithImplication 17.67 8 23.55 10.66SupportSeparationImplicationTracking 16.54 6 18.11 9.97SupportSeparationImplicationTrackingClosing 5.79 1 19.01 16.43SupportMainSyndrome 16.46 6 18.20 9.97SupportMainSyndromeScoring 11.92 6 19.65 10.00SupportOnlyPositive 13.44 5 18.08 10.00SupportOnlyPositiveClosing 5.18 1 18.34 10.48SupportOnlyPositiveCandidatesClosing 2.82 1 18.41 14.48SupportOnlyPositiveClosingDifSev 5.23 1 18.11 9.70SupportOnlyPositiveStrictClosing 3.39 1 18.14 13.82SupportLessMissingScore 0.43 0 38.97 13.91SupportMoreCoincidences 2.12 1 18.29 14.12SupportMoreCoincidencesPassThru_1 2.12 1 18.29 14.12SupportMoreCoincidencesPassThru_2 2.11 1 18.29 14.06SupportMoreCoincidencesPassThru_3 2.24 1 17.66 10.00SupportMoreCoincidencesPassThru_4 2.27 1 17.39 9.19

Conclusions and remarks

It was great doing this work because:

Enhancing the ExpertCare application could have a direct impact on the population’s health.

Automated strategies allow ExpertCare architecture to be used in other domains.

We applied a scientific research approach and techniques to this “real world” software problem.

We learned from Artificial Intelligence, Object-Oriented Programming and Medicine in an interdisciplinary work.

Conclusions and remarksSmalltalk proved to be an adequate tool because:

Representation of the knowledge base was almost trivial.

Building a virtual lab for essays and benchmarks was very easy.

Additional tools for exploring the knowledge base and studying it were easy to implement.

There were no barriers for implementing and testing several strategies with diverse heuristics.

It was easy to get feedback and to debug troublesome cases, in order to enhance and refine strategies

Future work (technical) A visual tool for representing the

session. It should be some navigational metaphor.

The tool could be enhanced for tracing during simulation runs.

More tools for developers to understand and interact with strategy/session.

More tools for better comparative benchmarking.

Future work (domain model)

Integrate with ExpertCare Incorporate exceptions and special

rules Test with real samples Try some adaptation to other

knowledge bases

The End

Questions?