GLARE (GuideLine Acquisition Representation and Execution)

GLARE (GuideLine Acquisition Representation and

Execution)

Paolo TerenzianiDipartimento di Informatica,

Universita’ del Piemonte Orientale “Amedeo Avogadro”, Alessandria, Italy

- Introduction- Representation formalism- Kernel Architecture: acquisition and execution modules

- Decision making- Temporal reasoning

- General Architecture (advanced features of GLARE)

- Verification

Introduction

Many different computer systems managing clinical guidelines (e.g., Asgaard, GEM, Gliff, Guide, PROforma,…)

Different roles:- support- critique- evaluation- education- …...

Clinical guidelines are a means for specifying the “best” clinical procedures and for standardizing them

Adopting (computer-based) clinical guidelines is advantageous

GLARE(GuideLine Acquisition Representation and

Execution)

-Joint project with:Gianpaolo Molino, Mauro TorchioLaboratorio di Informatica Clinica, Azienda Ospedaliera S. Giovanni Battista, Molinette, Torino, Italy

Stefania Montani, Alessio Bottrighi, Luca Anselma, Gianluca Correndo

- Domain independent

(e.g., bladder cancer, reflux esophagitis, heart failure)

- User-friendly (limited number of primitives)

GLARE (Guideline Acquisition, Representation and Execution)

Representation Formalism

A BC

B1 B2

B1.1

B1.2 B2.1

B2.2

B2.3

CG

B

B1

B2

DTree of graphs

Representation Formalism

A BC

B1 B2

B1.1

B1.2 B2.1

B2.2

B2.3

CG

B

B1

B2

DTree of graphs

Atomic actions

Representation Formalism

A BC

B1 B2

B1.1

B1.2 B2.1

B2.2

B2.3

CG

B

B1

B2

DTree of graphs

Atomic actions

Composite actions (plans)

Representation Formalism

Tree of graphs

Atomic actions

Composite actions (plans)

Control relations between actions:- sequence

A BC

B1 B2

B1.1

B1.2 B2.1

B2.2

B2.3

CG

B

B1

B2

D

Representation Formalism

Tree of graphs

Atomic actions

Composite actions (plans)

Control relations between actions:- sequence- “controlled” (e.g., during)

A BC

B1 B2

B1.1

B1.2 B2.1

B2.2

B2.3

CG

B

B1

B2

D

Representation Formalism

Tree of graphs

Atomic actions

Composite actions (plans)

Control relations between actions:- sequence- “controlled”- alternative

A BC

B1 B2

B1.1

B1.2 B2.1

B2.2

B2.3

CG

B

B1

B2

D

Representation Formalism

Tree of graphs

Atomic actions

Composite actions (plans)

Control relations between actions:- sequence- “controlled”- alternative- repetition (e.g. “3 times each 2

days for a month”)

A BC

B1 B2

B1.1

B1.2 B2.1

B2.2

B2.3

CG

B

B1

B2

D

Representation Formalism

Temporal constraint associated with atomic actions and control relations (see below)

Representation FormalismHierarchy of Action Types

Action

Plan QueryWork action ConclusionDecision

Clinical action

Pharmacol.prescription

Diagnosticdecision

Therapeuticdecision

Representation Formalismdescription of a clinical action

basicdescription

namedescription (text)

contextualcosttimeresources

repetitions

frame time

frequency

action-time

execution timeI-time

exit_condition

logical

must includemay includemust excludemay excludeconflicts

preconditions

goals (text)

delay time

Therapeutic decisions

Fixed set of parameters (effectiveness, cost, side-effects, compliance, duration)

Treatment choice for symptomless gallbladder stones

Treatment choice

Surgical treatment

Expectant management

Litholitic therapy

Local information associated with treatment choice (in the symptomless gallbladder stones guideline)

Strategy Effectiveness Cost Duration Compliance Side effects

Expectant

management

- - - ++++ -

Surgery +++ ++ + - ++

Litholytic therapy + ++ +++ ++ +

Diagnostic Decisions

* Decision parameters<finding, attribute, value>

* Decision criteriascore-based mechanism

For each alternativeFor each parameter score

(additive) threshold range

Diagnostic Decisions(Gastro Esophageal Reflux Disease)

PARAMETERS: heartburn absent (“no-hb”), heartburn lasted not more than 3 months (“hb=<3”), heartburn r lasted more than 3 months (“hb>3m”); dysphagia absent (“no-dys”); dysphagia present (“dys”); occurrence of weight loss (“wl”) or non-occurrence (“no-wl”); hemathemesis absence (“no-hem”); hemathemesis presence (“hem”); postural reflux absent (“no-ref”), postural reflux lasted not more than 3 months (“ref=<3”); postural reflux lasted more than three months (“ref>3m”). THRESHOLD: >9. (One should conclude “no GERD” only if heartburn, dysphagia, weight loss, hematemesis and postural reflux are all absent.)

Acquisition

Strict interaction with DB’s

Clinical DB hierarchically organized vocabulary Standardization Data sharing Support for semantic checks (e.g., legal attribute values)

NOTE: the organization (schema) of Patients DB is equal to the one of the Clinical DB During acquisition, GLARE gets the information used at execution-time to retrieve automatically the patient’s data (via automatically generated dynamic-SQL queries)

Acquisition“Intelligent” helps (syntactic & semantic checks)

- “legal” names & “legal” values for attributes

- “logical” design criteria (no unstructured cycles, well-formed alternatives & decisions)

- “semantic” checks: consistency of temporal constraints

Digression 1We only allow “structured” guidelines

Each block (including cycles) has just one entry and one exit

a crucial step in order to enforce the semantic clearness of a representation formalism, and, thus, its understandability and practical usefulness (see the discussion about programming languages in the 60’)

Architecture of the system(Acquisition part)

Clinical DB

Pharmac. DB

Resource DB

ICD-9-CM DB

ExpertPhysician

AcquisitionInterface

Guidelines DB

Knowledge Manager

AcquisitionGraphical Interface

Clinical Guidelines Execution Module

“Instantiate” a general guideline on a specific patient

Independent of the guideline

Independent of the task (general purpose)

Providing support to decisions

Basic Requirements

Architecture of the system

Clinical DB

Pharmac. DB

Resource DB

ICD DB

ExpertPhysician

AcquisitionInterface

Guidelines DB

Knowledge Manager

UserPhysician

ExecutionInterface

Guidelines Instantiation DB

PatientDB

ExecutionModule

Agenda-based execution

In the agenda- next actions to be executed- execution time (earliest and latest e.t.)

“On-line” execution: wait until the next e.t.(support physician in clinical activity)

“Simulated” execution: jump to the next e.t.(education, critique, evaluation)

Executing atomic actions

Work actions:- evaluate pre-conditions- execute action within its range of time- delete from the agenda

Query actions:- retrieve data from patient’s DB- wait for data not already in the DB, or for the update of “expired” data

Conclusion actions:- insert conclusion into the patient’s DB

Decision actions: (see alternatives)

Executing composite actions

Sequence:- evaluate next action e.t. (given current time and delay)- execute it

Concurrent actions:- execute actions according with the temporal constraints

Decision + alternative actions (e.g., diagnostic decision):- evaluate parameter values for each alternative, using patient’s DB- determine the score for each alternative- compare the score with the threshold- show the alternatives to the user-physician (distinguishing between

“suggested” and “not suggested” ones, and showing parameters and scores)

- execute the alternative chosen by the physician (warning available)

Executing Clinical Guidelines: other issues

Exits

Failuresreturn to previous decisions (chronological vs. guided backtracking)

Repeated actions and user-defined periodicities- expressive language for user-defined periodicities [IEEE TKDE, 97]- computing next execution time

A user-friendly graphical interface

GLARE: Architecture of the system

GLARE KERNEL

Decision-MakingModule

Temporal ReasoningModule

UpdateModule

ContextualizationModule

Model-CheckingModule

SPIN

GLARE: Contextualization

GLARE KERNEL

Temporal ReasoningModule

ContextualizationModule

UpdateModule

Model-CheckingModule

Decision-MakingModule

SPIN

GLARE: Contextualization

Gap between GL generality and the peculiarity of specific contexts of execution is one of the major obstacles to GL dissemination & exploitation

In GLARE: software module to automatically adapting (general) GL considering locally available resources

* INPUT: a “general” GL + a list of locally available resources

* OUTPUT: a “locally-adapted” GL: only paths for which resources are locally available are retained

GLARE: Model Checking

GLARE KERNEL

Temporal ReasoningModule

ContextualizationModule

UpdateModule

Model-CheckingModule

Decision-MakingModule

SPIN

GLARE: Model Checking

After the acquisition of a GL, it is important to verify its properties (e.g., correctness, patient class eligibility, patient applicability)

GLARE is loosely coupled with the model-checker SPIN

General-purpose solution! Given any GLARE GL, any property that can be expressed in LTL can be checked!

GLARE: Update

GLARE KERNEL

Temporal ReasoningModule

ContextualizationModule

UpdateModule

Model-CheckingModule

Decision-MakingModule

SPIN

GLARE: Update

GL content has to be adapted to local contexts (e.g., country rules) and continuously revised to be up-to-date

To grant for quality and EBM, a team of responsible has to evaluate update proposals

In GLARE (ongoing work): bitemporal Database support for managing:-proposals of update- acceptance\rejection

GLARE: Temporal Reasoning

GLARE KERNEL

Temporal ReasoningModule

ContextualizationModule

UpdateModule

Model-CheckingModule

Decision-MakingModule

SPIN

GLARE: Temporal ReasoningTemporal constraints (e.g., delays between actions) play a fundamental role within GL. - A formalism to represent them is needed!

- Correct & complete algorithms to propagate the constraints are also needed!

GLARE’s Temporal Reasoner can:

- check the consistency of constraints (acquisition phase)

- check whether actions are scheduled\executed accordingly to the GL constraints, and schedule next actions (execution phase)

GLARE: Decision-making

GLARE KERNEL

Temporal ReasoningModule

ContextualizationModule

UpdateModule

Model-CheckingModule

Decision-MakingModule

SPIN

GLARE: Decision-making

Decision making is a core issue in the clinical practice. In particular, therapy selection is a critical issue.

Different parameters (cost, effectiveness, expected utility) must be taken into account

In GLARE: a semi-automatic decision support system based on

-simulation capabilities-Decision Theory

GLARE: current status

A Java prototype implements the kernel of the system

-The prototype interacts with the (current) patient records (stored in Cache) during execution

-All software developed by students: no “fully engineered” and “supported” sowtware available yet

Different extensions of the prototype to demonstrate the feasibility of contextualization, decision making, model-checking, and temporal reasoning facilities

Ready for the step towards a commercial product

Remarks

Computer-based approaches to GL can provide crucial adavntages for

-Organizations (quality, service optimization, standardization)

-Physicians (decision support, quality)

-Patients (quality & standardization of the treatment, avoiding errors)

A strict and long-term cooperation of Physicians and Computer Scientists has lead to the development of GLARE

GLARE embodies advanced Artificial Intelligence techniques, providing automatic support for contextualization, decision making, model-checking, update management, and temporal reasoning

GLARE: Decision-making

“local information”: considering just the decision criteria associated with the specific decision at hand

“global information”: information stemming from relevant alternative pathways in the guideline

“What if” facility

Facility for gathering the chosen parameter (e.g, resources, costs, times) from the “relevant” alternative paths on the guideline

It provides an idea of what could happen in the rest of the guideline if the physician selects a given alternative for the patient, and supports for comparisons of the alternatives

Syntomless gallbladder stonestreatment choice: “global information”

Treatment choice

Surgical treatment

Expectant management

Litholitic therapy

Litholitic treatment

Choice ofsurgical appr.

Laparoscopy

Laparotomy

Laparoscopy

Laparotomy

Syntomless gallbladder stonestreatment choice: “global information”

Treatment choice

Surgical treatment

Expectant management

Litholitic therapy

Litholitic treatment

Choice ofsurgical appr.

Laparoscopy

Laparotomy

Laparoscopy

Laparotomy

Duration min:2 days Max:3 days

Syntomless gallbladder stonestreatment choice: “global information”

Treatment choice

Surgical treatment

Expectant management

Litholitic therapy

Litholitic treatment

Choice ofsurgical appr.

Laparoscopy

Laparotomy

Laparoscopy

Laparotomy

Duration min:6 days Max:8 days

Syntomless gallbladder stonestreatment choice: “global information”

Treatment choice

Surgical treatment

Expectant management

Litholitic therapy

Litholitic treatment

Choice ofsurgical appr.

Laparoscopy

Laparotomy

Laparoscopy

Laparotomy

Duration min:1 day Max: all life long

Syntomless gallbladder stonestreatment choice: “global information”

Treatment choice

Surgical treatment

Expectant management

Litholitic therapy

Litholitic treatment

Choice ofsurgical appr.

Laparoscopy

Laparotomy

Laparoscopy

Laparotomy

Duration min:2 months Max:1 year

Local information associated with treatment choice (in the symptomless gallbladder stones guideline)

Strategy Effectiveness Cost Duration Compliance Side effects

Expectant

management

- - - ++++ -

Surgery +++ ++ + - ++

Litholytic therapy + ++ +++ ++ +

Digression 2Why don’t we put “global info (about paths)” locally in the decision

actions?

Given “local info” in each node, collecting & storing might be automatic

HOWEVER:

- exponential space in each node

- data duplication (consistency after updates?)

- not user friendly (too many data!)

- not all aternatives are “relevant”

- data not always necessary

>> global data only at execution time, on request

1

2

3

4

5

6

7

8

910

11

14

1312

15

1

1-2-4-81-2-4-91-2-5-101-2-5-11

1-3-6-121-3-6-131-3-7-141-3-7-15

2

2-4-82-4-9

2-5-102-5-11

Enhancing Decision Making through Decision Theory

• Supporting therapy selection is a critical objective• Selection parameters (cost, side-effects) have to be balanced by the

physician • Several decisions in sequence (dynamic decision problem)• IDEA: Decision theory: a natural candidate for decision support• PROBLEM: Mapping concepts between the two areas is not

straightforward

Justification of the approach

• Shared assumption:– An (implicit or explicit) query action is always found before a decision

action• Each decision is based on a data collection completed at decision

time• Does not depend on the previous history

– Markov assumption– Discrete time process– Completely observable

• Markov Decision Process (MDP) Decision Theory

Correspondence between GL and DT concepts

• State: parameters that describe the patient’s clinical condition– Query action: means for observing the state

• State transition: changes in the patient’s state variables– Through GL actions (or exogenous changes)– Observation through query actions – Diagnostic decisions are observable but not needed– The effect of a single work action is typically not observable* State transition produced by the set of actions between two therapeutic decisions

• Utility: life expectancy corrected by QALYs– Information provided by the medical literature or by phisician interviews

• Cost: not only monetary expenses– Resources– Time

Selecting and adapting DT algorithmsto cope with GL

Goals:

1. Generation of the Markov model of GLs

2.a Optimal policy evaluation

2.b Path utility evaluation

Generation of the Markov model

- States: therapeutic decisions

- Transitions: macro-actions representing all actions between two consecutive therapeutic decisions

- Probability of transitions: literature or interview

Selecting and adapting DT algorithmsto cope with GL

• FocusingPath selection to focus on paths of interest

• Concurrent actionsSelection of one possible order of execution

• Repeated actions different strategies, depending on whether- the number of repetitions in known a priori- the number of repetitions in not known a priori

Evaluation of the optimal policy

1 The number of repetitions is known

- finite time horizon approaches

dynamic programming

2 The number of repetitions is not known

- infinite time horizon approaches

value iteration

Evaluation of path utility

Fixing a path = applying a specific policy• dynamic programming without maximizing the

expected value of the cumulative utility function wrt the different actions

• value iteration without maximizing wrt the different actions

Conclusions

Decision theory can provide crucial advantages for supporting therapeutic decision making in GL systems

1 Knowledge representation level:

analysing the correspondences between DT and GL concepts

2 Algorithmic level:

selecting and adapting DT algorithms to fit the GL context

General methodology (GLARE as an example)

Conclusions (Decision Theory-based approach)

Decision theory can provide crucial advantages for supporting therapeutic decision making in GL systems

1 Knowledge representation level:

analysing the correspondences between DT and GL concepts

2 Algorithmic level:

selecting and adapting DT algorithms to fit the GL context

General methodology (GLARE as an example)

GLARE: Temporal Reasoning

GLARE KERNEL

Temporal ReasoningModule

ContextualizationModule

UpdateModule

Model-CheckingModule

Decision-MakingModule

SPIN

Temporal Constraints in Clinical Guidelines

Temporal constraints are an intrinsic part of clinical knowledge (e.g., ordering of the therapeutic actions)

Different kinds of temporal constraints, e.g.,

- duration of actions (min / max)

- qualitative constraints (e.g., before, during)

- delays (min / max)

- periodicity constraints on repeated actions

Temporal Constraints in Clinical Guidelinesrepetitions

TheThe therapy for multiple mieloma is made by six cycles of 5-day treatment, each one followed by a delay of 23 days (for a total time of 24 weeks). Within each cycle of 5 days, 2 inner cycles can be distinguished: the melphalan treatment, to be provided twice a day, for each of the 5 days, and the prednisone treatment, to be provided once a day, for each of the 5 days. These two treatments must be performed in parallel.

Managing Temporal Constraint: the Problem

Temporal Constraints without Temporal Reasoning (constraint propagation)- are useless- clash against users’ intuitions/expectations

Both representation and inference are NEEDED

Managing Temporal Constraints: the Problem

Implied constraint (temporal reasoning):(1.6) C ends between 30 and 60 m after the start of A

(1.1) the end of A is equal to the start of B(1.2) the end of B is equal to the start of C(1.3) the duration of A is between 10 and 20 m(1.4) the duration of B is between 10 and 20 m(1.5) the duration of C is between 10 and 20 m

A B C10-20 10-2010-20

Correct (consistent) assertion:(1.7) C ends between 30 and 50 m after the start of A

Not correct (inconsistent) assertion:(1.8) C ends more than 70 m. after the start of A

However: Temporal Reasoning is NEEDED in order to support such an intended semantics!

Managing Temporal Constraints: the Problem

DESIDERATA for Temporal Reasoning Algorithms

- tractability “reasonable” response time

- correctness no wrong inferences

- completeness reliable answers

DESIDERATA for the Representation formalism

- expressiveness capture most temporal constraints in GL

TRADE-OFF!

SPECIALIZED APPROACHES (since ’80 in AI literature)

Digression: Why Completeness is fundamental?

Implied constraint (temporal reasoning):(1.6) C ends between 30 and 60 m after the start of A

(1.1) the end of A is equal to the start of B(1.2) the end of B is equal to the start of C(1.3) the duration of A is between 10 and 20 m(1.4) the duration of B is between 10 and 20 m(1.5) the duration of C is between 10 and 20 m

A B C10-20 10-2010-20

Suppose that temporal reasoning is NOT complete, so that (1.6) is not inferredThe answer to query (Q1) might be: YES(Q1) Is it possible that C ends more than 70 m. after the start of A?

Complete Temporal Reasoning is NEEDED in order to grant correct answers to queries!

Temporal Constraint Treatment

WHEN Temporal Reasoning is useful in Guidelines?

ACQUISITION

- to check consistency

EXECUTION

- to compare the duration of paths, in hypothetical reasoning (simulation) facilities

- to check that the time of execution of actions on patients is consistent with the constraints in the guideline

- to schedule next actions

Temporal Representation and Temporal Reasoning for Clinical Guidelines

• Different kinds of temporal constraints

• No current approach in the AI literature covers all of them

• Our proposal: an extension of the “consensus” STP approach [Dechter et al., 91]

• Our goal: expressiveness + correct, complete and tractable inferences

GLARE’s approach (representation)Labeled tree of STPs (STPs-tree)

Tree of STPs for the multiple mieloma chemotherapy guideline.The overall therapy (node N1) is composed by 6 cycles of 5 days plus a delay of 23 days . In each cycle (node N2), two therapies are executed in parallel: Alkeran (node N3: Sa and Ea are the starting and ending nodes), to be repeated twice a day, and Deltacorten (node N4: Sd and Ed are the starting and ending nodes), to be repeated once a day. Arcs between any two nodes X and Y in a STP (say N2) of the STP-tree are labeled by a pair [n,m] representing the minimal and maximal distance between X and Y.

Consistency checking on STPs-trees

ALGO1: temporal consistency of guidelines

Top-down visit of the nodes in the STPs-tree

For each node in the STPs-tree:1) the consistency of the constraints used to specify the

repetition taken in isolation is checked; 2) the “extra” temporal constraints regarding the repetition are

mapped onto STP constraints; 3) Floyd-Warshall’s algorithm is applied to the constraints in

the STP plus the “extra” STP constraints determined at step 2.

Property 1. ALGO1 is correct, complete, and tractable (it

operates in O(N3), where N is the number of actions in the guideline).

Temporal reasoning on guidelines + instantiationsALGORITHM (sketch)

ALGO2: temporal consistency on guidelines execution (1) the existence of non-observed instances whose occurrence is

predicted by the guideline is hypothesized; (2) all the constraints in the general guidelines are inherited by the

corresponding instances (considering both observed and hypothesized instances). This step also involves “non-standard” inheritance of constraints about periodicity; (3) constraint propagation is performed on the resulting set of constraints

on instances (via Floyd-Warshall’s algorithm), to check the consistency of the given and the inherited constraints;

(4) if constraints at step 3 are consistent, it is further checked that such constraints do not imply that any of the “hypothesized” instances should have started before NOW.

Temporal reasoning on guidelines + instantiations

Property 2. ALGO2 is correct, complete, and tractable.It operates in O((N+M)3), where N is the number of actions in the guidelineand M the number of instances of actions which have been executed(and observed).

CONCLUSIONS (Temporal Reasoning)

Many approaches to time in clinical guidelines in the literature, but properties of the inferential mechanisms (if present) quite neglected

GLARE (Guideline Acquisition, Representation and Execution): a proposal of solution based on an extension of well-consolidated Artificial Intelligence techniques

Temporal constraints: an essential part of clinical guidelines

Temporal constraints: need a principled treatment:

not only representation, but also

correct, complete (and tractable) inferential mechanisms

GLARE: Model Checking

GLARE KERNEL

Temporal ReasoningModule

ContextualizationModule

UpdateModule

Model-CheckingModule

Decision-MakingModule

SPIN

Goals of our model-checking work

• Checking properties (e.g., correctness, liveness,…) of clinical guidelines (GL) is an important task

• Only limited and ad-hoc verification approaches in the literature

• (LTL-based) model-checking techniques are successfully used in AI and CS

• We show how (LTL) model checking techniques can be applied to provide a general tool for the verification of clinical guidelines

• We extend the GLARE system by integrating it with the model checker SPIN

The model checking approach

domain

property

Domain model(formalism)

Property represent.(logic)

In SPIN:Promela lang.(agent based)

In SPIN:LTL logics

Interface tologic

Model checker

TRUE orCONTEREXAMPLE

Coupling GLARE with SPIN

• Tool to (automatically!) translate ANY GLARE clinical GL into a Promela model

• Details of translation in AMIA’06 paper • Flexible and general approach for automatic checking of clinical GL

properties

programmer

Property 1

Property k

……

Guideline 1 Guideline n

STANDARD approach(ad-hoc)

user

pgm1

Pgm.k

Property 1

Property k

……

Guideline 1 Guideline n

STANDARD approach(ad-hoc)

user

Our approach (general)

SPIN

GlareTo

Spin

PromelaGuideline 1

PromelaGuideline n

……

Guideline 1 Guideline n

STANDARD approach(ad-hoc)

Our approach (general)

SPIN

GlareTo

Spin

PromelaGuideline 1

PromelaGuideline n

user

Property 1

Property k

General approach to property checking

• No need to produce an “ad-hoc” pgm for each type of property!• Software tools built once and forall• ANY property (that can be expressed in SPIN –i.e. in LTL) can be

checked with NO additional effort

Model Checking: which properties?

• Properties concerning a guideline “per se”: one can check if the guideline contains a path of actions satisfying a given set of conditions

• Properties of a guideline in a given context: specific contexts of execution may impose several limitations on the executable actions of guidelines

• Properties of a guideline when applied to a specific patient: provided that the model checker has in input all the data in the patient record, the feasibility of a given action, or path of actions on the specific patient can be proved

• Integrated proofs: any combination of the above types of proofs is feasible

EXAMPLE: inconsistencies in a guideline

• If a recovery treatment has been excluded, later on the guideline cannot prescribe it

• Given the LTL formula:

□ (conclusion ==recovery_treatment_excluded →proc_recovery_treatment == started)

SPIN produces a counterexample to this property.

Conclusions (Model Checking)• General approach to the automatic check of properties in clinical

GL exploiting AI model checking techniques

• Software tool (translator) built once and forall, to check ANY propery that can be expressed in SPIN (i.e., in LTL)

• Experiments still ongonig, to fully explore the extremely wide range of properties that can be checked automatically

• Open issue:enhancing SPIN’s user interface to express (LTL) properties

Conclusions (general)

• Clinical guidelines can provide crucial advantages in the clinical practice

• Computer-based approaches can help• GLARE: a computer-based approach used to experiment

the application of formal (Artificial Intelligence, Logical and Database) techniques to accomplish cue goals (e.g., decision support, verification)