Comprehensive Evaluation of Radiation Oncology Information ...chapter.aapm.org/nccaapm/z_meetings/2008-04-25/... · 1Ammenwerth E, Graber S, Herrmann G, et al. ”Evaluation of health

NCCAAPM MeetingNCCAAPM Meeting Rochester, MN Spring 2008 Rochester, MN Spring 2008

Comprehensive Evaluation of Radiation Oncology Information

Systems (ROIS)Luis Fong, Ph.D.

Michael Herman, Ph.D.

Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting

Several Layers of Complexity

• Different types of cancer• Different treatment techniques

• Several technologies

Consultation Patient Information Prescription

Simulation

Treatment PlanQATreatment

TreatmentR & V

MainHospital

OtherSites

Technological Innovations:• EPID• OBI• CBCT• Other IGRT

Different users:• Physicians• Physicists• Therapists• Dosimetrists• IS Staff• Administrative

Staff

• Research• Clinical activities

Analysis:On-lineOff-line

Multi- vs. single-vendorenvironments Lots, Lots of Information …

Paper vs. Paperless


Several Layers of Complexity

• Different types of cancer• Different treatment techniques

• Several technologies

Consultation Patient Information Prescription

Simulation

Treatment PlanQATreatment

TreatmentR & V

MainHospital

OtherSites

Technological Innovations:• EPID• OBI• CBCT• Other IGRT

Different users:• Physicians• Physicists• Therapists• Dosimetrists• IS Staff• Administrative

Staff

• Research• Clinical activities

Analysis:On-lineOff-line

Multi- vs. single-vendorenvironments Lots, Lots of Information …

Functionality, Integration, Connectivity,

Robustness, High Performance and Stability

Paper vs. Paperless

Guarantee High Standards of Patient Care


Some History


“X-Otron”PMH/OCI 1958-1983

H.E. Johns et al.(1959)

kV x-ray tube housed

in the treatment

Head

www.imaginis.com/radiotherapy/radio_history.asp

http://www.slac.stanford.edu/cgi-wrap/getdoc/neal97-008.pdf

Clinac 4 control console

http://scaa.usask.ca/gallery/uofs_events/articles/1951.php

Paper Chart


Varian OBI™Elekta Synergy™

www.hermesmedical.com

TomoTherapy Hi-Art™

www.brachytherapy.com/tomotherapy.html

www.varian.com/prnt/orad/prd131.html

MRI-CT

PET-CT

ROIS


InformationManagement

TechnologicalInnovations

PatientTreatment

High Quality Level of

Patient Care

ROIS

The successful implementation of the ROIS depends greatly on a clear understanding of the site-specific processes and IS/IT infrastructure as well as the capacity of the ROIS to match the constraints and clinical

practice of a particular institution1,2.

1Ammenwerth E, Graber S, Herrmann G, et al. ”Evaluation of health information systems-problems and challenges”. International Journal of Medical Informatics 2003;71:125-135.2Lorenzi N.M, and Riley R. T Organizational Aspects of Health Informatics: Managing Technological Change, New York, Spring-Verlag, 1995, pp 11-13.


Goal• To develop a comprehensive methodology to

systematically assess the processes and information flow as well as the performance of radiation oncology information systems (ROIS).

Clinic Specific Road Map

Clinic SpecificPractice

andInfrastructure

ROIS


Peopleware

Radiation Oncology Classification Model

Radiation Oncology Practice

Clinical Process

Software - Hardware

ROIS

CP

IM TII

Team of Experts

Evaluation Categories: •Clinical Processes (CP), •Information Management (IM)•Technological Innovations Integration (TII)

Team of Expertsrepresenting the

different areas of the radiation oncology:

Physicians

Therapists

Physicists

IS team

Dosimetrists

AdministrativeStaff

Users

Physicians

Therapists

Physicists

IS team

Dosimetrists

AdministrativeStaff

Users


Analytical Methods


Analytical Methods• Analytical Method I

– IS/IT Infrastructure Maps. The purpose of these maps is to provide a clear understanding of all IS/IT processes and connectivity between the different components and internal systems in the department.

• Analytical Method II– Process Flow Maps. These maps include all the

patient care tasks and activities as well as the management of the patient information. These maps will be the basis for the development of a survey.


Evaluate and identify potential interoperability problems and lack of functionality and connectivity

Equipmentor

System A

Equipmentor

System B

ConnectivityProcesses

Analytical Method I - IS/IT Infrastructure Map Template


Analytical Method II - Process Flow MapTemplate

User(s)

Process X

SurveyGeneric:• Tasks• Activities• Sub-Processes

System A

INPUT

System B

OUTPUT


Three Major Areas• Clinical Processes (CP), • Information Management (IM)• Technological Innovations

Integration (TII)

ROIS XProcess XSurvey

Act 1Act 2

Act N

Tool 1Tool 2

Tool N

Analytical Method II – Survey

I. Quantitative Measurement of System Performance*II.Qualitative Measurement by Recording users Experiential Feedback

*where Performance measures the level of integration between the ROIS and the site-specific clinical practice, needs and infrastructure


Analytical Method I IS/IT Infrastructure Maps

Equipmentor

System A

Equipmentor

System B

ConnectivityProcesses


• Two Types of Infrastructures:– Clinical Multi-vendor:

• EXCI-IMPAC• 4DITC-IMPAC

– Test Single-vendor• 4DITC-ARIA

IS/IT infrastructures


EXCI-IMPAC

IMPAC dB (Pervasive):\\rdosrv01\accessrt\impac\db

RT Plan (*.rtp format):\\rdosrv01\accessrt\apps\Impac\rtp\

RT Image:\\rdosrv01\accessrt\apps\impac\stagedcm\

RT Image (per machine):\\rdosrv01\accessrt\apps\impac\pv*

QREADSMIDIALocally

CT-SIM SIM WS GE AdvantageTM

Eclipse/VARIS Vision

Multi-ACCESS /ViewStationVARiS Images

\\mfad.mfroot.org\rchdept\filedata

Brain Lab TPS

Mayo PACS system

Locally

VARIS dB (Sybase):\\varsrv04

PortalVisionWS

LINACConsole

MLC WS

LINACIAS

MLCController

DataTab

EPID

IMPAC image:\\mfad.mfroot.org\rchdept\impac

Record & Verification

IMPAC WS(Sequencer)

NPV6

IMPACDCM module

Eclipse Process

NAMER

RTP to

IMPAC


Locally



VARiS Images\\mfad.mfroot.org\rchdept\filedata

Brain Lab TPS

Mayo PACS system

RTP to

IMPAC

Locally

VARIS dB (Sybase):\\varsrv04

QREADSMIDIA

IMPAC dB (Pervasive):\\rdosrv01\accessrt\impac\db

RT Plan (*.rtp format):\\rdosrv01\accessrt\apps\Impac\rtp\

RT Image:\\rdosrv01\accessrt\apps\impac\stagedcm\NAMER

IMPAC images:\\mfad.mfroot.org\rchdept\impac

IMPACDCM module

4DITC-IMPAC

Eclipse ProcessMulti-ACCESS /ViewStation

IMPAC WS(Sequencer)

LINACConsole

LINAC

MLCController

VARIAN network switch

CBCTWS

EPID OBI

VerificationRecord

SupervisorLVI

DICOM4DITC

WSIAS

OBIWS

IAS


Locally


ARIA Brain Lab TPS

Mayo PACS system

Locally

ARIA dB(Sybase)

:\\varsrv05

QREADSMIDIA

4DITC-ARIA

Record

Daemons

ARIA Images

LINACConsole

LINAC

MLCController


CBCTWS

EPID OBI

Verification

SupervisorLVI

4DITCWS

IAS

OBIWS

IAS


Local

GE Advantage


Mayo PACS system

Local

Eclipse Process

IMPAC WS(Sequencer)

LINACConsole

MLCController


CBCTWS

VerificationRecord

SupervisorLVI

DICOM4DITC

WSIAS

4DITCWS

IAS

OBIWS

IAS

OBIWS

IAS

IMPAC dB (Pervasive)

RT Plan-Temporaryfrom TPSs

RT Images-Temporaryfrom TPSs

Multi-ACCESS

IMPAC images (DFS)

NAMER

RTP to

IMPAC

VARIAN Images (DFS)VARIAN dB

(Sybase)

IMPACDCM module

SIM WS

MV

LINACkV

CTSIM

BrainSCAN

IMPAC dB (Pervasive)

RT Plan-Temporaryfrom TPSs

RT Images-Temporaryfrom TPSs

RT Images-Temporaryper machine

Local

GE Advantage


Multi-ACCESSVARIAN

Images (DFS)

Mayo PACS system

Local

VARIAN dB (Sybase)

PortalVisionWS

LINACConsole

MLC WS

IAS

MLCController

DataTab

MVIMPAC images (DFS)

Record & Verification

IMPAC WS(Sequencer)

NPV6

IMPACDCM module

Eclipse Process

NAMER

RTP to

IMPAC

SIM WSCTSIM

LINAC

BrainSCAN

Local

GE Advantage

ARIA BrainSCAN

Mayo PACS system

Local

VARIAN dB(Sybase)

DICOM Daemons

VARIAN Images(DFS)

LINACConsole

MLCController


CBCTWS

Record&

Verification

SupervisorLVI

4DITCWS

IAS

4DITCWS

IAS

OBIWS

IAS

OBIWS

IAS

SIM WS

MV

LINACkV

CTSIM

Information System

Workstation(WS)

DFS(Distributed File

System)

Server(Database)

Connectivity Service

Imager

LocalStorage

Symbols Description:

IS/IT Infrastructure Maps EXCI-IMPAC

4DITC-IMPAC4DITC-ARIA


Analytical Method II Process Flow Maps

User(s)

Process XSystem A

INPUT

System B

OUTPUT


Clinical Processes (CP)

Information Management (IM)

Technological Innovations Integration (TII)


Site-Specific Clinical Processes

Mayo EMR Mayo GPAS

ROIS

Treatment Planning

Plan Second Check

Treatment Delivery

Weekly Chart CheckWeekly Management VisitCompletion of Treatment

QA (IMRT)

Reports

Rad Onc

IMRT-SReview of Images

Weekly Programming

Physician Consult Desk Notification Treatment Scheduling Simulation


Mayo GPASFront Desk

Desk Notification

Treatment Scheduling

Tx Scheduler ROIS

Therapist

Simulation

PhysicianROIS

Physician Consult

Mayo EMR

Physician


Physicist

Plan Second Check

ROIS

Report

Dosimetrist

Treatment Planning

Physician ROIS

Report

IMRT-S

Physicist

IMRT QA

ROIS

IMRTReport

IMRT-S


Therapist

Treatment Delivery

IMRTReport

Report

ROIS

Physicist ROIS

Weekly Chart Check

Therapist ROIS

Weekly Programming

Weekly Management Visit

Physician ROISMayo GPAS

Review Images

Physician ROIS

Daily

Weekly


Completion of Treatment

Front Desk

Tx Scheduler

TherapistIMRT

ReportReport

IMRTReport

Report

Storage

ROIS

Mayo EMRDose

Summary


Clinical Process Flow Map

Patient information and process management systems:(A) Mayo EMR, (B) Physician Internal Electronic Ordering form,(C) Mayo Schedule Manager(D) IMRT QA Manager(ROIS)


Therapist

Simulation

PhysicianROIS

System: ROIS X

Process: Simulation

Please evaluate the performance of the system for: ScoreAdding patient ID photoAdding field specific set-up photosAdding patient set-up informationCreating Reference FieldsSending reference images from GE Workstation into the systemSending images/field information from Acuity into the systemViewing/Accessing information from previous treatment coursesViewing/Accessing prescription and related notesAdding specific task lists to patient informationBilling appropriate items performed

Information System Evaluation Survey

Example of Process Flow

Map and corresponding

survey






Information Management:I. Database management II. Long-term archiving

solutionIII. Application availabilityIV. Customer supportV. System integrationVI. Data Conversion

Information Management

Survey






Technological Innovations Integration

I. New imaging modalities: I. Cone Beam CT (3D matching),II. EPID-MV (2D matching), III. OBI-kV, kV – MV (2D matching)

II. Portal Dosimetry.

• Implementation

• Connectivity with ROIS

• Operability with ROIS

• Ability to keep updating with ROIS

• Ability to perform off-line review for research purposes


Survey



Survey

Clinical ProcessSurvey


Survey

ROIS X

Physicians

Therapists

Physicists

IS team

Dosimetrists

AdministrativeStaff

Users

Physicians

Therapists

Physicists

IS team

Dosimetrists

AdministrativeStaff

Users


Performance-Importance of Patient Care Space

High Medium Low

5 4 3 2 1

• High. The performance of this activity is essential to guarantee an accurate and safe patient treatment. It has a high effect on dose delivery as well as in the information and process flow.

• Medium. The performance of this activity helps in the process and information flow, and has medium dosimetric effect on the patient treatment.

• Low. The performance of this activity has very small effect on process and information flow and does not have any dosimetriceffect at all on the patient treatment

Importance for Patient Care Weighting Factor

ExcellentAbove

Adequate AdequateBelow

Adequate Poor

0 if Not Available 5 4 3 2 1

• Excellent. The system performs the activity and has the capability of improving the efficiency and efficacy of our department processes flow.

• Above Adequate. The system performs the activity with some extra features that make the activity flow smoothly.

• Adequate. The system performs the activity adequately• Below Adequate. The system performs the activity with some

minor difficulties that could be overcome.• Poor. The system performs the activity in a way that could not

be implemented in our department. • Not Available. The system is unable to perform this activity.

Performance Scoring Scale:

Poor Adequate Excellent

Performance

Impo

rtanc

e fo

r Pat

ient

Car

e High

Medium

Low

RejectionRegion

(I)

RejectionRegion

(II)

AcceptanceRegion

(I)

AcceptanceRegion

(II)

Performance Patient Care Factor

Magnitude Direction


Performance Sorted by Patient Care Priority

PoorBelow

Adequate AdequateAbove

Adequate Excellent

1 2 3 4 5Performance

HighMediumLow

1 2 3 4 5

HighMediumLow

1 2 3 4 5

Importance for Patient Care

Region Evaluate the performance of the system for: PI IPCRR-I Review detailed treatment record 2.50 5.00RR-I Reviewing changes in plan per field 2.50 5.00AR-I Reviewing dose site summary 4.00 5.00AR-I Reviewing overrides 4.00 5.00AR-I Checking the chart since the last time checked 5.00 4.00RR-I If changes were made, tracking the history of the change 2.50 3.50RR-I If overrides were made, tracking the history of the override 2.50 3.50RR-I Reviewing action points 2.75 3.50RR-I Selecting BID patients 3.00 3.50AR-I Selecting patients per machine per site 4.50 3.50AR-I Signing off electronically that chart check was done 5.00 3.50


Clinic Specific Performance vs. Patient Care

PERFORMANCE

PA

TIE

NT

CA

RE


High

Medium

Low



Survey



Survey

Patient Care Site-Specific Master Priority List in terms

of Importance for Patient Care

Focus on High Importance for Patient

Care and Low Performance



Survey



Survey

Patient Care

Lens 2

Lens 3

Lens n

Alternative Priority Coefficient

nPriorityPerformace ∏×

Site-Specific Master Priority List in terms

of n-Priorities



High

Medium

Low

Performance

Impo

rtan

ce o

f Pat

ient

Car

e

RR-I

RR-II AR-II

AR-I


High

Medium

Low

Performance

Impo

rtan

ce o

f Pat

ient

Car

e

RR-I

RR-II AR-II

AR-I

Quantitative Results

ROIS A ROIS B

In our practice, both evaluated systems showed, in average, above adequate performance


Qualitative Results

Clinical Practice• Proficient knowledge and understanding of database management, DICOM RT and the

components, connectivity and infrastructure of any clinically relevant information systems linked to the ROIS is needed by physicist and IS team

Vendor• Development of efficient and effective customer services program as well as training resources

Vendor• It is highly desirable to have the vendors provide a virtual environment (beyond a canned demo)

ready for the users which will allow them to test the tools provided by the ROIS, simulating their current practice.

Clinical Practice &Vendor

• Effective channels of communication among users and between user and the vendor need to be developed in relation with ROIS issues, configuration and upgrades.

Clinical Practice

• Appropriate amount of resources need to be allocated toward the development of a strong IS team. The main role of the IS team will be to provide support for the everyday maintenance and basic trouble shooting of the non-clinical relevant hardware and software (e.g. virus configuration, operating system patches, workstations, etc.)


• Successful, effective and safe approach to ROIS issues shall be conducted by a team formed by: physics, IS staff and vendor service support.


• Clear definition of roles and responsibilities of vendor and user related to the management, service and support of ROIS need to be developed prior its deployment.

ResponsibleRecommendation


Who could benefit from this Evaluation Method?

Specific ClinicalPractice ROIS

Communication Channels&

Taxonomy• Vendors• Developers• Administrators

• Physicians• Physicist• Administrators• All Users

• Prioritize• Develop better

Processes

• Prioritize• Develop better

Solutions and Tools

Ultimately Improve

Patient Care


Conclusions• This work provides for the first time an objective tool to

understand and analyze clinical information flow and its compatibility with a given ROIS in a clinic specific patient care setting.

• Mapping provides insight into process communication, interoperability and efficiency.

• The hierarchy and importance of various characteristics are customizable to a given clinical practice and thus allow the tool’s broad applicability.

• Proper assessment of information flow and matching to an ROIS will provide a more efficient and more effective care delivery setting


Acknowledgements

• Lori Buchholtz• James Sorenson • Janelle Miller• Alan Kraling• Jill Rendler• Tim Peters• Jessica Schlichting• Michael Herman, Ph.D.• Jon Kruse, Ph.D.• Terry Harms• Bob Dahl• Janelle Molloy, Ph.D.• Jack T. Cusma, Ph.D.

• Chris Hanna, Ph.D.• Rob Mitchell• Matt Adler• Maureen Thompson• Jeremy Cohen• Richard Nusspickel• David Ackerson• Scott Johnson• Sean Lapinski• Chuck Pearce• Kenneth Dick• Cathy Tenda• Diana Jones

Project supported in part by Varian Medical Systems


Thanks!

Ready for some

questions

?

Documents

Comprehensive Evaluation of Radiation Oncology Information ...chapter.aapm.org/nccaapm/z_meetings/2008-04-25/... · 1Ammenwerth E, Graber S, Herrmann G, et al. ”Evaluation of health