MIMViewer

MedicalImagingMarkupViewer

TópicosAvançadosemEngenhariaInformática2ProgramaDoutoralemEngenhariaInformática

UniversidadedeAveiro

PEDROLOPES

pedrolopes@ua.pt

2

Abstract

Anewworldofdigitaldiagnosticcapabilitieshasbeenmadepossiblewiththeadvancesinmedical

imaging. These advances promoted communication and cooperation between several distinct

entitiestocreatestandardsandmodelsthatcanbeusedintheentireindustrytoimprovethequality

of the final service offered to users. Despite the growing quality in digital images, they are not

enoughalone.Thatis,itisrequiredthatahumanclinicianelaboratesareportcontainedhismedical

opinionaboutspecifictraitsfoundintheobtainedimages.Thesestructuredreportscomprisemostly

free text annotations that havenot beenexplored yet. Therefore, thedevelopmentof a standard

datamodelforstructuredreportsandimageannotationsisacrucialstepintheprocessofimproving

medicalimagingtechnologiesandservices.

3

TableofContents

Abstract ............................................................................................................................... 2

TableofContents ................................................................................................................ 2

1. Introduction ................................................................................................................. 4

1.1 Problem...............................................................................................................................4

2. DataModel .................................................................................................................. 5

2.1 GeneralAnnotation ............................................................................................................6

2.2 Calculation ..........................................................................................................................6

2.3 Ontology .............................................................................................................................7

2.4 Image ..................................................................................................................................8

2.5 Markup................................................................................................................................8

3. Architecture ............................................................................................................... 10

3.1 Components......................................................................................................................10

3.2 ExecutionScenario............................................................................................................12

4. MedicalImagingMarkupViewer ............................................................................... 13

4.1 Annotations.......................................................................................................................14

4.2 DICOMModule .................................................................................................................15

4.3 ViewerModule..................................................................................................................15

5. ConclusionsandFutureWork.................................................................................... 16

5.1 FutureWork......................................................................................................................16

5.2 Conclusion.........................................................................................................................16

References......................................................................................................................... 18

4

1. Introduction

Medicalimagesaresourceforvastamountsofclinicalknowledge.Theyofferthisknowledgethrough

not only visual information, which fulfils the main purpose of creating the images but also with

metadata:how,when,where the imagewasacquiredandwhat conditions the systemhad.Along

withtheimages,textualreportsareusuallyattached,containingthemedicaldiagnosticgivenwhen

theimagewasacquired.Althoughtheyonlycontaintextinformation,thesereportsarecrucialtoa

deeper analysis of the image data as specialized clinicians create them. The textual information

containedinthesereportsis,initsmajority,freetext.Freetexthasseveralmajordrawbacksderived

fromthefactthatmachinescannotunderstand it.This lackofautonomousunderstandingdisables

featureslikeautomaticspatialcoordinate’sdiscoveryorassociationsamongdistinctreportsorwithin

a simple report. Summarily, it isdifficult forbothmachinesandhumans to index,queryor search

free text annotated to images. This limits the valuable added value that image data, image

annotations and clinical reports can offer in interpretation for clinical, research, and teaching

purposes.

1.1 Problem

An imageannotationcanbe seenasa clinical report component thathas tobe stored inadigital

mediaandprovidedtocliniciansthroughoutdistinctoperatingenvironments.Inthiscase,thematter

inhandsisnotobtainingtheannotation,butstoringandexchangingit.

Creatingadatamodelthatsupportsanykindofannotationcansolvestorageproblems.Thiskind

ofmodelisautopiaandsothesmallergoalistosupportthevastmajorityofannotationsthatcanbe

writtenbyclinicians.Alongwiththisdatamodel,therehastobeastructuredreportthatcanbeused

toexchangedatainseveralscenarios:reportuploadtothestorageserver,reportretrievefromthe

storageserver,reportexchangesbetweendifferentapplications.

Tosolvethestoragedataissue,thecaBIGAIM[1]taskforceisdevelopingadatamodelthatcan

supportalargenumberofimageannotationsandaimstomakeitanindustrystandardformedical

imagingreports.Thedataexchange issue is solvedrecurring to theXMLfile format.XML iswidely

acceptedinthemedicalindustry,whetheroneisdealingwithwebapplicationsandstructuredforms

[2]orwiththecomplexorganizationofthemedicalimagingworkflow[3].

5

2. DataModel

The annotation storage problem may lead to several distinct software solutions and

implementations. An annotation is nothing more than a simple explanation or descriptive

information, produced by a human or a machine, that is directly related to the specific traits

detectedinanimageoracollectionof images.Thatis,anannotationdescribes,withasetoffixed

parameters,themeaningofsomerelevantpixelsinsideanimage.

Includingannotationsalongwiththeimagesempowersthecreationofasemanticlayerontopof

theimagesthatenablesthedevelopmentofnewsoftwaretoolsthataremoreautonomousandthe

implementationofnovelfeaturesliketextordatamining.

ThemodelpresentedinFigure1wasadaptedfromcaBIGworkandisthemostcompletemodel

to date. This complex model can be organized in various logical groups: general information,

ontologyfinding,imagereference,markupandcalculation.Thismodelwaschosenasthemaincase

studyespeciallybecauseitisastate‐of‐the‐artimplementationandtheonewithmoreprobabilityof

becomingadefactostandardinimageannotation.Forabetterunderstandingofthemodellogical

groupsaredescribedbrieflyinthenextsubsections.

Figure1‐CompleteObjectModel

6

2.1 GeneralAnnotation

TheGeneralAnnotation‐Figure2–groupdescribesthegenericpropertiesofaspecificannotation

orsetofannotationsthatarepartofareport.

Themainclass,Annotation,isanabstractclassthatderivesintwokindsofannotationsthatcan

beinstanced: ImageAnnotationandAnnotationofAnnotation,whichareusedtoannotateimages

or other annotations respectively. Annotation can also be related to User and Equipment, which

refer the annotation equipment. The Patient class canonly be relatedwith the ImageAnnotation

entity.

Figure2‐GeneralAnnotationModel

2.2 Calculation

The Calculation group ‐ Figure 3 – represents how calculations are stored in the data model. A

calculationisacollectionofresultsthatareobtainedautomaticallyfromtheimageannotations.For

instance, if an annotation contains a circle, the calculation group will store features like area in

squaremillimetresormaximumandminimumpixelvalues.

The Calculation Result is the main class that stores the type of the result, the number of

dimensionsanditsunitsofmeasure.Thisclassiscrucialasitmakesthemodelextremelygeneric:it

can storedistinct typesof results: scalar, vector,histogramorarray. Inaddition it is also required

thatthecalculationcoordinates–classCoordinate–arestoredandalsothefinalresult–classData.

7

Figure3‐ImageCalculationModel

2.3 Ontology

Ontologydefinesacollectionofrelatedterms,asemanticthesaurithatcontainsthetermsthatcan

beusedtodescribeacertainfeaturebelongingtoapredefinedscientificfield.TheOntologygroup‐

Figure4–describesthekeyclassesusedtoclassifythefeaturesbelongingtoanannotation.

In this particular scenario, the controlled terminologies derive from recognized ontologies:

RadLex, SNOMED‐CT or UMLS and are used to increase the semantic value of each set of

annotations.

Figure4‐OntologyModel

8

2.4 Image

The Image group ‐ Figure 5 – is the core of the annotation markup. This group stores a set of

identifiersthatallowaccesstotheDICOMimagethattheannotationsreferto.TheImageitselfisnot

stored in thedatabase; it is stored in an external PACS systemand is only referenced in thedata

model.

ImageReferenceisthemainimageabstractclass.WebImageReferenceisaninstancethatonly

storestheURItoaspecificimage(PNG,JPEG...),notaDICOMfile.DICOMImageReferencecontains

asubsetofclassesthatareusedtostoretheinformationrequiredtoaccessaparticularDICOMstudy

andtheannotatedimagescontainedinthestudy.

Figure5‐ImageModel

2.5 Markup

TheMarkup group ‐ Figure 6 – is used to store the image annotation elements: the textual and

graphical representations. Text Annotation is used to label the image and store the reported

annotation itself.Everyannotationneedsagraphical representation in the image. In thiscaseone

can have several basic shapes drawn in the image. In order to view these graphic annotations

correctly,italsorequiredthattheircoordinatesarestoredtoallowaredrawingintheexactposition

whenviewingtheannotation.

9

Figure6‐MarkupModel

10

3. Architecture

Severalarchitecturescanbeimplementedtosupporttheproposeddatamodelanddesiredfeatures.

Figure7representsabasicimplementationscenariowithseveralcomponentsthatarerequiredinan

environment working at this scale and in this scientific area. The architecture components are

complexelementsthatcanbecompletelyindependentandoperateheterogeneously.

Figure7‐MIMArchitecture

11

3.1 ComponentsTable1‐ArchitectureComponents

Component Description

PACS

APictureArchiving and Communication System is amesh of computers,mostly

servers,connectedwiththemainpurposeofexecutingalltasksrelatedtoDICOM

images: storage, retrieval, distribution andpresentation. These kinds of systems

areverycommon inmodernhospitalsandtheyaresteadily replacingtraditional

film‐basedimplementationsinmedicalenvironments.

RIS

ARadiologyInformationSystemisanorganizedframeworkthatisusedbymedical

radiology sections to collect, manipulate and distribute patient data and image

references. These services mostly store patient schedules and relevant past

historyandareoneofthemostimportantcomponentsintheradiologyworkflow.

HIS

AHospitalInformationSystemisaframeworksimilartoanRIS,exceptitworksat

a much wider environment: the hospital. In most cases, the HIS is the central

hospital information system that accommodates both the RIS and other similar

platforms. These systems usually store entire patient medical records and are

usedforaccountingoperationsandelectronicdataprocessing.

MIMEngine

TheMedical ImagingMarkupEngineisthecoreoftheimageannotationsystem.

Thiscomponent isresponsibleforconnectingwiththePACS,theRISandtheHIS

to obtain data and channel it to the clients. This engine is also responsible for

readingandwritingtheXMLexchangefileswheretheannotationsarestored.

MIMWriter

The MIM Writer is a desktop tool to annotate images. Clinicians use this

application to loadDICOMimagesandannotate themwhenmakingadiagnosis.

TheannotationisthenstoredandcanbeaccessedbyeithertheMIMWriterfor

further modifications or the MIM Viewer to analyze the image and respective

annotations.

MIMViewer

TheMedical Imaging Markup Viewer is a web application that is used to view

annotations.Thisapplicationconnects to theMIMEnginetoobtain therelevant

data,thenprocessesitandshowsittotheenduserinarichwebenvironment.

12

3.2 ExecutionScenario

A simpleexecution involves several steps from the imageacquisition to the imageanalysis. These

stepsinvolveoperationsexecutedinvariousenvironmentsthatareheterogeneousatseverallevels:

hardware,software,andgeographicallocation...Table2describeanoperationscenario:

Table2‐OperationScenarioDescription

Step Description

1Patientregistersinthe

hospitalThepatientarrivesatthehospitalandisregisteredintheHIS.

2 PatientissubmittedtoaCT

ThemedicalstaffordersaCTscanandthePatientdatais

addedtotheRIS.ThePatientgoestotheradiologylabandis

submittedtoaCTscan.

3CTisstoredinthehospital’s

PACS

WhentheCTisfinisheditisimmediatelystoredinhospital

PACSandmadeavailabletoothersystem.

4

DoctorAaccessestheCTto

makethediagnoseinMIM

Writer

LaterwhenDoctorAisexaminingthePatient,heaccesstheCT

scanwiththeMIMWriter.MIMWriterisadesktop‐based

applicationthatcommunicateswiththeMIMEnginetogather

thePatient’sstudy.

5DoctorAannotatesthe

imagesobtained

WhileusingMIMWriter,DoctorAannotatesthestudywith

severalnoticeabletraits.

6DoctorAsavesthe

annotations

Whenthejobisfinished,DoctorAsavestheannotations.MIM

WriterwillsavethereportintheMIMEnginedatabase.No

datafromthePACSorHISischanged.

7

DoctorAaskscolleague

DoctorBforasecond

opinion

DoctorAhassomedoubtsaboutthefinaldiagnosticand

contactshiscolleagueDoctorBtoobtainotheropinion.This

canbedoneusinganykindofcommunicationmethod.

8

DoctorBisathomeand

accessesMIMViewerinhis

browser

Atthemoment,DoctorBisathome.Duetothisfacthehasto

accessMIMViewerandauthenticateinthesystemtoanalyse

thePatient’sCT.MIMViewerwillcommunicatewiththeMIM

Enginetogathertherequiredinformation:DICOMimages,

PatientinformationandclinichistoryandshowsittoDoctorB.

9DoctorBanalysesthe

Patient’sCT

DoctorBnavigatesinMIMVieweranalysingandeditingDoctor

Areport.

10 DoctorBexchanges DoctorBandDoctorAexchangeinformationregardingthe

13

informationwithDoctorA Patient’sdiagnosticwhichisthencommunicatedtothePatient.

14

4. MedicalImagingMarkupViewer

MIMVieweristhemainclientcomponentthatwasprototypedinthiswork.Theinitialpurposefor

thedevelopmentofthisclientwastoexhibitthepotentialitiesthatthiskindofplatformcanoffer.

The developed application is available online, inside the University of Aveiro, at

http://expression.ieeta.pt:8080/MIMViewer.

Thissimpleapplicationcontainsonlyasmallsubsetofmodulescreatedtomimicthecomplete

platform. These modules include a XML file containing the annotations, a Viewer and a DICOM

component.

4.1 Annotations

The annotations were taken from clinical reports made by clinician Miranda Rodrigues and are

associatedwiththeDICOMimagesprovidedinadistinctmediaformat.TheXMLfilecontainingthe

annotations intends to offer a markup that can contain multiple annotations according to the

previouslydefineddatamodelscheme.

Figure 8 shows a simple image annotation containing the annotation ID, a set of elements to

accesstheDICOMimageandanannotationdescriptionaswellasacollectionofgeometricshapesto

describetheannotation.

Figure8‐SimpleAnnotationExample

15

4.2 DICOMModule

The DICOM module is a simple stub for a DICOM PACS server with WADO functionality. This

componentwillreceivetheDICOMidentifiersreadfromtheannotationsfileandretrievetheimage

referencedintheannotation.TheDICOMidentifiersarepassedintheHTTPquerystringallowinga

genericaccessbytheViewermoduleaswellasbyanyotherapplication.

AlthoughthisrepresentsaDICOMserver,theDICOMimagesarenotgatheredinreal‐time:the

DICOMmodulesimplyselectstheimage,fromastaticgroupof images,whichismoreadequateto

thestudyviewedonscreen.

4.3 ViewerModule

The Viewer module is the main application component. It allows access to the list of available

annotated studies andaccess toeach study setof annotations. Figure9 showsanexampleof the

Viewerinoperation.Ontopthereisthemainapplicationmenuthatprovidesaccesstotheseveral

availablefeatures.Attheleftthereisthestudylist,thecliniclistandroomforanyotherlistthatcan

beaccessedbytheMIMEngine. Inthemiddlethereisthemainworkspacethat isdividedinthree

sections.Thetopsectioncontainsasimpleannotationsummaryattheleftandtheannotationimage

at the right. At the bottom there is the complete annotation report with linkable elements that

dynamically change the annotation image in real time. This image is obtained from the DICOM

moduleand,ifclicked,providesaccesstoafullpreviewoftheannotatedimage.

Figure9‐ViewerInterface

16

5. ConclusionsandFutureWork

5.1 FutureWork

The developments made in this study promote several research lines that can be followed to

improveknow‐how in thisareaand to foster thedevelopmentof state‐of‐the‐artmedical imaging

platforms.Myrecommendationwiththispreliminarystudyisthatanopportunityshouldbegivento

anMScstudenttogodeeperinthissubjectandtoattainthegoalsdescribednext.

Enhancetheframeworkdatamodel,implementitphysicallyinadatabaseanddevelopa

dataaccesslayertostore,manipulateandretrievethestoredinformation.

ImprovetheXMLfileformatforannotationexchangesbetweenheterogeneoussystems.

Theseexchangescanbemadebetweenseveralsystems:importfromportableformatto

database, export from database to portable format and exchange between distinct

operationenvironments.

DesignandimplementMIMEnginewhichistheplatformcoreandmustworkasaproxy

andintegrationelementbetweenthesystemcomponents.

Develop communicationmethodsbetween theMIMEngineand thePACS,RISandHIS

platformsinordertoallowdirectreferencetodataanddataexchangesbetweenallthe

componentsintheenvironment.

Improve MIM Viewer to include new interface features that enhance visualization

capabilities and ease the clinicians’ tasks. Some of these novel features can be:

annotationedition,dynamicannotationviewer, real‐timeannotation,datamining, text

indexing and real‐time access to heterogeneous data (patient, clinician, equipment,

DICOM…).

DevelopaMIMWriterapplicationthatcanbeusedbycliniciansintheirlaboratoriesand

thatallowsthecreationofannotationsinamedicalenvironment.

17

5.2 Conclusion

Image annotations represent a powerful added value to disease diagnostics based on medical

imaging. Despite the quality of current computerized images, free text elements provided by

clinicianswillalwaysbeavaluableadditiontotheobtainedmedicaldata.

Creatingastandardmodelforthesetextualelementsiscrucialtoimprovethequalityofdigital

diagnosisandempowerthedevelopmentofnoveltoolsbasedoncommunicationandcollaboration

betweencliniciansandsupportedbyadvanceddataexchanges.

Thisreportpresentsasimpleinformationmodelforimageannotationandmarkupinhealthcare

basedoncaBIGAIMwork.Inalong‐termperspective,theAIMframeworkwillempowermulticenter

clinical trials where data collection and analysis spans multiple investigators and institutions. In

addition, caBIG’s AIM has the main goal of becoming an industry standard through providing a

foundation for standardized image annotation practice in the clinical, research, and translational

communities.

18

References

[1] D.Channin,P.Mongkolwat,V.Kleper,K.Sepukar,andD.Rubin,"ThecaBIG™AnnotationandImageMarkupProject,"JournalofDigitalImaging.

[2] W. Hur, J. Lee, and C. Kim, "Web‐based Diagnostic Imaging Service Using XML Forms,"JournalofDigitalImaging,vol.19,pp.328‐335,2006.

[3] V. Bashyam,W. Hsu, E. Watt, A. A. T. Bui, H. Kangarloo, and R. K. Taira, "Informatics inRadiology: Problem‐centric Organization and Visualization of Patient Imaging and ClinicalData,"Radiographics,p.292085098,January23,20092009.