Medical Multimedia Information Systems (ACMMM17 Tutorial)

MedicalMultimediaInformationSystems

KlausSchoeffmann1, BerndMünzer1, Pål Halvorsen2,MichaelRiegler2

1 InstituteofInformationTechnology

KlagenfurtUniversity,Austria

2 SimulaResearchLaboratory

Norway

• Introduction&Overview• MultimediaDatainMedicine• CharacteristicsofEndoscopicVideo• DifferentFieldsandCommunities

• Application1:Post-ProceduralUsageofSurgeryVideos• Domain-SpecificStorageforlong-termArchiving• VideoContentAnalysis• Visualization,Interaction&Annotation

• Application2:DiagnosticDecisionSupport• Knowledgetransfer• Analysis• Feedback

• Conclusions&Outlook

Agenda

ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS) 2

Introduction


Inspectionsandinterventionproducemanykindsofdata• Medicaltext

• ORreports,Patientrecords…

• Sensorsignals• ECG,EEG,vitalsigns

• Medicalimages(radiology)• Ultrasound,x-ray• CT,MRI,PET,…

• Medicalvideo• Opensurgery• Microscopicsurgery• Endoscopicinspections• Endoscopicsurgery

Multimedia Data in Medicine


Communities:

• SignalProcessing

• MedicalImaging

• Computer-AssistedSurgery/Robotics

• Multimedia

„HumanEEGwithout alpha-rhythm“by Andrii Cherninskyi /CCBY-SA

„Pankreatitis“by Hellerhoff/CCBY-SA„Ultrasound“,PublicDomain

• Traditionalopensurgery?

• Minimallyinvasiveinterventions

• Reducedtraumaforpatient

• Inherentlyavailablevideosignal

• Usefulfordocumentation

• Microscopicsurgery

Video Data Sources in Medicine


„Laparoscopy“,PublicDomain

„KussmaulGastroscopy“,PublicDomain

Diagnostic Endoscopy


• Diagnosis/Inspections• Gastroenterology(colonoscopy,gastroscopy)

• Bronchoscopy

• Hysteroscopy

• …

• Flexibleendoscope

• Naturalorifices

• WCE(Wirelesscapsuleendoscopy)„Colonoscopy“,PublicDomain

„Kolontransversum“by J.Guntau /CCBY-SA

Therapeutic Endoscopy


• Therapy/Surgery• Laparoscopy

• Cholecystectomy

• GynecologicalSurgery

• UrologicalSurgery

• …

• Arthroscopy

• …

• Rigidendoscope

• SmallIncisions„Laparoscopy“by BruceBlaus /CCBY

„Arthroscopy“,PublicDomain

Endoscopic Video Examples


Domain-specific Characteristics & Challenges


• FullHDor4K(evenstereo3D)

• Singleshot recordings

• Up to multiplehours

• Homogenous color distribution

• Visually very similar content

• Circular content area

• Restricted motion

• Geometric distortion

• Specular reflections

• Occlusions

• Smoke

• Noise,motion blur,blood,flying particles

ResearchFieldsandCommunities


Overview


Münzer,Bernd,KlausSchoeffmann,and LaszloBöszörmenyi."Content-based processing and analysis of endoscopicimages and videos:Asurvey."MultimediaToolsand Applications (2017):1-40.

Pre-Processing

• ImageEnhancement• Contrastenhancement,colormisalignmentcorrection…

• Cameracalibrationanddistortioncorrection• Specularreflectionremoval• Combstructureremoval&superresolution• …

• InformationFiltering• FrameFiltering• ImageSegmentation


T.Stehle.Removalofspecularreflectionsinendoscopicimages.ActaPolytechnica:JournalofAdvancedEngineering,46(4):32–36,2006.

J.Barreto,J.Roquette,P.Sturm,andF.Fonseca.AutomaticCamera Calibration AppliedtoMedicalEndoscopy.In20thBritishMachineVisionConference(BMVC’09),2009.

B.Münzer,K.Schoeffmann,andL.Böszörmenyi.RelevanceSegmentationofLaparoscopicVideos.In2013IEEEInternationalSymposium onMultimedia(ISM),pages84–91,Dec.2013.

A.Chhatkuli,A.Bartoli,A.Malti,andT.Collins.Liveimageparsinginuterinelaparoscopy.InIEEEInternationalSymposiumonBiomedicalImaging(ISBI),2014.

Real-time Support at Intervention Time

Applications

§ Diagnosissupport

§ Robot-assistedsurgery

§ ContextAwareness

§ Augmentedreality


“Roboticsurgicalsystem”,PublicDomain

T.Collins,D.Pizarro,A.Bartoli,M.Canis,andN.Bourdel.Computer-AssistedLaparoscopicmyomectomybyaugmentingtheuteruswithpre-operativeMRIdata.In2014IEEEInternationalSymposiumonMixedandAugmentedReality(ISMAR),pages243–248,Sept.2014.

„DaVinciSurgical System“by Cmglee /CCBY-SA

Slightlymodifiedfrom:M.P.Tjoa,S.M.Krishnan,etal.Featureextractionfortheanalysisofcolonstatusfromtheendoscopicimages.BioMedical EngineeringOnLine,2(9):1–17,2003.

• 3Dreconstruction

• Deformingtissuetracking

• ImageRegistration

• Instrumentdetectionandtracking

• Surgicalworkflowunderstanding

Enabling Techniques


L.Maier-Hein,P.Mountney,A.Bartoli,H.Elhawary,D.Elson,A.Groch,A.Kolb,M.Rodrigues,J.Sorger,S.Speidel,andD.Stoyanov.Opticaltechniques for 3Dsurface reconstruction incomputer-assisted laparoscopic surgery.MedicalImageAnalysis,17(8):974–996,Dec.2013.

S.Giannarou,M.Visentini-Scarzanella,andG.Z.Yang.Affine-invariantanisotropic detector for softtissue tracking inminimally invasivesurgery.InBiomedicalImaging:From Nanoto Macro,2009.ISBI’09.IEEEInternationalSymposiumon,pages 1059–1062,2009.

Post-Procedural Applications

Managementand Retrieval• Compression and storage• Content-based retrieval• Temporalvideo segmentation• Videosummarization• Visualization &Interaction

QualityAssessment§ Skillsassessment

§ Education&Training

§ ErrorRating

§ Assessmentof intervention quality


M.Lux,O.Marques,K.Schöffmann,L.Böszörmenyi,andG.Lajtai.Anovel tool for summarization of arthroscopic videos.MultimediaToolsandApplications,46(2-3):521–544,Sept.2009.

D.Liu,Y.Cao,W.Tavanapong,J.Wong,J.H.Oh,andP.C.deGroen.Quadrantcoveragehistogram:anewmethodformeasuringqualityofcolonoscopic procedures.InEngineeringinMedicineandBiologySociety,2007.EMBS

2007.29thAnnualInternationalConferenceoftheIEEE,pages3470–3473,2007.

J.Muthukudage,J.Oh,W.Tavanapong,J.Wong,andP.C.d.Groen.ColorBasedStoolRegionDetectioninColonoscopyVideosforQualityMeasurements.InY.-S.Ho,editor,AdvancesinImageandVideoTechnology,number7087inLectureNotesin

ComputerScience,pages61–72.SpringerBerlinHeidelberg,Jan.2012.

• Vision• Archivetogetherallrelevanttext,image,andvideodata• Usedataforinformationretrieval• Supportsurgeonsatdiagnosis,surgeryplanning,teaching,…• Combinedifferentkindofdata(e.g.,radiology-supportedsurgery)

• Challenges• Isolatedsystems/separationofdata• VeryBigData• Alotofirrelevantcontent• Veryspecificdomaincharacteristics• Needfor domain expertknowledge• Differentcommunities and views

Medical Multimedia Information Systems


Post-ProceduralUseofSurgeryVideos


• Videorecordingsofendoscopicsurgeriesshowthesameimagesthesurgeonusedforoperation

• Valuableinformationforpost-proceduraluse:• Laterinspectionofspecificmoments• Discussionofcriticalmoments(e.g.,withOPteam)• Informationtopatients• Preparationoffutureinterventions• Forensics&investigations(e.g.,comparisons)• Trainingandteaching• Surgicalqualityassessment(technicalerrors)

Video as the ’’Eye of the Surgeon’’


Full Storage of Endoscopic Videos

• Exemplaryhospital• 5departments(Lap,Gyn,Arthro,GI,ENT)• 2operationrooms,each4ops/day,eachopca.1-2h• à i.e.40interventionsperday,each~90mins.

• 60hoursvideoperday!• Assumption:HD1920x1080,H.264/AVC• 270GB/day(1h=4.5GB)• 1.9TB/week• 100TB/year(200TBMPEG-2) 4Kabouttwiceasmuch!

(unlessencodedwithH.265/HEVC)


Greatchallengeforahospital’sITdepartment!

How to Reduce Storage Requirements?

1. Spatial compression optimization

2. Temporal compression optimization

3. Perceptual quality based optimization

Transcoding


upto30%

upto40%upto93%

Study on Video Quality

• Subjectivequalityassessment• CatharinaHospitalEindhoven,NL• 37participants

• 19experiencedsurgeonsand18trainees• 7women,30men,averageage:40years

• Subjectivetestsregardingmaximumcompression1) Perceivablequalityloss

• Double-Stimulus(ITU-RBT.500-11)• Switchbetweenreferenceandtestvideo

2) Perceivablesemanticinformationloss• SingleStimulus(ITU-RP.910)• Assessingrandomvideos(incl.reference)

Münzer,B.,Schoeffmann,K.,Böszörmenyi,L.,Smulders,J.F.,&Jakimowicz,J.J.(2014,May).Investigationoftheimpactofcompressionontheperceptionalqualityoflaparoscopicvideos.In2014IEEE27thInternationalSymposiumonComputer-BasedMedicalSystems (pp.153-158).IEEE.

Session1 Session2


Assessment of Video Quality (Session 1)

-5

0

5

10

15

20

25

30

35

0

3000

6000

9000

12000

15000

18000

21000

24000

20 22 24 26 28 18 20 22 24 26 18 18

Diffe

renceMeanOpinion

Score(D

MOS)

Bitrate(Kb/s)

TestConditions

Averagebitrate Ratingdifference

1920x1080 1280x720 960x540 640x360

subjectivelybetterthanreference

Referencevideo(MPEG-2,HD,20(35)Mbit/s)

“lossless”


crf(constantratefactor)

Assessment of Video Quality (Session 2)

1. Visuallylosslesswith8Mbit/sQ1(incomparisonto20Mbit/s)Reduction:60%datavs.0%MOS

2. Goodqualitywith2,5Mbit/sandQ2reducedresolution(1280x720)Reduction:88%datavs.7%MOS

3. Acceptablequalitywith1,4Mbit/sQ3andlowerresolution(640x360)Reduction:93%datavs.31%MOS

1

23


Example Videos

1280x720Weakcompression

16MB

(crf 18)

640x360Strongcompression

0,8MB

(crf 26)

20x


EndoscopicVideoContentAnalysis


1000frames(sampledfrom17minwith1fps)

ACMMultimedia2017Tutorial

MedicalMultimediaInformationSystems(MMIS)

26

Content Relevance Filtering / Instrument Recognition


Münzer,B.,Schoeffmann,K.,&Böszörmenyi,L.(2013,December). Relevancesegmentationoflaparoscopicvideos.InMultimedia(ISM),2013IEEEInternationalSymposiumon(pp.84-91).IEEE.

Primus,M.J.,Schoeffmann,K.,&Böszörmenyi,L.(2015,June).Instrumentclassificationinlaparoscopicvideos.InContent-BasedMultimediaIndexing(CBMI),201513thInternationalWorkshopon(pp.1-6).IEEE.

Instrumentdetectionforcontentunderstanding(e.g.,opphasesegmentation,followinginstrumentsinrobot-assistedsurgery)

Out-of-patientScenes BlurryScenes BorderArea

Phase Segmentation (Cholecystectomy)


ManfredJ.Primus,KlausSchoeffmann andLaszloBöszörmenyi.“TemporalSegmentationofLaparoscopicVideosintoSurgicalPhases“,inProceedingsofthe 14thInternationalWorkshoponContent-BasedMultimediaIndexing(CBMI2016),Bucharest,Romania,2016

à Phasesegmentationthroughinstrumentrecognition(coloranalysis,imagemoments,rules/heuristics)

Instrument Recognition/Tracking


Classification of OP Scene (Cataract Surgeries)


ManfredJ.Primus,DorisPutzgruber-Adamitsch,MarioTaschwer,BerndMünzer,Yosuf El-Shabrawi,LaszloBöszörmenyi,andKlausSchoeffmann.“Frame-BasedClassificationofOperationPhasesinCataractSurgeryVideos“. Proceedingsofthe24thInternationalConferenceonMultimediaModeling2018(MMM2018),Bangkok,Thailand,2018,pp.1-12, toappear

Learning Medical Semantic (e.g., Surgical Actions)


1.105Segments/823.000Frames/9hannotatedVideo (outof111interventions)

Dissection – 58Segs /35.517Pics Coagulation – 212Segs /84.786Pics Cutting cold – 271Segs /26.388Pics

Cutting – 106Segs /92.653Pics Hysterectomy – 25Segs /68.466Pics Injection – 52Segs /52.355Pics

Suturing – 92Segs /321.851PicsSuction &Irrigation– 173Segs /73.977Pics

Petscharnig,S.,&Schöffmann,K.(2017).Learninglaparoscopicvideoshotclassificationforgynecologicalsurgery.MultimediaToolsandApplications,1-19.

WHY?• structurevideocontent,• automaticindexingforretrieval,• automaticsupervisionofsurgeries

Deep Learning Surgical Actions


ConfidenceThresholdslow high

Petscharnig,S.,&Schöffmann,K.(2017).Learninglaparoscopicvideoshotclassificationforgynecologicalsurgery.MultimediaToolsandApplications,1-19.

Deep Learning Surgical Actions


R...RecallP...Precision

Smoke Detection


Cauterizationin90%surgeries

Instruments:LaserorHF

(100° - 1200° C)

Currentfiltrationsystemmanual!

à AutomaticSmokeDetection&Removal?(Real-Time)

Automatic Smoke Detection


AchievablePerformancewithSaturationPeakAnalysis(SPA)

AndreasLeibetseder,ManfredJ.Primus,StefanPetscharnig,andKlausSchoeffmann.“Image-basedSmokeDetectioninLaparoscopicVideos“.Proceedingsof ComputerAssistedandRoboticEndoscopyandClinicalImage-BasedProcedures:4thInternationalWorkshop,CARE2017,and6thInternationalWorkshop,CLIP2017,heldinConjunctionwithMICCAI 2017,QuebecCity,QC,Canada,September14,2017,pp.70-87

Automatic Smoke Detection - Performance


20Kimages(DSA) 10Kimages(DSA)4.5Kimages(DSB)

SPA: SaturationPeakAnalysisGLNRGB:GoogLeNet usingRGBimagesGLNSAT:GoogLeNet usingsaturationonlyimages

Deep Learning


Real-Time Smoke Detection Prototype



VideoInteractionTools


Desired Status


BerndMünzer,KlausSchoeffmann andLaszloBoeszoermenyi.“EndoXplore:AWeb-basedVideoExplorerforEndoscopicVideos“. ProceedingsoftheIEEEInternationalSymposiumonMultimedia2017(ISM2017),Taipei,Taiwan,2017,pp.1-2

Special Content Visualization


Special Interaction Tools


MarcoA.Hudelist,SabrinaKletz,andKlausSchoeffmann.2016.AMulti-VideoBrowserforEndoscopicVideosonTablets.In Proceedingsofthe2016ACMonMultimediaConference (MM'16).ACM,NewYork,NY,USA,722-724.

MarcoA.Hudelist,SabrinaKletz,andKlausSchoeffmann.2016.ATabletAnnotationToolforEndoscopicVideos.In Proceedingsofthe2016ACMonMultimediaConference (MM'16).ACM,NewYork,NY,USA,725-727.

Surgical Quality Assessment (SQA) Software


• Integratingratingfeatures• Moreefficientvideonavigation/browsing

MarcoA.Hudelist,HeinrichHusslein,BerndMuenzer,SabrinaKletz andKlausSchoeffmann.“ATooltoSupportSurgicalQualityAssessment“,inProceedingsofthe ThirdIEEEInternationalConferenceonMultimediaBigData (BigMM),LagunaHills,CA,USA,2017,pp.238-239.

DiagnosticDecisionSupport

43ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)

ChallengesandRequirements



Medicalknowledgetransfer AutomaticDataanalysis/detection Feedback/visualization

• Medicalknowledgetransfers– needDATAw/GroundTruth

• Highdetectionaccuracy

• Fastandefficient:real-timefeedbackandlargescale

• Fitthenormalexaminationprocedures

• Adheretoethical,legal,privacychallenges&regulations


Key Challenges & Requirements

Gastrointestinal(GI)CaseStudy(challenges,systemsupport,datasets,diagnosticdecisionsupport,...)


• Manytypesofdiseasescanpotentiallyaffectthehumangastrointestinal(GI)tract– thedigestivesystem

• about2.8millionsofnewluminalGIcancers(esophagus,stomach,colorectal)aredetectedyearly• themortalityisabout65%

• ScreeningoftheGItractusingdifferenttypesofendoscopy…• iscostly(colonoscopyaccordingtoNYTimes:$1100/patient,$10billiondollars)• consumesvaluablemedicalpersonneltime(1-2hours)• doesnotscaletolargepopulations• isintrusivetothepatient• …

• Currenttechnologymaypotentiallyenableautomaticalgorithmicscreeningandassistedexaminationsà atrueinterdisciplinaryactivitywithhighchancesofsocietalimpact


GI Tract Challenges and Potential


WHO: Colorectal Cancer Mortality 2012

Women

Men

Colorectalcanceristhethirdmostcommoncauseofcancermortalityforbothwomenandmen,anditisaconditionwhereearlydetectionisimportantforsurvival,i.e.,a5-yearsurvivalprobabilityofgoingfromalow10-30%ifdetectedinlaterstagestoahigh90%survivalprobabilityinearlystages.

Colonoscopyitisnottheidealscreeningtest.Relatedtothecancerexample,onaverage20%ofpolyps(possiblepredecessorsofcancer)aremissedorincompletelyremoved.Theriskof gettingcancerlargelydependontheendoscopists abilitytodetectandremovepolyps.A1%increaseindetectioncandecreasetheriskofcancerwith3%.

ACMMultimedia2017Tutorial MedicalMultimediaInformationSystems(MMIS)

Live Automatic Detection

• Systemtoassistdoctorsduringliveendoscopyprocedures

• detectionaccuracydependonexperienceandskills

• havea“secondeye”,“better”detection

• automatictagging,annotationoflesions

• Betterprocedurefordocumentation,automaticreportgeneration

50

51MedicalMultimediaInformationSystems(MMIS)

Video Capsule (PillCam)

§ Standardcolonoscopy:§ expensive§ doesnotscale§ intrusive

§ WirelessVideoCapsuleendoscopy:§ betterscale§ lessintrusive§ possibletocombine

examinations

§ watchhoursofvideo§ lessexpensive?


System Overview

MedicalKnowledgeTransfer(DataCollection)


• Needmoredataandthereforetoolstoefficientlyannotateandtagdata


Available GI Datasets

Name Contain Annotation Size Type Usage

CVC-ClinicDB Polyps GTmasks 612images Trad. ©,bypermission

ETIS-Larib PolypDB Polyps,Normal GTmasks 1500images Trad. ©,bypermission

ASU-MayoClinicDB Polyps,Normal GTmasks 18videos Trad. ©,bypermission

ColonoscopyVideosDB VariousLesions Sorted 76videos Trad. Academic

CapsuleEndoscopyDB VariousLesionsandFindings Sorted 3170images, 47videos VCE Academic, byrequest

GastroAtlas VariousLesionsandFindings Sorted,Textannotations 4449 videos Trad. Academic

WEOAtlas VariousLesionsandFindings Sorted,Textannotations ? Trad. Academic

GASTROLAB VariousLesionsandFindings Sorted,Textannotations ? Trad. Academic

AtlasofGE VariousLesions Sorted,Textannotations 669 images Trad. ©,bypermission

• Whichimageisnotfromthesameclass?

…anditgetsworse…

• Makingamistakebetweencatsanddogsmaynotmatter,butamisclassificationheremayhavelethalconsequences

Why Can’t CS People Do the Annotation!?


PylorusZ-line Z-line Z-line Z-line Z-line

• Simpleandefficient

• Web-based

• Assistedobjecttracking


Video Annotation Subsystem"ExpertDrivenSemi-SupervisedElucidationToolforMedicalEndoscopicVideos"

ZenoAlbisser,et.al.ProceedingsoftMMSys,Portland,OR,USA,March2015

• Forlargecollectionofimages• VV/Kvasir dataset• Fullycleaned

• Featureextractionmechanisms

• Differentunsupervisedclusteringalgorithms

• Hierarchicalimagecollectionvisualization

• Opensource:ClusterTaghttps://bitbucket.org/mpg_projects/clustertag


ClusterTag: Image Clustering and Tagging Tool"ClusterTag:InteractiveVisualization,ClusteringandTaggingToolforBigImageCollections"

KonstantinPogorelov,et.al.ProceedingsofICMR,Bucharest,Romania,June2017

• Multi-ClassImageDatasetforComputerAidedGIDiseaseDetection• GIendoscopyimages• Someimagescontainthepositionandconfigurationoftheendoscope(scopeguide)• 8differentanomaliesandanatomicallandmarks

• v1:500imagesperclass,6pre-extractedglobalfeatures• v2:1000imagesperclass

• Newinformationaddedinthefuture:http://datasets.simula.no/kvasir/


The Kvasir Dataset"Kvasir:AMulti-ClassImage-DatasetforComputerAidedGastrointestinalDiseaseDetection"

KonstantinPogorelov,etal.ProceedingsofMMSYS,Taiwan,June2017

• BowelPreparationQualityVideo

• 21GIendoscopyvideos ofcolon

• Someframescontainthepositionandconfigurationoftheendoscope(scopeguide)

• 4classesshowingfour-scoreBBPS-definedbowel-preparationquality

• 0- verydirty• …• 3- veryclean

• http://datasets.simula.no/nerthus/


The Nerthus Dataset"Nerthus:ABowelPreparationQualityVideoDataset"

KonstantinPogorelov,etal.ProceedingsofMMSYS,Taiwan,June2017

GIAnomalyDetectionSystem


• Easytoextendwithnewdiseases• Easytoextendwithnewalgorithms• Easytotrain

• Resultsareexplainable?

• DiseaseLocalization?

• Real-time?


Detection and Automatic Analysis subsystem


State-of-The-Art: Some Example Detection Systems

Polyp-Alert• detectspolypsusingedgesandtexture• nearreal-timefeedbackduringcolonoscopy(10fps)• detected97.7%(42of43)ofpolypshotson53randomlyselected

(notperframedetection)• only4.3%ofafull-lengthcolonoscopyprocedurewronglymarked• oneofthefewend-to-endsystems• Wallapak Tavanapong – fromMMcommunity

• Featuresextractionusingopen-sourceLIRE(Lucene ImageRetrieval)• Indexer:

• IndexingimagesbyLIREfeaturesfor“training”

• Classifier:• Built-inbenchmarkingfunctionality• Outputtoconsole&JSON/HTML

• Verifiedwithdifferentdatasetsandusecases,e.g.,life-logging,recommendersystems,networkanalysis,etc.

• Opensourceproject– OpenSea


Global Features (GF)-Based Detection”EIR- EfficientComputerAidedDiagnosisFrameworkforGastrointestinalEndoscopies"

MichaelRiegler,et.al.ProceedingsofCBMI,Bucharest,Romania,June2016

• Searchforanoptimalcombinationofglobalimagefeaturedescriptors

• Combiningresultsbylatefusion• LIREimagefeaturedescriptorsJCDandTamuraarethebestchoice


Global Features (GF)-Based Detection

Originalpolyp Colorfeature Edgeandcolor Texture Edge



Featureextractors

Features

Features

Polyps

Cancer

Featureextractors

FeaturesNormal

Distancetothetrainingimages

Classselectionforeachfeature

Distance

Distance

Polyps

Cancer

DistanceNormal

Indexofthetrainingset

LatefusionImageclass

• WithmanyenoughCPUs,thedetectionrunsinreal-time

• GPU-acceleration



Java CUDAC++

""GPU-acceleratedReal-timeGastrointestinalDiseasesDetection"KonstantinPogorelov,et.al.

ProceedingsofCBMS,Dublin,Ireland/Belfast,NorthernIreland,June2016

• Tensorflow asbackend• BasedonInceptionv3• Lastlayersremoved• Modelretrainedonmedicaldata• Applyingsimpletransformationstoincreasesizeoftrainingset

• Verylongtrainingtime• Applyingmodelisfast


Basic CNN-Based Detection“Efficientdiseasedetectioningastrointestinalvideos- globalfeaturesversusneuralnetworks"

KonstantinPogorelov,et.al.MultimediaToolsandApplications,2017

Performance(accuracyandspeed)


§Mayodataset(18781images/frames)§ masksforallpolyps

• GF:• recall98.50%,precision93.88%,fps~300

• CNN:• ModifiedInceptionv3:recall95.86%,precision80.78%,fps:~30• Inceptionv3+WEKA:recall:88.87%,precision:89.16%,fps:~30


ASU Mayo Dataset: Polyp Detection ”EIR- EfficientComputerAidedDiagnosisFrameworkforGastrointestinalEndoscopies"

MichaelRiegler,et.al.ProceedingsofCBMI,Bucharest,Romania,June2016

• ResourceconsumptionandprocessingperformanceofGF:

• Neuralnetworks(alsoincludingGPUsupport)?• testssofar:~30fps(sameGPUasabove)

• butaddinglayers,morenetworks,…!??(newerGPU)• Inceptionv3TFL:66fps,plainCNN:~40-45fps• GAN:~12fps(for160x160)


ASU Mayo Dataset: Polyp Detection

• Processonlyframescontainingpolyps

• Performsimageenhancement

• Detectscurve-shapedobjectsandlocalmaximums

• Buildsenergymapandselects4possiblelocations

• Localizationperformance:• recall31.83%,• precision32.07%• ~30fps

• laterbetterGPU:~75fps(detection:300fps;localization100fps)


ASU Mayo Dataset: First Try for Polyp Localization

• Vestre Viken (VV)multi-diseasedataset(250imagesperclass)

• GF:• recall90.60%• precision91.40%• fps~30

• CNN:• recall:87.20%• precision:87.90%• fps:~30


VV Dataset: Multi-Disease Detection""Efficientdiseasedetectioningastrointestinalvideos- globalfeaturesversusneuralnetworks"


• GF

• CNN


VV Dataset: Multi-Disease Detection""Efficientdiseasedetectioningastrointestinalvideos- globalfeaturesversusneuralnetworks"


• 7differentalgorithms• Convolutionalneuralnetworks(CNN)(2)– trainedfromscratch

• 3-layers• 6-layers

• Transferlearning(1)– retrainedInceptionv3• Globalfeatures(4)

• 2globalfeatures(JCD,Tamura)• 6globalfeatures(JCD,Tamura,ColorLayout,EdgeHistogram,AutoColorCorrelogram andPHOG)• 2differentalgorithms(Randomforestandlogisticmodeltree)

• 2baselines• RandomForrestwithoneglobalfeature• Majorityclass

• 2-foldedcrossvalidation


Kvasir Dataset v1: Multi-Disease Detection





Dyed

and

Lifted

Polyp

Dyed

Resectio

nMargin



Cecum

Pylorus

• UsingsameGFandsomenewdeepfeatures,i.e.,• Pre-trainedImageNet datasetInceptionv3• ResNet50models

• UseddifferentMLclassifications;• randomtree(RT)• randomforest(RF)• logisticmodeltree(LMR)– performedbest

• Usesweightsof1000pre-definedconceptsasfeatures

• Toplayerinputasfeaturesvector(16384forInceptionv3and2048forResNet50)


Kvasir Dataset v1 à v2: Multi-Disease Detection

Pretrainedmodel

Outputortop-layerinputweights

WEKAforclassification

78

Team Approaches F1 FPS

SCL-UMD Global-features anddeep-features extraction,Inception-V3and VGGNet CNNmodels,followedby

machine-learning-basedclassificationusingRT,RF,SVMandLMR classifiers

0.848 1.3

FAST-NU-DS Global andlocalfeaturescombinedfollowedbydatasizereductionbyapplyingK-means clusteringandthanusing logisticregression model fortheclassification

0.767 2.3

ITEC-AAU TwodifferentcustomInception-likeCNNmodels 0.755 1.4

HKBU Amanifoldlearningmethod(bidirectionalmarginalFisheranalysis)learningacompactrepresentationofthedata,thenmachine-learning-basedmulti-classsupport

vectormachineisusedfortheclassification

0.703 2.2

SIMULA GF-featuresextraction,ResNet50 and Inception-V3CNNmodels andfollowedbymachine-learning-basedclassificationusingRT,RFandLMR classifiers

0.826 46.0

• 7differentalgorithms• Convolutionalneuralnetworks(CNN)(2)– trainedfromscratch

• 3-layers• 6-layers

• Transferlearning(1)– retrainedInceptionv3• Globalfeatures(4)

• 2globalfeatures(JCD,Tamura)• 6globalfeatures

(JCD,Tamura,ColorLayout,EdgeHistogram,AutoColorCorrelogram andPHOG)• 2differentalgorithms(Randomforestandlogisticmodeltree)

• 2baselines• RandomForrestwithoneglobalfeature• Majorityclass

• 2-foldedcrossvalidation


Nerthus Dataset: Bowel Cleanness Level





• Toolittledata• Blurryimagesduetocameramotion• Objectstooclosetocamera• Underoroverscenelighting• Flares• Artificialobjectsandnatural“contaminations”• Lowresolutionofcapsularendoscopes• …


Data Challenges: Preprocessing


Data Enhancements for CNN Training


Data Enhancements for CNN Training

DetectionFeedback



Detection Subsystem Outputs

• Visualizetheoutputofthesystemtothemedicaldoctors• Simpleandeasytounderstand• Livesupport• Useableforautomaticreports,etc.

• Polyps• Input:CameraorVideofiles

• Output:LivestreamandPerformancereports

• FullHD• Real-time:30FPS


Real-time Detection Feedback

So,allproblemssolved!!??


• Improvedetection,localizationandsystemperformance(retrieval,machinelearning,features,search,real-time,distributedcomputing,scale,visualization,neuralnetworks,userinteraction,objecttracking,…)

1. Exploitingdomainexpertknowledge– builddatasets2. Integrationofvariousdata,multi-modality– newsensors3. ExplainableAI4. Automatedreportsystem5. Fullsystemintegration6. Patientcontextinformation7. Visualization,decisionsupport8. Integrationofdatafromvarioussources/systems9. Otherareasinmedicine10. …

Manymore…


Many Open Challenges…"MultimediaandMedicine:TeammatesforBetterDiseaseDetectionandSurvival"

MichaelRiegler,et.al.ProceedingsACMMM,Amsterdam,TheNetherlands,October2016

• Wehavegivenseveralcase-specificexamples,butingeneral,theyarecommonforMMIS

• Doctorswanttouseallthedataforgeneralsupport:analysis,diagnostics,reporting,teaching,statistics,similaritysearch/comparisons,…

• Currently,…• moreandmorehighqualitydataisrecorded/produced

• dataanalysismethodsare(only)promising• goodvisualizationtoolsexist,butnotused(e.g.,AR,VR,…)• sometoolsaremissing• many(other)areasproduceseparate(isolated)methods• …

• but,weneedacompleteintegratedsystem!

Ø Ourmultimediacommunityisneeded

Summary



The End…