Sistem de gradare a cancerului mamar prin analiza ...daniraco.free.fr/pubs/Conference_Invitee/racoceanu2008cdcsr_invitee.pdfMedical Image Analysis, Indexing and Retrieval for Assisted

09/09/2009

1

Sistem de gradare a cancerului mamar

prin analiza imaginilor histopatologice

intr-un cadru microscopic virtual

Conferinţa Diaspora in Cercetare �tiinţifica Româneasca

Bucureşti, 16-19 septembrie, 2008

Daniel RACOCEANU

CNRS - Centre National de la Recherche Scientifique, France

IPAL - IMAGE PERCEPTION, ACCESS & LANGUAGEInternational research unit – UMI CNRS 2955National University of Singapore (NUS)Université Joseph Fourier (UJF), Grenoble, FranceInstitute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR),

Singapore

http://ipal.i2r.a-star.edu.sg

IPAL – Image Perception, Access & Language

French national Research CenterUMI CNRS, NUS, UJF, I2R/A*STAR


09/09/2009

2

3

• 2006 Dr. Habil. – Accreditation to Supervise Research

(Habilitation à Diriger des Recherches)Control and Computer Sciences

Keywords: Dynamic Monitoring, Artificial Intelligence, Ambient Intelligence, Dynamic Neural

Networks, Neuro-Fuzzy Systems, Fuzzy Petri Nets, E-maintenance, Diagnosis, Prognosis.

• 1997 PhD - Control and Computer Sciences University of Besançon, France (jury’s congratulations – THFJ : Très honorable avec les felicitations du jury)

Keywords: Stochastic Modeling, Markov Chains, Control, Reduction Methods, Principal

Component Analysis (PCA), Singular Perturbations

• 1993 Master of Science - Production and Control System SciencesNational Engineering School, Belfort, France (with distinction).

• 1992 Engineer’s Degree - Mechatronic Production Systems Production Syst. Dept., Politehnica University of Timisoara (TCM)

D. Racoceanu

Academic Backgound

4

D. Racoceanu

Professional Experience / Responsibilities

• Since 2008 Director of IPAL International Lab (CNRS, NUS, UJF, I2R/A*STAR)French National Research Center (CNRS)

• Since 2005 Principal Scientist, French National Research Center (CNRS)Invited Professor, National University of Singapore

• 1999 –2005 Associate Professor, Faculty of Sciences and TechnologyUniversity of Besançon, France

Principal Scientist, FEMTO-ST Institute, Besançon, France

• 1997 –1999 Project Manager, General Electric Energy Products – Europe

• 1993 –1996 Lecturer - Dept. of Production System Management, University Institute of Technology of Belfort, University of Franche-Comté, France

09/09/2009

3

Daniel RACOCEANU

Recent research topics / projects

• IPAL research axis : � Medical Image Analysis, Indexing and Retrieval for Assisted Diagnosis

• Projects� ONCO-MEDIA (ICT ASIA – CNRS/MAE) - wwww.onco-media.com

� MMedWeb (A*STAR/SERC)

• Research Topics� Application

� Breast Cancer Grading (NUH) -MMedWeb

� Early Detection of Stroke from brain CT (SGH) - ONCO-MEDIA

� Analysis of MR images of patients withParkinson disease

� Methodology� Content-Based Information Retrieval

(CBIR) versus Image/Case Base Reasoning (CBR)

� Area of expertise� Soft computing approaches (neuro-fuzzy,

bayesian, fuzzy logic) � Computer vision

Automatic Breast Cancer Grading

Early Detection of Stroke from brain CT

Image Perception, Access & Language (CNRS, I2R/A*STAR, NUS, UJF)

6

• CB(M)IR general overview

• Challenges & open issues

• Our solution

• Introduction. CBIR in Medical Applications

• Breast Cancer Grading

• Semantic Indexing Approach

• Experiments & Results

• Conclusions

• CBIR framework

User query

Semantic Extraction/Interpretation

Stxt Simg Simg Svid…

Text Imagemodality1

Imagemodalityn

Stream…

Structural Extraction/Interpretation

Ltxt Limg Limg Lvid…

Categorization

Primitive Extraction

PEtxt PEimg PEimg PEvid…

Medical Knowledge

Semantic (Ontology)

Features(Segmentation)

Structure (Objects)

INDEXING

Mining Extraction

trend

High-Level Fusion

QPQuery processing

( Retrieval )

Relevance feedback

09/09/2009

4

Content-Based Medical Image Retrieval

(CBMIR)

Query by image - example

CBMIR : retrieval example

09/09/2009

5

Retrieval : image information

CBMIR versus PACS and DICOM

CBMIR

(Similarity based medical images/cases access )

PACS(Picture Archiving and Communication Systems )

DICOM(Digital Imaging and Communications in Medicine)

Computer AssistedDetection/Diagnostic/Prognostic

MedicalEducation Medical

Research

09/09/2009

6

CBMIR and related fields

Medical Image Analysis

Medical Informatics

InformationRetrieval

Artificial Intelligence and Pattern Recognition

Knowledge Management

CBMIR

CBMIR - Applications

• Medical assistance� Quantification support for medical diagnosis / treatment� Similarity–based retrieval for detection / diagnostic / prognostic

and treatment assistance� Improve the patient healthcare using medical metadata

management and case-based similarity

• Education support� Medical image contextual browsing to improve the understanding of a

medical pathology and the related diagnosis/treatment issues� Visual-similarity- and symptom- based training of the students� Automatic assistance for building medical multimedia atlases

• Medical research � Medical image mining

• Extract and explore new pathological trends,

• Extract new correlations/co-occurences between different aspects and influence parameters of a pathology

• …

09/09/2009

7

Knowledge-based diagnosis assistance using

Medical Cases and A Priori Medical Knowledge

Add new features

Learning Algorithm

Add new rules

Medical Image

Knowledge guidedsegmentation

Relevant features (ROI) extraction

A priori rules(rule based system)

Rules propagation

Diagnosis / PrognosisAssistance

Medical Doctor, Radiologist, Pathologist, …

Incrementallearning

Flexible segmentation

algorithmsTraceability,

Management of uncertainties

Scientific challenges

• Incremental learning• Decision/Diagnosis traceability• Management of uncertainties in Diagnosis process• Flexible segmentation algorithms

Flexible segmentation

algorithms

Incrementallearning

Traceability,Management of

uncertainties

09/09/2009

8

Clinical scope

• Solve REAL CHALLENGING clinical problems� Early detection of the stroke from brain CT� Breast cancer grading� Ultrasound guided biopsy

Real Microscope Platform

• Available devices :� Olympus microscope with 4x to 100x magnifications

� prior x/y motorized stage and a Z focus

� MediaCybernetics Mega pixel digital camera

• Microscope stage computer control � traveling area,

� pattern selection,

� magnification …

• Camera parameters � white balance,

� exposure,

� background and Z axe focus …

• Stitching & blending slices for composite image generation

09/09/2009

9

Content based image retrieval

for Microscopic Medical images

• The images can be chosen from three different pathology database

• The retrieved images are indexed according to the similarity coefficient

19

Micromedical Image Analysis using a

Virtual Microscope Platform

• Actual approach : image query

• Next approach: domain specific semantic query

09/09/2009

10

21

• CB(M)IR general overview


• Our solution





• Conclusions

• Translational CBIR

� translate CBIR theoretical ideas into clinical practice

� apply CBIR advantages to medical domain

Knowledge-guided Semantic indexing (for Breast Cancer Grading)

Content-Based Medical Image Retrieval - CBMIR

22

• CBMIR general overview


• Our solution

• Our Paradigm framework





• Conclusions

Pathologist

Query, Edition, Validation

09/09/2009

11

23

• Subjective Knowledge

• Objective Knowledge

• BREAST CANCER GRADING – the Diagnosis Golden Standard

Pathologists – A priori domain knowledge





• Conclusions

a) Tubule formation b) Mitosis c) Nuclear pleomorphism

24

• Subjective Knowledge

• Objective Knowledge

Criteria ScoreHyperfield (Frame) Score

TF NPS MC

1 >75% tubules Small size and regular shape < 9

2 10 - 75% tubules Medium size and variated shape 10-19

3 <10% tubules Big size and irregular shape > 19

Composite score /10 frames Global score (TF+NPS+MC)

Grade I (well differentiated) 3- 5

Grade II (moderately differentiated) 6-7

Grade III (poorly differentiated) 8-9

• Nottingham Grading System (NGS) – the gold standard





• Conclusions

09/09/2009

12

25

• BCG Ontology Design

Medical Concepts CV concepts Protégé concepts type

Grading Grading Super class

Cells/CellsCluster Cells/Union of Cells Class inherited (mage) –relations

DarkCellsCluster/VeryDarkCellsCluster

Union of Cells Class inherited (Cells) - relations

hasIntensity (attributes-property)

Dark/VeryDark (instances of Intensity class)

Lumina White compact segments ofthe Image included in union ofdark Cells

Class inherited (WhiteBlobs)

hasIntensity (property) White (instance of Intensity class),

hasSize (property) Small (instance Size class),

hasLocalization (property) Included_In (instance of Localization class)

DarkCellsCluster (instance of Cells)

TF/Mitosis/NP Union of Cells/ Dividing Cellsnuclei/ dimension & shapefeatures of nuclei

Classes inherited (Grading)- relations

Local Grading/GlobalGrading(10 HPFs)

TF, MC, NP Grading computation single/ten images

Class inherited (TubuleFormation/MC/NPS)- relations

• BCG Ontology Representation

• BCG Rule-Based System Modeling

• Semantic Indexing Scheme





• Conclusions

• BCG Ontology Design using PROTÉGÉ








• Conclusions

09/09/2009

13

27

• BCG Ontology Design – OWLviz hierarchy format








• Conclusions

29

• Medical Knowledge (MK) - Computer Vision (CV)

Generic Translator Framework (GTF)

Medicalconcepts

Medical rules

CVconcepts

CV symbolicrules

CV intermediateSymbolic rules

Medical Knowledge-Guided Semi-automated BCG System

Concepts Translator

Rules Translator








• Conclusions

09/09/2009

14

30

� objects : Cells, CellsCluster, Blobs

� attributes : size, shape, intensity, localization

� values : small, medium, big, regular, variated, irregular,

White, Dark, VeryDark, ecc

� operators :

• Phase1. MK-CV Concepts Translator








• Conclusions

31

Medical Objects CV Objects

Slide Image (digitized)

Grading Grading

Cells Cells

CellsCluster Union of Cells

DarkCellsCluster/VeryDarkCellsCluster

Union of Cells

Lumina White compact segments of the Image included in union of dark Cells

TF/Mitosis/NP Union of Cells/ Dividing Cells nuclei/ dimension & shape features ofnuclei

Local Grading/Global Grading (10 HPFs)

Grading computation for TF, MC,NPfor a single image/ten images

• Phase1. MK-CV Concepts Translator








• Conclusions

09/09/2009

15

32

� Mitosis Pathologist Rule :

very dark diving cells nuclei from the peripheral area of neoplasm

� Mitosis CV symbolic Rule :

� Mitosis CV intermediate symbolic Rule :

• Phase2. MK-CV Rules Translator








• Conclusions

33

• Formal language theory applied to BCG rules








• Conclusions

A context free grammar:

� Alphabet Σ= {Cells, Lumina, Tubule, Mitosis, Nuclei, Blobs, etc}

� Formal language α = {DarkCells, VeryDarkCells, WhiteBlobs, TFROI, NPROI, MROI, etc}

� Productions (Definition Rules) :

Cells - > CellsClusterCellsCluster -> DarkCellsCluster| VeryDarkCellsClusterDarkCellsCluster-> NPROI| TFROIVeryDarkCellsCluster->MROI

09/09/2009

16

10x Exam

40x Exam High Power Fields

(HPF)

TF segmentation NP computation M segmentation

Cells segmentation

TF indexing / F NP indexing / F Frame MC

Epithelium and Neoplasm localization

FNPS

Average 10 HPF

NPS MCTFS

BCG

Fat Cells zone localization

Stroma zone localization

DCIS

Abnormal Tubule

FormationInvasive

40x Exam

Average

NPS

Average Average

Epithelium localization

Neoplasm localization

• Knowledge-based Multi-scale Decision Tree








• Conclusions

• Multi-scale Image Analysis








• Conclusions

09/09/2009

17

36

• Semantic Indexing in BCG

MitosisROI

TubuleFormationROI

NucleiROI

FNPS

FTFS

FMC FBCG

TopTen

Hyperfields

Original

Image

BCG

Semantic Indexing Local and global

grading computation

Translation of the

domain knowledge

rules in symbolic rules

and image analysis

procedures








• Conclusions

37

• Data set (first prototype)

� 6 breast cancer core-biopsy cases/slides with 7000 frames

� 1400 frames learning phase

� 5600 frames testing & validation phase

� 10X40 (400X) magnification

� 1080 X1024 resolution

• Prerequisites

• Comparative Results Analysis





• Conclusions

09/09/2009

18

38

Data type Case ID Tubule score Nuclear score Mitosis count BCG (path)

Training database(1400 frames)

1000 1 1 3 1

2000 1 2 1 1

4895 3 3 3 3

Testing database(5600 frames)

5020 2 3 3 3

5042 3 3 2 3

5075 3 2 1 2

Data type Case ID Tubule score Nuclear score Mitosis count automaticBCG

Training database(1400 frames)

1000 1 1 3 1

2000 2 2 1 1

4895 3 2 3 3

Testing database(5600 frames)

5020 3 2 3 3

5042 3 2 3 3

5075 3 2 1 2

Automatic grading results

Pathologic visual grading, configuration of training & testing database

• Prerequisites

• Comparative Results Analysis





• Conclusions

39

• Key ideas of our novel approach

� translational CBIR for BCG

�semantic indexing to bridge the semantic gap

� BCG ontology – modeling and validation (Protégé OWL-DL, Pellet)

� CV concepts & rules generation from MK concepts & rules

� 80% accuracy by semi-automated grading

• Contributions

• Future Perspectives





• Conclusions

09/09/2009

19

40

research, teaching

&

bona-fide

prognosis assistance

� multi-scale image processing & analysis

� automated image annotation using BCG ontology

� visual positioning

� Query by Semantic Example (QBSE)

� ontology segmentation, medical & OBO validation

� clinical assessment & treatment

• Focus on

• Contributions

• Future perspectives





• Conclusions

Semantic query Similarity research• Semantic indexing / retrieval in a virtual microscope environnement for BCG and diagnosis assistance

• Contributions

• Future perspectives





• Conclusions

Tubule formation score

Mitosis count

Nuclear pleomorphism

score

Frame grading

Individual frame

Semantic indexing

Pathologist

Semantic query Similarity research

“ Show me the frames with the most

mitotic cells”

Medical rules

09/09/2009

20

42

• ONCO-MEDIA project

(ONtology and COntext related MEdical image Distributed Intelligent Access)

ICT Asia International project, www.onco-media.com

• Thomas PUTTI, MD. , Teh MING, MD. � Pathology Department, National University Hospital of Singapore

• A/Prof. Wee-Kheng LEOW, M. Jean-Romain DALLE � School of Computing, National University of Singapore

• Adina TUTAC, Vladimir CRETU� Universitatea Politehnica Timisoara, Romania

Acknowledgement

Thank you for your attention !

43

IPAL - IMAGE PERCEPTION, ACCESS & LANGUAGE

International Mixed Research Unit French National Research Center (CNRS)National University of Singapore (NUS)Institut for Infocomm Research / A*STAR Singapore University Joseph Fourier, Grenoble, France (UJF)


Documents

Sistem de gradare a cancerului mamar prin analiza ...daniraco.free.fr/pubs/Conference_Invitee/racoceanu2008cdcsr_invitee.pdfMedical Image Analysis, Indexing and Retrieval for Assisted