Dr. Yu(Jade) Qian y.qian@mdx.ac.uk

Content• Introduction of MIRAGE project• Content-based 3D brain images retrieval and

visualization• Conclusion and future work• Demonstration

PART I

Introduction of MIRAGE Project

MIRAGE(Middlesex medical Image Repository with a CBIR ArchivinG Environment)

Phase 1: MIRAGE- from Creation to Archiving • Strand : Start-up Repository funded by JISC• Between Apr. 2009- Sep. 2010.• Aim: To develop a repository of medical images benefiting MSc and research students in the

immediate term and serve a wider community in the long term in providing a rich supply of medical images for data mining, to complement MU current online e-learning system OASIS+.

Phase 2: MIRAGE 2011 – from Archiving to Creation• Strand : Take-up and Embedding funded by JISC• Between Feb. 2011- Oct. 2011.• Aim: To enrich the current repository MIRAGE with two necessities of ‘3D Viewer’ and

‘Uploading’ to meet users’ needs, leading to a sustainable, usable and flexible model of data management.

Framework for MRIAGE I &II

Phase 1: MIRAGE – Online System(1)

• Server side: Image collection: Accommodating 100,000 2D images and 100 3D

images Visual feature extraction: Pre-processing off-line using C++ and Perl. Indexing file creation

• Client side: Interface based on PHP generate dynamic web pages.

• Client – Server communication protocol: MRML: a XML based protocol

Phase 1: MIRAGE – Online System(2)Interface

(1) Home page (2) Query and retrieval results

Phase 2: MIRAGE2011 – Online System(1)Image Uploading

MIRAGE MIRAGE2011

Phase 2: MIRAGE2011 – Online System(2)3D Viewer

MontageMIRAGE MIRAGE 2011

PART II

Content-based 3D Brain Images Retrieval and Visualization

Y. Qian, X. Gao , M. Loomes, R. Comley, B. Barn, R. Hui, Z. Tian, Content-based Retrieval of 3D Medical Images, eTELEMED 2011, February, 2011.

(Best paper award, has been invited to be extended to a journal paper).

2D brain images ----- 3D Brain

Shape-based Surface of a 3D object(e.g. tumor) Texture-based Inside of a 3D object( e.g.textures representing tissue structure properties)

Aim: To develop a fast texture-based 3D brain retrieval method

CBIR for 3D Brain Image ------Introduction

CBIR for 3D Brain Image ---Methodology(1)Proposed Framework

CBIR for 3D Brain Image ---Methodology(2)Pre-processing

1) Spatial Normalization---Statistical Parametric Mapping (SPM5) Transform each individual brain into a standard brain template

2) Divide 3D brain into 64 non-overlapping equally sized blocks

CBIR for 3D Brain Image ---Methodology(3)Extraction of Volumetric Texture Features

1) 3D Grey Level Co-occurrence Matrices (3D GLCM)

2) 3D Wavelet Transform (3D WT)3) 3D Gabor Transform (3D GT)4) 3D Local Binary Pattern (3D LBP)

Extraction of Volumetric Textures (1) ------ 3D Grey Level Co-occurrence Matrices (3D GLCM)

3D GLCM is two dimensional matrices of the joint probability p(i,j) of occurrence of a pair of gray values (i,j) separated by a displacement d = (dx,dy,dz).

Formula:

Feature: 52 Displacement vectors: 4 distance * 13 direction = 52 4 Haralick texture features: energy, entropy, contrast and homogeneity Feature vector: 208 components (=4 (features) * 52 (matrices)).

13 directions

Extraction of Volumetric Textures (2) ------ 3D Wavelet Transform (3D WT)

3D WT provides a spatial and frequency representation of a volumetric image.

Two scale 3D Wavelet Transform:

Feature:

Mean and Standard deviation Feature vector: 30 components (2 (features) +15 (sub-bands))

Extraction of Volumetric Textures (3) ------ 3D Gabor Transform (3D GT)

A set of 3D Gabor filters:

Gabor Transform:

Feature: 144 Gabor filters 4 (F) *6(θ)*6(Φ) =144 Mean and Standard deviation Feature vector: 288 components (2 (features) +144(filters))

zFyFxFjzyxgFzyxg cossinsincossin2exp,,,,,,, ^

iiii FzyxgzyxfGT ,,,,,*,,144...3,2,1i

Extraction of Volumetric Textures (4) ------ 3D Local Binary Pattern (3D LBP)

Local binary pattern(LBP) is a set of binary code Ci to define texture in a local neighborhood (p,r). A histogram Hi is then generated to calculate the occurrences of different binary patterns.

LBP on three orthogonal planes (LBP-TOP), i.e., XY, XZ, and YZ planes, expressed as

Feature: 59 binary patterns Feature vector: 177 components (=59(patterns)*3(planes)

CBIR for 3D Brain Image ---Methodology(4)Retrieval ---Similarity Measurement

Histogram Intersection(3D LBP)

Normalized Euclidean distance (3D GLCM,3D WT,3D GT)

i

ii IQIQD ,min,

2

,

i i

ii IQIQD

CBIR for 3D Brain Image ---Methodology(5) Lesion Detection

Assume bilateral symmetry of a normal brain along its mid-plane

Evaluation ---- Test Dataset

100 MR brain images

Size: 256 256 44

DICOM (Digital Imaging and Communications in Medicine)

format

Collected from Neuro-imaging Centre at Beijing General

Navy Hospital, China

Experimental Results(1) ------ Lesion Detection

Experimental Results(2) -------Retrieval

Comparative results demonstrate that LBP outperforms four 3D texture methods in terms of retrieval precision and processing speed.

Experimental Results(3) -------Query time

The query time with VOI selection offers 4 times faster operation than that without.

3D Brain Visualization(1)

3D Brain Visualization(2)

CBIR for 3D Brain Image------ On-line system(1):

CBIR for 3D Brain Image------ On-line system(2):

• Server side: 100 3D brain images(DICOM format to JPG format) 3D visual feature extraction(4 methods): Off-line pre-

processing using Matlab. 3D visualization: using Matlab

• Client side: Interface based on PHP generate dynamic web pages.

PART III

Conclusion and Future Work

Conclusion for MIRAGE (Middlesex medical Image Repository with a CBIR ArchivinG Environment)

• Create Middlesex medical Image repository ( ~100000 2D images and 100 3D brain images)

• Create CBIR archiving environment for 2D and 3D medical images.

Future Work(1)------Continue working on 3D Brain Image

• Test on the larger dataset and enrich our repository

• Research on clinical purpose (EC FP7)• ------ Collaborate with Neuro-imaging Centre at Beijing

General Navy Hospital, China.

UltrasonixTABLET Ultrasound scanner

Future Work(2)------Echocardiogram Video Clip

B-mode 2D Video Clip

B-mode and M-mode Video clip

Colour Doppler Video Clip

Enrich our repositoryResearch for clinical purpose(EC FP7) ------ Collaborate with First Hospital of Tsinghua University, China.

MDX Grid Machine

Future Work(3) ------- Grid Computing

PART VI

Demonstration