AUTOMATIC SEGMENTATION OF THE LOWER LIMB ANATOMY

Lena Paelinck Student number: 01307508 Promotor: Prof. Dr. Emmanuel Audenaert Copromotor: Prof. Dr. Christophe Pattyn A dissertation submitted to Ghent University in partial fulfilment of the requirements for the degree of Master of Medicine in Medicine Academic year: 2016 – 2018

AUTOMATIC SEGMENTATION OF THE LOWER LIMB ANATOMY

Lena Paelinck Student number: 01307508 Promotor: Prof. Dr. Emmanuel Audenaert Copromotor: Prof. Dr. Christophe Pattyn A dissertation submitted to Ghent University in partial fulfilment of the requirements for the degree of Master of Medicine in Medicine Academic year: 2016 – 2018

III

ACKNOWLEDGEMENTS

After one and a half years of hard work accompanied with laughter and tears, the process of

writing this dissertation comes to an end. Many people have played a crucial role in the

accomplishment of this work and to them I want to express my gratitude.

First of all, I would like to thank the University of Ghent and the department of Orthopaedics

and Traumatology of the UZ Ghent for giving me the chance to become part of a group of

professionals that has been investigating automatic segmentation for years and for giving me

the permission to help them in this research and to write this dissertation.

Secondly, I would like to express my gratitude to my supervisors Prof. Dr. Emmanuel

Audenaert and Prof. Dr. Christophe Pattyn for their guidance. Their insightful advices and

support through these eighteen months played an essential part in accomplishing this work.

Thank you for finding the time to help me in this process of researching and writing.

A special word of thanks to my mentor Dr. Jan Van Houcke is in order. His comments and

directives were essential to establish this work. He encouraged me to put my own ideas into

this dissertation but guided me when I was heading in the wrong direction. I appreciate the

time given into answering my questions, the reading over and making improvements to my

work.

Finally, a profound expression of gratitude towards my parents, sister, family and friends. Their

everlasting support and encouragement not only in this dissertation, but through all the years

of studying medicine, mean a lot to me. Thank you for finding the time to read through this

work and improving it. Without, I would not have been able to accomplish this work. Also, I

would like to say thanks, to my boyfriend, Brent Van Riet, for giving me the strength to complete

this work, for helping me through difficult times and for always being there whenever I needed

it.

A heartfelt thank you to all!

IV

TABLE OF CONTENTS

ACKNOWLEDGEMENTS ......................................................................................................... III

TABLE OF CONTENTS ...........................................................................................................IV

LIST OF FIGURES ....................................................................................................................VI

LIST OF TABLES .....................................................................................................................VI

LIST OF ABBREVIATIONS.....................................................................................................VII

ABSTRACT ................................................................................................................................ 1

NEDERLANDSE SAMENVATTING .......................................................................................... 3

1 INTRODUCTION................................................................................................................. 5

1.1 Anatomy of the lower limb ....................................................................................... 6

1.1.1 The Pelvis or the os coxae ........................................................................................... 6

1.1.2 The Femur or the thighbone ........................................................................................ 7

1.1.3 The Tibia or the shinbone ............................................................................................ 8

1.1.4 The Fibula or the splint bone ....................................................................................... 8

1.1.5 The Patella or the knee cap ......................................................................................... 9

1.1.6 The Talus or the tarsal bone ........................................................................................ 9

1.1.7 The Calcaneus or the heel bone ................................................................................. 9

1.1.8 The Lumbar Vertebrae................................................................................................ 10

1.1.9 The sacrum ................................................................................................................... 10

1.2 Imaging technology ................................................................................................. 11

1.2.1 Introduction to Computed Tomography ................................................................... 11

1.2.2 Pixels and voxels ......................................................................................................... 13

1.3 Introduction to segmentation................................................................................. 13

1.4 (Semi-) automatic segmentation methods ........................................................... 14

1.4.1 Thresholding (global vs local thresholding) ............................................................. 15

1.4.2 Gradient based method .............................................................................................. 18

1.4.3 Shape models .............................................................................................................. 19

1.4.4 Free deformation ......................................................................................................... 21

1.5 Practical applications of image segmentation ..................................................... 23

1.6 Validation metrics .................................................................................................... 24

1.6.1 Mean error distance (ASD & RMS) .......................................................................... 24

1.6.2 Maximum fault (HD) .................................................................................................... 24

V

1.6.3 Dice Similarity Index.................................................................................................... 25

2 MATERIALS AND METHODOLOGY .............................................................................. 27

2.1 Manual segmentation using Mimics...................................................................... 27

2.2 Construction of the SSM......................................................................................... 28

2.3 The pipeline of the automatic segmentation of the full lower limb ................... 29

2.3.1 Initialisation and segmentation .................................................................................. 29

2.3.2 Supervised training of the shape models and continuous updating of the

segmentation pipeline ................................................................................................................. 31

2.3.3 Solids and appearance ............................................................................................... 31

2.4 Used validation metrics .......................................................................................... 33

2.4.1 Mean error distance (ASD & RMS) .......................................................................... 33

2.4.2 Maximum fault (HD) .................................................................................................... 34

2.5 Inter- and intra-observer differences .................................................................... 34

3 RESULTS.......................................................................................................................... 35

4 DISCUSSION .................................................................................................................... 38

4.1 General ...................................................................................................................... 38

4.2 Comparison with previously developed programs ............................................. 39

4.3 Limitations ................................................................................................................ 44

4.4 Suggestions for further research .......................................................................... 44

5 CONCLUSION .................................................................................................................. 46

6 REFERENCES.................................................................................................................. 47

VI

LIST OF FIGURES

Figure 1: Anatomy of the pelvis ............................................................................................................................... 7 Figure 2: A) Anatomy of the Femur B) Anatomy of the Tibia C) Anatomy of the Fibula ......................................... 8 Figure 3: Anatomy of the Patella .............................................................................................................................. 9 Figure 4: A) Anatomy of the Talus B) Anatomy of the Calcaneus ............................................................................ 9 Figure 5: A) Anatomy of the Lumbar Vertebra B) Anatomy of the Sacrum ........................................................... 10 Figure 6: The Hounsfield scale................................................................................................................................ 12 Figure 7: Illustration of pixels ................................................................................................................................. 13 Figure 8: Common drawbacks on manual and automatic segmentation .............................................................. 14 Figure 9: Illustration of thresholding ...................................................................................................................... 15 Figure 10: Histogram based threshold ................................................................................................................... 16 Figure 11: Histogram of CT values .......................................................................................................................... 17 Figure 12: Illustration of gradient based method .................................................................................................. 18 Figure 13: Illustration of the SSM and the Eigenmodes ......................................................................................... 21 Figure 14: Illustration of the free deformation phase ........................................................................................... 22 Figure 15: Illustration of the Dice Similarity Index (DSI) ........................................................................................ 25 Figure 16: Average shape and appearance of the low- and high-resolution SSM ................................................. 28 Figure 17: Overview of the automatic segmentation pipeline .............................................................................. 31 Figure 18: Illustration of the correspondence feature ........................................................................................... 32 Figure 19: Illustration of solid cortical, cartilage and marrow volumes ................................................................ 32 Figure 20: Illustration of the Hausdorff Distance ................................................................................................... 34

LIST OF TABLES

Table 1: Accuracy and observer variation in segmentation of the respective segments ....... 36

Table 2: Comparison with previously developed automatic segmentation techniques .......... 40

VII

LIST OF ABBREVIATIONS

2D Two-dimensional

3D Three-dimensional

AD Average Distance

ASD Average Surface Distance

ASM Active Shape Model

CAOS Computer Aided Orthopaedic Surgeries

CAT Computed Axial Tomography

CT Computed Tomography

CTA Computed Tomography Angiography

DSI Dice Similarity Index

HD Hausdorff Distance

HT Higher Threshold

HU Hounsfield Unit

LAM Local Appearance Model

LT Lower Threshold

MR(I) Magnetic Resonance (Imaging)

OE Overlap Error

PCA Principal Component Analysis

RMS Root Mean Square

ROI Region of Interest

SD Standard Deviation

SSM Statistical Shape Model

1

ABSTRACT

Background: Image segmentation allows transforming a 2D image, obtained from CT or MRI,

into a 3D image. Hence, image segmentation has a growing interest, especially in diagnosis

and planning of surgery. It gains traction in orthopaedics as well as other specialities like

cardiology and neurology. However, image segmentation is mostly done manual - a laborious

and time-consuming task to be performed by trained professionals. The increase in the number

of segmentations results in a need for automatic segmentation techniques. The aim of this

study is to validate the accuracy of our proposed developed segmentation protocol compared

with the golden standard of manual segmentations. Secondly, the inter- and intra-observer

variability of the manual segmentation will be evaluated.

Materials and methodology: The pipeline of the proposed method for automatic

segmentation is presented. The fully automatic segmentation is divided into three phases.

First, there is an initialization and segmentation phase, separated in four steps. Afterwards,

the training of the shape model is supervised and the segmentation pipeline is continuously

updated. At last the appearance and solids phase occurs. Additionally, the method of validation

as well as the validation metrics and the inter- and intra-observer differences are explained.

Average surface distance (ASD) and Hausdorff distance (HD) are used as parameters for

validation. Inter- and intra-observer differences are reported.

Results: The mean error or the ASD, measured in 10 cases, ranges from 0,53 mm to 0,76

mm, with an average of 0,65 mm and the maximum error or the Hausdorff distance ranges

from 2,02 mm to 7,84 mm, with an average of 4,05 mm. The inter-observer variability,

measured in three cases, has the following results: an average error ranging from 0,39 mm to

0,61 mm, with an average of 0,44 mm and a maximum error ranging from 1,67 mm up to 3,74

mm, with an average of 2,29 mm. The intra-observer variability in one case was also measured

and the following results are found: an average error ranging from 0,17 mm to 0,32 mm and a

maximum error ranging from 0,78 mm till 2,29 mm.

Discussion: Several previously developed automatic segmentation techniques are compared

to ours. The results of these techniques range from 0,5 till 5,4 mm ASD. Automatic

segmentation techniques which only implement the segmentation of a part of a bone or a joint

are not included in the comparison. The limitations of our study are mentioned, such as the

small number of CT data sets included in the beginning of our statistical shape models, the

small number of manually segmented CT data sets as limitation for the validation of our

2

program and the small number of studies found to compare our results to. Finally, suggestions

for further research are described: a possibility to make an extension of our developed

automatic segmentation program to MRI, the full skeleton and other disciplines like cardiology

and neurosurgery.

Conclusion: It is important to note that this study is the first of its kind to develop an automatic

segmentation technique for the full lower limb. After comparing with previously developed

techniques, our automatic segmentation technique seems to generate highly competitive

results and, by consequence, was found to be the most accurate technique.

3

NEDERLANDSE SAMENVATTING

Achtergrond: Segmentatie laat toe om een 2D beeld, verkregen via CT of MRI, te

transformeren tot een 3D beeld. Bijgevolg is er een groeiende interesse in segmentatie, vooral

bij het stellen van diagnoses en plannen van chirurgie. Het is van toepassing in orthopedie,

maar ook in andere specialisaties zoals cardiologie en neurologie. Echter, heden ten dage

wordt segmentatie meestal manueel gedaan, wat een intensief en tijdrovend werk is, dat alleen

is weggelegd voor experten. Voorgenoemde zaken duiden op de nood aan automatische

segmentatie technieken. Het doel van deze studie is om ons nieuw ontwikkeld automatische

segmentatie programma te valideren door dit te vergelijken met de gouden standaard, in casu

de manuele segmentatie. Bijkomend worden ook de inter- en intra-observator variabiliteit van

de manuele segmentatie bestudeerd.

Materialen en methodologie: Het werkingsmechanisme van de voorgestelde methode om

automatisch te segmenteren wordt voorgesteld. Het voorgestelde automatische segmentatie

programma is onderverdeeld in drie fases. Eerst vindt de initialisatie en segmentatie plaats.

Deze eerste fase is dan nog eens onderverdeeld in 4 stappen. Daarna wordt de training van

het statistische vormmodel gesuperviseerd en de werking ervan voortdurend geüpdatet. De

laatste fase is diegene waarbij uiterlijke kenmerken aan het 3D model gegeven worden. Verder

wordt zowel de methode van valideren als de parameters die gebruikt worden voor de validatie

en de inter- en intra-observator verschillen bekeken. The gemiddelde oppervlakte afstand en

de maximale afstand worden gebruikt als parameters voor validatie. De inter- en intra-

observator verschillen worden beschreven.

Resultaten: De gemiddelde fout of de gemiddelde oppervlakte afstand, bestudeerd bij 10

patiënten, varieert van 0,53 mm tot 0,76 mm met een gemiddelde van 0,65 mm. De maximum

fout of de HD varieert van 2,02 mm tot 7,84 mm met een gemiddelde van 4,05 mm. De inter-

observator variabiliteit werd berekend bij 3 patiënten en gaf volgende resultaten: een

gemiddelde fout die varieert van 0,39 mm tot 0,61 mm, met een gemiddelde van 0,44 mm en

een maximum fout die varieert van 1,67 mm tot 3,74 mm met een gemiddelde van 2,29 mm.

De intra-observator variabiliteit in 1 CT data set werd berekend en volgende resultaten werden

bekomen: een gemiddelde fout variërend van 0,17 mm tot 0,32 mm en een maximum fout die

varieert van 0,78 mm tot 2,29 mm.

Discussie: Verschillende automatische segmentatie technieken, ontwikkeld door andere

onderzoeksgroepen, werden vergeleken met de onze. Het resultaat van deze technieken

4

varieerde van 0,5 tot 5,4 mm ASD. Studies waarbij de automatische segmentatie techniek zich

toelegde op een deel van een bot of op een gewricht werden uitgesloten uit deze vergelijking.

Verder worden ook de beperkingen van onze studie besproken, zoals o.a. het kleine aantal

CT data sets die opgenomen zijn in het begin van ons statistisch vormmodel, het kleine aantal

manuele segmentaties, die een limitatie vormen voor de validatie en het kleine aantal

gevonden studies waarvan de resultaten vergeleken kunnen worden met de onze. Als laatste

werden de mogelijkheden voor verder onderzoek beschreven. Zo is er onder andere een

mogelijkheid tot uitbreiding van deze techniek tot MRI, het volledige skelet en andere

specialisaties zoals cardiologie en neurochirurgie.

Conclusie: Het is belangrijk te vermelden dat deze studie de eerste is die een automatische

segmentatie techniek ontwikkelt voor het volledige onderste lidmaat. Na de vergelijking met

eerder ontwikkelde technieken, bleek deze studie sterk concurrerende resultaten te vertonen

en daarbij dus ook over de meest accurate automatische segmentatie techniek te beschikken.

5

1 INTRODUCTION

Ever since the Ancient World, several physicians studied the human body in multiple ways.

Hippocrates and Vesalius are just a few examples of famous physicians who had an enormous

impact on medicine from then and now (1,2). The skeleton, the organs and the details of the

human body have been long known, studied and illustrated before modern medicine existed.

An important difference between medicine from then and now is the way we study the human

body. Before, the human body was studied on corpses or during an operation. Nowadays with

the help of medical imaging, the human body can be studied without the need for invasive

actions.

Even though modern technology made it much easier to study the human body, Computed

Tomography (CT) and Magnetic Resonance (MR) scanners still depict it in two-dimensional

(2D) images. In order to allow for three-dimensional (3D) evaluation of the anatomical

structures, segmentation is required.

Image segmentation is the process of segmenting or partitioning a digital image into multiple

segments or in a series of pixels with the aim to change the representation of the image into

something that gives more information and is easier to scrutinize (3). It converts a complex

image to a simple image that is easy to interpret (4). In specific, image segmentation is the

process of labelling every pixel in the image so that pixels with the same label share the same

characteristics (5). In plain words, segmentation as described in this paper is the extraction of

the part one wants to see in 3D, e.g. the skin, the heart, bones, etc., from a set of 2D slices. It

is a mainly manual technique that involves selecting the region of interest in each 2D image

followed by computing the 3D shape.

However, manual segmentation is a laborious and time-consuming task that must be

performed by trained professionals with the right anatomical knowledge. It is estimated that

the segmentation of a full skeletal lower limb takes around 100 man hours (6). If 3D evaluation

of anatomy wants to reach more physicians, then automatic segmentation techniques need to

be developed.

Segmentation of individual anatomy is used to study the anatomic variation of structures.

Furthermore, it allows for running personalized computational models in mechanical simulation

studies.

6

Up till now, various semi- to automatic segmentation routines have been described in literature.

The average surface distance (ASD) ranges from 0,5 mm up to 5,4 mm. Hence, there is room

for improvement (7–13).

The aim of this thesis is to validate the accuracy of our newly developed segmentation protocol

and to compare this to the golden standard, manual segmentations, and previously developed

automatic segmentation techniques. Additionally, the inter- and intra-observer variability of the

manual segmentation will be evaluated. The current protocol is hypothesized to be superior to

previous techniques. Furthermore, the average of the inter- and intra-observer manual

segmentations are expected to be in the same range.

In light of this study, I performed the majority of the manual segmentations and performed the

statistical analysis comparing manual to automatic segmentation as well as the intra and inter

observer variability.

1.1 Anatomy of the lower limb

The bones of the human skeleton included in this study are the pelvic bones, the femur, the

tibia, the fibula, the patella, the calcaneus, the talus, lumbar vertebrae and the sacrum. In this

section, the anatomy of these bones is described extensively and the joint they take part in is

mentioned (14–18).

1.1.1 The Pelvis or the os coxae (Figure 1)

The pelvis consists of three parts: the os pubis or the pubic bone, the os ilium and the os ischii

or the ischial bone. In childhood, these three bones are connected by cartilage and by the age

of 15-17 the fusion begins. From the age of 20 to 25 the fusion is completed and together they

form an acetabulum, which is located at the lateral side of the hip bone. This is the cavity where

the femur (the thighbone) will fit in the pelvis. The acetabulum and the femur head form the hip

joint. The biggest bone in the pelvis is the os ilium. It is located at the top and the backside of

the pelvis and consists of two parts: the ala ossis ilii and the corpus ossis illii. The latter is the

part of the ilium that fusions with the other two bones. The second bone is the os pubis,

consisting of the corpus ossis pubis, the ramus superior ossis pubis and the ramus inferior

ossis pubis. The two latter form the foramen obturatum and again the corpus part of the os

pubis is the one that fuses with the os ilium and the os ischii. At last, we have the os ischii,

which consists of the corpus ossis ischii (the part that fusions) and the ramus ossis ischii. The

7

ramus forms the lower part of the foramen obturatum. Formed by the os pubis and the os ischii,

the foramen obturatum is the space where the obturator nerve, vein and artery pass.

As mentioned above, these three bones form the hip bone and each hip consists of two hip

bones which are connected by fibrocartilage at the symphysis pubica.

1.1.2 The Femur or the thighbone (Figure 2)

The femur consists of a proximal and a distal extremity, called the proximal and distal

epiphysis, and in between the corpus femoris or the diaphysis is located. At the upper or

proximal epiphysis, we find the caput femoris, which is the part that will rotate in the acetabulum

of the hip bone. The caput contains a depression which is called the fovea capitis femoris. This

fovea does not take part in the joint but serves as an attachment place for the ligamentum

teres. Between the caput and corpus femoris we find the collum femoris, which is also called

the neck of the femur. At the sides, we find two bulgings, the trochanter major at the lateral

side, and the trochanter minor at the medial-back side. These trochanters major and minor are

the insertion point of multiple muscles. At the lower or distal epiphysis is a surface, the facies

patellaris, where the patella touches the femur. At the lower end, we find 2 surfaces, the

epicondylus medialis and the epicondylus lateralis, where the tibia touches the femur. The

femur, the tibia and the patella form the knee joint.

Figure 1: Anatomy of the pelvis (18) (from Sobotta Atlas of Human Anatomy)

8

1.1.3 The Tibia or the shinbone (Figure 2)

The tibia has the same structure as the femur, it consists of a proximal and distal epiphysis

and a diaphysis or corpus tibiae which has a triangular structure. The proximal epiphysis has

a condylus medialis and a condylus lateralis, where the tibia touches the femur. It also has a

tuberositas tibia, which is a bulging for the insertion of the ligamentum patellae. The distal

epiphysis has an incisura fibularis, where the fibula touches the tibia, and a malleolus medialis,

which is also called the inner ankle.

1.1.4 The Fibula or the splint bone (Figure 2)

The fibula has the same structure as the other long bones like the femur and the tibia and

consists of a diaphysis and a proximal and distal epiphysis. The proximal epiphysis is the caput

fibulae, which has a protrusion called the apex capitis fibulae. This caput fibulae is connected

to the diaphysis by the collum fibulae. Like the tibia, the corpus of the fibula has a triangular

structure. At the distal epiphysis, there is again a bulging, called the malleolus lateralis or the

outer ankle.

Figure 2: A) Anatomy of the Femur B) Anatomy of the Tibia C) Anatomy of the Fibula (18)

(from Sobotta Atlas of Human Anatomy)

9

1.1.5 The Patella or the knee cap (Figure 3)

The patella is a little bone that has a role in the knee

joint. The other bones that form the knee joint are the

femur and the tibia. The patella has a heart shaped

structure and is located with the point (the apex

patella) downwards or distal. The basis patella is

located upwards or proximal. The dorsal side

articulates with the femur and the anterior side (the

facies anterior) is touchable through the skin.

1.1.6 The Talus or the tarsal bone (Figure 4)

The talus is the most important bone of the foot, since it carries the weight of the whole body.

It has a caput tali (the head) and a corpus tali (the body) are connected by a collum tali ( the

neck). On the corpus tali, there is a trochlea tali with a facies superior, a facies malleolaris

lateralis and a facies malleolaris medialis. The facies superior is where the talus touches the

tibia, the facies malleolaris lateralis touches the fibula and the facies malleolaris medialis

touches the tibia. Underneath, there is a facies articularis calcanea anterior and posterior,

where the talus touches the calcaneus.

1.1.7 The Calcaneus or the heel bone (Figure 4)

The calcaneus is the biggest bone of the foot. At the back, there is a tuber calcanei, also called

the heel and is the place where the Achilles tendon inserts. At the top (the cranial side), there

are three facies articularis, namely the facies articularis talaris anterior, media and posterior.

These three are the connection between the talus and the calcaneus.

Figure 3: Anatomy of the Patella (18) (from

Sobotta Atlas of Human Anatomy)

Figure 4: A) Anatomy of the Talus B) Anatomy of the Calcaneus (18) (from Sobotta Atlas of

Human Anatomy)

10

1.1.8 The Lumbar Vertebrae (Figure 5)

The lumbar vertebrae are a part of the vertebral column which consists of seven cervical

vertebrae, twelve thoracic vertebrae, five lumbar vertebrae, five sacral vertebrae (the sacrum)

and four or five coccyx vertebrae (the coccyx). The corpus of the lumbar vertebra is much

bigger than the corpora of the other vertebrae. The processus spinosus points to caudal. The

processi costales are the protrusions on the side and are typical for the lumbar vertebrae. After

the processus costales, the processus accessorius is located and together with the processus

mamillaris they are a remnant of the processus transversus. The pediculus and lamina arcus

vertebrae form the arcus of the lumbar vertebrae. In the central of this arcus there is a foramen

vertebrale for the spinal cord.

1.1.9 The sacrum (Figure 5)

The sacrum is the consequence of the fusion between the five sacral vertebrae and it consists

of 2 sides, the frontal concave side or the facies pelvina, and the dorsal convex side or the

facies dorsalis. The fifth lumbar vertebra leans on the only visible corpus vertebrae of the upper

sacral vertebra, the basis ossis sacri and on the ventral side also called the promontorium.

The apex ossis sacri touches the os coccyx. In the facies pelvina there are 4 geminated

foramina sacralia anterior (foramina sacralia posterior on the dorsal side). These foramina

sacralia are the apertures for the nervi spinales. Between the pairs of foramina sacralia anterior

there are lineae transversae, originated from the fusion of the sacral vertebrae.

Figure 5: A) Anatomy of the Lumbar Vertebra B) Anatomy of the Sacrum (18) (from Sobotta Atlas of Human Anatomy)

11

1.2 Imaging technology

1.2.1 Introduction to Computed Tomography

To study these bones without using an invasive procedure, there is a need for imaging

technology of which the purpose is to make a scan or copy of the human body in 2D. On this

scan, dependent on the type of scanning machine, one can see different structures of the body

like bones, muscles and/or organs. The two well-known scanning machines are the Computed

Tomography (CT) and the Magnetic Resonance Imaging (MRI). The CT scan is the only one

included in this study and will be explained shortly.

The CT (Computed Tomography) or CAT (Computed Axial Tomography) uses, just like the X-

ray radiography, X-radiation for making a copy of the human body. The X ray tube emits the X

rays and the detector, located on the other side of the patient and the X ray tube, receives the

radiation that goes through the body. Hence, the part of the body that is to be visualized, needs

to be located between the X ray tube and the detector. But instead of using one stationary X-

ray tube like X-ray radiography, the CT uses an x-ray tube that circles around the patient.

Equal to X-ray radiography, the CT scan makes a reconstruction of the density of structures in

the human body. The principle is that the radiation can go through some structures and while

others stop the radiation. It is comparable with visual light: visual light is also an

electromagnetic radiation, but contrary to X-rays it is visible and it cannot pass any tissue of

the human body. When you hold your hand towards the light, there is shadow on the floor. The

same applies for X-rays. Dense or radio-opaque structures like bone and metal (e.g.

pacemaker, artificial knee, etc.) stop the radiation, so the detector does not receive any

radiation. Bones and metal create a shadow on the detector and that is why bones and metal

appear white on the 2D image. Other low density or radiolucent structures such as air let the

radiation pass through the body what makes it appear black on the 2D image. Structures like

muscles, blood and fat are a shade of grey as they let a small amount of radiation pass through.

In contrast to the X-ray radiography, the CT scan detector gets more information from the

reception of X-rays. When the X-ray tube circles around the patient, it gives 3D information

(information in the depth) to the detector. Then it is the task of the computer algorithm to

transform this 3D information into a 2D image.

12

The attenuation coefficient is the measurement of how much a tissue can be penetrated by an

X-ray beam, i.e. it measures how much strength the beam has lost while passing through the

body. The intensity of the X-ray beam that went through the patient, is lower in dense or radio-

opaque structures. The Hounsfield scale is an alternative for the attenuation coefficient and is

expressed in Hounsfield Units (HU), where water is given a HU of 0 and air a HU of -1000.

The HU of water and air serve as reference values to which all other tissues are compared.

Bone has a HU of + 400 up to +3000. All the other tissues have a HU between – 1000 HU and

+ 1000 (19) (Figure 6).

When the X-ray tube finishes the circle, a single 2D slice is made by the information sent from

the detector to the computer. For the next 2D slice, the bed where the patient lies on, is moved

a little bit in the direction in which one wants to scan the human body. Then, the CT scan can

make a new full circle around the patient to create the next 2D image. In plain words, the

principle of a CT scan is to slice the body into multiple 2D images (20–24).

Just like the X-ray radiography, the CT has the disadvantage of exposing patients to radiation,

which can cause mutations in the DNA when the patient is frequently exposed. Compared to

the X-ray radiography, the CT scan gives a lot more information about soft tissues and bones.

The advantage compared to the MRI is the shorter waiting time as well as the time that is

needed for the patient to be lying in the CT scan to create a full copy of his body (21).

In this paper, as mentioned before, we only include CT imaging. MRI and X-ray radiography

are excluded. However, in addition to the conventional CT scan, the CTA (Computed

Tomography Angiography) is included as well. The principle of CTA is equal to a conventional

CT scan, but in addition a contrast agent is administered intravenously and this enables the

Figure 6: The Hounsfield scale (19) (Available from:

https://web.archive.org/web/20070926231241/http://www.intl.elsevierhealth.com/e-books/pdf/940.pdf)

13

visualisation of the blood vessels, which are normally difficult to see. The contrast agent (e.g.

iodine) enhances the visualisation of blood vessels by inhibiting the X-rays to pass through

(21).

The purpose of this study is to go from a set of 2D images obtained from a CT scan, to a 3D

image, which gives a lot more information than the individual 2D image. This can only be

achieved by using segmentation.

1.2.2 Pixels and voxels

A CT scan produces 2D axial pictures, consisting of pixels. A pixel is just one point on the

image, with certain coordinates (x,y), having a certain brightness or colour. A lot of adjacent

pixels toghether can form an image (Figure 7). To create a 3D image, there is a need of voxels.

The term voxel originates from the contamination of pixel and volume. A voxel is a pixel but in

3D, so it is not a square anymore but a cube with a certain coordinate (x,y,z) and a certain

brightness or colour. Due to the use of voxels instead of pixels, we can see the 3D image from

different angles (23,25).

1.3 Introduction to segmentation

As mentioned above, segmentation is a method that helps one to extract the bony structures

from a CT image. It used to be done, and often is still done, manually, asking a lot of time from

professionals, considering that an outsider who is not comfortable with the technique of

segmenting is unable to perform this. Equally, there are a couple of disadvantages of manual

segmentation, for example the long time it takes to segment, the subjective evaluation and the

low reproducibility (26). Low reproducibility means it is impossible to do the exact same

segmentation all over again, even by the same person. To minimalize these disadvantages,

there is a clear need of a (semi-) automatic segmentation method which does not include, or

only requires a minimal of human intervention.

Figure 7: Illustration of pixels (25) (from Matej “Retro”)

14

Manual segmentation is based on the contrast between the surrounding soft tissue and bone

tissue. The aim is to mark the bone, so the computer can extract it from the image. When

making a segmentation, a line must be drawn where the bone touches the soft tissue meaning

when the contrast is high, it is easy to draw a line between the two. For example, the density

of the long bones such as femur, tibia and fibula is high and the density of the surrounding

tissue is low which establishes a high contrast and makes segmentation effortless. But

segmentation, manual as well as automatic, has its difficulties in the joint epiphysis areas

where the contrast between the cancellous bone and the soft tissue is much lower, since the

cortical layer is thinner and thereby has a lower density (27). The cortical layer is the outer part

of the bone that makes the contrast with the surrounding tissue this high. The thinner the

cortical layer, the harder it is to see the line where the bone is in touch with the surrounding

tissue. Also, both manual as automatic segmentation deal with problems like low quality CT

scans and narrow space between the different bones taking part in the joints (10) (Figure 8).

A B C

1.4 (Semi-) automatic segmentation methods

In this section, the different methods of automatic segmentation are briefly explained. The

easiest method is thresholding, but because of its many disadvantages this method is no

longer the first-choice method of segmenting.

In manual segmentation as well as automatic segmentation, the first thing to do is select the

region of interest (ROI). The ROI is the region where one is interested in. For example, in a

CT scan of the full lower limb, where one is only interested in the femur, then the region of

interest is the region around the femur, including the femur. The image is transformed so that

only the femur is fully visible (28).

Figure 8: Common drawbacks on manual and automatic segmentation. A) Normal femur with high contrast

resulting in effortless segmentation B) Narrow spaces between the pelvic bones and the femoral head in

the hip joint C) A thin cortical layer in the knee joint (femur and tibia) that involves in less contrast

15

1.4.1 Thresholding (global vs local thresholding) (29)

Thresholding as a segmentation method uses the difference in

density between bone and the surrounding tissue (4,30–33). The

Hounsfield Units (HU), which are used to express the density of

any tissue in a CT scan, are used to define the type of tissue.

Bone has a HU between 700 (for trabecular bone) and 3000 (for

cortical bone) and soft tissue has a HU between 100-300. (19)

This allows the establishment of a threshold, a kind of boundary,

between 300-700 HU, so any tissue above this threshold will be

defined as bone and any tissue beneath will be defined as soft

tissue (34,35). For this reason, thresholding is a region based

segmentation method and is sometimes called binarizing.

The most important thing to do, is to choose the threshold value.

You can choose it manually or by calculating the mean CT value of all the pixels and taking

this CT value as a threshold (30,36). But a more appropriate way of choosing the threshold

value is based on creating a histogram of the image pixel densities. The most common used

technique of histogram based thresholding is Otsu’s method (37). The principle of the

technique proposed by Otsu is to search thresholds that maximize the interclass variation and

minimize the intraclass variation. For example in Figure 9, the histogram of a hip joint region

is calculated and two peaks will be found. The histogram curve shows us the number of voxels

that have a certain CT value or HU unit. In the ideal case the histogram curve shows us two

peaks with in between a deep valley. This valley should be used as a threshold so all the

voxels with a HU above the threshold will be defined as bone and all the voxels beneath will

be defined as soft tissue. But in most of the medical images, the two peaks are not that clear.

The lower peak correlates with the densities of the femoral head and acetabulum and the

higher peak correlates with the surrounding soft tissue. Afterwards two Gaussian curves,

representing the distribution of the bone on one hand and the soft tissue on the other hand,

are fitted to the highest values of those two peaks of the histogram curve. Finally, the

intersection between those two Gaussian curves is taken as the threshold value (35,37,38)

(Figure 10).

Figure 9: Illustration of thresholding.

A) Transversal slice of the diaphysis of

the femur B) Graphical illustration of

the variation in density of this picture

(the principle of thresholding)

A

B

16

Figure 10: Histogram based threshold: Two Gaussian curves representing the distribution of the bone on one hand

and soft tissue on the other hand are fitted in the histogram curve and the intersection of the two curves is taken as

the threshold value. (35) (from Cheng Y et. al.)

As mentioned above, thresholding is no longer the first-choice method of segmenting, since it

has many disadvantages like over-thresholding, segmenting bones where implants are

involved and diffused boundaries (35).

Over-thresholding means that not only bone will be defined as bone, but also a part of the

surrounding soft tissue is defined as bone. When the threshold of HU is taken a bit lower, e.g.

250 HU, a part of the surrounding tissue will be segmented too. Sometimes the threshold must

be placed at lower HU e.g. in the epiphyses where the bone density is lower and when the

threshold is higher, the bone will be defined as soft tissue.

When implants are present in the patient, which are often made of metal, it is impossible to

use a segmentation method based on thresholding because bone and metal have an overlap

in density and the implant could be defined as bone.

Narrowness in a joint and weak boundaries are also common problems of thresholding. For

example, in the hip joint where the femur and acetabulum of the pelvis almost touch each other

due to thin and often worn out cartilage layers. Thresholding is not able to make the distinction

between the femur and pelvis and both these bones will be defined as bone tissue. This could

be a problem if the aim is only to segment the femur and not the pelvis (35).

Local or adaptive thresholding can be a solution for the over-thresholding issue. Adaptive

thresholding is the situation where different threshold values are used for different regions in

the image (38,39). Therefore the image is divided into separated blocks or sections and each

section has a different threshold value, calculated by using for example the histogram method

explained above (32,33,40).

17

Another method, which is a variant of local or adaptive thresholding is calculating 2 threshold

values: LT (Lower Threshold) and HT (Higher Threshold) (Figure 11). Every voxel with a HU

below the LT is defined as soft tissue and every voxel with a HU higher than HT is defined as

bone. Every voxel within the LT and the HT will be assigned by a local adaptive formula. This

formula is based on the mean CT value (m) and the standard deviation (SD) of the 26

neighbour voxels of the voxel under examination (v)(41):

𝐶𝑇𝑣 = {< 𝑚 − 𝛼 ∙ 𝑆𝐷 => 𝑙𝑎𝑏𝑒𝑙 𝑜𝑓 𝑠𝑜𝑓𝑡 𝑡𝑖𝑠𝑠𝑢≥ 𝑚 − 𝛼 ∙ 𝑆𝐷 => 𝑙𝑎𝑏𝑒𝑙 𝑜𝑓 𝑏𝑜𝑛𝑒

(1)

with D typically equal to one (41).

At last, there is one more way to use thresholding as a segmentation method. In case of a lot

of background, you can define a second thresholding value so instead of binarizing the image,

you divide it in three regions which differ in density. The first threshold makes the difference

between the background (air = - 1000 HU) and the rest of the image. The second threshold

makes the difference between the rest of the image, so the soft tissue and the bone. This last

way of thresholding is used in our proposed automatic segmentation program.

Figure 11: Histogram of CT values. All voxels with CT values above the HT are classified as bone and all the

voxels with CT values below the LT are classified as soft tissue. In between LT and HT local adaptive thresholds

are used. (41) (from Kang Y et. al.)

. (

18

1.4.2 Gradient based method (29)

The image gradient is a parameter for directional change in an image. On one hand the

magnitude of the gradient represents the rapidity of the change and on the other hand, the

direction of the gradient represents in which direction the image is changing (42). The principle

of the gradient based method is based on the first derivative of the image which represents

changes in density in the image. The method only works if there is a clear change in density

between two pixels. These clear changes in density are mostly found in the edges between

surrounding soft tissue and bone and therefore can be used as a segmentation method (Figure

12) (32,33).

The magnitude of the gradient tells us more about the rapidity of

the change in density of the image. For example, when there is a

transition zone from soft tissue to cortical bone, there is a big

change in density. Therefore, the gradient will have a high peak

in that pixel. But when there is a transition zone between soft

tissue and trabecular bone whereas the HU difference is not that

big, the peak of the gradient will be much lower (3,42).

The direction of the gradient tells us more about the direction in

which the change in density occurs in the image. The direction

can be measured in the x-axis or in the y-axis (3,42). In Figure

12, the direction of the gradient is represented in the x-axis. In C)

the gradient has not included the direction, but just the change in

density and in D) the gradient has included the direction. To

calculate the gradient, we start from the first pixel on the left side.

The next pixel has the same density so there is no change and

the gradient stays the same. When we come across the first pixel

that includes the femur, there is a big change in density. The HU

increase from one pixel to the other, so the direction goes

upwards and the magnitude is high. When we are in the cortical

bone of the femur, the density does not change so the gradient

ascends to this isoline. But afterwards, when we come across the first pixel that does not

belong to the cortical bone of the femur anymore and therefore a decrease in HU, the gradient

direction goes downwards. After this change, the density stays the same for a long time, so

the gradient goes back to his isoline.

Figure 12: Illustration of gradient

based method. A) Transversal slice of

the diaphysis of the femur B) Graphical

illustration of the variation in density of

this femur (the principle of thresholding

C) The principle of gradient based

segmentation D) Gradient based

method considering direction

B

C

D

A

19

From the information above, we can conclude that the gradient based method is a

segmentation method based on edge detection, as the edges are most of the time the zones

with the biggest difference in density (25,26).

The advantage of gradient compared with thresholding is that it is way more sensitive. For

example, with the presence of a sesamoid bone in an image, having a HU below the threshold,

it will be noticed by the gradient but not by the thresholding method. In the thresholding method,

the threshold is often placed high so in order to not wrongly define soft tissue as bone. Because

of this high threshold value, the sesamoid bone will wrongly be defined as soft tissue. With the

same image segmented by the gradient based method, the gradient will recognize the abrupt

change in the density, even for the sesamoid bone, since the surrounding soft tissue has a

much lower HU. So, we can conclude that the gradient based segmentation method is more

sensitive as it will recognize such minimal difference in density.

This also explains the disadvantage of gradient based segmentation. Since it recognizes

minimal differences in density, it will also recognize noise as an edge and thus the gradient

will show us a difference in density. The edge detection method requires an equilibrium

between detection of the real edges and not detecting noise as an edge. So, if the level of

detection of the minimal of change is too high, noise can be wrongly defined as an edge. We

can conclude that gradient based method is suited for images with sharp edges and minimal

noise, because in noisy images it can create extra edges or, when the level of detecting density

changes is too low, miss edges (32,43).

Common used operators of the gradient based method are Sobel’s operator, Laplacian

operator, Laplacian of Gaussian operator and Canny edge detector. Sobel’s operator is based

on the first derivative of the image and therefore is not too sensitive for noise, but the other

three are based on the second derivative and are therefore very sensitive to noise as

derivatives amplify the noise (32,33,43–45).

1.4.3 Shape models (46–49)

In our proposed method of automatic segmentation, we use Statistical Shape Models (SSM)

for the automatic segmentation of the bones in the lower limb and therefore it is briefly

explained.

A statistical shape model describes the average shape distribution within a population as well

as the main modes of variation of that shape. The SSM is based on a training dataset.

20

First of all, there is the need to represent the shape of a bone. In this paper, it will be explained

with the concept of landmarks. Landmarks are multiple points distributed across the surface

creating a point density model (50). They need to be placed in exactly the same anatomical

location on every bone, in other words there is a need of corresponding positioning. This is

often done manually, where the landmarks are placed in the periphery of the bones in the

training dataset. To facilitate this time-consuming task, there is an easier way to place

landmarks: once there are a few landmarks placed, new ones can be placed between the

already placed ones by interpolation. However, this is one of the most difficult tasks of

constructing a statistical shape model since there is a need of correspondence. When there is

a lack of correspondence, the SSM will not be consistent. In the equation below 𝑥𝑖 stands for

the coordinates of multiple landmarks or in other words the shape of a bone.

With these landmarks, the average shape can be calculated as:

�̅� =

1𝑁 ∑ 𝑥𝑖

𝑁

𝑖=1

(2)

Where �̅� stands for the average shape and N for the number of samples included in the training

dataset.

Afterwards, a Principal Component Analysis (PCA) is performed. PCA calculates the most

important modes of variation of the average shape, also called the Eigen modes. The most

important mode of variation is mentioned first because it contains the biggest percentage of

shape variation (Figure 13). After PCA, the main modes of variation are known. Then each

bone of the training dataset, could be described as:

𝑥 = �̅� + 𝑎𝑏

(3)

Where x is the bone from the population that is described, �̅� the mean shape, a the contribution

(the score) of the mode of variation and b the mode of variation. The formula above considers

one mode of variation, but of course it can be applied to multiple modes of variation:

21

With c the total number of modes of variation.

As more modes of variation are included, the accuracy of the

description of the bone will increase. Furthermore, including

more data in the training dataset, will improve the relevance of

the SSM.

The next step is to fit those SSM into a new set of images (the

fitting process) and Active Shape Models (ASM) are one way

of doing this (47). The ASM is a statistical shape model which

iterative deform to fit the model obtained from the training data

set to the new image (49). Once these SSM/ASM are made

based on a training set, they can be used for a lot of medical

procedures (e.g. design of prosthesis/implants, correlation

between morphology and diseases (femur neck –

osteoarthritis), prediction of fracture risk, surgery planning,

computer aided surgery, …).

But segmentation using the SSM fitting alone does not allow to

segment aberrant shape data if the patient has anatomical

abnormalities (e.g. a tumour) or when too many differences

between the bone of the patient and the bones of the training

dataset occur. The SSM simply would not fit properly (46).

1.4.4 Free deformation

Free deformation is an example of a snake or an active contour model. Such snakes move

within the image and search for boundaries that are not included in the statistical shape model

(32).

In this paragraph, the process prior to the free deformation phase is explained. To begin, the

region of interest is selected, as mentioned above. Then, the first step is thresholding, which

𝑥 = �̅� + ∑ 𝑎𝑧𝑏𝑧

𝑐

𝑧=1

(4)

Figure 13: Illustration of the SSM and

the Eigenmodes. A) The percentage of

variation in appearance explained by

different shape modes in a 2D statistical

appearance model of the femur. B-D)

The representation of the first three

principal modes of variation of the full

SSM at the mean and at plus or minus

three standard deviations from the mean

(46) (from Sarkalkan N et. al.)

22

marks the bone and therefore the first step in segmentation is made. The second step is to fit

the ASM on the image. This way, the biggest part of the boundaries will be found and these

will be secured. Supposing for example that after application of the ASM, 95% of the founded

boundaries will be correct, but 5% cannot be found, because they are located outside the

restriction that the ASM dictates. The reason being is that the ASM is able to move within the

picture, but unable to move too far away from its average model. Thus, when the boundaries

are located over a couple standard deviations of the mean, the SSM will not allow the ASM to

go that far outside its average model. Therefore, we can conclude that the ASM has certain

restrictions.

Finally, the free deformation phase takes place, which is an active contour model that searches

the boundaries that cannot be found by the ASM. On the bases of an iterative process, the

free deformation searches in which direction it has to move to find the boundary, considering

its neighbours. This is an iterative process because also, the free deformation has its

restrictions. The active contour model allows a movement outside the ASM restriction, but it is

not allowed to move more than a few millimetres. So, when the real boundary is far away from

the boundary the ASM suggested, the free deformation needs more than one processing cycle

to reach the real boundary. It is clear that with each cycle, the found boundary is bit by bit

closer to the real boundary (Figure 14) (50–54).

The best way to visualize an active contour model is to see it as a rubber band that has the

possibility to deform or move during time. The aim of this deformation is to get as close as

possible to the target contour. During the iterative process, the active contour model is

attracted toward the target contour. The snake works with an energy functional, to measure

the suitability of the contour. When there is a good result, the energy functional is minimal. The

Figure 14: Illustration of the free deformation phase. Assuming that the circles represent the cortex of the femur.

The green circle shows us the real cortex of the femur and the blue circle shows us the model made by the ASM.

Because of its restrictions, the ASM is not able to reach the real green cortex. The free deformation phase, makes is

possible to reach the real cortex by an iterative process. The red circles show us that the free deformation phase

takes place in multiple steps. Each step, the real boundary is closer.

23

intention is to strive for a minimal energy functional. The energy functional is based on multiple

forces that control the location and the shape of the active contour model. These forces can

be divided into internal and external forces. The internal forces consist of the guarantying of

continuity of the contour (Econt: this energy term forces the snake to be continuous) and the

smoothness of the contour (Ecurv: this energy term forces the snake to make a smooth contour).

Together they control the alteration of the snake and keeps the model close to its original

shape. Interesting is that weighting the internal energy terms needs to be done very carefully,

since too much weight leads to a rigid model, that does not want to alter and too little weight

leads to a model that is too flexible. The external energy term is the attraction component. It

guaranties that the contour is attracted to the closest image edge and thereby controls the

fitting of the contour into the image (Eimage). The Eimage is based on image intensity or gradient

magnitude and will become very little when the snake is getting close to an edge. With this

information, the formula for the energy functional is the following: (50,52–54)

𝐸𝑠𝑛𝑎𝑘𝑒 = 𝐸𝑖𝑛𝑡𝑒𝑟𝑛𝑎𝑙 + 𝐸𝑒𝑥𝑡𝑒𝑟𝑛𝑎𝑙 = 𝐸𝑐𝑜𝑛𝑡 + 𝐸𝑐𝑢𝑟𝑣 + 𝐸𝑖𝑚𝑎𝑔𝑒

(5)

In the old model of Kass (52) there was one extra energy term. Econstraint is the constraint energy

and is based on interaction with the user. The user can give the snake restrictions by using

this energy term and push the snake towards or away from certain positions (50,51). Because

of the aim of this study to make segmentation automatic, the user interaction will not occur.

Using active contours has a couple of disadvantages. The initial contour must be placed close

to the real boundary because if not, the snake will not be able to find this boundary (55).

In our proposed program, the free deformation is used as final step after initializing the contour

to the image, near the real boundaries.

1.5 Practical applications of image segmentation

In this paragraph, a summary of applications in the context of image segmentation is

described. Image segmentation is or could be used in a lot of medical procedures such as

orthopaedic surgery: e.g. computer aided orthopaedic surgeries (CAOS), planning of surgical

procedures, design optimisation and the placing of prosthesis or implants. It can also be used

for diagnosis of e.g. anatomical abnormalities or pathogenic cellular growth, tumour

localisation and be of use for treatment planning. Furthermore, since segmentation is used to

study the anatomy of structures, it makes comparison between organs or bones from different

24

humans possible (e.g. male vs female bones) (10,32,34,35). The ability to see a 3D anatomical

structure instead of a 2D slice, gives us more information and allows us to calculate the risk

factors for certain diseases (e.g. osteoporosis and osteoarthritis) (46). Image segmentation is

not only used in the orthopaedic field, but also in other medical disciplines. For example, in

cardiology, where image segmentation makes heart surgery a lot easier. The same applies to

neurology and thereby brain surgery. Also in the diagnosis of brain tumours image

segmentation can be useful. In fact, image segmentation is capable of extracting every organ

in the human body. Thus, every single organ can be extensively studied for its normal variation

and its abnormalities (48).

Segmentation techniques are not only used in medicine, but also in crime investigation, e.g.

face recognition, finger print recognition and iris recognition (32).

The examples above are only a few of the applications of segmentation in the medical field. It

shows us that segmentation is, or is soon to become a huge tool in diagnosing procedures and

even in operation planning.

1.6 Validation metrics

For the validation of automatic segmentation programs multiple measurements are used. In

this paper three different metrics are explained and two of them will be used to compare our

program to other bone segmentation methods. The mean error distance (Average Surface

Distance (ASD) & Root Mean Square (RMS)) and the maximum fault (= Hausdorff Distance

(HD)) will be used in this study and are explained in section 2.4. The Dice Similarity Index

(DSI) and its derivative, the Overlap Error (OE), will not be used. As they are frequently used

as a validation metric in other, previously developed, automatic segmentation techniques they

are still explained below.

1.6.1 Mean error distance (ASD & RMS)

See section 2.4 for the definition of this validation metric.

1.6.2 Maximum fault (HD)

See section 2.4 for the definition of this validation metric.

25

1.6.3 Dice Similarity Index

The Dice Similarity Index (DSI) or the Dice coefficient is a measurement for the similarity

between the segmentations from the manual technique and the proposed automatic

segmentation technique. That is why it is also called the overlap index. The formula for the

DICE coefficient is:

𝐷𝑆𝐼 =2(¨𝐴 ∩ 𝐵¨)¨𝐴¨ + ¨𝐵¨

(6)

Where ¨𝐴¨is the manual segmentation and ¨𝐵¨ the automatic segmentation. 𝐴 ∩ 𝐵 is the

volume that the 2 segmentations have in common (Figure 15).

DSI is given in percent, i.e. if the result is 100%, it means that the manual and the automatic

segmentation completely overlap. 0% means there is no overlap between the two

segmentations (28,35).

Some papers use the volumetric Overlap Error (OE) as a metric. OE stands for the percentage

of segmentation between the manual segmentation and the automatic segmentation that does

not overlap. In other words, when the result is 0% it is a perfect segmentation, when it is 100%

there is no overlap between the two segmentations.

Since

𝑂𝐸 = 1 −(¨𝐴 ∩ 𝐵¨)(¨𝐴 ∪ 𝐵¨)

(7)

𝐴 ∩ 𝐵

Figure 15: Illustration of the Dice Similarity Index

(DSI)

𝐴 𝐵

26

and

(¨𝐴 ∪ 𝐵¨) = (¨𝐴¨ + ¨𝐵¨) − ¨𝐴 ∩ 𝐵¨

(8)

we can convert the volumetric overlap error to the Dice Similarity Index: (50)

𝐷𝑆𝐼 = 2 ×(1 − 𝑂𝐸)(2 − 𝑂𝐸)

(9)

We can conclude that these are good parameters to compare different studies, but it is

important to note that a comparison of the different methods is not easy and must be

interpreted with caution because of different datasets, different parameters to evaluate and

different methods being used (28,46,56,57).

27

2 MATERIALS AND METHODOLOGY

To start, the program that is used to do manual segmentation is briefly explained. Some

techniques that make the manual segmentation less time-consuming are mentioned. Then,

the pipeline of the proposed method for automatic segmentation is presented. Each step will

be explained and illustrated by figures. In the third part, the method of validation is explained.

At last, the validation metrics and the use of inter- and intra-observer differences are

mentioned.

The automatic segmentation technique performed 210 CT automatic segmentations. Only 10

of them were used for the validation since a manual segmentation was necessary for

comparison. Furthermore, two observers independently segmented three cases and one

observer segmented two times the same case by using Mimics. These inter-observer and intra-

observer variation results were used for the validation.

2.1 Manual segmentation using Mimics

The first part of my work during this study was to create manual segmentations, which could

be used for the golden standard. The manual segmentation is done using Mimics version 17.1,

a program developed by Materialise Leuven. Mimics is a 3D medical image processing

software that allows, with user interaction, to turn scans into accurate 3D models (58). First

there is a need of a full CT scan in DICOM files. These files need to be uploaded in Mimics.

Then, the manual segmentation can begin. The aim is to select every pixel that presents bone

and when this is accomplished, Mimics allows this selection to be turned into a 3D shape. To

make it somewhat less time consuming, Mimics has a couple of features to make manual

segmentation easier such as thresholding, the possibility of selecting multiple pixels at once

and region growing. Thus, in Mimics, it is possible to create a threshold: every pixel with a HU

above the threshold will be defined as bone. One should be careful not to place the threshold

at a too high HU value, because then soft tissue will be defined as bone, and this will create

more work to remove the label ‘bone’ of these wrongly defined pixels. Region growing makes

it possible to close open regions and fill holes. By using these features, the manual

segmentation goes faster, but still it takes 100 man hours to manually segment a full lower

limb (6). This is the reason why there is a need for automatic segmentation techniques, so

professionals can use their time more efficiently.

28

2.2 Construction of the SSM

Statistical shape models are used in our proposed automatic segmentation technique. First, a

low-resolution statistical shape model is used. This low-resolution SSM contains the full lower

limb. Second, a high-resolution statistical shape model is applied to the CT data set and only

encloses one bone. For the construction of these SSM models, manual segmentations are

necessary. As a result of the manual segmentation, explained above, a 3D model is formed.

This 3D model consists of landmarks, connected to each other. Each landmark is connected

with two neighbour’s landmarks with the aim to form triangles. All the triangles together form

the surface of the 3D model.

In this study, a minimal of 10 manual segmentations was made to construct the first version of

the SSM model. An average model of these 10 manual segmentations was computed and

Principal Component Analysis (PCA) was applied to determine the modes of variation of the

average shape model (see 1.4.3). The more cases segmented by the SSM, the more accurate

the SSM will be (see beneath).

Figure 16 shows us the average shape and appearance of the low-resolution full limb model

and the average shape of the high-resolution model of different bones. These statistical shape

models were used in our proposed automatic segmentation technique.

Figure 16: Average shape and appearance of the low-resolution full limb SSM on the left and the average

shape of the high-resolution SSM of the 6 lower vertebrae, the foot bones, the sacrum, the tibia and fibula

on the right.

29

Attention was paid to the exclusion of segmentations that were included in the statistical shape

models, because the automatic segmentation of a case that is included in the SMM will be

easier to recognise for the automatic segmentation program and therefore will affect the

results. Excluding these segmentations was done by applying the “leave one out” technique,

which means that when an evaluation needs to be done of a bone that is included in the SSM,

this bone will be left out of the SSM.

2.3 The pipeline of the automatic segmentation of the full lower limb

In the overview beneath, the automatic segmentation pipeline of our newly developed program

is illustrated. The fully automatic segmentation is divided into three phases: firstly, there is an

initialisation and segmentation phase. This first phase is separated in four steps. Secondly,

the training of the shape model is supervised and the segmentation pipeline is continuously

updated. Finally, there is the appearance and solids phase.

2.3.1 Initialisation and segmentation

To illustrate this phase, the reader is referred to Figure 17.

2.3.1.1 Step 1: Initialisation to estimate position, orientation and scale of the lower limb on an

isosurface reconstruction following simple thresholding

First, the volume of the CT images is down-sampled to reduce calculation time for the

initialisation phase. Afterwards, with the use of two thresholds corresponding to Otsu, the

image is segmented into three levels: background, soft tissue and bone (37). With the use of

Appearance and solids

Supervised training of the shape models and continuous updating of the segmentation pipeline

Initialisation and segmentationInitialisation of the lower limb on a

isosurface reconstruction following simple thresholding

Statistical shape adaption to provide a good local initialisation for the final in-

image segmentation processIn-image statistical shape adaptation Free form segementation

30

3D binary morphological operations, a binary dataset is generated. These operations include:

closing open regions, breaking small connections, filling holes and the removal of unconnected

noise. After the application of these operations, an isosurface is generated from this volume

data to provide a first estimation of the geometry of the cortical surfaces of the full lower limb.

2.3.1.2 Step 2: Statistical shape adaptation to provide a good local initialisation for the final

in image segmentation process

To create a first estimation on shape and position of the lower limb bones, the low-resolution

Statistical Shape Model (SSM) is fitted onto the isosurface. This fitting procedure iteratively

searches for the closest points from the source image to the target as well as from the target

to the source. It also excludes outlier points which can be caused by noise or other bony data.

2.3.1.3 Step 3: In-image statistical shape adaptation

The low resolution SSM gives a first good estimation of shape and location of the lower limb

bones. The third step includes local optimisation with the use of the high-resolution statistical

shape model fitting and is performed bone by bone. Penetration of surfaces in difficult areas

like joints is avoided by digital subtraction of the neighbouring bones during the fitting

procedure.

2.3.1.4 Step 4: The free form segmentation phase

This step is used to overcome the restrictions that SSM dictates and allows for the shape not

to be captured by the SSM, as explained above in 1.4.4.

31

2.3.2 Supervised training of the shape models and continuous updating of the segmentation pipeline

Each automatically segmented case is evaluated by a clinical expert. Segmentations which fit

well with the original scan data are added to the existing SSM and for each new segmentation

the updated versions are used. Cases that before did not fulfil quality requirements are

segmented once again with the updated pipeline.

2.3.3 Solids and appearance

With a correspondence feature, solids and appearance can be added to our 3D model of a

bone. Each bone has a periosteal surface, presented as an outer layer, and an endosteal

surface, presented as an inner layer. The correspondence feature makes it possible for our

automatic segmentation technique to connect one point of the outer mesh layer to the opposite

point of the inner mesh layer (Figure 18).

Figure 17: Overview of the automatic segmentation pipeline: A) Initial positioning of the

lower limb on an isosurface reconstruction B) fitting of the low-resolution shape model on

the isosurface C) followed by in-image statistical shape adaptation and D) free form

segmentation phase.

32

Therefore, during the construction of the shape model, the periosteal and endosteal surfaces

were non-rigidly registered onto each other to define dense correspondence between both

layers. The correspondence feature of inner and outer cortical mesh layers allows for

immediate construction of tetrahedron meshes as well as for the definition of local cortical

thickness. This cortical thickness can play an important role in the prediction of the fracture

risk. When bone is fragile, the risk of getting a fracture increases.

Tetrahedrons are created to form a 3D model. Every 3D bone model is constructed of

tetrahedrons, which are formed by interconnecting 4 points. These points are part of the

scatterplot or mesh, which is created after the pipeline was applied to the CT data set. Cartilage

layers can similarly be added by applying a distance map to the surface normal of a selected

area, and again tetrahedrons can be defined automatically (Figure 19).

Secondly, due to the correspondence feature, bone density in terms of Hounsfield units could

be directly measured during segmentation. This density or in other words the amount of bone

mineral in the bone tissue, is measured at the middle of corresponding outer-inner vertices.

For the 3D representation, different densities are given different colours. For example, low

density is given the colour blue and high density is given the colour red. Therefore, one can

16534

298 16534

9763 298

9763

Periostal surface

Endostal surface

Figure 18: Illustration of the correspondence feature

Figure 19: Illustration of solid cortical, cartilage and marrow volumes, presented by

tetrahedrons

33

predict the quality of the bone just by seeing the appearance of this 3D representation, since

bone density is the most important predictor for osteoporosis and bone fracture. The lower the

bone density, the higher the risk of osteoporosis and therefore the risk of bone fractures (59–

61).

2.4 Used validation metrics

As explained above, the mean error distance and the maximum fault are used as parameters

for validation of our newly developed automatic segmentation technique. DSI (Dice Similarity

Index) and OE (Overlap Error) will not be used in this study, but is frequently used in the

validation of other, previously developed programs.

2.4.1 Mean error distance (ASD & RMS)

The mean error distance is the Average Distance (AD) in mm, often called the Average Surface

Distance (ASD), between every point in the manual segmentation to the closest point in the

automatic segmentation.

�̅� =𝑋1 + 𝑋2 + 𝑋3 + ⋯ + 𝑋𝑛

𝑁

(10)

With �̅� the mean error or the AD, every 𝑋 distance between two points of the different

segmentation sets and N the number of the distances included in the formula (57).

A similar parameter of the average distance is the Root Mean Square (RMS). It is the square

of the formula above and then the square root is computed from the final result. Root mean

square is also called ‘the quadratic mean’.

�̅�𝑅𝑀𝑆 = √𝑋12 + 𝑋2

2 + 𝑋32 + ⋯ + 𝑋𝑛

2

𝑁

(11)

Where �̅�𝑅𝑀𝑆 is the root mean square (57,62).

34

2.4.2 Maximum fault (HD)

The Hausdorff Distance (HD) is the greatest distance of all distances from one point in the

manual segmentation to the correspondent point in the automatic segmentation and vice versa

(Figure 20). In other words, it measures the mismatch between the two segmentations (57,63).

2.5 Inter- and intra-observer differences

In manual segmentation, inter- and intra-observer differences were defined. Inter observer

differences are differences between the manual segmentations of the same bone, in the same

case made by two different clinical experts. The manual segmentation from one expert is the

golden standard and the manual segmentation of the other expert is compared to this golden

standard. Intra-observer differences are differences between the manual segmentation of the

same bone, in the same case by the same clinical expert. The intra-observer difference

measures in other words the reproducibility.

The inter- and intra-differences are measured to demonstrate that even the golden standard,

in other words the manual segmentation, has a degree of error. This makes it easier to interpret

the ASD and the maximum error that were found in our automatic segmentation technique

whilst comparing it to the golden standard.

Figure 20: Illustration of the Hausdorff Distance

35

3 RESULTS

A dell precision M6800 Laptop (Intel Core i7 - 4910MQ, 16 GB RAM, 64 bit) was used to

process a total of 210 CT data sets. Each scan data set consisted of an average of 1864

(ranging from 1320 to 2315) slices with an average pixel size of 0,703 mm (ranging from 0,575

mm to 0,975 mm). The processing of a full data set requires an average around 2 hours per

case. This time includes uploading all cases as well as exporting and writing of segmentation

results in obj. and stl. file format.

As mentioned in the introduction, different bones of the lower limb are included in this study.

Six lower vertebrae, sacrum, bilaterally the pelvic bone, femur, tibia, patella, fibula, talus and

calcaneus are included (21 osteological structures in total). A series of iterations and

progressive updating of the low-resolution as well as the high resolution SSM’s, following

quality clearance by the clinical experts was performed. Finally, all the scans were processed

and convergence of the algorithm was obtained in all the scans except one.

For the validation, manual segmentations were necessary to compare the automatically

segmented bones to the manually segmented bones, which are considered as the golden

standard. Because of this time-consuming task, 10 manual segmentations were made. These

manual segmentations included the pelvis, the femur, the tibia, the patella, the fibula, the

calcaneus and the talus. Since the six lower vertebrae and the sacrum were not manually

segmented, the results of the automatic segmentation are not given in Table 1 because they

are not comparable to the golden standard. Therefore, these bones are not included in the

validation of our automatic segmentation technique.

The results, expressed In average error (Average Surface Distance) and maximum error

(Hausdorff Distance) are presented in Table 1. The mean error or the ASD ranged from 0,53

mm to 0,76 mm, with an average of 0,65 mm and the maximum error or the Hausdorff distance

ranged from 2,02 mm to 7,84 mm, with an average of 4,05 mm. The smallest errors were found

in the foot bones (the talus and the calcaneus) and the largest errors were found in the pelvis.

36

Table 1: Accuracy and observer variation in segmentation of the respective segments

Automated segmentation (n = 10)

Manual segmentation

Inter-observer variability (n = 3)

Intra-observer variability (n = 1)

Average Error

(mm)

Maximum Error

(mm)

Average Error

(mm)

Maximum Error

(mm)

Average Error

(mm)

Maximum Error

(mm)

Pelvis 0,75 +/- 0,17 7,84 +/- 2,26 0,41 +/- 0,20 3,74 +/- 2,68 0,18 1,36

Femur 0,65 +/- 0,10 4,79 +/- 2,39 0,41 +/- 0,15 2,30 +/- 0,98 0,18 2,29

Tibia 0,63 +/- 0,11 4,07 +/- 2,15 0,39 +/- 0,19 1,88 +/- 0,47 0,23 0,89

Patella 0,65 +/- 0,15 2,02 +/- 0,39 0,42 +/- 0,11 1,96 +/- 0,29 0,18 0,78

Fibula 0,76 +/- 0,18 3,76 +/- 1,17 0,61 +/- 0,08 2,25 +/- 0,71 0,17 0,81

Calcaneus 0,53 +/- 0,16 2,90 +/- 0,77 0,40 +/- 0,12 1,67 +/- 0,34 0,32 1,12

Talus 0,57 +/- 0,12 2,97 +/- 0,59 0,44 +/- 0,08 2,21 +/- 0,41 0,26 0,78

The automatic segmentation of the pelvis has an average error of 0,75 mm and a Hausdorff

distance of 7,84 mm in comparison with the golden standard. The femur has an average

surface distance of 0,65 mm and a Hausdorff distance of 4,79 mm. The tibia has an average

surface distance of 0,63 mm and a maximum error of 4,07 mm. The patella has an average

error of 0,65 mm and a maximum error of 2,02 mm. The fibula has an average error of 0,76

mm and a Hausdorff distance of 3,76 mm. The calcaneus and the talus have an average

surface error of 0,53 mm and 0,57 mm respectively and a Hausdorff distance of 2,90 mm and

2,97 mm respectively (Table 1).

The inter-observer variability has the following results: an average error that ranges from 0,39

mm to 0,61 mm, with an average of 0,44 mm and a maximum error that ranges from 1,67 mm

till 3,74 mm, with an average of 2,29 mm. The intra-observer variability in one case was also

measured and the following results were found: an average error that ranges from 0,17 mm to

0,32 mm and a maximum error that ranges from 0,78 mm till 2,29 mm. These result show us

that even in the golden standard, degrees of error exist.

When we make a comparison between the errors made by the automatic segmentation and

the errors made by 2 experts whose manual segmentations of the same bone were compared

to each other, small differences are found. The maximum difference in average error between

the automatic segmentation and the inter observer variability research was found in the pelvis

since the average surface distance between the two manual segmentations, made by two

different experts, is 0,41 mm and the average error of the automatic segmentation is 0,75 mm.

37

Therefore, the difference in error is 0,34 mm. On the other hand, a minimum difference in

average surface distance of 0,13 mm is found in the calcaneus and talus since the average

surface distance between the two manual segmentations is 0,40 mm and 0,44 mm respectively

and the average surface distance of the automatic segmentation is 0,53 mm and 0,57 mm

respectively. The average difference between the automatic segmentation and the inter

observer variability research in average error is 0,21 mm and in maximum error is 1,76 mm.

The maximum difference in maximum error is 4,10 mm, found in the pelvis and the minimum

difference in maximum error is found in the patella with a difference of 0,06 mm.

Also in the intra-observer variability research errors are found. Here the maximum average

error difference between the intra-observer variability research and the automatic

segmentation is found in the fibula, with an average surface distance difference of 0,59 mm.

The minimal difference in average surface distance is 0,21 mm, found in the calcaneus. The

maximum difference in maximum error is 6,48 mm, found in the pelvis, and the minimum

difference in maximum error is found in the patella, with a difference of 1,24 mm. The average

difference between the automatic segmentation and the intra-observer variability research in

average error is 0,43 mm and in maximum error is 2,85 mm.

38

4 DISCUSSION

4.1 General

Image analysis is becoming an important tool in medicine, especially in diagnosis and medical

decision making. The progress in image analysis has caused an increase of diagnosis on the

basis of imaging technology. Image segmentation of Computed Tomography and Magnetic

Resonance Imaging plays an important role in the image analysis, but is often still done

manually, and by consequence is time-consuming (26). E.g. the time needed for extracting the

articular surfaces of the knee joint was estimated to approximately two days of work and a for

a full lower limb over 3 weeks of work were reported (6).

Because of this time-consuming task, several professionals (engineers and doctors in

medicine) have described different automatic segmentation techniques for image

segmentation. The segmentation technique that gives the most consistent results is the model

based technique, the technique based on using the SSM and ASM. But the disadvantage of

this technique is that there is a need of a large training data set to create an accurate shape

model. Other methods are less accurate in images with a lot of noise or artefacts (35,47,64).

Because of these disadvantages, there is a growing interest in automatic segmentation

programs which are able to process large data sets within minimal time and with a minimal

need of prior knowledge, and therefore a minimal need of human interaction (27).

An overall error of 0,65 mm was found using the current segmentation pipeline. The overall

error in the inter-observer variance was 0,44 mm, where the inter-observer variance are two

manual segmentations and therefore two golden standard segmentations compared to each

other. The absolute difference in overall error between the current segmentation pipeline and

the golden standard is therefore only 0,21 mm.

The current segmentation pipeline has an overall maximum error of 4,05 mm and the inter

observer variance has an overall maximum of 2,29 mm. The absolute difference in the overall

maximum error is therefore 1,76 mm.

On the basis of these results, we can conclude that a very good approximation of golden

standard segmentation can be guaranteed by using our proposed segmentation pipeline.

39

It is important to note that this study is the first of its kind to develop an automatic segmentation

program that segments the full lower limb. Multiple automatic segmentation techniques, which

were developed in the past, are able to segment only a few bones and organs or were

specifically developed to segment joints like the hip joint or knee joint (35,56,65).

4.2 Comparison with previously developed programs

As said above, multiple professionals have already developed semi-automatic or automatic

techniques for making segmentation a less time-consuming task. In Error! Reference source not found., the results of several studies will be presented and a comparison with our

proposed technique will be made.

The following studies are mentioned: Lamecker H. and colleagues (7), Seim H. and colleagues

(8), Almeida D.F. and colleagues (10), Krcah M. and colleagues (27), Uozumi Y. and

colleagues (26), Younes L.B. and colleagues (11), Wu D. and colleagues (9).

In the heading of the table, the following abbreviations are used: � patients stands for the

number of patients, ASD stands for average surface distance, RMS stands for root mean

square, HD stands for Hausdorff distance and OE stands for overlap error.

Lamecker H. et al. proposed a new statistical shape model of the pelvic bone, generated by

manually segmenting 23 CT data sets of male patients. They divided the pelvic bone into 11

regions. Just like our automatic segmentation pipeline, they performed a PCA (Principal

Component Analysis) and determined the main modes of variation. Also, they applied a gray-

value profile analysis where they use the thresholding technique. The results were described

for a “leave all in” segmentation as well as a “leave one out” segmentation. In the first case,

the SSM contains the CT data set that is going to be segmented. The ASD will be a lot smaller

if the SSM already knows the shape of the CT data set that needs to be segmented. Therefore,

these results are not comparable with ours, since our statistical shape model does not contain

any CT data sets that still need to be segmented. On the contrary, every new segmented CT

data set is added to the SSM. The “leave one out” segmentation on the other hand is fitted for

the comparison since the SSM does not know the shape in advance. The mean surface

distance in this study was 1,6 +/- 0,2 mm and the maximum distance was 14,6 +/- 3,8 mm. On

the basis of these results, we can conclude that our automatic segmentation is more accurate

(7).

40

Ta

ble

2: C

ompa

rison

with

pre

viou

sly

deve

lope

d au

tom

atic

seg

men

tatio

n te

chni

ques

41

Seim H. and colleagues presented an algorithm for an automatic segmentation of the pelvis.

Again, this study is based on a statistical shape model. The following three steps are used:

initial placement of the average shape (SSM) of the pelvis within the CT data, adaption of the

SSM to the image by a variation of the shape models and finally a free form deformation step,

which is used to overcome the restrictiveness of the SSM. We can conclude that this proposed

method has a lot of similarities with ours. For the validation, they used 50 manually segmented

CT data sets by performing the “leave one out” method. The results of this proposed automatic

segmentation pipeline were an average surface distance of 0,7 +/- 0,3 mm and a maximum

distance of 16,5 +/- 5 mm. In this study however, the pelvis contains the sacrum. We can

conclude that our research is an extension to their study since they suggested to include an

evaluation of the inter-user variability, which would help to compare the errors made by the

automatic segmentation to those made by professionals and therefore the golden standard.

For the results, their ASD is slightly better than ours with a minimal difference of 0,05 mm, but

the maximum distance is bigger, which makes this study equivalent to ours (8).

Almeida D.F. and colleagues described a new, fully, automatic segmentation pipeline for CT

images of the femur. They used an ASM that is based on a SSM and a LAM (Local Appearance

Model). Almeida D.F. et al. described the results for low-resolution CT scans as well as high-

resolution CT scans. Since we only used high-resolution CT scans in our study, we do not

mention the results of the low-resolution CT scans. 148 datasets were segmented, but only 10

were considered for the comparative analysis (due to the time consuming task of manual

segmentation) and resulted in an average surface distance of 1,014 +/- 0,474 mm and a

Hausdorff distance of 4,336 +/- 0,861 mm. Compared to our automatic segmentation pipeline

results, the ASD of our study is much lower but the HD of this study is with a difference of 0,36

mm slightly better than ours (10).

Krcah M. et al. proposed a fully automatic segmentation technique of the femur. Basically, they

worked with a threshold-like technique (graph-cuts and a bone boundary filter) and in the last

step they separated the individual bones, since working with thresholding, the risk of leakage

to adjacent bones exists. They performed this automatic segmentation on 197 femurs and had

as result an average Hausdorff distance of 5,4 mm. With a difference of 0,6 mm, we can

conclude that our automatic segmentation pipeline is more accurate (27).

Uozumi Y. et al. described an automatic bone segmentation method which was tested on six

patients. They worked in three different steps: the segmentation between the femur and the

tibia, the segmentation of the femur and the patella and the segmentation of the tibia and fibula.

Their main segmentation method is an automatisation of the thresholding technique and they

42

too, compare the segmentations, performed by the automatic segmentation program to the

manual segmentation or the golden standard segmentation. But due to their use of matching

rate (in %) as validation metric, the comparison with our proposed method is not useful.

However, it seemed to be an added value to mention their results, as their study showed good

results and was published recently (2013). The matching rates were 95,84 +/- 0,57 % for the

femur, 94,12 +/- 1,01 % for the tibia, 94,49 +/- 0,83 % for the patella and 86,37 +/- 4,28 % for

the fibula. Further extension of our results to the DSI (Dice similarity index) would be necessary

to compare these results (26).

Younes L.B. and colleagues presented a fully automatic segmentation of the femur using SSM.

The first step was a primitive shape recognition. The presented method is divided into three

steps: detection of the femoral head and the femoral shaft as a sphere and as a cylinder

respectively, registration between primitive shapes of the SSM and CT image to initialise the

SSM into the image and at last, the fitting of the SSM tot the CT image. 8 CT data sets were

segmented and had as result an average surface distance of 1,48 +/- 0,28 mm and a maximum

error of 10,53 +/- 3,19 mm. Both values are much higher than ours, which makes this technique

not the most accurate one (11).

At last, Wu D. et al. proposed an automatic segmentation method, based on a combination of

different techniques. First, they used a marginal space learning for bone detection and

deformed it with a SSM. Second, the deformed model was used as a shape prior in a graph

cut for refined segmentation. At last, a multi-layer graph cut is used because there was a need

for segmenting each bone separately since it is possible that their results overlap. They tested

their proposed method on 248 data sets and the following results were obtained: an average

surface distance of 0,82 +/- 0,33 mm for the femur, an ASD of 0,69 +/- 1,25 mm for the tibia,

an ASD of 0,96 +/- 4,29 mm for the fibula and an ASD of 0,68 +/- 2,06 mm for the patella.

When we compare these results to ours, we can conclude that our automatic segmentation

pipeline is more accurate (9).

Furthermore, many other studies were not included in this comparison since they only

segmented a part of the bone. Often, those studies focused on segmenting joints like the hip

joint or the knee joint. Therefore, their results are not comparable with ours. Still, a short

summary of their results is made since these studies had a big impact on other articles. The

segmentation methods will not be extensively mentioned as these studies will not be used for

comparison.

43

Chu C. et al proposed a fully automatic CT segmentation method for the hip joint. The main

method is the use of a SSM and for validation, the manual segmentation was taken as the

golden standard. An ASD of 0,52 +/- 0,10 mm for the pelvis, an ASD of 0,45 +/- 0,10 mm for

the left proximal femur and an ASD of 0,48 +/- 0,08 mm for the right proximal femur, were

obtained as results (56). Secondly, Cheng Y. et al described an automatic segmentation

technique for the acetabulum and the femoral head. Their method is a combination of

thresholding and an iterative process based on neighbourhood information with an average

ASD of 1,22 +/- 0,98 mm and a DSI of 91,55 +/- 4,82 % as result (35). Thirdly, Ramme A.J.

and colleagues presented a semi-automatic segmentation method for the knee joint. They

tested their method on the distal femur and the proximal tibia of 72 CT data sets. The following

results are obtained: A DSI of 95% for the femur as well as the tibia (66). Further, Zoroofi R.A.

et al. presented an automatic segmentation method for the hip joint (acetabulum and femoral

head), using 60 CT data sets. They used a combination of different techniques including

thresholding and obtained the following result: an ASD of 1,31 +/- 1,12 mm and a DSI of 90,36

+/- 5,31% (35,65). Moreover, Yokota F. and colleagues proposed an automated segmentation

method for a diseased hip. They made use of different SSM’s and had an ASD of 1,49 +/- 1,04

mm and a DSI of 90,14 +/- 1,95 % as results (35,64).

The only study found researching an automatic segmentation technique for the calcaneus was

the study of Görres J. and colleagues. They performed a segmentation of two calcaneal

surfaces on the basis of an ASM on 50 cases and have shown an ASD of 0,59 and 0,46 mm

respectively (67).

Unfortunately, there is no literature found on the automatic segmentation of the full calcaneus

and the talus. Thus, the results of our calcaneus and talus segmentation are not mentioned in

Error! Reference source not found. since comparison is not possible.

After comparing the results of the previously mentioned studies, which are the most important

studies comparable with ours, we can conclude that our proposed automatic segmentation

method is the most accurate one with the best results. Our segmentation method is similar to

the one used by Seim H. et al. (8). Their results were also close to ours, what makes this

technique the most promising one.

44

4.3 Limitations

This study was confronted with some limitations that have to be taken into account. Firstly, the

statistical shape models that are used (the low and high resolution statistical shape models)

contain in the beginning only a couple of CT data sets. As explained above, the SSM obtains

more accuracy by adding more CT data sets. After all, the ASM is a measurement of the mean

shape of the CT data sets that are included in the training data set. And it is logical that in case

of a large training data set, the average shape will have a more accurate shape, that ‘knows’

a lot of abnormalities and variations of the normal anatomy and therefore can be used for the

whole population. In case of a small training data set, the ASM will not be able to fit in a CT

data set with abnormalities, which is not included in the training data set. Since the program

adds every new segmented CT data set to the ASM, the ASM will become more accurate when

more CT data sets are segmented by the pipeline.

Secondly, the small number of manually segmented CT data sets is also a limitation for the

validation of our program. The manual segmentation is a very time-consuming task and for

validation there is a need for many manual segmentations, as the comparison between the

manual and the automatic segmentation are the basis of the validation. Because of the time-

consuming nature of this work, only a few full lower limb segmentations were performed

manually. Therefore, it was only possible to perform a global estimate of accuracy. Regional

analysis could not be performed, since the number of manual segmentations was too low for

this type of analysis. In the end, all other studies found, did not perform a regional analysis

either.

Finally, to compare our newly developed program to previously developed automatic

segmentations techniques, a literature study has been performed. Multiple studies that

included the automatic segmentation of the pelvis and femur were found. But on the other

hand, a very small number of studies were found that included the tibia, the fibula and the

patella. Not one study was found relating to the automatic segmentation of the calcaneus, talus

or the full lower limb. For this reason, a comparison with other programs could not be

performed optimally.

4.4 Suggestions for further research

The current automatic segmentation technique can be extended to the full human skeleton.

This study only includes the lower limb, but could be extended to the upper limb, thorax and

skull. Also, the automatization of segmentation is not only useful in orthopaedics, but can be

45

used in gastroenterology, cardiology, pneumology, gynaecology, neurology, oncology, etc.

This extension would imply that different statistical shape models must be created for each

organ, bone or tissue and for the validation, a lot of manual segmentations must be performed

in order to compare them to the automatic segmentation. As well as the creation of the

statistical shape model as the manual segmentation are time-consuming tasks and therefore

the extension of this program will cost a lot of time. But if the same results can be obtained,

segmentation and therefore analysis of the whole human body will be easier and quicker.

In this study, only the CT is included. It would be interesting to expand the segmentation

pipeline to other imaging techniques in particular Magnetic Resonance Imaging technology.

MRI is currently the most used imaging technology for the knee joint and the spine. However,

MRI will confront the researcher with some new challenges due to the different nature of

contrast intensity. The fact that one could start with elaborate SSM’s based on CT images

could facilitate the step towards to MRI segmentation.

46

5 CONCLUSION

The interest in automatic segmentation techniques is rising all over the world in various medical

specialties. Image segmentation makes it possible to see the region of interest in 3D, which

yields a lot more information than the 2D image obtained from CT or MRI. The need of

automatic segmentation techniques is high, since manual image segmentation is a time-

consuming task that can only be performed by trained professionals. With an average surface

distance ranging from 0,53 mm to 0,76 mm and a Hausdorff distance ranging from 2,01 mm

to 7,84 mm, the accuracy of our automatic segmentation technique has been illustrated.

Furthermore, the errors made by the automatic segmentation technique were compared to the

errors made by the manual segmentation or golden standard, resulting in a minimal average

difference in error of 0,21 mm.

Several scientists and engineers presented their own automatic segmentation techniques and

described an average surface distance ranging from 0,4 mm to 5,4 mm (10,12,13,27,35,56).

The ASD differs according to the different techniques, number of patients and quality of CT

scan, which makes comparison difficult. However, taking the differences into account, an

attempt to compare various studies with ours is made. We can conclude that our method has

highly competitive results and therefore is the most accurate one. Besides the study of Seim

H. and colleagues (8), neither one of the mentioned studies presented results similar to ours.

On one hand, it is important to note that this study is the first of its kind developing an automatic

segmentation technique for the entire skeleton of the lower limb. Previous studies focused on

one bone or on a joint only (56,64,65). On the other hand, further research can be performed.

For example, extension to MRI would be interesting. Also, the extension to the full skeleton

and to other specialties like cardiology and neurology would be an added value.

This study, with several advantages of automatic segmentation, can even be of use in other

specialties and even in non-medical professions like crime investigation (e.g. face and finger

print recognition) (32). The automatization of image segmentation has a lot of advantages for

professionals, since time is always a precious element. It makes transformation to 3D images

possible and therefore it can facilitate operations or it can allow for a diagnosis in an earlier

stage.

47

6 REFERENCES

1. Schiefsky MJ. Hippocrates On ancient medicine. Brill; 2005. 415 p. 2. O’malley CD. ANDREAS VESALIUS 1514-1564: IN MEMORIAM. Med Hist. 1964

Oct;8:299–308. 3. Shapiro L, Stockman G. Computer Vision. October. 2001. 608 p. 4. Saini S, Arora K. A Study Analysis on the Different Image Segmentation Techniques.

Int J Inf Comput Technol. 2014;4(14):1445–52. 5. Wikipedia, the free encyclopedia: Image segmentation [Internet]. [cited 2017 Oct 10].

Available from: https://en.wikipedia.org/wiki/Image_segmentation 6. Henak CR, Anderson AE, Weiss JA. Subject-Specific Analysis of Joint Contact

Mechanics: Application to the Study of Osteoarthritis and Surgical Planning. J Biomech Eng. 2013 Feb 11;135(2):21003.

7. Lamecker H, Seebass M, Hege H-C, Deuflhard P. A 3D statistical shape model of the pelvic bone for segmentation. In 2004. p. 1341.

8. Seim H, Kainmueller D, Heller M, Lamecker H, Zachow S, Hege H-C. Automatic Segmentation of the Pelvic Bones from CT Data Based on a Statistical Shape Model. In: Proceedings of the First Eurographics conference on Visual Computing for Biomedicine. 2008. p. 93–100.

9. Wu D, Sofka M, Birkbeck N, Zhou SK. Segmentation of multiple knee bones from CT for orthopedic knee surgery planning. Med Image Comput Comput Assist Interv. 2014;17(Pt 1):372–80.

10. Almeida DF, Ruben RB, Folgado J, Fernandes PR, Audenaert E, Verhegghe B, et al. Fully automatic segmentation of femurs with medullary canal definition in high and in low resolution CT scans. Med Eng Phys. 2016 Dec;38(12):1474–80.

11. Younes BL, Nakajima Y, Saito T. Fully automatic segmentation of the Femur from 3D-CT images using primitive shape recognition and statistical shape models. Int J Comput Assist Radiol Surg. 2014 Mar 8;9(2):189–96.

12. Almeida D, Folgado J, Fernandes PR, Ruben RB. 3D shape prior active contours for an automatic segmentation of a patient specific femur from a CT scan. In: Computational Vision and Medical Image Processing IV. 2014. p. 271–5.

13. Liu B, Zhang H, Hua S, Jiang Q, Huang R, Liu W, et al. An automatic segmentation system of acetabulum in sequential CT images for the personalized artificial femoral head design. Comput Methods Programs Biomed. 2016;127:318–35.

14. Kishner S, Chowdhry M. Hip Joint Anatomy: Overview, Gross Anatomy [Internet]. 2015 [cited 2017 Sep 30]. Available from: http://emedicine.medscape.com/article/1898964-overview#a2

15. Kishner S, Courseault J, Authement A. Knee Joint Anatomy: Overview, Gross Anatomy, Natural Variants [Internet]. 2015 [cited 2017 Sep 30]. Available from: http://emedicine.medscape.com/article/1898986-overview#a2

16. Kerckaert I. Locomotorisch stelsel en huid, Deel III: Osteologie. Gent: Academia Press; 2013. 58 p.

17. D’Herde K, Schoemaker E, Veening J. Sesam Atlas van de anatomie deel 1: Bewegingsapparaat. 10th ed. Amersfoort: ThiemeMeulenhoff; 2012. 467 p.

18. Paulsen F, Waschke J. Sobotta Atlas of Human Anatomy: General anatomy and musculoskeletal system. 15th ed. Munich: Elsevier GmbH,Urban & Fischer; 2011. 400 p.

19. Chapter 1: Introduction to CT physics [Internet]. 2004 [cited 2017 Oct 19]. p. 16. Available from: https://web.archive.org/web/20070926231241/http://www.intl.elsevierhealth.com/e-books/pdf/940.pdf

20. Understanding How MRI, CT and X-ray Are Different | CDI, Insight Imaging &amp; Affiliated Providers [Internet]. [cited 2017 Oct 2]. Available from: https://www.mycdi.com/knowledge_center/difference_between_mri_ct_and_x-ray/

48

21. Computed Tomography (CT) | National Institute of Biomedical Imaging and Bioengineering [Internet]. [cited 2017 Oct 2]. Available from: https://www.nibib.nih.gov/science-education/science-topics/computed-tomography-ct

22. Thurston M, Nadrljanski MM. Computed tomography | Radiology Reference Article | Radiopaedia.org [Internet]. [cited 2017 Oct 3]. Available from: https://radiopaedia.org/articles/computed-tomography

23. Seeram E. Computed tomography : physical principles, clinical applications, and quality control. 487 p.

24. X-rays | National Institute of Biomedical Imaging and Bioengineering [Internet]. [cited 2017 Oct 2]. Available from: https://www.nibib.nih.gov/science-education/science-topics/x-rays

25. Matej “Retro” J. Pixels and voxels, the long answer – Retronator Magazine – Medium [Internet]. 2014 [cited 2017 Oct 3]. Available from: https://medium.com/retronator-magazine/pixels-and-voxels-the-long-answer-5889ecc18190

26. Uozumi Y, Nagamune K. An automatic bone segmentation method based on anatomical structure for the knee joint in MDCT image. In: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE; 2013. p. 7124–7.

27. Krcah M, Szekely G, Blanc R. Fully automatic and fast segmentation of the femur bone from 3D-CT images with no shape prior. In: 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. IEEE; 2011. p. 2087–90.

28. Hosseini M-P, Nazem-Zadeh MR, Pompili D, Soltanian-Zadeh H. Statistical validation of automatic methods for hippocampus segmentation in MR images of epileptic patients. Conf Proc . Annu Int Conf IEEE Eng Med Biol Soc IEEE Eng Med Biol Soc Annu Conf. 2014;2014:4707–10.

29. Gonzalez RC, Woods RE. Digital image procession. 3rd ed. Horton Marcia J, McDonald Michael, Dworkin Alice, Disanno Scott, Kernan Rose, editors. Pearson Prentice Hall; 2008. 943 p.

30. Zaitoun NM, Aqel MJ. Survey on Image Segmentation Techniques. Procedia Comput Sci. 2015;65:797–806.

31. Umaa Mageswari S, Sridevi M, Mala C. An Experimental Study and Analysis of Different Image Segmentation Techniques. Procedia Eng. 2013;64:36–45.

32. Dass R, Priyanka, Devi S. Images Segmentation Techniques. IJECT. 1993;3(1):66–70. 33. Kang W-X, Yang Q-Q, Liang R-P. The Comparative Research on Image Segmentation

Algorithms. In: 2009 First International Workshop on Education Technology and Computer Science. IEEE; 2009. p. 703–7.

34. Pham DL, Xu C, Prince JL. Current Methods in Medical Image Segmentation. Annu Rev Biomed Eng. 2000 Aug;2(1):315–37.

35. Cheng Y, Zhou S, Wang Y, Guo C, Bai J, Tamura S. Automatic segmentation technique for acetabulum and femoral head in CT images. Pattern Recognit. 2013 Nov;46(11):2969–84.

36. Saleh Al-Amri S, Kalyankar N V, D KS. Image Segmentation by Using Thershod Techniques. J Comput. 2010;2(5):2151–9617.

37. Otsu N. A Threshold Selection Method from Gray-Level Histograms. IEEE Trans Syst Man Cybern. 1979 Jan;9(1):62–6.

38. Daniel P, Raju R, Neelima G. Image Segmentation by using Histogram Thresholding. IJCSET. 2012;2(1):776–9.

39. Zhang J, Yan C-H, Chui C-K, Ong S-H. Fast segmentation of bone in CT images using 3D adaptive thresholding. Comput Biol Med. 2010 Feb;40(2):231–6.

40. Pal NR, Pal SK. A REVIEW ON IMAGE SEGMENTATION TECHNIQUES. Pattern Recoynition. 1993;26(9):1277–94.

41. Kang Y, Engelke K, Kalender WA. A new accurate and precise 3-D segmentation method for skeletal structures in volumetric CT data. IEEE Trans Med Imaging. 2003 May;22(5):586–98.

42. Jacobs D. Image Gradients: Class Notes for CMSC 426. 2005 p. 3.

49

43. Kaur N, Singh J, Sharma V. Analysis and Comprehensive Study: Image Segmentation Techniques. 2015;2321–9653.

44. Heena A, Biradar N, Maroof NM. A Novel Approach to Review Various Image Segmentation Techniques. Int J Innov Res Comput Commun Eng An ISO Certif Organ. 2007;32975(2).

45. Khan AM, Ravi S. Image Segmentation Methods: A Comparative Study. Int J Soft Comput Eng. 2013;3(34):2231–307.

46. Sarkalkan N, Weinans H, Zadpoor AA. Statistical shape and appearance models of bones. Bone. 2014 Mar;60:129–40.

47. Heimann T, Meinzer H-P. Statistical shape models for 3D medical image segmentation: A review. Med Image Anal. 2009 Aug;13(4):543–63.

48. Cootes TF, Taylor CJ. Statistical Models of Appearance for Computer Vision. 2004. 49. Cootes TF, Taylor CJ, Cooper DH, Graham J. Active Shape Models - Their Training and

Application. Comput Vis image Underst. 1995;61(1):35–59. 50. Morse BS. Lecture 21: Image Understanding. 2000 p. 6. 51. Pichumani R. Snakes: an active model [Internet]. 1997 [cited 2017 Oct 25]. Available

from: http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/RAMANI1/node31.html

52. Kass M, Witkin A, Terzopoulos D. Snakes: Active contour models. Int J Comput Vis. 1988;321–31.

53. Bebis G. Deformable/Active Contours (or Snakes). 54. Khelif M, Derraz F, Beladgham M. Application of Active Contour Models in Medical

Image Segmentation. Int Conf Inf Technol Coding Comput 2004 Proceedings ITCC 2004. 2004;2:675–81.

55. Xu C, Prince JL. Gradient Vector Flow: A New External Force for Snakes. IEEE Proc Conf Comp Vis Patt Recog. 1997;66–71.

56. Chu C, Chen C, Liu L, Zheng G. FACTS: Fully Automatic CT Segmentation of a Hip Joint. Ann Biomed Eng. 2015 May 4;43(5):1247–59.

57. Yeghiazaryan V, Voiculescu I. An Overview of Current Evaluation Methods Used in Medical Image Segmentation. 2015;21.

58. 3D Medical Image Processing Software | Materialise Mimics [Internet]. [cited 2017 Nov 1]. Available from: http://www.materialise.com/en/medical/software/mimics

59. Paz ICL A, Ldg B, Bruno L. Bone mineral density: review. Brazilian J Poult Sci Rev Bras. 2006;8(2):69–73.

60. McClung MR. The relationship between bone mineral density and fracture risk. Curr Osteoporos Rep. 2005 Jun;3(2):57–63.

61. Assessment of Fracture Risk [Internet]. 2005 [cited 2017 Nov 20]. Available from: https://www.medscape.org/viewarticle/506083_2

62. Weisstein EW. Root-Mean-Square [Internet]. Wolfram Research, Inc.; [cited 2017 Oct 20]. Available from: http://mathworld.wolfram.com/Root-Mean-Square.html

63. Huttenlocher DP, Klanderman GA, Rucklidge WJ. Comparing Images Using the Hausdorff Distance. IEEE Trans Pattern Anal Mach Intell. 1993;15(9):850–63.

64. Yokota F, Okada T, Takao M, Sugano N, Tada Y, Sato Y. Automated segmentation of the femur and pelvis from 3D CT data of diseased hip using hierarchical statistical shape model of joint structure. Med Image Comput Comput Assist Interv. 2009;12(Pt 2):811–8.

65. Zoroofi RA, Sato Y, Sasama T, Nishii T, Sugano N, Yonenobu K, et al. Automated segmentation of acetabulum and femoral head from 3-D CT images. IEEE Trans Inf Technol Biomed. 2003 Dec;7(4):329–43.

66. Ramme AJ, Criswell AJ, Wolf BR, Magnotta VA, Grosland NM. EM segmentation of the distal femur and proximal tibia: A high-throughput approach to anatomic surface generation. Ann Biomed Eng. 2011;39(5):1555–62.

67. Görres J, Brehler M, Franke J, Vetter SY, Grützner PA, Meinzer HP, et al. Articular surface segmentation using active shape models for intraoperative implant assessment. Int J Comput Assist Radiol Surg. 2016;11(9):1661–72.