CAS, Srping 2002 © L. Joskowicz 3
CAS clinical applications
• Neurosurgery
• Orthopaedics
• Maxillofacial, craneofacial, and dental surgery
• Laparoscopic and endoscopic surgeries
• Radiotherapy
• Specific procedures in ophtalmology, othorhinolaringology, etc.
CAS, Srping 2002 © L. Joskowicz 4
Elements of CAS systems
CAS, Srping 2002 © L. Joskowicz 5
Technical elements of CAS systems1. Medical images
2. Medical image visualization
3. Segmentation and modeling
4. Virtual and augmented reality, tele-surgery
5. Preoperative analysis and planning
6. Image and robot registration
7. Medical mechanical and robotics systems
8. Real-time tracking
9. Safety, man-machine interface, human factors
CAS, Srping 2002 © L. Joskowicz 6
Key parameters for understanding and comparing solutions
• How many procedures are performed yearly?
• What is the rate of complications? What are their causes?
• In what aspects can a CAS system help?
• Does it address part of a clinically important problem?
• What stage is the system in: in-vitro, cadaver, clinical trials?
1. Medical Images
CAS, Srping 2002 © L. Joskowicz 8
Most common imaging modalities
• Film X-ray, Digital X-ray, Fluoroscopy, Digital Substraction Angiography (DSA)
• Ultrasound -- 2D and 2.5D (stack of slices)
• Computed Tomography (CT)
• Magnetic Resonance Imaging (MRI)
• Nuclear Medicine (NM) – PET -- Positron Emission Tomography– SPECT -- Single Photon Emission Tomography
CAS, Srping 2002 © L. Joskowicz 9
Medical images: characteristics (1)• Preoperative or intraoperative use
– depends on the size and location of imaging machine
• Dimensionality: 2D, 2.5D, 2D+time– projection, cross section, stack of projections, time
sequence
• Image quality– pixel intensity and spatial resolution– amount of noise; signal/noise ratio– spatial distortions and intensity bias
CAS, Srping 2002 © L. Joskowicz 10
Medical images: characteristics (2)
• Field of view
• Radiation to patient and to surgeon
• Functional or anatomical imaging– neurological activity, blood flow, cardiac activity
• What it’s best at for– bone, soft tissue, fetus, surface/deep tumors, etc
• Clinical use– diagnosis, surgical, navigation,
CAS, Srping 2002 © L. Joskowicz 11
X-ray images• Measure absorption of x-ray radiation from
source to set of receptors
• Film X-ray has very high resolution
Gray value proportionalto radiation energy
CAS, Srping 2002 © L. Joskowicz 12
X-ray Fluoroscopy
CAS, Srping 2002 © L. Joskowicz 13
Fluoroscopic images
CAS, Srping 2002 © L. Joskowicz 14
X-ray image properties• Traditional, cheap, widely available
• Two-dimensional projections (at least two required)
• High resolution, low noise (more fluoroscope)– film size, 64K gray levels– fluoroscopic images: TV quality, 20cm field of view
• Relatively low radiation
• Bone and metal images very well
• Fluoroscopy used for intraoperative navigation
CAS, Srping 2002 © L. Joskowicz 15
Ultrasound imaging (US)
• Measure refraction properties of an ultrasound wave as it hits tissue
• No radiation
• Poor resolution, distortion, noise
• Low penetration properties
• One 2D slice or several slices (2.5D)
• Relatively cheap and easy to use
• Preoperative and intraoperative use
CAS, Srping 2002 © L. Joskowicz 16
Ultrasound imaging
CAS, Srping 2002 © L. Joskowicz 17
Computed Tomography (CT)
CAS, Srping 2002 © L. Joskowicz 18
Computed Tomography Images
cuts
d = 35mmd = 25mm
d = 15mmd = 5mm
CAS, Srping 2002 © L. Joskowicz 19
Computed Tomography Principle
angle
intensity
X-rays
CAS, Srping 2002 © L. Joskowicz 20
Computed Tomography Properties
• Sepcifications:– 512x512 12bit gray level images; pixel size 0.5mm– slice interval 1-10mm depending on anatomy– 50-200 slices per study– noise in the presence of metal (blooming)
• All digital, printed on X-ray film
• Acquisition 1sec/slice (spiral models)
• 15mins for image reconstruction
• Costs about $250-750K, each study $500
CAS, Srping 2002 © L. Joskowicz 21
Magnetic Resonance Imaging• Similar principle and construction than CT
machine, but works on magnetic properties of matter– magnetic fields of 0.1 to 4 Teslas
• Similar image quality characteristics as CT
• Excellent resolution for soft tissue
• Costs $1-2M, each study $1,000
• Open MR: intraoperative device (only 15 to date)
CAS, Srping 2002 © L. Joskowicz 22
Magnetic Resonance Images
CAS, Srping 2002 © L. Joskowicz 23
Nuclear Medicine Imaging (NMI)
• Same slices principle
• Source of photons or positrons is injected in the body. Shortly after, radiation of metabolism is measured
• Poor spatial resolution
• Expensive machine AND installation ($4-5M)
• Expensive and time-consuming
• Provides functional info no other source does
CAS, Srping 2002 © L. Joskowicz 24
Nuclear medicine images
CAS, Srping 2002 © L. Joskowicz 25
Image Fusion: MRI and NMI
MRI (anatomy) NMI (functional)
CAS, Srping 2002 © L. Joskowicz 26
Video images from within the body
• Used in laparoscopic and endoscopic surgery
CAS, Srping 2002 © L. Joskowicz 27
Main medical imaging modalities
X-ray X-ray Fluoro US US Video CT MRI NMR OpenFilm Digital (2D) (2.5D) MR
Pre/Intraop2D/2.5DResolutionRadiationAnatomyProcedure
Establish a comparative table of modality properties
CAS, Srping 2002 © L. Joskowicz 28
The imaging pipeline
CAS, Srping 2002 © L. Joskowicz 29
2. Medical image visualization
• 3D visualization of complex structures
• image correlation and fusion
• quantitative measurements and comparisons
• visualization of medical and CAD data
Enhance diagnosis by improving the visualinterpretation of medical data
CAS, Srping 2002 © L. Joskowicz 30
Medical image visualization
CAS, Srping 2002 © L. Joskowicz 31
Visualization: Technical needs
• image enhancing and noise reduction
• image interpolation: images from new viewpoints
• 3D visualization from 2.5D data– volume rendering: display voxels and opacity values– surface rendering: explicit reconstruction of surface
• 3D modeling from 2.5D data
• 2D and 3D segmentation
• 3D+T visualization (beating heart)
CAS, Srping 2002 © L. Joskowicz 32
Medical image visualization
• Much activity! Radiologists are the experts
• Commercial packages– 3DVIEWNIX, ANALYZE, IMIPS
• Main technical topics:– 3D volume rendering techniques– 3D image filtering and enhancement – surface construction algorithms: Marching cubes, etc.
• Sources: chapters 3,9, and 10 in textbook
• Related fields: computer graphics, image processing
CAS, Srping 2002 © L. Joskowicz 33
3. Segmentation and modeling
• Isolation of relevant anatomical structures based on pixel properties
• Model creation for the next computational task– real-time interaction and visualization– simulation– registration, matching, – morphing
Extract clinically useful informationfor a given task or procedure
CAS, Srping 2002 © L. Joskowicz 34
Segmentation and modeling
CAS, Srping 2002 © L. Joskowicz 35
Segmentation and modeling: technical needs
• Segmentation:– landmark feature detection– isosurface construction (Marching cubes)– contour extraction, region identification
• Modeling:– points, anatomical landmarks, surface ridges– surfaces as polygon meshes, surface splines– model simplification methods (Alligator, Wrapper)
CAS, Srping 2002 © L. Joskowicz 36
Segmentation and modeling
• Medical images have very special needs!
• Commercial packages– 3DVIEWNIX, ANALYZE, IMIPS
• Main technical topics:– Volumetric segmentation techniques for CT, MRI– 2D and 3D segmentation with deformable elements – surface and model simplification algorithms
• Sources: chapters 4 and 8 in textbook
• Related fields: image processing, computer vision
CAS, Srping 2002 © L. Joskowicz 37
4. Virtual and augmented reality
• Create a virtual model for viewing during surgery
• Project the model on the patient or integrate with surgeon’s view
• Useful for intraoperative anatomy exploration and manipulation
• Telesurgery systems
Use images to create or enhance a surgical situation
CAS, Srping 2002 © L. Joskowicz 38
Virtual and augmented reality
CAS, Srping 2002 © L. Joskowicz 39
Virtual and augmented reality• Part manipulation, visual and sensory feedback
• Interaction devices: goggles, gloves, etc
• Only a handful of systems exist
• Main technical topics:– a couple of the working systems; simulators– telesurgery systems
• Sources: chapters 14 and 15 in textbook
• Related fields: computer graphics
CAS, Srping 2002 © L. Joskowicz 40
5. Preoperative analysis and planning
• Task and procedure dependent
• Spatial and volume measurements
• Stress and fracture analysis
• Implant and tool selection and positioning
• Surgical approach planning: bone rearrangement, angle evaluation, radiation dose planning, etc
Use images and models to assist surgeons in planning a surgery and evaluate options
CAS, Srping 2002 © L. Joskowicz 41
Preoperative analysis and planning
CAS, Srping 2002 © L. Joskowicz 42
Preoperative analysis and planning• About a dozen planners exist for different
procedures
• Main technical topics:– planning systems for orthopaedics, neurosurgery– application of engineering analysis techniques: finite-
element methods, stress analysis, etc
• Sources: chapters 11, 25, 33, 41, 52--56in textbook
• Related fields: CAD, computational geometry, engineering analysis
CAS, Srping 2002 © L. Joskowicz 43
6. Image and robot registration
• Define correspondance features– point-to-point, point-to-line, surface-to-surface
• Establish correspondances between features
• Establish a similarity measure
• Formulate and solve dissimilarity reduction problem
• Related tasks: image fusion, morphing, atlas matching
Establish a quantitative relation between different refererence frames
CAS, Srping 2002 © L. Joskowicz 44
Multimodal registration problems• Great differences depending on
– the type of data to be matched– the anatomy that is being imaged– the specific clinical requirements of procedures
• Feature selection and extraction: stereotactic frame, implanted fiducials, anatomical landmarks and surfaces, contours and surfaces in
• Manual vs. automatic feature selection, pairing• Rigid vs. deformable registration• Nearly similar vs. dissimilar images• Noiseless vs. noisy images (outlier removal)
CAS, Srping 2002 © L. Joskowicz 45
Registration chain example
3D surface model
X-rays
CT
Patient
Tracker
Instruments
Infrared tracker
CAS, Srping 2002 © L. Joskowicz 46
Image and robot registration• Rich topic of very great importance!
• Types of registration methods vary widely
• Main technical topics:
• rigid registration methods: three points and more– deformable registration: local and global methods– intensity-based registration
• Sources: chapters 5-7 in textbook, many papers
Book on Medical Image Registration
• Related fields: vision, robotics
CAS, Srping 2002 © L. Joskowicz 47
7. Medical robotics devices
• Task and procedure dependent
• Accurate, steady, and repeatable 3D positioning
• Navigation and localization aids
• Cutting and milling, biopsies
• Key issues are:– kinematic design, trajectory planner– controller, safety provisions
Semi-active and active mechanicaldevices for improving surgical outcome
CAS, Srping 2002 © L. Joskowicz 48
Medical robotics devices
CAS, Srping 2002 © L. Joskowicz 49
Medical robotics devices
• Mostly passive and semiactive devices
• Rich topic of very great importance!
• Main technical topics:– compare features and functionalities of systems– discuss and compare design considerations – devices for specific surgeries laparoscopy)
• Sources: chapters 16-18, 22, 29, 34, 39, 45, 47, and 48 in textbook
• Related fields: robotics, mechatronics
CAS, Srping 2002 © L. Joskowicz 50
8. Real-Time tracking devices
• Ideally, an accurate Global Positioning System!
• Current technologies offer only partial solution
• Based on different principles– video: follow known objects– optical: follow light-emitting diodes– magnetic: measure the variation of – acoustic: works like a radar
Hardware to follow in real time the precise position and orientation of anatomy and instruments during surgery
CAS, Srping 2002 © L. Joskowicz 51
Optical and video tracking devicescamera
instrument
Passive markers
Instrument has infrared LEDs attached to it Active markers
CAS, Srping 2002 © L. Joskowicz 52
What kind of accuracy?
CAS, Srping 2002 © L. Joskowicz 53
9. Safety, man-machine interfaces
• Medical systems have very stringent safety requirements
• Reported cases of radiation overdose due to faulty system design
• Important issues in man-machine interfaces
• Ideas for presentations– the radiotherapy accident– chapters 12-15 and 19 in textbook
CAS, Srping 2002 © L. Joskowicz 54
10. Systems integration• Complete systems that address specific clinical
problems in domains
• Use available technology to develop the system
• The hard part: make it all work!
• Main technical topics:– systems in orthopaedics, neurosurgery, etc
• Sources: chapters in each section of
• Related fields: all!