Image and Video Processing with MATLAB.pdf

1© 2011 The MathWorks, Inc.

Image and Video Processing Using

MATLAB

April 12, 2011

Rochester, NY

2

Agenda

9:00 a.m. Welcome and MathWorks Overview

9:15 a.m. Image Processing with MATLAB

10:30 a.m. Break

10:45 a.m. Video Processing with MATLAB

11:30 a.m. System Design and Hardware Targeting

12:00 p.m. Summary and Wrap-up

3

From MathWorks

Sr. Account Manager

– Ed Donovan

– [email protected]

– 508-647-4251

Application Engineer

– Stephanie Kwan

mailto:[email protected]

4

Agenda



10:30 a.m. Break




5

Where is image processing used?

6

Common Image Processing Tasks

Perform segmentation

Deblur and remove noise

Measure image features

Perform edge detection

Apply morphological operations

Enhance image contrast

Perform image registration

Apply transforms

7

Common Image Processing Challenges

Reading and writing to various file formats

Visualizing images and intermediate results

Testing algorithms with what-if scenarios

Identifying causes of algorithm failure

Processing large images with limited memory

Executing algorithms faster

8

Technical Computing Tasks

Reporting and

Documentation

Outputs for Design

Deployment

ShareExplore and Discover

Data Analysis

and Modeling

Algorithm

Development

Application

Development

Files

Software

Hardware

Access

Code and Applications

Automate

9

Explore and Discover

MATLAB – A Platform for Technical Computing

Reporting and

Documentation

Outputs for Design

Deployment

Share

Data Analysis

and Modeling

Algorithm

Development

Application

Development

Files

Software

Hardware

Access

Automate


10

Core MathWorks Products

The leading environment for

technical computing

The industry-standard, high-level

programming language

for algorithm development

Numeric computation

Data analysis and visualization

Toolboxes for signal and image

processing, statistics, optimization,

symbolic math, and other areas

Foundation of MathWorks products

11

Image Processing Toolbox

Perform image processing, analysis, visualization, and algorithm development

Image analysis

Image enhancement

Spatial transformation

Image registration

Morphological operations

ROI-based processing

Image display and exploration

12

Demo: Camera PipelineFrom Sensor Data to Image

Scene Image

13

1. Noise Reduction

2. Demosaic

3. Tone Mapping

4. White Balance

5. Gamma Correction

Demo: Camera Pipeline

14

Image Acquisition Toolbox

Acquire images and video directly into

MATLAB and Simulink

Configure device properties

Perform background acquisition

Synchronize multimodal devices

Configure, acquire, and preview live

video data using a graphical interface

15

Image Acquisition Toolbox Hardware Support

Industry standard support:

– Frame grabbers

Analog

Camera Link

– DCAM compatible FireWire (IIDC 1394)

– GigE Vision

– Common OS webcam interfaces

Operating system support:

– Windows

– Linux

– Macintosh

16

Image Acquisition Toolbox Hardware Support

Manufacturers include:

– Allied Vision Technologies

– Basler

– Baumer

– DALSA

– FLIR

– Hamamatsu

– Lumenera

– Matrox Imaging

– National Instruments

– PixeLINK

– Point Grey

– Qimaging

– Sony

See Supported Hardware Pages for more information

17

Where is Large Imagery Used?

Satellite imagery

Aerial surveys

Super-resolution

Image sequences and stacks

Volumetric data

Multispectral and hyperspectral

Mosaics and panoramic imagery

18

Common Large Image Challenges

“Out of memory” errors

Slow processing

19

Some Steps to Handle Large Images

Try memory command to see available space

Preallocate space for variables

Minimize numerical data types

Load small portions of an image

Visualize only what is necessary

Incorporate more hardware resources

Can lead to

complex code

20

blockproc and rsetwrite

Block processing

– Automatically divides an

image into blocks for

processing

– Reduces memory usage

– Processes arbitrarily large

images

Reduced resolution data set

– Avoids memory demands in

visualizing large images

21

Demo: Cell Segmentation with blockproc

22

Improving Performance

MATLAB

– Preallocate space for variables

– Identify bottlenecks with Profiler

– Vectorize code


– Many toolbox functions are faster as of R2010a

Parallel Computing Toolbox

– Utilize multicore processors, clusters, and GPUs

23

Worker Worker

Worker

Worker

WorkerWorker

Worker

Worker


Going Beyond Serial MATLAB Applications

TOOLBOXES

BLOCKSETS

24

Parallel Computing enables you to …

Larger Compute Pool Larger Memory Pool

11 26 41

12 27 42

13 28 43

14 29 44

15 30 45

16 31 46

17 32 47

17 33 48

19 34 49

20 35 50

21 36 51

22 37 52

Speed up Computations Work with Large Data

25

Distributing Tasks (Task Parallel)

Time Time

Pro

cesses

26

Demo: Camera Pipeline and Parallel Computing

27

Parallel Computing Toolbox Support for GPUs

Automatically use GPU’s with supported built-in

MATLAB functions– conv2, filter2

Run supported MATLAB code on the GPU– arrayfun

Create kernals from existing CUDA code and PTX files

Run kernals on the GPU from MATLAB

28

Why would you want to use a GPU?

Speed up execution of computationally intensive

simulations

For example:

– Performance: A\b with Double Precision

29

Why use MATLAB for Image Processing?

Read and write many image file formats

Visualize and explore images interactively

Connect directly to cameras and frame grabbers

Use a large library of built-in functions

Quickly build custom image processing algorithms

Block-process large images to avoid memory issues

Process images faster with multiple cores and clusters

30

Agenda



10:30 a.m. Break




31

Agenda



10:30 a.m. Break




32

Common Video Processing Tasks

Convert between color spaces

Enhance video contrast

Perform segmentation

De-interlace video frames

Calculate running statistics

Overlay text and graphics

33

Common Video Processing Challenges

Reading and writing to various file formats

Managing buffers and intermediate storage

Testing video algorithms with test cases

Accessing standard video algorithms

Visualizing results and intermediate steps

Displaying text and graphics

34

Demo: Working with Video in MATLAB

Read in and write out data using VideoReader and

VideoWriter

What happens if the video file is very long or the frame

size is large?

35

System Objects

MATLAB objects that represent algorithms and I/O

capabilities that process data frame by frame

Instantiate and configure– reader = vision.MultimediaFileReader(‘viptraffic.avi’)

Execute within a loop– step(reader)

Standard API– reset

– isDone

– close

36

Video Processing in MATLAB

Need to maintain buffer

Explicit indexing

Explicit state management

myVid = VideoReader(‘myvideofile.avi’);

numFrames = myVid.NumberOfFrames;

numIter = 10;

opticalFlowIn = zeros([size(currentFrame) 5]);

opticalFlowOutput = zeros([size(currentFrame) numFrames]);

i = 1;

while i <= numFrames

opticalFlowIn(:,:,2:end) = opticalFlowIn(:,:,1:end-1);

opticalFlowIn(:,:,1) = read(myVid,i);

flow = opticalFlow(opticalFlowIn(:,:,1),opticalFlowIn(:,:,5),…

‘horn-schunck’,numIter,‘magitude-squared’);

opticalFlowOutput(:,:,i) = flow;

i = i+1;

end

implay(opticalFlowOutput,30)

37

Video Processing with System Objects

Initialize objects

“In-the-loop” code is much simpler

Implicit states, buffering, and indexing

Video player runs in-the-loop

reader = vision.VideoFileReader

reader.Filename = ‘myvideofile.avi’;

viewer = vision.DeployableVideoPlayer(‘framerate’,30);

optical = vision.OpticalFlow

optical.Method = ‘horn-schunck’;

optical.OutputValue = ‘Magitude-squared’;

optical.ReferenceFrameDelay = 3;

optical.MaximumIterationCount = 10;

while ~isDone(reader)

currentFrame = step(reader);

OF = step(optical, currentFrame);

step(viewer, OF);

end

38

Demo: Car Detection Using Optical Flow

Use optical flow to detect and count moving

vehicles on a road

39

Motion Estimation

Motion estimation applications:

Object tracking

Segmentation

Interpolation

Compression

Many methods include:

Optical flow

Block matching

Template matching

40

Demo: Interpolation with Block Matching

Use block matching to perform motion estimation

and interpolate video

41

Computer Vision System Toolbox

Design and simulate computer vision and video

processing systems

Read and write multimedia files

Display videos

Overlay text and graphics

Perform pre- and post-processing

Perform motion-based processing

Detect and track objects

Recognize features

Design computer vision systems

42

What is Computer Vision?

Automating the process of extracting information from

images and video to understand a real world scene

Computer

VisionInterpretation

Detect

Identify

Classify

Recognize

Track

Pedestrian

Bicyclist

Truck

Car

Traffic violation

Accident

Image

Processing

Remove noise

Adjust contrast

Measure

…

43

Common Computer Vision Applications

Detect and track objects

Automatic target recognition

Count objects in a scene

Classify or cluster objects

Automatically register images

Create mosaics and panoramas

Perform stereo vision

44

Statistics Toolbox

Perform statistical analysis, modeling,

and algorithm development

Principle components analysis

K-means clustering

Gaussian mixture models

Naïve Bayes classification

K-nearest neighbor search

Support vector machines

Boosted decision trees

45

Why use MATLAB for video processing and

computer vision?

Read and write to many video file formats

Manage buffers, indexing, and states with System objects

Integrate live data from cameras and frame grabbers

Access algorithms for video processing and computer vision

Overlay text and graphics annotations on video data

Provides tools for designing computer vision systems

46

Agenda



10:30 a.m. Break


11:30 a.m. System Design and Implementation


47

Technical Computing Tasks

Reporting and

Documentation

Outputs for Design

Deployment

ShareExplore and Discover

Data Analysis

and Modeling

Algorithm

Development

Application

Development

Files

Software

Hardware

Access


Automate

48

MATLAB Compiler

COM

ExcelAdd-in

Javawww

.NET.exe.dll

.lib

COM ExcelAdd-in

Javawww

MATLABBuilder EX

MATLABBuilder NE

MATLABBuilder JA

.NET.exe.dll

.lib

MATLAB Compiler

From MATLAB to Implementation

C

Code

Simulink

Coder

HDL

Code

C

Code

MATLAB Coder

.exe.dll

.mex

SimulinkHDL Coder

Simulink & Stateflow

49

MATLAB Compiler

Automatic conversion of your MATLAB programs into

standalone applications and shared libraries

Support for the full MATLAB language and most

toolboxes

Royalty-free deployment

50

MATLAB Coder

Generate C and C++ Code from MATLAB Code

Maintain one design in MATLAB

Design faster and get to C/C++ quickly

Verify behavior of generated code

Accelerate computationally intensive

portions of MATLAB code

Test systematically and frequently

Spend more time improving algorithms

in MATLAB

Compatible with Computer Vision

System Toolbox

51

Deploy MATLAB algorithms on Windows/Linux desktop PC

Integrate MATLAB algorithms with existing C code

Hand-off code to software engineers for embedded

processor implementation

Accelerate user-written MATLAB algorithms

Common MATLAB to C Tasks

52

Broad support for Features and Functions

Broad set of language features and functions/system

objects supported for code generation.

Matrices and

ArraysData Types

Programming

ConstructsFunctions

• Matrix operations

• N-dimensional

arrays

• Subscripting

• Frames

• Persistent

variables

• Global variables

• Complex numbers

• Integer math

• Double/single-

precision

• Fixed-point

arithmetic

• Characters

• Structures

• Numeric classes

• Variable-sized data

• System objects

• Arithmetic,

relational, and

logical operators

• Program control

(if, for, while,

switch )

• MATLAB functions and sub-

functions

• Variable length argument lists

• Function handles

Supported algorithms

• > 400 MATLAB operators and

functions

• > 200 System objects for

• Signal processing

• Communications

• Computer vision

53

Demo: Camera Pipeline Revisited

Generate C code from MATLAB code

54

Targeting DSPs and FPGAs

MATLAB and Simulink

• Algorithm development

• Debugging and profiling

• System design

Generate code Verify design

Fixed-Point Modeling

Link to Embedded Software

Integrated Development Environment

• Compiler

• Build automation tools

• Debugger

DSP/

FPGA

55

Demo: Camera Pipeline System DesignFrom Sensor Data to Image

Scene Image

56

Why use MATLAB and Simulink for embedded

hardware design?

Automatically convert MATLAB to C code

Reduce the number of versions of code

Simplify the design process

Reuse MATLAB and legacy code in models

Convert floating-point algorithms to fixed point

Optimize algorithms for DSPs and FPGAs

Test and validate an algorithm’s implementation

57

Agenda



10:30 a.m. Break




58

What You’ve Seen Today

MATLAB for image and video processing


– Visualize, analyze, and process images

Image Acquisition Toolbox

– Test image processing algorithms with live data

Computer Vision System Toolbox

– Visualize, analyze, and process video

– Motion estimation, tracking, and stereo vision


– Process images faster using multiple core and clusters

MATLAB Coder

– Convert MATLAB code to C/C++ code

59

Doheny Eye Institute Develops

Next-Generation of Retinal Prosthesis

with MathWorks Tools

ChallengeDevelop next-generation, higher-resolution

retinal prostheses

SolutionUse MathWorks tools to develop, simulate, and

automatically generate code for real-time image

processing algorithms

Results Development time reduced from months to weeks

DSP deployment streamlined

Patient testing improved

“With Video and Image Processing

Blockset and Target Support

Package, we rapidly prototype our

image and video processing

algorithms on the DM642 board. This

can save me days or weeks of time.”

Neha Parikh

Doheny Eye Institute

Illustration of a retinal prosthesis

prototype.

60

Given Imaging Develops Camera-in-a-

Capsule Using MATLAB to Improve the

Diagnosis of Gastrointestinal Disorders

ChallengeCreate an alternative to endoscopy and other

invasive gastrointestinal imaging procedures

SolutionUse MATLAB and companion toolboxes to develop

and implement a swallowable video capsule

Results Fast, efficient development

Easy access to precise diagnostic information

Improved patient care

“With MATLAB, we simulated the

intended system and fine-tuned it at

the early stages of implementation,

enabling us to develop critical

engineering programs that met

requirements on the first iteration.”

Rafi Nave

Given Imaging

Link to user story

The PillCam SB 2 video capsule.

http://www.mathworks.com/company/user_stories/userstory2342.html?by=industry

61

Additional Articles and User Stories

Newsletter: Robot Soccer Newsletter: JPL Mars rover

User Story: Dutch Epilepsy Clinics Foundation Newsletter: Parking assistant at BMW

62

From a Leading Textbook Author ...

"I have used a number of commercial

image processing packages over the

years, and prefer the MathWorks

Image Processing Toolbox for several

reasons: the wide variety of functions it

provides, the user’s ability to write

additional functions with minimal effort,

the quality of the software, and the high

level of support."

Rafael C. Gonzalez

Professor Emeritus

University of Tennessee

63

Support and Community

64

Migration Planning

Component Deployment

Full Application Deployment

Co

nti

nu

ou

s Im

pro

ve

me

nt

Consulting ServicesAccelerating return on investment

A global team of experts supporting every stage of tool and process integration

Supplier InvolvementProduct Engineering TeamsAdvanced EngineeringResearch

Advisory Services

Process Assessment

Jumpstart

Process and Technology

Standardization

Process and Technology

Automation

65

Training ServicesExploit the full potential of MathWorks products

Flexible delivery options:

Public training available worldwide

Onsite training with standard or

customized courses

Web-based training with live, interactive

instructor-led courses

Self-paced interactive online training

More than 30 course offerings:

Introductory and intermediate training on MATLAB, Simulink,

Stateflow, code generation, and Polyspace products

Specialized courses in control design, signal processing, parallel computing,

code generation, communications, financial analysis,

and other areas

http://www.mathworks.com/training

66

MATLAB Central

Community for MATLAB and Simulink

users

Over 1 million visits per month

File Exchange– Upload/download access to free files

including MATLAB code, Simulink models,

and documents

– Ability to rate files, comment, and ask questions

– More than 12,500 contributed files, 300

submissions per month, 50,000 downloads

per month

Newsgroup– Web forum for technical discussions about

MathWorks products

– More than 300 posts per day

Blogs– Commentary from engineers who design, build,

and support MathWorks products

– Open conversation at blogs.mathworks.com

Based on February 2011

data

http://blogs.mathworks.com/

67

Connections Program

More than 400 add-on products and services that

complement and extend MathWorks products:

Specialized third-party toolboxes for MATLAB

Interfaces to third-party software and hardware products

Specialized training courses and consulting services

System integrators and suppliers that incorporate

MathWorks products

68

Book Program

More than 1200 books for educational and professional

use, in 26 languages

Mathematics

Aerospace engineering

Environmental sciences

Chemistry

Finance

Electronics

Controls

Signal processing

Image processing

Biosciences

Communications

Mechanical engineering

69

Technical Support

Resources

Over 100 support engineers

– All with MS degrees (EE, ME, CS)

– Local support in North America,

Europe, and Asia

Comprehensive, product-specific Web

support resources

High customer satisfaction

95% of calls answered

within three minutes

70% of issues resolved

within 24 hours

80% of customers surveyed

rate satisfaction at 80–100%

70

For More Information

Experiment with product by downloading a trial

Peruse videos, webinars, user stories, and demos

online

Contact us

– Talk to a sales representative

to get answers to your questions

– Discuss your projects

with MathWorks applications

engineers

Documents

Image and Video Processing with MATLAB.pdf