2013-1 Machine Learning Lecture 01 - Pattern Recognition

Ch01_Introduction to Pattern Recognition (modified from 패턴인식개론/한학용)

2

Contents

01_ Philosophical Debates on AI

02_ Pattern Recognition (PR)

03_ Features and Patterns

04_ Components of PR and Design Cycle

05_ Category of PR and Classifiers

06_ Performance Evaluation of PR Algorithms

07_ Approaches of PR and its Application Areas

08_ Example of PR Applications

3

01_Philosophical Debates on AI

Questions Is computer merely a calculating machine?

Can computer think and understand languages like human?

4

01_Philosophical Debates on AI

Positive opinions on the possibility of AI

Negative opinions on the possibility of AI

Imitation Game A.M. Turing(1912~1954)

Chinese Room Arguments John Searle(1932~ )

5

02_Definition of Pattern Recognition

What is PR? An area of AI that deals with the problems to make computable machines

(Turing Machines) to recognize certain objects

PR

AI

Cognitive Science

6

03_Features and Patterns

What is feature? Discernible aspects, qualities, characteristics that a certain object has

What is pattern? A set of traits or features of individual objects

7

03_Features and Patterns

Easy features and difficult features

Categories of patterns

8

04_Components of PR and Design Cycle

Components of PR System and its Process

9

04_ Components of PR and Design Cycle

Design steps of PR system

Step 1 : Data gathering Most time-consuming tedious process in PR tasks

Necessary step to ensure stable PR performance

For stable performance, we need to consider how many samples are needed before the gathering.

Step 2 : Feature selection Essential part regarding PR system’s performance

We need to decide what features to choose through sufficient prior analysis on the object patters.

Step 3 : Model selection To decide what approach (model and algorithm) is to be constructed and applied

Need prior knowledge on the features

Need to set up parameters for the model according to the approach

10

04_ Components of PR and Design Cycle

Step 4 : Learning Using the feature sets extracted from the collected data and chosen models, the

learning algorithm generates or fills up the model (or hypothesis, classifier)

According to the methods, there are supervised learning, unsupervised learning and reinforcement learning.

Step 5 : Recognition Given a new feature set, the generated hypothesis decide a class or category that

the feature set belongs to.

11

05_ Category of PR and Classifiers

Categories of problems

Classification In classification problem, the system needs to output one label in a set of finite

number of labels.

Regression Generalized version of classification

Through regression, the PR system will return a real value score (usually between 0 and 1)

Clustering The problem of organizing a small number of multiple groups from a certain set

The output of clustering system is a set of pairs (example and its class).

The clustering can be processed in an hierarchical manner such as in phylogenetic tree.

Description The problem of expressing an object using a set of a prototype or primitive terms

12

05_ Category of PR and Classifiers

Classifier Most classification task in PR is done by classifiers

Classification is to partitioning a feature space composed of feature vectors into decision regions of nominal classes.

We call the boundaries of the regions as decision boundaries

Classification of a feature vector x is to decide what decision region the feature vector belongs to, and to assign x to the class that represents the region

13

05_ Category of PR and Classifiers

Classifier can be represented as a set of discriminant functions

∀j=i, if 𝑔𝑖 𝑥 > 𝑔𝑗 𝑥 , then we decide that the feature vector 𝑥 ∈ class 𝜔𝑖

14

06_ Performance Evaluation of PR Algorithms

Confusion Matrix

Recall rate = TP

TP+FN

Precision = TP

TP+FP

True Positive Rate (TPR) = TP

TP+FN

False Positive Rate (FPR) = FP

FP+TN

Actual Positive Actual Negative

Predicted Positive TP FP

Predicted Negative FN TN

15

06_ Performance Evaluation of PR Algorithms

Receiver operating characteristic (ROC) Curve

16

06_ Performance Evaluation of PR Algorithms

AUROC Ares under the region of ROC Curve

Closer the curve to top-left corner, more accurate the recognition algorithm

The performance can be evaluated by the amount of AUROC

17

07_ Approaches of PR and its Application Areas

Approaches of PR Template matching

Oldest and easiest

First, prepare the template for the object to compare.

Normalize the pattern to recognize for matching it with the template.

And calculate similarity value such as cross-correlation or distance to perform the recognition

Most important task is to prepare the most general template that explains all the samples in a certain category.

Fast running time, but weak in variation of features

Statistical approaches Decide the class of unknown pattern bases on decision boundaries of pattern sets.

Each of the pattern sets represent a certain class.

The statistical model of the patterns is a probability density function 𝑃 𝑥|𝑐𝑖 .

Learning is a process of creating a probability density function and calculating its parameters for each class

Neural networks Model the relation of connection and integration of the biological neurons

Calculate the response process of neural network for input stimulus

Classify patterns based on the responses

18

07_ Approaches of PR and its Application Areas

Knowledge of the patterns is stored as weights that represent the connection strength of synapse.

Learning is performed similar to biological ways, but the learning process is not a serial algorithm.

The learned knowledge is considered as a black box.

Minimal need for prior knowledge.

With sufficient number of neurons, theoretically any complicated decision boundaries can be constructed, so this approach is very attractive.

Structural approaches Instead of quantitative features, we consider the relationship among the basic

prototypes what construct the pattern.

Examples: Character, Fingerprint, Chromosome

19

07_ Approaches of PR and its Application Areas

Approaches of PR

20

07_ Approaches of PR and its Application Areas

Applications of PR Character recognition

Convert a scanned text image into character codes which can be edited in a computer

Mail classification, Handwriting recognition, Check and banknote recognition, License plate recognition

Biological recognition and human behavioral pattern recognition Voice recognition, fingerprint recognition, face recognition, DNA mapping, walking

pattern analysis and classification, utterance habit analysis and classification

Diagnostic systems Car malfunction, medical diagnostics, EEG, ECG signal analysis and classification, X-

Ray image pattern recognition

21

07_ Approaches of PR and its Application Areas

Prediction system

Weather forecasting based on satellite data, earthquake pattern analysis and earthquake prediction, stock price prediction, etc.

Security and military area Intrusion detection based on network traffic pattern analysis, security screening

system, search and attack of terrorist camp and targets using satellite terrain image analysis, radar signal classification, Identification Friend or Foe (IFF)

22

07_ Approaches of PR and its Application Areas

Related Areas Application Areas

•Adaptive signal processing •Machine learning •Artificial Neural networks •Robotics and Vision •Cognitive science •Mathematical Statistics •Nonlinear optimization •Exploratory Data analysis •Fuzzy and Genetic System •Detection and Estimation Theory •Formal language •Structural modeling •Biological cybernetics •Computational neuroscience

•Image processing/segmentation •Computer Vision •Speech recognition •Automatic target recognition •Optical character recognition •Seismic Analysis •Man-machine interaction •Bio recognition (fingerprint, vein, iris) •Industrial inspection •Financial forecast •Medical analysis •ECG signal analysis

23

08_ Example of PR Applications

Simple English character recognition system feature V : # of vertical lines

feature H : # of horizontal lines

feature O : # of slopes

feature C : # of curves

Character Feature

V H O C

L 1 1 0 0

P 1 0 0 1

O 0 0 0 1

E 1 3 0 0

Q 0 0 1 1

24

08_ Example of PR Applications

Automatic fish classification (Sea Bass or Salmon) A: Conveyor belt for fish

B: Conveyor belt for classified fish

C : Robot arm for grabbing fish

D: Machine vision system with CCD camera

E : Computer that analyze fish image and control the robot arm

25

08_ Example of PR Applications

Automatic fish classification Assume that fish is either salmon or sea bass

Using machine vision system for acquiring new fish image

Normalize the intensities of new fish image using image processing algorithm

Segment fish from the background in the image processing process

Using the prior knowledge that sea bass is bigger than salmon, extract features in the image to measure the length of the new fish

From the training samples of the two fish categories, calculate the distribution of the length, and decide the threshold of decision boundary that minimize the classification error

Accuracy : 60%

26

08_ Example of PR Applications

Adding features for enhancing recognition rate The accuracy should be over 95% for stable pattern recognition system

We find that average intensity level is a good feature.

27

08_ Example of PR Applications

Enhancing the recognition rate We generate 2 dimensional feature vector with length and average intensity.

Using a simple linear discriminant function, we enhance the recognition rate.

Accuracy : 95.7%

28

08_ Example of PR Applications

Cost vs. Classification Rate To minimize the cost, we adjust the decision boundary

29

08_Example of PR Applications

Generalization problem Using neural network, the performance can be enhanced to 99.9975%

Is this a good result?