Binary code-based Human Detection

Binary code-based human detection

Yuji Yamauchi, Hironobu Fujiyoshi Chubu university

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features

Human detection

•Classify and locate human in an image

2


Basic approach

3


Basic approach

4


Basic approach

Classify cropped image

5

識別器

classifier

Pos

Pos

Neg


Basic approach

識別器

Pos

Pos

Neg

Classify cropped image

6

Detection result

classifier


Basic approach

識別器

Classify cropped imageDetector

Detection result

7

classifier

Pos

Pos

Neg

Clustering result

識別結果の統合処理

Clustering


Processes of training and testing

Training samplesHuman Background

8

Training



Training samplesHuman Background

Compute features

9

Training



Training samples

Compute features

Human Background

Training classifier

Classifier

10

Training



Training samples

Compute features

Classifier

Training classifier

Unknown sample

Human Background

Compute features

11

TrainingTesting



Training samples

Compute features

Classifier

Training classifier

Unknown sample

Human Background

Compute features Human / non-human

12

TrainingTesting


Why human detection is difficult?

Appearance Pose

Viewpoint Occlusion Background clutter

Due to combined multiple factors, variance of appearances of human images become large

13


Image local features

Image local features based on gradients in local region 14

(d) HOG feature [Dalal CVPR2005]Input image Block Gradients HOG

(a) EOH feature [Levi CVPR2004] (b) Edgelet feature [Wu ICCV2005]

Line Arc Symmetric pair

Input image Edges

Templates

(c) LBP histogram feature [Mu CVPR2008]

Input image

Gradients

vs.

patch

LBP

50 105

95255 200

80220180

75 0 1

1 1

01 1

0

100 80

110100 150

10080 90

220 0 0

0 1

00 0

1

00000000

11111111

Input image 3x3 pixels LBP Histogram


Memory usage for HOG feature

•HOG feature • Gradient magnitude accumulates in each orientation within a local region

15

# of features : 3,360 dim. Floating type : 8 byte

Image size : 640 x 480 pixels # of windows : 50,000

26.8KB 1.25GB

Memory usage per a window Memory usage per a image

Imput image Gradients image gradient in each pixel Histogram of oriented gradientsCell Orientation

Magnitude

1 2 3 4 5 6 7 8 9

Block

A Study of Improving Human Detection Based on Co-occurrence of Image Local Features 16

To achieve implementation to low end device, the memory usage have to be reduce

For example - Cyclone Ⅲ - Logic cells : 119,088 - Memory : 0.48MB

•For implement to a low end device

Practical application of human detection


Reduction method of memory usage

•Reduction of feature dimension • Vector Quantization • Feature dimensions are reduced by clustering

• Principal component analysis • By projection to low-dimensional subspace, feature dimensions are reduced

!!!

•Quantization to low bit • Scalar quantization • Represent a feature at few bits

• Binary code

17

Minimal loss of information Computational cost is high

Computational cost is low Severe loss of information


Binarized HOG feature (B-HOG)

•B-HOG feature is obtained by thresholding •Represent by unsigned integer type

18

11100001

Input image

B-HOG feature

HOG feature

ThresholdC1

B-HOG feature can reduce memory usage to 1/8 Threshold is necessary to be optimized


Relational HOG feature (R-HOG)

•Comparing two HOGs •Normalization is unnecessary in HOG

19

Threshold is unnecessary R-HOG feature can reduce computational cost If obtained similar histograms, binary become unstable

Input image

00111110R-HOG feature

HOG feature

>C1

C2


Difference

0

1

Magnitude

Magnitude

Orientation

Orientation1 2 3 4 5 6 7 8

1 2 3 4 5 6 7 8

0 方向シフト 10011111

Problem of R-HOG feature

•If obtained similar histograms, binary become unstable

20


Shifted R-HOG feature (SR-HOG)

•Orientation in a histogram is shifted

21

Difference

0

1

Magnitude

Magnitude

Orientation


1 2 3 4 5 6 7 8

0 orientation shift 100111111 orientation shift 11000011


Shifted R-HOG feature (SR-HOG)

•Orientation in a histogram is shifted

22

Difference

0

1

Magnitude

Magnitude

Orientation


1 2 3 4 5 6 7 8

0 orientation shift 100111111 orientation shift 11000011

7 orientation shift 01111000


Effect of SR-HOG feature

23

Difference

0

1

Magnitude

Magnitude

Orientation


1 2 3 4 5 6 7 8

0 orientation shift : 10011111

Difference

0

1


Magnitude

5 orientation shift : 111110000


Experimental overview

•Comparison methods • HOG feature + Real AdaBoost • B-HOG feature + Real AdaBoost • R-HOG feature + Real AdaBoost • SR-HOG feature + Real AdaBoost !

•Dataset • INRIA Person Dataset • http://pascal.inrialpes.fr/data/human/ !

•Evaluation method • Detection Error Tradeoff (DET) carve

24

http://pascal.inrialpes.fr/data/human/


INRIA Person Dataset

•Training samples • Positive sample : 2416 • Negative sample : 12180

25

•Testing samples • Positive sample : 1126 • Negative sample : about 2 million

Positive sample

Negative sample


Experimental results

26

HOG B-HOGR-HOGSR-HOG


Memory usage and computational cost

•Comparing necessary memory usage in feature extraction •Comparing the processing time until classification by feature extraction

27

Memory and computational cost per 1 window (64 x 128 pixels)

•R-HOG and S-HOG features can reduce computational cost to about 50% •Binary code can reduce memory usage to about 1/8


Problems of binary code

•Quantization residual (QR) • A lot of information drops out !!!!

•Binary code is represented at discrete variables

28

閾値

HOG特徴量

量子化残差

00111100B-HOG特徴量

00111100

01011000

閾値

閾値

HOG特徴量

HOG特徴量

B-HOG feature

Quantization residual

B-HOG feature

B-HOG feature

HOG

HOG

HOG


Proposed method

•Binary code • Binarizing a feature represented by real value

A lot of memory usage can be reduced Information drops out Binary code is represented at discrete variables

29

•Classifier introducing transition likelihood model based on quantization residual(QR) • A transition likelihood model represents the likelihood of an observed binary code x transitioning to another binary code x ’ Effective utilization of quantization residual Relationship between binary codes is represented Computational cost and memory usage does not increase


Classifier Introducing Transition Likelihood Model Based on QR

•Relationship between binary codes is represented by transition likelihood model

30

(a) 遷移尤度モデル Transition likelihood model

HOG特徴量HOG




31

(a) 遷移尤度モデル

HOG特徴量

Transition likelihood model

HOG

1111000100111110111000012値符号列 x

量子化残差

Binary code x





32


HOG特徴量

1111000100111110111000012値符号列 x

量子化残差 Transition likelihood model

Binary code x

HOG


高い

遷移尤度

000000000000000100000010

11111111

000000000000000100000010

11111111

Transition likelihood

High

Low




33


HOG特徴量

1111000100111110111000012値符号列 x

量子化残差

高い

遷移尤度

000000000000000100000010

11111111

000000000000000100000010

11111111 Transition likelihood model

Binary code x

HOG



High

Low

2値符号列 x11000001Binary code x

(b) 識別器(b) 識別器 Classifier

入力画像Input image




34


(b) 識別器

HOG特徴量

1111000100111110111000012値符号列 x

量子化残差

高い

遷移尤度

000000000000000100000010

11111111

000000000000000100000010

11111111

入力画像

2値符号列 x11000001

遷移尤度

× 0.06

× 0.02

× 0.12× 0.62

遷移尤度

Binary code x

HOG



High

Low

Binary code x

Input image

00000000

11111111

0000000111000001

遷移後の 2値符号列 x’

-0.17 × 0.06

-0.68 × 0.02

0.02 × 0.120.43 × 0.62

対数オッズ　Binary code x’ Log odds Transition likelihood

+ 1.6弱識別器 h(x)Weak classifier


Classifier



•A transition likelihood model represents the likelihood of an observed binary code x transitioning to another binary code x ’

35

111000012 値符号列 x


00000000

11111111

00000001

111000010.45,0.50,0.40,0.20,0.40,0.50,0.25,0.90

量子化残差Quantization residuals

binary code x

Binary code x’ after transition prediction



•A transition likelihood model represents the likelihood of an observed binary code x transitioning to another binary code x ’ Step1. Compute non-invert scores z of binary

36

111000012 値符号列 x


00000000

11111111

00000001

111000010.45,0.50,0.40,0.20,0.40,0.50,0.25,0.90

量子化残差２値符号の非反転度

0.6,0.4,0.6,1.2,1.4,1.6,1.2,0.20.6,0.4,0.6,1.2,1.4,1.6,1.2,1.8

1.4,1.6,1.4,1.2,1.4,1.6,1.2,1.8

1.4,1.6,1.4,0.8,0.6,0.4,0.8,1.8Quantization residuals

binary code x


Non-invert scores


Non-invert score z of binary

•Probability of transition from x to x’ Point1. Whether or not the binary invert Point2. magnitude of the quantization residual q

37

0.0

2.0

-1.0 0.0 1.0

Quantization Residual q

non-invert score z

1.0

Concave function F()Convex function F()



•A transition likelihood model represents the likelihood of an observed binary code x transitioning to another binary code x ’ Step1. Compute non-invert scores z of binary Step2. Compute transition scores e of binary code

38

111000012 値符号列 x


00000000

11111111

00000001

111000010.45,0.50,0.40,0.20,0.40,0.50,0.25,0.90


0.6,0.4,0.6,1.2,1.4,1.6,1.2,0.20.6,0.4,0.6,1.2,1.4,1.6,1.2,1.8

1.4,1.6,1.4,1.2,1.4,1.6,1.2,1.8

1.4,1.6,1.4,0.8,0.6,0.4,0.8,1.8 0.88遷移スコア

18.21

0.840.09

Transition scores

Quantization residuals

binary code x


Non-invert scores



•A transition likelihood model represents the likelihood of an observed binary code x transitioning to another binary code x ’ Step1. Compute non-invert scores z of binary Step2. Compute transition scores e of binary code Step3. Create transition likelihood distribution E

39

111000012 値符号列 x


00000000

11111111

00000001

111000010.45,0.50,0.40,0.20,0.40,0.50,0.25,0.90


0.6,0.4,0.6,1.2,1.4,1.6,1.2,0.20.6,0.4,0.6,1.2,1.4,1.6,1.2,1.8

1.4,1.6,1.4,1.2,1.4,1.6,1.2,1.8

1.4,1.6,1.4,0.8,0.6,0.4,0.8,1.8 0.88遷移スコア

18.21

0.840.09

Quantization residuals

binary code x


Non-invert scores Transition scores


Create transition likelihood distribution E

•Create transition likelihood distribution E from transition scores e of all training samples I

40

00000000

10000000

11111111

00000000

10000000

11111111

input binary codes xB

ina

ry c

od

es a

fter tra

nsitio

n p

red

ictio

n

x’

0111111101111111

Low

High

Tra

nsitio

n lik

elih

oo

d




41


(b) 識別器

HOG特徴量

1111000100111110111000012値符号列 x

量子化残差

高い

遷移尤度

000000000000000100000010

11111111

000000000000000100000010

11111111

入力画像

2値符号列 x11000001

遷移尤度

× 0.06

× 0.02

× 0.12× 0.62

遷移尤度

Binary code x

HOG



High

Low

Binary code x

Input image

00000000

11111111

0000000111000001


-0.17 × 0.06

-0.68 × 0.02

0.02 × 0.120.43 × 0.62

対数オッズ　Binary code x’ Log odds Transition likelihood

+ 1.6弱識別器 h(x)Weak classifier


Classifier


Classifier introducing transition likelihood model

•Week classifier h(x) in Real AdaBoost

42

Transition likelihood mode P(x’| x)

W : Probability density function of each class




43


➡Weak classifier h(x) predict transition of binary code x➡By using lookup table, computational cost is same asconventional method





44


➡Weak classifier h(x) predict transition of binary code x

•P(x’| x) is unobservable, therefore E (x’| x) is commuted

➡By using lookup table, computational cost is same asconventional method




45

HOGB-HOGR-HOG


HOGB-HOGR-HOGB-HOG based proposed method


46

Proposed method enables pedestrian detection that is more accurate than that of previous methods



47

Proposed method enables pedestrian detection that is more accurate than that of previous methods

HOGB-HOGR-HOGB-HOG based proposed methodR-HOG based proposed method


Comparison of performances in each weak classifier

•Error rate of weak classifiers are plotted • If the detection performance is higher with the proposed method, the point is plotted to the right and below the red line

About 95% of weak classifiers improved performance 48

20

30

40

50

20 30 40 50Error before using transition likelihood[%]

B-HOG R-HOG

20

30

50

20 30 40 50Err

or a

fter u

sing

tran

sitio

n lik

elih

ood[%]

Error before using transition likelihood[%]

40


Implementation on FPGA

49

- Cyclone Ⅲ - Logic cells : 119,088， Memory : 0.48MB


Conclusion

•For reducing memory usage, HOG feature is binarized • B-HOG feature • R-HOG feature, SR-HOG feature ➡Proposed method reduce memory usage to 1/8, and reduce computational cost to 50% !

•Classifier introducing transition likelihood model based on quantization residual • Weak classifier h(x) predict transition of binary code x ➡Proposed method enables pedestrian detection that is more accurate than that of previous methods !

• Future work • To expand the idea of the proposed method to other learning methods 50

Documents

Binary code-based Human Detection