1

Real Time Pedestrian Detection, Tracking and Distance Estimation

# of slides : 30

Keywords: HOG, Lukas Kanade, Pinehole Camera, OpenCV

Omid.Asudeh@mavs.uta.eduUniversity of Texas At Arlington

2

Problem definition (Goal): In this project, given a stream of video, we want to detect people, track them, and find their distance in a real-time manner.

Issues:● Pedestrian Detection

● Pedestrian Tracking

● Distance Detection

3

People Detection

What is HOG (Histograms of Oriented Gradients)?:

Answer: It is a kind of feature descriptor.

Then what is a feature descriptor?

The goal of a feature descriptor is to generalize an object in a way that same objects (e.g. pedestrians) make as close as possible descriptors. That way the task of classification becomes easier.

4

HOG

Objects Descriptors

?

Classifier (SVM)

No

Yes Is Pedestrian

Is Not Pedestrian

Pedestrian detection Process

5

Challenges of HOG and SVM Classifier:

1. HOG is Computationally Complex

3. Classifiers are not exact (it may classify wrongly)

HOG in Open CV :

Input : an image

Output : a list of rectangles each bounding a pedestrian

Idea : Read a video stream frame by frame and do HOG on each frame and show the results

Slow

Lets try the idea! To make sure

2. It needs the whole body to be detected

6

Feature point Tracking

What Does a feature tracker do ?

Answer: Suppose we are given two images. We have the position of some points on the first image. In the second image the scene has changed a little. The goal of the tracker is to find the position of the points in the second image.

Benefit : It is not computationally complex

7

Tracking in OpenCV: OpenCV has implemented the Lucas-Kanade (LK) traking algorithm

Input: Image A, list of feature points in image A, Image B

Output : List of tracked feature points in the Image B

Idea : Forget about the HOG. Read the video stream frame by frame, do corner detection, track corners using LK algorithm from the current frame to the next in a loop.

seems that you forgot about the pedestrian. Using this algorithm, the program will track all the feature points (corners) of the image and not specially those related to the people.

Not that easy!

Lets Not try the idea To make sure!

8

● HOG detects people but is not fast enough to be used in real-time applications

● Tracking corners is a relatively fast operation but it is not specified to people

Idea :

1)Read the video stream frame by frame

2)Do people detection using HOG until finding people.

3)Do corner detection for each people

4)Track corners using LK from the current frame to the next.

Good idea, but it still have some issues!!

But Why??

9

ImgA = Get Current frame

People-list = Find People

(HOG) in ImgA

People found?No

Feature-list + = findFeaturePoints(p)

1.Tracked-feature-list = Track (ImgA,ImgB,Feature-list)

2.Feature-list = Tracked-feature-list 3. ImgA = ImgB4.Show (Feature-list)

ImgB = Get Next frame

Yes

∀ p∈people−list

Initial Idea

Detection Tracking loop

10

What if some feature points lost during tracking loop?

11

ImgA = Get Current frame

People-list = Find People

(HOG) in ImgA

People found?No

Feature-list + = findFeaturePoints(p)

1.Tracked-feature-list = Track (ImgA,ImgB,Feature-list)

2.Feature-list = Tracked-feature-list 3. ImgA = ImgB4.Show (Feature-list)

ImgB = Get Next frame

Yes

∀ p∈people−list

Developing Idea...

DetectionTracking loop

# of tracked features<Threshold

yes

No

12

ImgA = Get Current frame

People-list = Find People

(HOG) in ImgA

People found?No

Feature-list + = findFeaturePoints(p)

1.Tracked-feature-list = Track (ImgA,ImgB,Feature-list)

2.Feature-list = Tracked-feature-list 3. ImgA = ImgB4.Show (Feature-list)

ImgB = Get Next frame

Yes

∀ p∈people−list

Developing Idea...

Detection

Tracking loop

# of tracked features<Threshold

yes

No

This part is not clear, how do you deal with finding features of each person?

13

Okay, Let's talk about more details

What does HOG give us in openCV?

HOG

A list of rectangles

14

What is a rectangle? <X,Y,W,H>

So, HOG gives us : <(x1,y1,w1,h1);(x2,y2,w2,h2);(x3,y3,w3,h3),...>

15

Crop each rectangle and do feature detection for it

Then map features of each rectangle to the original image

16

X = X1+x'

Y = Y1+y'

Coordinates in original image

17

It seems that you tackled the complexity issue of HOG so far, and the algorithm can detect and track people relatively fast. Also it can track people even if their full body is not in the screen.

Yes, I did.

You know what? I think there is something badly wrong with your algorithm

What??????

18

Challenge :

● The tree is not moving, so it rarely loses its feature points during tracking loop.

● Therefore, the algorithm recur tracking loop infinite times (trap), and it will never go to the detection phase again!!

● HOG is not a perfect algorithm. It may consider wrong objects (like a tree ) as people.

You may say this is the end of the story, but there is always hope!

19

● The wrongly recognized object, is not moving, so it rarely loses its feature points.

This is the key!

If the number of the feature points are not changing during a while, say in 200 frames, then the algorithm should suspect a trap state and break the tracking loop.

Idea:

Therefore, we can tackle this problem by adding a simple condition to the previous algorithm

20

ImgA = Get Current frame

People-list = Find People

(HOG) in ImgA

People found?No

Feature-list + = findFeaturePoints(p)

1.Tracked-feature-list = Track (ImgA,ImgB,Feature-list)

2.Feature-list = Tracked-feature-list 3. ImgA = ImgB4.Show (Feature-list)

ImgB = Get Next frame

Yes

∀ p∈people−list

Dealing with wrong object detection challenge

DetectionTracking loop

(# of tracked features<Threshold)

yes

No

OR (# of features are not changing for a while )

21

Distance recognition

22

Problem definition: Using a pinhole camera, we want an strategy to estimate the distance of the pedestrians

Challenge: that is impossible to get the distance of objects using just a pinhole camera!!

Yes, but what if we have a piece of information?

What kind of information are you talking about?

Like the height of the camera from the ground!

23

h

Z

fImage plane

v0

v It can be proved that:

Z=f∗k v∗h

v−v0

Where f and k_v are the focal length and the pixel density(pixel/meter) of the camera.

24

Challenge: We should find V which is the height of the base point (where feet of the pedestrian touch the ground) in the image plane

But How to find the bottom point?

25

Base line

h_i

h_j

Let's define the bottom of the rectangle obtained from detection phase as the base line.

Idea: the average height of the feature points from the base line remains unchanged in the video stream.

26

i

j

d_i d_ j

Detection Rectangle

The whole image while tracking

j'

i'

Keep <d_i, d_j>

V_ j'V_ i'

V'_b

V'_b = Average(V_ j'+d_ j , V_ I' +d_ i)

Base line EstimationExample for 2 points

Goal

27

Scaling

The closer the pedestrian to the camera the more difference between the points.

j'

i'

V_ j'V_ i'

α=Average∀ i, j ∈ people [ p]

V j '−V i '

d i−d j

Scaling factor:

V b=α∗V ' b

28

Let's see some videos of the results of this work!

29

● Learning opencv, 1st edition , O'Reilly Media, Inc. ©2008 ISBN: 9780596516130, Dr. Gary Rost Bradski, Adrian Kaehler

References:

● http://en.wikipedia.org/wiki/Histogram_of_oriented_gradients

● http://en.wikipedia.org/wiki/Lucas%E2%80%93Kanade_method

● https://chrisjmccormick.wordpress.com/2013/05/09/hog-person-detector-tutorial/

30

Omid.Asudeh@mavs.uta.edu