The Recognition of Human Movement

Using Temporal Templates

Liat Koren

- 2 -

Lecture subjects

• Introduction• Prior work• The Temporal

Templates• Usage example

- 3 -

Introduction

• Computer vision trends

– Less image or camera motion

– More on labeling of action

• Reasons

– More computational power

– Wireless application

– Interactive environments

- 4 -

Introduction – cont.

• Recent efforts are in Three Dimensional object reconstruction– Assuming it will have to be used in the recognition

of human motion.

• This article claims otherwise– View-based approach– Direct recognition

- 5 -

Motivating Example

- 6 -

Motivating Example

- 7 -

Motivating Example

• Static pictures– Hard to recognize.

• Sequence on motion– Human can recognize without three dimensional

reconstruction.

• Conclusion– It is possible to recognize movement using only

the motion itself.

- 8 -

• Process– Recover the pose of the person at each time

instant using a 3D model.– The model’s projected image should be as close

as possible to the object(e.g. edges of body in the image)

• Drawbacks– Complicated process– Human interference is usually required– Special imaging environment

- 9 -

2D Based recognition

• Action is a sequence of static poses of

object.

• Requires

– Normalization

– Removal of background

- 10 -

Wilson and Bobik’s approach• Actions are usually hand gestures• Representation

– Actual image– Grayscale– No background

• Benefits:– Hand appearance is fairly similar over a wide range of

people

• Problems– Actions that include the appearance of the whole body

are not visually consistent across different people.

- 11 -

Yamato’s et al. approach• Representation

– No background– Black and white silhouettes

• Matching– Vector quantize – Usage of a mathematical method

• Benefits– Help handling the variability between people

• Problems– Disappearance of movement inside the silhouette

- 12 -

Summery of prior work

• Action is a sequence of static poses.• Requires individual features or properties that

can be extracted and tracked from each frame.

• Recognition of movement from a sequence of images is a complicated task.

• Usually requires previous recognition and segmentation of the person.

- 13 -

Motion based recognition• Attempt to characterize the motion itself

without reference to the underlying static poses of the body.

• Possible approaches– Blob like representation– Tracking of predefined regions (e.g., legs, head,

mouth) using motion.• Face expression patches• Whole body patches

– Measure typical patterns of muscle activation

- 14 -

Terms

• Movement– where – motion has occurred in image

sequence. • MEI – Motion Energy Image

– how – the motion is moving.• MHI – Motion History Image

+

Temporal Templates

- 15 -

Temporal Templates

• Representation of movement– View specific– Movement is motion in time– Vector image that can be matched against stored

representations of movements.

• Assumptions– Background is static– Camera movements can be removed– Motion of irrelevant objects can be eliminated

- 16 -

Motion-Energy Imageswhere did the movement occurred ….

- 17 -

Motion-Energy Images

• Notice that:– If τ is very big, all the differences are accumulated– Τ has a vast influence on the temporal

representation of a movement.

- 18 -

Motion-Energy Images

• Smooth change in the viewing angle causes a smooth change in the viewed image, thus coarse sampling of the viewing circle is enough (30°)

- 19 -

Motion-History Images

• Intensity of a pixel represents the temporal

history in that pixel.

• Newer movement is brighter.

- 20 -

Motion-History Images

• A time-window of size τ is used – movement older than τ is ignored.

• The results of the article uses a simple

replacement and decay operator:

Notice that MEI can be calculated out of MHI by paintingin white any non-black pixel

One may wonder, why not use only

MHI ?Answers will be

given later…

- 21 -

MEI and MHI in a nutshell

• MEI and MHI are two vector images designed to encode a variety of motion properties.

• Benefits in this representation is that the calculation is recursive, thus only up-to-date information need to be stored, making the computation both fast and space efficient.

- 22 -

Matching Temporal Templates

• Collect training examples of each movement from a variety of viewing angles.

• Compute statistical representation of the MHI/MEI images (Hu moments)

• Given an input movement:– Calculate a statistical representation– Use mahalanobis distance to find a stored

movement, that is the nearest to the input.

- 23 -

Mahalanobis Distance Example

- 24 -

Reasoning for the algorithm

• Mahanobis distance provides:– Good matching as shown in the results of the article.

– Simple calculation which makes real-time applications feasible.

• Hu moments allow representation of images, that is invariant to scale or translation.One problem with Hu moments is that: “Hu moments are difficult to reason about intuitively” (the authors)

- 25 -

Testing the system

18 exercises performed by experienced aerobicinstructor.

MEIs are on the bottomrows.

- 26 -

Why both MHI and MEI ?

Because MHI and MEI

perceive two different

characteristics of the

movement (the “where”

and the “how”) they look

different ,and thus, both

essential.

- 27 -

First experiment• Input 30° left of the subject• Match against all seven

views of all 18 moves• 12 out of 18 are correctly

recognized

0°

30° 60° 90° 120°150°

180°

- 28 -

Analyze the results of 1st exp.

Move 13 in 30 ° Move 6 in 0 ° The correct match

input

false correct

- 29 -

Combining multiple views

• Two cameras with orthogonal views• Minimize the sum of the mahalanobis

distance between the two input templates and two stored views of movement that have 90° between them.

• Hidden assumption: we know the angular relationship between the cameras.

- 30 -

Second Experiment

0°

30° 60° 90°120°

150°

• Input with two cameras:• 30° left of the subject• 60° right of the subject

• Match against all seven views of all 18 moves

• 15 out of 18 are correctly recognized

- 31 -

Analyze the results of 2nd exp.

Move 16 Move 15 The correct match

input

false correct

- 32 -

Segmentation and Recognition• Problem : speed of performance is different

among different people.• Solution: Segmentation

– When training the system, calculate τmax and τmin for each movement.

– Use algorithm to match over a wide range of τ.

- 33 -

Problems

• Problems with current system– One person partially occludes another

• Solution: Use several cameras

– More than one person appears in the view point

• Solution: use a tracking bounding box

- 34 -

More Problems• Motion of part of the body is not specified

during a movement– Possible solutions

• Automatically mask away regions of this type of motion• Always include them

• Camera motion– Rather easy to eliminate since camera motion is

limited.

• Person is performing the movement while locomotion

- 35 -

The KidsRoom: An Application• room is aware of the children (at most 4)• The room takes the children

to a story.• The room’s reaction is influenced

by the actions of the children.• Current story : adventurous tour to monster land• In the last scene the monsters teach the children to

dance.• Then, the monsters follow the children if they perform

movements they “know”• The narration coerces the children to room locations

where occlusions is not a problem

- 36 -