Multimedia and Human Computer Interfaces

CS335 Principles of Multimedia Systems

Multimedia and Human Computer Interfaces

Hao Jiang

Computer Science Department

Boston College

Nov. 20, 2007


Outline

Human Computer Interface. Multimedia interfaces and applications. Vision based interfaces

– Systems using vision based interfaces.– Some basic methods in vision interfaces

• Background subtraction.

• Color blob detection and tracking.

• Face detection (Viola and Jones face detector).


Human Computer Interface

Different meanings of HCI– The study of people, computer technology and the ways

they interactive with each other.– The design, evaluation and implementation of computer

systems for human use.– The research to study how to make computer systems more

useable.


Aspects of HCI

Design– Intuition, experiences and design principles.

Human factors– Experiments testing the perception, memory and cognition

models of human being.

Devices– Physical devices such as mouse, keyboards and other fancy

HCI hardware.

Software and systems– Such as windows GUI, shell commands in Unix etc.


HCI Studies Computers

– What are they good at and bad at?– Input, output, processing, networking, etc.

People– What are people good at?– What is the limitation of human abilities?– Input, output, memory, perception and cognition model.– No average answer. Must accommodate diversities.

Usability– More than the size of widgets, where are the buttons and

syntax of commands. Context

– Environment, physical constraints, cognition constraints, task, etc.


The Quality of HCI

There are fives factors that affect the design and implementation of HCI– Time to learn.– Speed of performance.– Rate of errors.– Retention over time.– Subjective satisfaction.


Multimedia Interfaces

Multimedia interface combines different sorts of media modalities to achieve more natural and more efficient human computer communication.– Speech, audio, touch,

video input (patterns, posture, gesture) etc.

“Put that there” is an example multimedia HCI that combines speech, gestures and graphics.

http://www.media.mit.edu/speech/sig_videos.html

Put that there, MIT 1981


Early and Late Integration

Mode 1

Mode 2

ClassificationFeatures Output

Mode 1

Mode 2

Features

Classification

Output

Classification

Early integration

Late integration


Advantages of Multimedia Interface

More alternative methods. The flexibility of combining different schemes. More efficient for specific tasks. Greater precision of information. Help handicapped people to use computers.

http://www.youtube.com/watch?v=d7eGypGOlOc&feature=related http://www.youtube.com/watch?v=Gnq-8iaOcXk

Pen gesture interface Virtual Reality


Vision Based Interfaces

Vision based interfaces uses computer vision techniques to achieve more effective HCI.

Vision based interfaces are based on techniques such as:

Hand tracking,Hand gesture,Arm gestureHand Pointing

Face detectionHead trackingGazeExpressionLip reading

Posture, body gesture and action recognition


Example Systems

MotionCapturing


Example Systems

The MIT Kidsroom

Posture recognition

Actiondetection


Methods in Vision Interfaces

Background subtraction.

Estimating the background

B(n) = B(n)*alpha + I(n)*(1-alpha)

Foreground Map = |Image – Background|>threshold

alpha is a positive floating point number close to 1.


Color Blob Detection

We would like to compare the template color histogram witheach location in the image.

We first convert each image pixel by its bin number.s(x,y) = s(x,y-1) + i(x,y)h(x,y) = h(x-1,y)+ s(x,y)

(x,y)

h(x,y) is the color histogram

a b

c d


Blob Tracking (Filtering Scheme)

Search window

t1t2 t3

t4

: Predicted location. : detected object.

: Final estimation.p

x

y

x = p * alpha + y * (1-alpha)


Blob Tracking (DP Approach)

t2 t3 t4

AB

BC

BD

t1

AB

C

D E

F

CF

CE

DF

DE

t

cost(F) + alpha ||2C-B-F||

const


Mean Shift Tracking

Instead of exhaustive search in a window, if object moves slowly, we can use a more efficient location updating scheme – Mean shift.

In the following equation, x(i) are points in the search window and y is the location of object, w(i) are weights.


Mean Shift Tracking Result

http://www.caip.rutgers.edu/~comanici/Papers/KernelTracking.pdf


Face Detection

Viola and Jones face detector

These features can be computed using integral images similarly

to the color histograms discussed before.


Face Detection

It uses boosting scheme to automatically choose the optimal weight for the classifier based on each kind of feature.

To further improve the efficiency, a cascaded classifier is used.


Summary

We only touch the very surface of multimedia interfaces.

Currently, the dominant HCI is still GUI with keyboard and mouse.

Multimedia interfaces are beginning to attract more and more interests.

We need more robust and efficient media recognition schemes to support effective multimedia interfaces.

Documents

Multimedia and Human Computer Interfaces