stanford hci group / cs376

research topics in human-computer interaction http://cs376.stanford.edu

Vision-based Interaction

Scott Klemmer17 November 2005

2

cs547: Blake Ross and Asa Dotzler

Mozilla: Creating simple software in a geek-driven culture

3

The first vision-based interface Myron Krueger used computer

vision to create Responsive Environments (1970s)

“Reaction is the Medium” http://www.artmuseum.net/w2vr/

timeline/videoplace_video.html

4

How it works

Video and background are separated in analog using chroma key techniques(think broadcast news)

The first and last points of each raster are stored in the computer, and represent the person’s outline

5

Vision-based UIs: “Verbs” Detecting and Tracking elements of a

certain type in a scene Capturing contents of detected objects Recognizing individual members in an

object class

6

Vision-based UIs: “Verbs” Detecting and Tracking elements of

a certain type in a scene

7

Vision-based UIs: “Verbs” Capturing contents of detected

objects

8

Vision-based UIs: “Verbs” Recognizing individual members in a

class

9

Vision-based UIs: “Nouns” People (one or multiple)

Bodies Faces Hands

Documents Objects

10

Vision-based UIs: “Nouns” People (one or multiple)

Bodies Faces Hands

Documents Objects

11

Vision-based UIs: “Nouns” People (one or multiple)

Bodies Faces Hands

Documents Objects

12

Background SubtractionI N F R A S T R U C T U R E

13

Image Moments (of Inertia) 0th moment is mass

(total number of pixels)

14

Image Moments (of Inertia) 1st moment is center

15

Image Moments (of Inertia) 2nd moment is orientation

16

Tools for Vision apps Intel’s OpenCV

C API to highly optimized image processing functions (threshold, dilate, optical flow, …)

http://www.intel.com/research/mrl/research/opencv Fast to run! Slow to develop Great for vision folks; too low-level for app folks

Papier-Mâché Java API (and to some extent visual UI) for vision

(and other physical input) http://guir.berkeley.edu/papier-mache Fast to develop! Slow to run Great for app folks; ~5 fps can sometimes be too

slow

17

Good Vision Books

Computer Vision: A Modern Approach David Forsyth and Jean Ponce (2003) Fantastic book; but goal is more

theoretical understanding than practical application

Robot Vision Berthold Horn (1987) More focused on apps and interactive

algorithms Somewhat out of date