PortableVision-based HCI A Hand Mouse System on Portable

Devices

連矩鋒 (Burt C.F. Lien)

Computer Science and Information Engineering Department

National Taiwan University

Problems

• A Portable Vision-based HCI– Hand mouse operating on a projected interface– Real-time detection of user hand motion from a

user PDA/SmartPhone’s video camera (target platform)

• A part of “mTeller” project– Natural storytelling support system

Why important

• Vision-based HCI is a more instinct way to manipulate data

• Mobility– Steerable interface everywhere

Related Works I

• A Portable System for Anywhere Interactions – Sukaviriya et al., IBM Research

• Real-time hand tracking using a set of cooperative classifiers based on Haar-like features– Barczak1 et al., Institute of Information &

Mathematical Sciences Massey University

Everywhere Display (IBM)

Figure 1: Interactive store application

Related Works II

• Real-Time Hand-Arm Motion Analysis using a single Video Camera– Hienz et al.

• MMX-Accelerated Real-Time Hand Tracking System– Liu et al. (2001)

Related Works III

• Rapid Object Detection Using a Boosted Cascade of Simple Features.– Viola, P., & Jones, M. (2001).

• Robust real-time object detection.– Viola, P., & Jones, M.

• Robust real-time face detection – P. Viola and M. Jones.

• Adaboost-based real-time pedestrian detection – P. Viola, M. Jones, and D. Snow.

Target Devices

System Configuration

Hand motion capture and

interpretation

Wireless projector

data transmission

Interactive Interface

What is new

• A portable vision-based HCI on handheld device

• Design for general data manipulation

System Implementation

• Platform (prototype)– “Laptop” + “Low Cost Camera (USB)”

• Software tools– “MS VC++” + “Intel OpenCV library”

Technical Challenges and approaches

• Real-time hand motion tracking under an intense lighting condition (projector light)– Adaboost

• Hand position vs. Corresponding programs on host– Comparison between PDA’s screen and the projected screen(??)

• An efficient algorithm to run detection system on handheld devices (limited computation)– Lower framing rate (2fps?) + fewer classifier

• Noise filter– A sliding window or a simple filter to filter misbehaving hand

motion

Expect Result

• A generic vision-based HCI software program running on a laptop (1st phase)

• A user can manipulate the laptop directly on the projected image– Hand mouse (one-click function only)

The End