WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based...

WiGest: A Ubiquitous WiFi-based Gesture Recognition System

Heba Abdelnasser†, Moustafa Youssef‡, and Khaled A. Harras*Alexandria Univ.†, Egypt-Japan Univ. of Sc. and Tech.‡, Carnegie Mellon Univ.*

Motivation

● Rendering touch input – user is wearing gloves.

● Wet/dirty hands.

Motivation

● Rendering touch input – user is wearing gloves.

● Wet/dirty hands.

● Special sensors● E.g. Samsung Galaxy S5

WiGest Approach

● Free the user from specialized sensors● Using the ubiquitous WiFi● Works with any phone

● Leveraging natural human movements to control devices

Basic Idea

• Leveraging the impact of hand motion on the received WiFion the phone to control WiFi-enabled devices.

WiGest Example

Challenges

• WiFi is a noisy signal• Avoiding false positives – human

interference• Handling the variations of

gestures and their attributes

• Energy consumption• No training

– Handle different humans– Same human a different times

-36

-33

-30

-27

-24

1 2 3 4 5 6 7 8

RS

SI

dB

m

Time in Seconds

-41

-40

-39

-38

-37

-36

-35

-34

0 1 2 3 4 5 6 7 8 9 10

RS

SI

dB

m

Time in Seconds

Far

Near

Agenda

Introduction• WiGest System• Results• Conlusions

Agenda

Introduction• WiGest System• Results• Conlusions

WiGest System

WiGest System

Primitive Extraction

Primitive Extraction

1. Denoising– Using Wavelet denoising

2. Edges detection,– Extract different primitives.

3. Parameters extraction,– Magnitude: near/far.– Speed: fast/slow.

-36

-33

-30

-27

-24

1 2 3 4 5 6 7 8

RS

SI

dB

m

Time in Seconds

Input Output

Noise Reduction

Time in sec

Scale

2.5 3 3.5 4 4.5 5 5.5 6 6.5 7

5

10

15

20

25

30

35

-36

-33

-30

-27

-24

1 2 3 4 5 6 7 8

RS

SI

dB

m

Time in Seconds

Noise Reduction

• Removing noise using wavelet denoising– It can preserve signal details while filtering out the noise

and variations in the signal– Consists of three stages:

• Decomposition the signal to approximation and detail coefficients.

• Thresholding detail coefficients• Reconstruct the denoised signal by adding approximation and

detail after thresholding

Noise Reduction

Raw signal After denoising

-33

-30

-27

-24

1 2 3 4 5 6 7 8

RS

SI

dB

m

Time in Seconds

-36

-33

-30

-27

-24

1 2 3 4 5 6 7 8

RS

SI

dB

m

Time in Seconds

Edges Extraction

• Detecting gesture edges using wavelet analysis– In detail (high-pass filter) coefficients

• Falling edge causes a local maxima• Rising edge causes a local minima

Edges Extraction

Denoised signal

DWT detail coefficient

-33

-30

-27

-24

1 2 3 4 5 6 7 8

RS

SI

dB

m

Time in Seconds

-6

-4

-2

0

2

4

6

1 2 3 4 5 6 7 8

Time in Seconds

Edges Extraction

Denoised signal

DWT detail coefficient

-33

-30

-27

-24

1 2 3 4 5 6 7 8

RS

SI

dB

m

Time in Seconds

Edges Extraction

Denoised signal

DWT detail coefficient

Edges Extraction

Denoised signal

DWT detail coefficientExtracted primitives

Parameters Extraction

Magnitude

-41

-40

-39

-38

-37

-36

-35

-34

0 1 2 3 4 5 6 7 8 9 10

RS

SI

dB

m

Time in Seconds

Far

Near

-39

-38

-37

-36

-35

0 2 4 6 8 10 12 14 16 18

RS

SI

dB

m

Time in Seconds

Slow Fast

Speed

Parameters Extraction

Magnitude

-41

-40

-39

-38

-37

-36

-35

-34

0 1 2 3 4 5 6 7 8 9 10

RS

SI

dB

m

Time in Seconds

Far

Near

-39

-38

-37

-36

-35

0 2 4 6 8 10 12 14 16 18

RS

SI

dB

m

Time in Seconds

Slow Fast

Speed

System Logic Flow

Segmentation

• Silence period determines the start and end of a gesture

• Preamble (unique signature), to start the communication channel

Frequency =

3 /2 Hz,

Count = 2

Frequency =

5 /2 Hz,

Count = 3

-39

-36

-33

-30

0 1 2 3 4 5 6 7 8

RSSI

dBm

Time in SecondsSilence period Gesture delimiter

Pattern Encoding

• The extracted gesture primitives are converted to string sequence:

– Rising edge : +– Falling edge : -– Pause : 0

• Gesture matching– String matching to identify the gesture

Extracted gesture pattern : - + - +Gesture : Infinity

Gesture Identification

Extracted gesture pattern : - + - +Gesture : Infinity

Preamble • Energy efficiency• Human interference

and noise

Pattern encoding• Efficient matching• Error tolerance

Gestures ParametersFrequency =

3 /2 Hz,

Count = 2

Frequency =

5 /2 Hz,

Count = 3

-39

-36

-33

-30

0 1 2 3 4 5 6 7 8

RSSI

dBm

Time in Seconds

Count• Number of gesture

repetitions

• E.g. double click and single click

Frequency• Number of gesture

repetitions per unit time

• E.g. speed of basketball

dribble

System Logic Flow

Action Mapping

• At the end, the extracted gesture is mapped to an

application action .

Gesture Action

Application

Agenda

IntroductionWiGest System• Results• Conlusions

Implementation and Evaluation

• Test environments• Typical apartment• Engineering building at our campus

• More than 1000 experiment• Off-the-shelf WiFi-equipped devices• Hardware:

• Cisco Linksys

• Android-based cell phone.

• HP laptop

39 ft

34 ft

115 ft

49 ft

Experiments Performed

• Distance effect for different scenarios• Orientation• Number of APs• Human interference

• Whole home case study

• More results in the paper

Distance Impact

0.5

0.6

0.7

0.8

0.9

1

3 7 10 13 16 20 23 26 39 52 62-70

-60

-50

-40

-30

Acc

ura

cy

Av

erag

e R

SS

I

Distance in feet

Accuracy Average RSSI

0.5

0.6

0.7

0.8

0.9

1

3 7 10 15 20 26 30 39 56-70

-60

-50

-40

-30

Acc

ura

cy

Av

erag

e R

SS

I

Distance in feet

Accuracy Average RSSI

0.5

0.6

0.7

0.8

0.9

1

7 10 13 16 20 30 36-70

-60

-50

-40

-30

Acc

ura

cy

Av

erag

e R

SS

I

Distance in feet

Accuracy Average RSSI

• 87% accuracy for distances up to 26ft

• Using a single AP

Orientation Impact

• Overall accuracy of 90.5%• The highest accuracy is in West – body not blocking the

signal

0

0.2

0.4

0.6

0.8

1

North South East West Overall

Acc

ura

cy

Number of APs Impact

0

0.2

0.4

0.6

0.8

1

1 APs 3 APs 5 APs 7 APs

Acc

ura

cy

Number of APs used in voting

• Accuracy 96% using 3 APs

• Reaches 100% with seven APs

Human interference

• Accuracy 89% in the presence of four

interfering humans

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4

Det

ecti

on

Acc

ura

cy

Number of interfering humans

Whole Home Media-player Case Study

• Accuracy of 96% using 2 APs

0 .9

0 .96

0 .93

0 .97

0 .99

1

0 .98 0.02

0.01

0.025

0.03

0.04

0.090.005 0.005

0.005

0.005

0.03 0.005

0

0 0 00

0 0000

0 0000

00 0000

0000

000

00

0

0

0

00

00

nu

ll

Ac

tu

al

Ac

tio

n P

erfo

rm

ed

Act ion Classif ied

39 ft

34 ft

Agenda

IntroductionWiGest SystemResults• Conlusions

Conclusion

• WiGest is a gesture recognition system• Robust to noise in the environment and interfering humans• Does not require any training• Energy-efficient• Works with any WiFi-enabled device

• Primitives detection accuracy is 87.5%– Using a single AP

• Accuracy increases to 96% using three overheard APs.

Future Work

• Leveraging CSI for finer grained gestures• Other wireless technologies, e.g. cellular and

Bluetooth• Other applications

Thank You

Come and see our demo tomorrow10am to 1pm

wrc-ejust.org wrc_ejust wrc.ejust