Embed Size (px)
Citation preview
MTA SZTAKI
Department of Distributed
Systems
An assistive interpreter tool using glove-based hand gesture recognition
Péter MátételkiMáté PatakiSándor TurbuczLászló Kovács
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
2
DSD MTA SZTAKI DSD
MTA founded in 1827 SZTAKI founded in 1968 DSD founded in 1994 pioneers of web
technology in Hungary main fields
collaboration systems digital libraries web accessibility IoT crisis management plagiarism detection
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
3
DSD Helen Keller
„Blindness separates us from things, but deafness separates us from people.”
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
4
DSD
Target groups hearing and speech impaired people deaf people (native language: sign language)
Social problems problematic social integration isolated group
To overcome the language barriers human sign language interpreter or: a good assistive tool
Challenges
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
5
DSD
Silent Speech Translation (interACT)
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
6
DSD
Eyes-Free Keypad Input
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
7
DSD
Keyboard Glove – University of Alabama
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
8
DSD
Our assistive tool is controlled by sign language (intuitive) talks for the deaf everyday use
Features sign language to speech real time seamless
Talking Hands
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
9
DSD TalkingHands
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
10
DSD How does it work?
sign language custom gestures and text motion capture with the glove gesture descriptor stream signal processing language processing text to speech
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
11
DSD Glove: gesture capturing
9DOF sensors signal fusion absolute position relative angles virtual hand gesture descriptors 30 Hagdil descriptor/sec Bluetooth
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
12
DSD Mobile application: processing
Segmentation algorithm simple similarity repetition unkown descriptors sliding window kinetics based algorithm
Context-sensitive auto-correction modified Levenshtein n-gram (1- & 2-grams) confusion matrix speech
raw text
descriptors
text
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
13
DSD Scenarios, situations
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
14
DSD Scenarios, situations
work banking shopping public services healthcare education free time …
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
15
DSD Future plans
TalkingHands as a product two-handed gesture recognition capture motion dynamics research in assistive technologies and IoT robotics remote manipulation
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
16
DSD The project
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
17
DSD
Péter Mátételki
MTA SZTAKI DSD
http://dsd.sztaki.hu/projects/TolmacsKesztyu
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
18
DSD
www.youtube.com/watch?v=NhTUeZ16ZTw
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
19
DSD Dactyl fingerspelling alphabet
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
20
DSD Hagdil gesture descriptor
ck
MTA SZTAKIDepartment of Distributed Systemsdsd.sztaki.hu
21
DSD Architecture
