AGENDA ITEM: VII Florida Polytechnic University Board of ......PI: Sherif Rashad Co-PIs: Ryan Integlia and Elhami Nasr . Graduate Student: Rabeet Fatmi Undergraduate Students: Gabriel

AGENDA ITEM: VII

Florida Polytechnic University

Board of Trustees December 7, 2016

Subject: Faculty/Student Presentation: (Design and Implementation of an Innovative System for Automatic Conversion from American Sign Language to Speech)

Proposed Board Action

No action required- Information only.

Background Information

The main objective of this research project is to design and implement a new practical and reliable system for automatic real-time conversion from ASL to speech. The researchers involved in this project are using a multidisciplinary approach to build a novel system that will be able to capture and analyze hand gesture motions in real-time using motion sensors and machine learning algorithms. The proposed system will be able to learn new signs and to expand and improve the sign language dictionary. This system can have a wide range of applications for healthcare, education, gamification, entrainment, and many other services and applications. Funding Source: Internal Seed Grant, Florida Polytechnic University PI: Sherif Rashad Co-PIs: Ryan Integlia and Elhami Nasr Graduate Student: Rabeet Fatmi Undergraduate Students: Gabriel Hutchison and Xin Wang

Supporting Documentation: Presentation Prepared by: Dr. Sharif Said Rashad, Associate Professor of Computer Engineering, Computer Science

Board of Trustees Meeting 12.07.16 Page 25

Design and Implementation of an Innovative System for Automatic Conversion from

American Sign Language to SpeechPI: Sherif Rashad

Co-PIs: Ryan Integlia and Elhami NasrGraduate Student: Rabeet Fatmi

Undergraduate Students: Gabriel Hutchison and Xin Wang

12/7/2016


Introduction

• Gesture recognition is a very efficient means of communication, providing a natural form of expression for the user.

• Millions of hearing-impaired people worldwide communicate through sign languages every day.

• In the same way that voice recognition provides a simple communication platform for most computer users, gesture recognition is a natural means of communication for of hearing-impaired people


American Sign Language

• American Sign Language (ASL) is a well-defined collection of gestures.

• ASL gestures are clearly defined as a means of communication, and the meaning of most gestures is very clear since the language is already created for communication between humans.


Related Work

• Sensor gloves used to recognize the hand gestures.– The person needs to wear gloves with several sensors to capture the motions. – The main disadvantage of this approach is that it is not practical and it is difficult to

use the sensor gloves in natural settings for many users.

• Most recent research has attempted to invent methods which eliminate this requirement.

• ASL alphabet recognition was introduced using a Microsoft’s Kinect as a low-cost depth camera.


Proposed Technique

• Current focus is on using motion sensors and analyzing motion signals• Machine learning techniques are applied to recognize different gestures

Image Processing

Motion Sensors

Imaging Devices

Motion Analysis

Evaluation of Generated Models

Data Fusion

Machine Learning Techniques

Networked Gamification

Text to Speech Modules


• Myo Wearable Device:– Medical grade stainless steel EMG

sensors– Highly sensitive nine-axis IMU containing

− Three-axis gyroscope− Three-axis accelerometer− Three-axis magnetometer.

– ARM Cortex M4 Processor.

Myo Wearable Gesture Control Armband Board of Trustees Meeting 12.07.16

Page 31

Data from Myo Armband

• Spatial data:− Orientation data: roll, pitch, and yaw.− Acceleration vector data− Angular velocity data

• EMG data from the eight sensors


Sample Data

RollRight PitchRight YawRight RollAccelRight PitchAccelRight YawAccelRight GyroRollRight GyroPitchRight GyroYawRight Class

161.2454335 193.8987434 44.61407429 -0.123046875 -0.31640625 0.938964844 -2.1875 1.25 3.6875 table

152.0511368 188.7658686 33.9266243 -0.078613281 -0.466308594 0.876953125 2.75 1.125 -2.5 above

151.9580826 142.0753625 24.51713538 0.408203125 -0.382324219 0.784667969 34.125 7.5625 -36.5 below

144.5047546 218.5517203 24.50601605 -0.268554688 -0.602539063 0.73828125 0.9375 24.5625 22.3125 drink

110.3546439 148.6133252 353.0980963 0.268554688 -0.901855469 0.330566406 -3.0625 -0.5625 -1.1875 thank you

• A combination of values define what gesture was being performed• Data tells us what feature values explicitly differentiate between classes (words)


0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

J48 KNN ANN naive Bayes

97.23%96.96%

94.03%

78.18%

Prediction Accuracy

Prediction Accuracy

• What percentage of test data was correctly classified by a classifier• Classifiers used: J48, K nearest-neighbor, Artificial neural network, Naive Bayes


Sample Confusion Matrix (J48)

Confusion Matrix describes the performance of a classifier



Words (classes) that are outputted by the classifier



Confusion row for the word ‘table’



Total instances of actual words



Correctly classified instances


Android Mobile Application Board of Trustees Meeting 12.07.16

Page 40

Demo

DEMO


Future Work

• Add more words to the dictionary

• Work with sentences

• Continue to work on the development of the mobile application

• Include additional sensors (Leap Motion)


Thank You

Questions?


Documents

AGENDA ITEM: VII Florida Polytechnic University Board of ......PI: Sherif Rashad Co-PIs: Ryan Integlia and Elhami Nasr . Graduate Student: Rabeet Fatmi Undergraduate Students: Gabriel