Sign language recognizer

Sign Language

Recognition

Neural NetworkUsing

PRESENTED BYBikash Chandra Karmokar (0707019)

&Md. Kibria Siddiquee (0707024)

SUPERVISED BY:Dr. Kazi Md. Rokibul AlamAssociate professor,CSE,KUET,khulna,Bangladesh.

An intelligent approach to recognize sign language for deaf and dumb people of the world

Dedication

First of all we would like to remember the deaf and dumb people of the world for whom we tried to develop a Sign language Recognizer (SLR).

Outline

• Sign language• SLR & its necessity• Helping process of SLR• Working procedure of SLR• Block Diagram of SLR• BP training time & graph• Recognition accuracy • Limitations• Future plan• Papers

What is Sign Language ??

Communicating language used primarily by deaf people.

Uses different medium such as hands, face, or eyes rather than vocal tract or ears for communication purpose.

Communication using sign language

What is SLR ??

Sign language recognizer (SLR) is a tool for recognizing sign language of deaf and dumb people of the world.

Why we need SLR ??

Problems:

• About 2 million people are deaf in our world• They are deprived from various social

activities• They are under-estimated to our society• Communication problem

Continued..

Proposed Solution: SLR

SLR can be a desirable interpreter which can help both the community general and deaf.

How SLR help ?? An Example.....

Suppose a deaf customer went to a shop. She is trying to express her demands to the shopkeeper using sign language but the shopkeeper can not understand her demands.

??

shopkeeper Deaf customer

Continued..

SLR brings the solution for this problem>>

• SLR capture signs shown by deaf man• Convert the signs to text• This text is shown to shopkeeper

Now the shopkeeper can understand the deaf man’s demands

Sign to text conversion using SLR

Sign Converted text

Continued..

Text to sign conversion

When shopkeeper replied to the deaf customer SLR • Convert text to sign• This sign is shown to the deaf customer

Continued..

Now the deaf man can understand the shopkeeper’s speeches

Text to sign conversion using SLR

Continued..

Shopkeeper speech/text Sign

Text to Sign Conversion

Process

Separate each letters

Showing sign

Collecting Text• Text from the writing

place are collected

Continued..

Separate each letter

Showing sign

Collecting Text• From the sentences

each letter are separated and put into an array.

Continued..

Separate each letters

Showing sign

Collecting Text• For each letter a

predefined sign image are shown.

Sign to Text Conversion

How SLR works ??

Normalization

Sign recognition

Sign to text conversion

Image processing & sign detection

Continued..


Normalization

Sign recognition


• Image capture

• Skin color detection

Continued..


Normalization

Sign recognition


• Hand gesture detection

• Sign detection

Continued..


Normalization

Sign recognition


• Reducing image size

200x200 30x33

Continued..


Normalization

Sign recognition


• Backpropagation implementation

Continued..


Normalization

Sign recognition


• Converting sign language to Bengali or English text

কv

Block diagram of the SLR

BP Training

Figure: Training error versus number of iteration

Training time for BP

Input size of pixelTraining

Time (min)

30*33 1.545*48 2.860*63 3.7

We have used 50 signs as training input where each sign has 5 samples that make 50 x 5 = 250 samples.

Recognition Accuracy

No. of inputAvg.

Accuracy (%)

10 74

20 65

30 60

Limitations

• Due to brightness and contrast sometimes webcam can hardly detect the expected skin color.

• Because of the similarity of tracking environment background color and skin color the SLR gets unexpected pixels.

• Due to almost similar pattern its become hard to take decision.

Continued..

Continued..

Future Plan

• Real time word recognition of ASL & BSL• Implementing neural network Ensembles • Implementing Genetic algorithm for sign recognition

Required Tools

• Visual studio 2008• XML• Avro Keyboard installed• Aforge .Net• Open CV• Webcam

References

http://www.lifeprint.com/ http://engineeringproject2011.webs.com/ www.c-sharpcorner.com www.codeproject.com http://en.wikipedia.org www.aforgenet.com

Published papers

1. Bikash Chandra Karmokar, Kazi Md. RokibulAlam, Md. KibriaSiddiquee, “An intelligent approach to recognize touchless written Bengali characters”, International Conference on Informatics, Electronics & Vision (ICIEV), ISSN: 2226-2105, 2012, Dhaka, Bangladesh

2. Kazi Md. RokibulAlam, Bikash Chandra Karmokar, Md. KibriaSiddiquee, “A comparison of constructive and pruning algorithms to design neural networks”, Indian Journal of Computer Science and Engineering (IJCSE), ISSN : 0976-5166 Vol. 2 No. 3 Jun-Jul 2011

Technology

Sign language recognizer