Upload
gyles-marshall
View
219
Download
2
Embed Size (px)
Citation preview
Craig NewmanCurtin University of Technology
Translating Australian Sign Language into Speech
Translating Australian Sign Language into Speech
QuickTime™ and a Video decompressor are needed to see this picture.
• Introduction To Auslan• Elements Of Auslan Signs• Hardware Used• Hardware Control Algorithm• Elements Of Sign Recognition Algorithm• Future Intentions
Presentation OverviewPresentation Overview
• British and Irish origins• More than 4000 commonly used signs • 0.1% of Australians use Auslan
Introduction to AuslanIntroduction to Auslan
• Handshapes• Location• Orientation• Movement• Expression
Elements of Auslan signsElements of Auslan signs
• Large Scale• Straight Line• Series of Straight Lines• Arcs• Circles
• Small Scale• Changes in Orientation• Changes in Fingers New Handshape
MovementMovement
• Known through the P5• Degree of Finger Bend• Position Primary Locations, Movement• Orientation
• Required• Contacting Secondary Locations
What was Needed?What was Needed?
• Collect Data Every 17ms• 60 Hertz• Synchronisation with P5
• Test Pins Not Logged• Write Pin High, Read Remaining Pins• Log Connected Pins• Debounce: Test For Consistency• Encapsulate for Transmission• Transmit When Required
Contact Sensor AlgorithmContact Sensor Algorithm
Working Example 1Working Example 1
QuickTime™ and a Video decompressor are needed to see this picture.
Working Example 2Working Example 2
INSERT CLIP: IM
QuickTime™ and a Video decompressor are needed to see this picture.
• 3 Samples Stored in stop_buffer• Position• Orientation• Finger Bend• Contacts
• Check Last 2 Samples within Range
When was the hand considered Stopped?
When was the hand considered Stopped?
• Wait Until Hands Stop• Take First Sample From stop_buffer• Filter Through if/else Structure to find
Correct Combination• Return Handshape
Identifying HandshapesIdentifying Handshapes
Working Example 1Working Example 1
INSERT CLIP: GUNQuickTime™ and a Video decompressor are needed to see this picture.
Working Example 2Working Example 2
INSERT CLIP: MOTHERQuickTime™ and a DV - PAL decompressor are needed to see this picture.
• Record Position of • Head• Shoulder• Bicep• Forearm
• Extrapolate Primary Locations• Else is Neutral Space
• Problem• Requires Dynamic Position Tracking
Identifying Primary LocationsIdentifying Primary Locations
• Wait Until “not stopped”• While “not stopped” copy stop_buffer
• Allow for Slight Pauses
• Filter Data By Testing For Inherent Geometric Structure
• Output Movement Type
Identifying Movement TypeIdentifying Movement Type
• Identify Initial Handshape, Location, Orientation
• Filter Through if/else Structure
• If Sign Recognised
• Output Sign
• Else
• Identify Movement
• Identify Final Handshape, Location, Orientation
• Filter through if/else Structure
• Output Sign
Putting it all together: Recognizing Signs
Putting it all together: Recognizing Signs
Working ExampleWorking Example
INSERT CLIP: WRAPUP2QuickTime™ and a Video decompressor are needed to see this picture.
• Design Composite Glove• Dynamic Tracking Of Primary Locations• Introduce Learning Algorithms
• Identifying Handshapes• Identifying Movement
• Translate Signing Context Into English Text
• Integrate Text To Voice Synthesizer
Future IntentionsFuture Intentions