A Deaf-to-hearing communication aid on

a mobile phone

By Mr. Muyowa MutemwaSupervisor: Dr W D TuckerCo-Supervisor: Mr. M Norman

Deaf with a capital 'D' is different from deaf or hard of hearing.

Deaf people primarily use sign language to communicate

And define their sense of culture, as opposed to the other groups that use spoken, and textual, languages like English or Xhosa..

User trials done with Deaf Community of Cape Town (DCCT).

Introduction (1)

SignSupport is a system that will ease communication between a Deaf and hearing users.◦ A set of web pages,◦ Each page has a combination of SASL videos and

English text ◦ To enable a Deaf person to tell a hearing person

(e.g. doctor) how s/he is feeling. Requires a mobile phone with:

◦ A data connection ◦ And a browser that supports Adobe Flash video

(FLV).

Introduction (2)

Introduction (3)

Introduction (4)

The central problem that this research work seeks to address is “can we develop a model using Adobe Flash technology on a mobile phone that will ease communication between hearing users and users who primarily use SASL for communication? ”.

Research question

Communication between a hearing person and a Deaf person requires an interpreter.

Most public workers such as doctors, train ticket sellers, policemen cannot sign.

Technology in South Africa is not as supportive towards Deaf people like in England, they have TESSA in public places such as post offices.

Most Deaf people in South Africa rely on neighbors and family members for emergency situations.

Rationale (1)

Most Deaf people can use computer, but they have to travel for long distances to use of them

While they carry their mobile phones with them at all times.

By combining video with the text, it can be for educational purposes.

I intend on learning sign language while doing this study.

Rationale (2)

provide a semi-synchronous communication aid on a mobile phone for SASL users to be able to communicate with hearing users in public places.

Sub-aims:◦ put the Signsupport on a mobile phone.◦ experimenting with the video quality for SASL

visualization.◦ create internet pages for Wireless Access Protocol (WAP) ◦ create a lookup dictionary of videos for the hearing

user’s response.◦ This system is not designed to a replace a public worker

Aims of my Research

Deaf telephony projects have been conducted with DCCT members at the Bastion over the years.

As a result Deaf people there have become accustomed to using information and communication technologies (ICT) at a small 'internet café’.

Most Internet usage at the Bastion involves visual media, educational programmes in sign language, and Deaf channels on Youtube.

Results indicate that Deaf users prefer to use SASL to communicate, they can also use text but their skills are limited.

BACKGROUND (1)

DCCT members tend to own low-end mobile phones with low-resolution cameras and narrow bandwidth data connectivity.

Deaf users do not take advantage of low cost text messaging like Mxit.

They might be more inclined to use data services if content were available in SASL on a mobile phone.

BACKGROUND (2)

TESSA (UK), is a Text and Sign Support Assistant, for people who are deaf or hard of hearing.

TESSA combines speech recognition technology and virtual human animation.

Video calling is available on advanced mobile phones,

but the poor resolution and low frame rate of the video remains unacceptable for SASL communication, speech is still a priority here.

RELATED WORK (1)

RELATED WORK (2)

The YouTube client is available for a limited number of phones and mobile operating systems.

Most video playbacks of on mobile devices come in the form of third party media players.

Skyfire (www.skyfire.com) is a cross platform browser that enables Video streaming.

Flash Lite 3 brings engaging experiences like web content and streaming video to a mobile phone.

RELATED TECHNOLOGY

Looijesteijn’s industrial design

PRELIMINARY USER TRIAL & RESULTS (1)

Testing◦ We tested a mock-up of the PC-based prototype of

SignSupport at the Bastian.◦ 4 Deaf users split into pairs◦ Storyboard◦ At the end of trial semi structured interview◦ Recording using a video camera for further

analysis Results

◦ Overall, the users indicated that they would like to see the prototype developed further.

PRELIMINARY USER TRIAL & RESULTS (2)

Participants◦ Deaf people who use sign language as their primary

source of communication with other deaf people and hearing people who do not sign.

◦ DCCT staff member who were involved in the computer-based prototype done by Looijesteijn.

◦ 10 Deaf people will help in the re-designing of Signsupport, focus group.

◦ 4 Deaf people will evaluated the mobile-based prototype.

Project Design (1)

Materials◦ mobile phone,◦ mobile phone browser,◦ a video camera,◦ & Adobe Flash program.

Project Design (2)

Design

Project Design (3)

Data collection◦ semi-structure interviews,◦ questionnaires,◦ discussion like interview,◦ & a video camera to capture the user trail.

Project Design (4)

Using Skyfire our intention is to design the next prototype with 'plug and play' scenarios, of which the doctor's visit is just one.

Experiment will be done with content stored on (client-side video storage)

and off (server-side video storage) the device.

text lookup vs. sign lookup, by designing a clickable image of the entire body and having questions for part that can be clicked.

Mobile prototype design (1)

Mobile prototype design (2)

1. Introduction2. Overview of project3. Literature survey4. Prototype design5. Research findings and analysis6. Conclusion

Chapter Outline

Project Schedule (1)

Project Schedule (2)

The initial SignSupport mock-up on the PC showed a potential to help Deaf users to communicate with people that do not sign.

The next version of SignSupport will run in a mobile browser and allow multiple content modules.

We will continue to conduct user trials with Deaf participants to provide a solution that is useful for them.

Testing will resemble that of the PC mock-up, but on a mobile phone.

CONCLUSION

Q & A