Texas InstrumentsInnovation Challenge India Design Contest 2015

SELF ASSESSMENT

“We, authors of the report entitled Spatial Augmented Reality- GESTAIR, confirm that this report has not been

submitted to any other forum such as another contest or conference for publication. We hereby state that we will not

submit the same work for any other contest in the future. We understand that Texas Instruments has the right to use this

report in its conferences/publications. We will seek TI’s permission before we submit the report for publication in an

external forum.”

1. Comment on the originality of your idea. Did you derive inspiration from any other work? Provide the appropriate

references.

Ans: The motivation of the project was to help motor people with voice impairments. Assistive technology for speech

disorders could consist of equipment or a device that will supplement a user’s attempt to verbally communicate with

others. It could also be an unaided method of augmenting or even replacing speech (an alternative method of verbal speech).

An example of alternative speech is sign language. The existing systems available require the efforts of other person to learn

sign language .Thus providing a better solution to the society in the form of augmented reality can help them not only

interact with their surrounding better but also prove assistive in their verbal communication through inclusion of speech of

the recognized text .

But the prototype involved utilized two technologies to realize motive: a) Displair

b) digital pen utilizing triaxial accelerometers

2. List any persons who helped you in the course of the project and explain their contribution.

Ans: Our mentor guided us through the project and helped us coming out with a prototype for the proposed solution.

Also our seniors helped us in understanding the relevant circuits that could help accomplish the task. Also in making the

final video and in setting up the arena we took help from our friend and her contribution stands vital in successful

demonstration of the final prototype.

1. Highlight at least two technical challenges you faced and how you overcame them.

Ans: The technical challenges faced:

i. Inspite of intensive study , programming the Launchpad and understanding of syntax of ENERGIA and getting

desired output was a benchmark in itself. But further a more comprehensive study, experimentation and taking

help from seniors provided a hands on experience.

ii. Setting the boundary for the x,y,z coordinates

iii. Developing the fogscreeen module. Since the module was bulky and complex in hardware making the interface

was tough. Also other parameters like air-flow density hindered the project completion.

2. Please highlight at least two non-technical challenges you faced and how you overcame them.

Ans: None

3. Did you use WEBENCH to design power supply, filter etc in your project? If yes, share your experience of using

WEBENCH

Ans: No

4. Explain how the experience of the TI India Analog Design Contest helped you.

Ans: It helped us realize our potentials to develop a prototype of our own . Also in the process ,we learnt more about the

components manufactured by Texas Instruments and helped us analyse the various applications of these components in

realizing endobjectives and building smart and sophisticated systems.

5. List two things that could have added further value to your project.

Ans: A more sophisticated and portable system could have been developed . The idea endorses a new perspective to the

application of augmented reality driven systems to providing assistive technologies to the people in need. Also the project

still lacks composite characters and words all at the same time then passing single glyphs. The addition of the above could

have made the idea more comprehensive and could have facilitated ease of use.

6. Please tick all aspects of your project that you believe are now complete.

Paper design of hardware Algorithm/software design

Hardware implementation on breadboard System-level testing with examples

Hardware implementation on PCB Benchmarking/Performance Analysis

Hardware Testing Short Video on Project

Anshita Agarwal-

Diksha Sharma-

Divyanshi Rastogi

Deshraj Yadav-

Names and signatures of student team members

Mr.Sumitkhandelwal- Name and signature of the mentor

Spatial Augmented Reality-GESTAIR

Divyanshi Rastogi, Anshita Agarwal,Diksha Sharma, Deshraj Yadav SumitKhandelwal

JSS Academy of Technical Education Noida

C-20/1, Sector-62 Noida, U.P.-201301

Email: anshitaagarwal24@gmail.com

Abstract—If buttons are a thing of the past and touch screens are the present, what are the screens of the future? It’s not a riddle, but it is a trick question: the screens of the future won't be screens at all but interactive images floating in mid-air. The Digital Pen Technology and fogscreen projection for Virtual Projection is the new generation method for trajectory recognition based input device. The digital pen helps users to use the pen to write digits or make hand gestures, and the accelerations of hand motions measured by the accelerometer are transmitted to a computer for online trajectory recognition. In this technology, the implementation of a new concept of image drawing and mouse controlling is done using image processing along with the speech after recognition of the text that hangs in the mid air. Every day we use our gestures to interact with everyone. Gestures play an important role in our life. Thus Gestairis helpful in using the digital world through our gestures for interaction. It provides a link between the digital world and the real object world. And when this combines with the spatial coordinates, it can allow people to display and interact with information without cluttering the physical environment.Thus augmented reality blurs the line between what is real and what we see; by augmenting what we see, hear and feel. Thereby, iteliminates the need of multiple interface and creates a single universal interface allowing us to perform wide range of tasks. Keywords—Digital pen, accelerometer, fogscreen, text

recognition,augmented reality

I. INTRODUCTION

The main idea behind developing this system is to develop an

advance human computer interaction system using trajectory

recognition algorithm and image processing tools where system

can look for human behavior or action through use of

accelerometer, process it and consequently perform predefined

task or action and display floating images in mid-air.Main aim

of proposed system is to construct a system for online hand

writing character recognition written in air and a fogscreen that

gives us the video feedback in the mid-air. Here accelerometer

gives response for every slight deflection or movement in the

system. Accelerometer is developed by using MEMS

technology. A significant advantage of accelerometer for

general motion sensing is that they can be operated without any

external reference and limitation in working conditions

Thus, the final layout of the system proposed would look

like the following:

Figure 1: Basic functional block diagram

II. TECHNICAL BACKGROUND

The proposed solution is inspired by the digital pen that can

write in a 3-D plane and involves application of basic

handwriting recognition algorithm to identify the glyphs made

by the user. Also the second part of the project derives its

inspiration from an existing technology called Displairwhich is

a 3D interactive raster display technology developed by a

Russian company of the same name. The Displair projects

images onto sheets of water droplets suspended in air, giving

the illusion of a hologram. Our project is an amalgamation of both the

technologies and not only makes the use of hand gestures to

communicate but also aims at displaying the communicated

messages in the air.

Another added feature is the text to speech which reads out the

text aloud. Due to its motivation of helping motor people with

speech impairments, it helps rendering them a virtual voice to

communicate to others and also makes it easier for the person

on the other side to understand without making extra efforts of

learning sign language.

III PROPOSED SOLUTION

The existing technologies like displair use expensive cameras

and detect the hand movements. The prototype thus would be

quite expensive. The solution we have provided uses an

accelerometer as a wearable device that makes the prototype

comparatively cost effective.

Also since it is an assistive technology to the people with voice

impairments it is a boon to them by equipping them with a

virtual voice.

The overall execution flow is as follows:

Figure 2.Top-level view of the proposed solution

We assume that a handwriting recognizer has access to a

handwriting profile based on a large number of samples of a

user’s handwriting. In the case of letters, upper and lower case

must be written separately and in all cases the glyphs must be

formed in the same manner. Also, the main constraint factor

included was that the user could write only one letter at a time

and not a complete string. The training data is scaled to fit

within a fixed dimension set to a constant. Since we are dealing

with pixels of finite size, the variable sized dimension of the

bounding box never takes on a zero value. For recognition, the

user writes an unknown symbol which is then scaled to the

same bounding box as the training data. The recognizer has

access to all of the training data and precomputed statistics

about this data. By comparing aggregate statistical information

from the training data with statistics from the unknown symbol,

and by directly engaging in a pointwise comparison between

the unknown symbol and all known symbols in the training set,

the recognizer identifies what it believes is the best match.

IV IMPLEMENTATION

A. Hardware Implementation

a. Pattern Formation:

Digital pen consist of Tri-axial accelerometer& TI

microcontroller (MSP430G2553).

The MEMS based accelerometer measures the acceleration

signals generated by a user’s hand motions. The

microcontroller converts the analogacceleration signals to

digital ones via the A/D converter.The raw accelerationsignals

of hand motions are generated by the tri-axial accelerometer are

given to microcontroller. Our hand always trembles slightly

while moving due to human nature, which causes certain

amount of noise. The signal pre-processing consists of

calibration, a moving average filter, a high-pass filter, and

normalization. Using moving average filter we collect set of 10

value received from accelerometer & takes average of this

value because of that the signal become smoother and if there is

any sudden change in signal due to hand movement is avoided

with the help ofhigh pass filter.The normalization is to start the

signal from start point.

b.Text-to-speech module: The TTS256 is an 8-bit microprocessor programmed with letter-to-sound rules. This built-in algorithm allows us for the automatic real-time translation of English ASCII characters into allophone addresses compatible with the MagnevationSpeakJet Speech Synthesizer IC. It is then combined with the SpeakJet to build a complete text-to-speech solution.In our proposed system we are using TTS256 IC whichuses predefined speech rules to break up text into various sound components (called allophones) and then translates these into the numbers. The combination of TTS256 and SpeakJet allow us to turn text strings stored in the processor into speech. The circuit usedfor the same is:

Figure 3. Text to speech Module

The SpeakJet is connected to the audio conditioning circuit that

reads out the written text. It consists of a pre-amplifier stage , a

band pass filter employing NE5532A IC’s followed by a

power amplifier circuit using LM833 connected to the speaker

at its output. The circuit is shown below:

Figure 4. Pre-amplifier circuit

Figure 5.Band-pass filter

Fig 6.Power amplifier

c. Fogscreen Projection:

The last part of project is the projector which gets the video feedback after processing of the image that was extracted from the glyphs written by the user.Fogscreen is an exciting new projection technology that allows to project image and video screen of “dry fog” giving a false illusion that the objects are floating in mid-air. It is one type of advanced projecting device that consumes water and electricity to form fogs on which the images are projected. It is formed by ordinary tap water and digital technology like ultrasonic devices(usually a ultrasonic humidifier) to create a thin layer of dry fog which is sandwiched between two air curtains. Fogscreen is suspended fog generating device and has no

chemicals apart from ordinary tap water. It creates a dry fog by

ensuring water droplets are in range of 2-3 microns in size and

are electrostaticallycharged so they move around and away

from other objects.It is thus ahigh-tech-version of cool air

humidifier.

The system we used employed a mini fogger machine that was made using dry ice and tap water and a small fan about 60mm in diameter to direct the fog into the inlet of the fogscreen box. The fogscree box had one set of fans to blow the fog downwards while the other two sandwiches the fog between air curtains so that it becomes a smooth projection screen.

B. Software Implementation

The algorithm for handwriting recognition provides a

multiple character recognition output indication which includes

compressing the velocity indicating output, separately

analyzing information for the x and y velocity components,

analyzing the directions of the velocity, calculating velocity

thresholds, digitizing velocity components, comparing and

matching between a digital dictionary and a digitized velocity

record and using only part of the digitized velocity record,

comparing and matching between a digital dictionary and a few

different representation forms of digitized velocity using a

prioritization procedure which takes place in case of

disagreement between different comparing and matching

results, filtering out short duration segments of the velocity

components, indicating velocity value changes and ignoring

time durations between the changes, performing a merger

operation on velocity segments and binarizing velocity

segments. But all this happens internally within the inbuilt

package of python TESSERACT OCR. Thus the gestures made

by user are recorded and sent to computer for processing that

uses a python code to read, reconstruct and identify the

characters as written by the user.

V. CONCLUSIONS

The project as suggested used an accelerometer to

translate gestures into meaningful images and convey what was

exactly written by the user. The concentration was on the

improvement of the interface that connects the digital world

and the real world and this was successfully achieved by

implementing this gesture based device.

Also the idea aimed at creating a more clutter free

interaction of humans with their physical world and provide an

environment friendly and chemical-free interaction of

technology by making the traditional concept of pen and paper

redundant , but due to technical difficulties and insufficient air

flow , the fogscreen couldn’t be successfully implemented.

A.Future Scope:

The project has many applications: 1.One of the major contributions is to help the people with

speech disorders and impairment. School-aged children who

have special needs relating to speech or communication

disorders may be entitled to assistive technology services in the

classroom. Assistive technology for speech disorders could

consist of equipment or a device that will supplement a child’s

attempt to verbally communicate with others. It could also be

an unaided method of augmenting or even replacing speech (an

alternative method of verbal speech). An example of alternative

speech is sign language. Children who use assistive technology

will profit greatly. They will be able to communicate their

needs and desires and will perform better at school. They may

also see social benefitsand canbe proved to be a great

assistance in interacting and conveying messages to the people

around them thus making the extra efforts of learning sign

language redundant.

2.Also one of the major outcomes of this is the efforts of

making the concept of pen and paper redundant.The use of

writing texts virtually into the mid air and saving it into your

portable devices can greatly contribute to the environment.

3.Another aspect of the proposed system is its ability to take

education to the next level. With its power to be visually

appealing, it could open the doors to new education

opportunities. Students of all ages could be entertained by

interacting with it while also being challenged academically.

You can do anything from drafting documents to building

interactive 3D models.

B.Limitations:

1.It works with only single glyphs.

2.Thefogscreen module employed is bulky and is not portable

for ease of use. But, further development and research can make

device more economic and portable.

ACKNOWLEDGMENTS

We would like to thank our mentor Mr.SumitKhandelwal

for his constant support and guidance. We also thank our

seniors in helping us in coming out with the prototype and

guiding us in understanding the various circuits involved. We

thank our department and our HoD, Mr.SampathKumar.V in

providing us an access to labs ,components and equipments to

complete our project testing and realizing our potential in

developing the prototype.

REFERENCES

. [1] Sung-Do Choi, Lee, A.S., Soo-Young Lee, On-Line

HandwrittenCharacter Recognition with 3D Accelerometer, 2006 IEEE International Conference on Information Acquisition, 20-23 Aug.2006.

[2] Jeen-Shing Wang, Yu-Liang Hsu, Cheng-Ling Chu, Online

Handwriting Recognition Using an Accelerometer-Based Pen Device, 2nd International Conference on Advances in Computer Science and Engineering (CSE 2013)..

[3] http://prosauce.org/blog/2012/6/10/how-to-diy-improved-inexpensive-fog-screen.html

APPENDIX A

# [max size of image be let 900*900]

from PIL import Image

frompytesser.pytesser import *

import sys

import cv2

importos

importnumpy as np

coord = [(1,2,3),(2,3,4),(4,5,6),(6,7,8)]

img = cv2.imread('white.jpg')

fori in coord:

x,y = i[0],i[1]

img.itemset((x,y,0),0)

img.itemset((x,y,1),0)

img.itemset((x,y,2),0)

# plot the given gesture into the image

cv2.imwrite('test.jpg',img)

image_file = 'test.jpg'

im = Image.open(image_file)

text = image_to_string(im)

text = image_file_to_string(image_file)

text=image_file_to_string(image_file, graceful_errors=True)

# the identified character is

print "THE OUTPUT IS"

print text

img1 = cv2.imread('test.jpg', 0)

# pick the character printed from the database

img2 = cv2.imread(text+'.jpg', 0)

#merge the two images

h1, w1 = img1.height,img1.width

h2, w2 = img2.height,img2.width

vis = np.zeros((max(h1, h2), w1+w2), np.uint8)

vis[:h1, :w1] = cv2.GetMat(img1)

vis[:h2, w1:w1+w2] = cv2.GetMat(img2)

vis2=cv2.CreateMat(vis.shape[0],vis.shape[1],cv2.CV_8UC)

cv2.CvtColor(cv2.fromarray(vis),vis2, v2.CV_GRAY2BGR)

cv2.ShowImage("test", vis2)

cv2.waitKey(0)

cv2.destroyAllWindows()

APPENDIX C – BILL OF MATERIALS

Component Manufacturer Cost per

compon

ent

Quant

ity

Total

cost of

compon

ent

TI Supplied/

Purchased

1 ADXL 335 Analog

Devices

4.95

USD

2 9.95

USD

Purchased

2 TTS256C Magnevation 21.95 USD

1 21.95 USD

Purchased

3 SPEAKJET

IC

Magnevation 24.95

USD

1 24.95

USD

Purchased

4 LM833 Texas

Instruments

1.05

USD

1 1.05

USD

TI

supplied

5 LM386 Texas

instruments

0.5

USD

1 0.5

USD

TI

supplied

6 NE5532A Texas

instruments

0.5

USD

2 0.5

USD

Purchased

7 PC Fans Cooler

master

5 USD 10 50

USD

Purchased

8 Fogscreen

module

components

1 50

USD

Purchased

Total Cost of the Project 159

USD

Youtube Link:- https://www.youtube.com/watch?v=HMHPLSf3ZdA

TEAM ID:-1298 TEAM LEADER:-Anshita Agarwal TEAM MENTOR:- Sumit Khandelwal