1
Outline
Introduction
Components
Technology Behind Sixth Sense.
How It Works?
Applications
Cost
Advantages & Disadvantages
Conclusion And Future2
“Sixth Sense is a wearable gestural interface device that
augments the physical world with digital information and lets
people use natural hand gestures to interact with that
information. ”
It was developed by PRANAV MISTRY,
3
4
A pocket projector,
A mirror,
A camera.
Mobile component
Colored markers5
The Projector: projects visual information, enabling surfaces, walls and physical objects around the wearer to be used as interfaces;
The Camera And Hands: recognizes and tracks the user's hand gestures and physical objects using computer-vision based techniques.
The Software Program: processes the video stream data took by the camera and tracks the locations of the colored markers (visual tracking of colored fingers) at the tip of the user’s fingers using simple computer-vision techniques.
The maximum number of tracked fingers is only constrained by the number of unique fiducials, thus Sixth Sense also supports multi-touch and multi-user interaction.
6
Technology behind sixth sense
Hardware Setup.
Software Setup.
Events & Gestures.
System Architecture.7
Hardware setup
MOBILE
Nokia n95 smart phone is used (running symbian O.S s60
edition).
It has multitasking capability.
Built-in camera provides execution of both:
Gesture tracking engine.
Gesture enabled application.
8
PROJECTOR Pocket projector Pk101 from Optoma is used.
It augments near by surfaces.
It is a LED based projector.
Suitable for mobile usage.
Hardware setup
9
10
Software setup
Applications are implemented using JAVA 2 MICRO
edition.
Computer vision library is written in symbian c++(used in
Gesture tracking).
The software for the sixth sense prototype is developed on a
Microsoft Windows platform using C#, WPF and open CV.
11
The software works on the basis of
computer vision.
A small camera acting as an eye, connecting
us to the world of digital information.
Processing is happening in the mobile phone,
and basically works on computer vision
algorithms.
Approx 50,000 lines of code are used.
12
13
Events & Gestures
Events & Gestures are detected using colored markers
attached to fingers.
The user can zoom in or out by moving his hands/fingers
farther or nearer to each other, respectively
The user can draw on any surfaces using the movement of
the index finger used as a pen.
System also supports freehand gestures (postures). one
example is to touch both the index fingers with the opposing
thumbs, forming a rectangle or framing gesture
14
This gesture activates the photo taking application.
Another example of such gestures is the ‘Namaste’ posture
that lets the user navigate to the home screen.
System lets the user draw icons or symbols in the air using
the movement of the index finger.
15
16
Images Pictures Gestures
Info on surface
captureSend for processing
Reflect on desired surface
Project image
Send Info
17
The hardware that makes Sixth Sense work is a pendant like mobile wearable interface
It has a camera, a mirror and a projector and is connected wirelessly to a Bluetooth smart phone that can slip comfortably into one’s pocket
The camera recognizes individuals, images, pictures, gestures one makes with their hands
Information is sent to the Smartphone for processing
The downward-facing projector projects the output image on to the mirror
Mirror reflects image on to the desired surface
Thus, digital information is freed from its confines and placed in the physical world
18
19
You hold up your left hand, fingers pointing to the right.
The system recognizes that you want to make a call, and
projects a dialing pad onto your fingers. You tap the virtual
keypad with your right hand to dial the call.
20
21
22
23
Real Time Clock
24
Browsing Net
25
The current prototype system costs
approximately $350 to build, mainly due to
the micro-projector. The software may be
available for free on the model of open and
editable freeware
26
Portable
Inexpensive
Multi-sensory
Connectedness between the world and information
It is an open source
Data access directly from machine in real time
27
Software does support the ability to use real time video streams in order to produce augmented reality.
Hardware limitations of the devices, that we currently carry around with us. For example many phones will not allow the external camera feed to be manipulated in real time.
Post processing can occur however.
28
Interactive Advertisements. 3d visualizations. Solar batteries via small solar panel. Camera can act as a third eye for the blind
person
“According to researchers, after 10 years we will be here with the ultimate sixth-sense brain implant.”
29
Sixth Sense recognizes the objects around us, displaying information automatically and letting us to access it in any way we need.
The Sixth Sense prototype implements several applications that demonstrate the usefulness, viability and flexibility of the system.
Allowing us to interact with this information via natural hand gestures.
The potential of becoming the ultimate "transparent" user interface for accessing information about everything around us.
30
31
32