05/02/2023 1

Seventh Sense Technology

Under the guidance of:Mr. Mohammed SaifuddeenAsst. ProfessorPACE Mangaluru

Presented by :Sahal Hashim P.USN: 4PA11EC091Branch : Electronics & Communication

05/02/2023 2

Contents :-INTRODUCTION : - What is Sixth Sense What is Seventh Sense PROBLEM STATEMENT :-• Need for Sixth/Seventh SenseLITERATURE SURVEY :-• Overview of Sixth Sense Technology• Flaws in Sixth SensePROPOSED METHOD :-RESULTCONCLUSION & FUTURE SCOPEREFERENCES

05/02/2023 3

What is Sixth Sense.?• Sixth Sense is a technology that augments the physical world around us with digital

information and lets us use natural hand gestures to interact with that information.

• It was developed by Pranav Mistry, a PhD student in the Fluid Interfaces Group at MIT Media Lab.

• It is so called Wear Your World (WUR) Device that uses Natural hand gestures to interact with the digital world.

• Sixth Sense comprises a mobile device, pocket projector, a mirror and a camera. The hardware components are coupled in a pendant like mobile wearable device

05/02/2023 4

What is Seventh Sense.?• Seventh Sense Technology is a technology used for the human

electronic interaction using natural hand gestures• The hand gestures are used to control any micro controller based

devices or robots at distant places using Zigbee or GSM Module or Satellite transceiver• Seventh Sense Technology uses the Sixth Sense Technology and its

advancements to control the autonomous robots or any microcontroller based devices on human will.

05/02/2023 5

Need for Sixth/Seventh Sense.• The bridge gap between the Physical and digital world which needs to

be overcomed.

• Much simpler and cost efficient approach is needed to interact with the digital world.

• The world of Robotics is limited at present and it needs to be explored by making them Smarter, Intelligent and Human friendly.

• Bulky gadgets needs to get smaller yet providing Broad Functions.

05/02/2023 6

Overview of Sixth Sense TechnologySixth Sense is supported by techniques such as..

Gesture recognition

Computer vision

Radio Frequency Identification

Augmented reality

05/02/2023 7

Device used in Sixth Sense TechnologyThe hardware components are coupled in a pendant like mobile wearable device.

Camera

Projector

Mirror

Mobile Component

Colored Markers

05/02/2023 8

05/02/2023 9

Flaws in Sixth Sense • Sixth Sense is human oriented and cannot be implied at a particular

location without his physical presence.• The present Technology cannot control any kind of Robots or

microcontrollers.• The person in charge of sixth sense device can only access the

Information around him.• Gestures cannot be recognized with bare hands and the user needs

the colored markers at all time.• The main flaw of the Sixth Sense algorithm is the movement of

objects with same color of the marker in the background.

05/02/2023 10

Proposed Method • Seventh Sense is focused on forecasting information and making an

alteration somewhere he is not virtually present.

• Seventh Sense technology is built upon Robotics and Advanced Sixth Sense Technology on the basis of Computer Vision.

• It comprises of a projector and a simple camera at the users place and remotely placed Robot or Nano robot or any micro controlled device.

• We can control the activities of the robot or micro controlled devices along with a help of camera attached to the robot or device.

05/02/2023 11

Seventh Sense Flow Diagram

05/02/2023 12

Design of Seventh Sense TechnologyDesign of the seventh sense can be broadly classified into three

components.

1. Seventh sense technology based hardware2. Second component is related to Computer vision and Artificial

neural networks.3. Robotics

05/02/2023 13

HARDWARE COMPONENTS OF SEVENTH SENSE TECHNOLOGY

• Camera• Projector• Mobile Computing Device• Micro controlled Device or ROBOTS with Wireless Camera• ZIGBEE, GSM or Satellite Transceiver Module

05/02/2023 14

Working of an Autonomous Robot

05/02/2023 15

SOFTWARE IMPLENENTATION OF SEVENTH SENSE TECHNOLOGY • To develop an application based on seventh sense technology one can

use:

1. Languages used are Java, OpenCV, JavaCV. 2. Image processing Software used is Matlab 3. Embedded programming in Embedded C

05/02/2023 16

SYSTEM IMPLEMENTATION• Seventh Sense Technology is based upon mainly two types of gesture

recognition

1. Gesture recognition with markers 2. Gesture recognition with bare hand (without markers)

05/02/2023 17

Hand gesture recognition process execution

05/02/2023 18

METHODS OF IMPLEMENTATION Gesture Recognition with Markers/Bare Hand

• Capture every single frame from the video. • Process each frame obtained. • Get the two red/blue channels from the frame by setting threshold. • Subtract the gray scale image from the channel. • Convert the subtracted image to binary image. • Find bounding box of definite height and definite distance from each

other.

05/02/2023 19

For Bare Hand Gesture Recognition• The frame obtained from the video is first captured and converted to

YCC and applied threshold for skin color and then the Image is converted back to RGB and finally to the binary image.

05/02/2023 20

Different stages of background subtraction

A Real time video after Background Subtraction.

05/02/2023 21

Pattern Recognitionwe use back ground subtraction followed by canny edge detection and Scale Invariant Feature Transform (SIFT) process to store each gestural actions

Output obtained after Canny edge detection applied on background subtraction output Hand gesture training process using SIFT

05/02/2023 22

Algorithms Applied• Threshold setting code for

Background Subtraction

threshold=25for i=1:width for j=1:height if Fg(i,j)>threshold Fg(i,j)=255; else Fg(i,j)=0;

• Threshold setting for Canny Edge Detection

I_max=max(max(NVI));I_min=min(min(NVI));level=alfa*(I_max-I_min)+I_min;subplot(3,2,5);Ibw=max(NVI,level.*ones(size(NVI)));imagesc(Ibw);title('After Thresholding');

05/02/2023 23

Result• The Combination of Sixth Sense technology with Robotics were tested

and the autonomous robots were able to act more intelligent and were more human friendly providing all the information needed by its operator with just few hand gestures.

05/02/2023 24

Conclusion and Future Scope• Seventh Sense is more likely installing a digital system (computer) into

our body and making it as seventh sense of our body. • It allows us to interact with the information via natural hand gestures

and move robots or small devices at our will.• This Seventh sense can be extended to Nano robotics and can find

different diseases and cure at will, especially cancer.• This technology can be further developed to work as a Fifth Sense for

disabled person.

05/02/2023 25

References• M. K. Saha and S. Hore, “Sixth Sense Technology: A Brief Literary Survey,” Proceedings of International Journal of Engineering Research

Technology (IJERT), Vol. 2 Issue 12, December 2013. • V. J. Oniga, S. and I. Orha, “Intelligent human-machine interface using hand gestures recognition,” In the proceedings of the IEEE Trans.

On Automation Quality and Testing Robotics (AQTR), pp. 559 - 563, 24-27 May, 2012. • I.-L. Jung, N. Akatyev, and Won-Dong Jang, “Touch less user interface based on marker detection and tracking for real-time mobile

applications,” In the proceedings of the International Journal of Innovative Computing(ICIC), Information and Control, Vol. 9 Issue 2, February- 2013.

• www.cs.cmu.edu/cil/vision.html • R. Z. Khan and N. A. Ibraheem, “Hand gesture recognition: A literature review,” In the proceedings of International Journal of Artificial

Intelligence Applications (IJAIA), Vol.3, No.4, July 2012. • H. Park, “A Method for Controlling Mouse Movement Using a Real –Time Camera,” 2008. • Robertson P., Laddaga R., Van Kleek M., “Virtual mouse vision based interface”, In the Proceedings of the nineth U international

conference on intelligent user interfaces, pp. 177 U 183, January 2004. • H. D. Nasser, “Hand gesture interaction with a 3d virtual environment,” The Research Bulletin of Jorden, ACM , ISSN: 2078 U 7952 , Vol

II(III), Page - 86. • T. Kirishima, K. Sato, and K. chihara, “Real -Time Gesture Recognition by Learning Selective Control of Visual Interest Point,” IEEE Trans.

on Pattern Analysis and Machine Intelligence, Vol. 27, No. 3, pp. 351- 364, March 2005. • M. Fiala and C. Shu, “3d Model creation using self-identifying markers and sift keypoints,” In the proceedings of the IEEE Trans. on

Haptic Audio Visual Environments and their Applications, pp. • 118 - 123, 2006.

05/02/2023 26

• www.learnartificialneuralnetworks.com • Murthy and G.R.S., “Hand gesture recognition using neural networks”, In the Proceedings of IEEE Trans. on Advance Computing

Conference(IACC), pp. 134 - 138, 19-20 • February,2010. • Yubing Dong, Mingjing Li and Jie Li., “Image retrieval based on improved Canny edge detection algorithm”, In the Proceedings of the

International conference on Mechatronic Sciences, Electric Engineering and Computer (MEC), pp. 1453 - 1457, 2013. • Ghosh D.K. and Ari S., “A static hand gesture recognition algorithm using k-mean based radial basis function neural network”, In the

Proceedings of 8th IEEE Trans. on Information, Communications and Signal Processing (ICICS), pp. 1-5, 2011. • Kumar, S.P. and Pandithurai, O., “Sixth sense technology”, In the Proceedings of IEEE Trans. on Information Communication and Embedded

Systems (ICICES), pp. 947 - 953, 2013. • Bhowmick S., Kumar S. and Kumar A., “Hand gesture recognition of English alphabets using artificial neural network”, In the Proceedings of

IEEE Trans. on Recent Trends in Information Systems (ReTIS), pp. 405 - 410, 2015. • Hsien-I Lin, Ming-Hsiang Hsu and Wei-Kai Chen, “Human hand gesture recognition using a convolution neural network”, In the Proceedings of

IEEE Trans. on Automation Science and Engineering (CASE), pp. 1038 - 1043, 2014. • Chang Tan, Xiao and Nanfeng, “Improved RCE neural network and its application in human-robot interaction based on hand gesture

recognition”, In the Proceedings of 2nd IEEE Trans. on Information Science and Engineering (ICISE), pp. 1260 - 1263, 2010. • Jalab, H.A. and Omer, H.K., “Human computer interface using hand gesture recognition based on neural network”, In the Proceedings of 5th

National Symposium on Information Technology: Towards New Smart World (NSITNSW), pp. 1 - 6, 2015. • Canny, John., “A Computational Approach to Edge Detection”, In the Proceedings of IEEE Trans. on Pattern Analysis and Machine Intelligence,

pp. 679 – 698,1986. • https://in.mathworks.com/discovery/edge-detection.html• GitHub profile adjecon (https://github.com/ajdecon/gradschool_matlab/blob/4d1c9d247021019b0a0d229112f8e90c6cf96564/bpass.m)• Li Cheng, M. Gong, D. Schuurmans, and T. Caelli. Real-time Discriminative Background Subtraction. IEEE Trans. Image Processing, 20(5), 1401-

1414, 2011(http://web.bii.a-star.edu.sg/~chengli/BkgSbt.htm)• Wikipedia on YCC (https://en.wikipedia.org/wiki/YCbCr)

05/02/2023 27

Credits:-

Pranav Mistry Sidharth Rajeev