Mini JarvisLIVE ANNOTATIONS WITH GOOGLE GLASS

August 14, 2014

Saumya Soman Jin Bing Lin Anthony Yang Yash Sanghavi

Outline

• Background• Architecture Overview• Hardware• Software• Challenges Faced• Demonstration• Conclusion• Future Work• Acknowledgements

Background

• Object and Facial Detection technology have existed for a number of years• Glass presents an exciting new platform for object and facial detection• Mini Jarvis performs searches using facial and object recognition to learn more

information about those people or objects

Architecture Overview

Architecture Overview

Hardware

Software

● Google GDK (Glass Development Kit)● OpenCV Library (Open Source Computer Vision)● Apache Tomcat 7● Eclipse Juno IDE● RESTful webservice● Google Custom Search API

OpenCV Library

● OpenCV is a software library for computer vision● Used to train the server to detect an object or face● Cross-platform i.e. it runs on Windows, Android, Linux, etc.● It is written mainly in C++● Also has interfaces for Java anad Python

Apache Tomcat 7

● Tomcat is a Servlet container (Web server) licensed under the Apache Software License

● Used for deploying and running web applications written in Java● Open source non-commercial project● Lightweight and easy for server management

RESTful webservices

• A web service is a web page meant for a computer to request and process.

• REST (Representational State Transfer) is an architectural style which is based

on web-standards and the HTTP protocol.

• REST based architecture everything is a resource.

• Every resource should support the HTTP common operations (GET, POST, PUT

and DELETE)

Google Custom Search API

• Google offers Custom Search API

• Use color and name of entity as search terms

• Returns JSON response of search results

• Allows 100 requests per day

Training

• Requires positive and negative images• Positive images contain the object

• Negative images do not contain the object

• OpenCV

• create_samples generates samples

• train_cascade uses samples to make classifier XML

• Successfully trained for one object (banana) and one person (Anthony)

Color Extraction

• Extracts color of object recognized using OpenCV libraries

• Detected image is enclosed in a rectangle

• Rescaled to 32 x 32

• Returns dominant color within pixels

• Accuracy of detection

• Long training time

• Accuracy of color extraction

• Frequent overheating of the Glass

• Delay to server

• Frequent updates to Glass

Challenges Faced

Demonstration

Conclusion• Mini Jarvis is proof-of-concept of object and facial detection on

Google Glass• Takes advantage of the built in hardware of Glass• Privacy Issues?

Future Work• Add more objects and faces• Develop methodology to train with better accuracy• Optimize the request-response time from server• Reverse Image Search• Multiple Simultenous Annotations

Acknowledgements• Ivan Seskar• Roy Yates• Shridatt (James) Sugrim

Questions?