Smart Camera Applications

Mobile Monday in SwitzerlandSmart Camera Applications

Mobile Monday – April ’09 1

Juha K. Laurila ([email protected])

Nokia Research Center (Lausanne)

April 6th - 2009

Nokia, NRC and Smart Camera Applications • Nokia – Devices and Services

• www.ovi.com

• Biggest camera manufacturer• Camera in 60% of manufactured devices - in many of them two

• higher volume than overall (stand-alone) digital camera market

Markets

Group Executive Board

Corporate Development Office

Corporate Funct ions

Services

Devices

Nokia Siemens Networks

• Rich set of sensors in mobile devices –


• Rich set of sensors in mobile devices –endless opportunities for context-aware services and applications

• Camera = sensor

• NRC – commercial impact from deep science

• Outline:• Mobile augmented reality (MARA)

• Mirror world (Image Space)

Shopping

Weather

HotelInformation

Restaurant reviews& menus Calla taxi

Imagine a world where anythingcan be considered interactive …

Motion detector Gestures Keyboard Augmented reality Identification Measurements

Mobile Augmented Reality • Sensors• GPS

• compass

• accelerometers

• Camera• used to give context over which icons are overlaid


Mobile Augmented Reality Applications, is one approach combining reality with information about it. The key technical components are location (where the user is, via GPS) and orientation (which way the user is pointing the device, magnetic compass and accelerometers). It uses camera images as backdrop over which icons and information about POIs, points of interest, can be overlaid. This is a very robust approach, though the accuracy in most cases is limited to annotating targets that are about the size of a building. Here are some ways of interacting with the system. At first, if the device is held so it’s level with the ground, the user sees a normal map view with icons for points of interest. The device can detect this using the accelerometer. When she raises it so the camera points forward, the system can indicate POIs by overlaying icons over the camera view. The icons are placed based on the users current location and compass bearing. Finally, if you flip the device upright, you can access more detailed information about one of the POIs.

Augmented Reality Based on Image RecognitionSystem Overview

GPS


Server

Here is different project that is still ongoing in my team together with people from Stanford University. Now we use camera as a sensor, try to recognize what camera is seeing, and then provide a link to the user. Again, no typing in keywords, no browsing through hierarchical menus, just point your camera and get back information. Since recognizing all the different locations just visually is a very complicated vision task, we simplify and use rough location as a cue. Phone gets position from GPS satellite (or other means, it’s good enough to know the rough location within a city block or two) Phone prefetches data relevant to current position Then the user snaps a photo the image is then matched with the prefetched data And the result is displayed to the user

Nokia Point & Find: Press Release, April 1st - 2009

Nokia Introduces Nokia Point & Find, a new way to Connect With Information and Services on the go Apr 01, 2009

New platform enables businesses to engage with consumers in real time with relevant content

San Francisco, CA, USA - Nokia introduces an innovative service concept that enables people on the move to access relevant information and services on the internet, simply by pointing their mobile phone camera at real-life objects. A beta version of Nokia Point & Find, focusing on movies, is now available in the UK and US. Capabilities will later expand into other services and countries.

"Simply by pointing their camera phone at a poster for a new movie, people can watch the trailer, read reviews, and find the closest cinema where it is playing," said Philipp Schloter, General Manager, Nokia Point & Find.

Nokia Point & Find is an open service platform on which other companies can build innovative customized experiences to drive better engagement with potential


innovative customized experiences to drive better engagement with potential customers. Nokia is inviting businesses, content providers and agencies to discuss how the Nokia Point & Find service could help them with specific applications, campaigns or promotional activities. Unique experiences can be created through the self-service Nokia Point & Find Management Portal, or by working with the Nokia Point & Find professional service team.

Nokia Point & Find uses advanced real time image processing and recognition technologies to link the user to digital content and services. It also recognizes bar codes, integrates GPS positioning technology, and supports category-specific text-entry search. When the phone is pointed at an object, Nokia Point & Find uses a variety of the phone's capabilities including the camera, internet connectivity, and GPS positioning to evaluate the object. Then, by rapidly searching through a database of virtually tagged items, the system identifies the object and returns a set of links to associated content and services.

Companies interested in using Nokia Point & Find for their own campaigns can visit http://pointandfind.nokia.com to request further information. Nokia Point & Find's beta release is available for download into select mobile devices in the UK and US at http://pointandfind.nokia.com as well.

Nokia is now trying to find a way to make some of these ideas to a real service. Point and Find is a service that was at first incubated in our team, and was spun off to a business development program almost a year ago. The team is based here in San Francisco, with a few guys in London. Point your camera at an object and find out information about it. The first use case relates to movie advertising, once the system recognizes a movie poster, the user can access additional information about the movie such as ratings, view trailers, even buy tickets. They hope to be able to launch it fairly soon.

Other Possible Use CasesLandmark-based navigation


As we needed to collect a database of geolocated and tagged images, we also thought of other uses for the data. This system provides an alternative to the normal map-based navigation. Instead of just showing a route on a map, find for each route segment a good image, automatically create a textual annotation, automatically add an arrow pointing where you should go. Prototype runs on N95.

Nokia Image Space Experience places and merge digital and real world via mobile device

Browser•Intuitive navigation and participation via the Flash based browser•Enjoy, create and share Scenes –photo enabled narratives, supported by the community content in the service

Browser•Intuitive navigation and participation via the Flash based browser•Enjoy, create and share Scenes –photo enabled narratives, supported by the community content in the service

Overview

•User takes photos, uploads them to service and the system automatically locates them and sets the attitude (heading, yaw and roll)

Overview

•User takes photos, uploads them to service and the system automatically locates them and sets the attitude (heading, yaw and roll)

Summary: A service which offers an easy to generate and intuitive to navigate image-based mirror world. Any type of media content can be placed to this world for exploring and experiencing via a web browser and mobile client. Strong community aspect.

Mobile Client•Records the camera pose when taking the photo•Enables viewing digital information with sensors in situ and a far

Mobile Client•Records the camera pose when taking the photo•Enables viewing digital information with sensors in situ and a far

community content in the servicecommunity content in the serviceand roll)

•Unbroken, immediate chain from taking photos to enjoying them in enriched and contextualized way

•Brings together people, places and content into an easy to grasp reference model

•From dense network of photos the system creates a point-cloud model and provides enhanced navigation interface

and roll)

•Unbroken, immediate chain from taking photos to enjoying them in enriched and contextualized way

•Brings together people, places and content into an easy to grasp reference model

•From dense network of photos the system creates a point-cloud model and provides enhanced navigation interface

SummarySome augmented reality and mirror world opportunities:• Content centric communities

• With contextualization• Dynamic content – from users and commercial providers

• Improve and simplify photowork and other media management• Navigation• Commercial use cases

• Real estate agencies etc.


• Blended realities – Why?• Main metaphor: magic lens - annotate view of real world• Access information visually, on location

• Devices• hand-held camera phone NOW• data glasses FUTURE

Thank you !

Acknowledgements to Mixed Reality research teams in NRC-Tampere, NRC-Palo Alto and NRC-Helsinki


NRC-Tampere, NRC-Palo Alto and NRC-Helsinki

Background Material


Background Material

Nokia Research Center (NRC)

• Global corporate research unit

• Unique partner network

• Open innovation mode

• Industrial value from deep International organizations Corporations

Open source developers

Venture capitalcompanies

Operators &service providers

Universit ies & research institutes


• Industrial value from deep science

• Contribute to renewal and transition of the company

• Research in four focus areas

• Presence in Switzerland since 2008

Rich Context ModelingRich Context Modeling New User InterfaceNew User Interface

High Performance Mobile PlatformsHigh Performance Mobile Platforms

Cognit ive RadioCognit ive RadioCognit ive Radio

organizations Corporations

Virtuality Continuum*) – Our FocusSingle-playergames

Face-to-facediscussion


REALITY VIRTUAL REALITYAUGMENTEDREALITY

MIXEDREALITY

SOCIALLY-DEFINED, DIGITALLY-ENRICHED ENVIRONMENTS

INFORMATION OF THE PHYSICAL IS ENHANCING THE VIRTUAL WORLDS

AUGMENTEDVIRTUALITY

*) P. Milgram, et al., “Augmented Reality: A class of displays on the reality-virtuality continuum”

Mixed Reality Components – Our Focus

AUGMENTATION

LIFELOGGINGAUGMENTEDREALITY

… of people, objectsand places

Context-orchestrated rich content experiences and management, Experience Explorer

Interface & interact with your contextualized environment in situ, MARA Platform

INTERFACE & NETWORKS

INTERACTION

PERSONAL

INTERACTION

INTERACTION BETWEEN IN-PERSON

ESENTATIONS

Mixed reality services with sophisticated contextual interfaces for experiencing and managingyour digital life and your contextualized surroundings with the richest dynamic content

SIMULATION

INTIMATEEXTERNAL

VIRTUALWORLDS

MIRRORWORLDS

User-created mirror worlds as reflections of the physical world, Image Space

Extension of self-creativity and avatar interaction outside games

MODELS & IMMERSIONIDENTITY &

INTERACTION

SENSORS &

EVERYWARE

LARGE-SYSTEM

MONITORING

INTERACTION BETWEEN IN

AND VIRTUAL REPRESEN

Business

Smart Camera Applications