Augmented RealityPossibilities and Challenges

Parth Darji

Senior AR Engineer

Kagiso Interactive Media

Augmented RealityPossibilities and Challenges

Introduction to Augmented reality

Types of Augmented Reality

Abilities and Scope

Current Research

Demo applications

Introduction to Virtual Reality

Applications of Virtual Reality

Qualcomm Vuforia™ AR SDK

Hands-on experience of developing an AR application

Choosing AR and VR as career path

Conclusion

Questions and suggestions

Annexure: Further Reading

Annexure: Useful links

Quoting Wikipedia, “Augmented reality (AR) is a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data.”

Google says “Augmented reality (AR) a technology that superimposes a computer-generated image on a user's view of the real world, thus providing a composite view.

But I think Jay Wright, business-development director at technology company Qualcomm Inc. describes it best when he says – ‘It’s the real world – only better’.

Introduction to Augmented Reality

GPS and Compass based:

• This one uses the on-board GPS and compass to determine a user’s current location and augment the camera screen – intelligently.

• Usually for mobile phones and other hand-held devices.

Advantages:

• It can give the user an intelligent data about his/her surroundings by augmenting it with location markers, navigation, etc.

Disadvantages:

• The current limitations of GPS technology makes

• tracking inaccurate.

Examples?

Future scope?

Types of Augmented Reality

Marker-based

• Probably the most common AR type in the market currently.

• It uses a camera and a visual marker to determine the center, orientation and range of its spherical coordinate system.

Advantages

• Easily affordable, lots of SDKs in the market.

Disadvantages:

• Comparatively weak

• Few useful applications,

• Usually ends up being just a ‘wow’ factor.

New SDKs have entered the which allow

using real-world 3D objects as markers.

Types of Augmented Reality

Marker-less

• Currently the best technology for tracking.

• Augments real world environments without the need of specially placed markers.

• Allows more complex applications of AR.

Examples?

Types of Augmented Reality

Wearable techs:

• This one blurs the lines between augmented reality and virtual reality, in that wearable devices not only augment the environment around the user, but also try to immerse the user into a partly computer-generated environment.

• The recent buzz around Google Glass has been a pointer to what this technology is going to be in the near future.

• There is plenty of research going on on wearable techs and their applications in Augmented Reality and Virtual worlds.

• Other Examples: Oculus VR, Meta, DAQRI Smart Helmet, etc.

Types of Augmented Reality

Special mention – Vuforia Smart Terrain™

• A special type of AR which combines both marker-based and markerless.

• Here, one marker is used to calibrate the real world and the augmented world coordinate systems, and then anything else in the camera view can be used to reconstruct and augment the physical environment into its graphical counterpart, to create new kinds of gaming and visualization applications.

Types of Augmented Reality

Smart Terrain™ in action

• More examples?

Video Link: https://www.youtube.com/watch?v=9ZcB3-35VFE

So, after looking at the basic theory around AR, what could we possibly achieve using this technology?

• Entertainment

• Ease of access to information using graphically put content

• Road safety (iOnRoad)

• Games

• The “wow” factor

• Others?

Abilities and Scope of AR

Scope:

• AR is a relatively new technology in practice and there’s a lot of research going on in this area. Although as a concept, AR emerged as early as 1990 and a first head mounted display was built as early as 1996, AR has been available for open development very recently. With the current crop of wearable technology and cutting edge computer vision, AR is developing exponentially each passing day. I personally believe that within the next decade, it would be a technology we would use in our day-to-day lives.

Quick fact: AR and VR market is expected to grow at a CAGR of 15.18% from 2013 to 2018 and reach $1.06 Billion in 2018.

Abilities and Scope of AR

Majority of current research revolves around wearable techs. A lot of companies are working on Computer Vision, transparent displays, low powered processors, depth sensors, etc. to make the AR experience as immersive and interactive as possible. A lot of these projects are crowd-funded and others are funded by tech giants like Google and Facebook.

Israel is doing a tremendous job in terms of Computer Vision R&D and development of nano-sized depth sensors which we’ve started seeing in our smart phones (HTC One m8).

The next crop of smart phones would commonly have these depth sensors which would allow us to gather more information of the real world around us and develop applications which would use this data and do meaningful processing out of it.

A major role in AR application is played by how the application interacts with the users. So a lot of research is also going on in the field of Human Computer Interaction. Presenting such a complex technology to laymen is a tremendous task and researchers are working to make it all the more accessible.

A look at current research

Head mounted displays are becoming more and more popular, thanks to the advent of Google glass. The days are not far when we would be able to play our favorite RPG games where we would be the heroes of the game actually being inside it.

Smart watches are being a common thing these days and almost every big smart phone manufacturer is trying its hands on pushing a new smart watch into the market every now and then. With processors becoming smaller and more battery intensive, these watches would be able to do wonders. Can you think of a smart watch which has a camera on it and as you move your hand, it shows you the Facebook profile of the person you’re looking at? Fancy much, eh?

Think of your smart phone as a dashboard mounted GPS device. You can see the distance from the car ahead of you along with the speed at which it is moving so you can control your speed accordingly. You enter a destination and it colors the entire road green on your mobile phone so you no longer have to worry about missing a right-turn along the way. You’re busy talking to the person next to you and your phone suddenly shows you a red flag on the road telling you that you’re going off track?

Okay, how about a tablet for each student at school where he/she can scan the textbook and all the content comes up as an animated 3d model or a movie? How about that for learning? The DNA molecule that you had problems cramming would be popping out of your textbook and rotating in 3d and you can zoom in/out on particular areas which you find difficult to understand? Would that help you remember it better?

All this has been possible or will be possible only because of the rigorous research being put into the area of Augmented Reality and Virtual Worlds. The opportunities are endless and so are the possibilities. The only bottleneck to this technology would be your imagination and your creativity.

A look at current research

A lot of research is being put into using this technology for military purposes. AR could provide a lot of information to the troops about their surroundings.

Gaming: AR can drastically change the course of gaming. Current research is based on involving real world objects into a game as well as placing computer generated content in the real world which the user can interact with. Example: Zombie killer.

Medical: A lot of research is going on in the field of medical applications of AR. One such example would be superimposing MRI scan data directly on to a patient’s head so as to pinpont the location of a tumor. Similarly, AR can also help achieve greater insight into some of the most unreachable parts of the human body. Education: Researchers are trying to make lab experiments more immersive and educational. AR can help students learn more about the experiments they are participating in.

A look at current research

GPS and Compass based:

• Wikitude

• Yelp monocle

• iOnRoad

• Google Ingress

Marker based:

• Layar

• Blippar

Markerless:

• Kinect, Xbox games

Demo applications

“Virtual Reality (VR), sometimes referred to as immersive multimedia, is a computer-simulated environment that can simulate physical presence in places in the real world or imagined worlds.” – Wikipedia

Virtual reality can recreate sensory experiences, which include virtual taste, sight, smell, sound, touch, etc., but here we will primarily be looking at vision based virtual reality.

Basically, virtual reality is a technology where a user is tried to be placed in an environment other than the real world. For example, when you put on a head mounted display, you see a computer generated graphic content around you in a 360 degree view. You still are in the real world, but your senses perceive it as a virtual world.

Introduction to Virtual Reality!

Lets look at a video which exactly emphasizes the value of this technology.

Introduction to Virtual Reality!

Video Link: https://www.youtube.com/watch?v=dO4k2Rvs94I

Introduction to Virtual Reality!

Let’s look at another video – a demo video for Oculus VR and we would be able to better understand the concept, possibilities and scope of VR.

Video Link: https://www.youtube.com/watch?v=kznm_qMR0-4

Virtual Reality has been very popular with the Hollywood film industry. A lot of movies are made using green screen chroma-keying techniques and inserting virtual reality content in the real world.

Education and training

• Training and education can be done using VR because it can prepare you for dangerous situations and put the worker in risky real life situation without the risk of them being hurt.

• Examples: driving simulator, flight simulator, etc.

Museums

• Visitors can have a virtual tour to history by making them the characters of a 3d environment which depicts the details of history as it happened.

Others?

Other Applications of Virtual Reality

Now, lets delve into some practical stuff where we would pick an SDK and try to develop a working AR application from it. For simplicity, we will use the Android platform, but you can convert the application to iOS with relative ease.

For this, we shall choose the Qualcomm Vuforia™ SDK because of the following reasons

• Free to use (almost)

• Vuforia™ delivers best-in-class computer-vision, advanced detection, tight optical tracking, and cloud-based image recognition.

• Flawless Unity3d integration.

• Multiple platform support – Android and iOS.

• Variety of markers you can use for tracking.

• A community of over 1,00,000 developers so help is just a forum post away.

Qualcomm Vuforia™ Augmented Reality SDK

• Choosing Unity3d over native development?

• Advantages– Cross-platform development.

– Support of powerful .NET libraries

– Easy integration of 3D objects

• Disadvantages– Unity3d is an expensive software

– Increases application size

– Cannot use native library support

Qualcomm Vuforia™ Augmented Reality SDK

• Looking at all the prospects with this technology and the impressive things it can do, I think anyone with a passion of doing research in an evolving field would be greatly interested here.

• You get to work on something as interesting as space travel or virtual teleportation – More on this later.

• You get to take yourself and the world to whole different level of interaction with the technology around us.

• There is so much going on in the crowd funding domain that almost anything that is vaguely interesting would find a decent amount of money to work on the idea. All the tech giants are investing millions of dollars in this area of research.

• Tremendous research opportunities: There are universities that have dedicated AR and VR labs – MIT, GaTech, UCSD, UCB, Stanford, Cornell, etc.

• Do you see a future here?

Choosing AR as a career path

Conclusion

AR systems will instantly recognize what someone is looking at, and retrieve and display the data related to that view.

There are hundreds of potential applications for such a technology, gaming and entertainment being the most obvious ones.

Any system that gives people instant information, requiring no research on their part, is bound to be a valuable to anyone in any field.

As AR evolves, the fine line between real and computer generated data would become more and more blurry and our senses of sight, smell, hearing would be enhance.

So what do you think? Like it? Don’t like it? Indifferent?

Questions?

Suggestions?

Questions and Suggestions

We looked at one of the many AR platforms available in the market. You can go ahead and look at other SDKs that specifically cater to the need of your applications.

Let me list down a few of them here:

• Metaio

• OpenCV

• Total Immersion

• Wikitude

• PointCloud

You can also go ahead and check out gesture based SDKs like Kinect, Intel Perceptual Computing, xBox, etc. Details about these SDKs is out of the scope of this talk, but you can always get back to me if you’re interested and want more info

Annexure: Further Reading

• Really cool AR apps• Promising AR devices• Augmented World Expo - 10 best AR SDKs for

developers• Download links:

– Unity3d download– Vuforia Download - Registration required

• Tutorials:– Vuforia - Getting Started– Vuforia - Unity3d integration

Annexure: Useful Links