-

-

AR is a game changer that is boosting operational efficiency, transforming customer engagements and creating new business opportunities

Exploiting the Potential of Augmented Reality

2

Figure 1: AR offers a modified view of reality using computer-generated sensory inputs

Augmented Reality (AR) is a promising emerging technology that is capturing the imagination of many businesses. This paper aims to present an overview of AR, its applications and underlying technologies, and explore how AR can improve productivity and efficiency and potentially transform the nature of work as we know it.

Augmented Reality is a field of computer science where “a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. It is related to a more general concept called mediated reality, in which a view of reality is modified (possibly even diminished rather than augmented) by a computer. As a result, the technology functions by enhancing one’s current perception of reality.”1

AR can be viewed as a natural consequence of digital evolution. It’s an example of how technology has the power to change the way we perceive and interact with our environment. The potential of AR is made possible by the mobile

revolution, which brought technology into the hands of people in the forms of smartphones, tablets and now wearable devices. Devices such as cameras are embedded into these devices with programmable interfaces. With touch as a primary input method, mobile devices are very cogent in the experience they provide. In the un-augmented real world, it is not possible to interact physically with every object or parts of objects. This could be due to obstacles in terms of size, location, safety or physical characteristics, such as fragility.

Consider a car showroom. There are several models of cars for display and the customer can be overwhelmed when a salesperson explains the details of each model. AR lets the customer learn about the various models on their own terms, using an immersive experience that can provide the information they need to make the buying decision.

AR technology involves digitally recognizing an object and augmenting it, for example by enhancing it with relevant content including imagery, videos and documents. It allows users to engage with real-world objects by taking advantage of sophisticated visual assimilation techniques. It removes obstacles and delivers a deep level of interaction with the object and the environment. Depending on the type of object, the augmentation can create an immersive experience leading to an improved cognition of the object. With mobile devices, the experience is enhanced with various input techniques such as touch and voice.

Consider again the application of AR in the car showroom. When a customer enters a showroom, they are handed a mobile device such as a tablet with predefined software applications deployed. The customer launches an app that internally launches the device camera. When the customer approaches a particular car, the app intelligently identifies the car and augments the live frame on the device with relevant content. This might include text about the key features of the model, a video demonstrating the innovative engine technology, or an animation of the interior.

AR can be applied in countless scenarios—factories, offices, and commercial and public spaces—with the objective of delivering rich, experiential information, insight, and education unmatched by any other

The Case of AR01

Reality

AugmentationContent

EnhancedPerception

1. Augmented Reality, Wikipedia, https://en.wikipedia.org/wiki/Augmented_reality

3

Figure 2: Information Augmentation for machines using AR helps employees in a manufacturing facility

Business Use Cases

Both large and small businesses are at the forefront when it comes to adopting and applying AR. The technology is being used in a variety of scenarios from employee training to customer engagement. It has the potential to improve operational efficiency and open new avenues to innovate both products and services that enhance customer engagement and loyalty, and drive competitiveness. What follows are four examples of how AR is being applied today in the business world.

Manufacturing

Optimizing operations by increasing efficiency is a constant focus in the manufacturing sector. Many production tasks require quick and easy access to the right data at the right time to perform a specific task. For example, consider a large shop floor with a variety of different machines. The operation of each of these machines includes monitoring them while in use and scheduling regular maintenance. Given the large amount of disparate information available on each machine, it is usually not possible to have instant access to information on a specific machine’s operational settings or maintenance and repair records.

Using AR, all of this information is accessible instantly to the employee. Each machine can be defined in the AR system and all the relevant data associated with it can be filed. Using a mobile device, the employee can learn about the machine, access relevant data, and take necessary actions to operate, maintain or repair

Warehouse Management

Large warehouses and distribution centers are common in many industries, acting as inventory hubs for manufacturers and retailers, including e-commerce retailers. Some double as sales centers accessible by customers. With millions of SKUs in inventory, it can be challenging for workers to find accurate information about the products and manage the picking and replenishment of inventory. Warehouses that service customers directly face the added challenge of accessing detailed information about the products.

AR can be a very effective solution to help warehouse workers/customers manage the products on the shelves so that they can make accurate selections. For example, the company can provide workers with an app to use on their mobile devices that augments the image of a product with the relevant specifications when viewed through the live camera frame. This can increase productivity by decreasing the chance of workers selecting the wrong product to be shipped. In warehouses that allow customers direct access, the app can significantly improve their shopping experience by helping them find,

technology. AR can be a game changer for the business world, given the need for companies to boost productivity, increase the skills of their employees, and create deeper engagements with their suppliers and customers.

the machine. They can take advantage of technologies such as virtual assistants along with AR that collect and analyze data continuously, providing insights and actionable tasks that lead to simpler and more efficient solutions that boost both capital and labor productivity.

4

Figure 3: Warehouse workers can manage products on shelves using AR

Figure 4: Pre-loaded AR on mobile devices enhances training by providing interactive wizards

have to provide detailed instructions and information to workers and educate them on each individual machine and its components.AR is very useful in such scenarios. Trainers are given mobile devices preloaded with AR-enabled training modules, where each module acts as an interactive wizard to guide the trainees through each step. It also expedites the process of learning as the trainees can repeat the wizard until they are proficient in the module. The learning process can take place independent of the trainer.

Training

Hands-on training is an essential part of learning and development process at many companies. The more complex the machines or processes are, the more challenging the training process and the longer it takes. For example, consider a product assembly process in a manufacturing plant that requires workers to manage a variety of complex machines. Trainers

Product Marketing

Product marketers are constantly trying to renew and reinvent the product marketing process to engage and excite the customer. Storytelling is integral in the marketing process. Businesses aim to continuously reorient themselves to adapt

evaluate and select the right products with minimal involvement from staff. Also, the app can provide accurate information about the kind of products customers buy and the factors involved in the decisions they make. This data can be analyzed and used, for example, to manage inventory levels and store popular products closer to the customer service desk.

to changing market conditions and tell compelling stories about their products to existing and potential customers. Successful storytelling is the starting point for increasing sales and market share and driving customer loyalty.

AR helps marketers create a “wow” factor that is integrated into the storytelling through powerful visuals and content. The AR solution enhances the customer experience and can increase the likelihood of a sale. Consider the car showroom example described above. A customer is provided with a mobile device that has a preloaded AR module to use as they tour the showroom floor. As they learn about the features and attributes of the car models, they are also being marketed to through the imagery and the messaging of the content.

Figure 5: AR helps marketers provide immersive experiences for their customers

The opportunity is similar in other industries, such as retailing. AR solutions can help retail marketers present their products in innovative ways. For instance, a retail clothing store that provides shoppers with AR-equipped mobile devices might include the option of selecting the product category the shopper is looking for and directing them to that department. The visual enhancements might include information and visuals about items on sale and the quality of the products they are shopping for, as well as direct them to articles of clothing that are complementary.

6

Figure 7: Object definition in an AR solution

Solutions02Solution Model

There are a variety of models included in the range of AR solutions, from those that are device-only to others that include a client and server. Regardless of the type, the typical solution comprises the following four categories: Object Definition Object Classification Object Detection and Tracking Object Augmentation

Object Classification

Once the objects of interest are identified and defined in the system, they are classified into specific groups based on either predefined or user-defined parameters. The goal of the classification is to enable best possible detection of the object when it is detected by the device. The object classification is done by extracting features from the visuals—still images and videos—that were collected during the definition of the object, either using an automated or user-triggered process. These features can be shape, contour, color, histogram, texture or other characteristics. Further, content creators use algorithms to extract features from digital images, including the size/rotation/illumination invariant. SURF (Speeded Up Robust Features), SIFT (Size Invariant Feature Transform), GLOH (Gradient Location and Orientation Histogram), and ORB (Oriented FAST and Rotated BRIEF) are some of the algorithms widely used to create image descriptors and feature extractions.

Object Definition

Object Definition is a key step in developing an AR solution. Content creators first define the object and collect and save visual representations of the object that may include still images and videos. The visuals are captured in as many dimensions as possible with the best possible resolution and quality and are used in subsequent process steps to classify and detect the object. Content creators also define tags or areas of interest in the object that identify specific parts of the object and associated content. This content is used to augment the live scene that is captured on the device. This content can include plain text,

01 —

02 —

03 —

04 —

documents, videos, images, animations, and other visual constructs.

Figure 6: The four elements of a typical AR solution

Object Definition

Object Augmentation

Object Classification

Object Detectionand Tracking

Visual Elements

Attribute 1Attribute 2

Attribute N

.

.

.

Object

Content

Aa

Figure 8: Object classification in an AR solutionFigure 9: Object detection and tracking in an AR solution

Figure 10: Object Augmentation in an AR solution

7

Object Detection and Tracking

Detection of objects is an end-user triggered process. A device with a camera is established to detect and augment the objects of interest that have been predefined and classified. The camera captures the current live frame and provides it as an input to the detection algorithms that are associated with the classification being performed. Several combinations of features and image descriptors on the live frame are used along with feature-matching algorithms such as FLANN matcher/brute force to detect the object.

Once the object is detected with the movement of the device in the user’s hand and the change in the user’s position, the object is tracked in the live frame.

Object Augmentation

Augmentation of the detected object on a live camera frame involves enhancing the object in the live camera frame with visual elements that are predefined or generated based on the properties defined for the object. The associated tags and content that are defined along with the object can be used to construct these augmented visual elements. Animations and graphics can also be used together with standard elements—such as text, images and video—to enhance the live scene.

Objects

Shape based Classification

Classifier

7

Objects

Shape based Classification

Classifier Objects

Color based Classification

Classifier

Figure 11: A device-only AR solution

Figure 12: A client-server based AR solution

8

Solution Architecture

AR solutions involve understanding reality and converting it into a digital form. The most efficient and widely available transducer to do the job is the camera. A live feed from a digital camera is used to capture and convert the image of an object into an understandable format that AR uses to detect and recognize the same live object when the camera is focused on it again. AR solutions can be implemented differently based on factors such as how and where the solution is being applied and the scale of use. The type of solutions can be classified as following: Device Only Device and Server

Device Only

A device-only solution will have all the components associated with the process steps of AR described in the solution model above but will reside on a single device. A simple, yet robust AR solution can be implemented with a camera that has an acceptable minimum resolution, and good processing and storage capability. The following diagram illustrates a device-only solution:

Device and Server

A device-and-server solution has components associated with the four process steps of the AR solution but they are distributed across a device and a server. This typically is a client/server model in the application world. The following diagram illustrates the device-and-server solution:

The main disadvantage of the device-only solution is the limited capacity—both processing and storage—of the device. The solution can only scale as much as the device capacity can support. For example, on a device with 64 GB storage, the size of the image, video and classifier data will be limited to a maximum of around 56 GB—excluding the operating system—and the processing speeds will be limited by the number of cores on that device.

01 —

02 —

Figure 11: A device-only AR solution

This solution has the advantage of utilizing the power of a server or cluster of servers with higher processing and storage capabilities than the device for feature extraction and training. This enables the solution to scale. The detection process can be initiated either on the server or on the local device depending on the capability of the network available to the device.

As with any client/server application, network latency is a disadvantage for the AR device-and-server solution. When the process of detection is executed on the device, it leads to a solution closer to the device-only solution with the exception that the image classification and training is performed on the server and all the defined objects and their associated attributes are stored on the server. The required data for detection and augmentation can be synchronized with the device, thereby exploiting the capabilities of both the device and the server(s).

Device-only solutions are useful when they need to be executed independently of a network. For example, if a small, packaged product is to be marketed to customers across different locations by augmenting information about the product and creating a compelling AR customer experience, the marketer can use a device-only solution and create this experience without dependency on the network.

Benefits

9

While physical paper catalogs and manuals are

useful, it would be difficult for anyone to

remember every detail of an object that they deal

with, based only on what they have read. AR

solutions help enhance the assimilation of

information by utilizing visual images, sound and

touch. These solutions render complex sensory

information about objects that enhance learning.

03Enhanced Assimilation

In a world where digital systems have redefined

the way we capture and store information about

various objects, being able to render such

information as required plays a significant role in

improving productivity. Conventional interfaces

such as web-based and mobile interfaces that

communicate digital content can enhance

learning and focus attention. However, an

interactive experience has the capability to more

fully capture and hold the user’s attention and

keep them engaged. Interactive experiences are

the core of AR solutions and contribute to

increasing productivity.

Increased Productivity

Information about complex objects and systems

is difficult to visualize and communicate to

others. This could be due in part to the sheer

volume of information about, and the structural

complexity of the object or system. AR solutions

can help manage this complexity by providing

content for augmentation for different part and

Managing Complexity

levels of the object. This layered approach helps

users learn the details of each part and

component while progressively building an

understanding of the overall object or system.

For example, consider a complex office all-in-one

printer that provides capabilities such as image

scanning, messaging and photocopying. The

printer has several components such as paper

trays and toners. Understanding these

components is necessary for maintenance. AR

can help visualizing the attributes of each of the

components of the machine cascading down from

the most complex part to the least complex part.

Users of digital systems in general prefer to

collaborate with other users—family, friends and

co-workers—to shape opinions and make

decisions about the products they purchase. AR

solutions can accommodate collaboration by

including live interaction between multiple users

who share their views and opinions during the

AR experience.

Collaborative experiences are particularly useful

for remote training. For example, consider a

scenario where a trainer is explaining the

functionality of a sophisticated machine to a

group of people in a different location. When the

trainer focuses a live camera device on the

machine and its key components, AR can help

trainees view augmented content such as a video.

With an appropriate solution, the trainees can

also point to an area of interest in the video and

ask the trainer for more information through

augmentation.

Collaborative Experiences

One of the key challenges in developing AR

solutions is creating object-detection algorithms

that can identify objects in all orientations, sizes

and lighting conditions. However, there is no

one-stop, sure-fire algorithm that can

consistently deliver a 100% match during the

object detection. Hence, the aim in detecting

objects is to maximize the probability of correctly

detecting the object. This uncertainty requires

solution designers to make assumptions about

certain scenarios, which may limit how well the

solution can work beyond the scenario they are

designing for.

04

10

Challenges

Accurate Detection and Recognition of Objects

Another challenge with object detection is the

issue of “false positives.” A false positive is a

match to an object that does not exist in the

template/definition within the solution. That is,

while defined objects are correctly detected as

expected, the objects that aren’t valid or not

defined are incorrectly detected to be valid

objects. This issue can be resolved by providing a

larger sample of images and videos to train the

system and by applying customized algorithms

that can filter with greater granularity during

detection.

For example, consider two objects such as an IP

phone and a notebook computer that are the

same color. If there are not enough images for

either of these objects, it is possible that the IP

phone could be detected as a notebook and vice

versa.

False Positives

Mobile devices come in various screen sizes and

dimensions. The amount of information that can be

augmented for any given object on a device depends

on the screen’s available real estate. For example,

when there is a defined amount of information that

can be used to augment the object, it becomes more

challenging to manage the augmentation the smaller

the device’s screen size. A trade off needs to be made

between how generic and pared down the solution

can be made versus how many tailored solutions can

be made for different devices.

For instance, consider an iPad and an iPhone. The

screen sizes of these two devices are quite different.

For the same object, the amount of information that

can be augmented on an iPad will be greater that on

an iPhone. If the same amount of information is

attempted to be shown on the iPhone as the iPad, it

may lead to information clutter that will distort the

view and hamper the user experience.

Size of Mobile Device

Accurate tracking of an object in a live frame as the

user moves the mobile device around the object in

the contextual space is dependent on the quality of

the sensors and GPS in the device. These factors

can limit the deployment of the AR solution on a

limited set of devices based on their hardware

capabilities.

Accurate Object Tracking

Augmented Reality solutions rely on visual

input—images and video—of objects captured and

trained to eventually be detected and recognized. It

is important to ensure the sources of these inputs

are authorized to collect and store the information.

While this is relatively low risk for in-house worker

training and education solutions, it can be an issue

for public-facing solutions.

Security

11

The recent release of the Pokémon Go app is an

evidence of how geo-location is being married

with AR to provide an immersive user experi-

ence.

All the object recognition challenges for AR

solutions can be reduced to some extent by using

the user’s location as one of the dimension for

overlaying the data and identifying the object.

For example, if the user is in front of a building

that is a point of interest (POI) and the AR

application is able to display the building’s floor

plan, then the AR app need not rely on image

recognition. Instead it can use the location of the

user as the primary image recognition input to

identify the building.

05 Location and AR In certain cases, AR can utilize the camera’s depth

sensor, a new component that is starting to be

introduced into tablets. As devices get more sophisti-

cated, and accessories such as glasses and wearables

mature, usage of AR solutions will increase.

One area of location-based AR of interest to develop-

ers is the automotive market. Auto use cases will use

location-based input to stitch the live stream from

multiple cameras in the car with POI data to display

augmented images on the windshield.

Augmented Reality on various visual interfaces is

a promising paradigm providing an opportunity

to create immersive experiences for users. Given

the pervasiveness of mobile devices such as

phones and tablets, creating these experiences

The Way Forward06

12

can help companies grab a larger mind share of

users and increase brand loyalty. As the

Internet-of-Things wave spawns new devices and

enhances existing ones, AR is poised to play a

potentially pivotal role in creating the next

generation of user experiences. With the

technology evolving rapidly, now is the time to

invest resources to develop AR solutions that will

define the future of the user experience.

Aricent is a global design and engineering services provider working with market leaders to

develop best-in-class products and services in the areas of communication infrastructure,

software and internet services, and embedded systems. Aricent has developed a software

framework that provides application programming interfaces (APIs) for developing integrated

Augmented Reality applications and solutions on mobile platforms for different domains.

The key features of Aricent’s Augmented Reality Framework (ARF) include:

Image Processing Engine

Feature Extraction

Multi-Class Object Detection

Gesture Recognition

Proximity Sensing, Ambient Location, Absolute Location (for both indoor and outdoor

locations)

Content/Template Definition

Floor Map Integration

Activity Mapping (Location, Image/Text, Content, Gesture Events, Point of Interest)

Alerts and Notification

Barcode/QR Code/OCR-Bbased Detection of Objects

About AricentAricent is a global design and engineering company innovating for the digital

era. With more than 12,000 design and engineering talent and over 25 years of

experience, we help the world’s leading companies solve their most important

business and technology innovation challenges - from Customer to Chip.

© 2016 Aricent. All rights reserved.

All Aricent brand and product names are service marks, trademarks, or registered marks of Aricent in the United States and other countries.

13

Contacts

Raghu Kishore Vempati, Principal Systems Engineer, InnovationEmail: raghu.vempati@aricent.com