Application of Wearable Sensors and Augmented Reality in

Safety and Maintenance functions at Tata Steel Limited

Abhishek Kumar, Abhilash Agnihotri

System Integration Group, Automation Division, Tata Steel Limited

Jamshedpur, Jharkhand -831001.

Phone- +919234511868

E-mail: abhishek.kumar@tatasteel.com

In any manufacturing industry, maintenance crews work in hazardous areas such as

around moving conveyors , live power cables ,open pits and trenches or even poisonous

gases and fumes to name a few.

To mitigate this methods and apparatus for Personal Protective equipment (PPE) such as

Helmets and High Visibility jackets fitted with wearable sensors which can detect and alert

proximity to live power cables andtrack and monitor health and environment of the worker

indoors and outdoors has been developed. Helmet mounted camera system transmits live

video to the control room with immersive viewing facilities.

The alerts and hazards generated from the Smart PPE are monitored centrally and expert

viewing the live video feed provides assistance immediately to the field operator from the

central control room. This system provides equipment based specific condition monitoring

and maintenance data, thus increasing productivity and safety.

INTRODUCTION

Rapid advances have been made in digitalization, cybernetics and micro –electromechanical

Sensor technologies .Internet of Things leveraging the ubiquitous cellular and Radio

Frequency backbone with highly miniaturized electronics have now made possible wearable

technologies and Augmented Reality applications which were in the realms of science

fiction only a few years back.

ABSTRACT

Inevitably as these nascent technologies evolve, they find multitudinous applications in the

Manufacturing Industry. Some areas of applications where these technologies are getting

rapidly deployed are for inventory tracking through the supply chain, condition monitoring

of equipment, for Preventive maintenance and for personnel safety. As low hanging fruits

these technologies can be quickly and cheaply deployed to give organizations sustainable

competitive advantages and new insights to improve their operational efficiencies.

Several solutions leveraging the above technologies have been developed and deployed in

Tata Steel as part of its digitization initiative. This paper discussed a system which has been

developed recently which utilizes Wearable Sensors, Augmented Reality and associated

technologies for plant maintenance and personnel safety.

RELATED TECHNOLOGIES

Augmented Reality:-

Augmented Reality or Mediated Reality can be defined as Real World/Physical environment

supplemented or Augmented with sensory data (Sight /Sound/Smell/Vibration etc.) with the

help of computers and electronics. A Typical example/application ofaugmented reality

technology is the Heads UP display provided in the helmets of fighter pilots which project

critical flight data on the windscreen of the Helmets worn by fighter pilots so that they do

not have to look down at the control panel. Now this technology is commercially available

(See Fig.1.HUD for a Car) and has changed the paradigm of the conventional Human

Machine Interfaces or the computer screens familiar to us. From Interacting conventionally

with a computer HMI using a keyboard and mouse the operator can be shown the entire

process in 3 dimensions projected on their consoles and the keyboard can be projected on

their desk or any flat surface (See Fig1B--Laser Keyboard). They can rotate, dissect and

inspect the 3 dimensional holographic image of the equipment (See Fig.1.C&D) to diagnose

problems and also obtain expert help at remote locations using these immersive viewing

/display technologies.

Fig.1.0: Examples of Augmented Reality

Wearable Sensors:–

With miniaturization of micro controllers and sensors it is now possible to have a button

sized micro controllers with 8 I/O‟s sown on the Protective jacket or pasted on the helmet of

any personnel. These low cost micro controllers can be used to monitor the ambient

temperature, Air quality and health of the personnel and also transmit in real time situational

data such as locationGPS or LBS and position(upright or fallen) from micro accelerometers

and GPS devices embedded in the Garments. This information transmitted to the control

room can provide lifesaving assistance if such as case arises.

Wearable sensors such as products from „Adafruit‟ (Fig 2) and Grover provide low cost

wearable wireless sensors used in this system which can be stitched into clothes using

conductive wires and fabric these low powered devices connect to micro controllers using

the Arduino programming environment. These micro controllers can be stacked with various

devices and modules known as shields which range from sensors to WiFi, Zigbee, Blue

tooth or other communication modules as well. These are open source systems and are

available for integration off the shelf.

A B

C D

Fig.2.0: Wearable Sensor from Adafruit Industries.

Tracking/Tagging /Positioning Systems –

The essence of the augmented reality system for maintenance is to provide contextual,

relevant information to the maintenance personnel based upon their location. Location

information is also necessary to track / monitor the personnel working in hazardous areas.

Of the several technologies which have been incorporated in the system, some have been

available for several years such as RFID, GPS (Global Positioning System) and LBS and

Barcoding and these do not need much explanation. Some new Technologies which have

been adapted for this system areBluetooth Beacon technology called Eddystone API which

is an open platform developed by Google, originally for proximity based advertising

tracking

Fig.3.0: QR Code and NFC Beacon Schematic

Blue tooth beacons using the “Eddystone” profile (See Fig.4) developed by Google provide

similar location and beacon functionality. These are used to broadcast advertisements and

store locations and special offers inside shopping malls or location of booths in a conference

with many stalls. These beacons can deliver advertisements to the Bluetooth enabled smart

phones nearby.

Fig. 4.0: Google Eddystone Beacon Data Format and Encoding

QR codes or 2D which are an advanced version of the Bar codes and NFC(Near Field

Communication Tags (See Fig.3) are also used for providing tagging or location based

information in the system as a cheaper alternative to the Bluetooth Beacons or RFID (Radio

Frequency Identification) systems .

Telemetry Networks

For data transfers from the field to the Central control room multiple RF technologies have

been used for Telemetry and data communication .As cell phones are available with all

personnel, the wearable sensors piggyback data on the GSM/GPRS networks of the cell

phone through blue tooth connectivity to the cell phone. For the Immersive viewing and

remote monitoring components of the system, 5.8 Ghz License Free ISM band has been

utilized for video transmission .Blue tooth Technology and Wi-Fi on 2.4 Ghz have been

used for indoor tracking of personnel.

Table 1.0: Performance characteristics of various Telemetry technologies

Multiple RF bands have been selected on the basis of their propagation characteristics such

as Range, bandwidth and resilience to interference .For critical information with low data

volume Cellular networks have been preferred while for high Data volume short range Wi-

Fi and Wi-Max networks have been deployed.

Inductive Proximity Sensing

Inductive fencing systems have been developed by companies for controlling live stocks and

pets .Companies such as pet safe market products in which a wire buried underground

transmits a signal which is received by a receiver. More recently wireless systems which are

located centrally and radiate an Omni directional signal .These do not require the

cumbersome process of burying an antenna wire. Such systems work on the Chirp spread

spectrum technology implementation.

Immersive Viewing Systems

Micro CCTV system with head tracking and First Person viewing using 5 Ghz or 2.4 Ghz

band have been developed by Aero-modeling companies for hobbyists flying drones and

remote control air crafts. These devices are marketed by companies such as Boscam and

Fatshark . The GoPro Company manufactures cameras and gimbals which can be mounted

on helmets and mountain bikes or skate boards while playing adventure sports or even for

water sports but do not have a video transmission or pan tilt facility.

Fig: 5.0: Immersive viewing and Head Tracking System

FEATURES OF THE SYSTEM

System Overview

In a manufacturing industry such as steel industry carrying out maintenance work is fraught

with hazards. It is also desired to track the work team if they are working in such hazardous

locations. While doing maintenance work to improve productivity it is desirable to provide

the correct documents and SOP (Standard Operating procedure) for the specific equipment

maintenance history of the equipment and location and availability of spares and

replacement schedules etc. This can improve the productivity of the equipment and

maintenance teams by reducing maintenance times. It is also desirable to track the people

working in underground cable tunnels for their safety.

There are certain work stations in a plant, where round the clock vigilant monitoring of the

equipment is required. In case personnel leave their station for any reason, a reliever should

be present or the system should be shut down safely, examples being conveyors and

crushers in mining industry.

To address these problems a system was conceived where- in wearable sensors are

incorporated in the mandatory safety equipment worn by the workers. The sensors on the

invention can detect when the worker is approaching a hazard or leaving a work place.

If a person falls or become incapacitated the system triggers a SOS to the control room

automatically.

Live streaming video over an immersive viewing system provides for remote assistance and

diligent monitoring of the work by experts from a central location. It tracks people when

they are working under ground by means of beacons and above ground by GPS.

During maintenance work pertinent and relevant maintenance manuals and documentation

are made available to the team when in proximity of equipment by means of the beacons,

NFC tags or QR Code stickers. By Scanning the QR code or NFC tag, the Hand Held device

of the Maintenance crew is automatically connected to the Maintenance Server. The relevant

maintenance checklist is downloaded and information on Maintenance history, list of earlier

problems, spares availability, dismantling guide etc. is downloaded on the Hand Held device

of the Maintenance crew.

Using Proximity based tags; it is also ensured that the maintenance team has to be physically

present near the machine to access the checklist, thus preventing proxy or offline entries.

Also by means of the Beacons and Wireless Networks , location of persons working

underground or indoors can be tracked, while movement outside can be tracked using GPS

sensors. These sensors stitched on the High Visibility jackets of the workers. Accelerometer,

Gyroscope, pulse sensors indicate to the control room if the person has fallen or

incapacitated in any way. Carbon Monoxide detectors alert the wearer and control room of

gas leakage.

A micro CCTV system enables live video transmission to a safe location from where

assistance or expert help can be provided. This system thus enhances safety and productivity

of workers in a hazardous environment.

Another objective of the Smart PPE system is to implement a positive isolation system

whereby hazardous areas are demarcated with a buried conductor or a conductor embedded

in a rope or metal cordon, which acts as a radiating antenna. The buzzer vibrates to alerts the

wearer of the receiver when it nears this cordon. This antenna may also be a centrally

located Omni directional radiator without requiring a need for burying a conducting cable.

This allows users to provide variable lengths of cordoned area possible by enabling

transmission of a proportional frequency through the tunable Signal generator depending on

cordon length which is the antenna, (antenna length being a function of frequency) and

thereby reception of several frequencies by the receiver simultaneously through mechanisms

of filters and associated circuitry.

This is done so that alarms are generated only from those area/power lines considered as

hazardous and create false positive alerts are reduced.

As process areas are inherently noisy, audible visual alarms may be overlooked, therefore a

piezo buzzer is embedded inside the helmet which vibrates near the temple of the wearer

and cannot be ignored or overlooked.

The Smart Helmet also has a feature toprovide live video transmission to an expert located

centrally. The expert can remotely control the pan tilt of the camera and can have an

immersive viewing experience by using immersive viewing goggles. This live video can

also be seen on a video monitor and recorded

To conserve the battery and also to monitor if the personnel are wearing the smart PPE,

conductive cloth has been used along with pressure switches to fabricate the Chin strap. The

movement the chin strap goes slack, information is triggered to the control room that the

Smart PPE has been removed and after a defined interval the system is turned off

automatically.

Description of the Wearable Sensors and Smart PPE system

The Smart Helmet (Fig6) incorporates a battery pack 11 with a USB charging port 10,an on

off switch 8, a battery indicator 9 and charging port 10.This battery pack contains a suitable

Lithium polymer battery in explosion proof casing, which powers the other devices a micro

-camera 13 with pan tilt features and a proximity sensor /receiver 6 .This battery pack 11 is

so designed that it can be installed and removed easily for replacement.The micro camera

with pan tilt facility is used to transmit video and receive data respectively from an

Immersive viewing goggles 4 with head tracking facility and a LCD monitor with Diversity

antenna 15, over a 5Ghz frequency band.

Fig.6.0: Schematic of the Smart Helmet System

The range of this system is typically 300 meters in line of sight outdoors or about 100

meters indoors.A conducting media such as thin single strand copper or metal wires of

armored sheath can be used as the transmitting media or the signal generator can be

connected to a monopole antenna which transmits radially from a central location. The

signal generator 2 can be battery operated or connected to the mains power. The receiver 6

denoted as an inductive proximity sensor mounted on the said helmet 14 receives this

transmitted frequency when it nears the radiating antenna 3 which earmarks the boundary or

fence of the hazard area. An amplifier circuit embodied in the receiver 6 and induction coil

amplify the signal as the fence is neared. Depending upon the proximity of the receiver 6 an

escalating level of vibration and buzzer intensity can be provided to the wearer. The chin

strap of the helmet14 incorporates pressure sensor which can detect if the helmet if removed

from the persons head and turn off the system to conserve power, while sending an alert

message to control room.

Fig.7.0: Smart Jacket with Wearable Sensors

As summarized in Fig 7 a high visibility jacked 22 has incorporated within several wearable

sensors. These sensors known as shields are stitched using conducting thread and cloth. The

sensors such as Accelerometer, CO gas detector, Gyroscope, pulse monitor etc. 19 are

connected to the input ports of a micro controller 21. This micro controller is enabled with a

Wi-Fi/GSM module which is used to transmit the sensor data to the control station. A GPS

sensor 20 is also incorporated in the jacket and connected to the micro controller 21. These

devices are powered by a battery pack which hasa trickle charger and is connected with a

solar panel 18 stitched into the jacket.

The Central control system is on the LAN 17 and runsproprietary software which denotes

the sensor status, location data received from each person wearing the Invention, reports etc.

Fig 8.0: Indoor Personnel Tracking System using Beacons and Wi-Fi infrastructure

As shown in 8.hen the worker is moving indoors in a cable tunnel or confined space, Blue

tooth beacons enumerated as 25 in .8 will be used to track location. The beacon 25 based on

the Eddystone profile developed by Google Inc. are devices which can detect blue tooth

low energy beacons and transmit the same over a Wi-Fi network 27 or transmit a beacon

with parameters such as location id, signal and health of the device and other messages. A

person can be tracked in such location by either carrying a beacon which is detected and

transmitted over Wi-Fi network to the central station or by carrying a smart phone which

can receive these beacons broadcasts. Also as indicated in the figure 8 the beacon 25 when

located near any equipment can enable the smart phone to connect to the Wi-Finetwork and

connect to the central computer. The Smart PPE and Augmented Reality system may be

configured in a variety of ways and is not limited to the representation described here

INDUSTRIAL APPLICATIONS AND CONCLUSION:

The Smart PPE and Augmented Reality based Maintenance system can be used in a multiple

configurations and combinations. The beacon, smartphone Wi-Fi combination can be

utilized for effective maintenance of the equipment. The helmet and Beacon system can be

used to ensure positive isolation of hazardous areas and prevent unauthorized entry at such

locations while the CCTV system can be used to provide remote expert assistance and

guidance.

This system can be used to track and monitor workmen entering hazardous area with

chances of exposure to poisonous gases and in locations where conventional GPS cannot

work in confined spaces, indoors and underground.

Fig.9.0:Integrated Plant Maintenance using Augmented Reality Technologies

The system can be marketed as a combined unit or in separate modules at different price

points for enhancing safety and productivity for workmen exposed to hazardous

environments in all industries/agencies engaged in maintenance of equipment generally and

steel industry specifically.

The challenge is to drive down cost by economies of scale and modular design to make this

affordable and cost effective proposition. The cost of the system using discrete components

is around 3000 INR, excluding the immersive viewing system which alone costs around

Rs.30,000.The challenge is to bring this price to below 300 Rs. using indigenous

components.

REFERENCES

1. https://en.wikipedia.org/wiki/Augmented_reality

2. Newman, Richard L. “Head Up Displays: Designing the Way Ahead” , ISBN 0-291-39811-1

3. T. Olsson, D. Gaetano, J. Odhner, and S. Wiklund.”Open Softwear 2nd

Edition”.

4. https://github.com/google/eddystone

5. Denso.”QR Code Essentials”.

6. Robert M Dondero . ” Programming the Webserver”

7. J Annuzi, Lauren Darcy, Shane Conder .”Advanced Android Development,4th

Edition .Addison-

Wesley”

8. PanagiotaFatourou.” Introduction to Sockets Programming in C using TCP/IP”

9. Jacob Millman,ChristosHalkias,Chetan D Parikh.”Integrated Electronics AnalogAnd Digital Circuits

And Systems Paperback (English) 2011”.

Abhishek Kumar