1 Copyright © 2014 Tata Consultancy Services Limited
Fusing Personal Context with Physical and Physiological Context for creating value-added crowd-sensing applications
22nd May 2015
Arpan PalPrincipal Scientist, Innovation LabsTata Consultancy Services Ltd.
2
Pioneer & Leader in Indian IT
TCS was established in 1968
One of the top ranked global software service provider
Largest Software service provider in Asia
300,000+ associates
USD 15Billion+ annual revenue
Global presence – 55+ countries, 119 nationalities
First Software R&D Center in India
Tata Consultancy Services (TCS) at a Glance
Bangalore, India1
Chennai, India2
Cincinnati, USA3
Delhi, India4
Hyderabad, India5
Kolkata, India6
Mumbai, India7
Peterborough, UK8
Pune, India9
2000+ Associates in Research, Development and Asset Creation
Singapore10
Innovation @ TCS
TCS Connected Universe Platform (TCUP)• M2M Communication• Distributed Computing• Sensor Integration and Management• Analytics ServicesContext-aware Applications• Healthcare• Insurance• Retail• Manufacturing• Smart Building / Campus• Smart Villages / Cities
Overview
10 Corporate Innovation Labs
Co-innovation Network (COIN) with Academia and Industry
Internet-of-Things Research
Three stage Innovation Process – Explore, Enable. Exploit
3
Agenda
Context Discovery using IoT
Application Use Cases – Physical Context
Evacuation, Insurance, Retail
Physiological Sensing – Mobile and Wearable
HRV, BP, EEG, GSR
Behavioral Model from Physiological Sensing
4
The Internet of Everything
Humans
Physical Objects and Infrastructu
re
Computing Infrastructu
re
Peo
ple
Con
text
Dis
cove
ry
PhysicalContext Discovery
INTERNET OF EVERYTHING
Physical Context
DiscoveryWhat is happening,
where and when
People Context Discovery
Who is doing what, where and when, who is
thinking what
Internet of
Digital
Internet of
Things
Internet of
Humans
ABI Research. May 7, 2014
New Business / Pricing Models
5
Understanding the People Context
Non-intrusive, un-obtrusive sensing
Identity, Location, Activity, Physiology
Understand Behavior – Individuals / Groups
Quantified Self
Customer becomes the focus, not the product or service – key is understanding the Customer, Extend B2B to B2B2C
Using Wearable's and Nearables
(mobile phone, camera, mic, ….)
6
Context Discovery - Multi-dimensional Fusion
• Panic• Stress• Like / Dislike
• Weather• Environment
• Network• Likes and
Dislikes
• Location• Activity• Proximity
Physical Social Media
PhysiologySurroundings
Contextual Information
7
Click to edit Master title styleApplication Use Cases
8
Application Use Cases
• Floor plan based capacity planning
• Location based recommendation
• Behavioral Sensing – panic / proximity
Emergency Evacuation
• Hard Cornering / Braking / Harsh Acceleration from Accelerometer
• Driver Scoring• Road / Traffic /
Weather Condition
• Behavioral Sensing - Stress
Driving Behavior
• User profiling from usage / social media
• Location based Recommendation
• Environmental Effect
• Behavioral Sensing – Buying urge / group behavior
Consumer Behavior
Wearable sensing, nearable sensing and crowd sensing
9
Sensing Physical Context of People – Location and Activity
Indoor Localization – Bldg, Mall• Entry-Exit using RFID and
Magnetometer• Zoning using Wi-Fi• Fine-grained positioning using Inertial
Navigation
Activity Detection - Wellness• Walking / Brisk Walking / Jogging /
Running using Accelerometer Signature
• Orientation and Placement agnostic• Calorie Burnt using Activity based
models
Magnetometer – Entry/Exit
RFID Fusion
WiFi -Zoning Bluetooth -Proximity
98% 99.7% 97% 96%
(Accuracy ~2m)
(Accuracy ~ 98%)
Publicationso Nasimuddim Ahmed et. al., ""SmartEvacTrak: A People Counting and Coarse-Level Localization Solution for Efficient
Evacuation of Large Buildings“, CASPER'15 workshop of IEEE Percom 2013o Vivek Chandel et.al., "AcTrak - Unobtrusive Activity Detection and Step Counting using Smartphones“, Mobiquitous 2013
10
Sensing Physical Context of People – Traffic and Driving
Traffic Sensing – City Authority• Congestion Modeling from historical
location data crowd sensed from vehicles
• Honk Detection from crowd sensed audio data
• Road Condition Monitoring from crowd sensed Accelerometer data
Driving Behavior - Insurance• Hard Cornering / Breaking / Harsh
Acceleration from Accelerometer Analytics
Publicationso Maiti, Santa, et al. "Historical data based real time prediction of vehicle arrival time." ITSC 2014o Ghose, Avik et. al., "Road condition monitoring and alert application: Using in-vehicle smartphone as internet-connected
sensor.“, Percom Workshops 2012.o Tapas Chakravarthy et. al., “MobiDriveScore — A system for mobile sensor based driving analysis: A risk assessment
model for improving one's driving”, ICST 2013
11
Physiological Sensing – Mobile Phone and Wearable
12
Behavioral Sensing using Physiology
Referenceso Levenson, Robert W. "Blood, sweat, and fears." Annals of the New York Academy of Sciences 1000, no. 1 (2003):o Näätänen, R et.al., "A model for the role of motivational factors in drivers' decision-making." Accident Analysis &
Prevention 6, no. 3 (1974)o GW Evans, “Environmental stress”, 1984o Bechara, Antoine et. al., "Emotion, decision making and the orbitofrontal cortex." Cerebral cortex 10, no. 3 (2000):o Mauss, Iris B et, al., "The tie that binds? Coherence among emotion experience, behavior, and physiology." Emotion 5, no.
2 (2005): 175.
• Heart Rate Variability
• Blood Pressure
• EEG• GSR
13
Physiological Sensing – Heart Rate, BP and HRV
PPG Signal
Field Trials at TCS Office and Indian VillagesTie-up with HospitalsWearable variant pilot for Crane Operator Monitoring in Factories
14
rMSSD, DSD, SDNN, nn50, PNN50, nn20, pNN20
Physiological Sensing – Results
~2 bpm error in
Heart
Rate~92% Accuracy in blood
pressure
Publicationso Arpan Pal et. al., "A Robust Heart Rate Detection using Smart-phone Video", in MobileHealth workshop of Mobihoc
2013o Aishwarya Visvanathan et. al., "Smart Phone Based Blood Pressure Indicator", in MobileHealth workshop of Mobihoc
2014.o Anirban Duttachoudhury et.al., "Demo – Estimating Blood Pressure and ECG from Photoplethysmograph
using Smart Phones", SenSys 2014 – BEST DEMOo Banerjee, Rohan et al. "Noise Cleaning and Gaussian Modeling of Smart Phone Photoplethysmogram to
improve Blood Pressure Estimation“, ICASSP 2015o Nasim Ahmed et al. “Feasibility Analysis for Estimation of Blood Pressure and Heart Rate using A Smart
Eye Wear”, WearSys workshop in Mobisys 2015
~89% Accuracy in HRV -
SDNN
15
Psycho-Physiological Sensing – EEG and GSR
GSR
Mental tasksCognitive LoadVisual Attention(VA)Memory(M)Logic(L)Arithmetic(A)Other(O)
Emotion(E) Stress(S)
EE
G
arte
fact
re
mov
al
AP
I (V
A),
AP
I(M
),…
AP
I(S
)
Application
Fus
ion
Fea
ture
E
xtra
ctio
n (
ind
ivid
ual t
ask
)
16
Psycho-Physiological Sensing – Results
~80% Accuracy, showing
hierarchical keyboard is easy to
use than QWERTY
~75% Accuracy
o “Evaluation of Different onscreen keyboard layouts using EEG signals”, SMC 2013o “EEG-Based Fuzzy Cognitive Load Classification”, FUZZ IEEE 2013o “Unsupervised Approach for Measurement of Cognitive Load using EEG Signals”, BIBE
2013
17
Physiological Sensing for Behavior Modeling
18
Behavioral Modeling using Physiology
Pietro Cipresso et. al., “Psychometric modeling of the pervasive use of Facebook through psychophysiological measures: Stress or optimal experience?”, Computers in Human Behavior , 49 (2015) 576–587, Elsevier
19
Behavioral Modeling using Physiology – Early Results
Tetris-like game designed for Bored and Flow State Stimulio Submitted: “Dynamic Assessment of Learners' Mental State for an
Improved Learning Experience “, Frontiers of Education 2015
Should be extendable to other use cases
20
Looking Ahead - Challenges
Need to take care of Battery Power Issue
Need to address Privacy Issue
Each sensor may be very accurate on its own – fusion is the key
Right feature selection for the given use case would be critical
Lack of multi-sensor Dataset needs to be addressd
Option• Do controlled experiments on diverse set of sample subjects using physiological
sensing and create simplified aggregate models • Use the Model in the field (e.g. - % of people who do not follow the evacuation
recommendation can help in creating a probabilistic model)• Would need Individual training or constant wearing of sensors for individual
models – Driving / Shopping Behavior
TCUP – the TCS IoT
platform can be used to
collect multi-sensor data in an
efficient way
21
More References
o Karel A. Brookhuis, Dick de Waard, Monitoring drivers’ mental workload in driving simulators using physiological measures, Accident Analysis & Prevention, Volume 42, Issue 3, May 2010, Pages 898-903, ISSN 0001-4575, http://dx.doi.org/10.1016/j.aap.2009.06.001.
o J.A. Healey and R.W. Picard, "Detecting Stress during Real-world Driving Task using Physiological Sensors", Intelligent Transportation System, IEEE Trans, , Vol. 6, No. 2, June (2005) 156-166.
o Jordan Smith, Neil Mansfield, Diane Gyi, Mark Pagett, Bob Bateman, Driving performance and driver discomfort in an elevated and standard driving position during a driving simulation, Applied Ergonomics, Volume 49, July 2015, Pages 25-33, ISSN 0003-6870, http://dx.doi.org/10.1016/j.apergo.2015.01.003.
o Gianluca Borghini, Laura Astolfi, Giovanni Vecchiato, Donatella Mattia, Fabio Babiloni, Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness, Neuroscience & Biobehavioral Reviews, Volume 44, July 2014, Pages 58-75, ISSN 0149-7634, http://dx.doi.org/10.1016/j.neubiorev.2012.10.003.
o David P. Wyon, Inger Wyon, Fredrik Norin, Effects of moderate heat stress on driver vigilance in a moving vehicle, Ergonomics, Vol. 39, Iss. 1, 1996.
o Markku Kilpeläinen, Heikki Summala, Effects of weather and weather forecasts on driver behaviour, Transportation Research Part F: Traffic
o Psychology and Behaviour, Volume 10, Issue 4, July 2007, Pages 288-299, ISSN 1369-8478, http://dx.doi.org/10.1016/j.trf.2006.11.002.
o Mauss, Iris B., Robert W. Levenson, Loren McCarter, Frank H. Wilhelm, and James J. Gross. "The tie that binds? Coherence among emotion experience, behavior, and physiology." Emotion 5, no. 2 (2005): 175.
o Levenson, Robert W. "Blood, sweat, and fears." Annals of the New York Academy of Sciences 1000, no. 1 (2003): 348-366.o Bechara, Antoine, Hanna Damasio, and Antonio R. Damasio. "Emotion, decision making and the orbitofrontal cortex."
Cerebral cortex 10, no. 3 (2000): 295-307.
Thank You
IT ServicesBusiness SolutionsConsulting
With Avik Ghose, Aniruddha Sinha, Tanushyam Chattopadhyay, Arindam Pal, Debatri Chatterjee