Designing Physical/Digital Objects for The Internet of Things: From Sensors to User Interaction

DESIGNING PHYSICAL/DIGITAL OBJECTS FOR THE INTERNET OF THINGSFROM SENSORS TO USER INTERACTION

Gerd Kortuem

17 June 2009 Infolab21 | Lancaster [email protected] | @kortuem | www.kortuem.com

mailto:[email protected]


http://www.kortuem.com


Gerd Kortuem | [email protected] | @kortuem | www.kortuem.com | Infolab21 | Lancaster University

SOME TWITTER USERS ARE NOT HUMAN





















Andy Stanford Clark (IBM)






Source: Cartoon by Peter Steiner. The New Yorker, July 5, 1993 issue (Vol.69 (LXIX) no. 20) page 61






ON THE INTERNET, NOBODY KNOWS YOU ARE A

THING

Source: http://www.botanicalls.com/kits/





http://www.botanicalls.com/kits/

http://www.botanicalls.com/kits/


• ATmega1280, 5V• Digital I/O Pins 54 (of which 14 provide PWM output)• Analog Input Pins 16• DC Current per I/O Pin 40 mA• DC Current for 3.3V Pin 50 mA• Flash Memory 128 KB of which 4 KB used by bootloader• SRAM 8 KB• EEPROM 4 KB• Clock Speed 16 MH

Arduino is an open source microcontroller board, plus a free software development environment, to make objects that can sense inputs from switches, sensors, and computers, and control motors, lights, and other physical outputs in the real world!

ARDUINO






Source: Clive Thomas / Wired http://www.wired.com/techbiz/startups/magazine/16-11/ff_openmanufacturing

OPEN-SOURCE HARDWARE





http://www.wired.com/techbiz/startups/magazine/16-11/ff_openmanufacturing



Source: Clive Thomas / Wired http://www.wired.com/techbiz/startups/magazine/16-11/ff_openmanufacturing

OPEN-SOURCE HARDWARE








Blogjects track and trace where they are and where they’ve been

Blogjects have self-contained (embedded) histories of their encounters and experiences

Blogjects always have some form of agency — they can foment action and participate; they have an assertive voice within the social web.

BLOGJECTSA Manifesto for Networked Objects — Cohabiting with Pigeons, Arphids and Aibos in the Internet of ThingsBY JULIAN BLEEKER






THINGS AND PEOPLE

PEOPLE AND THINGS

An Internet of Things that is more than just sensors attached to things






1. CASE STUDIES2. HUMAN DIMENSION






BP Chemical storage facility, Saltend, Northeast England






Road maintenance crew, South England

trial users anonymized






Breaker Circular Saw & Wacker Plate






Vibration White Finger






Vibration Dosimeter






Duration of use (measured)

Vibration characteristics (known for each equipment, measured by manufacturer)

Soil condition (assumed)

operating heavy vibrating machinery such as hydraulic drillsand breakers. Our experimental system comprises a collec-tion of tools, augmented with wireless sensor nodes, personaluser devices and mobile computers that collaborate in anad-hoc manner in order to collect HAV exposure informa-tion. This information is available in real-time to workersin the field and subsequently to management via a back-enddatabase. Our design allows health and safety regulationsspecified at the enterprise level to be embedded within thewireless sensor nodes, allowing them to operate without anyinfrastructural support. This is crucial for workers who oftenhave to operate in areas with limited or no wireless commu-nication infrastructure. The system was evaluated througha two week field trial that took place with the collaborationof a major UK construction and maintenance company.

2. DOMAIN ANALYSIS

2.1 The Hand Arm Vibration ProblemLong-term exposure to hand arm vibration can lead to

serious health conditions such as “vibration white finger”(VWF) and in extreme cases to life-long disability. VWF istriggered by excessive use of vibrating machinery such as hy-draulic drills and breakers, and causes the fingers to becomenumb and begin turning white. As the disease progresses itcan become irreversible; the person su!ers increasingly fre-quent painful attacks at any time and may even lose theirfingers (typically this only happens in extreme cases, for ex-ample, when people are working with vibrating machineryin very cold conditions, as is the case in the forestry industryamong people working with chainsaws).

Extensive health and safety regulations exist to limit work-ers’ exposure to HAV. For example, in 2005 the UK Parlia-ment introduced the Control of Vibration at Work Regula-tions [10]. These regulations set limits for daily exposureto vibration, define methods for their calculation and for-mulate guidelines for monitoring an employee’s exposure tovibration. The guidelines place a responsibility on employ-ers to assess each individual employee’s exposure risk and toconsider the specific working conditions of each employee.When it is not possible to eliminate a worker’s exposure tovibration then suitable health surveillance must be put inplace and immediate action be taken if specific vibrationlimits are exceeded.

The damage caused by exposure to vibrations is a combi-nation of both the frequency of the vibrating tool and theduration of the exposure. Using a tool that vibrates at alow frequency for a long time can be as damaging as usinga heavily vibrating tool for a short time. Thus regulationsspecify two figures for exposure duration, a limit to the over-all daily trigger time (Daily exposure limit), and a limit onshort-term exposure to very high levels of vibration (Dailyexposure action). These limits are defined in terms of theaverage daily exposure dose A(8):

• Daily exposure limit value = A(8) of 2.5m/s2

• Daily exposure action value = A(8) of 5.0m/s2

A(8) is defined as ahv

p

T/8 where

ahv = actual vibration acceleration rate expressed inm/s2

T = actual exposure duration expressed in hours

In turn, ahv is composed of the root–mean–square accel-eration magnitudes in three orthogonal directions, x, y andz, at the vibrating surface in contact with the hand. Wheredaily vibration exposure A(8) is above 2.5m/s2 but below5.0m/s2, the following steps should be taken: Inform work-ers of risk, carry out regular health surveillance of worker,record assessment. Where daily vibration exposure exceeds5.0m/s2, the following actions must be taken: Limit usageimmediately, rotate workers, introduce other working meth-ods to reduce HAV. Since in practice ahv is not known fora specific piece of work, equipment manufacturers publishestimated ahv values for each individual tool, which can beused to estimate an operator’s exposure.

There is a distinct lack of automated solutions for assistingwith HAV H&S rule compliance. For example, vibration ex-posure data is typically manually recorded by operatives onpaper sheets, which are then entered by hand into a healthand safety information system. Moreover, key data such astrigger time must be estimated by operatives, most oftenhours after work has been completed. This of course raisesserious concerns with respect to completeness, accuracy andconsistency of captured data. The current practice can beimproved using mobile data entry solutions based on hand-held wireless computers. Yet, while mobile solutions reducethe need for paper forms they still su!er from the fact thatthey rely on human information gathering and recordingin the field. In addition, current solutions are tailored foro!-line processing of data in the back o"ce, ignoring thepotential benefits for real-time information in the field.

2.2 Requirements CaptureIn close collaboration with a major UK company that car-

ries out road maintenance and construction operations, weset out to design, implement and evaluate a Hand Arm Vi-bration (HAV) monitoring system, based on our approach ofaugmenting work artefacts. Our team consisted of computerscientists, psychologists, ethnographers and organisationalmanagement experts. Together, we engaged with the com-pany on various levels, ranging from high-level managementto workers in the filed, to investigate current work practices,identify technical and usability requirements, and anticipatedeployment challenges. The following gives an outline of ourapproach and introduces the requirements that we gathered.

2.2.1 MethodologyTo inform the design and elicit requirements we followed

a two-pronged approach. On the one hand, we undertookextensive work place studies and interviews to understandcurrent work practices. On the other hand, we generateddesign sketches to elicit concrete feedback.

In total we conducted over twenty formal interviews withmanagers and employees and 16 full days of work obser-vation. Field observation days also involved informal dis-cussions with upwards of 25 operatives going about theirdaily work, with the fieldwork being conducted across threedi!erent regional sections of the company, each of whichwith its own safety culture and traditions. An importantresult with respect to risks from vibration and noise expo-sure (“invisible risks”) was that it emerged that these wereappropriately assessed and their mitigation documented atan organisational level, yet were often underestimated orviewed as insignificant by operatives due to the apparentlyloose linkage between the risk and its outcomes. For exam-

Actual Exposure

Maximum Legal Exposure

?

Hand-arm vibration syndrome (HAVS)

• Neurological component

– numbness, tingling, reduced sense of

touch & temperature

– main cause of disability in advanced

cases

• Vascular component

– “vibration white finger”

– episodes of finger blanching

– typically cold-induced attacks

• Musculoskeletal component

– pain, weak grip, etc.






Static domain model+

Dynamic world model

Operator

IN-SITU ASSISTANCE

encode rules query

REAL-TIME ASSESSMENT

inform

CODIFICATION

Health & Safety Policies






PERSONAL HEALTH HISTORIES

0

37.5

75.0

112.5

150.0

13:50 14:00 14:10 14:20 14:30 14:40 14:50 15:00 15:20 15:30 14:20

Dai

ly E

xpos

ure

Act

ion

Valu

e

Time

0

10

20

30

40

10:50 11:00 11:10 11:20 11:30 11:40 11:50 12:00 12:20 12:30 12:40

Dai

ly E

xpos

ure

Lim

it Va

lue

Time

How dangerous is work for each individual?

Daily limit Intensity limit






The recorded usage time had an average error of 3.2% against the actual usage as recorded by the video footage.






SITE-SPECIFIC HEALTH LOGSWhere are safety hotspots?

Is health managed differently across work-sites?






ACTIVITY PATTERNSWhen do safety incidents occur?

When is a task finished?

1 5 9 13 17 21

Tool 1Tool 2MotionSoundLightTask

9:00 9:30 10:00 10:30 11:00 11:30






Mod

el

Awarness

Interactivity

Activity-awa

re

Process-awar

ePolicy

-aware

Functions

Rules

Work-Flows

POLICY-AWARE OBJECTS






Mod

el

Awarness

Interactivity

Activity-aware

Process-awar

ePolicy-

aware

Functions

Rules

Work-Flows

ACTIVITY-AWARE OBJECTS






RENTAL COMPANY CONSTRUCTION COMPANY

UseCost calculation

Recording usage

LeaseCost = (T "U) + Cost(m)ii=1

|M |

#






USAGE-BASED PRICINGRental equipment is charged per actual use, not per time

0

1

2

3

4

5

6

7

0 1 2 3 4 5 6 7

Tool

Worker

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Tool 3Tool 2Tool 1

d1

When, how, how often? By whom?

d1 d2 d3






THE SAFETY-AWARE BARREL

System software

Sensors

Display

Application software

Memory

Processor

Wireless Network

Storage






THE SAFETY-AWARE BARREL

Don’t shake me so much!I am too hot!

I shouldn’t be here!

There are too many of us!

I am standing too close to my neighbours

Don’t put me next to these barrels!






COOPERATIVE REASONING

Chemical A alone is ok Chemical A next to chemical B is not ok

Reactive chemicals may not be stored next to each other

m1 + m2 < safety limit

Total mass of a chemical must be below critical threshold

m1 + m2 + m3 > safety limit

©TECO Karlsruhe University












Chemical knowledge













Chemical knowledge

Spatial awareness







DEVICE ARCHITECTUREDomain Knowledge

reactive(<chemical>,<chemical>)critical_mass(<chemical>,<number>)content(me,<chemical>)mass(me,<number>

Safety Ruleshazard:- content(me, CH1), content(C, CH2), reactive(CH1, CH2), min_dist(CH1, CH2, D1), distance(me, C, D2), D2 > D1.

Observational Knowledgedistance(<container>,<container>,<dist>)

Smart-Its Sensor Node

PIC18F6720 @ 20 MHz (5 MIPS)Program Memory: 128KB

Data Memory: 3.8KB RAM1KB internal EEPROM

Memory: 32 KB EEPROM






Trial at BP’s chemical plant in Hull, UK (July 2006)Smart-its sensor node:

PIC18F6720 @ 20 MHz (5 MIPS)Program Memory: 128KB

Data Memory: 3.8KB RAM







1. CASE STUDIES2. HUMAN DIMENSION






DESIGN PROCESS

Google Maps http://maps.google.co.uk/maps/ms?f=q&hl=en&geocode=&ie=UTF8&om=1&msa=0&msid...

2 of 3 18/9/07 17:44







DESIGN PROCESS

Google Maps http://maps.google.co.uk/maps/ms?f=q&hl=en&geocode=&ie=UTF8&om=1&msa=0&msid...

2 of 3 18/9/07 17:44







INVISIBLE RISKSRisk perception is poor

Goal: increasing awareness






SELF-IMAGE AND SOCIAL PRESSURE Safety equipment often not used

Personal and social “invisibility” becomes key design goal






PAST EXPERIENCES & NARRATIVESGPS had been experienced as surveillance technology

Technology narrative must be part of the design

= or or ?






INFORMATION DISCLOSURE

Sensing-only Awareness display

Unit w/o display was perceived as surveillance technology






TECHNOLOGY ARCHETYPES

Sensor network(sensing + data)

Smart object(interaction)

Developers view problem in light of well-known system models






TECHNOLOGY ARCHETYPES

Management

Operatives' behavior

Data logs

Exception reports

Health & Safety Rules

Operatives

Feedback

Automatic data capture

Enterprise

System Personal Health & Safety Record

Operative'sbehavior

Context-sensitive notices

Sensor-network inspired H&S system

(top-down)

Smart-object inspired H&S system

(bottom-up)

Different system models imply different H&S solutions






BEHAVIOR-BASED SAFETY

Personal Health & Safety Record

Operative'sbehavior


Personal Health & Safety Record

Operative'sbehavior


Operatives as source of expertise and behavioral change

Collaborative safety






POINTS TO TAKE HOME

Internet of Things = Things + People






THANK YOUGerd Kortuem

@[email protected]

www.comp.lancs.ac.uk/~kortuem







http://www.comp.lancs.ac.uk/nemo

http://www.comp.lancs.ac.uk/nemo


PressSmart tags hail the web of things. BBC. http://news.bbc.co.uk/2/hi/technology/7207514.stmScientificDaniel Fitton, Fahim Kawsar and Gerd Kortuem. Exploring The Design of a Memory Model for Smart Objects. First International Workshop on Digital Object Memories. To appear.Daniel Fitton, Vasughi Sundramoorthy, Gerd Kortuem, James Brown, Christos Efstratiou, Joe Finney, Nigel Davies. Exploring the Design of Pay-Per-Use Objects in the Construction Domain. EuroSSC 2008: 192-205Kortuem et al. Sensor Networks or Smart Artifacts? An Exploration of Organizational Issues of an Industrial Health and Safety Monitoring System. Proceedings International Conference on Ubiquitous Computing (Ubicomp 2007), October 2007, Innsbruck, Austria Brown et al. Network Interrupts: Supporting Delay Sensitive Applications in Low Power Wireless Control Networks. Proceedings ACM MobiCom workshop on Challenged Networks (CHANTS 2007), September 2007, Montreal, CanadaEfstratiou et al. Experiences of Designing and Deploying Intelligent Sensor Nodes to Monitor Hand-Arm Vibrations in the Field. Proceedings of MobiSys 2007, San Juan, Puerto Rico Iszatt White, M. Catching Them At It? An Ethnography of Rule Violation. Symposium on Current Developments in Ethnographic Research in the Social and Management Sciences, 13th-14th September 2006, Liverpool, UK Busby, J. and Iszatt-White, M. Pushing the Boundaries of HRO Thinking: Non-complex and Uncoupled but still Deadly. SRA Annual Meeting - Risk Analysis in a Dynamic World: Making a Difference, 3rd-4th December 2006, Baltimore, Maryland Davies et al. Sensing Danger – Challenges in Supporting Compliance in the Field. 8th IEEE Workshop on Mobile Computing Systems and Applications (HotMobile 2007), February 2007, Tucson, Arizona Lowton M. and Finney, J. Finding NEMO: On the Accuracy of Inferring Location in IEEE 802.15.4 Networks. Proceedings Workshop on Real-World Wireless Sensor Networks (RealWSN 2006), June 2006, Uppsala, Sweden

PUBLICATIONS





http://news.bbc.co.uk/2/hi/technology/7207514.stm

http://news.bbc.co.uk/2/hi/technology/7207514.stm

http://www.informatik.uni-trier.de/~ley/db/indices/a-tree/f/Fitton:Dan.html

http://www.informatik.uni-trier.de/~ley/db/indices/a-tree/f/Fitton:Dan.html

http://www.informatik.uni-trier.de/~ley/db/indices/a-tree/s/Sundramoorthy:Vasughi.html

http://www.informatik.uni-trier.de/~ley/db/indices/a-tree/s/Sundramoorthy:Vasughi.html

http://www.informatik.uni-trier.de/~ley/db/indices/a-tree/b/Brown:James.html

http://www.informatik.uni-trier.de/~ley/db/indices/a-tree/b/Brown:James.html

http://www.informatik.uni-trier.de/~ley/db/indices/a-tree/e/Efstratiou:Christos.html

http://www.informatik.uni-trier.de/~ley/db/indices/a-tree/e/Efstratiou:Christos.html

http://www.informatik.uni-trier.de/~ley/db/indices/a-tree/f/Finney:Joe.html

http://www.informatik.uni-trier.de/~ley/db/indices/a-tree/f/Finney:Joe.html

http://www.informatik.uni-trier.de/~ley/db/indices/a-tree/d/Davies:Nigel.html

http://www.informatik.uni-trier.de/~ley/db/indices/a-tree/d/Davies:Nigel.html

http://www.informatik.uni-trier.de/~ley/db/conf/eurossc/eurossc2008.html#FittonSKBEFD08

http://www.informatik.uni-trier.de/~ley/db/conf/eurossc/eurossc2008.html#FittonSKBEFD08