Prof. Joe ParadisoResponsive Environments Group, MIT Media Lab

http://www.media.mit.edu/resenv BSN - 6/08

On-Body Wireless Sensing for Human-Computer Interfaces

At a fascinating intersection

Sensor Networks

Wearable ComputingImplantables

Ubiquitous Computing

3

JAP7/05

Topics

• Wearable Sensors– Sports Medicine & Entertainment

• Low Power Techniques– Quasi-Passive & Groggy Wakeup

• Applications in Health and Supply Chain Monitoring– Passive RFID Tags for Precise Localization

• X Reality– Sensor networks connect real and virtual spaces– Virtual worlds for browsing reality

• Sensor-Enhanced Media Creation SensorNetworks for Social Computing

4

JAP7/05

Sensing as Commodity

• Sensors are now becoming a commodity, and soon caneasily be designed into most any device.

– Rather than omitting them from a cost/complexity viewpoint, itbegins to make more sense to just include them if there’s anysuspicion that they could be needed.

– This causes a shift in how sensors are used – rather than rely ononly 1 or 2 sensors made a priori to measure particular quantities,many sensors will be used that don’t necessarily exactly measurethe quantity of interest (especially as applications will becomemore general and evolve over time).

1997 - Expressive Footwear17 Data Channels

Tilt, shock, rotation, height, bend,location, multipoint pressure

The Wearable Gait Laboratory

Stacy Morris

7

JAP7/05

Measures 18 Parameters per foot

2

Left Foot Right Foot

PVDF Toe

FSR Metatarsal, Lateral

FSR Metatarsal, MedialFSR Heel, Medial

FSR Heel, Lateral

Bend - Ankle(out of the plane)

Layout of Insole Sensors

Bend - Insole

Capacitive(height)

PVDF Heel

Axes for Gyroscopesand Accelerometers

x

y

z

8

JAP7/05

“Shuffle Index” vs. “Stride Excitement” via CART

0.012 0.014 0.016 0.018 0.02 0.022 0.024 0.026 0.028 0.03-40

-35

-30

-25

-20

-15

Variation in Insole Force [normalized by bodyweight]

Minimum pitch [degrees]

NL F Y

NL F

PD F

Min

imum

Foo

t Inc

linat

ion

(deg

)

9

JAP7/05

Real-Time Auditory Feedback

• Parametric Personal Music System– Always play ambient music

• Provide meaningful (and correcting) musical feedbackwhen problematic gait changes arise.

The ParkinsonianAntiShuffle

The PronationPreventer

The PressureAvoider

Erik Asmussen

Scaling to several dancers…

High Speed Sensor FusionVocabulary of features

Capacitive proximity to 50 cm6-axis IMU - 1 Mbps TDMA radio

100 Hz Full State Updates from 25 nodes

Capacitive Sensing System

• Capacitive system measuresrelative spacing between nodes

• Employs a “transmit mode”configuration

• One node transmits a sinusoidalpulse at 90kHz, others measureamplitude of received pulse

• Nodes trade roles as transmitters and receivers as arbitrated by thewireless basestation

• Sensitive up to 0.5 meter with bracelet-sized electrode and sharedground through the body

Typical Capacitive Sensor Response With Strap Electrode and Body Grounded

0

500

1000

1500

2000

2500

3000

3500

4000

4500

0 10 20 30 40 50 60

Spacing (cm)

Raw

Sen

so

r V

alu

es

New Developments

• Full deployment at a rehearsal with 20 sensor nodes running onfive dancers

• Feature extraction and mapping algorithms which are fedlogged data at the sample rate to simulate real time operation

Rough Demo Mapping

DirectFeatures

IntuitiveFeatures

MusicalParameters

• Audio rendered in real-time asfeatures are calculated

• Runs on a 1.6 GHz G5 w. 1GB RAM

Network In Action

SportSemble

Red Sox Spring Training

Modified sensors for high range and high sample rates

- 3-axis 10,000 °/sec gyro- 3-axis 120 G accelerometer- 3-axis 10 G accelerometer- 3-axis magnetometer- 1 kHz synchronized sampling per node

- 12 seconds recorded on flash at each node

Biomotion measurement of a Red Sox Pitcher

Preliminary Results - Red Sox Spring Training

• Acceleration at the wristpeaks well above 100g

• Most of this accelerationoccurs in a 30ms window

• Equates to 30 samples forthe modified inertial system,but only 5 frames on a180Hz video capture system

Acceleration of the Pitch Above Captured at Three Critical Locations - Hand, Wrist, & Biceps

>100 g’s!!

New collaboration, Red Sox Grant - Spring Training in February

Players Cluster

19

JAP7/05

Miniaturizating & Distribution

• Future of athletic broadcasts– Get content directly from point of expression

• Sensor packs in figure skates, boxing glove, snowboard…– Map dynamic content (music, graphics, specs…)

• Future of exercise– Monitor lower, upper limbs, heart rate, etc.– Map interactive content

• Synchs up and nudges participant to stay on track• Therapy with interactive feedback

Interactive Music on N800

• Zigbee interface for N800– Easy implementation of wearable sensors (inertial, etc.)

• Pd (PureData) compiler for N800– Allows artists to graphically compose music interaction

& synthesis– Produces C code (not interpreted)– Fast, efficient execution

Robert Jacobs’ M.Eng. Thesis - demos coming

Groggy Wakeup for Efficient Smart Sensor Systems

Processed

DataS

tate

Enable

Sensor 1

Sensor 2

Sensor N

Microcontroller

State Analysis

Data

Sensor Controller

Algorithm Selector

Data Compression Storage

Transceiver

Ari Y. Benbasat

Power-efficient detection of gait phaseGeneral on-node implementation

Accel1

Gyro1Gyro2

Accel2Accel3

Gyro3

TiltAccel1Tilt

Accel1Tilt

Accel2Accel1Tilt

Accel3Accel2Accel1Tilt

Accel1Tilt

DETECT

Gyro1 Gyro1Gyro2

Accel3Accel2

Power

Time

Log and/orProcess

Automated Power-Efficient Sensor HierarchyKeep higher-power sensors OFF unless needed for detection decision

Groggy Wakeup for Efficient Smart Sensor Systems Ari Y. Benbasat

Power-efficient detection of walking up stairs

Automated Power-Efficient Sensor HierarchyTrade Power Consumption with Detection Efficiency

“A Framework for the Automated Generation of Power-Efficient Classifiers for Embedded Sensor Nodes,” Ari Y. Benbasat and Joseph A. Paradiso, Proc. of SENSYS 2007

Groggy Wakeup for Efficient Smart Sensor Systems Ari Y. Benbasat

Power-efficient detection of gait phase

Automated Power-Efficient Sensor HierarchyTrade Power Consumption with Detection Efficiency

24

JAP7/05

Tree Execution Running on the Stack in Real Time

2-level gait classifier tree during actual walking

The Disposable Wireless Sensors

• Very simple “featherweight” motion sensor– Cantilevered PVDF piezo strip with proof mass– Activates CMOS dual monostable when jerked– Sends brief (50 µs) pulse of 300 MHz RF– 100 ms dead timer prevents multipulsing– Can zone to within ~10 meters via amplitude– Ultra low power – battery lasts up to shelf life– Extremely cheap – e.g., under $1.00 in large

quantity

CargoNet: A Low-Cost MicroPower Sensor Node Exploiting Quasi-Passive

Wakeup for Adaptive Asynchronous Monitoring of Exceptional EventsMalinowski, Moskwa, Feldmeier, Laibowitz, and Paradiso - Presented at ACM Sensys 2007

• Dynamically adaptable thresholds- Adapts to environments with persistent stimuli

• Small and inexpensive• Microampere current draw

- 5 years on a single coin cell battery

Power Harvesting ShoesPVDF Stave

Molded into soleEnergy from bend

Ppeak ≅ 10 mW

≅ 1 mW

Flex PZT UnimorphUnder insole

Pressed by heelPpeak ≅ 50 mW

≅ 10 mW

Responsive Environments GroupMIT Media Lab

1998 IEEE Wearable Computing Conference

Raw Powercirca 1% efficient

unnoticable

Mobility in Sensor Networks• Forefront research where sensor nets meet robotics

and control• Sensor clusters move to places to optimally:

– Measure dynamic phenomena– Position relays to repair or patch broken network– Dump information at access points (portals)– Get recovered or recharged

What does power harvestingmean in a mobile system?

Energy cost of moving atoms is muchhigher than moving bits…

AES Roadmap“

Parasitic Mobility in Sensor NetworksImplications- Sensor clusters hitch rides to placeswhere they need to be to optimally:

- Measure relevant phenomena- Relay information peer-peer- Dump information into portals- Get recovered or recharged

Paradiso & Laibowitz

Innovations and Architecture- Interpretation of Energy Harvesting in mobile networks- Two flavors:

- The Tick (e.g., jumps onto a car, attaches magnetically, then disengages)- The Bur (e.g., sticks to passing object, then shakes off)

- Contains GPS, RF, basic sensor suite

- Rapid diffusion of sensors across an environment

- System self-organizes to auto-dispatch nodes to desired regions

Phoresis

31

JAP7/05

Parasitic Mobility Research (ParaMoR)

• Paramor Hardware – smallnodes with sensor suite (light,microphone, inertial,proximity, temperature, heat),GPS, RF communication,rechargeable power source,and minimal actuation forattachment/detachment

• Active nodes (ticks)• Passive nodes (burs)• Value-added nodes (pens)• ParaSim – Software

simulator to study behaviorand evaluate controlalgorithms for parasiticallymobile sensor nodes

Mat Laibowitz

Active Node

Passive Node

Utags : Passive Real-Time 3-d Localization & Remote Sensing

• Utilizing passive RFID surface acoustic wave (SAW) and low-cost radar technology.

• Target short-range (3-100m) applications– Expect single-measurement resolution of under 10 cm

• Multipath from reader dies out before tag responds• 900 MHZ devices coming out of MIT MTL and nano labs

– Now being characterized and tested

Precise, ultra low cost wide-area tracking of small passive tags for indoor localizationWill revolutionize asset tracking and supply chain operation, search & rescue, etc.

Jason LaPenta

Google for Reality

34

JAP7/05

Public Misinformation…

“Human” Energy harvesting will do little for sustainabilityIt will only be valuable in extending/eliminating batteries in

portable devices, wearable sensors, etc.

35

JAP7/05

Sensor networks for energy conservation

Humidity

Temp.

Comfort

• Leveraging dense sensor networks for optimal urban energymanagement– 40% of US energy is spent in buildings– Pervasive sensor/actuator network can work to minimize this

• Optimize heating, AC, lighting for Person not room• Anticipating behavior & build usage models over time

– My RA, Mark Feldmeier is now a MIT Martin Energy Fellow• Exploring ways to sense “comfort” to optimize distributed utility control

Marshall McLuhan, 1911-1980 "After three thousand years of explosion, by means of fragmentaryand mechanical technologies, the Western world is imploding. Duringthe mechanical ages we had extended our bodies in space. Today,after more than a century of electric technology, we have extended ourcentral nervous system itself in a global embrace, abolishing bothspace and time as far as our planet is concerned. Rapidly, weapproach the final phase of the extensions of man - the technologicalsimulation of consciousness, when the creative process of knowing willbe collectively and corporately extended to the whole of humansociety, much as we have already extended our senses and ournerves by the various media."

Marshall McLuhan - Understanding Media (1964)

Electronic media (a.k.a.television) as an extension ofthe central nervous system

Sensor Networks as Extension of theNervous System

Cast our awareness across space, time, scale, modality…

38

JAP7/05

Bootstrapping a Ubiquitous Sensor Infrastructure

• Sensor networks are the foot soldiers at the frontlines of ubiquitous computing

• At this point, few if any customers will buy anensemble of “UbiComp” sensors

• They will aggregate from established devices– Home security, appliances, utility devices, entertainment…

Just as the web sprouted from a networked ensemble ofpersonal computers, true “ubicomp” will arise from an

armada of networked devices installed for other purposes.

39

JAP7/05

Power Strips are Everywhere

• Needed in Homes, offices, especially the Media Lab!• Sensors are becoming commodity items

– Cost of adding sensors to a design is becoming incremental• Power strips are ideal base platforms for hosting a sensor

network– Ready access to power– Power line can be a network port– Can monitor the status of devices that are plugged in

40

JAP7/05PlugPoint – Power Strips as the backbone of a UbiComp Sensor InfrastructureJ. Lifton, M. Feldmeier, Y. Ono (Ricoh)

Power Line providesenergy & comm

Monitor current profiles,Switch individual socketsHosts basic sensors (mic,

light, motion)Expansion Port for others

Hub for wireless sensornetwork

Collaboration with Ricoh Research

Army of Plugs

35 ON MEDIA LAB THIRD FLOOR

Darkerimpliesmore

sound &movement

MSR - 4/08

Sensor Networks for Linking Virtual and Real Worlds

X-Reality

45

JAP7/05

46

JAP7/05

47

JAP7/05

Real World can be “browsed” in virtual space

Second Life as a the Virtual End of Dual Reality

Shadow Lab - Binding real sensor data to virtual worlds

Josh Lifton - Building a virtual Media Lab in SecondLife

Simple sensor apparitions to explore basic ideas- Energy use ➔ height of fire

- Activity (sound, motion) ➔ whirlwinds- Active regions ➔ higher walls

- “Ghost” face ➔ individual presence

Third floor of ML built inSecond Life

ResEnv Lab rendered indetail - other areas currently

derived from map

Sensor data piped in andinterpreted as real-time

graphic phenomena

Lifton 07

50

JAP7/05

Mobile User Interfaces for Sensor Networks

Browsing, querying,navigating through

sensor net dataWhat are the interfaceaffordances, displays,

interactionmodalities?

Privacy & Security?Mobile platform

inside of network vsfixed platform

outside?McLuhan Extension

of human senses

51

JAP7/05

Nokia 770 Sensor Net Tricorder - Mk. I

SPINNER

Sensate Pervasive Imaging Network for NarrativeExtraction from Reality

Unites wearable human sensing with video capture

Maps sensor data to high-level concepts for creation ofmeaningful videoInvestigates how humans perform this mapping (i.e., how they

create stories and narrative)

Use of wearable sensing allows access to subject/datachannels far beyond what can be achieved withstandard image pixel processing

Mat Laibowitz - PhD in Progress

Overall System Diagram

Behavior

Models

Sensor /Action

Models

Captured

Events

(video clips w /

sensor data )

Assembling of

Narrative

(video )

Meaningful

Narrative

Discourse

(video)

Story

Model

Design /Input

User

Interface

Sensor Network

Internal Mappings

Sensor /Imaging

Physical Network

Sensor Data

Training

DataFE

EDB

ACK

FEED

BAC

K

Camera

System

Sensor Network Devices

Back-endServer

Video/SensorDatabase

WiFi or Ethernet

Imaging Sensor Nodes

Participant

WristSensor

AudioSensor

Participant

WristSensor

AudioSensor

Participant

WristSensor

AudioSensor

WearbleNarrative

Sensor Nodes

Zigbee

Device Overview

Spinner devices includeWrist mounted sensor gesture- and bio- sensingCollar mounted sensor social signaling and audio analysis/recordingCamera system

Wearable devices functions and capabilitiesCamera system controlSensor data capture for video footage cataloguingMultimedia browsing

All devicessupport mesh networkingAre equipped with a location/orientation systemHave dedicated DSP processing for real-time classification of event data

Device Details – Camera System

Deployed to cover entirebuilding (>100 nodes)

3MP CameraMotorized Panning and FocusDedicated Video DSP/ARM - (TI Davinci chip)Real-time Linux OSLCD displays what is being

captured at all timesContains Spinner

Gateway/Sensor board(detailed on next slide)

Device Details – Spinner GatewayWorks with or without camera

boardCommunicates with

wearable/mobile devices inmesh network

Serves as reference beacon forlocation system

EthernetAudio system with DSP

AVR32Environmental Sensing

MotionHumidity and TemperatureLightInfrared Communication and

Detection/Proximity

Device Details – Spinner Social SensorWearable on collar or as

pin/badgeAudio system with DSP for

analytics and CD qualityrecording

Compass for orientation3-axis AccelerometerIR communication and line of

sight detection/proximityLocation engineCaptures social signal and group

dynamics

Device Details – Spinner Wrist SensorWrist worn device3-axis AccelerometerGalvanic Skin Response (GSR) SensorLocation engineUI for interacting with networkStores and plays videos, providing

ownership of video to end userCaptures gesture and indications of affective

state

Expanded Reality

Not just a pipeTechnology removes boundaries from Real World to

allow new content

Contributions

A novel methodology for humans and machines tofilter, organize, and understand large streams ofvideo data

A new form of entertainment and communication thatallows you to create media with your social andpersonal behavior

Toolkit for documentation of daily life that may lead tonew and unexpected insights about random events

These capabilities could lead to a new form of onlinecommunity

The UberBadge• 16-bit MCU w/ 64kb flash, 2k

RAM, and GCC support• 45-LED Display intended to be read

at distance• IR communications• RF communications with second

processor to handle MAC• Up to 256MB of data memory• Audio input and output with

onboard microphone• Onboard accelerometer• Pager motor for vibratory feedback• Multiple ports for expansion

– Accepts Sensor Stack Modules• Optional LCD display

Mediate Group Interaction & Behavior Modeling

Mat LaibowitzResponsive EnvironmentsGroup

63

JAP7/05

Broadcast MessagesInformation on your badge is mainly for other people, not you!

64

JAP7/05

Audience Voting in Auditorium

• Red = No, Green = Yes

Yes

63%

No

37%

65

JAP7/05

Exchange Business Card, Bookmark Demos

66

JAP7/05

Bookmark data posted on website

67

JAP7/05

Bookmark data posted on website

68

JAP7/05

Badge Accelerometer & Audio Data

Accelerometers

Microphone

69

JAP7/05

Auditorium Fidgeting

• Medium-good correlation with length of talk• “Resets” with every presentation

70

JAP7/05

Interest Meter & Group Dynamics

Collaboration with Human Dynamics Group

Badge-Badge Badge-Demo

Uninterested

Interested

Uninterested

Interested

Classification Value = f (voice, motion, face-face time…)

80%Accuracy

71

JAP7/05

Affiliated Wearers

Affiliated people tend to spend more time face-faceand move together!

Accuracy of inferred affiliation: 93%

Face-Face Time (IR)Correlated Motion (accels)

72

JAP7/05

Affiliated Wearers from Energy Only

74

JAP7/05

Timekeeping for Talks

T – 3 mins T – 2 mins T – 1.5 mins T – 1 mins T – 30 sec

Out of time!!• 24 talks in the morning (research updates)• 5 Minute time limit on each!• Audience badges flashed time queues• We didn’t run over (first time ever…)!!

Smart pendants to Amulets

76

JAP7/05

Conclusions• Sensing, computation, and communication become tightly integrated and

commonly embedded• Low power and energy scavenging enable active nodes to be embedded and

“forgotten”• The Ubicomp infrastructure permiates our cities, our dwellings, our objects, our

clothing, and eventually our bodies– Pervasive-Wearable-Implantable

• The 5 human senses locked into our body are augmented by interfaces intoubiquitous sensor network data– Marshall McLuhan for real– Interface devices now - implantables some day– Omniscience…

• This infrastructure mediates everything– Collaboration, business, social interaction…– Context engines filter, represent, and manifest information

• Google for reality• Brave New World

– Privacy, security, promises vs. perils…

77

JAP7/05