User interaction engineering - Engineers Australia · User interaction engineering Measuring cognitive load for safety and efficiency Ronnie Taib 11 June 2015 - Joint Electrical Institutions

User interaction engineering

Measuring cognitive load for safety and efficiency

Ronnie Taib11 June 2015 - Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET

Acknowledgements

• Fang Chen

• Natalie Ruiz

• Eric Choi

• Julien Epps

• Yu Shi

• Bo Yin

• Peng Wang

• James Constable

• Asif Khawaja

• Kun Yu

• Ling Luo

• Jeremy Tederry

• Pega Zarjam

• Siyuan Chen

• Yang Wang

• Ben Itzstein

• Jessica Jung

• Anne Hess

2© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET

What is NICTA?

• Australia’s National Centre of Excellence in ICT• 700 Staff, 5 labs, $100m/y revenue

• NICTA objectives• Research Excellence in ICT• Wealth Creation for Australia

• Transforming Industry• Delivering over $3bn/y impact on GDP• Projects of national scale and impact

• New Industries• Fourteen spin-outs, on every three months

• Skills and Capacity• 22 University partners, 280 PhD Students

Kernel

device software

seL4 microkernel

Hardware

seL4

Hardware

seL4 isolates critical components from software failures

critical componentisolated and protected

untrusted, complex user interface

© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 3



“One of the top 5 places on the planet to go

for detailed understanding of ITS”

Outline

• The problem

• Research

• Real-life applications

• Opportunities


The problemCognitive load theory


http://www.bartleby.com/107/illus1197.html


The problem

• The lack of “at least some cognitive availability and understanding of the situation” may have led pilots to ignore continuing alarms during the fatal accident on the Rio to Paris flight AF447


Baddeley’s modal model of working memory

• Definition• Level of perceived effort

associated with learning, thinking and reasoning (including perception, memory, language, etc)

• Available ‘space’ in working memory in comparison to the ‘space’ needed by a user to complete the task successfully

9

Cognitive Load Theory [Sweller et al. 98]:

© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET

Spare capacity

Germane loadFor schema building, effort of learning

Extraneous loadRepresentation of the task, unnecessary for task completion

Intrinsic loadConcepts of the task

Working memory

Measuring cognitive load

• Subjective measures• Questionnaire: introspection, e.g. NASA TLX• Traditionally most consistent, but disrupts the task

• Physiological measures• Heart rate, EEG, galvanic skin response, eye activity• Intrusive/obtrusive, costly, difficult to analyse

• Performance measures• Intrusive, post hoc, requires a task metrics

• Behavioural measures• Eye gaze, mouse/pen movement, speech, posture• Response-based, non-obtrusive, real-time


Adapted from [Hockey, 2003]

Cognitive load and human interactionLong-term memory

Shared space (7±2)

Perception

Response

Visual processing(Visuospatial sketchpad)

Linguistic processing(Phonological loop)

Central executive

Muscular action

Excitation + vocal tract configuration

Gesture…

Multi-sensory

perception . . .

. . .

Short-term memory


e.g. latency, pitch, jittering

Response

Stimuli

http://www.bartleby.com/107/108.html#i380

http://www.bartleby.com/107/108.html#i380



The big picture: interaction design


System output

Perception

Memory

Response

System input

Content adaptation



Challenges

• Signal acquisition• Collection: obtrusive

• Processing: video, speech

• Calibration• Complex signals

• No direct mapping to cognitive load

• User, time and location dependent

• Real-time output• Signal sensitivity and time scale


Approach

• Baseline

• Signal fluctuations

• Classification using machine learning

Measure cognitive load unobtrusively and in real-time


Multimodal cognitive load measurement


Cognitive load

Subjective ratings

Task Performance

Physiological Behavioural

Mouse

Body movement

Eye activity

Pen input

Gesture

Speech

Linguistic

Fusion

Data-driven Knowledge-based

EOG

EMG

BVP

Temp

MEG

GSR

EEG

Generic process

Raw signalFiltering, cleaning

Feature extraction

Classification, regression, clustering

Cognitive load


Models

ResearchCognitive load measurement


Communicative Indicators of High Load

• Multimodality [e.g. Oviatt et. al., 2007]

• Increased use of integrated multimodal communication as load increases

• Speech signal [e.g. Yin 2008]

• Fundamental frequency, pitch, prosodic features

• Linguistic patterns [e.g. Khawaja 2008]

• Word types, sentence structures, pauses, pronoun use…

• Manual gesture [e.g. Ruiz, 2006]

• Semantic structures (redundant vs. complementary)

• Pen input [e.g. Ruiz, 2007; Yu, 2010]

• Geometric + temporal features of trajectories

• Eye-gaze [e.g. Chen, 2010]

• Pupil dilation


Study 1: traffic monitoring

• Tasks• Incoming information (text)

• Tagging Incident, Accidents, Events

• Notifying and Deploying crews

• Modalities• Speech

• Freehand gesture

• Hand Shapes

• 4 levels of difficulty

Function Buttons (for point and dwell gestures)

System Feedback Area

Task Display Area Main Map Area

Function Buttons (for point and dwell gestures)

System Feedback Area

Task Display Area Main Map Area


Difficulty

Study 1: working memory strategy

• As difficulty increases• Redundant Hybrid Complementary

• High load: multimodal input• Semantic chunks channelled to different

modalities• Least replication possible• Shift to areas marked exclusively for modal use• Similar to data acquisition “modality effect”

• Working memory strategy• Maximise modal working memory• Use less central executive resources, which can

be utilised for higher-order processes such as planning, understanding and hypothesis testing

Redundant

Spatial VerbalExecutiveA

B

A

B

Hybrid


B

A

Complementary


B

0

10

20

30

40

50

60

70

80

90

Level1 Level2 Level4

Q1MinMeanMaxQ3

Proportion of Purely Redundant Turns by Level


Study 1: automated gesture segmentation

• Manual annotation of gestures• Timestamping, annotating

• Pointing, dwelling

• Hand shapes

• Trajectory extraction from video

• Segmentation• Accuracy under 50ms


13000

Sp

ee

d

Time (ms)

Study 1: automated gesture segmentation

Gesture

identified

Start time End time

Ditch 1

Ditch 2 Ditch 3

Ditch 1

Ditch 2 Ditch 3

Start time before

Start time nowAuto Start time Auto End time

Manual start

time

Manual end time

1. Segment gestures using speed threshold and noise level

2. Compare result with manual segmentation

3. Spot and filter out ditches

4. Trace back to the previous local minima

Start time at the beginning

Start time after ditches were filtered out

Start time now© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 22

Study 2: reading data collection

• Primary task• Read 3 stories (1 page each), then oral Q&A session• Lexile ratings to quantify complexity (www.lexile.com)• From primary school level to post-graduate

• Secondary task• Listening/count beeps/numbers

• 4 levels of difficulty (incl. baseline)

• Modalities and data• Speech: reading + Q&A• Performance• Galvanic skin response (GSR)

Martin Luther King, Jr.'s "I Have a Dream" Speech

March on Washington, DC, August 28, 1963

I am happy to join with you today in what will go down in history as

the greatest demonstration for freedom in the history of our nation.

Five score years ago, a great American, in whose symbolic shadow

we stand today, signed the Emancipation Proclamation. This

momentous decree came as a great beacon light of hope to millions

of Negro slaves who had been seared in the flames of withering

injustice. It came as a joyous daybreak to end the long night of

their captivity.

But one hundred years later, the Negro still is not free. One

hundred years later, the life of the Negro is still sadly crippled by

the manacles of segregation and the chains of discrimination. One

hundred years later, the Negro lives on a lonely island of poverty in

the midst of a vast ocean of material prosperity. One hundred years

later, the Negro is still languishing in the corners of American

society and finds himself an exile in his own land. So we have come

here today to dramatize a shameful condition.

In a sense we have come to our nation's capital to cash a check.

When the architects of our republic wrote the magnificent words of

the Constitution and the Declaration of Independence, they were

signing a promissory note to which every American was to fall heir.

This note was a promise that all men, yes, black men as well as

white men, would be guaranteed the unalienable rights of life,

liberty, and the pursuit of happiness.

It is obvious today that America has defaulted on this promissory

note insofar as her citizens of color are concerned. Instead of

honoring this sacred obligation, America has given the Negro people

a bad check, a check which has come back marked "insufficient

funds." But we refuse to believe that the bank of justice is

bankrupt. We refuse to believe that there are insufficient funds in

the great vaults of opportunity of this nation. So we have come to

cash this check — a check that will give us upon demand the

riches of freedom and the security of justice. We have also come to

this hallowed spot to remind America of the fierce urgency of now.

This is no time to engage in the luxury of cooling off or to take the

tranquilizing drug of gradualism. Now is the time to make real

promises of democracy. Now is the time to rise from the dark and

desolate valley of segregation to the sunlit path of racial justice.

Now is the time to lift our nation from the quick sands of racial

injustice to the solid rock of brotherhood. Now is the time to make


http://www.lexile.com/

Study 2: results

• Linguistic features

• Pauses: more and longer, increased latency with difficulty

• Complexity: decreased lexical density, longer and broken

sentences

• Speech-based classification

• 71% accuracy (speaker independent, up to 84% speaker

dependent)

Martin Luther King, Jr.'s "I Have a Dream" Speech

March on Washington, DC, August 28, 1963

I am happy to join with you today in what will go down in history as

the greatest demonstration for freedom in the history of our nation.

Five score years ago, a great American, in whose symbolic shadow

we stand today, signed the Emancipation Proclamation. This

momentous decree came as a great beacon light of hope to millions

of Negro slaves who had been seared in the flames of withering

injustice. It came as a joyous daybreak to end the long night of

their captivity.

But one hundred years later, the Negro still is not free. One

hundred years later, the life of the Negro is still sadly crippled by

the manacles of segregation and the chains of discrimination. One

hundred years later, the Negro lives on a lonely island of poverty in

the midst of a vast ocean of material prosperity. One hundred years

later, the Negro is still languishing in the corners of American

society and finds himself an exile in his own land. So we have come

here today to dramatize a shameful condition.

In a sense we have come to our nation's capital to cash a check.

When the architects of our republic wrote the magnificent words of

the Constitution and the Declaration of Independence, they were

signing a promissory note to which every American was to fall heir.

This note was a promise that all men, yes, black men as well as

white men, would be guaranteed the unalienable rights of life,

liberty, and the pursuit of happiness.

It is obvious today that America has defaulted on this promissory

note insofar as her citizens of color are concerned. Instead of

honoring this sacred obligation, America has given the Negro people

a bad check, a check which has come back marked "insufficient

funds." But we refuse to believe that the bank of justice is

bankrupt. We refuse to believe that there are insufficient funds in

the great vaults of opportunity of this nation. So we have come to

cash this check — a check that will give us upon demand the

riches of freedom and the security of justice. We have also come to

this hallowed spot to remind America of the fierce urgency of now.

This is no time to engage in the luxury of cooling off or to take the

tranquilizing drug of gradualism. Now is the time to make real

promises of democracy. Now is the time to rise from the dark and

desolate valley of segregation to the sunlit path of racial justice.

Now is the time to lift our nation from the quick sands of racial

injustice to the solid rock of brotherhood. Now is the time to make


Study 3: traffic incident management

• Task• Creating detours • Creating green light corridors• Requires calculations: scratchpad available

• Modalities

• Digital pen input

(tablet monitor)

• Speech • GSR

• Data• Performance• Subjective ratings

3 levels of difficulty1. Easy: 6 streets2. Medium: 10 streets3. Hard: 16 streets


Study 3: degeneration of interactive shapes

• Geometric analysis of trajectory• 12 features incl. angle at start stroke,

angle and end stroke, duration, length, sharpness etc. [Rubine 1991]

Selection Examples Shape ExamplesStandard form and Sample inputs


Study 3: degeneration of interactive shapes

• Malahanobis distance (MDIST)• Weighted Euclidean distance

• Stdev away from standard form captured during training

As difficulty increases, the curve moves away from 0, indicating a greater degree of degeneration (statistically significant)


Study 4: basketball skill acquisition (with AIS)

• Player Formation Recall• 10 second video played• Recalling increasing numbers

• Modalities• Pen input

(Tablet monitor)• Speech • GSR• Eye tracking

• Data• Performance and subjective ratings

3 levels of difficulty- Low (Easy): 3 players- Medium (Med): 6 players- High (Hard): 10 players


Study 4: pen input

Attacker

Defender

Ball carrier


Study 4: pen gesture temporal features

• Duration• Decreased with tasks difficulty in

82% of gestures

• Significant decrease from Low to High (t-test, p<0.05) in 50% gestures tested

• Velocity• Increased with difficulty in 78% of

gestures

• Significant increase from Low to High (t-test, p<0.05) in 44% gestures tested

0

100

200

300

400

500

600

cross circle ball carrier

Subject 9 Gesture Duration

easy

medium

hard

0

0.05

0.1

0.15

0.2

0.25

0.3

cross circle ball carrier

Subject 12 Velocity


Study 5: Stroop test

Simply read aloud each word

RED YELLOW BLUE GREEN BLACK

PINK ORANGE BROWN GRAY PURPLE

GREEN RED BLACK BLUE YELLOW

PURPLE GRAY PINK ORANGE BROWN



Cognitive Load Instructions Example

Low Read colour

words

Blue, green

Yellow, red

Medium Name word

colour

Blue, green,

red

High Same as

medium load

except with time

pressure

Blue, green,

red



• Speech only classification: 79% accuracy (3 levels)

• iPhone implementation


Real-life applicationsImproving efficiency and safety


Real-life case studies

• Emergency communications centre in North America (ambulance dispatch)• Speech classification: 82% accuracy

• High load detection rate: 96%

• Contact centre operator in Australia (5000+ seats)• Reduced attrition rate by 50% by automated operator selection

• www.braingauge.com

• Air traffic controllers (training centre)• Speech classification: 88% accuracy

• Estimates produces every 2 minutes


http://www.braingauge.com/

Our safe future

• Adaptive cruise control

• Adaptive headlamps

• Advanced automatic collision notification

• Automatic parking

• Automotive night vision with pedestrian detection

• Distance control assist

• Traffic sign recognition

• Lane departure warning system

• Drowsiness warning

• Blind spot monitoring

• Driver monitoring system

• Dead man's switch

• Platooning

• Robotic car or self-driving car

Source: EU Commission’s Intelligent Car Initiative (2006)


But inattention is an omnipresent danger

Sleeping can be dangerous


Inattention at the wheel

• 10% of car crashes in USA• An Examination of Driver Distraction as Recorded in NHTSA Databases [NHTSA

2009]

• Over 25% of crashes involved a form of driver inattention8.3% of drivers were distracted at the time of their crash• The Role of Driver Distraction in Traffic Crashes [Stutts 2001]

• Technology-based distraction is only 15% of distraction related incidents• The impact of driver inattention on near-crash/crash risk: An analysis using

the 100-car naturalistic driving study data [Klauer 2006]


Driver mental state monitoring

• Identify mental states at risk• Don’t wait until it is too late

• Not just phones and distractors

• Use existing sensors: dashboard camera, seat/wheel sensors, body sensors

• Unique capability• Collaboration NICTA, Emotiv, Fraunhofer

• Cognitive load expertise Clever simulator tasks

• Machine Learning expertise Models out of noisy data


Theoretical challenges

• Cognitive load• How to quantify judgement

• Causality between task, performance, physiology and behaviour

• Simulator vs. test track vs. FOT vs. reality

• Machine learning• What features? Physiological, behavioural

• Time series with varying sampling rate

• Features with different delay and duration after stimulus

• Online learning, user independent seed models?


Practical challenges

• Task design• Realistic tasks• Controlled cognitive load, emotions, UX• Elicit right amount of speech, movement

• Signal acquisition• Sensors• Loggers: NICTA’s OML• Machines and synchronisation issues• Reducing manual annotation

• Implementation• A lot of work to create the virtual scene


Approach


Preliminary user study

• 19 voluntary participants

• 45 min each

• Sensors• Physiological: GSR, EEG, Pupil dilation, temperature

• Behavioural: wheel, pedals, speech, eye gaze, wrist and head motion

• Performance: task markers, repeat loops with varying conditions, use of devices (in/out)

• Audio-visual: speech, face video, glasses camera, pupils


Driving simulator


Instrumentation

Eye gazePupil dilationSkin conductanceEEG

PostureUpper body videoSpeechWheel and pedals



On the phone


Adjusting the radio


Sending a text message

Preliminary study: glossy outcome

www.forthebetter.com.au


Frustration study


Frustration study: challenges

• Frustration in simulator • Simulated frustration?

• Sensors• Positioning?

• Features?


Frustration study: challenges

0

0.2

0.4

0.6

0.8

1

1.2

-20000 0 20000 40000 60000 80000 100000 120000 140000

actions

P01S04T0

P01S04T1

P01S04T2

P01S04T3

P01S04T4

P01S04T5

P01S04T6

P01S04T7

P01S04T8

P01S04T9

Action on the pedals Moving on the seat

Action on the wheel


Frustration study: results

• Experiment• 11 subjects

• 128 = 16 sensors * 8 features

• 1241*128 samples (10s each)

• Analysis• Multinomial regression with forward feature selection

• Support Vector Machine

• Infinite Gaussian Mixture Model

• Neural nets


Frustration study: results

• Accuracies

• Features selection• Two of the classes perform much better


Method MRMR + FFS

SVMSVM+GM

MGMM

Neural Network

Accuracy 41% 39% 33% 1% 15% 35%

class

1

class

2

class

3

class

4

class

5

class

6

class

7

class

8

class

9

class

10

class

11

macro

average

Standard

deviation

precision

0.20

0

0.24

3

0.25

0

0.30

0

0.28

3

0.13

3

0.28

6

0.69

0

0.57

6

0.34

2

0.13

3 0.312

0.173

recall

0.18

0

0.18

0

0.12

0

0.30

0

0.26

0

0.08

0

0.24

0

0.80

0

0.68

0

0.52

0

0.22

0 0.325

0.236

F-Score

0.18

9

0.20

7

0.16

2

0.30

0

0.27

1

0.10

0

0.26

1

0.74

1

0.62

4

0.41

3

0.16

5 0.312

0.203

Driver mental state monitoring: opportunities

• Driver assistance systems• Real-time detection of mental states at risk: overload, fatigue, daydreaming…

• Testing and training drivers• Assessing driver reactions, ensure appropriate level of proficiency

• Leverage current sensors• Dashboard camera, seat/wheel sensors, body sensors


OpportunitiesUser interaction design


Opportunities

• Internet of things• Personal monitoring trend

• Car sensors

• Smart buildings

• Smart fabrics

• Analytics technology is mature• Multimodal

• Real-time

• Simply need to connect

© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 57 © 2015 Emotiv

Thank [email protected]

www.linkedin.com/in/taibr


Documents

User interaction engineering - Engineers Australia · User interaction engineering Measuring cognitive load for safety and efficiency Ronnie Taib 11 June 2015 - Joint Electrical Institutions