Session 48 Johan Karlsson

ALR-JKAR/Jan2011/Transportforum - 1 Copyright Autoliv Inc., All Rights Reserved

Fordonsstrategisk Forskning och Innovation FFI – D4SF

Driver Drowsiness and Distraction Detection bySensor Fusion

D4SFJohan Karlsson, AutolivTransportforum 2011



Overview

�Background�Goals

� Drowsiness detection� (Distraction detection)

�Method� Data collection� Training/optimization of classifier� Sensor fusion

�Results� Reference – ground truth� Improvement by (f)using parallel detectors



Driver Drowsiness detection

�Drowsy driving is a road safety problem- drowsiness contributing in 10-30% of accidents (Anund & Patten 2010)

�What can be done?� Commercial fleet traffic

� Fatigue Risk Management� Work time regulation� Detection and warning

� Privately owned vehicles� Detection and warning

�Detection?� Detection systems offered as option from several OEMs� So far, performance is far from ideal...



Target and Goals

�Different indicators exist- ’Physiology’ measures - blink duration etc. - Driving performance measures - lane keeping measures- Environment measures - time of day, traffic, road type

(previous sleep possible in commercial fleet vehicles??)

� Various indicators have different strengths and weaknesses

�Improve performance by fusing data from multiple indicators

� The fusion algorithm shall show an improvement in:- Overall performance- Reduced number of faulty detections- Increased number of correct detections

+ +

+

+

–

Specificity

–+Indicator C

+ ++ +Fusion

+–Indicator B

++Indicator A

AvailabilitySensitivity



� Data collection

� Relevant vehicle data

� Speed, lane position, SW angle, pedals etc.

� Video based gaze direction, eyelid opening, head pos

� KSS value every 5 minute

� EEG, EOG and EMG

� Video recordings (road scenery and cabin)

� In total: 43 drivers have completed 3 drives each

� Procedure: Each driver drove three times during one day (day, evening and night). Trip duration 80-90 minutes

Data collection�On-road tests were conducted with governmental approval (N2007/5326/TR) and ethical approval by Regional ethics approval board (EPN 142-07 T34-09).



Test Route

Road RV34Mostly 9 m width Driving lane width 3,75 m Speed limit - mostly 90 km/h

Numbers on map are Yearly day traffic volume in January 2002

We know of only a few similar studies performed on public roads

90 minute driving, approx 115 km distance

Rested safety driver –dual command



Ground Truth – KSS

+ Simple to collect+ Simple to understand – immediately ready for analysis

- Training needed for participants- Some offset for inexperienced participants?

KSS Description in Swedish Verbal description

1 extremt pigg extremely alert

2 mycket pigg very alert

3 pigg alert

4 ganska pigg rather alert

5 varken pigg eller sömnig neither alert nor sleepy

6 första tecknen på sömnighet some signs of sleepiness

7 sömnig, ej ansträngande vara vaken sleepy, but no effort to keep alert

8 sömnig, viss ansträngning vara vaken sleepy, some effort to keep alert

9 mycket sömnig, ansträngande vara vaken, kämpar mot sömnen

very sleepy, great effort to keep alert, fighting sleep



Blink duration (AS): Mean blink duration

Lane keeping variability (Lane): Variability in Steering wheel position or Lane Position. e.g. using Generic Variability Indicator (Sandberg 2008) .

Time-of-day (TPM): Expected drowsiness with regard to time of day (circadian rythm)

* Each indicators has several parameters that needs to be tuned for optimal performance

))1(( pxxz ii δδ −+−=

ki

N

ii zzw

NG ||)(

1

1∑

=

=

)()( 11)(

RRLL zR

zL

e

c

e

czw βαβα −−−− +

++

=

GVI (Sandberg 2008)

Short Blink Long Blink

400 ms

200 msOpen

Closed

OpeningClosed

Closing

Amplitude

Example indicators of driver sleepiness



Video examples

Video examples



� SVM (Support Vector Machine):

� Machine learning method using data from field tests to calculate “best fit” function between indicator values and ground truth (KSS rating scale)

� Indicator parameters optimized simultaneously with training of SVM

� Data sets for SVM training and validation are from separate drivers. Thus, validation is done on truly “never-before-seen” data.

Sensor fusion

Indicator A

Indi

cato

r B

Drowsy data

Alert data

Goal: Find the maximum margin hyperplane



Evaluation Method� Assuming a binary classification,

alert or drowsy� Performance is the mean value of

sensitivity and specificity� Performance is related to the

proportion of the time where the algorithm is correct

DrowsyNon-

Drowsy

Detect A(hit)

B(false hit)

Non-Detect

C(miss)

D(correct reject)

Sum A + C B + D

2

yspecificitysensitiviteperformanc

DB

Dyspecificit

CA

Aysensitivit

+=

+=

+=

KS

S

Ground truth

Alg

orith

m o

utpu

t

KSS = ground truth

Binary Algo output

Ground truth cutoff

Time



Example of results from sensor fusion

0.84 (0.89)0.76 (0.81)0.80 (0.85)Blink + Steer + Circ.

0.92 (0.88)0.68 (0.68)0.80 (0.78)Blink + Lane + Circ.

0.96 (0.95)0.36 (0.32)0.66 (0.64)Blink

0.83 (0.87)0.77 (0.79)0.80 (0.83)Blink + Circadian

SpecSensFitnessModel

First figure is training data performancesecond figure is test data performance � Decision every 1 minute

� KSS >= 7 � drowsy� KSS < 7 � alert



Fulfillment of goals

� The fusion algorithm shall show an improvement in:

- Improved performance true

- Increased number of correct detections true

- Reduced number of faulty detections (?)

Clearly improved overall performance

– Minor differences between different combinations



Summary

�Controlled experiment on public roads� 43 drivers so far�What is ideal performance?

� Method developed with focus on mathematical performance� Most important goal is to have relevant warnings

�More data is needed: � Different road types� Different conditions (weather, drive duration etc.)� Different driver types (age, cultural differences etc.)



Thank you for you attention!

Education

Session 48 Johan Karlsson