- 1. CarSafe Alerting Drowsy and Distracted Drivers using Dual
Cameras on Smartphones Chuang-Wen (Bing) You, Nicholas D. Lane,
Fanglin Chen, Rui Wang, Zhenyu Chen, Thomas J. Bao, Martha
Montes-de-Oca, Yuting Cheng, Mu Lin, Lorenzo Torresani, Andrew T.
Campbell 0
2. Outline OutlineMotivation Approach Design &
implementation Evaluation Related work Conclusion 3. Outline
OutlineMotivation Approach Design & implementation Evaluation
Related work Conclusion 4. CarSafe videoCarSafe video 5. What do
you do if you cant afford a top end car with all those safety
features? 6. Outline OutlineMotivation Approach Design &
implementation Evaluation Related work Conclusion 7.
CarSafeDual-camera appWhat are detected: 1) The following distances
2) Lane trajectory categoriesWhat are detected: 1) Face directions
2) Eye statesWhat are detected: 1) Speed 2) Turns 3) Lane
trajectory categories GPSAccelerometerGyroscope 8. Dangerous
driving eventsDrowsy drivingInattentive drivingTailgatingLane
weavingCareless lane change 9. Limited dual camera access ` `A
blind spot in the frontTime A blind spot in the back Back
cameraSwitching delay Front camera 10. Switching delay & frame
processing time About 500 ms ~ 3 secondsOverheadAbout 50 ms ~ 2
secondsSwitching delay (Front-Back (ms))Switching delay (Back-Front
(ms))Frame processing time (Face detection (ms))Nokia
Lumia8042856.32032.5Samsung Galaxy S3519774301.2HTC One
X1030939680.3iPhone 4S44650370.92iPhone 546752958.48Model 11.
Challenges for real-time processing of dual camera video streams on
smartphones Limited dual-camera accessCamera switching algorithm
Events occurring in blind spotsSensor fusion techniques to provide
blind spot hints Adapt existing vision algorithms Varying mobile
environment Real-time performanceUtilize multicore computation
resources 12. Outline OutlineMotivation Approach Design &
implementation Evaluation Related work Conclusion 13. CarSafe
architectureDriver, road, & car classification pipelines The
Overview of CarSafe alerts user interface dangerous driving
conditionsdangerous driving event engine driver statesroad
conditionscar eventsmulticore computation planner driver
classification pipeline road classification pipeline car
classification pipeline front/back images lane proximity blind spot
hints context-driven camera switching front images front-facing
cameraback imagesback-facing camerasensor readingsGPS, accel, gyr
GPS, accel, gyro & compass o & compass 14. Driver
classification pipelineEye state classification Driver
Classification Pipeline{closed, open}Frontal face detection -
Haar-like feature - Adaboost classifier for frontal facesEye region
estimation - Active shape modelEye center localization -
Gradient-based approachEye state classification - SURF features -
SVM classifier 15. Driver classification pipelineFace
directionClassification Pipeline Driver
classificationfacing.rightFrontal face detection - Haar-like
feature - Adaboost classifier for frontal facesSide face detection
- Haar-like feature - Adaboost classifier for side facesFace
direction classification - Left face facing right - Right face
facing left 16. Road classification pipelineLane trajectory
detectionLane crossing eventsDecision treeCrossingLane marker
detectionLane crossing detectionLane changeLane weavingTrajectory
classification 17. Road classification pipelineFollowing distance
estimation Image plane NMd1MFocal point (F)d2 fN R Road surfaceCar
recognition - Haar-like feature - Adaboost classifier for carsZ1S
Z2Distance estimation - Pin-hole camera projection 18. Road
classification pipelineSpeed, turn, and trajectory inferences GPS
samples 2Inertial sensor readingsd3 1 d2Multi-variate Gaussiand1
Lane change / weaving class Speed estimation & Turn turn
detectionOther classTrajectory classification - Provide as blind
spot hints 19. CarSafe architectureDangerous driving eventof
CarSafe The Overview engine alerts user interface dangerous driving
conditionsdangerous driving event engine driver statesroad
conditionscar eventsmulticore computation planner driver
classification pipeline road classification pipeline car
classification pipeline front/back images lane proximity blind spot
hints context-driven camera switching front imagesfront-facing
cameraback imagesback-facing camerasensor readingsGPS, accel, gyr
GPS, accel, o & compass gyro & compass 20. Dangerous
driving event engineDangerous Driving Event EngineDrowsy Driving
Measuring alertness, PERcentage of CLOSure of the eyelid (PERCLOS),
and declares the driver drowsy if PERCLOS exceeds a
thresholdInattentive Driving Not facing forward for longer than 3
seconds while the car is moving forwardTailgating The safe
following distance is not respected for a period longer than 3
secondsLane Weaving and Drifting The classifier infers lane weaving
continuously for longer than 2 secondsCareless Lane Change No head
turn corresponding to a lane change eventDrowsy drivingInattentive
drivingTailgatingLane weavingCareless lane change 21. CarSafe
ArchitectureContext-driven camera switching The Overview of CarSafe
alerts user interface dangerous driving conditionsdangerous driving
event engine driver statesroad conditionscar eventsmulticore
computation planner driver classification pipeline road
classification pipeline car classification pipeline front/back
images lane proximity blind spot hints context-driven camera
switching front images front-facing cameraback imagesback-facing
camerasensor readingsGPS, accel, gyr o & compass 22.
Context-driven camera switchingScheduled switching Predict when to
switch based on current context (PERCLOS, speed or following
distance)Tb TfTfTimeFront camera Back camera Switching delay 23.
Context-driven camera switchingPre-emptive switching Pre-empted by
blind spot hints or lane proximity information Original switching
pointTfTf TbTimePre-empted by a blind spot hint Front camera Back
camera Switching delay 24. CarSafe architectureMulticore
computation planner The Overview of CarSafe alerts user interface
dangerous driving conditionsdangerous driving event engine driver
statesroad conditionscar eventsmulticore computation planner driver
classification pipeline road classification pipeline car
classification pipeline front/back images lane proximity blind spot
hints context-driven camera switching back images front images
front-facing cameraback-facing camerasensor readingsGPS, accel, gyr
o & compass 25. Multicore computation plannerMulti-core
Computation Planner Leverage the multicore architecture of new
smartphones to perform classificationqueue managerdispatcherdrop
outdated framesdemultiplexer events 26. CarSafe architectureUser
interface alerts user interface dangerous driving
conditionsdangerous driving event engine driver statesroad
conditionscar eventsmulticore computation planner driver
classification pipeline road classification pipeline car
classification pipeline front/back images lane proximity blind spot
hints context-driven camera switching front images front-facing
cameraback imagesback-facing camerasensor readingsGPS, accel, gyr o
& compass 27. User interface & implementationUser Interface
28. Outline OutlineMotivation Approach Design & implementation
Evaluation Related work Conclusion 29. EvaluationEvaluation
Demonstrate CarSafe under real-world conditions where people use
the application in the wild Overall accuracies of CarSafe &
individual pipelines Effectiveness of the context-driven camera
switching Performance improvement of the multicore computation
planner 30. Data collectionData Collection Collecting datasets to
adequately evaluate CarSafeis challenging Two distinct experiments
and datasets 12 participants (11 males and 1 female) Controlled car
maneuvers (6 males) Normal daily driving (5 males and 1 female)
Manually labeled dangerous driving events 31. Overall CarSafe
accuracyOverall CarSafe AccuracyCondition# of true # of false
positives positives# of ground truthPrecisionRecallDrowsy
driving1812250.75 0.60.72Tailgating628780.890.79Careless lane
change122140.860.86Lane weaving160221.000.72Inattentive
driving164250.80.64Overall--1640.830.75smilingsquinting 32. Overall
accuracy for detecting low-level events # of true positivesEvent#
of false positives# of ground truthPrecisionRecallPart 1: events
detected from the driver classifier
facing.right2110310.680.68facing.left236260.790.88Part 2: events
detected from the road classifier
lane.change211240.950.88lane.weaving162221.000.73Part 3: events
detected from the car classifier
turn.right310351.000.89turn.left222250.920.88Overall---0.890.82 33.
Overall accuracy for classifying lane trajectory events Mean
precision and recall are 84% and 76% respectively Detected # of
data segmentsRealLane change / weavingOtherLane change /
weaving19030Other1091127 34. Effectiveness of the context-driven
camera switching Compare carsafe to a static strategy (baseline)
carsafe outperforms baselineThe optimal parameter settingbaseline
carsafe 35. Multicore computation planner benchmarksMulti-core
Computation PlannerFront fps: 5 10 Back fps: 4 11 36. Outline
OutlineMotivation Approach Design & implementation Evaluation
Related work Conclusion 37. Related workRelated Work CostFixed
vehiclemounted devices$$DeviceDetected eventFixed camerasDriver
drowsiness, Lane departure, or following distanceCameras, radar,
and ultrasonic sensorsCollision avoidance, night vision, and
pedestrian detectionTop-end cars$$$$Existing phonebased
systems$SmartphonesCollision & off-road warnings$Dual-camera
SmartphonesDrowsy driving, inattentive driving, tailgating, lane
weaving, and careless lane changeCarSafe 38. Outline
OutlineMotivation Approach Design & implementation Evaluation
Related work Conclusion 39. ConclusionConclusion Propose the design
and implementation of CarSafe and evaluate CarSafe in a small field
trial Explore how to design computation-intensive mobile apps Where
are the performance bottlenecks? Apply and tune vision algorithms
for mobile sensing apps How well existing vision algorithms can
achieve under varying mobile settings? Our future plans Improve the
current prototype Test CarSafe on other phone models or platforms
Stimulate dual camera app interest Encourage major platform vendors
to solve this dual camera problem 40. Drive Safe.Be Safe.Think
CarSafe.0
