ARTIFICIAL PASSENGER (A Sleep Prevention Dialogue Based Car System)

Submitted ByMutabunga Bibentyo

Christian(1313026)

Introduction IBM (International business machines corporation, NY) has

developed a software that holds a conversation with the driver to determine whether the driver can respond alertly enough, called “Artificial Passenger”.

This was designed to make long solo journeys safer and more bearable.

Why Such System? According to a national survey in UK and

USA, it is observed that the driver fatigue annually causes

• 10000 crashes• 1500 deaths • 7100 injuries

Majority of off-road accidents observed, were caused by eye closure of half and even 2-3 seconds, where the normal human eye blinks at 0.2-0.3 seconds.

What is an Artificial Passenger?

Natural language e-companion.

Sleep preventive device in cars to overcome drowsiness.

Life safety system.

What Does It Do? Detects alarm conditions through sensors.

Broadcasts pre-stored voice messages over the speakers.

Captures images of the driver.

Condition SensorCondition Sensor

Mobile Indicator Mobile Indicator DeviceDevice

About Artificial Passenger The AP is an Artificial Intelligence based companion that will

be resident in software and chips embedded in the automobile dashboard.

The system has a conversation planner that holds a profile of you, including details of your interests and profession.

A microphone picks up your answer and breaks it down into separate words with speech-recognition software.

A camera built into the dashboard also tracks your lip movements to improve the accuracy of speech recognition.

A voice analyzer then looks for signs of tiredness by checking to see if the answer matches your profile. Slow responses and lack of attention are signs of fatigue.

If you reply quickly and clearly, the system judges you to be alert and tells the conversation planner to continue the line of questioning.

If your response is slow or doesn’t make sense, the voice analyzer assumes you are dropping off and acts to get your attention.

If driver displays signs of fatigue, the artificial passenger might be programmed to open all the windows, sound a buzzer, increase background music volume, or even spray the driver with ice water.

Devices Used in Artificial Passenger Eye tracker Voice recognizer or speech recognizer Natural language processor Driver analyzer Conversational planner Alarm Microphone Camera

Working ComponentsEye-TrackerEye-Tracker

Collecting eye movement data requires both hardware and software.

Hardware:: Head-Mounted systems or remote systemsHardware:: Head-Mounted systems or remote systems

Both systems measure the corneal reflection of the infrared LED, which illuminates and generates a reflection off the surface of the eye. This action causes the pupil to appear as a bright disk in contrast to surrounding iris and creates a small glint underneath the pupil which is used for calibration and tracking. Head mounted systemsHead mounted systems use multiple data points to record

eye movement and measures pupil glint from multiple angles. Remote systemsRemote systems ,in contrast, measure the orientation of the

eye relative to a fixed unit such as camera.

Monitoring System

Software:Software: Data collection and analysis is handled by eye-tracking

software. Software catalogs eye-tracking in one of the two ways1. ERICA’s Eye Gaze software uses a small red x to

represent eye movement.2. In other, data is stored as a series of x/y coordinates

related to specific grid points on the computer screen

Eye Tracker

Working Components( cont…) Automatic Speech Automatic Speech

RecognitionRecognition There are two ASRs used in the system:

First one is “speaker independent” and used for decoding voice signals of the driver.

Second one operates with voice car media and decodes tapes, audio , telephones, mails etc.

Working Components( contd…)Natural Language ProcessorNatural Language Processor

Processes the decoded signals of voice or text data provided from the ASR. Identifies related meanings from the contents of the decoded messages.

Produces variant of responses.

This output goes to the driver analyzer as an input.

Working Components( contd…)Driver AnalyzerDriver Analyzer

Receives the textual data and voice data.

Measures the time response using a clock.

Time responses, conclusions about driver’s alertness will be passed on to the conversational planner.

Working Components( contd…)Conversational PlannerConversational Planner

It is the heart of the system.

Instructs the language generator to produce the response.

If the driver is in perfect condition CP instructs the language generator to continue the conversation otherwise it will be instructed to change the conversation.

Working Components (contd…)

AlarmAlarm If the CP (conversational planner)

receives information that the driver is about to fall asleep an alarm system is activated.

MicrophoneMicrophone For picking up words and separate them by some internally used software for conversation.

Working Components (contd…)CameraCamera

This will track the lip movements of the driver. Used to improve the accuracy of the speech

recognition.

Steps Involved in Tracking Automatically initialize lips and eyes using color predicates

and connected components. Track lip corners using dark line between lips and color

predicate even through large mouth movement like yawning. Construct a bounding box of the head.

Determine rotation using distances between eye and lip feature points and sides of the face.

Determine eye blinking and eye closing using the number and intensity of pixels in the eye region.

Determine driver vigilance level using all acquired information.

Detecting Driver Vigilance Aiming a single camera at a head of the driver.

Detecting frequency of up and down nodding and left to right rotations of the head within a selected time period with the camera.

Determining frequency of eye blinking and eye closing.

Determining frequency of yawning of the driver within the selected time period with the camera

Generating an alarm signal in real time if the frequency value of the up and down nodding, the left to right rotations, the eye blinking, the eye closings, the yawning exceeds a selected threshold value.

Applications Interface with Neighboring Interface with Neighboring CarsCars -Determines if a

driver presents a high safety risk for ex. falling asleep, tired, inexperienced or under the influence of alcohol and signals the cars nearby to be careful of the driver.

Medical ApplicationMedical Application -The system can monitor a driver and detect if they are sick ,for ex. having a stroke or heart attack.

In any problem it alerts the vehicles near by , so the driver there can become alert.

Opens and closes the doors and windows of the car automatically.

It is also used for the entertainment. Also used in cabins in airplanes, trains, boats etc

Future Implementation

Will provide us with shortest time routing based on road conditions changing because of weather and traffic, information about the cars on the route, destination requirement (as-flight has been delayed etc) and nearby incentives (“e-coupons” for restaurants along your way).

ConclusionMethod for monitoring driver

alertness Sufficient time to avert an accident. Successful implementation of Artificial

passenger would allow use of various services in car like reading emails, navigation, downloading music files, voice games etc without compromising on driver safety.

Questions?Questions?