Project Proposal Document · ,1752'8&7,21 %dfnjurxqg /lwhudwxuh 6xuyh\ ,q wrgd\¶v idvw prylqj zruog gulylqj kdv ehfrph d qhfhvvlw\ lq hyhu\rqh¶v olih ,q wkh olih ri prvw jhqhudo

$Page 1: Project Proposal Document · ,1752'8&7,21 %dfnjurxqg /lwhudwxuh 6xuyh\ ,q wrgd\¶v idvw prylqj zruog gulylqj kdv ehfrph d qhfhvvlw\ lq hyhu\rqh¶v olih ,q wkh olih ri prvw jhqhudo$
A Smart Driving Assistance System to Elevate the Driving

Experience of Drivers in Sri Lanka

18-018

Project Proposal Report

Nuwan Kodagoda

Mohamed Rauf Aaquiff Ahnaff

Mohammed Musthafa Mohammed Munsif

Saranki Magenthirarajah

Kiruthiga Balendran

BSc. (Hons) in Information Technology

Department of Software Engineering

Sri Lanka Institute of Information Technology

Sri Lanka

April 2018

A Smart Driving Assistance System to Elevate the Driving

Experience of Drivers in Sri Lanka

18-018

Project Proposal Report

(Proposal documentation submitted in partial fulfilment of the requirement for the

Degree of Bachelor of Science Special (honors)

In Information Technology)

BSc. (Hons) in Information Technology

Department of Software Engineering

Sri Lanka Institute of Information Technology

Sri Lanka

April 2018

i

DECLARATION

We declare that this is our own work and this proposal does not incorporate without

acknowledgement any material previously submitted for a degree or diploma in any

other university or Institute of higher learning and to the best of our knowledge and

belief it does not contain any material previously published or written by another

person except where the acknowledgement is made in the text.

The supervisor/s should certify the proposal report with the following declaration.

The above candidates are carrying out research for the undergraduate Dissertation

under my supervision.

……………………………………….. …..…...………….………………

Signature of the supervisor Date

……………………………………….. …..…...………….………………

Signature of the co-supervisor Date

Name Student ID Signature

M.R. Aaquiff Ahnaff IT15048738

M. M. M. Munsif IT15015754

M. Saranki IT15101266

B. Kiruthiga IT15135308

ii

ABSTRACT

Driving requires one's utmost concentration to be safe and effective. It is impossible for people to concentrate on mundane repetitive tasks for a long period of time and thus leads to distracted driving. There are systems that have tried to achieve this but limited themselves only to the higher-end market through vendor-specific implementations. Other applications have tried to achieve this but have not succeeded. We propose a system that will assist the drivers with these tasks in order to achieve a safer and more efficient driving experience. By identifying road signs through image processing, the user would be alerted to road signs that might go unnoticed. The driver would also be alerted to pedestrian crossings without signs. A real-time lane detection algorithm will ensure that the driver stays in the lane and would be alerted if otherwise. Recording of speed data from the users and formulating safe speeds and driving statistics for a user will help the drivers to maintain a safe speed throughout the journey.

Keywords: Driving Style, Road signs detection, Pedestrian crossing detection, Lane

detection

iii

TABLE OF CONTENTS

DECLARATION .................................................................................................... i

ABSTRACT ........................................................................................................... ii

TABLE OF CONTENTS ...................................................................................... iii

LIST OF FIGURES ................................................................................................v

LIST OF TABLES ................................................................................................ vi

LIST OF ABBREVIATIONS .............................................................................. vii

1 INTRODUCTION ...........................................................................................1

1.1 Background & Literature Survey ................................................................1

1.1.1 Driving style ........................................................................................2

1.1.2 Pedestrian crossing detection ...............................................................3

1.1.3 Road signboards detection ...................................................................5

1.1.4 Lane detection .....................................................................................8

1.2 Research Gap ............................................................................................ 10

1.2.1 Speed and driving style ...................................................................... 10

1.2.2 Pedestrian crossing detection ............................................................. 11

1.2.3 Road signboards detection ................................................................. 13

1.2.4 Lane detection ................................................................................... 16

1.3 Research Problem ..................................................................................... 18

1.3.1 Problems in detecting speed information and driving patterns ............ 18

1.3.2 Hurdles in identifying pedestrian crossings ........................................ 19

1.3.3 Difficulties in identifying the road signboards .................................... 20

1.3.4 Obstacles in identifying lane separators ............................................. 21

2 OBJECTIVES ............................................................................................... 23

2.1 Main Objectives........................................................................................ 23

2.2 Specific Objectives ................................................................................... 23

2.3 Research Questions ................................................................................... 23

3 METHODOLOGY ........................................................................................ 25

3.1 Data Gathering ......................................................................................... 25

3.2 Driving Style ............................................................................................ 26

3.3 Pedestrian Crossing Detection .................................................................. 28

3.4 Road Signboards Detection ....................................................................... 29

3.5 Lane Detection ......................................................................................... 31

3.6 Testing...................................................................................................... 32

iv

3.6.1 Application performance and accuracy .............................................. 32

3.6.2 Driving style ...................................................................................... 32

3.6.3 Pedestrian crossing detection ............................................................. 33

3.6.4 Road signboards detection ................................................................. 33

3.6.5 Lane detection ................................................................................... 34

3.6.6 User Testing ...................................................................................... 34

3.7 Gantt Chart ............................................................................................... 35

4 BUSINESS PLAN ......................................................................................... 36

5 DESCRIPTION OF PERSONNEL AND FACILITIES ............................. 37

6 BUDGET AND BUDGET JUSTIFICATION .............................................. 38

7 REFERENCE LIST ...................................................................................... 39

8 APPENDIX .................................................................................................... 43

v

LIST OF FIGURES Figure 1 - High-Level Architecture ......................................................................... 26 Figure 2 - Procedure of Driving Style Analyzer ...................................................... 27 Figure 3 - Procedure of Pedestrian Crossing Detection and Presentation ................. 28 Figure 4 - Procedure of Road Signboard Detection and Presentation ....................... 29 Figure 5 - Procedure of Lane Detection and Presentation ........................................ 31 Figure 6 - Gantt Chart ............................................................................................. 35 Figure 7 - Work Breakdown Structure..................................................................... 46

vi

LIST OF TABLES

Table 1 - Comparison of driver speed related applications....................................... 11 Table 2 - Comparison of Pedestrian Crossing Detection System .............................. 12 Table 3 - Comparison of Road Sign Detection Applications .................................... 15 Table 4 - Comparison of Lane Detection and related Application ............................ 17 Table 5 - Components and corresponding research area .......................................... 25 Table 6 - Description of Personnel and Facilities..................................................... 37 Table 7 - Table of Budget and Justification ............................................................. 38 Table 8 - Research and Product Comparison Summary ........................................... 45

vii

LIST OF ABBREVIATIONS

DB – Database

IP – Image Processing

ML – Machine Learning

DUI – Driving under the influence

GPS – Global Positioning System

MMS – Mobile Mapping System

LDW – Lane Departure Warning

FPS – Frames per Second

ROI – Region of Interest (in Image Processing Context)

UI – User Interface

CAN – Controller Area Network

AWS – Amazon Web Services

ANN – Artificial Neural Networks

LOG – Laplacian Of Gaussian

GUI – Graphical User Interface

1

1 INTRODUCTION

1.1 Background & Literature Survey

In today’s fast-moving world, driving has become a necessity in everyone’s life. In the

life of most general public use of their private vehicles for their day to day activities

has increased drastically. Therefore, safe driving is something to be taken very

seriously and it is important to be responsible as well.

Drivers are tasked with many responsibilities while driving a motor vehicle. Being

fully aware of the surroundings and staying focused on the road are the main

responsibilities. However, concentration is the key necessity to take up these

responsibilities. There are plenty of reasons for a driver failing to concentrate while

driving. Work stress, health and mental condition, alcohol, use of mobile phones and

other environmental factors affect a person's current mentality and pave the way to

lose concentration.

In Sri Lanka, most of the accidents take place due to the carelessness of the driver.

They fail to pay attention to the road rules and end in colliding with another vehicle.

In morning rush hours, drivers are in a hurry to get to their destination. Most of the

pedestrians neglect to cross the road in the pedestrian crossing. This also has led to

several road accidents.

In a long journey, high level of concentration and the necessity of assistance are very

much needed for drivers. In situations like driving in a new route, a lone driver will be

in need of assistance and might need support in places of difficulty and emergency. A

guide who suggests on average speed to be taken at the bends, forewarns of the nearby

road signs and pedestrian crossings, alerts when the vehicle is departing from the lane

and directs on the path and navigations would be a great advantage to the drivers.

There are systems that provide the drivers with some of the above-stated guidance.

These systems help to solve some of the issues faced by the drivers and they reduce

the level of causing accidents in minor level. Thus, the proposed system is there to

bind all the major difficulties addressed by the drivers and to be a guide to assist in the

2

long journey to make smart decisions. This will help in reducing the cause of accidents

and ensures that drivers are obeying the law.

A smart driving experience includes the following components which are the main key

factors for a safe drive.

1.1.1 Driving style

Driving behavior affects traffic safety, fuel consumption, and emissions according to

the publication by J. Ferreira et al [1]. Several research papers have been published to

identify driving behavior from users using smartphones. One of the most important

attributes of driving behavior is speed. It is tricky for a driver to maintain the same

speed throughout the entire journey. This is because different parts of a road have

different safe speeds. Exceeding this can be risky and going well below it can increase

the time of the journey. Maximum speed limits posted on road signs are merely

guidelines not to exceed the speed limit and cause a violation of indicating safe speeds.

DriveScribe is an android application that turns a smartphone into a driving coach [2].

Based on the speed maps that have been included in the application, it warns the user

when they exceed the speed limit at that site. It uses different colors to indicate if the

driver is close to the speed limit, exceeding it a little or too much. It also records the

violations in the user’s online profile. These are also some other applications that warn

users when they exceed the speed limits based on the speed signs. But these road signs

only consider the maximum speeds posted and might not be the best guideline for a

safe drive. The factor these applications have missed to take into is the experience of

the previous drivers and the speed of the majority around those sites.

Trapster [3] is a free navigation social networking mobile application that alerts users

of police speed traps and Driving under the influence (DUI) checkpoints. It also allows

the user to publish their trip data on social media such as Facebook. The app uses

Global Positioning System (GPS) or Wi-Fi to find the current location of the user and

send the audio or text alerts when approaching a threat. While it is a very helpful

application, it also encourages bad driving behaviors since it reduces the risk of getting

caught. Instead, we could use a similar concept to share the experiences of the previous

drivers so that the next driver could benefit. Instead of showing speed traps, we could

3

show the average speed at that site, so that the users would reduce their speed to

comply with the behavior of the majority.

Driving Coach is a research about an android application by R. Araújo et al [4] to

evaluate driving efficient patterns. This application gathers data from the device using

sensors like GPS and the vehicle Controller Area Network (CAN) bus and then uses

that to extract driving features and statistics. This data is then used to classify the

driving style according to the fuel efficiency of the driver behavior. The goal of this

application is to induce fuel-efficient driving patterns among drivers. However, this

application lacks the social comparison and thus will not appeal to the majority of the

users. Social comparison theory explains that people like to evaluate themselves by

comparing themselves to others [5]. Introducing a social comparison method like

sharing on social media can have a huge impact on the users thus leading to more

efficient and safer driving habits.

The paper by Kuei-Chung Chang and Po-Kai Liu [6] discusses the implementation of

a real-time speed limit sign recognition and warning system for over speeding for

android mobile devices. The resulting application uses image processing to identify

road signs with speed limits and then uses the GPS sensor to warn the user of over

speeding so that they could avoid speeding tickets or dangerous accidents. This is quite

similar to Driving Coach in terms of the output, but what differs is that this uses image

processing to find out speed limits from road signs instead of the speed maps that have

been programmed into the map.

Driver behavior profiling [1] is a research conducted to compare different smartphone

sensors and machine learning algorithms to find the best method to profile driver

behavior. This comprehensive research collected information from many sensors like

accelerometer, linear acceleration, magnetometer, and gyroscope, used them in

machine learning algorithms to classify

1.1.2 Pedestrian crossing detection

Pedestrian crossings play a bigger role in the transport structure than one might

understand, which serves as a safety measure to be practiced by not only the

pedestrians but also the motorists on the road, to keep the traffic flow in an orderly

4

manner. Due to numerous reasons, the pedestrian crossing is one of the most

disregarded aspects by drivers in Sri Lanka, resulting in extremely unfavorable

circumstances to occur such as accidents, injuries (minor and fatal) to the pedestrians

and drivers, damages to the vehicle and undeniably they (driver and the pedestrian)

will be scarred for life.

There is a handful of research conducted based on the detection of pedestrian

crossings, having a vision of keeping pedestrian safety. If the driver can be notified of

an oncoming pedestrian crossing beforehand, then he could take safety measures,

which could drastically reduce accidents that occur due to drivers’ unawareness. Most

of the research done are to develop a guidance system for the visually impaired or

blind, to safely cross the road. One of the most prominent research is done using the

Bipolarity feature of the Pedestrian Crossing by Uddin et al [7], which evaluates the

object in front of the camera to be a crossing based on the even distribution of the black

and white stripes. Similarly taking the bipolarity feature and extending it further in

2009, Cao Y et al [8] conducted another research with the aim of increasing the rate of

accuracy of recognition and to reduce false positives. Another research in 2016 done

by Mascetti S et al [9] focused on helping the visually impaired by extending on their

own algorithm which was published in 2014 [10] which enhanced on the groundwork

for application of this technology in mobile devices. This is specifically engineered to

run using mobile GPUs, also claiming 93% accuracy with no false positives. But is

limited to recognize only the black and white patterns (bipolarity) similar to the

aforementioned research.

Our system sends an early warning signal to the driver of the existence of a pedestrian

crossing, which does not require the images to be taken at a specific angle. But the

mode of gathering data by applying the Mobile Mapping System (MMS) principle

requires us to do so from the drivers’ perspective. Since the video data will be collected

via a vehicle mounted-camera, which deems more efficient than going around stopping

by a pedestrian crossing and recording it in the map database. Since most of the related

research are done from the pedestrians’ perspective to aid in crossing, only very few

research are done from the drivers’ perspective. In 2015 Wang C et al [11] conducted

a research, motivated by the usage of intelligent vehicles based on the Self-Similarity

5

feature. Self-Similarity is the attribute that any selected sub-region of a pedestrian

crossing is similar to each other of the zebra-crossing. They had used Inverse

Perspective mapping, which is proven to be efficient when processing images from a

bird-eye view (based on the processing time of a single frame). Boudet L et al in 2009

[12] directed a research was based on the Evidential fusion of video sensors that uses

the non-dedicated cameras and sensors to be used to collect information regarding

pedestrians and the actual crossings. This was done by exploiting the redundant data

that was being collected by traffic monitoring cameras and sensors but requires access

to traffic cameras which is not an accessory that is readily available. In 2017 [13] an

Impairment analysis of pedestrian crossing research was conducted by Liu X et al

keeping the focus on 3D street reconstruction which can be used by relevant authorities

to monitor the wear and tear of the pedestrian crossings. This proposed system uses

Artificial Neural Networks (ANNs) and Target detection to analyze the status quo

(focusing on flaking paint, defilement, and impairment) of the pedestrian crossings.

1.1.3 Road signboards detection

Road signboards play a significant role in the transport sector. They are designed to

serve both pedestrians and drivers not only to alert or warn them about the journey but,

also to inform them about the road conditions and regulate the common traffic issues

by managing the bottlenecks and to maintain the sign inventory. Every country has its

own road laws and methods implementing them in real scenarios. Reasons for

countries having road laws are as follows:

1. To ensure the safety of the citizens

2. To monitor and manage the traffic

3. To develop the road conditions based on necessity

4. To reduce the issues related to transport (e.g. accidents, damages)

5. To plan and manage the nation's’ expenses on road development and

regulations adhering to the laws

Road signboards are the symbolic representation of many road rules and regulations.

More than stating the road laws in the text, presenting them in the form of road signs

has a huge impact. The reason is that our human brain will be attracted more towards

the pictures, signs, and symbols than the text. The design of a road signboard has

6

numerous factors to be considered. The factors that need to be considered when

designing a road signboard are as follows as per the research paper evaluating the

effects of bilingual traffic signs on driver performance and safety by S. L. Jamson et

al [14]:

1. Should be easily recognizable and prominent even in a difficult scenario (e.g.

Nighttime, rainy day)

2. Should convey the message status at a glance with the aid of colors, the shape

of the board and features (e.g. In Sri Lanka, the danger warning signs are in

yellow background and black object lines)

3. Should be properly placed so that the visibility range should cover a large

number of people in the scene.

4. Should be simple and contain less number of symbols so that it will not take

much time for the audience to comprehend the message.

5. Should not place many signboards in the same location as it will confuse the

people in prioritizing them.

Due to the failure of these reasons and other human mistakes many problems and

discomforts ascended in the field of transport. To overcome these problems many

individuals, organizations and nations came up with different research with the aim of

identifying solutions for them. As a result, many technologies to solve the same

problem was proposed and proven. One of the common technology that handles the

road sign detection is the Image Processing, which is incorporated into many of the

Machine Learning algorithms. Although the field of digital image processing officially

started in the 1960s as per Handbook of Pattern Recognition and Computer Vision

[15], due to the demand of the trending problems, the field is getting updated

frequently. Many algorithms, models, and concepts of digital image processing

evolved in early 2000 with the feasible technologies at that time as discussed in Study

and Comparison of Various Image Edge Detection Techniques by R. Maini et al [16].

Most of the existing road sign detection research mainly concerns about identifying

and classifying the signs. The research, Traffic sign recognition application based on

image processing techniques by R. Laguna et al [17], produced an application that

could process an input image and detects and validate the road signs in the input image.

7

The flow of the research initiates with the image preprocessing by converting the

image into grayscale and applying the Laplacian Of Gaussian (LOG) filter to extract

the edges. This was followed by candidate signs comparison with the existing signs to

qualify the road signs and then classification was done with the aid of cross-correlation

algorithm. Finally presenting the UIs in order to manage the mentioned task. It is clear

that the system doesn’t really handles research problem identified here but follows a

procedure in order to build a system that could identify and classify the road signs.

This research was based on a real-time traffic sign recognition from video by class-

specific discriminative features by A. Ruta et al [18].

Unlike the research by R. Laguna et al [17], both static images and real-time traffic

video are taken in as the source of road signs from which the sign detection was done.

A traditional three-stage framework, which incorporates detection, tracking and

recognition were used to complete this research. For every several frames of the video,

the system will be triggered and captures the candidates qualifying to be a road sign.

This is to minimize the probability of missing a sign from the scene. This initial

procedure will be done by the detector. Discriminative and Novel Image

Representation feature selection algorithms were used on top of the framework for

feature extraction. The tracker predicts the position and scale of the detected sign. In

order to detect the signs, feature extraction is performed using template approach,

where the captured signs are compared with the existing signs. This is done by the

classifier accordance with the of one-vs-all dissimilarity maximization principle which

is based on the Color Distance Transform that enables robust discrete-color image

comparisons. This is useful to overcome the issues regarding the image/video colors,

because during the acquisition of the images/video, depending on the capturing device

being used, the color of the entire scene might differ in research by A. Ruta et al [18].

Another interesting research, Road sign detection and recognition in panoramic images

to generate navigational maps by R. N. Sadekov et al [19] was carried out in this

domain to identify the road signboards. This approach seems to tackle very minimal

aspects of the problem. In this research, the image gathering is done directly from

Google Street View panoramic images. The classification is done based on the Neural

Network. Unlike the other research discussed above, this research has taken the

8

signboards’ locations into consideration. This was possible since they used the Google

Street View panoramic images.

Research in identifying the road signboards are mainly done in paper levels [16, 17].

In practice, there are several applications getting launched to the market in recent

times. Most of these applications use mobile as the platform. Few such applications

are Traffic Sign Detector [20], Road sign Recognition Free [21] and aCoDriver 5 [22].

The Traffic Sign Detector [20] is an android application which was developed based

on Neural Network that detects the road signs with the aid of the mobile camera. The

sign detection is done in real-time and it alerts the driver [20]. A similar app which

does the road sign detection is the Road Sign Recognition Free [21]. This application

works only for speed limit signs with a red circle. So more or less this handles the

speeding problems through detecting the road signboards. The next app to be

considered on this list is the aCoDriver 5 [22]. This application handles the lane

detection, speed alert and informs about the distance to the traffic in front. The speed

alert is accomplished by detecting the road signs and filtering the signboards. This is

done in real-time. This app was developed to detect the signs in few parts of Europe.

All these research, systems and applications provide a source for commencing this

research and show the need to continue this research work and take it to the next level

in both speed and accuracy aspects which will be discussed under the research gap

with clear comparison regarding the existing systems and our research title.

1.1.4 Lane detection

In intelligent vehicle systems, lane detection is one of the most important parts.

Observing lane discipline is an essential safety measure on the road. Paying more

attention to the lane position is essential as there are so many accidents take place.

Everyone wants to move fast so that to avoid getting stuck in traffic. This leads to

carelessness and not to obey the road rules.

Most of the countries face serious traffic problems [23]. Driver inattention results in

roadway departure collisions. In fact, 10% of all road accidents are caused by changing

lanes. The newspaper article by Sarah Muiz [24] has reported that there are 1700 fatal

accidents, 653 bike accidents and 507 involving pedestrians have occurred in Sri

9

Lanka in 2015. The following are the main essential cues in detecting lanes for a smart

driving.

Lane boundaries

Lane boundaries are identified by different shapes and colours. These identifiers may

vary from solid lines through dashed lines. In Sri Lanka, these line marks subject to

change. Some places in our country use white and orange whereas some use yellow.

Lane departure warning

An alert which tries to make a road driving safer and prevent accidents related to a

moving vehicle outside of its lane due to driver inattention. The idea behind this is to

notify a driver in a situation when a vehicle is moving outside of its lane but this

movement is not desired by a driver [25].

The paper by A.B.Hillel et al [26] discusses different models and algorithms that can

be applied to detect lane boundaries. The algorithms were discussed in detail with

procedures on how the lanes can be detected from a video clip. However, these are

done at offline. G.Kaur and D.Kumar [27] have explained the limitations of existing

lane methods in this paper. They have compared the results done using Hough

Transform and H-Maxima Transform.

There are few number of products developed using smartphone video camera and used

for real-time detection. The iOnRoad [28] was developed to run on both Android and

iOS platforms. One of the main functions of this application is lane detection. The

drawback that we figured in this app is that it detects only the solid line and warns the

driver whenever crossing a solid line. The aCoDriver 5 [22] is an Android app

developed to warn the driver when the vehicle tries to cross the lane separator.

However, this application is designed to detect lanes in Europe only.

Making an excellent decision on which lane to follow when there are multiple lanes

on the road is important. Being undecided leads to excess and unnecessary lane switch

which can cause a crash. Following the road rules and keeping lane discipline avoid

collisions on the road. Proper lane usage will reduce road fatalities and assure every

driver a smooth and safe ride.

10

1.2 Research Gap

There has been a constant growth of new technologies, concepts, principles, and

theories in the field of transport and driving. Along with the arrival of the

aforementioned, the number of problems in the field of transport is also getting

increased radically. So, the corresponding solutions for those problems require number

of research. It is evident that one single problem related to the domain can have

multiple different solutions which depend on different aspects such as the geographical

concerns, feasibility of technologies, financial and human resources, laws and

regulations in a country, recent trends in road accidents, social norms and values [30].

The main requirement for pointing out the gap between existing systems, thesis and

principles with the proposed system is to come up with the most feasible solution that

will be implemented through the proposed system and to enhance it for better results

(and to driver satisfaction). Therefore, this part of the document will further elaborate

on each of the components’ research gap. So that all possible causes will be analyzed

to proceed with the research.

1.2.1 Speed and driving style

Many applications exist that try to warn the users of speeds like DriveScribe [2],

however, these applications limit themselves to identifying the legal speed limits using

road signs and alerting the user when they exceed them. While this is handy, speed

limits indicate a maximum speed before a violation rather than safe speeds around

some locations (sections of a route). Thus, the maximum legal speed around a bend

might not be the safe one. Drivers with experience in a route find it easy to handle

these bends since they are used to it and know roughly at what speed they should drive

around these corners.

What we propose in this research is to build a system that provides the user with

average speeds around sites, so that the driver would reduce their speeds to match the

behavior of the majority of drivers and also to feel safe of their speed. The average

speed around the site is obtained by processing the driving information fetched from

the drivers using their mobile phones with their consent.

11

Features\Apps Existing Research / Products Drive Assist

Driving Coach

[2]

Trapster [3]

Real-time speed detection and

warning system [6]

Driver behavior

profiling [1]

Speed Limit warning based on road signs

X - X - X

Geo-location-based alerts

- X - - X

Social Media Integration

- X - - X

Classification of driving behavior.

- - - X X

Average speed warning

- - - - X

Table 1 - Comparison of driver speed related applications

This system would also compare the driving behavior of the user to the average driving

behavior and give a feedback score on how close they have been to the average speeds

around sites.


A multitude of research have been taken place in terms of identifying pedestrian

crossings, most of them are done from the actual pedestrian’s point-of-view and a very

limited amount from the drivers’ view. Most of the research, if not all of have limited

themselves as research and not applied themselves to be used in the real world, except

for a few driver assistance systems and the world of autonomous vehicles. While some

of them are accessible, most are proprietary making it reachable only by those who

can afford such luxury. Our aim being, enabling this technology to be used by regular

commuters by their readily available mobile devices makes us stand apart from the

rest. From the research view, each of them uses majorly one or two aspects of the

12

pedestrian crossing and produce a result but not a definitive solution that can be used

on a daily basis.

From the research view, each of them uses majorly one or two aspects of the pedestrian

crossing and produce a result but not a definitive solution that can be used habitually.

Usage of ANNs result in a driver assistance system requires huge datasets and trained

models else train for months before being able to use on roads. Majority of the research

are based on images captured from the pedestrians’ point of view, making it difficult

to process and if subjected to impairment the algorithm simply provides a false

negative. Since none of the research are implemented as a fully-fledged system, they

cannot alert the driver (except in the autonomous vehicles) and do not make use of the

geospatial data. All are aimed at identifying only the zebra crossing, taking advantage

of their bipolarity feature.

Feature/Research Existing Research/ Products Drive Assist

Bipolarity feature [8]

ZebraRecognizer [9]

Self-Similarity

Feature [11]

Impairment Analysis [13]

Identify zebra crossing

X X X X X

Identify partially visible crossings

X - - - X

Identify at various lightings

- X - X X

Identify impaired crossing

- - - X X

Identify puffin crossing

- - - - X

Identify multiple crossings at once

- - - X -

Process geospatial video data

- - - - X

Table 2 - Comparison of Pedestrian Crossing Detection System

13


Numerous readings and experiments have been/are taking place in terms of identifying

the road signboards both in real-time and offline mode. Each research follows and

tackles different approaches to identify the road signboards. The major approaches are

based on Image Processing and Machine Learning. However, there is no one definite

solution or methodology to adhere to follow for the detection of signboards. The need

for road sign detection in this research is analyzed through the research gap that is to

be identified in this part of the document, by comparing this research title with the

existing systems and research.

The main similarity in almost all the research, is the procedure followed to identify the

road signs. The common practice would be the acquisition of images, image

preprocessing, edge and color detection, and finally the presentation of the result of

[17, 21]. However, almost all the studied research only handles the road sign detection

part but no other driving assistant features or aspects attach to them. But the proposed

system handles four different components such as detection of road signs, pedestrian

crossing, speed information and lane separators that could assist the driver while

driving. Therefore, the main difference between the proposed Drive Assistant and the

existing research is the diversity of features they handle.

The main difference in research by R. Laguna et al [17], compared to the proposed

system is it handles only the static images in offline mode. But in reality, capturing the

entire journey as a video clip is far convenient and accurate than taking pictures. The

main reason for such approach is that the possibilities of missing some important sign

candidates will be reduced in video clips. Also, the outcome of it is a set of Graphical

User Interface (GUIs) which will allow the user to input an image, visualize it, identify

the interest regions, get the histogram and detect the signs. So, there is nothing beyond

the identification of the signboards. This system doesn’t support the user while driving.

But the research topic proposed in this document, incorporates more features and

aspects such as taking video clips of the journey as the input for the system, attaching

the GPS coordinates for the recognized road signs, periodically fetching the vehicle’s

GPS data, using voice assistance to alert the users about the oncoming road signs

within the minimum range of distance as per research by R. Laguna et al [17].

14

In the research by A. Ruta et al [18], is more concerned about the real-time traffic

video from which the signs are extracted. This also functions as an app that helps to

identify the road signs but not while driving. This particular app uses various types of

algorithms and mechanisms such as Novel Image Representation, Discriminative

feature selection algorithms, one-vs-all dissimilarity maximization principle. Since

this app deals only with the road sign detection and doesn’t retrieve or fetch data in

real-time, utilizing so many algorithms won’t show any latency issues. But the

proposed application is a combination of four prominent components. Therefore, using

so many heavy computational algorithms, mechanisms and principal will slow down

the system and due to one component’s heavy usage of algorithms might affect another

component’s outcome.

The main similarity between the proposed system and research by R. N. Sadekov et al

[19] is that both are concerned about finding the exact location of an identified

signboard. But the procedure that has been followed in it is more of a static approach;

that is to get the location coordinates directly from the Google Street View panoramic

images. One of the main drawback is that the image quality will not meet the expected

quality requirements therefore the accuracy of the result might be less. In case if there

is a change in the placement of the signboards, the updated information will not be

incorporated with the system.

According to research by R. Laguna et al [17], the average success rate is 91.07%,

based on the number of traffic signs that the image contains. In research by A. Ruta et

al [18], the obtained classification error rate does not exceed more than 9%, which

shows the accuracy of classifying the signs. This error rate was calculated mainly

based on the performance of the classifier generated based on the exhaustive image

comparison (as though no region selection was performed), and image comparison

with respect to the sign representations obtained for the optimal setting. In Traffic Sign

Detector [20] the output is in the form of voice and image of road signboards but leads

to wrong identification of boards. Rather than prompting the correct oncoming

signboard, the app keeps on repeating all the possible signs, which will distract the

driver.

15

The following table shows the comparison between the existing road signboard

detection systems with the proposed research.

Feature/Research Existing Products / Research Drive Assist

Traffic Sign Detector

[20]

Road sign Recognition

Free [21]

aCoDriver 5 [22]

Traffic sign recognition application based on

image processing

techniques by R. Laguna et

al [17]

Identify road signboards

X X X X X

Process geospatial embedded video data

- - - - X

Alert the driver of oncoming danger warnings and mandatory signs

- - X - X

Takes journey video as the input

X X X - Only static

images

X

Prioritize the outcomes of each components

- - - Has lane detection compone

nt

- X

Output in a form of voice with sign meaning and distance to reach the board

-

- - Only the

sign meaning

- X

Real-time road sign detection

X X X - -

Table 3 - Comparison of Road Sign Detection Applications

16


First versions of Lane Departure Warning (LDW) systems were introduced by

Japanese car manufacturers. Later in 2007-2009 Audi and BMW also introduced LDW

system with sound, vibration and visual notification of a driver but not intervene in

driving [25]. Many research works have proved that lane detection understanding

could be done in many ways. These progress works have used several sensing

modalities, including video camera, Light Detection and Ranging, stereo and

information obtained through GPS [26]. Sensing through video camera is the most

common path in lane detection.

Even though, there are existing products like iOnRoad and aCoDriver 5, they limit

themselves in showing some important features. In most of the roads, lane separators

are drawn in dashed lines and lane boundaries are drawn in solid lines. Moreover, the

paint of the lines will be faded and detecting lanes would not be an easy task. Some

researchers have explained on how to detect lanes from a video clip which is not done

at real-time. Although there are some systems which is used for lane detection but

identifies only the straight lines.

In our research, we are going to implement a real-time lane detection system which is

varied from other researches.

1. Our system identifies the lane boundaries whereas other research works use a

constant width for lane. Consequently, it controls all types of lanes like

merging and splitting lanes

2. Our system incorporates both lane detection and tracking into one structure that

can efficiently handle lane changes

17

Refer the Appendix for the Summarized Version of the Comparisons Table

Features/ Apps Existing Products / Research Drive Assist iOnRoad

[28] aCoDriver

5 [22] Adjacent Lane Detection and

Lateral Vehicle Distance

Measurement [23]

Recent Progress in Road

and Lane Detection

[26] Identify solid lines

X X X X X

Identify dashed lines

- X X - X

Identify lines at night time or low light

X - - X X

Warn driver with both audio and visual alerts

X Only audio

- Only visual

X

Real-time lane detection

X X - - X

Table 4 - Comparison of Lane Detection and related Application

18

1.3 Research Problem

The emerging problem with the current scenario is the uncertain traveling experience.

Whenever a person is on a road journey, despite the regulations on road, the

discomforting experiences seem to exist and increase. Since factors related to a road

journey such as vehicle speed, traffic, crossroads that are not visible, routes taken to

reach the destination and weather are all highly dynamic; proposing one general

solution for it, is impractical. So, it is implicit that no one solution or system could

handle all the probable problems a driver could face during driving.

Though there are various issues related to traveling and driving, this research targets

only on the main common issues that any driver faces recurrently during the phase of

a journey.

1. Difficulties in identifying the road signboards especially during bad weather

conditions, night times and also when they are covered due to some obstacles.

2. Complications in detecting the pedestrian crossing when no signboards are

placed to indicate them, when they are faded or when driving fast.

3. Predicting the optimal speed with no previous experience, knowing the average

speeds at roads.

4. Being aware when crossing the lane separators unexpectedly in the middle of

the journey.

The comprehensive description of the mentioned problems are as follows:

1.3.1 Problems in detecting speed information and driving patterns

Driving consists of many routine tasks that the user needs to perform. One of these is

to guess the speed at different points in the route. The driver must make split-second

decisions to slow down when they approach an unsafe bend in the road. This requires

utmost concentration from the user, especially when new to the route. We can notice

that a driver with experience in a route can drive that route faster than a new driver

since he/she knows all the tricky parts of the road and the speeds to get around them

efficiently.

Drivers adhering to safe speeds should be rewarded. In order to achieve this, we need

to classify the driving behavior and score the drivers on some scale. Social comparison

19

is a powerful tool and competing with others for the best driving style will improve

the safety and efficiency of journeys and help reduce overall traffic accidents.

This component of the research focuses on providing the drivers with average speeds

throughout the journey so that they can adjust their speed to match with the majority

and investigate how this improves the driving experience. It will also focus towards

algorithms that can be used to classify or rate driving behaviors using the data collected

from the user.

1.3.2 Hurdles in identifying pedestrian crossings

This part of the problem definition concerns with the complications with regard to

recognizing pedestrian crossings by the driver, primarily during night times, heavy

traffic, when the paint is faded or when merely driving hastily. The absence of

compliance with pedestrian rights has been one of the major culprits for this problem

with ignorance and lack of common sense playing a parallel part in the act. Some of

these acts could and at oftentimes result in accidents where the pedestrians get gravely

injured, bedridden or killed, and on the other hand, the driver faces a situation where

he is arrested, detained or fined and almost always is scarred for life.

Every driver who passes through the driving exams and receives the license has the

undeniable basic knowledge of how to conduct around pedestrian crossings, yet the

Sri Lankan police statistical reports taken through 2006 – 2015 [29] show that nearly

44.24% of the road users who are fatally injured or killed are drivers/riders and

followed by pedestrians at a staggering 32.66%.

Some of the identified issues that act as a blocker between the driver and his senses to

identify an oncoming pedestrian crossing:

1. At night times crossings have a lower visibility

2. During traffic, vehicles block or be over a crossing

3. The paint of the crossing on the road may be faded

4. Driving at higher speeds and ignoring the reality that anyone could leap in at

any moment

5. The driver is distracted (personal issues, children in car, traffic, stress and

more)

20

6. The driver is expecting a road signboard to indicate an oncoming pedestrian

crossing, but it is unavailable or misplaced

7. At heavily raining times

It is undeniable that educating people of the rules and regulations regarding pedestrian

crossings and pedestrian rights is crucial but, this requires a different form of media to

convey the message and to enforce the laws which require a vast support from the

general public and the government. Even then, the irregularities with the physical

pedestrian crossings are to be amended by the authorities and cannot be easily fixed

by common man. Some high-end vehicles have tackled similar problems using

specialized hardware, but the intent of this research part is to solve this problem for

the whole of general commuters.

1.3.3 Difficulties in identifying the road signboards

This part of the document describes the difficulties in identifying the road signboards,

especially during bad weather conditions, night times and also when they are covered

due to some obstacles [34]. The importance of road signboards is tremendous when it

comes to driving. Therefore, it is a must that the signboards are to be displayed to the

driver providing visibility even at a longer range without any distractions. A driver is

placed in a position where he/she might not notice the road signboards due to various

reasons. The difficulties in identifying the road signboards can be analyzed in 2

different aspects [34, 35].

1. The ignorance of drivers in following the road rules

To discuss the ignorance of drivers in the Sri Lankan context, factors mentioned below

have a very prominent impact.

Carelessness

Aggressive driving

Drunken drive

Usage of mobile phones while driving

Personal attitude

21

Though there are laws prevailing in terms of driving, most of the laws are only on

papers and not practiced properly. This is one aspect of the drivers’ ignorance in terms

of following road signs which the driver should take measures to safeguard him/herself

during a journey as well as to get a pleasant driving experience.

2. The problems related to road signboards

The next aspect under difficulties in identifying the road signboards is the problems

related to road signboards. Some of the common reasons for such problems are as

follows:

Placing the signs in wrong places where people won’t be able to pay their

attention

Placing too many signboards in the same place

People tend to lose their tolerance in reading the signboard with text in it

Presenting the road sign in both symbol and text [36]

Obstacles such as bigger vehicles, trees and other advertisement boards

covering the signboards

This research topic mainly focuses on the second aspect of the issue regarding

identifying the road signboards. The idea behind this concern is to analyze the need

for providing accurate details of road signs and their locations, to amend the existing

loopholes in the field of road rules and regulations and to stress on the importance of

following them.

1.3.4 Obstacles in identifying lane separators

This section focuses on the issues in detecting the lanes. Most of the accidents in the

world take place due to driver inattention. The main causes of collisions can be driver

error, drowsiness, impaired driving due alcohol or distractions due to a phone call or

tuning the radio [32].

Driver inattention can influence lane-keeping ability. Improper detection of lanes leads

to a collision. Sometimes, it is very easy to detect lanes with the human eyes but in

some conditions due to the following external effects, it is difficult to identify the

separators and boundaries on the road [33].

22

Road lines that in bad condition

Different shapes of lanes

Low light areas

Moving vehicles

Merging and splitting lanes

Pedestrian crossing and other writings on the road

Various types of slopes

Shadows projected from tall buildings and trees

Lane detection plays a vital role in accidents. Concentrating on all the above factors is

a tremendous task to the driver. It is impossible to focus both on driving and

environmental factors. Therefore, this research is more concerned with the important

factors in detecting lanes and avoid accidents by using improved driving assistance.

This lead warning the driver and saving a considerable number of lives.

23

2 OBJECTIVES

2.1 Main Objectives

To implement a system that could assist the driver in common tasks such as looking

out for road signs, keeping an eye for pedestrians, assessing a safe speed and keeping

track of lanes. All these could be consumed by the user equipped with a smartphone.

2.2 Specific Objectives

To implement a mobile app that can collect real-time speed information from

users and their corresponding geo-location to be used for analysis.

To identify algorithms and machine learning techniques to analyze the raw

driving data and classify driving styles.

While driving, communicate optimal speeds at current location to the users

based on the previous data.

Provide the information to the driver in a non-distracting manner using speech

and display.

Collect driving information such as the tendency to accelerate, average speed

etc. to build a driver profile and use it for social comparison to encourage a

positive driving behavior.

To identify lane boundaries and separators to detect the lanes on the road and

alert the driver when he departs from his lane.

To alert the drivers on the upcoming danger warning and mandatory road

signboards at least before 50-75 m of distance

To forewarn the driver of oncoming pedestrian crossing, so he/she could take

caution when nearing the area.

2.3 Research Questions

How to collect speed information from the users in an efficient manner?

Finding optimal machine learning algorithms to classify driving styles using

raw data.

Calculating optimal speeds at certain coordinates using the raw data collected.

How to find the best algorithm for identifying the lane?

How to implement it in real-time?

24

How to fetch road signboard information from DB based on vehicle’s current

geolocation in real-time?

How to prioritize the output from each research components and present it in

voice?

How to fragment the video in a way to incorporate the GPS coordinates?

How to identify the frames that contain road signboards?

How to notify the driver about the oncoming road signboards without

overlapping the google map voice assistance?

25

3 METHODOLOGY

This system is planned to be an end to end system that consists of an android

application, server to provide data and handle processing of raw data.

Since the project consists of four core components, the methodology for each

component would be discussed separately in detail.

No. Component Description Research Area

1 Analyzing driving pattern and classifying Machine Learning (ML)

2 Pedestrian crossing identification Image Processing (IP)

3 Road signboards detection Image Processing (IP)

4 Lane separation detection Image Processing (IP)

Table 5 - Components and corresponding research area

3.1 Data Gathering

From the above list, pedestrian crossing detection and road signboards detection

requires clear video footage to analyze and extract the information at specific

locations. Since these footages will be of huge chunks of data, capturing and uploading

it to our servers to process will be done by a dedicated team, applying the MMS

practice. Since this process requires vast amounts of computing power, which is not

easily available on a mobile device, considering the limited RAM and battery. For the

real-time user alerts, the driver’s current location must be updated in a given time

interval so that the alerts of oncoming pedestrian crossings and signboards can be

passed on. This will require very minimal data usage for the driver, considering the Sri

Lankan market.

Each Component’s process is shown as Block diagrams below, please refer the

appendix for symbols and meaning.

26

Figure 1 - High-Level Architecture

3.2 Driving Style

To collect the driving information from the driver, an android application can be used.

GPS, accelerometer, data connectivity and other sensors can be used to fetch critical

information such as geolocation, acceleration, speed, and bearing. This information is

then posted to the server. The application will not send information each time the

geolocation information is received, instead, it will collect the data into a temporary

file and send them as a file. This will reduce the load on the device which is critical in

a resource-constrained environment that is a smartphone.

27

Figure 2 - Procedure of Driving Style Analyzer

The data collected from the users can be used to build a database. This has to be a

relational database since we need to maintain the relationship between drivers and their

speeds at certain locations. Some relational databases like MySQL could be used for

this purpose.

The data collected in the database can then be used to calculate safe speeds at relevant

coordinates. We could average out the speeds of all the users at a given coordinate P

and store this information in the database. When doing these calculations, we should

also consider outliers in order to avoid over speeding drivers and slow drivers.

The speed information collected through the methodology which is stored in the

database is then serviced to an android application through web services. The app

frequently updates itself corresponding to the current coordinate.

The driving information collected can be used to build a profile of drivers. These

profiles could be used by machine learning algorithms to classify the driving behavior

of drivers.

28

3.3 Pedestrian Crossing Detection

Forewarning the driver of an oncoming pedestrian crossing, will be handled through

our system in a non-disruptive yet informative way.

Figure 3 - Procedure of Pedestrian Crossing Detection and Presentation

The pre-collected video clips of a certain journey are broken down into sets of frames,

which are then preprocessed in the server (cloud) after uploading. These will be fed

into some selected image processing algorithm and techniques which includes image

enhancement, object segmentation and feature extraction which will then identify the

pedestrian crossing from the given frames. This information about the pedestrian

crossing are saved in the central database with the coordinates.

Due to some reasons, like uploading pre-processed video clips might cut of layers

which may deem unnecessary (due to compression) on the device which is used to

record the video, the pedestrian crossing might be not completely visible or might have

lost some information and due to the paint of the crossing might be faded, thus needs

to be reconstructed to confirm the availability and produce an accurate result.

29

To identify the existence of a pedestrian crossing, initially to enhance the image to

work further, some general spatial preprocessing techniques need to be applied such

as Mean filter [31] for noise reduction and Histogram Equalization [31] for contrast

correction, likewise for image segmentation, techniques like the Sobel operator [31]

and Canny Edge detector [31] algorithms can be used. Since there are two color

schemes being used for the pedestrian crossings in Sri Lanka, yellow and white, the

latter being introduced recently, both color schemes must be accommodated in order

to provide with a precise information.

Once the processing is completed, the information of where the pedestrian crossings

are located is saved in the central database which will alert the driver of the oncoming

crossings beforehand when the driver is in about a 50m range of the crossing, as a

voice guide.

3.4 Road Signboards Detection

In order to alert the drivers about the oncoming road signboards the following

procedures will take place.

Figure 4 - Procedure of Road Signboard Detection and Presentation

30

The procedure of identifying road signboards has two main parts.

1. Capturing the video and processing it to collect the possible road signs with

corresponding GPS coordinates

2. Fetching the GPS of the vehicle and delivering the result to the user in a form

of a mobile app

Initially, the video clip of the journey (from the start point to the destination) will be

recorded and fed into the system. This video will be embedded with the GPS

coordinates in order to identify the exact location of the road signboards. Next would

be to split the video clip into frames so that identifying the road signboards could be

achieved.

The preprocessing procedures can be triggered once the source images are prepared.

Image Preprocessing is essential at this stage because the quality of the video can vary

from place to place depending on the video captured time and other environmental

distractions. Therefore, to produce accurate results, enhancing the input images is very

important. In order to remove image noise, enhance the contrast and equalize

illumination several techniques such as Median Filtering Algorithm, Contrast

stretching technique can be used. Once the image has been preprocessed, with the aid

of algorithms such as Laplacian of Gaussian and Canny’s Edge Detection Algorithm

the signboard objects can be identified.

From the identified road signboards, the danger warning and mandatory signboards

will be taken into consideration since they fall into the target criteria. The main reason

for considering only these two road sign criteria is because there are lot many boards

available on road to indicate the road signs. So, if the system is to alert the driver for

each and every sign then that will result in deviating the driver’s concentration. So,

considering this fact the proposed system will mainly target the mentioned road sign

categories. The available road signs in Sri Lanka will be stored in the database (DB).

Once the captured signs are ready, they can be compared with the existing sign patterns

in the store using algorithm such as cross-correlation algorithms and qualified signs

will be stored in the DB. All these procedures will take place in the background.

31

Once the driver starts using the proposed system, in real-time the vehicle’s GPS

coordinates will be captured recurrently and the oncoming road signboards will be

notified to the driver in advanced within the distance of 50-75m through a voice

command. This entire process is done in cloud platform and the outcome of it will be

delivered to the user as an app that uses mobile platform. This methodology will help

the drivers to continue their journey with less trouble and even if there are obstacles

blocking the signs, the driver will still be informed about them.

3.5 Lane Detection

Lane detection algorithms are implemented to identify lane boundaries and separators

so that to calculate the position of the vehicle on the road. The following figure shows

the real-time lane tracking process.

Figure 5 - Procedure of Lane Detection and Presentation

Vision-based approach is used for video acquisition. In this project, we will be using

Android phone camera instead of an external camera to capture the video stream to be

used as an input to the system. Then, the size of the image frame is reduced to limit

the enormous space of frames so that facing extremely poor frames per second (fps)

rate is avoided. This is done by a concept called Scale Space [32]

Converting the image to Grayscale will be done to make the process faster and less

computations. Next, a technique called Region of Interest (ROI) is applied to help in

segmentation of the space and remove all the areas that do not contain any lane

markings. Further, Canny Edge Detection and Hough Transform [32] methods are

implied to identify the edges and the lines of the lanes. Using these techniques in real-

time, driver will get a warning within 100ms if changed lane unexpectedly.

32

3.6 Testing

Accuracy and speed in identifying road signs have very significant roles. Many other

research have followed different techniques to calculate and validate the system

accuracy level and speed. For the proposed component, road sign detection the

accuracy level and speed of processing output will be determined as follows:

3.6.1 Application performance and accuracy

Existing information for the road signs, pedestrian crossings and average speed will

be fetched from the DB based on the app user’s current location.

Speed

The speed of the system depends on how fast the GPS coordinates are being

sent to the DB and in return how long it takes to fetch the details from it. So,

the relationship between the total time taken to fetch GPS from the vehicle and

alert the user will be inversely proportional to the system’s speed.

Accuracy

Loading data from real-time while the vehicle is moving is tricky. If the

previous request was not finished the next coordinates will be missed and won’t

request for information. We can measure the number of coordinates that the

request was not made as the basis for the accuracy of the application and to try

reducing it using bulk requests by intelligently anticipating oncoming

coordinates.

3.6.2 Driving style

Speed

Algorithms to calculate speed and classifying driver information using the raw

data must be efficient in order to handle a large number of transactions. We

need to test the performance of different algorithms to find the most efficient

algorithms since these are computationally heavy tasks. A factor here could be

the time that the user ends the journey and the time to classify his driving style.

Accuracy

Predicting average speeds requires high accuracy since an incorrect speed can

misguide the user and lead to accidents or speed tickets. To test the accuracy

33

of the predicted speed we can compare it with the speed limit to know if it falls

within the range.


Speed

Detecting pedestrian crossings from raw video footage must be efficient due

reasons such as large number of frames being processed, processing will be done

in the cloud (costs more) and this will be presented to drivers to forewarn them of

the oncoming crossings. This can be achieved by using many of the available

image processing algorithms, and to conform to one algorithm, the process must

be tested against each. Also, comparisons between previous researches can deem

to show the speed in this program.

Accuracy

Outlining the accuracy of this functionality is considered important due to the fact

that this is to alert the driver of approaching pedestrian crossings so he/she could

take necessary actions to safely maneuver the road. Accuracy can be measured by

running the algorithm for a set of images and obtain the positive hits rate i.e. the

number of pedestrian crossings successfully identified out of the given set.


Speed

The input video will be sent to the cloud and get processed. Once the GPS

coordinates are attached to the identified road signs, the validated details will be

stored in the DB. These methods are handled by algorithms in the field of Image

Processing. Each process will be tested against with the possible algorithms.

Depending on the algorithm’s complexity the speed will differ. This will be the

key factor to calculate the speed of the stated procedure.

Accuracy

Accuracy in road sign detection will be affected by various factors such as the

video quality, number of captured signs, number of possible road signs and GPS

coordinates. Each sign image’s matching percentage with the original image in the

DB will be calculated on pixel-by-pixel basis.

34

So more the accurate rate for an image comparison will contribute to the system

accuracy level as a whole. Also, in the proposed methodology of detecting road

signboards the source of the input video is taken by a dedicated team with higher

frame rate camera. So, the video quality will add on to the accuracy level of the

output. Meanwhile, the possibilities of missing a sign candidate will be at its

lowest. This can be validated by getting the road signboards’ count in a specific

route manually and checking the output of the identified signboard count by the

proposed system.


Speed

Lane detection is done in real-time. The preferred method to test how long the

system takes to alert the driver when he departs lane could be shown as follows:

o Use the application for ten trips

o Cross the lane mark at each trip and calculate the time that it takes to

alert the driver

o Find the percentage of the speed from the ten trips

Accuracy

Accuracy also can be measured by the above methodology. Out of the ten trips

taken, the number of times alert message is given to the driver when the vehicle

has departed from the lane can be counted. The final percentage could be

calculated and test for accuracy would be made.

3.6.6 User Testing

Our vision is to provide regular commuters with a better driving experience. But,

“better” is not quantifiable and from a product level looking from an overview, we

cannot show how our system can help the drivers in achieving a better experience.

This can be achieved via user research and testing. To get the before and after of using

our system, we need to have a fixed target user group from the beginning, for which

we have decided to bring together a small group of drivers from Colombo since one

major route will be covered for the purpose of research. Primary user research will be

done by providing questionnaires to gather related information for the major four

components of the system. Periodically we will be giving our product to the users to

35

test and then compare their reaction and timings for certain driving-related tasks, with

and without the usage of our system. This can be collected as videos of the same driver,

driving on the same route under these two conditions. Ideas for improvement and

feedback can be composed to finally make this research justifiable.

3.7 Gantt Chart

Figure 6 - Gantt Chart

Refer the Appendix for the Work Breakdown Structure

36

4 BUSINESS PLAN

There are existing systems such that specifically embedded systems in high-end

vehicles and a few systems that run detached, which costs a great deal of capital or

simply are not providing enough with regards to the driving experience. Our proposed

system is a fully integrated driver assistance system that will guide the driver though

out an entire journey as if a knowledgeable friend is sitting next to you. The user

benefit is that every regular commuter with access to a smartphone will be able to

experience the high-end driver experience. Some features of the application require a

large amount of user data to work. Collecting this data requires an investment in the

early stages of the product.

1. Create a data collection team and equip them with capable devices to record

the information from them.

2. Collect data from users in a round robin manner so that one user does not have

to bear the data cost.

Since the user must bear the cost of data, they need an enticing reason to get into the

platform. Because of their availability right away, classification of driving behavior

and lane detection can be the enticing features that can get users into the platform to

build a larger user group. Therefore, the application is distributed for free including

these features. This will enable us to collect enough data from the user group that can

be used in pedestrian crossing detection and road sign detection. We could also reward

users in terms of in app credit or additional features if they are willing to provide data.

We plan to release this to the general public mainly targeting the regular commuters.

We are selling the experience, the luxury that was only accessible to the higher end

vehicle owners. This product will be deployed to the Android world at a considerable

low price to attract users from Sri Lanka. When we reach about a 10% to 20% of the

Sri Lankan population, making an identity for ourselves, we plan to come in terms to

an agreement with an Insurance company, where the product could be used to enhance

their customer experience by introducing a possible reward system. Through the

above-mentioned strategies, we will be able to capture the oligopoly market in Sri

Lanka create a brand for our product.

37

5 DESCRIPTION OF PERSONNEL AND FACILITIES

Member Component Task

M. R. Aaquiff Ahnaff - IT15048738

Collection of driving information and classification of driving behaviors.

Acquiring driving information through a mobile application.

Classify driving behavior using machine learning algorithms.

Build average speed map to alert users of them while driving.

M. M. M. Munsif - IT15015754

Recognition of pedestrian crossings and forewarn the driver to take safety measures

Acquiring the video clip embedded with geolocation data

Preprocess the images Recognize pedestrian crossings Embed the location of each

pedestrian crossing in the DB Warn the driver when he nears a

crossing, by tracking his position

M. Saranki - IT15101266

Identification of road signboards and alerting the driving about the oncoming signboards

Acquisition of video with geographic information

Preprocessing the images Signboards identification Signboards filtering Vehicle GPS fetching Alerting the driver about the

oncoming signboards

B. Kiruthiga - IT15135308

Identify lane boundaries, detect lanes and warn drivers when departing from the track

Get the video as an input to the system

Identify lane boundaries and separators

Track the position of the vehicle Warn the driver when departing

lane

Table 6 - Description of Personnel and Facilities

38

6 BUDGET AND BUDGET JUSTIFICATION

Item Need Cost (Rs.) Mobile Phone An android mobile phone is

required to test the application usage including the real-time lane detection.

15000

AWS SageMaker Required for classification of driving behaviors.

7.22(SLR/Hour) * 500 = 3610

AWS Recognition Required libraries for image processing to analyze the video for road signs and pedestrian crossings

15(SLR/Minute) * 240 = 3600

Total 22210 Table 7 - Table of Budget and Justification

39

7 REFERENCE LIST

[1] J. Ferreira, E. Carvalho, B. Ferreira, C. de Souza, Y. Suhara, A. Pentland and G.

Pessin, "Driver behavior profiling: An investigation with different smartphone sensors

and machine learning", PLOS ONE, vol. 12, no. 4, p. e0174959, 2017.

[2] R. Scott, "Review: Drivescribe, The App That Will Improve Your Driving (Video)

– TechGuySmartBuyTechGuySmartBuy", TechGuySmartBuy, 2018. [Online].

Available: http://techguysmartbuy.com/2014/04/review-drivescribe-the-app-that-

will-improve-your-driving-video.html. [Accessed: 12- Feb- 2018].

[3] U. SL, "Trapster (Android)", Uptodown.com, 2018. [Online]. Available:

https://trapster.en.uptodown.com/android. [Accessed: 08- Mar- 2018].

[4] R. Araújo, Â. Igreja, R. de Castro and R. Araújo, "Driving Coach: a Smartphone

Application to Evaluate Driving Efficient Patterns", Intelligent Vehicles Symposium,

2018.

[5] C. Gruder, "Determinants of social comparison choices", Journal of Experimental

Social Psychology, vol. 7, no. 5, pp. 473-489, 1971.

[6] K. Chang and P. Liu, "Design of real-time speed limit sign recognition and over-

speed warning system on mobile device", 2015 International Conference on Consumer

Electronics-Taiwan (ICCE-TW), pp. 43, 44, 2015.

[7] M. S. Uddin and T. Shioyama, “Detection of Pedestrian Crossing Using Bipolarity

Feature—An Image-Based Technique,” IEEE Trans. Intell. Transp. Syst., vol. 6, no.

4, pp. 439–445, Dec. 2005.

[8] Y. Cao, L. Chen, and S. Jia, “An Image Based Detection of Pedestrian Crossing,”

in 2009 2nd International Congress on Image and Signal Processing, 2009, pp. 1–5.

[9] S. Mascetti, D. Ahmetovic, A. Gerino, and C. Bernareggi, “ZebraRecognizer:

Pedestrian crossing recognition for people with visual impairment or blindness,”

Pattern Recognit., vol. 60, pp. 405–419, Dec. 2016.

40

[10] D. Ahmetovic, C. Bernareggi, A. Gerino, and S. Mascetti, “ZebraRecognizer:

Efficient and Precise Localization of Pedestrian Crossings,” in 2014 22nd

International Conference on Pattern Recognition, 2014, pp. 2566–2571.

[11] C. Wang, C. Zhao, and H. Wang, “Self-Similarity based Zebra-crossing Detection

for Intelligent Vehicle,” Open Autom. Control Syst. J., vol. 7, pp. 974–986, 2015.

[12] L. Boudet and S. Midenet, “Pedestrian crossing detection based on evidential

fusion of video-sensors,” Transp. Res. Part C Emerg. Technol., vol. 17, no. 5, pp. 484–

497, Oct. 2009.

[13] X. Liu, Y. Zhang, and Q. Li, “AUTOMATIC PEDESTRIAN CROSSING

DETECTION AND IMPAIRMENT ANALYSIS BASED ON MOBILE MAPPING

SYSTEM,” ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci., vol. IV-2/W4, pp.

251–258, Sep. 2017.

[14] S. L. Jamson, F. N. Tate, and A. H. Jamson, “Evaluating the effects of bilingual

traffic signs on driver performance and safety,” Ergonomics, 2005.

[15] Handbook of Pattern Recognition and Computer Vision. 2009.

[16] R. Maini and H. Aggarwal, “Study and Comparison of Various Image Edge

Detection Techniques,” Himanshu Aggarwal Int. J. Image Process., no. 31.

[17] R. Laguna, R. Barrientos, L. Felipe Blázquez, and L. J. Miguel, “Traffic sign

recognition application based on image processing techniques.”

[18] A. Ruta, Y. Li, and X. Liu, “Real-time traffic sign recognition from video by

class-specific discriminative features,” Pattern Recognit., 2010.

[19] R. N. Sadekov, K. A. Asatryan, V. E. Prun, V. V Postnikov, F. G. Kirdyashov,

and M. R. Koren, “ROAD SIGN DETECTION AND RECOGNITION IN

PANORAMIC IMAGES TO GENERATE NAVIGATIONAL MAPS.”

[20] "Play.google.com", 2018. [Online]. Available:

https://play.google.com/store/apps/details?id=pl.jesmanczyk.android.driverassistant

&hl=en. [Accessed: 27- Feb- 2018].

41


https://play.google.com/store/apps/details?id=road.signs.recognition.free&hl=en.

[Accessed: 17- Mar- 2018].


https://play.google.com/store/apps/details?id=com.evotegra.aCoDriver. [Accessed:

09- Mar- 2018].

[23] C. F. Wu, C. J. Lin, H. Y. Lin and H. Chung, “Adjacent Lane Detection and

Lateral Vehicle Distance Measurement Using Vision-Based Neuro-Fuzzy

Approaches”, Journal of Applied Research and Technology, vol. 11, Apr., pp. 251-

258, 2013

[24] "Road accidents: from BAD to WORSE", Dailymirror.lk, 2015. [Online].

Available: http://www.dailymirror.lk/87477/road-accidents-from-bad-to-worse.


[25] Akson, “Lane Departure Warning” Nov., 2013 [Online]. Available:

https://github.com/Akson/LaneDepartureWarning, [Accessed Feb. 16 2018].

[26] A.B.Hillel, R.Lerner, D.Levi and G.Raz, “Recent Progress in Road and Lane

Detection - A survey”, Machine Vision and Applications, vol. 25, Apr. , pp. 727 - 745,

2014

[27] G.Kaur and D.Kumar, “Lane Detection Techniques: A Review”, International

Journal of Computer Applications, vol. 112, no.12, Feb., pp. 975-8887, 2015

[28] iOnRoad Augmented Driving Lite. iOnRoad, 2011.

[29] "Traffic Statistics - Fatalities and No", Police.lk, 2018. [Online]. Available:

https://www.police.lk/index.php/traffic-police/112-traffic-statistics-fatalities-and-no.


[30] Strengthening road safety legislation. Geneva 27 Switzerland: World Health

Organization, 2013.

42

[31] "Image Processing Operator Worksheets", Homepages.inf.ed.ac.uk, 2003.

[Online]. Available: https://homepages.inf.ed.ac.uk/rbf/HIPR2/wksheets.htm.


[32] W. Haq, "Computer vision on nomadic devices for human factors engineering",

Academia.edu, 2012. [Online]. Available:

https://www.academia.edu/11329425/Computer_vision_on_nomadic_devices_for_h

uman_factors_engineering. [Accessed: 20- Mar- 2018].

[33] M. Aly, "Real time Detection of Lane Markers in Urban Streets", Intelligent

Vehicles Symposium, IEEE, pp. 7-12, 2008.

[34] "Nighttime Visibility of Traffic Signs: Chapter 1 - Introduction", 2018. [Online].

Available:

https://safety.fhwa.dot.gov/roadway_dept/night_visib/sign_visib/fhwasa03002/chapt

er1.htm. [Accessed: 20- Mar- 2018].

[35] G. M. S. B. Gajanayake Level, “Study of Road Sign Ignorance among Sri Lankan

Drivers Media and Communication Design Topic Explanation,” 2013.

[36] S. L. Jamson, F. N. Tate, and A. H. Jamson, “Evaluating the effects of bilingual traffic signs on driver performance and safety,” Ergonomics, 2005.

43

8 APPENDIX

Features\ Apps

Existing Research and Products

Dri

ve A

ssis

t

Dri

ving

Coa

ch [

2]

Tra

pste

r [3

]

K. C

hang

et a

l [6]

Prof

iling

[1]

Bip

olar

ity [

8]

Zeb

raR

ecog

nize

r [9

]

Self

-Sim

ilari

ty [

11]

Impa

irm

ent [

13]

Sign

Det

ecto

r [2

0]

Sign

Rec

og. [

21]

aCoD

rive

r 5

[22]

R.L

agun

a et

al [

17]

iOnR

oad

[28]

C. F

. Wu

et a

l [23

]

Lan

e de

tect

ion

[26]

Speed Limit warning based on road signs

X - X - - - - - - - - - - - - X

Geo-location-based alerts

- X - - - - - - - - - - - - - X

Social Media Integration

- X - - - - - - - - - - - - - X

Classification of driving behavior.

- - - X - - - - - - - - - - - X

Average speed warning

- - - - - - - - - - - - - - - X

Identify zebra crossing

- - - - X X X X - - - - - - - X

Identify partially visible crossings

- - - - X - - - - - - - - - - X

Identify at various lightings

- - - - - X - X - - - - - - - X

44

Identify impaired crossing

- - - - - - - X - - - - - - - X

Identify puffin crossing

- - - - - - - - - - - - - - - X

Identify multiple crossings at once

- - - - - - - X - - - - - - - -

Process geospatial video data

- - - - - - - - - - - - - - - X

Identify road signboards

- - - - - - - - X X X X - - - X

Alert the driver of oncoming warnings and required signs

- - - - - - - - - - X - - - - X

Takes journey video as the input

- - - - - - - - X X X - - - - X

Prioritize the outcomes of each components

- - - - - - - - - - - - - - - X

Output in voice with sign meaning with information

- - - - - - - - - - - - - - - X

45

Real-time road sign detection

- - - - - - - - X X X - - - - -

Identify solid lines

- - - - - - - - - - X - X X X X

Identify dashed lines

- - - - - - - - - - X - - X - X

Identify lines at night time or low light

- - - - - - - - - - - - X - X X

Warn driver with both audio and visual alerts

- - - - - - - - - - - - X - - X

Real-time lane detection

- - - - - - - - - - X - X - - X

Table 8 - Research and Product Comparison Summary

Symbols used for the Block Diagram

1. Process/ Storage

2. Input/ Output

46

Figure 7 - Work Breakdown Structure

Impl

emen

tatio

n of

Driv

er

assi

stan

ce sy

stem

1. Implementing Average speed prediction and driver style recognition

1.1 Mobile application to fetch driving data from user and present speed and other information

1.1.1 Android application to fetch data and send to server

1.1.2 Android application to provide information to the user based on the current coordinates

1.2 Backend server to handle transactions from the mobile application

1.2.1 Database schema preperation to store large amount of data for fast acces

1.2.2 Node server to handle requests from mobile application

1.3 Machine learning algorithms to work on the raw data to classify driving behaviour and build user profile

1.3.1 Implementation of ML algoithm.

1.3.2 Integrate with raw database and information database.

2. Pedestrian Crossing Detection

2.1 Implement image processing software to extract pedastrian crossings from GPS embedded video

2.1.1 Record video of a journey with embedded geospatial data

2.1.2 Implement IP algorithms to detect pedestrian crossings

2.2 Store extracted information to the information database with the relevant coordinates

2.2.1 Implement a schema to hold only relevant information show the driver

2.2.2 Get the coordinates and the corresponding data from the DB and alert the driver

3. Road Sign Detection

3.1 Implement image processing software to extract road signs from GPS embedded video

3.1.1 Application of relevant IP techniques to identify important road signs

3.2 Store extracted information to the information database with the relevant coordinates

3.2.1 Get the coordinates and the corresponding data from the DB and alert the driver

4. Lane Departure Warrning

4.1 Andriod implementation to process real time camera feed to identify lanes

4.1.1 Choose the best IP algorithm to identify lane marks

4.1.2 Track the position of the vehicle on the lane

4.2 Alert to the user when departing the lane

Documents

Project Proposal Document · ,1752'8&7,21 %dfnjurxqg /lwhudwxuh 6xuyh\ ,q wrgd\¶v idvw prylqj zruog gulylqj kdv ehfrph d qhfhvvlw\ lq hyhu\rqh¶v olih ,q wkh olih ri prvw jhqhudo