Upload
owen-timpson
View
221
Download
3
Tags:
Embed Size (px)
Citation preview
Traffic Light Control Using Reinforcement Learning
Daniel Goldberg
Andrew Elstein
Final Presentation
The Problem
• Traffic congestion is estimated to cost Americans $121 billion in lost productivity fuel, and other costs.
• Traffic Lights are imperfect and contribute to this
• Usually statically controlled
• A better method of controlling them can reduce waiting times significantly
Approach
• Implement a “Reinforcement Learning” (RL) algorithm to control traffic lights
• Create a simulation of traffic to tweak and test traffic light optimizations
Implementation• If minor adjustments
were made to the algorithm, it could operate within existing infrastructure
• Optimally, a camera system and would be added
Simulation
Insert picture of visualization
Simulation Structure• To build the simulation we created the follow Data
Structures:
• Cars
Position, Destination, Velocity, Map, Color
• Roads
• Lanes
• Individual Cells
• Intersection location matrix
• Intersections
• Position, Traffic Lights
• In total, the simulation is coded in
MATLAB with 3100 lines of code
Cars Struct
Simulation Dynamics
• Cars are spawned randomly
• They follow an randomly generated path to destination
• Cars follow normal traffic rules
• Road Cells are discretized to easily simulate traffic, only one car can exist in each road cell. Cars move ahead one or two cells in each time-step, depending on the car's max velocity and whether there is an open spot.
Demo
Reinforcement Learning
• Weiring - Multi-Agent Reinforcement Learning for Traffic Light Control
• It introduced an objective function to minimize or maximize a goal value
tl = traffic lightp = current position d = destinationL = light decision = discounting constant ‘ = next
Reinforcement Learning Theory
• Coordinating a system of lights to respond to current conditions can reap exceptional benefit
• The theory cleverly merges probability, game theory and machine learning to efficiently control traffic
• In our case, the expected value of each of a light’s possible states are calculated
• With this value function a game is played to maximize it, in turn minimizing waiting time
Results
Wrote a script to compare the smart algorithm to static On-Off-On-Off lights.
Our algorithm reduced average waiting time—and thus traveling time—for a system with any number of cars
Travelling time for our implementation was reduced by an average of 10%. There was a 15% reduction for sparse traffic systems from a static control, but only a 3% decrease for heavy congestion.
Results cont.
Extensions• Fairness-weighted objective:
•
• ω = weighting constant
• t = current time
• ti = time of arrival for car i
•
• If F(t) > 1, cars on road 1 get to go
• If F(t) < 1, cars on road 2 get to go
Further Extensions
• Car Path optimization and rerouting
• Model expansion to traverse an entire city
• Inter-traffic-light communication
• Retesting with increased computational resources for modeling accuracy and robustness
RL In the News• Samah El-Tantawy, 2012 PhD recipient from the
University of Toronto, won the 2013 IEEE best dissertation award for her research in RL.
• Her RL traffic model showed reduced rush-hour travel times by as much as 26 percent and is working on monetizing her research with small MARLIN-ATSC (Multi-agent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers) computers.
Challenges
• Difficult to understand data structures and how they would interact
• Object Oriented Approach vs. MATLAB’s index-based structures
• Understand how cars would interact with each other
• Understanding RL algorithm
• Adapting our model to use RL algorithm
• Limited computational resources