Prospect Eleven: Princeton Universitys Autonomous Vehicle Entry 2005 DARPA Grand Challenge Thursday, February 6, 2006 National Physical Laboratory Teddignton,

Prospect Eleven: Princeton University’s

Autonomous Vehicle Entry 2005 DARPA Grand Challenge

Thursday, February 6, 2006National Physical LaboratoryTeddignton, Middlesex, UK

Alain L. KornhauserTeam Leader, Prospect Eleven

Professor, Operations Research & Financial Engineering Princeton University

Co-founder & Board Chair, ALK Technologies, Inc.

Applications of Knowing “Where am I”

Two Examples:

CoPilot|MinETAReal-Time Dynamic

Minimum ETA Sat/Nav

Prospect Eleven: The Making, Testing and Running of

Princeton’s Entry in the 2005 DARPA Grand Challenge

Alain L. KornhauserTeam Leader, Prospect Eleven

Professor, Operations Research & Financial Engineering

Andrew Saxe’08 Gordon Franken’08 Scott Schiffres’06

Brendan Collins’08 Bryan Cattle’07 Rachel Blair’06

Josh Herbach’08 Anand Atreya’07 Kamil Choudhury’06

Jan 26, 2006

The DARPA Grand ChallengeDefense Advanced Research Projects Administration

• DARPA Grand ChallengeCreated in response to a US Congressional and DoD mandate, it was a field test intended to accelerate research and development in autonomous ground vehicles that will help save lives on the battlefield. The Grand Challenge brought together individuals and organizations from industry, the R&D community, government, the armed services, academia, students, backyard inventors, and automotive enthusiasts in the pursuit of a technological challenge.

• The First Grand Challenge: Across the Mojave, March 2004From Barstow, California to Primm, Nevada offered a $1 million prize. From the qualifying round at the California Speedway, 15 finalists emerged to attempt the Grand Challenge. However, the prize went unclaimed as no vehicles were able to complete the difficult desert route.

• The 2005 Grand ChallengeOctober 8, 2005 in the desert near Primm. Prize increased to $2 million.

Jan 26, 2006

18 -month Life Cycle of Prospect Eleven

2004 2005

Jan Jan Automate Brakes

Feb Feb Install Computers

Mar 2004 Grand Challenge Mar Mount GPS

Apr Announcement of 2005 GC Apr Work on Machine Vision

May Decide to get involved May 1st Site Visit

Jun Hire 5 summer interns Jun Not top 40, Alternate Status

Jul Literature survey Jul Re-Do Stereo Vision

Aug Literature survey Aug 15 2nd Site Visit

Sep Submit Application Sep 27 National Qualifying Event

Oct Oct 8 2005 Grand Challenge

Nov Receive salvaged 05 Canyon Fall Break

Take care of unfinished business

Dec Automate Steering Dec

Jan 26, 2006

Prospect Eleven & Competition

Jan 26, 2006

Constraints• Very little budget

• Simplicity

Guiding Principles

Objective• Enrich the academic experience of the students

Jan 26, 2006

http://www.pcmag.com/slideshow_viewer/0,1205,l=&s=1489&a=161569&po=2,00.asp

Homemade“Unlike the fancy “drive by wire” system employed by Stanford and VW, Princeton’s students built

a homemade set of gears to drive their pickup. I could see from the electronics textbook they were using that they were learning as they went.”

Jan 26, 2006

Teamwork

Real-time Decision SystemAndrew Saxe’08

Object Detection SystemBrendan Collins’08

Mechanical SystemsGordon Franken’08

Planning SystemsJosh Herbach’08

Electronic SystemsBryan Cattle’07

Computing SystemsAnand Atreya’07

Control SystemsScott Schiffres’06

Organizational SystemsRachel Blair’06

Team LeaderAlain Kornhauser*71P03

Jan 26, 2006

Mechanics

Design and Fabrication of:• Vehicle Actuators

– Steering– Brakes– Transmission

• Sensor Housings– Stereo Camera– GPS Antennae

• System Protection– Shock-absorbing computers– Secure installation of components

Jan 26, 2006

Vehicle Actuators• Steering

– Motor from Bosch cordless drill.– Modified gears mounted directly to

steering wheel.– Rotary encoder used for position

feedback.• Brakes

– Bicycle brake cable used to depress pedal.

– Motor-driven linear actuator.– Load-cell used for tension feedback.– Also: pneumatic e-brake, for fail-safe

operation.• Transmission

– Ability to shift from forward to reverse.– Not used due to several issues:

• Slow actuation• Difficulty interfacing with vehicle

transmission• Not open up an Achilles heel.

Drill motor

Steel Gears

Rotary Encoder

bike cable

brake pedalmotormotor

Load cell

Jan 26, 2006

Sensor Housings

• Stereo Camera– Camera rated for “indoor

office use only” !!!– Water & air tight enclosure– Mounts for photographic

filters

• GPS Antennae– 4 GPS receivers– Need good location on top

of vehicle for best data.– Wooden boom also

supported rotating beacon

Jan 26, 2006

System Protection

• Shock Absorption for computers.– Old way: Computers were

wrapped in foam.– New way: Computers are

rack-mounted on shock-isolating feet.

• Secure installation of other components.– Batteries, wires, power

supplies, air compressor…– The “shake test”

Shock-absorbing

feet

Jan 26, 2006

Stereo Vision System

• Principle of Operation: Difference between two cameras gives depth information

• Steps:– Compute disparity image

– Find obstacles in each column

– Approximate with rectangles

– Filter in time domain

Jan 26, 2006

Challenges

– Quantization

– Noise

– Lighting condition

– Field of view

– Occlusion

– Range

Jan 26, 2006

The left camera sees:

Point Grey Bumblebee

Focal Length: 4mm

Resolution: 640x480

Black & White

Jan 26, 2006

The right camera sees:

Point Grey Bumblebee

Focal Length: 4mm

Resolution: 640x480

Black & White

Jan 26, 2006

Yielding a disparity map:

Intensity indicates distance: the lighter, the closer

White indicates an invalidated location.

Jan 26, 2006

Processing in each column:

Intensity indicates confidence that an obstacle exists at that location. A darker line indicates a higher confidence.

Jan 26, 2006

Bounding with rectangles:

Darker rectangles indicate higher confidence.

Jan 26, 2006

Data Representation

Reference:

http://www.darpa.mil/grandchallenge/TechPapers/Stanford.pdf

vs.

Jan 26, 2006

Tracking

Jan 26, 2006

Navigation

• Blind-Man’s Cane Approach:– Find best instantaneous velocity vector (heading and

speed) within visible cone ahead.

? Desired V

Jan 26, 2006

Step 1: Tube Projection

Jan 26, 2006

Angle

Distance

Gap Identification

Shoot the Gap

V

Jan 26, 2006

Examples of “Shoot the Gap”

‘Gotta’ Reverse

Not without Problems

May Get Squeezed Yippes!

Jan 26, 2006

Interfacing to Existing Vehicle DataExisting electronic information available on ’05 GMC Canyon:• Engine status – temp, RPM, diagnostic codes,...• Transmission – what gear are we in?

4WD on/off• Car traction control – are wheels slipping,

are we really moving?• Wheel speed/odometry – how far have we moved?

how fast?• Throttle is electronic

All monitored & used by Prospect Eleven

Jan 26, 2006

OBD-II Interface10101110101...

Throttle voltage

ABS Wheel Encoders Left

Right

Counters

“Data-acquisition card”

Digital inputs

ADC

101001...

DAC

101001...

1001 11011001 0101

Transmission position

Tension on thebrake pedal

100 lb

Jan 26, 2006

Other thoughts

Jan 26, 2006

Feedback Control SystemsSpeed Control Module:

– Receives desired speed from Navigation– Decides how to modulate throttle and brake– Requirements

• Gracefully reach and maintain desired speed• Smooth acceleration and braking

– To minimize skidding on loose sand wet grass

• Strict adherence to speed limit– Minimize overshoot

Steering Control Module– Receives desired heading angle and speed from Navigation– Converts to desired steering wheel angle– Constrains wheel angle as function of speed to avoid roll-over.– Control Coefficients tuned to trade off response and overshoot

Jan 26, 2006

Speed Control -- Implementation

• Proportional Integral Control– Output = KPev + ∫KIevdt + KD (dev∕dt)

– Where evelocity = Vdesired – Vcurrent

• KP term deals with the bulk of error

• KI integrates up error to eliminate steady state error

• KD reduces overshoot and ringing by slowing response

Speed ControlDesired Speed

Current Speed

Latest Speeds

Percent Throttle

Percent Brake Tension

Jan 26, 2006

Why P11 stopped after 9.4 miles

• We store a list all obstacles we are currently concerned about– Each time a new obstacle is received, we add it to this list– Each obstacle needs to have its relative position updated each time

we get new vehicle position information• It gets these updates by subscribing to the RelativeFrameUpdated event

• When we have passed an obstacle by more than twenty feet, we remove it from the list– However, we were not unsubscribing the obstacle from the

RelativeFrameUpdated eventThus, nine miles down the road, our computer was still processing a bush it saw near the beginning of the course!

C# is a powerful language, but it has to be used carefully

Timeline of Accomplishments of Princeton’s “Prospect Eleven” DARPA Grand Challenge Team

May 2004 - Nov 2005

Segment Description Images Videos

“Going Back” Oct 30-Nov 2, ‘05

After completing 10 miles in GCE, Prospect Eleven returns to the desert to “complete” the 2005 and the 2004 Grand Challenge coursesGPS tracks for 3 days

2004 DGCNov 2, ‘05

GPS Tracks 0.3mbRun Images 3mb

Cruisin’04 2.5mb

Return to Beer Bottle Pass, Nov 1, ‘05

GPS Tracks 0.3mb

2005 DGC, Oct 31, ‘05

GPS Tracks 0.1mbRun Images 3mb

Cruisin’05 16mbGate’05 3mb, Gate2 ’05 4mb

Changing “one line”, Oct 30, ‘05 Images 0.2mb FixingCalibrating

2005 GCEOct. 8, ‘05

DARPA Grand Challenge Event, 132 mile course in desert around Primm, NV; 23 entries; Prospect Eleven is #10 seed

Run Summary Start_11mbPassBy_29mbSummary 17mb

NQESep 27-Oct 5, 05

National Qualifying Event held at California Speedway. 43 “bots” competing for 23 positions on 2.2 mile course

NQE Course CrashOutside_7mbCrashInside_4mbPrefectRun5_9mb

Run-up to NQEAug 17-Sep 26

Modifications and testing after invitation as one of three “Alternates” to the NQE

2nd Site VisitAug 16, ‘05

2nd chance at demonstrating capabilities of P11 to DARPA officials at West Windsor athletic fields after earning “Alternate status

Pre-2nd_SiteVisit_6mb

1st Site VisitMay 3, ‘05

Demonstration of Prospect Eleven’s capabilities and potential to DARPA officials, mixed performance

Automation of P11Nov ’04-May ‘05

Conversion of a GMC 05 Canyon to Prospect Eleven: Automation of brake, throttle, steering; GPS; vision

P11 Tech Paper

Application & PrepMay ‘04-Nov ’05

Putting the team together, planning, organization, & literature search.

DAPRPA Grand Challenge Event, October 07, 2005

The Day Before 2005 Grand Challenge Event

L2R: Rachel Blair’06, Dan Chiou’05, Prof. Alain Kornhauser*71 P03, Ben Essenburg’05, Bryan Cattle’07, Josh Herbach’08. Jeff Jones*05, Kamil Chihoudy’06, Scott Schiffres’06, Anand Atreya’07; Launch Team: Andrew Saxe’08, Brendan Collins’08, Gordon Franken’08


The DARPA Grand Challenge Event Team 2005 Grand Challenge Event


The Launch of Prospect Eleven 2005 Grand Challenge Event


Return of Prospect Eleven @ 8 Mile Mark

2005 Grand Challenge Event

DAPRPA Grand Challenge, Unfinished Business, October 31, 2005

Approaching Beer Bottle Pass 19:02

2005 Grand Challenge Course


Beer Bottle Pass 19:10 PST

2005 Grand Challenge Course

DAPRPA Grand Challenge, Unfinished Business, November 1, 2005

The Assent: 17:00Rerun to Beer Bottle

Pass


The Assent: 17:01Rerun to Beer Bottle

Pass


12:252004 Grand Challenge Course


14:262004 Grand Challenge Course


The Finish 18:44 2004 Grand Challenge Course

L2R: Prof. Alain Kornhauser*71 P03, Andrew Saxe’08, Bryan Cattle’07, Scott Schiffres’06

Prospect Eleven’sUnfinished Business

October 31-November 2, 2005DARPA Grand Challenge ’04 & ’05 Courses

GPS Tracks and Timing Maps


Finish, ’05 Course

Jan 26, 2006

Start 7:53:30 PSTFinish 19:35:49Elapsed Time 11:43:49Pause Time ~2:55:49Autonomous Time 8:48:59

8:44:48 1:10 Pause

10:48:04 3:45 Open & pass thru gate

9:52:53 8:14 Open gate & pass through

13:18:531:10 Pause

18:20:53 38:50 prepare inside video

19:00:21 – 0:56 Adjust Camera19:02:53 – 1:52 Wait for chase car

10:37:37 5:54 Open & pass thru gate

18:02:502:08 Pause

11:15:155:14 Pause at ranch, go thru gate

11:29:5242:00 Pull chase car out of mud

13:16:051:32 Rejoin course

10:53:332:15 Pause

14:16:032:23 remove rubble blocking I-15 underpass

12:42:354:40 Pause

12:33:511:19 Pause

15:13:311:30 Wait for traffic to clear on NV 161

16:03:071:53 Pause

16:07:484:32 to take pictures

16:12:5011:44 negotiate bulldozed area

16:31:172:12 Survey another bulldozed area

8:20:555:20 Pictures at Primm return

8:31:428:53 Take down fence & pass through

GPS Tracks & Timing of Prospect Eleven’s Autonomous run of the 05 GC Course 10/31/05

Jan 26, 2006

Red: 8:00 Outbound 10/31Deviation to avoid “lake”

Yellow: 16:30 Return 10/31Deviation to avoid “lake”

Blue: 8:00 Outbound 11/2Lake dried sufficiently, no deviation required.

GPS Tracks & Timing of Prospect Eleven Diverting Around Not-so-dry Lake 05 GC Course 10/31/05


Return to Beer Bottle Pass

GPS Tracks of return to Beer Bottle Pass 11/01/05


Return to Beer Bottle Pass 2005 Grand Challenge Course, 1.9 miles

Start Down: 17:10:17

End Up: 17:04:48Duration: 0:10:21

Start Up: 16:54:27

End Down: 17:18:04

Duration: 0:7:47

GPS Tracks & Timing of Prospect Eleven’s Autonomous run up then back down Beer Bottle Pass 11/02/05

Jan 26, 2006

7:28:334:35 negotiate bulldozed area

7:55:2633:08 Divert to get gas for P11

8:39:1311:50 take pictures


10:13:432:10 cross road




7:08:36Start ’04 GC Course



Jan 26, 2006

14:50:xx4:00 Avoid Washout in 4 spots

15:10:102:04 Inspect burial monument

12:05:250:30 Xing CA 127


13:08:541:05:56 Fix steering encoder




12:53:360:54 Pause





Jan 26, 2006

17:16:3315:28 Replace Front Left Blowout

16:12:070:50 Xing road




16:33:152:45 Letting traffic pass on Randall Drive

16:35:141:00 Xing Yermo Road

17:32:274:00 Inspecting Silt Filled underpasses < 4 ft. clearance

18:41:261:10 Pause let cars pass on Stoddar Valley Rd


18:18 Top

18:35 Bottom

Daggett Pass

Start 7:08:36Finish 18:44:19 – Arrival at gate of Slash X CafeElapsed Time 11:37:43Pause Time ~03:28:00Autonomous Time 08:09:00

File photo

File photo

File photo

Finish, ’04 Course

Jan 26, 2006

Jan 26, 2006

Accomplishments:• Invited to National Qualifying Event

• Seeded 10th for Grand Challenge

• Accomplished 10 miles of Autonomous Driving in GC

• “Completed” the 2005 & 2004 Courses during Fall Break

Overview:http://orfe.princeton.edu/~alaink/Prospect11/Movies/P11SummaryMovie.wmv

Wasn’t so easy, Outside: http://orfe.princeton.edu/~alaink/Prospect11/Movies/NQE2CrashOutsideView.mov

Wasn’t so easy, Inside: http://orfe.princeton.edu/~alaink/Prospect11/Movies/NQE2CrashInside.mov

Essentially Perfect:http://orfe.princeton.edu/~alaink/Prospect11/Movies/NQE5PerfectRun.mov

Launch at Grand Challenge:http://orfe.princeton.edu/~alaink/Prospect11/Movies/GCE-Start.AVI

Doing Better than 2004’s Best:http://orfe.princeton.edu/~alaink/Prospect11/Movies/GCE_PassBy8Miles.avi

Return to the Desert; 2005 Course:http://orfe.princeton.edu/~alaink/Prospect11/Movies/PGC05_10h29mFastDryLake.AVIhttp://orfe.princeton.edu/~alaink/Prospect11/Movies/PGC05_10h34mFastDryLake.AVI

Return to the Desert; Return to Beer Bottle Pass:

http://orfe.princeton.edu/~alaink/Prospect11/Movies/PGCBB_Small.mov

Return to the Desert; 2004 Course: http://orfe.princeton.edu/~alaink/Prospect11/Movies/PGC04_11h59mCruisin.AVI

Some Videos:

Jan 26, 2006

Lessons Learned• It is non-trivial to “Just Do It”

– You must respect Uncertainty (and plan for it)– Harmonize Accuracy– Time is your Friend

• (only know what you need to know when you need to know it)

– More is not necessarily Better– Always assume your code has bugs

• Stereo Vision Does Work!• Three (3) Regimes of Autonomous Control

– Under “7” mph– Between 7- 25 mph– Above 25 mph

Jan 26, 2006

Operational Processes used byCapital District Advanced Traveler Information System

(CD-ATIS)

CoPilot|MinETA: Real-Time Dynamic Minimum ETA Sat/Nav

by

Alain L. KornhauserCo-founder & Board Chair, ALK technologies, Inc.

Jan 26, 2006

1 6

1

1 1

2

2

2

3

3

3

4

3

5 5

6

6

8

8

9

6

PROBLEM: How to get from A to B•Many Paths

•Each with a Different Value to the Decision Maker

•Each Segment Changing with Uncertainty over Time

Things Change!!

4

Jan 26, 2006

Link Travel TimesHistoric, Actual & Forecast During Day One week-day on one link

Things change!

Jan 26, 2006

Real-Time Dynamic Minimum ETA Sat/Nav

•250 Volunteers using CoPilot|Live commuting to/from RPI

• CoPilot continuously shares real-time probe-based traffic data

• CoPilot continuously seeks a minimum ETA route

“Advance” project Illinois Universities

Transportation Research Consortium

The late 90s

Conducted its version of the abandoned “ADVANCE” (Advanced Driver and Vehicle Advisory Navigation ConcEpt )project

Link

&

Jan 26, 2006

Project Objectives

• Create: real-time data collection from vehicles and dissemination to vehicles of congestion avoidance information which is used to automatically reroute drivers onto the fastest paths to their destinations

• Target locations: small to medium-sized urban areas

• Aspects: operations, observability, controllability, users, information transfer to travelers

Jan 26, 2006

3-month field test

Capital District (Albany), NY, USA

Journey-to-work

200 participants80 Tech Park employees120 HVCC staff & students“Techy” travelers

Network:Freeways & signalized arterialsCongested linksPath choices exist

I 90

I 787

US

HW

Y 4

STATE

HW

Y 32

STATE HWY 2

STATE HWY 151US HW

Y 9

STATE HWY 378

2ND

ST

STATE HWY 155

WASHINGTON AVE

WINTER ST

15T

H S

T

STATE HWY 43

8TH

ST

STA

TE

HW

Y 3

77B

RO

AD

WAY

AV

E

CAMPBELL AVE

ACCESS RD

1ST ST

STATE HWY 136

COUNTY HWY 130

TIBBITS AVE

LIN

CO

LN A

VE

RA

MP

LOU

DO

N R

D

STATE HWY 43

RAMP

RA

MP

RA

MP

RAMP

RA

MP

HVCC

Rensselaer Technology

Park

TroyColonie

North Greenbush

Albany

Experiment Details

Jan 26, 2006

Basic Operational ArchitectureTwo-way cellular data communications between

Customized Live|Server

at ALK

Customized CoPilot|Live

In vehicles

6

Destination

12, 4

3 57

8

Jan 26, 2006

The In-Vehicle “Device”

Sprint PCS Vision card and battery pack: Communication

with the server

CoPilot GPS unit:Determines location

Pocket PC with 256 MB SD card: Software platform and audio

communication with the driver

Other Sat/Nav Platforms

Jan 26, 2006

Every Second

CoPilot|Live Determines “Where am I”, Then…

CoPilot|Live “Where Am I”,

Then…

ALK Server Updates:

TT(mi, mj )

If Momument, mj , is passed

Send mi , mj , ttk(mi, mj )= t(mi) - t(mi)

(52 bytes)

Set i=j

Jan 26, 2006

Every “n” Minutes

ALK Server Builds: set Uk

Sends: TT(mi, mj ) for every (i,j) in Uk

CoPilot|Live … Send… Current Location & Destination,

Last update time (42 bytes)

ALK Server …Send… New TT(mi, mj ) for every (i,j) in Uk

(280 bytes/100arcs)

CoPilot|Live …Updates TT(mi, mj ) in Uk , ETA on current route, Finds new

MinETA route, if MinETA “substantially” better then… Adopt new route

ALK Server …Determines Uk : set of TT(mi, mj ) within “bounding polygon”

of (Location;Destination)k that have changed more than “y%” since last update.

CoPilot|Live Sends: “Where am I”, Dest., Last update Receives/Posts: updatesComputes: MinETA Updates route, if better

Jan 26, 2006

When Available

ALK Server …Receives: Other congestion information from various

source, blends them in TT(mi, mj )

ALK Server Updates:

TT(mi, mj )

Jan 26, 2006

What We HeardI find it interesting how willing I am to listen to a machine tell me

which route to take

I like using it for when I have no idea on how to get somewhere, and it is good for my normal

route because it keeps me out of traffic on route 4.

It is great, it took a while to trust it telling me where to go, but i like it because i

cant get lost! Thanks.

This thing is awesome. I was a little skeptical at first but once i got the hang

of it I don’t know how I went along without it. I think any student commuting

to school will benefit from this.

I'm very impressed with the CoPilot program thus far. The

directions are accurate and it adapts quickly to route changes.

Jan 26, 2006

1

2

3

also Can Watch Vehicles

Jan 26, 2006

North American Monument Network• ~125,000 North American “Monuments”

• ~106 (mi, mj)

• 1st Commercial application in PC*Miler v19 (contains Median Travel Time by Time-of-Day for all NA)

(mi, mj) near Troy (mi, mj) larger area

Jan 26, 2006

Thank youAlain L. Kornhauser

[email protected]

www.princeton.edu/~alaink/

Documents

Prospect Eleven: Princeton Universitys Autonomous Vehicle Entry 2005 DARPA Grand Challenge Thursday, February 6, 2006 National Physical Laboratory Teddignton,