Embed Size (px)
Citation preview
Lorenzo Orecchia Department of Computer Science
Challenges in Optimization for Data Science
Combinatorial meets Convex Leveraging combinatorial structure to obtain faster algorithms, provably
6/30/15 1
A Tale of Two Disciplines
A Tale of Two Disciplines
Combinatorial Optimization !!
••Asks about paths, flows, cuts, clustering, routing !
••Questions and solutions techniques tend to be discrete/combinatorial
A Tale of Two Disciplines
Combinatorial Optimization !!
••Asks about paths, flows, cuts, clustering, routing !
••Questions and solutions techniques tend to be discrete/combinatorial
Numerical and Convex Optimization !!
•• Asks about matrices, convex programs (e.g. LPs, SDPs)
••Questions and solution techniques tend to be continuous/numerical
NEW INSIGHT: Deep connections between core concepts Use techniques from one to help the other
A Tale of Two Disciplines
Combinatorial Optimization !!
••Asks about paths, flows, cuts, clustering, routing !
••Questions and solutions techniques tend to be discrete/combinatorial
Numerical and Convex Optimization !!
•• Asks about matrices, convex programs (e.g. LPs, SDPs)
••Questions and solution techniques tend to be continuous/numerical
NEW INSIGHT: Deep connections between core concepts Use techniques from one to help the other
A Tale of Two Disciplines
Combinatorial Optimization !!
••Asks about paths, flows, cuts, clustering, routing !
••Questions and solutions techniques tend to be discrete/combinatorial
Numerical and Convex Optimization !!
•• Asks about matrices, convex programs (e.g. LPs, SDPs)
••Questions and solution techniques tend to be continuous/numerical
Fast Algorithms for Fundamental Graph Problems
Why Graph Algorithms?
Computer Networks
Transportation Networks
Social Networks
Representations of Physical Objects
Why Fast Graph Algorithms• Classical Algorithms (1970s-‐1990s): Standard notion of efficiency is polynomial running time • Today: Graphs of interest are getting larger and larger … Even quadratic running time is unfeasibly large for many instances
!!!!
Why Fast Graph Algorithms
Why Fast Graph Algorithms• Classical Algorithms (1970s-‐1990s): Standard notion of efficiency is polynomial running time • Today: Graphs of interest are getting larger and larger … Even quadratic running time is unfeasibly large for many instances
!• New Efficiency Requirement: as close as possible to linear
NEARLY-‐LINEAR RUNNING TIME
!!!
Input Size Running Time
Why Fast Graph Algorithms• Classical Algorithms (1970s-‐1990s): Standard notion of efficiency is polynomial running time • Today: Graphs of interest are getting larger and larger … Even quadratic running time is unfeasibly large for many instances
!• New Efficiency Requirement: as close as possible to linear
NEARLY-‐LINEAR RUNNING TIME
!!!
Input Size Running Time
GOAL: Build library of primitives running in almost-‐linear time • What can you do to a graph in nearly-‐linear time?
Nearly-‐Linear-‐Time Algorithms
GOAL: Build library of primitives running in almost-‐linear time • What can you do to a graph in nearly-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
REACHABILITY PROBLEM: Is there a path between vertices u and v in the graph G?
u v
GOAL: Build library of primitives running in almost-‐linear time • What can you do to a graph in nearly-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
Reachability
REACHABILITY PROBLEM: Is there a path between vertices u and v in the graph G?
u v
YES, by depth-‐first search
GOAL: Build library of primitives running in almost-‐linear time • What can you do to a graph in nearly-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
Reachability
YES, by depth-‐first search
SHORTEST PATH PROBLEM: What is the shortest path from u to v?
u
v
GOAL: Build library of primitives running in almost-‐linear time • What can you do to a graph in nearly-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
ReachabilityShortest Path
YES, by depth-‐first search
SHORTEST PATH PROBLEM: What is the shortest path from u to v?
u
v
YES, by breadth-‐first search
GOAL: Build library of primitives running in almost-‐linear time • What can you do to a graph in nearly-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
ReachabilityShortest PathConnectivity
Minimum Cost Spanning Tree…
GOAL: Build library of primitives running in almost-‐linear time • What can you do to a graph in nearly-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
ReachabilityShortest PathConnectivity
Minimum Cost Spanning Tree…
Probe Graph Structure in Simple Way
• What can you do to a graph in almost-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
Reachability, Shortest Path, Connectivity Minimum Cost Spanning Tree, …
ELECTRICAL FLOW PROBLEM: What is the current flow when one unit of current enters s and one leaves t?
1 1
• What can you do to a graph in almost-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
Reachability, Shortest Path, Connectivity Minimum Cost Spanning Tree, …
Laplacian Systems of Linear Equations [Spielman, Teng’04]
ELECTRICAL FLOW PROBLEM: What is the current flow when one unit of current enters s and one leaves t?
1 1
• What can you do to a graph in almost-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
Reachability, Shortest Path, Connectivity Minimum Cost Spanning Tree, …
Laplacian Systems of Linear Equations [Spielman, Teng’04]
• What can you do to a graph in almost-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
Reachability, Shortest Path, Connectivity Minimum Cost Spanning Tree, …
Laplacian Systems of Linear Equations [Spielman, Teng’04]
Undirected Flow Problems • s-‐t Maximum Flow [Kelner, Orecchia, Sidford, Lee ‘13][Sherman’13] • Concurrent Multi-‐commodity Flow [Kelner, Orecchia, Sidford, Lee ’13] • Oblivious Routing [Kelner, Orecchia, Sidford, Lee ’13] !… and Undirected Cut Problems • Minimum s-‐t cut [Kelner, Orecchia, Sidford, Lee ‘13][Sherman’13] • Approximate Sparsest Cut [Kelner, Orecchia, Sidford, Lee ’13] [Sherman’13] • Approximate Minimum Conductance Cut [Orecchia, Sachdeva, Vishnoi ’12]
A Faster, Simpler Laplacian Solver
+8 +8
A Faster, Simpler Laplacian Solver
+8 +8
INITIALIZATION:• Choose a spanning tree T of G.
A Faster, Simpler Laplacian Solver
+8 +88
INITIALIZATION:• Choose a spanning tree T of G.• Route flow along T to obtain initial flow f0.
88
8
A Faster, Simpler Laplacian Solver
+8 +88
INITIALIZATION:• Choose a spanning tree T of G.• Route flow along T to obtain initial flow f0.
88
8
0
0
0
0 0
0
+8 +88
88
8
0
0
0
0 0
0
A Faster, Simpler Laplacian Solver
+8 +88
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
NOTE: Flow f0 is electrical flow if and only if Ohm’s Law holds for all edges e=(b,a) !• Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. !
!!
88
8
0
0
0
0 0
0
A Faster, Simpler Laplacian Solver
+8 +88
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
NOTE: Flow f0 is electrical flow if and only if Ohm’s Law holds for all edges e=(b,a) !• Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. !
!!
88
8
0
81624
32
0
0
0
0 0
0
A Faster, Simpler Laplacian Solver
+8 +88
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
NOTE: Flow f0 is electrical flow if and only if Ohm’s Law holds for all edges e=(b,a) !• Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. !
!!
88
8
0
81624
32
24 24
24
0
0
0
0 0
032
A Faster, Simpler Laplacian Solver
Our Fastest, Simplest Laplacian Solver
+8 +88
88
8
0
81624
32
24 24
24
0
0
0
0 0
032INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
NOTE: Flow f0 is electrical flow if and only if Ohm’s Law holds for all edges e=(b,a) !• Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. !
• Check if voltages and flows obey Ohm’s Law on off-‐tree edges. !!
Our Fastest, Simplest Laplacian Solver
+8 +88
88
8
0
81624
32
24 24
24
0
0
0
0 0
032INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
NOTE: Flow f0 is electrical flow if and only if Ohm’s Law holds for all edges e=(b,a) !• Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. !
• Check if voltages and flows obey Ohm’s Law on off-‐tree edges. !!
Our Fastest, Simplest Laplacian Solver
+8 +88
88
8
0
81624
32
24 24
24
0
0
0
0 0
032INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
NOTE: Flow f0 is electrical flow if and only if Ohm’s Law holds for all edges e=(b,a) !• Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. !
• Check if voltages and flows obey Ohm’s Law on off-‐tree edges. !!
FAIL
FAIL
Our Fastest, Simplest Laplacian Solver
+8 +88
88
8
0
81624
32
24 24
24
0
0
0
0 0
032
Our Fastest, Simplest Laplacian Solver
+8 +88
88
8
0
81624
32
24 24
24
0
0
0
0 0
032INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e
Our Fastest, Simplest Laplacian Solver
+8 +8
8
88
8
0
0
0
0 0
0
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e
Our Fastest, Simplest Laplacian Solver
+8 +8
8
88
8
0
0
0
0 0
0
Our Fastest, Simplest Laplacian Solver
+8 +8
8
88
8
0
0
0
0 0
0
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e • Repeat
Our Fastest, Simplest Laplacian Solver
+8 +8
88
8
0
20/3
4/3
4/3 4/3
-‐4/3
-‐4/3
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e • Repeat
4/3
-‐4/3
Our Fastest, Simplest Laplacian Solver
+8
+8
88
8
0
20/3
4/3
4/3
-‐4/3
0
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e • Repeat
4/3
-‐4/3
Our Fastest, Simplest Laplacian Solver
+8
+8
88
8
0
20/3
4/3
4/3
-‐4/3
0
81624
92/3
84/3 80/388/3
76/3
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e • Repeat
4/3
-‐4/3
Our Fastest, Simplest Laplacian Solver
+8
+8
88
8
0
20/3
4/3
4/3
-‐4/3
0
81624
92/3
84/3 80/388/3
76/3
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e • Repeat
4/3
-‐4/3
Our Fastest, Simplest Laplacian Solver
+8
+8
88
8
0
20/3
4/3
4/3
-‐4/3
0
81624
92/3
84/3 80/388/3
76/3
OK
FAIL
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e • Repeat
4/3
-‐4/3
Our Fastest, Simplest Laplacian Solver
+8
+8
88
8
0
20/3
4/3
4/3
-‐4/3
0
81624
92/3
84/3 80/388/3
76/3
OK
FAIL
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e • Repeat
4/3
-‐4/3
Our Fastest, Simplest Laplacian Solver
+8
+8
88
8
0
20/3
4/3
4/3
-‐4/3
Working in the Space of Cycles
16
f0
f*
Subspace of flows routing required current input/output
START: Geometric interpretation of electrical flow algorithm
Initial Flow
Electrical Flow
Working in the Space of Cycles
16
f0
f*
Subspace of flows routing required current input/output
START: Geometric interpretation of electrical flow algorithm
Initial Flow
Electrical Flow
NB: Our iterative solutions never leave this subspace thanks to cycle updates !
f1
Cycle update on C1
Working in the Space of Cycles
16
f0
f*
Subspace of flows routing required current input/output
START: Geometric interpretation of electrical flow algorithm
Initial Flow
Electrical Flow
NB: Our iterative solutions never leave this subspace thanks to cycle updates !
f1
f2
Cycle update on C2
Working in the Space of Cycles
16
f0
f*
Subspace of flows routing required current input/output
START: Geometric interpretation of electrical flow algorithm
Initial Flow
Electrical Flow
NB: Our iterative solutions never leave this subspace thanks to cycle updates !
f1
f2
Cycle update on C2
Linear combination of cycles
Coordinate Descent in the Space of Cycles
17
GOAL:
f0 f*
Electrical Flow
Coordinate Descent in the Space of Cycles
17
GOAL:
f0
• Pick a basis of the space of cycles
f*
Electrical Flow
Coordinate Descent in the Space of Cycles
17
GOAL:
f0
• Pick a basis of the space of cycles• Fix a coordinate at the time, i.e., coordinate descent
f*
Electrical Flow
f1
Coordinate Descent in the Space of Cycles
17
GOAL:
f0
• Pick a basis of the space of cycles• Fix a coordinate at the time, i.e., coordinate descent
f*
Electrical Flow
f1
f2
Coordinate Descent in the Space of Cycles
17
GOAL:
f0
• Pick a basis of the space of cycles• Fix a coordinate at the time, i.e., coordinate descent
f*
Electrical Flow
f1
f2
Coordinate Descent in the Space of Cycles
18
GOAL:
f0
EXAMPLE: Ill-‐conditioned basis yields slow convergence
f*
Electrical Flow
Coordinate Descent in the Space of Cycles
18
GOAL:
f0
EXAMPLE: Ill-‐conditioned basis yields slow convergence
f*
Electrical Flow
Low-‐Stretch Spanning TreesG
Low-‐Stretch Spanning TreesGspanning tree T
Low-‐Stretch Spanning TreesGspanning tree T
e
Low-‐Stretch Spanning TreesGspanning tree T
e
Stretch of e =
Length of cycle Ce
st (e) = 5
Low-‐Stretch Spanning TreesGspanning tree T
e
Stretch of e =
Length of cycle Ce
Low-‐Stretch Spanning TreesGspanning tree T
e
Stretch of e =
Length of cycle Ce
Low-‐Stretch Spanning TreesGspanning tree T
e
Average Stretch:
Stretch of e =
Length of cycle Ce
Low-‐Stretch Spanning TreesGspanning tree T
Fact [Abraham, Neiman ’12]: It is possible to compute a spanning tree with average stretch O(log n loglog n) in almost-‐linear time.
Average Stretch:
Stretch of e =
Length of cycle Ce
Low-‐Stretch Spanning TreesGspanning tree T
Fact [Abraham, Neiman ’12]: It is possible to compute a spanning tree with average stretch O(log n loglog n) in almost-‐linear time.
Average Stretch:
Stretch of e =
Length of cycle Ce
Electrical Flow: Algorithmic Components
24
Continuous Optimization:
Randomized Coordinate Descent ITERATIVE METHOD
Electrical Flow: Algorithmic Components
24
Continuous Optimization:
Randomized Coordinate Descent
PROBLEM REPRESENTATION
ITERATIVE METHOD
Combinatorial Optimization:
• Space of cycles
• Basis given by Low-‐Stretch Spanning Tree
Electrical Flow: Algorithmic Components
24
Continuous Optimization:
Randomized Coordinate Descent
PROBLEM REPRESENTATION
ITERATIVE METHOD
Combinatorial Optimization:
• Space of cycles
• Basis given by Low-‐Stretch Spanning Tree
Joint Design of Components
A Framework for Algorithmic Design
Leverage Continuous Optimization Ideas Fast convergence requires smooth problem
ITERATIVE METHOD
PROBLEM REPRESENTATION
Not all representations are created equal: Use combinatorial techniques to produce a smooth representation
A Framework for Algorithmic Design
Leverage Continuous Optimization Ideas Fast convergence requires smooth problem
ITERATIVE METHOD
PROBLEM REPRESENTATION
Not all representations are created equal: Use combinatorial techniques to produce a smooth representation
Efficiency and simplicity rely on combining these two components in the right way
Connection with Electrical Flow
312
2
43
2
1
2
1
s-‐t Maximum Flow
Congestion Minimization
Connection with Electrical Flow
312
2
43
2
1
2
1
Electrical Flow s-‐t Maximum Flow
Energy Minimization Congestion Minimization
Connection with Electrical Flow
312
2
43
2
1
2
1
Electrical Flow s-‐t Maximum Flow
Set resistances as:
An Extra DifficultyObjective function:
An Extra Difficulty
PROBLEM: OBJECTIVE IS EXTREMELY NON-‐SMOOTH
Objective function:
An Extra Difficulty
PROBLEM: OBJECTIVE IS EXTREMELY NON-‐SMOOTH
Objective function:
An Extra Difficulty
PROBLEM: OBJECTIVE IS EXTREMELY NON-‐SMOOTH
Objective function:
An Extra Difficulty
PROBLEM: OBJECTIVE IS EXTREMELY NON-‐SMOOTH
Objective function:
An Extra Difficulty
PROBLEM: OBJECTIVE IS EXTREMELY NON-‐SMOOTH
Objective function:
An Extra Step: RegularizationObjective function:
An Extra Step: Regularization
SOLUTION: Change objective Find function that is close to objective but somewhat smooth
Objective function:
An Extra Step: Regularization
SOLUTION: Change objective Find function that is close to objective but somewhat smooth
Objective function:
An Extra Step: Regularization
SOLUTION: Change objective Find function that is close to objective but somewhat smooth
Objective function:
PROBLEM: OBJECTIVE IS EXTREMELY NON-‐SMOOTH
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed Regularize to softmax
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed
Use basis given byLow-‐stretch spanning tree
Regularize to softmax
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed
Use basis given byLow-‐stretch spanning tree
Regularize to softmax
Which basis to use?Little interference in
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed
Use basis given byLow-‐stretch spanning tree
Regularize to softmax
Which basis to use?Little interference in
Surprising equivalence: Basis is
OBLIVIOUS ROUTING SCHEME
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed
Use basis given byLow-‐stretch spanning tree
Regularize to softmax
Which basis to use?Little interference in
Surprising equivalence: Basis is
OBLIVIOUS ROUTING SCHEME
Oblivious Routing
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
Oblivious Routing
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
Oblivious Routing
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
Oblivious Routing
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
Oblivious Routing
Routing = Probability Distribution over Paths = Flow
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
Oblivious Routing
Routing = Probability Distribution over Paths = Flow
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
Oblivious Routing
Routing = Probability Distribution over Paths = Flow
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
Oblivious Routing
Routing = Probability Distribution over Paths = Flow
Oblivious RoutingGOAL: Route traffic between many pairs of users on the Internet
Minimize maximum congestion of a link DIFFICULTY: Requests arrive online in arbitrary order How to avoid global flow computation at every new arrival
Oblivious Routing
SOLUTION: Oblivious Routing: Every request is routed obliviously of the other requests
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
DIFFICULTY: Requests arrive online in arbitrary order How to avoid global flow computation at every new arrival
Oblivious Routing
SOLUTION: Oblivious Routing: Every request is routed obliviously of the other requests
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
DIFFICULTY: Requests arrive online in arbitrary order How to avoid global flow computation at every new arrival
Oblivious Routing
SOLUTION: Oblivious Routing: Every request is routed obliviously of the other requests
PRE-‐PREPROCESSING: Routes are pre-‐computed
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
DIFFICULTY: Requests arrive online in arbitrary order How to avoid global flow computation at every new arrival
Oblivious Routing
SOLUTION: Oblivious Routing: Every request is routed obliviously of the other requests
PRE-‐PREPROCESSING: Routes are pre-‐computed
MEASURE OF PERFORMANCE: Worst-‐case ratio between congestion
of oblivious-‐routing and optimal a posteriori routing
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
DIFFICULTY: Requests arrive online in arbitrary order How to avoid global flow computation at every new arrival
COMPETITIVE RATIO
Oblivious Routing: A New Scheme
SOLUTION: Oblivious Routing: Every request is routed obliviously of the other requests !!PRE-‐PREPROCESSING: Routes are pre-‐computed !!MEASURE OF PERFORMANCE:
Worst-‐case ratio between congestion of oblivious-‐routing and optimal a posteriori routing
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
DIFFICULTY: Requests arrive online in arbitrary order How to avoid global flow computation at every new arrival
COMPETITIVE RATIO
Oblivious Routing: A New Scheme
SOLUTION: Oblivious Routing: Every request is routed obliviously of the other requests !!PRE-‐PREPROCESSING: Routes are pre-‐computed !!MEASURE OF PERFORMANCE:
Worst-‐case ratio between congestion of oblivious-‐routing and optimal a posteriori routing
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
DIFFICULTY: Requests arrive online in arbitrary order How to avoid global flow computation at every new arrival
SUBLINEAR COMPETITIVE RATIO
NEARLY-‐LINEAR RUNNING TIME
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed !Use basis given by low-‐stretch spanning tree
Regularize to softmax !Use basis given by oblivious routing scheme
Non-‐Euclidean Gradient Descent
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed !Use basis given by low-‐stretch spanning tree
Regularize to softmax !Use basis given by oblivious routing scheme
Coordinate Descent ?Non-‐Euclidean Gradient Descent
Euclidean Gradient Descent
Contour map of over feasible subspace
Euclidean Gradient Descent
Contour map of over feasible subspace
Euclidean Gradient Descent
Contour map of over feasible subspace
Euclidean Gradient Descent
Contour map of over feasible subspace
Euclidean Gradient Descent
Contour map of over feasible subspace
Euclidean Gradient Descent
Contour map of over feasible subspace
Euclidean Gradient Descent
Contour map of over feasible subspace
Euclidean Gradient Descent
Contour map of over feasible subspace
Euclidean Gradient Descent
Contour map of over feasible subspace
Euclidean Gradient Descent
Contour map of over feasible subspace
Non-‐Euclidean Gradient Descent
Contour map of over feasible subspace
Non-‐Euclidean Gradient Descent
Contour map of over feasible subspace
Non-‐Euclidean Gradient Descent
Contour map of over feasible subspace
Non-‐Euclidean Gradient Descent
Contour map of over feasible subspace
Non-‐Euclidean Gradient Descent
Contour map of over feasible subspace
Non-‐Euclidean Gradient Descent
Contour map of over feasible subspace
Non-‐Euclidean Gradient Descent
Contour map of over feasible subspace
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed !Use basis given by low-‐stretch spanning tree
Regularize to softmax !Use basis given by oblivious routing scheme
Coordinate Descent ?Non-‐Euclidean Gradient Descent
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed !Use basis given by low-‐stretch spanning tree
Regularize to softmax !Use basis given by oblivious routing scheme
Coordinate Descent ?Non-‐Euclidean Gradient Descent
NEARLY-‐LINEAR-‐TIME FOR BOTH PROBLEMS
Where Do We Go From Here? Future Directions and Open Problems
A New Algorithmic Approach• A novel design framework
for fast graph algorithms !
• Incorporates and leverages idea from multiple fields !
• Based on radically different approach !
• Has yielded conceptually simple, powerful algorithms !
• Combinatorial insight plays a crucial role -‐ Low-‐stretch spanning trees -‐ Oblivious routings !
• Numerous potential applications in Algorithms and other fields
Combinatorial Optimization !!
Numerical Scientific Computing !!ITERATIVE METHOD
PROBLEM REPRESENTATION
Limits of Nearly-‐Linear Time? Almost-‐Linear Time Super-‐linear Time
Reachability Shortest Path Connectivity
Minimum Cost Spanning Tree …
Directed Flow Problems Directed Cut Problems
!All-‐pair Shortest Path Network Design
…
Laplacian Systems of Linear Equations
Undirected Flow Problems: • s-‐t Maximum Flow • Concurrent Multi-‐commodity Flow • Oblivious Routing !
Undirected Cut Problems: • Minimum s-‐t cut • Approximate Sparsest Cut • Approximate Minimum Conductance Cut
Limits of Nearly-‐Linear Time? Almost-‐Linear Time Super-‐linear Time
Reachability Shortest Path Connectivity
Minimum Cost Spanning Tree …
Directed Flow Problems Directed Cut Problems
!All-‐pair Shortest Path Network Design
…
Laplacian Systems of Linear Equations
Undirected Flow Problems: • s-‐t Maximum Flow • Concurrent Multi-‐commodity Flow • Oblivious Routing !
Undirected Cut Problems: • Minimum s-‐t cut • Approximate Sparsest Cut • Approximate Minimum Conductance Cut
Limits of Nearly-‐Linear Time? Almost-‐Linear Time Super-‐linear Time
Reachability Shortest Path Connectivity
Minimum Cost Spanning Tree …
Directed Flow Problems Directed Cut Problems
!All-‐pair Shortest Path Network Design
…
?
Laplacian Systems of Linear Equations
Undirected Flow Problems: • s-‐t Maximum Flow • Concurrent Multi-‐commodity Flow • Oblivious Routing !
Undirected Cut Problems: • Minimum s-‐t cut • Approximate Sparsest Cut • Approximate Minimum Conductance Cut
Limits of Nearly-‐Linear Time? Almost-‐Linear Time Super-‐linear Time
Reachability Shortest Path Connectivity
Minimum Cost Spanning Tree …
Directed Flow Problems Directed Cut Problems
!All-‐pair Shortest Path Network Design
…
?
Laplacian Systems of Linear Equations
Undirected Flow Problems: • s-‐t Maximum Flow • Concurrent Multi-‐commodity Flow • Oblivious Routing !
Undirected Cut Problems: • Minimum s-‐t cut • Approximate Sparsest Cut • Approximate Minimum Conductance Cut
Recent Partial Progress: -‐ Improved running time for directed flow problems [Madry’13] !
-‐ Conditional lowerbounds for All-‐Pair Shortest Path [Williams’13]
Challenges in Optimization for Data Science
Recent Developments Beyond Graph Algorithms
• Interpretation of Nesterov’s Acceleration as Gradient + Mirror Descent [Allen-‐Zhu, O ’14]
APPLICATIONS: Faster Approximate Solver for Packing Linear Programs [Allen-‐Zhu, O ’15] Faster Parallel Algorithms for Positive Linear Programs [Allen-‐Zhu, O ’14] !• Coordinate Descent on Non-‐smooth Objectives to Construct Sparse Objects APPLICATIONS: Fast Graph and Matrix Sparsification [Allen-‐Zhu, Liao, O ’15] !!• Many more from computational complexity to quantum computing
7/2/15 42
