Practical Network Coding for the Internet and Wireless Networks

Philip A. Chouwith thanks to Yunnan Wu, Kamal Jain, Pablo Rodruiguez

Rodriguez, Christos Gkantsidis, and Baochun Li, et al.

Globecom Tutorial, December 3, 2004

Outline Practical Network Coding

Packetization Buffering Results for SprintLink ISP

Experimental comparisons to multicast routing

Internet and Wireless Network Applications Live Broadcasting, File Downloading, Messaging,

Interactive Communication

Network Coding applicable to real networks?

Internet IP Layer

Routers (e.g., ISP) Application Layer

Infrastructure (e.g., CDN) Ad hoc (e.g., P2P)

Wireless Mobile ad hoc multihop wireless networks Sensor networks Stationary wireless (residential) mesh networks

Theory vs. Practice Theory:

Symbols flow synchronously throughout network; edges have integral capacities

Practice: Information travels asynchronously in

packets; packets subject to random delays and losses; edge capacities often unknown, varying as competing communication processes begin/end

Theory vs. Practice (2) Theory:

Some centralized knowledge of topologyto compute capacity or coding functions

Practice: May be difficult to obtain centralized

knowledge, or to arrange its reliable broadcast to nodes across the very communication network being established

Theory vs. Practice (3) Theory:

Can design encoding to withstand failures,but decoders must know failure pattern

Practice: Difficult to communicate failure pattern

reliablyto receivers

Theory vs. Practice (4) Theory:

Cyclic graphs present difficulties, e.g., capacity only in limit of large blocklength

Practice: Cycles abound. If A → B then B → A.

Need to address practical network coding in real networks

Packets subject to random loss and delay Edges have variable capacities due to

congestion and other cross traffic Node & link failures, additions, &

deletions are common (as in P2P, Ad hoc networks)

Cycles are everywhere Broadcast capacity may be unknown No centralized knowledge of graph

topology or encoder/decoder functions Simple technology, applicable in practice

Approach Packet Format

Removes need for centralized knowledge of graph topology and encoding/decoding functions

Buffer Model Allows asynchronous packets arrivals &

departures with arbitrarily varying rates, delay, loss

[Chou, Wu, and Jain, Allerton 2003][Ho, Koetter, Médard, Karger, and Effros, ISIT 2003]

Standard Framework Graph (V,E) having unit capacity edges Sender s in V, set of receivers T={t,…} in V Broadcast capacity h = mint Maxflow(s,t)

y(e) = ∑e’ e(e’) y(e’) (e) = [e(e’)]e’ is local encoding vector

s v

y(e’)

te

y(e’1) = x1

y(e’h) = xh

y(e’2) = x2

... ... ...

y(e1)

y(e2)

y(eh)

y(e)

Global Encoding Vectors

By induction y(e) = ∑hi=1 gi(e) xi

g(e) = [g1(e),…,gh(e)] is global encoding vector Receiver t can recover x1,…,xh from

h

t

hhhh

h

h x

x

G

x

x

egeg

egeg

ey

ey

11

1

1111

)()(

)()(

)(

)(

s v

y(e’)

te

y(e’1) = x1

y(e’h) = xh

y(e’2) = x2

... ... ...

y(e1)

y(e2)

y(eh)

y(e)

Invertibility of Gt

Gt will be invertible with high probabilityif local encoding vectors are randomand field size is sufficiently large If field size = 216 and |E| = 28

then Gt will be invertible w.p. ≥ 1−2−8 = 0.996

[Ho et al., 2003][Jaggi, Sanders, et al., 2003]

Packetization

Internet: MTU size typically ≈ 1400+ bytes y(e) = ∑e’ e(e’) y(e’) = ∑h

i=1 gi(e) xi s.t.

Nhhh

N

t

hNhh

N

h xxx

xxx

G

eyeyey

eyeyey

e

e

,2,1,

,12,11,1

21

112111

)()()(

)()()(

)(

)(

y

y

s

x1=[x1,1,x1,2,…,x1,N]

...

x2=[x2,1,x2,2,…,x2,N]

xh=[xh,1,xh,2,…,xh,N]

v

y(e’)

te

... ...

y(e1)

y(e2)

y(eh)

y(e)=[y1(e),y2(e),…,yN(e)]

Key Idea Include within each packet on edge eg(e) = ∑e’ e(e’) g(e’); y(e) = ∑e’ e(e’) y(e’)

Can be accomplished by prefixing i th unit vector to i th source vector xi, i=1,…,h

Then global encoding vectors needed to invert the code at any receiver can be found in the received packets themselves!

Nhhhh

N

t

hNhhhhh

Nh

xxx

xxx

G

eyeyeyegeg

eyeyeyegeg

,2,,

,12,11,1

211

11211111

10

01

)()()()()(

)()()()()(

Cost vs. Benefit Cost:

Overhead of transmitting h extra symbolsper packet; if h = 50 and field size = 28,then overhead ≈ 50/1400 ≈ 3%

Benefit: Receivers can decode even if

Network topology & encoding functions unknown Nodes & edges added & removed in ad hoc way Packet loss, node & link failures w/ unknown

locations Local encoding vectors are time-varying & random

Erasure Protection Removals, failures, losses, poor random

encoding may reduce capacity below h

Basic form of erasure protection:send redundant packets, e.g.,last hk packets of x1,…, xh are known zero

Nhhhh

Nhk

t

kNkkkhk

Nh

xxx

xxx

G

eyeyeyegeg

eyeyeyegeg

,2,,

,12,11,1

211

11211111

10

01

)()()()()(

)()()()()(

Priority Encoding Transmission (Albanese et al., IEEE Trans IT ’96)

More sophisticated form: partition data into layers of importance, vary redundancy by layer

Received rank k → recover k layers Exact capacity can be unknown1 2 3 4 5 Data layer h=6Global encoding vectors

...1 0 0 x1,10 0 0 x1,N...x1,2 x1,3 x1,4 x1,5 x1,6 x1,7 x1,8 x1,11 x1,12 Source packet 1

0 1 0 00 0 0 x2,2 x2,N...x2,3 x2,4 x2,5 x2,6 x2,7 x2,8 ... x2,11 x2,12 Source packet 2

0 0 1 0 00 0 0 x3,3 x3,N...x3,4 x3,5 x3,6 x3,7 x3,8 ... x3,11 x3,12 Source packet 3

1 0 0 0 00 0 0 00 x4,5 x4,N...x4,6 x4,7 x4,8 ... x4,11 x4,12 Source packet 4

1 0 0 0 0 0 0 00 0 0 0 0 x5,8 x5,N...x5,9 x5,10 x5,11 x5,12 ...

1 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 x6,Nx6,12 ... Source packet h=6

Asynchronous Communication

In real networks, “unit capacity” edges grouped Packets on real edges carried sequentially Separate edges → separate prop & queuing delays Number of packets per unit time on edge varies

Loss, congestion, competing traffic, rounding Need to synchronize

All packets related to same source vectors x1,…, xh

are in same generation; h is generation size All packets in same generation tagged with same

generation number; one byte (mod 256) sufficient

Buffering

random combinatio

n

Transmission opportunity: generate packet

buffer

node

arriving packets (jitter, loss, variable rate)

asynchronous

transmission

asynchronous reception

edge

edgeedge

edge

Decoding Block decoding:

Collect h or more packets, hope to invert Gt

Earliest decoding (recommended): Perform Gaussian elimination after each packet

At every node, detect & discard non-informative packets

Gt tends to be lower triangular, so can typically decode x1,…,xk with fewer more than k packets

Much lower decoding delay than block decoding Approximately constant, independent of block length h

Flushing Policy, Delay Spread,and Throughput loss

Policy: flush when first packet of next generation arrives on any edge Simple, robust, but leads to some

throughput loss

Time of arrival at node v

s vFastest path

Slowest path

...

...

... ...

...

...

Flushing time

X

X X X

Delay spread

Ih

(pkt/s) rate sending (s) spreaddelay

(s)duration generation

(s) spreaddelay (%) loss throughput

Interleaving Decomposes session into several

concurrent interleaved sessions with lower sending rates

Does not decrease overall sending rate Increases space between packets in

each session; decreases relative delay spread

0 0 0 201 1 1 1 2 2 23 3 3 3 ...

Simulations Implemented event-driven simulator in C++ Six ISP graphs from Rocketfuel project (UW)

SprintLink: 89 nodes, 972 bidirectional edges Edge capacities: scaled to 1 Gbps / “cost” Edge latencies: speed of light x distance

Sender: Seattle; Receivers: 20 arbitrary (5 shown) Broadcast capacity: 450 Mbps; Max 833 Mbps Union of maxflows: 89 nodes, 207 edges

Send 20000 packets in each experiment, measure: received rank, throughput, throughput loss, decoding

delay vs.sendingRate(450), fieldSize(216), genSize(100), intLen(100)

Received Rank

0 20 40 60 80 100 120 140 160 180 20040

50

60

70

80

90

100

Generation number

Re

ceiv

ed

ra

nk

Chicago (450 Mbps)Pearl Harbor (525 Mbps)Anaheim (625 Mbps)Boston (733 Mbps)SanJose (833 Mbps)

0 5 10 15 20 2540

50

60

70

80

90

100

Field size (bits)

Avg

. re

ceiv

ed

ra

nk

Chicago (450 Mbps)Pearl Harbor (525 Mbps)Anaheim (625 Mbps)Boston (733 Mbps)SanJose (833 Mbps)

Throughput

400 450 500 550 600 650 700 750 800 850400

450

500

550

600

650

700

750

800

850

Sending rate (Mbps)

Th

rou

gh

pu

t (M

bp

s)

Chicago (450 Mbps)Pearl Harbor (525 Mbps)Anaheim (625 Mbps)Boston (733 Mbps)SanJose (833 Mbps)

400 450 500 550 600 650 700 750 800 850400

450

500

550

600

650

700

750

800

850

Sending rate (Mbps)

Th

rou

gh

pu

t (M

bp

s)

Chicago (450 Mbps)Pearl Harbor (525 Mbps)Anaheim (625 Mbps)Boston (733 Mbps)SanJose (833 Mbps)

Throughput Loss

20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60

70

80

90

100

Generation Size

Th

rou

gh

pu

t Lo

ss (

Mb

ps)

Chicago (450 Mbps)Pearl Harbor (525 Mbps)Anaheim (625 Mbps)Boston (733 Mbps)San Jose (833 Mbps)

0 10 20 30 40 50 60 70 80 90 1000

50

100

150

200

250

300

Interleaving Length

Th

rou

gh

pu

t Lo

ss (

Mb

ps)

Chicago (450 Mbps)Pearl Harbor (525 Mbps)Anaheim (625 Mbps)Boston (733 Mbps)SanJose (833 Mbps)

Decoding Delay

20 30 40 50 60 70 80 90 1000

20

40

60

80

100

120

140

160

180

200

Generation Size

Pa

cke

t de

lay

(ms)

Pkt delay w/ blk decodingPkt delay w/ earliest decoding

0 10 20 30 40 50 60 70 80 90 1000

20

40

60

80

100

120

140

160

180

200

Interleaving Length

Pa

cke

t de

lay

(ms)

Pkt delay w/ blk decodingPkt delay w/ earliest decoding

Distributed Flow Algorithms

Work in progress…

Comparing Network Codingto Multicast Routing

Throughput Computational complexity Resource cost (efficiency) Robustness to link failure Robustness to packet loss

Multicast Routing

Standard multicast: union of shortest reverse paths Widest multicast: maximum rate Steiner tree Multiple multicast: Steiner tree packing

sender

receivers

Complexity and Throughput of Optimal Steiner Tree Packing Packing Steiner trees optimally is hard [e.g., JMS03] Network Coding has small throughput advantage

[LLJL04]:

Z. Li, B. Li, D. Jiang, and L. C. Lau, On achieving optimal end-to-end throughput in data networks: theoretical and emprirical studies, ECE Tech. Rpt., U. Toronto, Feb. 2004,

reprinted with permission.

Complexity and Throughput of Optimal Steiner Tree Packing (2)

Z. Li, B. Li, D. Jiang, L. C. Lau ‘04: Optimally packed Steiner trees in 1000

randomly generated networks (|E|<35) Network Coding advantage = 1.0 in every

network “The fundamental benefit of network

coding is not higher optimal throughput, but to facilitate significantly more efficient computation and implementation of strategies to achieve such optimal throughput.”

Complexity ofGreedy Tree Packing Find widest (max rate) distribution tree

in polynomial time [Prim]: Grow tree edge by edge Postprocess tree

Repeatedly pack widest distribution trees

Reached nodes

Non-reachednodes

Choose the edge with maximum capacity

Comparison of Achievable Throughput

0100200300400

500600700800

1 2 3 4 5 6

ISP#

(Mb

ps

)

Path-packing

Max-rate Steiner tree (Prim's algorithm)

Greedy tree-packing based on Prim's algorithm

Greedy tree-packing based on Lovasz's proof to Edmonds' theorem

Practical network coding

Multicast capacity [Wu, Chou, and Jain, ISIT 2004]

Comparison of Efficiency(#receivers * throughput / bandwidth cost)

Z. Li, B. Li, D. Jiang, and L. C. Lau, On achieving optimal end-to-end throughput in data networks: theoretical and emprirical studies, ECE Tech. Rpt., U. Toronto, Feb. 2004,

reprinted with permission.

Network codingWidest tree

Robustness to Link Failure Ergodic link failure: packet loss Non-ergodic link failure: G G-e

Compute throughput achievable by Network Coding

Multicast capacity of G-e Adaptive Tree Packing

Re-pack trees on G-e Fixed Tree Packing

Keep trees packed on G Delete subtrees downstream from e

[Wu, Chou, and Jain, ISIT 2004]

Robustness to Link Failures

Robustness to Packet Loss Ergodic link failure:

Routing requires link-level error control to achieve throughput 95% of sending rate

Network Coding obviates need for link-level error control

Outputs random linear combinations at intermediate nodes

Delays reconstruction until terminal nodes

5% loss 5% loss

Comparing Network Codingto Multicast Routing

Throughput √ Computational complexity √ Resource cost (efficiency) √ Robustness to link failure √ Robustness to packet loss √

Manageability

Network Coding for Internet and Wireless Applications

AkamaiRBN

ALM

Sp

litStr

ea

mC

oopN

et

Dig

ital

Fou

nta

in

Gn

ute

lla

Kaza

a

Bit

Torr

ent

Peer

Net

Win

dow

s M

ess

enge

r

Xbox

LiveAd Hoc

(P2P)

Infra-structure (CDN)

Inst

ant

Mes

sagi

ngwireles

sInterne

t Live

broa

dcas

t

Med

ia o

n

dem

and

Downl

oad

from

pee

r

Coded

downl

oad

Para

llel

downl

oad

Inte

ract

ive

Com

mun

icatio

n

(con

fere

ncin

g,

g

amin

g)

File download

Live Broadcast State-of-the-art: Application Layer Multicast

(ALM) trees with disjoint edges (e.g., CoopNet) FEC/MDC striped across trees

Up/download bandwidths equalizeda failed node

Live Broadcast (2) Network Coding [Jain, Lovász, and Chou, 2004] :

Does not propagate losses/failures beyond child

ALM/CoopNet average throughput: (1–ε)depth * sending rateNetwork Coding average throughput: (1–ε) * sending rate

failed node

affected nodes(maxflow: ht → ht – 1)

unaffected nodes(maxflow unchanged)

File Download State-of-the-Art: Parallel download (e.g.,

BitTorrent) Selects parents at random Reconciles working sets Flash crowds stressful

Network Coding: Does not need to reconcile working sets Handles flash crowds similarly to live broadcast

Throughput download time Seamlessly transitions from broadcast to download mode

File Download (2)

Mean

Max

LR Free 124.2

161

LR TFT 126.1

185

FEC Free

123.6

159

FEC TFT 127.1

182

NC Free 117.0

136

NC TFT 117.2

139C. Gkantsidis and P. Rodriguez Rodruiguez, Network Coding for large scale content

distribution, submitted to INFOCOM 2005, reprinted with permission.

Instant Messaging State-of-the-Art: Flooding (e.g., PeerNet)

Peer Name Resolution Protocol (distributed hash table) Maintains group as graph with 3-7 neighbors per node Messaging service: push down at source, pops up at

receivers

How? Flooding Adaptive, reliable 3-7x over-use

Network Coding: Improves network usage 3-7x (since all packets

informative) Scales naturally from short message to long flows

Interactive Communication in mobile ad hoc wireless networks State-of-the-Art: Route discovery and

maintenance Timeliness, reliability

Network Coding: Is as distributed, robust, and adaptive as flooding

Each node becomes collector and beacon of information Can also minimize energy

a+b a+bba

ba

a

a

b

b

a

aa

a

a a

Network Coding Minimizes Energy (per bit)

Wu et al. (2003); Wu, Chou, Kung (2004)Lun, Médard, Ho, Koetter (2004)

optimal multicast

transmissions per packet = 5

network codingtransmissions per

packet = 4.5

a

a a,b

a a ba,b b,a

Network Coding in Residential Mesh Network (simulation)

a+b a+b

Physical Piggybacking

Information sent from t to s can be piggybacked on information sent from s to t

Network coding helps even with point-to-point interactive communication

throughput energy per bit delay

a b

s t

Network Coding in Residential Mesh Network (multi-session)

Network Coding with Cross-Layer Optimization

Network Coding has > 4x bits per Joule

Summary Network Coding is Practical

Packetization Buffering

Network Coding outperforms Routing on Throughput Computational complexity Resource cost (efficiency) Robustness to link failure Robustness to packet loss

Network Coding can improve performance in IP or wireless networks in infrastructure-based or P2P networks for live broadcast, file download, messaging, interactive

communication applications