Datacenter Scale Load Balancing for MultipathTransport

Vladimir OlteanuUniversity Politehnica of Bucharest

Costin RaiciuUniversity Politehnica of Bucharest

AbstractMultipath TCP traffic is on the rise with recent deploy-ments by Apple and Samsung on mobile phones. De-spite this, MPTCP adoption on servers is falling behindand the major problem is that Multipath TCP does notwork with existing datacenter load balancers.

In this work we present MPLB, a distributed load bal-ancer for Multipath TCP traffic. MPLB uses statelesssoftware multiplexers to direct traffic to backend serversand is resilient to mux and network failures, as well asbackend server churn. We have implemented and testedMPLB, finding it can handle 6Mpps per machine withminimum-sized packets, seamlessly scale-out or in andgracefully handle mux failures.

1. INTRODUCTIONLoad balancing is an indispensable tool in modern

datacenters: Internet traffic must be evenly spread acrossthe frontend servers that deal with client requests, andeven internal datacenter traffic between different ser-vices is load balanced to ensure independent scaling andmanagement of the different services in the datacenter.

Multipath TCP (MPTCP) is a recent extension toTCP [5] that allows endpoints to utilize multiple pathsthrough the network in the same transport connectionand is a drop in replacement for TCP. Multipath TCPadoption is gaining pace. Apple IOS (and OSX) imple-ments MPTCP, Samsung and LG offer Android versionswith Multipath TCP implementations and several Ko-rean operators use MPTCP to“bond”LTE and 802.11acto achieve gigabit speeds on mobile phones. Current de-ployments use a proxy to terminate MPTCP and talkTCP to the servers, but this is only a stop-gap solution.

Permission to make digital or hard copies of all or part of this work for personalor classroom use is granted without fee provided that copies are not made ordistributed for profit or commercial advantage and that copies bear this noticeand the full citation on the first page. Copyrights for components of this workowned by others than the author(s) must be honored. Abstracting with credit ispermitted. To copy otherwise, or republish, to post on servers or to redistribute tolists, requires prior specific permission and/or a fee. Request permissions frompermissions@acm.org.

HotMIddlebox, August 22-26 2016, Florianopolis , Brazilc© 2016 Copyright held by the owner/author(s). Publication rights licensed to

ACM. ISBN 978-1-4503-4424-1/16/08. . . $15.00

DOI: http://dx.doi.org/10.1145/2940147.2940154

The next logical step is for MPTCP to be deployed indatacenter servers, but load balancer support is missing.

The job of a load balancer is to send the traffic froma single connection to the same server and is typicallyachieved by hashing on the packet five-tuple to decidethe destination server for both TCP and UDP traffic.There is currently no scalable load balancer solution forMultipath TCP traffic: an MPTCP connection can havean arbitrary number of subflows, each of which lookslike an independent TCP connection, and only SYNpackets contain connection-identifying information. Us-ing a hash of the five-tuple to load balance MPTCPsubflows would result in different subflows arriving atdifferent servers and breaking the protocol.

In this paper we design, implement and test MPLB, ascalable load balancer for Multipath TCP traffic. In oursolution, each load balancer acts completely indepen-dently and holds no per-connection state. This ensuresgreat scalability: we can add or remove load balancersvery quickly and without any impact on existing connec-tions, and can seamlessly tolerate failures. To supportstateless load balancing we developed a few novel tech-niques, including: a) load balancers choose connectionkeys instead of servers, b) servers can redirect packetsto other servers to ensure smooth handovers for scale-out and scale-in and c) each server encodes its identifierin the least-significant bits of the timestamp option.

We have implemented a prototype of MPLB and ranit on our local testbed. Our preliminary results showthat each mux can handle around 6Mpps MPTCP SYNpackets or around 26Gbps with Internet MTUs; MPLBcan quickly scale up and down to track demand and isrobust to failures.

2. BACKGROUNDServices in datacenters are assigned public IP ad-

dresses called VIPs (virtual IP). For each VIP, the ad-ministrator configures a list of private addresses of thedestination servers called DIPs (direct IPs). The job ofthe load balancer is to distribute connections destinedto the VIPs across all the DIPs.

Hardware load balancing appliances have long beenaround and are still in use in many locations; howeverthey are difficult to upgrade or modify and rather ex-

http://dx.doi.org/10.1145/2940147.2940154

MUX   MUX  …  

Server   Server   Server  …  

Border    router  

ECMP  VIP   VIP  

DIP1   DIP2   DIP3  

1 C   VIP  srcIP  dstIP  

2

3 VIP   DIP3   C   VIP  

payload  

4 VIP   C  C   VIP  

Figure 1: Load balancing: traffic to a VIP is spreadacross a pool of servers. Return traffic bypasses muxes.

src  dst  

IP  

X   80  src  dst  port  

S  flags  op2ons  

MPC  (KA)  

TCP    

B  A  80   X   S,A   MPC  (KB)  

X   80  

Y   80   S   JOIN(TB)  

80   Y   S,A   JOIN(TA)  

Y   80   DATA  

DATA  

Cellular    Interface  

Wifi  Interface  

B  A1  A1  B  

B  A1  

B  A2  A2  B  

B  A2  

Figure 2: MPTCP Operation: only SYN packets iden-tify the MPTCP connection.

pensive. That is why load balancers based on commod-ity hardware have been proposed [1–3,6,7,9,11]. Thereare two types of software load balancers: type 1, thatterminate the client TCP (or MPTCP) connections andopen new connections to the servers, and type 2, thatdo not terminate the connections.

Type 1 load balancers such as HAProxy [1] or Ng-inx [2] terminate TCP connections and could supportMPTCP by a simple kernel upgrade; unfortunately theyscale poorly since they must process both client-to-serverand server-to-client traffic (ten times larger then client-to-server traffic) and can only handle a limited numberof active connections (300K is an estimate for HAProxy).

Type 2 load balancers do not terminate TCP con-nections and can scale to large datacenters, and thisis the focus of this work. At a high level, our solutionuses the same architecture proposed by Ananta [11] andMaglev [3] and shown in Fig. 1. Load balancing is per-formed only for client-to-server traffic using a combi-nation of routing and software muxes running on com-modity x86 boxes. All muxes speak BGP to the borderdatacenter router and announce the VIPs they are incharge of as accessible in one hop. The border routerthen uses equal-cost multipath routing (ECMP) to splitthe traffic equally to these muxes. When a packet ar-rives at a mux, the mux decides by some mechanismwhich DIP it should be destined for.

Upon leaving the mux, the original packet is encap-sulated and sent to the DIP. The server decapsulatesthe packet, changes the destination address from VIPto DIP, and then processes it in its TCP stack. Whenthe stack generates a reply packet, the source addressis changed from DIP to VIP and the packet is sent di-rectly to the client, bypassing the mux (this is calledDirect Source Return, or DSR) to reduce its load.

Multipath TCP. Existing load balancer designs donot support Multipath TCP traffic. To understand theadditional requirements posed by MPTCP, we providea brief description of the relevant parts of the protocol.

Consider the example in Fig. 2. To start an MPTCPconnection, mobile A could use its cellular interface tosend to B a TCP SYN segment that has a multipathcapable option (MPC). The option signals to B that Awishes to use MPTCP for this connection and also con-veys KA, A’s key for this connection. To use MPTCP,B will reply with a SYN/ACK segment that also car-ries an MPC option with its own key, KB . The keys are

used by each endpoint to derive a token for the new con-nection; this token must be locally unique. The tokenis obtained by taking the 32 most significant bits of aSHA1 hash of the key, i.e. TA = MSB32(SHA1(KA)).

After the initial subflow is setup, A (or B) can nowadd subflows to the existing connection. A could useits Wifi interface to send a SYN/JOIN message, whichtells B that the new subflow is part of an existing con-nection. The JOIN option carries B’s token, allowing Bto associate the subflow to the correct connection.

When an MPTCP sender wishes to send data, it canuse any of the available subflows. The sender will createa TCP segment with the appropriate subflow addressesand ports, together with an option that informs the re-mote end of the connection level sequence number ofthis data (not shown). No connection identifying infor-mation is carried in data packets.

When load balancing MPTCP traffic, all subflows ofthe same connection must be sent to the same DIP. Ex-isting load balancers such as Ananta will treat MPTCPsubflows as independent TCP connections: a hash onthe five-tuple will decide the DIP for each subflow. Inmost cases the different subflows will be sent to differentservers, breaking all subflows except the first one.

3. OVERVIEW OF MPLBIdeally, muxes should not keep per flow state: the fate

of each packet should be decided independently, withoutany mux-specific state. If this were the case, any muxcould load balance any packet, and mux failures wouldhave no adverse effects on ongoing connections. Thechanges needed to ensure fault-tolerance underpin ourwhole design, so we present them first.

The statring point of our solution is very simple: in-stead of hashing the SYN and then remembering thedecision locally (as in Ananta or Maglev), muxes applya hash function to each packet and choose the targetserver by computing hash(5 − tuple)%N , where N isthe number of DIPs. As long as the set of DIPs doesn’tchange, the load balancing decision will be the same re-gardless of the mux, and mux failures or additions donot impact the flow-to-DIP allocations. Unfortunately,when a single server fails (or is added), most connec-tions will break because the modulus used changes.

Stable hashing. To avoid this issue, we add a levelof indirection by using “buckets” as follows. First, wechoose a number of buckets B that is strictly larger than

MUX                                                              

A   B   C  a)  Stable  hashing  with                    three  servers  

b)  Server  B  fails  

Fixed  number  of  buckets  

hash   MUX                                                              

A   B   C  X  X  

hash  

A   B   C  A   A   C   C  A  

Figure 3: Stable hashing is resilient to serverB failing: only B’s flows are moved.

A  

MUX  

C  B  

MUX  

SYN&data  FIN   FIN  

Figure 4: Load balanc-ing MPTCP statelessly

MUX  2                                                            

MUX  1                                                            

B  A   B  a)  Bucket  assigned  to  A   b)  Bucket  moved  to  B,  

Inconsistent  mappings.  

B   B  A   B   B  MUX  1                                                          

 B   B  MUX  2                                                          

  B  A   B   B  

A  

A  

Figure 5: Daisy chaining enables scalingwithout affecting ongoing flows.

N , the number of servers. Each bucket is“assigned”to asingle server, and one server may have multiple bucketsassigned. The number of buckets and the buckets toserver assignments are known by all muxes, and theyare disseminated via a separate mechanism (see below).When a packet arrives, muxes hash the connection toa bucket by computing b = hash(5 − tuple)%B, andthen forward the packet to the server currently assignedbucket b. As the set of buckets is constant, server churndoes not affect the hashing result: if a server fails, onlythe flows it hosted will be affected.

A centralized controller manages the mappings be-tween buckets and servers. These mappings change in-frequently: on server failure or explicitly by the admin-istrator for load-balancing and maintenance purposes.A discussion of how the mappings are managed is be-yond the scope of this paper.

We show an example of stable hashing in Figure 3.When server B fails, the controller will map the bucketto server C; the mapping is then disseminated to allmuxes. After the mapping is updated, flows initiallygoing to A or C are unaffected, and flows destined forB are now sent to C. Only the connections that werealready open on B before the failure are broken.

3.1 Load balancing Multipath TCPMPTCP load balancing is hard because there is no

discernible relationship between the initial connectionSYN sent by the client and the following subflow SYNpackets, as shown in Fig. 2: the initial SYN containsA’s key, and the second subflow SYN contains B’s token.

There are two possible types of solutions to solve theproblem. When a new connection is setup, B could in-form the associated mux of its token (either directly, orby using a distributed key-value store). Unfortunately,such solutions either force the mux to store per-flowstate, reducing fault tolerance, or add latency to everypacket because of distributed state lookups.

A better solution is for muxes to load balance JOINpackets using the token directly: treat the token as ahash and select a DIP by performing a lookup in thebuckets array (i.e. token%B). For this solution to workcorrectly, all connections destined to server B must havea token (chosen at connection startup) that points toB. This is not the case for MPTCP: tokens used by Bdepend on the randomly generated per-connection key.

A brute force solution is for B to randomly generatemultiple keys instead of just one per connection and

select the first key that generates an appropriate token.The time complexity of this method depends on thenumber of DIPs and is quite high: 2ms per connectionwhen load balancing across 1000 DIPs.

Instead, we move key generation to the muxes: themux will deterministically generate a key for every newconnection, hash it to find the token and the appropri-ate DIP. The key is then communicated to the DIP inthe encapsulated packet. The DIP does not generate anew key for the MPTCP connection, and simply usesthe one provided by the mux.

To load balance data packets in a stateless way weuse two separate mechanisms, shown in Figure 4.1. Redirect traffic. When a new subflow is setup, themux computes a hash of the five-tuple and selects theDIP (say C) that is responsible for the subflow underplain five-tuple hashing. It communicates this DIP toB, who in turn, instructs C to forward to B all packetsit receives for that five-tuple. Redirection is enough toensure correctness; however it is also inefficient: trafficwill hit two servers instead of one.2. Embed the token in regular segments. To reduce thecosts of redirection, we change the server stack to em-bed the token in its outgoing segments and “trick” thestandard MPTCP client to echo these tokens in returntraffic. In particular, our servers embed the token inleast significant bits of the timestamp option carriedby most TCP segments except the FIN. Clients echothese values sent by the server in its timestamp op-tions, and the muxes then use the least significant bitsof the TSecr part of the timestamp to properly loadbalance the packets. Only packets that do not havethe timestamp options are load balanced using the five-tuple and thus redirected (FIN packets). More details ofthe timestamp implementation, including a discussionon its correctness and safety, are given below.

The pseudocode of our mux algorithm is in Fig. 6.This algorithm has very small memory usage as for mostpackets in the fast path it does one packet memory ac-cess to retrieve the token and then indexes in the buck-ets array called srv, where each entry stores the DIPaddress of the server in charge of that bucket. Thepacket is then encapsulated and sent to the server. ForSYN(JOIN) packets, the mux does more work: it hashesthe five-tuple and informs the server of the redirectionpoint. Finally, SYN(MPC) packets are the most expen-sive, since a key is generated and the token is computed.

if (MP_CAPABLE) {k = SHA1(fivetuple, client_key, secret);t = SHA1(k);h = hash(fivetuple);dip = srv[t % BUCKETS];redir_dip = srv[h % BUCKETS];encap(p, k, redir_dip, dip);

} else if (MP_JOIN) {t = retrieve from JOIN option;h = hash(fivetuple);dip = srv[t % BUCKETS];redir_dip = srv[h % BUCKETS];encap(p, redir_dip, dip);

} else {if (timestamp)

t = LSB13(timestamp);else

t = hash(fivetuple);dip = srv[t % BUCKETS];encap(p,dip);

}

Figure 6: Mux load balancing algorithm. No per-flowstate is used to decide the fate of packets.

Daisy-chaining for smooth server handover. Thereis a natural amount of churn of servers behind a VIP dueto failures, scaling-out or in or planned maintenance.To allow smooth handovers, we need a mechanism tomigrate connections from a source server (A) to a tar-get one (say B). Supporting migration is easy for thecontroller: it must simply re-map the A’s buckets to Band then inform the muxes. The muxes will then startsending the bucket traffic to B.

For a truly smooth migration, however, there are twocomplications that need to be taken in account: existingconnections at A will be broken during the migrationand there may be temporary disagreements amongstmuxes in the buckets-to-servers mappings. To solveboth issues we use daisy chaining, a transitory periodwhere both the new server and the old one are activeand servicing flows that hit the migrated bucket, asshown in Fig. 5. We aim to move all new connec-tions to B, the new server, but allow old connections tocontinue by forwarding them to A.

To start daisy chaining, the controller informs bothservers of the bucket migration. When daisy-chaining,B will: 1) Locally service packets if they are SYN(MPC)or belong to a local connection. 2) Redirect the packetsto the appropriate server if they match an MPTCP-redirect rule, and 3) Daisy-chain: send all other packetsto server A. Server A will process traffic as usual untilall of its ongoing connections are closed or have timed-out. At this point A informs B there is no more needto do daisy chaining, completing the migration.

Daisy chaining adds robustness to our whole design.Consider what happens if the two muxes in Fig. 5 tem-porarily disagree on the server now in charge of thebucket being moved. Flows that hit mux 1 are load bal-anced according to the old mapping and will be directedto A, which processes them as usual (the black flow).Meanwhile, B will locally service the red connection andforward the blue connection to A. This robustness re-

moves the need for strict synchronization of all muxeswhen buckets are remapped.

Embedding the token in timestamps. We needto encode the MPTCP token in every packet to allowmuxes to forward MPTCP correctly without costly redi-rection. To ensure quick datacenter adoption, we seeka solution that is deployable at the servers and does notrequire client or protocol changes.

We rely on TCP timestamps for our purpose. Times-tamps are present in all packets except the FIN pack-ets and contain two parts, one set by the sender calledTSval and another called TSecr that contains the valuemost recently received from the remote end. In particu-lar, the server embeds X in the TSval part of the times-tamp on all its outgoing segments, where X is mono-tonically increasing in time, and the client will echo themost recent X it has seen in return packets.

In our solution, the server simply embeds the thirteenlowest order bits of the MPTCP token in the lowestorder bits of the TSval field as follows:

TSval = 0xFFFFFFFF & (TSval

CoresSYN Data (86B)

TCP MPC JOIN TS TS(Worst) No TS1 4.07 1.01 3.64 4.36 3.79 4.232 6.48 2.02 6.65 6.99 6.89 7.003 6.46 2.98 6.92 6.98 6.96 6.966 6.30 5.87 6.62 6.69 6.69 6.67

Table 1: Mux performance vs. packet type (Mpps)

4. PRELIMINARY EVALUATIONWe have implemented the mux in the Click modular

router suite [4] and we run it using the FastClick distri-bution that relies on netmap [12] to bypass the kernelfor improved performance. Our servers run Linux ker-nel 3.18 with the MPTCP patch (version 0.90) modifiedsuch that the server uses the connection keys generatedby the mux and tokens are embedded in all outgoingtimestamps. We have also implemented a host agentkernel module that decapsulates and encapsulates pack-ets, applies redirection and daisy chaining rules. Theserules are managed by a userspace daemon.

We have tested the performance and correctness ofour prototype in a small testbed.

Performance evaluation. We first tested a mux inisolation. The server we used has a six core Intel XeonE5645 processor running at 3GHz, 8GB of RAM andtwo dual-port ten gigabit Intel 82599 NICs.

Our traffic generator is based on the pktgen utilityfrom the netmap [12] suite: we modified it to gener-ate multiple types of packets. The traffic generator cansaturate a 10Gbps link with minimum sized packets. Ineach experiment we generate a single packet type andwe measure performance at the receiver using pktgen.

Table 1 lists performance of our software multiplexer.As expected, the single core results show that process-ing SYN(MPC) packets is the most expensive: the muxdoes two SHA1 computations and packet encapsulation.In comparison, for all other packet types the encapsu-lation is the most expensive operation

Processing MPTCP SYN(JOIN)s is slightly more ex-pensive than regular TCP SYN packets, as we have toperform one additional hash operation and encapsulateone more field. For smallest-size data packets (86B),direct hashing (shown as “no timestamp”) is slightlyslower than using timestamps for load balancing, be-cause of the hash calculation. We also measure a worst-case scenario for the timestamp option, placing it atthe end of the options field and preceding it with nops:throughput is now 3.79Mpps, 14% lower compared tothe default placement used by Linux.

We then increased the number of cores used to servicethe single NIC, spreading the NIC queues across all theparticipating cores. The results for two cores show anincrease in throughput for all packet types. From threecores and up, only the SYN(MPC) packets improve inperformance, and performance for other types of pack-ets hits a plateau at around 6Mpps. To understand theissue, we kept removing complexity until we were leftrunning only the netmap bridge, but the problem still

0

5

10

15

20

25

30

64 128 256 512 1024 0

2

4

6

8

10

12

14

Thro

uhgput (G

bps)

Packet ra

te (

Mpps)

Packet size (B)

ThroughputPacket Rate

Figure 10: Mux forwarding performance vs. packet size.Our prototype can forward 26Gbps with 512B packets.

persisted; the netmap paper reports a similar pathologyon the receive path: when packet sizes are between 65Band 127B receive throughput drops to 7Mpps (see [12]Fig. 6). The bottleneck is, most likely, due the highnumber of PCIe transactions.

All our experiments so far have used minimum sizedTCP packets and a single NIC. In our next experiment,we measured throughput for a variety of packet sizesin a mux with four ten gigabit NICs. The results arepresented in figure 10 and show that the mux saturatesthree ten gigabit NICs when packet sizes reach 512B,and that performance for 86B TCP packets is around11Mpps, confirming our PCIe link bottleneck theory.

How many servers can one of our muxes handle?Muxes are mainly used to handle HTTP requests fromclients, so we can get an estimate by examining real-lifeweb traffic characteristics. We examined recent traf-fic traces from the MAWI backbone link, finding thatclient to server traffic (to be serviced by the muxes) isoverwhelmingly composed of small packets: more than90% of packets are 128B or less. Of these, only 5% areSYN or FIN packets; the wide majority are TCP acksand small TCP data segments. These findings, corrobo-rated with the performance results above imply that themux should handle close to 10Gbps of client-to-servertraffic. The MAWI traces also show that server-to-clienttraffic is 15 times larger than client to server traffic: asingle mux can, in principle, load balance for a pool ofservers that together serve 150Gbps of downlink traf-fic. We expect one server to source around 1-10Gbps oftraffic, so a single mux should cater for 15-150 servers.

Ring size. In Fig. 7 we vary the ring size and measurethe throughput achieved when forwarding SYN(MPC)packets. Ring size directly affects the performance ofbatching heavily used in FastClick and netmap: smallerrings will generate more interrupts and PCIe transac-tions, reducing performance; ring sizes above 256 slotsdon’t help throughput but increase latency.

Latency. Have we sacrificed packet latency in our pur-suit of speed? We setup an experiment where our muxis running on a single core and processing SYN(MPC)packets sent at different rates. In parallel, we run a pingwith high frequency between two idle machines, and theecho request passes unmodified through the mux. Weshow a CDF of ping latency for different packet ratesin Figure 8. As long the CPU is not fully utilized, me-

0

2

4

6

8

10

64 128 256 512 1024 2048

Thro

ughput (M

pps)

Interface ring size

1 core2 cores3 cores6 cores

Figure 7: Effect of NIC ring size onperformance.

0

20

40

60

80

100

0.01 0.1 0.5 1 2 3 5 10

CD

F (

%)

Ping latency (ms)

14% loss

10% CPU,100Kpps50% CPU,500Kpps

90% CPU,0.9Mpps100% CPU,1.1Mpps

Figure 8: Latency induced by ourmux

0

1

2

3

4

5

6

7

0 5 10 15 20 25 30 35 40

Packet ra

te (

Mpps)

Time (s)

Figure 9: Scaling out muxes instantlyincreases available capacity.

dian and worst-case packet latencies stay below 0.5ms.When we overload the mux, the latency jumps to 3-4msand 14% of packets are dropped. The 4ms latency is aworst case and is mostly due the time it takes one coreto process all the packets stored in the 6 receive queuesused by netmap (256 packets per queue). With appro-priate provisioning, overloading the mux CPU can beavoided, and the latency inflation is at most 0.5ms.

Timestamp evaluation. To ensure our timestampmodifications are safe, we ran experiments varying theRTT between 1-500ms and loss rates between 0.01% to5%, comparing the throughput of single-path MPTCPusing regular timestamps or the modified timestamps.Our timestamp changes are benign: in all experimentsthe differences in throughput are under 10%-20%, andexplained by typical TCP performance variability.

Scaling out. In a large datacenter, scaling out entailsstarting a new mux and sending a BGP announcementfor the VIPs it serves. The border routers will then starthashing traffic to the new mux. We have a local clusterwhere we run experiments on machines connected byan Openflow-enabled IBM G8264 Rackswitch. In ourcontext, using BGP is overkill; instead, we insert staticrules in the switch to load balance traffic across ourmuxes using the source address prefix for redirection.

We generate and spread 10Gbps of SYN(MPC) pack-ets across our muxes, which forward them to a singleDIP measuring throughput. In Fig. 9 we start with asingle mux running on one core and processing 1Mpps;after that, every 10s we start one more mux and splitthe traffic across muxes. With every additional mux,there is an increase of 1Mpps of processed traffic, asexpected. In a large scale deployment we expect to seemore latency before the capacity increases, perhaps onthe order of seconds, due to BGP propagation delays.

5. RELATED WORKPaasch et. al [10] discuss the problems posed by Mul-

tipath TCP traffic to datacenter load balancers. Theiranalysis focuses on ensuring SYN(MPC) and SYN(JOIN)packets reach the same server, and it assumes muxeskeep per flow state after the initial placement decision.They provide two possible solutions to ensure the SYN(JOIN)token can be used to select the correct server: changethe MPTCP handshake mechanism such that the tokenis announced explicitly (and not derived from the ses-sion key) or change the way the token is derived from

the key, using a block cipher instead of a hash function.Our solution sidesteps these changes to the MPTCPprotocol by having muxes choose per connection keys,and also allows muxes to be stateless for data packets.

6. CONCLUSIONSLoad balancing Multipath TCP is a necessity, yet de-

signing a scalable load balancer is far from trivial. Wehave used three main ideas to tackle this challenge: sta-ble hashing via buckets, generating connection keys atmuxes and embedding the MPTCP token in TCP times-tamps on every packet. We have designed and imple-mented MPLB, a stateless software load balancer thatcan scale seamlessly and tolerates mux faults withoutdropping any connections. Our prototype can forward10Mpps for an expected HTTP client-to-server load ona single modest Xeon machine with two NICs.

AcknowledgementsThe authors would like to thank Mark Handley for hissuggestion to encode the server identifier in the times-tamp. This work was partly funded by SSICLOPSH2020 (644866) and UEFISCDI project Mobil4 (11/2012).

7. REFERENCES[1] HAProxy - The Reliable, High Performance TCP/HTTP

Load Balancer. http://www.haproxy.org/, 2016.

[2] NGINX - HTTP Server. http://nginx.org/en/, 2016.[3] Daniel E. Eisenbud et al. Maglev: A fast and reliable

software network load balancer. In NSDI, Santa Clara, CA,2016.

[4] E. Kohler et al. The Click modular router. ACM Trans.Computer Systems, 18(1), 2000.

[5] Ford, Alan et al. RFC6824:TCP Extensions for MultipathOperation ... https://tools.ietf.org/html/rfc6824.

[6] Gandhi Rohan et al. Duet: Cloud scale load balancing withhardware and software. In SIGCOMM, 2014.

[7] Gandhi Rohan et al. Rubik: Unlocking the power oflocality ... In ATC, 2015.

[8] V. Jacobson, R. Braden, and D. Borman. RFC 1323: TCPExtensions for High Performance, May 1992.

[9] Nanxi Kang et al. Efficient traffic splitting on commodityswitches. In CONEXT, 2015.

[10] Paasch, Christoph et.al. MPTCP behind Layer-4loadbalancers (ID). draft-paasch-mptcp-loadbalancer-00,Sep 2015.

[11] Patel, Parveen et. al. Ananta: Cloud scale load balancing.In SIGCOMM, 2013.

[12] L. Rizzo. netmap: A novel framework for fast packet i/o. InProc. USENIX Annual Technical Conference, 2012.

http://www.haproxy.org/http://nginx.org/en/https://tools.ietf.org/html/rfc6824

IntroductionBackgroundOverview of MPLBLoad balancing Multipath TCP

Preliminary EvaluationRelated workConclusionsReferences