The FuzzyLog: A Partially Ordered Shared Log The FuzzyLog: A Partially Ordered Shared Log Joshua Lockerman

  • View

  • Download

Embed Size (px)

Text of The FuzzyLog: A Partially Ordered Shared Log The FuzzyLog: A Partially Ordered Shared Log Joshua...

  • This paper is included in the Proceedings of the 13th USENIX Symposium on Operating Systems Design

    and Implementation (OSDI ’18). October 8–10, 2018 • Carlsbad, CA, USA

    ISBN 978-1-939133-08-3

    Open access to the Proceedings of the 13th USENIX Symposium on Operating Systems

    Design and Implementation is sponsored by USENIX.

    The FuzzyLog: A Partially Ordered Shared Log Joshua Lockerman, Yale University; Jose M. Faleiro, UC Berkeley; Juno Kim, UC San Diego; Soham Sankaran, Cornell University; Daniel J Abadi, University of Maryland, College Park;

    James Aspnes, Yale University; Siddhartha Sen, Microsoft Research; Mahesh Balakrishnan, Yale University / Facebook

  • The FuzzyLog: A Partially Ordered Shared Log

    Joshua Lockerman Yale University

    † Jose M. Faleiro UC Berkeley

    † Juno Kim UC San Diego

    † Soham Sankaran Cornell University

    Daniel J. Abadi University of Maryland,

    College Park

    James Aspnes Yale University

    Siddhartha Sen Microsoft Research

    † Mahesh Balakrishnan Yale University / Facebook

    † Work done while authors were at Yale University


    The FuzzyLog is a partially ordered shared log abstrac- tion. Distributed applications can concurrently append to the partial order and play it back. FuzzyLog appli- cations obtain the benefits of an underlying shared log – extracting strong consistency, durability, and failure atomicity in simple ways – without suffering from its drawbacks. By exposing a partial order, the FuzzyLog enables three key capabilities for applications: linear scaling for throughput and capacity (without sacrificing atomicity), weaker consistency guarantees, and tolerance to network partitions. We present Dapple, a distributed implementation of the FuzzyLog abstraction that stores the partial order compactly and supports efficient ap- pends / playback via a new ordering protocol. We im- plement several data structures and applications over the FuzzyLog, including several map variants as well as a ZooKeeper implementation. Our evaluation shows that these applications are compact, fast, and flexible: they retain the simplicity (100s of lines of code) and strong semantics (durability and failure atomicity) of a shared log design while exploiting the partial order of the Fuzzy- Log for linear scalability, flexible consistency guarantees (e.g., causal+ consistency), and network partition toler- ance. On a 6-node Dapple deployment, our FuzzyLog- based ZooKeeper supports 3M/sec single-key writes, and 150K/sec atomic cross-shard renames.

    1 Introduction

    Large-scale data center systems rely on control plane ser- vices such as filesystem namenodes, SDN controllers, coordination services, and schedulers. Such services are often initially built as single-server systems that store state in local in-memory data structures. Proper- ties such as durability, high availability, and scalability are retrofitted by distributing service state across ma- chines. Distributing state for such services can be dif- ficult; their requirement for low latency and high respon- siveness precludes the use of external storage services

    with fixed APIs such as key-value stores, while custom solutions can require melding application code with a medley of distributed protocols such as Paxos [29] and Two-Phase Commit (2PC) [21], which are individually complex, slow/inefficient when layered, and difficult to merge [40, 60].

    A recently proposed class of designs centers on the shared log abstraction, funneling all updates through a globally shared log to enable fault-tolerant databases [9– 11, 19, 51], metadata and coordination services [8, 12], key-value and object stores [3, 41, 57], and filesystem namespaces [50, 56]. Services built over a shared log are simple, compact layers that map a high-level API to append/read operations on the shared log, which acts as the source of strong consistency, durability, failure atom- icity, and transactional isolation. For example, a shared log version of ZooKeeper uses 1K lines of code, an order of magnitude lower than the original system [8].

    Unfortunately, the simplicity of a shared log requires imposing a system-wide total order that is expensive, often impossible, and typically unnecessary. Previous work showed that a centralized, off-path sequencer can make such a total order feasible at intermediate scale (e.g., a small cluster of tens of machines) [7, 8]. How- ever, at larger scale – in the dimensions of system size, throughput, and network bandwidth/latency – imposing a total order becomes expensive: ordering all updates via a sequencer limits throughput and slows down operations if machines are scattered across the network. In addi- tion, for deployments that span geographical regions, a total order may be impossible: a network partition can cut off clients from the sequencer or a required quorum of the servers implementing the log. On the flip side, a total order is often unnecessary: updates to disjoint data (e.g., different keys in a map) do not need to be ordered, while updates that touch the same data may commute because the application requires weak consistency guar- antees (e.g., causal consistency [5]). In this paper, we explore the following question: can we provide the sim-

    USENIX Association 13th USENIX Symposium on Operating Systems Design and Implementation 357

  • B D


    C J



    A G J A

    F JL G H

    app servers

    B E IC H D

    red shard

    remote red chain (half-propagated)

    local red chain

    local blue chain


    F L NM

    remote blue chain (half-propagated)

    blue shard

    FuzzyLog append / play order

    … append sync





    Figure 1: In the FuzzyLog, each color contains updates to a data shard, while each chain contains updates from a geographical region.

    plicity of a shared log without imposing a total order? We propose the FuzzyLog abstraction: a durable, iter-

    able, and extendable order over updates in a distributed system. Crucially, a FuzzyLog provides a partial order as opposed to the total order of a conventional shared log. The FuzzyLog is a directed acyclic graph (DAG) of nodes representing updates to a sharded, geo-replicated system (see Figure 1). The FuzzyLog materializes a happens-after relation between updates: an edge from A to B means that A must execute after B.

    The FuzzyLog captures two sources of partial ordering in distributed systems: data sharding and geo-replication. Internally, nodes in the FuzzyLog are organized into colors, where each color contains updates to a single application-level data shard. A color is a set of inde- pendent, totally ordered chains, where each chain con- tains updates originating in a single geographical region. Chains within a color are connected by cross-links that represent update causality. The entire DAG – consisting of multiple colors (one per shard) and chains within each color (one per region) – is fully replicated at every re- gion and lazily synchronized, so that each region has the latest copy of its own chain, but some stale prefix of the chains of other regions. Figure 1 shows a FuzzyLog de- ployment with two data shards (i.e,. two colors) and two regions (i.e., two chains per color).

    The FuzzyLog API is simple: a client can append a new node by providing a payload describing an update and the color of the shard it modifies. The new node is added to the tail of the local chain for that color, with outgoing cross-links to the last node seen by the client in each remote chain for the color. The client can synchro- nize with a single color, playing forward new nodes in the local region’s copy of that color in a reverse topological sort order of the DAG. A node can be appended atomi- cally to multiple colors, representing a transactional up- date across data shards.

    Applications built over the FuzzyLog API are nearly as simple as conventional shared log systems. As shown in Figure 1, FuzzyLog clients are application servers that maintain in-memory copies or views of shared objects. To perform an operation on an object, the application appends an entry to the FuzzyLog describing the muta- tion; it then plays forward the FuzzyLog, retrieving new entries from other clients and applying them to its lo- cal view, until it encounters and executes the appended entry. The local views on the application servers consti- tute soft state that can be reconstructed by replaying the FuzzyLog. A FuzzyLog application that uses only a sin- gle color for its updates and runs within a single region is identical to its shared log counterpart; the FuzzyLog de- generates to a totally ordered shared log, and the simple protocol described above provides linearizability [23], durability, and failure atomicity for application state.

    By simply marking each update with colors corre- sponding to data shards, FuzzyLog applications achieve scalability and availability. They can use a color per shard to scale linearly within a data center; transac- tionally update multiple shards via multi-color appends; obtain causal consistency [5] within a shard by using a color across regions; and toggle between strong and weak consistency when the network partitions and heals by switching between regions.

    Implementing the FuzzyLog abstraction in a scalable and efficient manner requires a markedly different design from existing shared log systems. We describe Dapple, a system that realizes the FuzzyLog API over a collection of in-memory storage