Beyond Strong ConsistencyModels for Consistency in High Performance Distributed Systems

John L. Singleton

University of Central Florida

“Represents”

What Happens if we delete a friend from my social graph?

A Motivating Example

The Problem of Scalability

del(fn)

query(f)

Strong Consistency: With a single database,

all users see the update after it is committed

The Problem of Scalability (continued)

Strong Consistency: As more

database servers are added to

handle the load, update latency

increases

DB1

del(fn)query(f)

DBn

sync(f)

query(f)

The Problem: Scaling a Model of Consistency

• Strong Consistency requires that every update is immediately propagated to all nodes.

• All users have the same view of the data at any moment.

• However, this is bad for performance and is hard to scale.

How can we implement a model of consistency that scales?

The CAP “Theorem”

• Uttered by Eric Brewer at PODC circa 2000

• Consistency: All users see the same view of the data

• Availability: The system is always available

• Partition Tolerance: The system can operate in the presence of network failures between nodes.

• Not really a theorem because no proof of its validity exists.

Some Criticism of the CAP Theorem

• In the world of NoSQL databases, CAP has been used as an excuse to give up consistency

• However, the cases they are optimizing for (for example, partition tolerance) are in fact the most rare

• Network partition in a local cluster

• Failure of a single node is much more common

“One should not throw out the C so quickly, since there are real error scenarios where CAP does not apply and it seems like a bad tradeoff in many of the other situations.” – Michael Stonebraker

One Possible Solution: Eventual Consistency

Asks the question: “What if we omit the C?”

• Do updates locally and propagate the changes

• No foreground synchronization

• Resists partitioning

• Requires consensus

• Consistency is eventual, no definite point in the future.

Some Preliminaries: State-Based Objects

• Two Views of an Object: State Based and Operation Based• State Based: Object is a collection of state transitions

The State-Based View

Causal History of State-Based Objects

• We need an abstraction to reason about the state an object may be in (or may have come from)

• Each replica can maintain a copy of the causal history.

Definition of Eventual Consistency

Eventual Delivery: An update delivered at some correct replica is eventually delivered to all correct replicas:

Convergence: Correct replicas that have delivered the same updates eventually reach equivalent state:

Termination: All method executions terminate

Problems with Eventual Consistency

The Strong Consistency Model

• In the strong consistency model, users get a

single view in a specific order, therefore no

conflicts

s1.del(a)s2.add(a,b)

a

b

time (increasing)

Problems with Eventual Consistency

The Eventually Consistency Model

• This update is invalid!

• Measures must be taken to handle conflicts

and perform rollbacks to ensure consistency

• This results in overhead to the system and

should be avoided

s1.add(a,b)s2.del(a)

a

b

time (increasing)

Eventual Delivery: An update delivered at some correct replica is eventually delivered to all correct replicas:

Convergence: Correct replicas that have delivered the same updates eventually reach equivalent state:

Termination: All method executions terminate

Strong Convergence: Correct replicas that have delivered the same updates have equivalent state:

Strong Eventual Consistency

• Strong Eventual Consistency is a stronger subset of Eventual Consistency

• By placing restrictions on the way in which replicated types are constructed, we can provide stronger guarantees.

• Most importantly: SEC is a solution to the CAP problem• Can tolerate up to n-1 concurrent crashes

• Does not require consensus

• Is weaker than strong consistency, but provides performance benefits and a well defined consistency model.

• Removes conflicts

Conflict Free Replicated Data Types

• The key to Strong Eventual Consistency is the Convergent Replicated Data Type (CvRDT)

• Constructed with a monotonic semi-lattice object• A set with a partial order

• And an operation that can find the least upper bound on any two objects in the set such that:

CvRDTs and Monotonic Semi-Lattices

If these conditions are satisfied, you have a CvRDT

(technical note: State-Based RDTs are equivalent to

Operation Based RDTs)

Sample CvRDT: A Integer Vector

Set-Based CvRDT

• How could we apply this idea to a more complex object like a set?

• Create one set, the Addition set, and record all elements added

• Create another set, the Remove set, and record all elements removed.

• Both sets are monotonic

• Define the query function as:

Conclusions

• Introduced the idea of multiple models of consistency:• Strong Consistency

• Eventual Consistency

• Strong Eventual Consistency

• Discussed the CAP Theorem and showed how performance problems with Strong Consistency lead to Eventual Consistency

• Demonstrated how problems with Eventual Consistency can arise

• Showed how Strong Eventual Consistency is a solution to the CAP problem, and can improve the performance and reliability of distributed data store systems.

References[1] Deftu, Andrei, and Jan Griebsch. "A Scalable Conflict-Free Replicated Set Data Type." In Distributed Computing Systems (ICDCS), 2013 IEEE 33rd International Conference on, pp. 186-195. IEEE, 2013.

[2] Shapiro, Marc, Nuno Preguiça, Carlos Baquero, and Marek Zawirski. "Conflict-free replicated data types." In Stabilization, Safety, and Security of Distributed Systems, pp. 386-400. Springer Berlin Heidelberg, 2011.

[3] Bouajjani, Ahmed, Constantin Enea, and Jad Hamza. "Verifying eventual consistency of optimistic replication systems." In Proceedings of the 41st annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages, pp. 285-296. ACM, 2014.

[4] Vogels, Werner. "Eventually consistent." Communications of the ACM 52, no. 1 (2009): 40-44.

[5] Stonebraker, Michael. "Errors in database systems, eventual consistency, and the cap theorem." Communications of the ACM, BLOG@ ACM (2010).

[6] Brewer, Eric A. "Towards robust distributed systems." In PODC, p. 7. 2000.

[7] Lamport, Leslie. "Time, clocks, and the ordering of events in a distributed system." Communications of the ACM 21, no. 7 (1978): 558-565.

[8] Gilbert, Seth, and Nancy Lynch. "Brewer’s Conjecture and the Feasibility of Consistent." Available, Partition-Tolerant Web Services. doi 10, no. 564585.564601 (2002).

Questions?