Apache CassandraDurability, Durability, Durability ...

Matthew F. Dennis // @mdennisCassandraSFAugust 8, 2012

Cassandra Data Model

KeyspacesColumn FamiliesRowsColumns (tuples)Name, Value, Timestamp, TTL

Yeah Matt, you told us ...

A Banking Application(the canonical example)

credit_account(acct, delta)get_account_balance(acct)xfer_funds(from, to, delta)

“Work Backwards From Your Queries” --everyone

The only “query” we really have is get_account_balance

accounts column family

acctXall_balls xact_id0 xact_id1 ...

$123.45 “details” “details” ...

00:00:00 etc hash(“details”) - a unique, one-oneidempotent id for xact0

from, to, delta, sessionId, timestamp, amount, check number, order number, et cetera

as a JSON/ProtoBuf/XML blob(i.e. everything about a “change” to the acct)

current “base” total

get_account_balance(acct)

● read the entire row

● apply deltas to “base”

credit_account(acct, delta)

● write “details” to accounts CF

get_account_balance(acct)

● obviously the row for each account grows unbounded

● need to safely recalculate the “base” to avoid unbounded growth

accounts column family

● a standard single master setup for consolidating works well here (because the system is slower, not broken, while the master is down)

● lets be clear on that; the master being up or down is independent of the correctness of the system(otherwise master => SPOF => bad)

accounts column family(consolidation, WOT form)

● pick number of processors, hash acctId mod num_consolidators(clearly other options exist to assign accounts to consolidators)

● only the assigned processor can update the base

● read row for account, calculate new base, write new base + delete columns that went into the base

● read at CL.ALL the first time an account is seen by the processor after boot (in memory, BDB, etc)

● on failure of a write at CL.Q, do not continue processing for that account until a read at CL.ALL for that account has completed

● adding consolidators is easy and requires no down time; shut them down, reconfigure with a new number, start them up

● essentially check pointing

accounts column family(consolidation)

acctXall_balls xact_id0 xact_id1 ...

{x} {y} {z} ...

xact0

xact1current “base” total

if “base” was written at CL.Q, then a read at CL.Q will returnthe most most current version.

accounts column family(consolidation)

acctXall_balls xact_id0 xact_id1 ...

{x, y, z} ...

xact0 tombstone

xact1 tombstonenew “base” total

processor calculates new base and deletes corresponding deltas

accounts column family(consolidation)

acctXall_balls xact_id0 xact_id1 ...

{x, y, z} ...

xact0 tbmbstone

xact1 tombstonenew “base” total

row level isolation guarantees that if the base includes the deltathen delta is absent from the delta list. Likewise, if the base does not include the delta then the delta is in the delta list

accounts column family(durability, consistency)

● writes can be at any any consistency level that meets your durability, consistency and availability requirements (CL.Q?)

● base updates and the queries to calculate them must be at CL.Q with the initial read at CL.ALL

why the CL.ALL read?

node0 node1 node2

left set is base, right is delta list {} {} {} {} {} {}

concurrent write for x {} {x} {} {} {} {}

CL.Q response from node0 and node1calculate base as {x}, CL.Q write fails {x} {} {} {} {} {}

concurrent write for y {x} {} {} {} {} {y}

CL.Q response from node1 and node2calculate base as {y} {x} {} {} {} {y} {}

node2 propagates base={y}node0 propagates deltas={}resulting in x missing from base *and* from deltas

{y} {} {y} {} {y} {}

accounts column family(xfers)

● clearly source account and destination account can be on different nodes

● so, how do you maintain consistency across them when doing transfers?

accounts column family(xfers)

● the common approach is to use a transaction log (go go wikipedia)

● Oracle uses one● PGSQL uses one● C* uses one● we should have one too!

the xact_log column family

node_tokentimeuuid(xact0) timeuuid(xact1) timeuuid(xact2)

“details” “details” “details”

same “complete” details as previously

timeuuid of when xact0 occurred

randomly chosen from set of nodes (or from a known range, e.g. 0-100)

the xact_log column family

● a durable (e.g. multi node) place to write changes● a write to xact_log CF ~= “commit”● each node runs a crond job that periodically (e.g.

every minute +/- 15 seconds) queries a slice of its corresponding row(s) and those of it’s neighbor (could improve on polling)

● node replays any messages found in their entirety and deletes the column

● normally, the query returns no results

xfer_funds(from, to, delta)(the interesting one)

● write “details” to xact_log CF● in parallel, write “details” for from

and to account rows● delete “details” from xact_log CF

(could be done after client response)

● failures?

xfer_funds(from, to, delta)(failures)

● before insert● after insert● after from xor to is applied● after from and to is applied● after delete from xact_log CF

consistency?(eventually)

● partitions between data centers?● failures for xacts in flight?● maintenance? ● upgrades? ● you have requirements, be honest about

what they are …● do not page your ops team at 4am unless

required (which *should* be rare)

accounts column family settings

● normal gc_grace_seconds● row cache friendly*● key cache friendly (~everything is)● level compaction strategy

(IO “now” or IO “later”?)● should probably use

commit_log_sync=batch(not a per CF setting)

* in general you should probably just avoid the row cache all together

xact_log column family settings

● gc_grace_seconds = 0● row cache unfriendly● key cache friendly, but not needed● level compaction strategy

(or sized with min_threshold=2)

other uses

● “base” and “deltas” need not represent money

● character inventory/trading● portfolios● escrow exchanges● anything combinable

(you control the consolidate code)

is this the best way?

● not always, of course not, depends on your requirements and goals

● could use C* for xact_log, Oracle for balances

● could use zookeeper instead of CL.Q and CL.ALL for consolidators

● C* solutions favors availability, scalability and durability over other desirable traits

Q?Matthew F. Dennis // @mdennis

Thank You!(now go prep for your lighting talk)