Upload
martin-dunn
View
27
Download
2
Tags:
Embed Size (px)
DESCRIPTION
RAMBO: A Reconfigurable Atomic Memory Service for Dynamic Networks Nancy Lynch, MIT Alex Shvartsman, U. Conn. Boston University April 23, 2003. Goal of the Research. - PowerPoint PPT Presentation
Citation preview
RAMBO:RAMBO:A Reconfigurable Atomic A Reconfigurable Atomic
Memory Service for Dynamic Memory Service for Dynamic NetworksNetworks
Nancy Lynch, MIT Nancy Lynch, MIT Alex Shvartsman, U. Conn.Alex Shvartsman, U. Conn.
Boston UniversityBoston UniversityApril 23, 2003April 23, 2003
Goal of the ResearchGoal of the Research• An algorithm to implement atomic read/write shared An algorithm to implement atomic read/write shared
memory in a dynamic network setting.memory in a dynamic network setting.– Participants may join, leave, fail during
computation.– Mobile networks, peer-to-peer networks.
• High availability, low latency.High availability, low latency.• Atomicity for all patterns of asynchrony and change.Atomicity for all patterns of asynchrony and change.• Good performance under reasonable limits on Good performance under reasonable limits on
asynchrony and change.asynchrony and change.• Applications:Applications:
– Battle data for teams of soldiers in military operation.
– Game data for players in a multiplayer game.
Approach: Dynamic Approach: Dynamic QuorumsQuorums
• Objects are replicatedObjects are replicated at several network locations.at several network locations.• To accommodate small, transient changes:To accommodate small, transient changes:
– Uses quorum configurations: • members, read-quorums, write-quorums.
– Maintains atomicity during stable situations.– Allows concurrency.
• To handle larger, more permanent changes:To handle larger, more permanent changes:– Reconfigure– Maintains atomicity across configuration
changes.– Any configuration can be installed at any time.– Reconfigure concurrently with reads/writes; no
heavyweight view change.
RAMBO RAMBO • RAMBO: Reconfigurable Atomic Memory for Basic RAMBO: Reconfigurable Atomic Memory for Basic
ObjectsObjects (dynamic atomic read/write shared (dynamic atomic read/write shared memory).memory).
• Global service specification: Global service specification:
• Algorithm: Algorithm: – Reads and writes objects.– Chooses new configurations, notifies members.– Identifies, removes (“garbage-collects”) obsolete
configurations.– All concurrently.
RAMBO
• Main algorithm + reconfiguration serviceMain algorithm + reconfiguration service• Loosely coupledLoosely coupled
• Recon service: Recon service: – Provides the main algorithm with
a consistent sequence of configurations.• Main algorithm: Main algorithm:
– Handles reading, writing.– Receives, disseminates new configuration
information; no formal installation.– Garbage-collects old configurations.– Reads/writes may use several configurations.
ReconNet
RRAMBO
Recon
RAMBO algorithm structureRAMBO algorithm structure
Main algorithm: Reads/writesMain algorithm: Reads/writes
• Uses two-phase strategy:Uses two-phase strategy:– Phase 1: Collect object values from read-quorums
of active configurations.– Phase 2: Propagate latest value to write-quorums
of active configurations.• Operations may execute concurrently.Operations may execute concurrently.• Quorum intersection properties guarantee atomicity.Quorum intersection properties guarantee atomicity.• Our communication mechanism:Our communication mechanism:
– Background gossiping– Terminate by fixed-point condition, involving a
quorum from each active configuration.
Removing old configurationsRemoving old configurations
• Main algorithm removes old configurations by Main algorithm removes old configurations by garbage-collecting them in the background.garbage-collecting them in the background.
• Two-phase garbage-collection procedure:Two-phase garbage-collection procedure:– First phase:
• Inform write-quorum of old configuration about the new configuration.
• Collect object values from read-quorum of the old configuration.
– Second phase: • Propagate the latest value to a write-quorum of
the new configuration.• Garbage-collection concurrent with reads/writes.Garbage-collection concurrent with reads/writes.• Implemented using gossiping and fixed points.Implemented using gossiping and fixed points.
Consensus
Recon
Net
Implementation of ReconImplementation of Recon• Uses distributed consensus to determine successive Uses distributed consensus to determine successive
configurations 1,2,3,…configurations 1,2,3,…
• Members of old configuration propose new Members of old configuration propose new configuration.configuration.
• Proposals reconciled using consensusProposals reconciled using consensus• Consensus is a heavyweight mechanism, but:Consensus is a heavyweight mechanism, but:
– Used only for reconfigurations, infrequent.– Does not delay Read/Write operations.
Implementation of consensusImplementation of consensus
• Use a version of the Paxos algorithm Use a version of the Paxos algorithm [Lamport 89, 98, [Lamport 89, 98, 02].02].
• Agreement, validityAgreement, validity guaranteed absolutely. guaranteed absolutely.• TerminationTermination guaranteed if/when underlying system guaranteed if/when underlying system
stabilizes.stabilizes.
• Leader chosen using failure detectors.Leader chosen using failure detectors.• Leader conducts two-phase algorithm with retries.Leader conducts two-phase algorithm with retries.
decide(v)init(v)
init(v)
Consensus
Models and analysisModels and analysis
• Timed I/O automata models.Timed I/O automata models.• Prove atomicity for arbitrary patterns of asynchrony Prove atomicity for arbitrary patterns of asynchrony
and change.and change.• Analyze performance conditionally, based on failure Analyze performance conditionally, based on failure
and timing assumptions.and timing assumptions.– Reads and writes take time at most 8d, under
reasonable “steady-state” assumptions.
Other approachesOther approaches• Use consensus to agree on total ordering of operations: Use consensus to agree on total ordering of operations:
[Lamport 89…][Lamport 89…]– Not resilient to transient failures. – Termination of r/w depends on termination of
consensus.• Totally-ordered broadcast over group communication: Totally-ordered broadcast over group communication:
[Amir, Dolev, Melliar-Smith, Moser 94], [Keidar, [Amir, Dolev, Melliar-Smith, Moser 94], [Keidar, Dolev 96]Dolev 96]– View formation takes a long time, delays reads/writes.– One change may trigger view formation.
• Dynamic quorums over GC: Dynamic quorums over GC: [De Prisco, et al, 99][De Prisco, et al, 99]– View formation delays reads/writes.– New view must satisfy intersection requirements.
• Single reconfigurer: Single reconfigurer: [Lynch, Shvartsman 97], [Lynch, Shvartsman 97], [Englert, Shvartsman 00][Englert, Shvartsman 00]
Outline of talkOutline of talk1. Introduction 1. Introduction 2. Reconfigurable Atomic Memory (RAMBO) 2. Reconfigurable Atomic Memory (RAMBO)
specificationspecification
3. Reconfiguration service (Recon) specification3. Reconfiguration service (Recon) specification
4. Implementation of RAMBO using Recon 4. Implementation of RAMBO using Recon
5. Proof of atomicity5. Proof of atomicity
6. Implementation of Recon6. Implementation of Recon
7. Conditional performance results7. Conditional performance results
8. Conclusions8. Conclusions
2. RAMBO Service Specification2. RAMBO Service Specification• I, infinite set of participants’ locationsI, infinite set of participants’ locations• X, set of objectsX, set of objects• C, configuration identifiersC, configuration identifiers• External actions for each i and x:External actions for each i and x:
– Inputs: joinx,i, readx,i, write(v)x,i, recon(c,c’)x,i
– Outputs: join-ackx,i, read-ack(v)x,i, …, report(c)x,i
• Ignore joins in this talk.Ignore joins in this talk.• Behavior:Behavior:
– Assuming basic well-formedness conditions, RAMBO guarantees atomicity.
– Liveness replaced by latency bounds.
RAMBO
Atomicity Atomicity • AKA linearizabilityAKA linearizability• Definition:Definition: Each operation appears to occur at some Each operation appears to occur at some
point between its invocation and response.point between its invocation and response.• Sufficient condition:Sufficient condition: For each object x, all the read For each object x, all the read
and write operations for x can be partially ordered and write operations for x can be partially ordered by by , so that:, so that: is consistent with the order of invocations and
responses: there are no operations such that 1
completes before 2 starts, yet 2 1 .
– All write operations are ordered with respect to each other and with respect to all the reads.
– Every read returns the value of the last write preceding it in .
Implementing RAMBOImplementing RAMBO• Composition of separate service for each x.Composition of separate service for each x.• RAMBO (for x) uses separate Recon service (for x):RAMBO (for x) uses separate Recon service (for x):
Net
Recon
recon
read,write RAMBO
new-config
3. Recon Service 3. Recon Service SpecificationSpecification
• External actions for each i:External actions for each i:
– Inputs: recon(c,c’)i
– Outputs: recon-acki, report(c)i, new-config(c,k)i
– And some joining actions (ignore)• Behavior:Behavior:
– Assuming well-formedness, Recon produces consistent configuration identifiers at participating locations:• Agreement: Two configs never assigned to
same k.• Validity: Any announced new-config was
previously requested by someone.• No duplication: No configuration is assigned to
more than one k.
Outline Outline 1. Introduction 1. Introduction 2. Reconfigurable Atomic Memory (RAMBO) 2. Reconfigurable Atomic Memory (RAMBO)
specification specification 3. Reconfiguration service (Recon) specification 3. Reconfiguration service (Recon) specification 4. Implementation of RAMBO using Recon 4. Implementation of RAMBO using Recon
5. Correctness (atomicity)5. Correctness (atomicity)
6. Implementation of Recon6. Implementation of Recon
7. Conditional performance results7. Conditional performance results
8. Conclusions8. Conclusions
4. Implementing RAMBO using 4. Implementing RAMBO using ReconRecon
• ReconRecon– Chooses configurations– Tells members of the previous and new
configuration.– Informs Reader-Writer components (new-config).
• Reader-WriterReader-Writer– Conducts read and write operations
• Two-phased quorum-based algorithm.• Uses all current configurations.
– Garbage-collects obsolete configurations.
Static Reader-Writer protocolStatic Reader-Writer protocol
• Quorum configuration for I:Quorum configuration for I:– read-quorums, write-quorums, collections of subsets of I
– For any RR in read-quorums, WW in write-quorums:
RR WW ..
• Replicate the object x at all locations in I.Replicate the object x at all locations in I.• At each i in I, keep:At each i in I, keep:
– value– tag, consisting of (sequence number, location)
• Read, Write use two phases:Read, Write use two phases:– Phase 1: Read (value, tag) from a read-quorum– Phase 2: Write (value,tag) to a write-quorum
Static Reader-Writer protocolStatic Reader-Writer protocol• Write at location i:Write at location i:
– Phase 1: • Read (value, tag) from a read-quorum.• Determine largest seq-number among the tags that are
read. • Choose new-tag := (larger sequence-number, i).
– Phase 2: • Propagate (new-value, new-tag) to a write-quorum.
• Read at location i:Read at location i:– Phase 1:
• Read (value, tag) from a read-quorum.• Determine largest (value,tag) among those read.
– Phase 2:• Propagate this (value,tag) to a write-quorum.• Return value.
• Highly concurrent.Highly concurrent.• Quorum intersection implies atomicityQuorum intersection implies atomicity
Why Readers propagateWhy Readers propagate
• If not, then atomicity can be violated:If not, then atomicity can be violated:
• Note: The value after phase 1 is reliable---could use Note: The value after phase 1 is reliable---could use it “optimistically”.it “optimistically”.
Write(1). . . slow
0
1
Read returns 1 Read returns 0
00
00
Extend to dynamic settingExtend to dynamic setting
• Any member of current configuration can propose a Any member of current configuration can propose a new configuration.new configuration.
• Recon produces consistent configurations.Recon produces consistent configurations.• Reader-Writer processes run Reader-Writer processes run two-phase static two-phase static
quorum-based algorithm, using all current quorum-based algorithm, using all current configurations.configurations.
• Uses gossip and fixed-point tests.Uses gossip and fixed-point tests.• When Recon provides new configuration, Reader-When Recon provides new configuration, Reader-
Writer doesn’t abort reads/writes in progress, but Writer doesn’t abort reads/writes in progress, but does extra work to access additional processes does extra work to access additional processes needed for new quorums.needed for new quorums.
Configurations and Config Configurations and Config MapsMaps
• Configuration Configuration cc– members(c) --“owners” of the data in configuration c– read-quorums(c)– write-quorums(c)
• Configuration map Configuration map cmcm– Sequence of configurations cm(k) – Can be defined, undefined (), garbage-collected
(±)
± ± c c c c ... ...
GC’d Defined Mixed Undefined
c
Configuration mapsConfiguration maps
c0
c0 c1
c0 c1 c2 ck
± c1 c2 ck
± ± c2 ck
. . .
. . .
. . .
. . .
. . .
± ± ± c3 ck . . .
± ± ± ± ± c c c c . . .
. . .
Reader-Writer stateReader-Writer state
• worldworld• value, tagvalue, tag• cmapcmap• pnum1, pnum1, counts phases of locally-initiated operationscounts phases of locally-initiated operations• pnum2, pnum2, records latest known phase numbers for all records latest known phase numbers for all
locationslocations• op-record, op-record, keeps track of the status of a current keeps track of the status of a current
locally initiated read/write operationlocally initiated read/write operation– Includes op.cmap, consisting of consecutive
configs.• gc-record, gc-record, keeps track of the status of a current keeps track of the status of a current
locally-initiated garbage-collection operationlocally-initiated garbage-collection operation
Reader-Writer protocolReader-Writer protocol
• One kind of message, gossiped nondeterministically.One kind of message, gossiped nondeterministically.• Message Message W, v, t, cm, ns, nrW, v, t, cm, ns, nr from i to j, where: from i to j, where:
– W is i ’s world– v,t are i’s value and tag – cm is i’s cmap – ns is i’s phase number, pnum1– nr is the latest phase number i knows for j,
pnum2(j)• ((ns,nrns,nr) used to identify “fresh” messages.) used to identify “fresh” messages.• Key actions are taken when “enough” information has Key actions are taken when “enough” information has
been gathered (fixed-point).been gathered (fixed-point).
When When W,v,t,cm,ns,nrW,v,t,cm,ns,nr arrives from j: arrives from j:
• world world := := world world WW• if if t t >> tag tag then ( then (value,tagvalue,tag) := () := (v,tv,t))• cmap cmap := update(:= update(cmap,cmcmap,cm))
– Updates cmap with newer information in cm.• pnum2pnum2(j)(j) := max(:= max(pnum2pnum2(j),(j), ns ns))• gc-record: If the message is “fresh”, then record gc-record: If the message is “fresh”, then record
the sender.the sender.• op-record: If message is “fresh” then:op-record: If message is “fresh” then:
– Record the sender.– Extend op.cmap with newly-discovered
configurations.
Processing reads and writesProcessing reads and writes
• Reads and Writes perform Query and Propagation Reads and Writes perform Query and Propagation phases using known configurations, stored in phases using known configurations, stored in op.cmap.op.cmap.– Query phase: Obtains fresh value, tag, cmap
information from read-quorums. – Propagation phase: Propagates up-to-date
(value,tag) to write-quorums; obtains fresh cmap information from write-quorums.
– Both phases: Extend op.cmap with newly-discovered configurations; new configurations are also used in the phase.
• Each phase ends with a fixed point, after hearing from Each phase ends with a fixed point, after hearing from quorums of all the configurations currently in quorums of all the configurations currently in op.cmap. op.cmap.
Reader-Writer: Fixed pointsReader-Writer: Fixed points
Garbage collectionGarbage collection• A process can try to GC config k when its A process can try to GC config k when its cmapcmap looks looks
like:like:
• Phase 1: Phase 1:
– Informs a write-quorum of ck about ck+1.
– Collects latest (value, tag) from a read-quorum of ck.
• Phase 2: Phase 2:
– Propagates (value, tag) to a write-quorum of ck+1.
– Set cmap(k) to ±.
• GC operates concurrently with reads and writes. GC operates concurrently with reads and writes. • Uses gossiping and fixed points.Uses gossiping and fixed points.
± ck ck+1 . . .. . . . . .
Outline Outline 1. Introduction 1. Introduction 2. Reconfigurable Atomic Memory (RAMBO) 2. Reconfigurable Atomic Memory (RAMBO)
specification specification 3. Reconfiguration service (Recon) specification 3. Reconfiguration service (Recon) specification 4. Implementation of RAMBO using Recon 4. Implementation of RAMBO using Recon 5. Correctness (atomicity) 5. Correctness (atomicity)
6. Implementation of Recon6. Implementation of Recon
7. Conditional performance results7. Conditional performance results
8. Conclusions8. Conclusions
5. Proof of Atomicity5. Proof of Atomicity• Atomicity holds for:Atomicity holds for:
– arbitrary patterns of asynchrony,– arbitrary crash-failures and message loss,– arbitrary joins.
• Proof: Construct partial order Proof: Construct partial order of read and write of read and write operations satisfying:operations satisfying: is consistent with the order of invocations and
responses.– All write operations are ordered with respect to
each other and with respect to all the reads.– Every read returns the value of the last write
preceding it in .• Let Let be the lexicographic order on the operations’ be the lexicographic order on the operations’
tagtags, and order write with s, and order write with tagtag t before all reads with t before all reads with tagtag t. t.
Showing consistencyShowing consistency
• Lemma 1:Lemma 1: Tags of GC operations are nondecreasing Tags of GC operations are nondecreasing with respect to the configuration index.with respect to the configuration index.
• Proof: GC is done sequentially.Proof: GC is done sequentially.
• Lemma 2:Lemma 2: If the first GC of config k completes If the first GC of config k completes before a read/write operation before a read/write operation begins, then the tag begins, then the tag of the GC is less than or equal to the tag of of the GC is less than or equal to the tag of (< if (< if is a write).is a write).
• Lemma 3:Lemma 3: If If 11 and and 22 are two read/write operations are two read/write operations
and and 1 1 completes before completes before 22 begins, then the tag of begins, then the tag of 11
is less than or equal to the tag of is less than or equal to the tag of 22 (strictly less if (strictly less if 22
is a write). is a write).
Proof of Lemma 3Proof of Lemma 3• Assume Assume 11 and and 22 are two read/write operations and are two read/write operations and 1 1
completes before completes before 22 begins. begins.• Each phase uses consecutive configurations.Each phase uses consecutive configurations.• Case 1:Case 1: prop-cmap( prop-cmap(11) and query-cmap() and query-cmap(22) share a ) share a
configuration c.configuration c.– Quorum intersection for c yields the tag inequality.
• Case 2:Case 2: All configurations in prop-cmap( All configurations in prop-cmap(11) are less than ) are less than all those in query-cmap(all those in query-cmap(22).).– Then the tag inequality follows from a chain of tag
inequalities, following a chain of GC operations for the intervening configurations. Uses Lemmas 1 and 2.
• Case 3:Case 3: All configs in prop-cmap( All configs in prop-cmap(11) are greater than all ) are greater than all those in query-cmap(those in query-cmap(22).).– Impossible.
Outline Outline 1. Introduction 1. Introduction 2. Reconfigurable Atomic Memory (RAMBO) 2. Reconfigurable Atomic Memory (RAMBO)
specification specification 3. Reconfiguration service (Recon) specification 3. Reconfiguration service (Recon) specification 4. Implementation of RAMBO using Recon 4. Implementation of RAMBO using Recon 5. Correctness (atomicity) 5. Correctness (atomicity) 6. Implementation of Recon6. Implementation of Recon
7. Conditional performance results7. Conditional performance results
8. Conclusions8. Conclusions
6. Implementing Recon 6. Implementing Recon • Recon algorithm uses (static) consensus services Recon algorithm uses (static) consensus services
to determine configurations 1, 2, 3,… to determine configurations 1, 2, 3,…
• Cons(k,c): Used to determine config k, if config k-1 is Cons(k,c): Used to determine config k, if config k-1 is c.c.
• Consensus is used only for reconfigurations, does not Consensus is used only for reconfigurations, does not delay read or write operations.delay read or write operations.
Consensus
Recon
Net
reconrecon-ack
Implementing ReconImplementing Recon• Simple---no atomicity issues.Simple---no atomicity issues.• Members of old configuration may propose a new Members of old configuration may propose a new
configuration; proposals reconciled using consensus.configuration; proposals reconciled using consensus.– recon(c,c’): Request for reconfiguration from c to c’. If c is the k-1st configuration (and is current), then send
init message to members; invoke Cons(k,c) with initial value c’
– Receive an init message: Participate in consensus.– decide(c’): Tell Reader-Writer the new configuration;
send config message to members of c’.– Receipt of config message: Tell Reader-Writer the new
configuration.• Consensus implemented using Paxos Synod algorithm.Consensus implemented using Paxos Synod algorithm.
7. Latency Analysis7. Latency Analysis
Consider a subset of timed executions:Consider a subset of timed executions:• Gossip occurs:Gossip occurs:
– Periodically, and– At certain key times:
• At beginning of operation phase.• Just after receiving a message from someone
with a new phase number. • Just after certain join and reconfiguration
events.• Perform local steps immediately.Perform local steps immediately.• Reliable message delivery, bounded delay.Reliable message delivery, bounded delay.• Normal timing for consensus services.Normal timing for consensus services.
Additional assumptionsAdditional assumptions
• e-Configuration-viability for time parameter ee-Configuration-viability for time parameter e– A read-quorum and a write-quorum of
configuration k remain alive, until at least time e after configuration k+1 is “installed” (decided upon by all non-failed members of configuration k).
• e-Reconfiguration-spacing e-Reconfiguration-spacing
– recon(c,*)i occurs at least e time after report(c)i
• e-Join-connectivity e-Join-connectivity – If i and j join by time t then they learn about each
other by time t+e
Latency results Latency results
• Reconfiguration:Reconfiguration:
– 13d, if recon(c,c’)i occurs and no members of c subsequently fail.
• Garbage-collection of cGarbage-collection of ckk by process i: by process i:
– 4d, if process i, a read-quorum and a write-quorum of ck, and a write-quorum of ck+1, do not fail.
• Read or write operation by process i in a “stable” Read or write operation by process i in a “stable” system:system:– 4d, if no reconfigurations occur, and process i’s cmap
is “up-to-date”.• Learning about configurations: Learning about configurations:
– If i and j are “old enough” and don’t fail, then information from i is conveyed to j within time 2d.
Latency results Latency results • Garbage-collection, in executions with 6d-Garbage-collection, in executions with 6d-
reconfiguration-spacing and 5d-configuration-reconfiguration-spacing and 5d-configuration-viability:viability:– If report(c) occurs at i and i does not fail then any
non-failed process that is old enough learns about c and garbage-collects any older configuration within time 6d.
• Read and write operations, in executions with 12d-Read and write operations, in executions with 12d-reconfiguration-spacing and 11d-configuration-reconfiguration-spacing and 11d-configuration-viability:viability:
– 8d, for an operation managed by a process that is old enough and does not fail.
8. Conclusions8. Conclusions• RAMBO algorithmRAMBO algorithm• Composed of R/W algorithm, Recon service, Composed of R/W algorithm, Recon service,
ConsensusConsensus• Atomicity in all executions. Atomicity in all executions. • Good latency bounds: Good latency bounds:
– For reading, writing, garbage-collection.– Under assumptions about timing, joins, failures,
and rate of reconfiguration.
Algorithmic innovationsAlgorithmic innovations• Dynamic configurations:Dynamic configurations:
– Members can be changed dynamically.– Any current member may request
reconfiguration.– Arbitrary configurations can be installed; no
intersection requirements.• Loosely-coupled reconfiguration:Loosely-coupled reconfiguration:
– Concurrent reading, writing, reconfiguration.– Reads/writes can use several configurations; can
complete during reconfiguration.• Efficient “steady-state”:Efficient “steady-state”:
– Assuming bounded delays, infrequent reconfiguration, and periodic gossip, read and write operations complete in time 8d.
– Each phase involves at most 2 configurations.
Comparison with other Comparison with other approachesapproaches
• Using consensus to agree on a total ordering of Using consensus to agree on a total ordering of operations:operations:– We use consensus only for the configurations.– Consensus termination impacts only
reconfiguration latency, not read and write latency.
• Group communication:Group communication:– Our reads/writes work during “new view”
establishment.• Dynamic quorum configurations over GC:Dynamic quorum configurations over GC:
– We allow arbitrary new configurations.• Single reconfigurer approaches:Single reconfigurer approaches:
– We allow multiple reconfigurers.– We uncouple introduction of new configurations
and garbage-collection of old configurations.
Current and future workCurrent and future work• LAN implementations LAN implementations [Musial, Shvartsman 03] [Bachmann 03]
– Experiments, toy applications• More analysis:More analysis:
– “Normal behavior” starting from some point• Algorithmic improvements and additions:Algorithmic improvements and additions:
– Concurrent garbage-collection [Gilbert, Lynch, Shvartsman 03]– Limiting communication.– Eliminating second phase of reads, or first phase of writes, in
special cases.– Choosing good configurations.– Better join protocol, explicit “leave” protocol.– Early return of read values.– Backup strategies for when configuration-viability fails.– Extensions to other data types?
• Corresponding lower bounds, impossibility results? Corresponding lower bounds, impossibility results? • Implementations in mobile networks, peer-to-peer systems Implementations in mobile networks, peer-to-peer systems