March 8th, 2016 Aysylu Greenberg Distributed Systems in ... · Confidential + Proprietary...

Confidential + ProprietaryConfidential + Proprietary

Distributed Systems in Practice, in TheoryAysylu GreenbergMarch 8th, 2016

Confidential + Proprietary

Confidential + Proprietary

Confidential + Proprietary

Aysylu Greenberg

@aysylu22

Confidential + Proprietary

Towards Distributed Build System

Confidential + Proprietary

Towards Distributed Build System

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

Confidential + Proprietary

http://opencv.org/

Towards Distributed Build System

Confidential + Proprietary

Confidential + Proprietary

Papers We Love SF

Confidential + Proprietary

Aysylu Greenberg

@aysylu22

Confidential + Proprietarymeetup.com/Papers-We-Love-London

Confidential + Proprietary

Today

● Staged Event-Driven Architecture

Confidential + Proprietary

Today

● Staged Event-Driven Architecture● Leases

Confidential + Proprietary

Today

● Staged Event-Driven Architecture● Leases● Inaccurate Computations

Confidential + ProprietaryConfidential + Proprietary

Computer Science ResearchInDistributed Systems Industry

Confidential + Proprietary

Operating systems research

Confidential + Proprietary

Operating systems research

Confidential + Proprietary

Operating systems research

Concurrency

Confidential + Proprietary

Operating systems research

Concurrency

Concurrency primitives: mutex & semaphore

Confidential + Proprietary

Operating systems research

Concurrency

Concurrency primitives: mutex & semaphore

Processes execute at different speeds

Confidential + Proprietary

Time in distributed systems

Confidential + Proprietary

Time in distributed systems

Confidential + Proprietary

Time in distributed systems

Pipelining

Confidential + Proprietary

Time in distributed systems

Pipelining

Confidential + Proprietary

Internet

Confidential + Proprietary

Internet

Distributed consensus

Confidential + Proprietary

Internet

Distributed consensus

Confidential + Proprietary

Internet

Distributed consensus

Confidential + Proprietary

Internet

Distributed consensus

Paxos

Confidential + Proprietary

Reconsider large systems

Confidential + Proprietary

Reconsider large systems

Platform as a service

Confidential + Proprietary

CS Research is Timeless

Inform decisions

Mitigate technical risk

Confidential + ProprietaryConfidential + Proprietary

Staged Event Driven Architecture&Deep Pipelines

2001

Confidential + Proprietary

Hardware Pipelines

Confidential + Proprietary

Data Pipelines

https://en.wikipedia.org/wiki/Graphics_pipeline

Confidential + Proprietary

Search Indexing Pipelines

Confidential + Proprietary

Search Indexing Pipelines

Confidential + Proprietary

Search Indexing Pipelines

Confidential + Proprietary

Search Indexing Pipelines

Confidential + Proprietary

Search Indexing Pipelines

Confidential + Proprietary

Search Indexing Pipelines

Confidential + Proprietary

Search Indexing Pipelines

Confidential + Proprietary

Search Indexing Pipelines

Confidential + Proprietary

Search Indexing Pipelines

Confidential + Proprietary

Search Indexing Pipelines

Confidential + Proprietary

Search Indexing Pipelines

Confidential + Proprietary

Confidential + Proprietary

Staged Event Driven Architecture

Confidential + Proprietary

Single-machine pipeline

generalizes to distributed pipelines

Staged Event Driven Architecture

Confidential + Proprietary

Staged Event Driven Architecture

● Dynamic resource controllers

Confidential + Proprietary

Staged Event Driven Architecture

● Dynamic resource controllers

+-

Confidential + Proprietary

Staged Event Driven Architecture

● Dynamic resource controllers○ automatic tuning

+-

Confidential + Proprietary

Staged Event Driven Architecture

● Dynamic resource controllers○ automatic tuning

■ thread pool sizing+

-

Confidential + Proprietary

Staged Event Driven Architecture

● Dynamic resource controllers○ automatic tuning

■ thread pool sizing■ event batching +

-

Confidential + Proprietary

+-

Staged Event Driven Architecture

● Dynamic resource controllers○ automatic tuning

■ thread pool sizing■ event batching

○ load shedding via queue

Confidential + Proprietary

+-

Staged Event Driven Architecture

● Dynamic resource controllers○ automatic tuning

■ thread pool sizing■ event batching

○ load shedding via queue■ backpressure

Confidential + Proprietary

+-

Staged Event Driven Architecture

● Dynamic resource controllers○ automatic tuning

■ thread pool sizing■ event batching

○ load shedding via queue■ backpressure■ threshold, filter, reorder, aggregate

Confidential + ProprietaryConfidential + Proprietary

Leasesas Heart Beat in Distributed Systems 1989

Confidential + Proprietary

Confidential + Proprietary

Leases

● Distributed locking

Confidential + Proprietary

Leases

● Distributed locking● Lease term tradeoffs

○ short

Confidential + Proprietary

Leases

● Distributed locking● Lease term tradeoffs

○ short■ delay from client and server failures

minimized■ reduced false write-sharing

Confidential + Proprietary

Leases

● Distributed locking● Lease term tradeoffs

○ short vs long

Confidential + Proprietary

Leases

● Distributed locking● Lease term tradeoffs

○ short vs long■ more efficient for frequently accessed

data■ minimized lease extension overhead on

server and client

Confidential + Proprietary

Leases

● Distributed locking● Lease term tradeoffs

○ short vs long● Use of leases in modern applications

○ Leader election TTL (in etcd)

Confidential + Proprietary

Leases

● Distributed locking● Lease term tradeoffs

○ short vs long● Use of leases in modern applications

○ Leader election TTL (in etcd)○ Liveness detection

Confidential + Proprietary

Confidential + Proprietary

Do the build, s'il vous plaît!

Build System

Dita Roque-Gourary

Confidential + Proprietary

Do the build, s'il vous plaît!

Build System

OK/D'Accord!

Dita Roque-Gourary

Confidential + Proprietary

Do the build, s'il vous plaît!

Build System

OK/D'Accord!

Waiting for the results, merci.

Dita Roque-Gourary

Confidential + Proprietary

Do the build, s'il vous plaît!

Build System

OK/D'Accord!

Waiting for the results, merci.

Build is in progress

Dita Roque-Gourary

Confidential + Proprietary

Do the build, s'il vous plaît!

Build System

OK/D'Accord!

Waiting for the results, merci.

Build is in progress

Dita Roque-Gourary

Waiting for the results, merci.

Confidential + Proprietary

Do the build, s'il vous plaît!

Build System

OK/D'Accord!

Waiting for the results, merci.

Build is in progress

Build is finished/fini.Dita Roque-Gourary

Waiting for the results, merci.

Confidential + Proprietary

Leases in Build System

Confidential + ProprietaryConfidential + Proprietary

Inaccurate Computations&Serving Search Results

Confidential + Proprietary

From Accurate to"Good Enough"

Confidential + Proprietary

Confidential + Proprietary

Confidential + Proprietary

Confidential + Proprietary

Confidential + Proprietary

Confidential + Proprietary

Inaccuracy for Resilience

1. Task decomposition

Confidential + Proprietary

Inaccuracy for Resilience

1. Task decomposition2. Baseline for correctness

Confidential + Proprietary

Inaccuracy for Resilience

1. Task decomposition2. Baseline for correctness3. Criticality Testing

Confidential + Proprietary

Inaccuracy for Resilience

1. Task decomposition2. Baseline for correctness3. Criticality Testing4. Distortion and timing models

Confidential + ProprietaryJeff Dean "Building Software Systems at Google and Lessons Learned", Stanford, 2010

Confidential + ProprietaryJeff Dean "Building Software Systems at Google and Lessons Learned", Stanford, 2010

Confidential + Proprietary

Confidential + Proprietary

Confidential + Proprietary

Robust & scaleable pipelines

Confidential + Proprietary

Robust & scaleable pipelinesLeases for sharing & heartbeat

Confidential + Proprietary

Robust & scaleable pipelinesLeases for sharing & heartbeat

Trade off inaccuracy for resilience & performance

Confidential + Proprietary

Robust, scaleable pipelinesLeases for sharing & heartbeat

Trade off inaccurate for resilient

CS research is timeless:

use it to mitigate risk

Confidential + Proprietary

ThanksInes SombraDavid GreenbergAlex HutchesonScott ZawalskiKaran Parikh

Confidential + Proprietary

References● T. Wurthinger, C. Wimmer et al. "One VM to Rule Them All"● M. Rinard "Probabilistic Accuracy Bounds for Fault-Tolerant

Computations that Discard Tasks"● F. Corbato, M. Daggett, R. Daley "An Experimental Time-

Sharing System"● E. Dijkstra "Cooperating Sequential Processes"● L. Lamport "Time, Clocks, and the Ordering of Events in a

Distributed System"

Confidential + Proprietary

References● B. Oki, B. Liskov "Viewstamped Replication: A New Primary Copy

Method to Support Highly-Available Distributed Systems"● L. Lamport "The Part-Time Parliament"● M. Welsh, D. Culler, E. Brewer "SEDA: An Architecture for Well-

Conditioned, Scalable Internet Services"● C. Gray, D. Cheriton "Leases: An Efficient Fault-Tolerant

Mechanism for Distributed File Cache Consistency"● S. Agarwal, B. Mozafari et al. "BlinkDB: Queries with Bounded

Errors and Bounded Response Times on Very Large Data"

Confidential + Proprietary

Should I read papers?

YES