4th PFI System reading

  • View
    479

  • Download
    0

Embed Size (px)

Text of 4th PFI System reading

  • How Soccer Players Would do Stream Joins

    3/4/2015 @nobu_k

    1

  • Who? (@nobu_k)

    CTO@Preferred Networks America, Inc.

    Speciality

    DBMS, Search engine

    Distributed Systems (consensus)

    beatmania IIDX SP/DP (DP)

    2

  • How Soccer Players Would do Stream Joins

    Jens Teubner, Rene Mueller, SIGMOD 2011

    Handshake Join

    Window-based stream joins supporting any join predicate

    Very high degrees of parallelism

    multi-core CPUs

    FPGA

    Massively Parallel Processor Arrays (MPPAs)

    3

  • JoinsJoins() combine two or more relations(tables) in RDBMS

    A join is a cross product of relations followed by a selection() Many methods

    Nested-loops joins

    Sort-merge joins

    (Recursive, hybrid) hash joins

    4

  • Stream JoinsProblems

    Unbounded "infinite" input data

    Solution: (sliding) window-based joins

    tuple-based/time-based

    Latency of the output

    Solution: online, symmetric evaluation

    How can it be scalable?

    5

  • Handshake Joins

    Streams flow by each other in opposite directions.

    Each core locally evaluates tuples.

    Core 1 Core 2 Core 3

    6

  • A newly arrived tuple( ) will be compared to all tuples( ) in the other stream in the same core.

    Any comparison algorithm(predicate) can be used.

    Evaluation Strategy

    7

  • Strategies

    Lock Step Forwarding

    Two-Phase Forwarding using Async-MQ

    Asymmetric protocol

    Synchronization

    ba

    a and b miss each other.

    8

  • Two-Phase Forwarding Using Asynchronous Message Queue

    ba

    b

    ab

    Leaving the tuple with a special mark.

    FIFO queue

    9

    1.

    2.

  • Two-phase forwarding: ACK

    b

    ab

    b

    ab

    b

    When the left core receives tuple b, it sends an ack to the right core before sending any other tuples.

    The right core deletes b when it receives the ack.

    10

    2.

    3.

  • Two-Phase Forwarding: when a and b miss each other

    bba

    bba

    a will be compared to tuples in the right core including b.

    11

    4.

    5.

  • Load Balancing

    Automatic load balancing without centralized control.

    Each core can handle an arbitrary number of tuples.

    Core 1 Core 2 Core 3

    12

  • Software Implementation

    AMD Opteron 6174 2.2GHz

    libnuma

    quoted from page 8

    13

  • Scalability

    page 8 page 9

    14

  • FPGA Implementation

    Assume the system has to provide a throughput of 500ktuples/sec with a window size of 100 tuples. Config- urations with 1, 2, 5, and 10 join cores can guarantee this throughput if operated at clock frequencies of 50, 25, 10, or 5 MHz, respectively. from page 10

    15

  • Summary

    Handshake join

    Window-based stream join

    Flexible and scalable

    Working well with FPGA

    16