Distributed (Operating) Systems -Message-oriented Communication-

-Stream-oriented Communication-

Computer Engineering DepartmentDistributed Systems Course

Asst. Prof. Dr. Ahmet SayarKocaeli University - Fall 2014

Communication Issues

• Message-oriented communication– Transient vs Persistent– Synchronous vs asynchronous

• Stream-oriented communication

Message-oriented Transient Communication

• Many distributed systems built on top of simple message-oriented model– Example: Berkeley sockets – transport level sockes

Berkeley Socket Primitives

Message-Passing Interface (MPI)

• Sockets are designed for network communication (e.g., TCP/IP)– Support simple send/receive primitives

• Abstraction not suitable for other protocols in clusters of workstations or massively parallel systems– Need an interface with more advanced primitives

• Large number of incompatible proprietary libraries and protocols– Need for a standard interface

• Message-passing interface (MPI)– Hardware independent– Designed for parallel applications (uses transient communication)

MPI

• Key idea: communication between groups of processes– Each endpoint is a (groupID, processID) pair

• MPI are intent to be used in clusters• It is basically used for scientific applications and parallel

systems• It overcomes some of the TCP communications• MPI basically run on UDP• Socket based communication is also transient but it is

based on synch communication– but here MPI also support asynchronous communication

MPI Primitives

Message-oriented Persistent Communication

• Message queuing systems or Message oriented Middleware (MOM)– Support asynchronous persistent communication– Intermediate storage for message while sender/receiver are

inactive– Example application: email

• Communicate by inserting messages in queues• Sender is only guaranteed that message will be

eventually inserted in recipient’s queue– No guarantees on when or if the message will be read– “Loosely coupled communication”

Message-Queuing Model (1)

Basic interface to a Queue

• Put: append a message to a specified queue

• Get: block until the specified queue is nonempty and remove the first message

• Poll: check a specified queue for messages and removes the first . Never block

• Notify: install a handler to be called when a message is put into the specified queue.

Message-Queuing Model

General Architecture of a Message-Queuing System (2)

• Queue manager and relays– Relays use an

overlay network– Relays know

about the network topology and how to route

Message Brokers

• Message broker: application level gateway in MQS– Convert incoming messages so that they can be understood by

destination (format conversion)– Also used for pub-sub systems

IBMʼs WebSphere MQ

• Queue managers manage queues– Connected through message channels

• Message channel agent (MCA)– Checks queue, wraps into TCP packet, send to receiving MCA

Stream Oriented Communication

• Communication as discussed so far – has concentrated on exchanging more-or-less independent and

complete units of information.– It does not matter at what particular point in time communication

takes place.– Timing has no effect on correctness.

• Continuous media: the temporal relationship between different data items are fundamental to correctly interpreting what the data actually means.– Audio and video streams

• Discrete media: temporal information is not important, ex. text and still images

Stream Oriented Communication

• Timing is crucial in certain forms of communication– Examples: audio and video (“continuous media”)– 30 frames/s video => receive and display a frame every 30ms– If there is increasing delay there might be some frames

dropped but human eye can see it• TCP does not guarantee short time latency• TCP enables retransmitting but frame should be

displayed at right time and right place– Retransmit does not make sense here

• UDP is actually better protocol for streaming

Data Streams

• A sequence of data units• Discrete vs. continuous data streams• Tcp/ip connections – discrete data streams – file

transfers• Playing an audio – continuous data streams– Isochronous communication : Data transfers have a

maximum bound on end-end delay and jitter– Push mode: no explicit requests for individual data units

beyond the first “play” request• Streaming live data vs. streaming stored data

To Push or Pull ?

• Client-pull architecture– Clients pull data from servers (by sending requests)– Example: HTTP– Pro: stateless servers, failures are easy to handle– Con: limited scalability

• Server-push architecture– Servers push data to client– Example: video streaming, stock tickers– Pro: more scalable, Con: stateful servers, less resilient to

failure• When/how-often to push or pull?

Examples

Streaming

• Live stream based comm, real time issues come up, ex- video conferencing, skype

• Stored stream based comm, latency does not matter, ex- youtube

• Live streaming has even more stringent constraints

• Unicast: N sends for n receiver• Multicast: 1 send for n receiver

Streams and Quality of Service

• Time-dependent and other requirements are specified as quality of service (QoS)– Requirements/desired guarantees from the underlying systems– Application specifies workload and requests a certain service quality– Contract between the application and the system

• Properties for Quality of Service:– The certain loss rate– The required bit rate at which data should be transported.– The maximum delay until a session has been set up– The maximum end-to-end delay .– The maximum delay variance, or jitter.– The maximum round-trip delay.– The quality of video

Specifying QoS: Token bucket

• The principle of a token bucket algorithm– Parameters (rate r, burst b)– Rate is the average rate, burst is the maximum number of packets that can

arrive simultaneously

Enforcing QoS (1)• Observation: There are various network-level tools, such as

differentiated services by which certain packets can be prioritized.

• Also: use buffers to reduce jitter

Enforcing QoS (2)•Problem: How to reduce the effects of packet loss (when multiple samples are in a single packet)?•Solution: simply spread the samples:

Stream synchronization

• Multiple streams:– Audio and video; layered video

• Need to sync prior to playback– Timestamp each stream and sync up data units

prior to playback• Sender or receiver?• App does low-level sync– 30 fps: image every 33ms, lip-sync with audio

• Use middleware and specify playback rates

• Problem: Given a complex stream, how do you keep the different substreams in synch?

• Example: Think of playing out two channels, that together form stereo sound. Difference should be less than 20–30 μsec!

• Alternative: multiplex all substreams into a single stream, and demultiplex at the receiver. Synchronization is handled at multiplexing/demultiplexing point (MPEG).

Synchronization Mechanism

Multicasting

• Multicast: sending data to multiple receivers• Network- and transport-layer protocols for

multicast bogged down at the issue of setting up the communication paths to all receivers.

• Group communication– IP multicast versus application-level multicast– Construct an overlay multicast tree rooted at the sender– Send packet down each link in the tree

• Issues: tree construction, dynamic joins and leaves

Application-Level Multicasting

• IP multicasting works in IP layer.– Why not just use that?– Often difficult to configure, requires too much

administrative support, agreement, etc.• Application-level seeks to address those issues.

• Peer-to-peer communication using structured overlays can use application-layer protocols to support multicast

• The overlay network is used to disseminate information to members

Application-Level Multicasting

• Essence: Organize nodes of a distributed system into an overlay network and use that network to disseminate data.

• Two possible structures:– Tree: unique path between every pair of nodes– Mesh: multiple neighbors ensure multiple paths

(more robust)