Advanced Distributed Systems - ERNETsrsarangi/csl860/docs/intro.pdf · Message-Oriented Communication Smruti R. Sarangi Advanced Distributed Systems. Plan Introduction Building a

PlanIntroduction

Building a Distributed System

Advanced Distributed SystemsCourse Plan and Overview

Smruti R. Sarangi

Department of Computer ScienceIndian Institute of Technology

New Delhi, India

Smruti R. Sarangi Advanced Distributed Systems

PlanIntroduction


Outline

1 Plan

2 IntroductionOverviewDeveloping Distributed Systems: Pitfalls

3 Building a Distributed SystemComputationCommunicationRemote Procedure CallsMessage-Oriented Communication


PlanIntroduction


Lectures - I

Lecture Topic1 Introduction and Structure2 Communication – Client Server, RPC

Message Queue, Stream3 Multicast Communication – Epidemic, Gossip4 Naming : DNS, Chord5 Naming : Pastry, Tapestry, LDAP6 Physical Clock , Logical Clock, Totally ordered multicast7 Mutual Exclusion Algorithms8 Leader Election Algorithms9 Consistency - I10 Consistency - II11 Fault Tolerance – Introduction, Log based recovery12 Byzantine Fault Tolerance


PlanIntroduction


Lectures - II

Lecture Topic13 Paxos14 Distributed Commit15 Security – Channels, Concepts16 Security Applications – Kerberos

Diffie Helmann Key Exchange17 Corba, EJB18 AFS and NFS19 Akamai and Corona20 Dynamo and Voldemort21 Coda, Fawn, and Google FS22 Web Services23 Dryad-LinQ


PlanIntroduction


Grading Scheme

Component WeightageAttendance 15%Midterm 15%End term (take home) 20%Programming Assignment 1 15%Programming Assignment 2 15%Programming Assignment 3 20%


PlanIntroduction


OverviewDeveloping Distributed Systems: Pitfalls

Outline

1 Plan




PlanIntroduction



Distributed System: Definition

A distributed system is a piece of software that ensures that:

a collection of independent computers appears to itsusers as a single coherent system

Two aspects:1 independent computers2 single coherent system ⇒ middleware .


PlanIntroduction



Goals of Distributed Systems

GoalsMaking resources availableDistribution transparencyOpennessScalability


PlanIntroduction



Distribution Transparency

Transp.Description

Access Hides differences in data representation and invocationmechanisms

Location Hides where an object residesMigration Hides from an object the ability of a system to change that

object’s locationRelocation Hides from a client the ability of a system to change the

location of an object to which the client is boundReplication Hides the fact that an object or its state may be replicated

and that replicas reside at different locationsConcurrency Hides the coordination of activities between objects to

achieve consistency at a higher levelFailure Hides failure and possible recovery of objects

NoteDistribution transparency is a nice a goal, but achieving it is a different story.


PlanIntroduction



Distribution Transparency

Transp.Description

Access Hides differences in data representation and invocationmechanisms

Location Hides where an object residesMigration Hides from an object the ability of a system to change that

object’s locationRelocation Hides from a client the ability of a system to change the

location of an object to which the client is boundReplication Hides the fact that an object or its state may be replicated

and that replicas reside at different locationsConcurrency Hides the coordination of activities between objects to

achieve consistency at a higher levelFailure Hides failure and possible recovery of objects

NoteDistribution transparency is a nice a goal, but achieving it is a different story.


PlanIntroduction



Degree of Transparency

Aiming at full distribution transparency may be too much

Users may be located in different continents

Completely hiding failures of networks and nodes is (theoreticallyand practically) impossible

You cannot distinguish a slow computer from a failing oneYou can never be sure that a server actually performed anoperation before a crash

Full transparency will cost performance , exposing distribution ofthe system

Keeping Web caches exactly up-to-date with the masterImmediately flushing write operations to disk for fault toler-ance


PlanIntroduction











PlanIntroduction











PlanIntroduction











PlanIntroduction



Openness of Distributed Systems

Open distributed system ⇒ Be able to interact with servicesfrom other open systems, irrespective of the underlying en-vironment:

Systems should conform to well-defined interfacesSystems should support portability of applicationsSystems should easily interoperate

Achieving openness ⇒ At least make the distributed systemindependent from heterogeneity of the underlying environ-ment:

HardwarePlatformsLanguages


PlanIntroduction



Openness of Distributed Systems

Open distributed system ⇒ Be able to interact with servicesfrom other open systems, irrespective of the underlying en-vironment:

Systems should conform to well-defined interfacesSystems should support portability of applicationsSystems should easily interoperate

Achieving openness ⇒ At least make the distributed systemindependent from heterogeneity of the underlying environ-ment:

HardwarePlatformsLanguages


PlanIntroduction



Scale in Distributed Systems

ObservationMany developers of modern distributed system easily use the adjective“scalable” without making clear why their system actually scales.

Scalability – At least three components:

Number of users and/or processes (size scalability)

Maximum distance between nodes (geographical scalability)

Number of administrative domains (administrative scalability)

ObservationMost systems account only, to a certain extent, for size scalability. The(non)solution: powerful servers. Today, the challenge lies in geograph-ical and administrative scalability.


PlanIntroduction











PlanIntroduction











PlanIntroduction



Techniques for Scaling

Hide communication latencies – Avoid waiting for responses;do something else:

Make use of asynchronous communicationHave separate handler for incoming responseProblem: not every application fits this model


PlanIntroduction




DistributionPartition data and computations across multiple machines:

Move computations to clients (Java applets)Decentralized naming services (DNS)Decentralized information systems (WWW)


PlanIntroduction




Replication/cachingMake copies of data available at different machines:

Replicated file servers and databasesMirrored Web sitesWeb caches (in browsers and proxies)File caching (at server and client)


PlanIntroduction



Scaling – The Problem

ObservationApplying scaling techniques is easy, except for one thing:

Having multiple copies (cached or replicated), leads to in-consistencies : modifying one copy makes that copy differ-ent from the rest.Always keeping copies consistent and in a general way re-quires global synchronization on each modification.Global synchronization precludes large-scale solutions.

ObservationIf we can tolerate inconsistencies, we may reduce the need forglobal synchronization, but tolerating inconsistencies is applica-tion dependent .


PlanIntroduction





Having multiple copies (cached or replicated), leads to in-consistencies : modifying one copy makes that copy differ-ent from the rest.

Always keeping copies consistent and in a general way re-quires global synchronization on each modification.Global synchronization precludes large-scale solutions.



PlanIntroduction





Having multiple copies (cached or replicated), leads to in-consistencies : modifying one copy makes that copy differ-ent from the rest.Always keeping copies consistent and in a general way re-quires global synchronization on each modification.

Global synchronization precludes large-scale solutions.



PlanIntroduction








PlanIntroduction








PlanIntroduction



Outline

1 Plan




PlanIntroduction



Types of Distributed Systems : Structure

Centralized Layered

Unstructured Peer to Peer Structured Peer to Peer


PlanIntroduction



Types of Distributed Systems : Synchronous vsAsynchronous

DefinitionSynchronous System The requester waits for the response be-fore placing other requests.

DefinitionAsynchronous System The requester does not wait for the re-sponse before placing other requests. One example of suchsystems are publish/subscribe systems. A given set of nodessubscribe for a service. When a node is ready to publish somedata, it looks up the list of subscribers and sends them mes-sages.


PlanIntroduction



Developing Distributed Systems: Pitfalls

ObservationMany distributed systems are needlessly complex caused bymistakes that required patching later on. There are many falseassumptions :

The network is reliableThe network is secureThe network is homogeneousThe topology does not changeLatency is zeroBandwidth is infiniteTransport cost is zeroThere is one administrator


PlanIntroduction





The network is reliable

The network is secureThe network is homogeneousThe topology does not changeLatency is zeroBandwidth is infiniteTransport cost is zeroThere is one administrator


PlanIntroduction





The network is reliableThe network is secure

The network is homogeneousThe topology does not changeLatency is zeroBandwidth is infiniteTransport cost is zeroThere is one administrator


PlanIntroduction





The network is reliableThe network is secureThe network is homogeneous

The topology does not changeLatency is zeroBandwidth is infiniteTransport cost is zeroThere is one administrator


PlanIntroduction





The network is reliableThe network is secureThe network is homogeneousThe topology does not change

Latency is zeroBandwidth is infiniteTransport cost is zeroThere is one administrator


PlanIntroduction





The network is reliableThe network is secureThe network is homogeneousThe topology does not changeLatency is zero

Bandwidth is infiniteTransport cost is zeroThere is one administrator


PlanIntroduction





The network is reliableThe network is secureThe network is homogeneousThe topology does not changeLatency is zeroBandwidth is infinite

Transport cost is zeroThere is one administrator


PlanIntroduction





The network is reliableThe network is secureThe network is homogeneousThe topology does not changeLatency is zeroBandwidth is infiniteTransport cost is zero

There is one administrator


PlanIntroduction





The network is reliableThe network is secureThe network is homogeneousThe topology does not changeLatency is zeroBandwidth is infiniteTransport cost is zeroThere is one administrator


PlanIntroduction


ComputationCommunicationRemote Procedure CallsMessage-Oriented Communication

Computation and Communication

Components of a distributed SystemNodes that run the distributed software. ( Computation )Communication network. ( Communication )


PlanIntroduction



Outline

1 Plan




PlanIntroduction



Computing Nodes

A compute node can either be a process or a thread

ThreadA thread is a light weight process that shares the address spacewith other threads.

ProcessA process is the runtime image of a program. It does not shareits address space.

We can use regular memory reads and writes to commu-nicate across threads. However, inter-process communica-tion requires OS intervention .In practice, there are thousands of threads and processesdistributed across hundreds of sites.


PlanIntroduction



Computing Nodes - II

Generic solution: Message passing across threads/processes.Minimize shared data.Fetch shared data through messages from a database.


PlanIntroduction



Outline

1 Plan




PlanIntroduction



How do Nodes Talk to Each Other?

Distributed systems predominantly use application layer pro-tocols.This protocol layer is known as middleware .They use standard sockets to send messages to other nodes.Paradigms for sending messages

1 Synchronous vs Asynchronous2 Transient vs Persistent


PlanIntroduction



Middleware Layer

ObservationMiddleware is invented to provide common services and proto-cols that can be used by many different applications

A rich set of communication protocols(Un)marshaling of data, necessary for integrated systemsNaming protocols , to allow easy sharing of resourcesSecurity protocols for secure communicationScaling mechanisms , such as for replication and caching

NoteWhat remains are truly application-specific protocols...such as?


PlanIntroduction



Outline

1 Plan




PlanIntroduction



Basic RPC operation

Application developers are familiar with a simple procedure modelWell-engineered procedures operate in isolation (black box)There is no fundamental reason not to execute procedures on sep-arate machines

Call local procedureand return results

Call remoteprocedure

Returnfrom call

Client

Request Reply

ServerTime

Wait for result

ConclusionCommunication between caller & callee can be hidden by usingprocedure-call mechanism.


PlanIntroduction



Basic RPC operation

Implementationof add

Client OS Server OS

Client machine Server machine

Client stub

Client process Server process1. Client call to

procedure

2. Stub buildsmessage

5. Stub unpacksmessage

6. Stub makeslocal call to "add"

3. Message is sentacross the network

4. Server OShands messageto server stub

Server stubk = add(i,j) k = add(i,j)

proc: "add"int: val(i)int: val(j)



1 Client procedure calls client stub.2 Stub builds message; calls local OS.3 OS sends message to remote OS.4 Remote OS gives message to stub.5 Stub unpacks parameters and calls

server.

6 Server returns result to stub.7 Stub builds message; calls OS.8 OS sends message to client’s OS.9 Client’s OS gives message to stub.

10 Client stub unpacks result and returnsto the client.


PlanIntroduction



RPC: Parameter passing

Parameter marshalingThere’s more than just wrapping parameters into a message:

Client and server machines may have different data repre-sentations (think of byte ordering)Wrapping a parameter means transforming a value into asequence of bytesClient and server have to agree on the same encoding :

How are basic data values represented (integers, floats,characters)How are complex data values represented (arrays, unions)

Client and server need to properly interpret messages ,transforming them into machine-dependent representations.


PlanIntroduction



RPC: Parameter passing

RPC parameter passing: some assumptions

Copy in/copy out semantics: while procedure is executed, noth-ing can be assumed about parameter values.

All data that is to be operated on is passed by parameters. Ex-cludes passing references to (global) data.

ConclusionFull access transparency cannot be realized.

A remote reference mechanism enhances access transparency:

Remote reference offers unified access to remote data

Remote references can be passed as parameter in RPCs


PlanIntroduction



Asynchronous RPCs

EssenceTry to get rid of the strict request-reply behavior, but let the clientcontinue without waiting for an answer from the server.

Call local procedure


Returnfrom call

Request Accept request

Wait for acceptance

Call local procedureand return results


Returnfrom call

Client Client

Request Reply

Server ServerTime Time

Wait for result

(a) (b)


PlanIntroduction



Deferred synchronous RPCs

Call local procedure


Returnfrom call

Client

RequestAcceptrequest

ServerTime

Wait foracceptance

Interrupt client

Returnresults Acknowledge

Call client withone-way RPC

VariationClient can also do a (non)blocking poll at the server to seewhether results are available.


PlanIntroduction



RPC in practice

C compiler

Uuidgen

IDL compiler

C compiler C compiler

Linker Linker

C compiler

Server stubobject file

Serverobject file

Runtimelibrary

Serverbinary

Clientbinary

Runtimelibrary

Client stubobject file

Clientobject file

Client stubClient code Header Server stub

Interfacedefinition file

Server code

#include#include


PlanIntroduction



Client-to-server binding (DCE)

Issues(1) Client must locate server machine, and (2) locate the server.

Endpointtable

Server

DCEdaemon

Client1. Register endpoint

2. Register service3. Look up server

4. Ask for endpoint

5. Do RPC

Directoryserver

Server machineClient machine

Directory machine


PlanIntroduction



Outline

1 Plan




PlanIntroduction



Message-Oriented Communication

Transient MessagingMessage-Queuing SystemMessage BrokersExample: IBM WebSphere


PlanIntroduction



Transient messaging: sockets

Berkeley socket interface

SOCKET Create a new communication endpointBIND Attach a local address to a socketLISTEN Announce willingness to accept N connectionsACCEPT Block until request to establish a connectionCONNECT Attempt to establish a connectionSEND Send data over a connectionRECEIVE Receive data over a connectionCLOSE Release the connection


PlanIntroduction



Transient messaging: sockets

connect

socket

socket

bind listen read

read

write

write

accept close

close

Server

Client

Synchronization point Communication


PlanIntroduction



Message-oriented middleware

EssenceAsynchronous persistent communication through support ofmiddleware-level queues. Queues correspond to buffers at com-munication servers.

PUT Append a message to a specified queueGET Block until the specified queue is nonempty, and re-

move the first messagePOLL Check a specified queue for messages, and remove

the first. Never blockNOTIFY Install a handler to be called when a message is put

into the specified queue


PlanIntroduction



Message broker

ObservationMessage queuing systems assume a common messaging pro-tocol : all applications agree on message format (i.e., structureand data representation)

Message broker: Centralized component that takes care ofapplication heterogeneity in an MQ system

Transforms incoming messages to target formatVery often acts as an application gatewayMay provide subject-based routing capabilities ⇒ Enter-prise Application Integration


PlanIntroduction



Message broker

Queuinglayer

Brokerprogram

Repository with conversion rules and programsSource client Destination client

OS OSOS

Message broker

Network


PlanIntroduction



IBM’s WebSphere MQ

Application-specific messages are put into, and removedfrom queuesQueues reside under the regime of a queue managerProcesses can put messages only in local queues, or throughan RPC mechanism


PlanIntroduction




Message TransferMessages are transferred between queuesMessage transfer between queues at different processes,requires a channelAt each endpoint of channel is a message channel agentMessage channel agents are responsible for:

Setting up channels using lower-level network communica-tion facilities (e.g., TCP/IP)(Un)wrapping messages from/in transport-level packetsSending/receiving packets


PlanIntroduction




MCA MCAMCA MCA

MQ Interface

Stub StubServerstub

Serverstub

Send queue

Program ProgramQueuemanager

Queuemanager

Routing table

Enterprise networkRPC(synchronous)

Local network

Message passing(asynchronous)

To other remotequeue managers

Client's receivequeueSending client Receiving client

Channels are inherently unidirectionalAutomatically start MCAs when messages arriveAny network of queue managers can be createdRoutes are set up manually (system administration)


PlanIntroduction




Routing

By using logical names, in combination with name resolution to localqueues, it is possible to put a message in a remote queue

SQ1SQ2

SQ1

SQ1SQ1SQ2

QMBQMCQMD

SQ1

SQ1SQ1

SQ1

SQ2SQ1

SQ1

SQ1SQ1

QMA

QMA QMA

QMC

QMCQMB

QMD

QMBQMD

Routing tableRouting table

Routing table Routing table

QMB

QMC

QMA

LA1LA1

LA1

LA2LA2

LA2

QMCQMA

QMA

QMDQMD

QMC

Alias tableAlias table

Alias table

QMD

SQ1

SQ2

SQ1


Documents

Advanced Distributed Systems - ERNETsrsarangi/csl860/docs/intro.pdf · Message-Oriented Communication Smruti R. Sarangi Advanced Distributed Systems. Plan Introduction Building a