CSE 8383 - Advanced Computer Architecture
Week-12April 8, 2004
engr.smu.edu/~rewini/8383
Contents Dynamic Networks Message Passing Mechanisms Message Passing in PVM
Dynamic Network Analysis Parameters:
Cost: number of switches Delay: latency Blocking characteristics Fault tolerance
MIMD Distributed Memory Systems
Interconnection Networks
M M M M
P P P P
Interconnection Network Taxonomy
Interconnection Network
Static Dynamic
Bus-based Switch-based1-D 2-D HC
Single Multiple SS MS Crossbar
Dynamic Interconnection Networks Communication patterns are based
on program demands
Connections are established on the fly during program execution
Multistage Interconnection Network (MIN) and Crossbar
Switch Modules A x B switch module A inputs and B outputs In practice, A = B = power of 2 Each input is connected to one or
more outputs (conflicts must be avoided)
One-to-one (permutation) and one-to-many are allowed
Binary Switch
2x2Switch
Legitimate States = 4
Permutation Connections = 2
Legitimate Connections
Straight Exchange
Upper-broadcast
Lower-broadcastThe different setting of the 2X2 SE
Group WorkGeneral Case ??
Multistage Interconnection Networks
ISC1ISC1 ISC2ISC2 ISCnISCn
switches switches switches
ISC Inter-stage Connection Patterns
Perfect-Shuffle Routing Function Given x = {an, an-1, …, a2, a1} P(x) = {an-1, …, a2, a1 , an}
X = 110001P(x) = 100011
Perfect Shuffle Example000 000001 010010 100011 110100 001101 011110 101111 111
Perfect-Shuffle000001010011
100101110111
000001010011
100101110111
Exchange Routing Function Given x = {an, an-1, …, a2, a1}
Ei(x) = {an, an-1, …, ai, …, a2, a1}
X = 0000000E3(x) = 0000100
Exchange E1
000 001001 000010 011011 010100 101101 100110 111111 110
Exchange E1
000001010011
100101110111
000001010011
100101110111
Butterfly Routing Function Given x = {an, an-1, …, a2, a1} B(x) = {a1 , an-1, …, a2, an}
X = 010001P(x) = 110000
Butterfly Example000 000001 100010 010011 110100 001101 101110 011111 111
Butterfly000001010011
100101110111
000001010011
100101110111
Multi-stage network000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
MIN (cont.)1
2
3
4
5
6
7
8
9
10
11
12
001010011
100101
110111
000000001
010011
100101
110111
An 8X8 Banyan network
Min ImplementationControl (X)
Source (S) Destination (D)
X = f(S,D)
Example
X = 0 X = 1
(crossed) (straight)
A
B
C
D
A
B
C
D
Consider this MINS1S2
S3
S4
S5
S6
S7
S8
D1D2
D3D4
D5
D6
D7
D8
stage 1 stage 2 stage 3
Example (Cont.) Let control variable be X1, X2, X3 Find the values of X1, X2, X3 to connect:
S1 D6 S7 D5 S4 D1
The 3 connectionsS1S2
S3
S4
S5
S6
S7
S8
D1D2
D3D4
D5
D6
D7
D8
stage 1 stage 2 stage 3
Boolean Functions X = x1, x2, x3 S = s2, s2, s3 D = d1, d2, d3
Find X = f(S,D)
Crossbar Switch M1 M2 M3 M4 M5 M6 M7 M8
P1
P2
P3
P4
P5
P6
P7
P8
Analysis and performance metricsdynamic networks Performance comparison of dynamic networks
Networks Delay Cost Blocking
Degree of FT
Bus O(N) O(1) Yes 0Multiple-bus O(mN) O(m) Yes (m-1)
MIN O(logN) O(NlogN) Yes 0Crossbar O(1) O(N2) No 0
Message Passing Mechanisms Message Format
Message arbitrary number of fixed length packets
Packet basic unit containing destination address. Sequence number is needed
A packet can further be divided into flits (flow control digits)
Routing and sequence occupy header flit
Message, Packets, Flits
Message
Packet
Data flit
Destination
Sequence
Store and Forward Routing Packets are the basic units of
information flow Each node uses a packet buffer A packet is transferred from S to D
through a sequence of intermediate nodes
Channel and buffer must be available
Wormhole Routing Flits are the basic units of information
flow Each node uses a flit buffer Flits are transferred from S to D through a
sequence of intermediate routers in order (Pipeline)
Can be visualized as a railroad train Flits from different packets cannot be
mixed up
Latency Analysis L packet length (in bits) W Channel bandwidth (bits/sec) D Distance (number of hops) F flit length (in bits) TSF = D * L/W TWH = L/W + D* F/W L/W if L>>F
(independent of D)
Communication Patterns Point to Point 1 - 1 Multicast 1 - n Broadcast 1 - all Conference n - n
Routing Efficiency Two Parameters
Channel Traffic (number of channels used to deliver the message involved)
Communication Latency (distance)
Multicast on a mesh (5 unicasts)
Traffic ?
Latency ?
Multicast on a mesh (multicast pattern 1
Traffic ?
Latency ?
Multicast on a mesh (multicast pattern 2)
Traffic ?
Latency ?
Broadcast (tree structure)3 2 3 4
2 1 2 3
1 1 2
Message Passing in PVM
User application
Library
Daemon
1
2 3
4
User application
Library
Daemon
5
6 7
8
Sending Task Receiving Task
Standard PVM asynchronous communication A sending task issues a send command
(point 1) The message is transferred to the
daemon (point 2) Control is returned to the user
application (points 3 & 4) The daemon will transmit the message
on the physical wire sometime after returning control to the user application (point 3)
Standard PVM asynchronous communication (cont.) The receiving task issues a receive
command (point 5) at some other time In the case of a blocking receive, the
receiving task blocks on the daemon waiting for a message (point 6). After the message arrives, control is returned to the user application (points 7 & 8)
In the case of a non-blocking receive, control is returned to the user application immediately (points 7 & 8)
Send (3 steps)1. A send buffer must be initialized2. The message is packed into the
buffer3. The completed message is sent
to its destination(s)
Receive (2 steps)1. The message is received2. The received items are unpacked
Message Buffers Buffer Creation (before packing)
Bufid = pvm_initsend(encoding_option)
Bufid = pvm_mkbuf(encoding_option)
Encoding option Meaning0 XDR1 No encoding2 Leave data in place
Message Buffers (cont.) Data Packing
pvm_pk*() pvm_pkstr() – one argument
pvm_pkstr(“This is my data”); Others – three arguments
1. Pointer to the first item2. Number of items to be packed3. Stride
pvm_pkint(my_array, n, 1); Packing functions can be called multiple
times to pack data into a single message
Sending a message Point to point (one receiver)
info = pvm_send(tid, tag)
broadcast (multiple receivers)info = pvm_mcast(tids, n, tag)info = pvm_bcast(group_name, tag)
Pack and Send (one step)info = pvm_psend(tid, tag, my_array, length, data type)
Receiving a message Blocking
bufid = pvm_recv(tid, tag)-1 wild card in either tid or tag
Nonblocking bufid = pvm_nrecv(tid, tag) bufid = 0 (no message was received)
Timeout bufid = pvm_trecv(tid, tag, timeout)bufid = 0 (no message was received)
Different Receive in PVM
Pvm_recv()
wait
Time
Funcitonis called
Time is expired
Message arrival
Blocking
Pvm_nrecv()
Continue execution
Non-blocking
Pvm_trecv()
wait
Timeout
Resume execution
Resume execution
Data unpackingpvm_upk*() pvm_upkstr() – one argument
pvm_upkstr(string);
Others – three arguments1. Pointer to the first item2. Number of items to be unpacked3. Stride
pvm_upkint(my_array, n, 1);