*Lecture 3 Innerconnection Networks for Parallel ComputersParallel ComputingSpring 2010
*Innerconnection Networks for Parallel ComputersInterconnection networks carry data between processors and to memory. Interconnects are made of switches and links (wires, fiber). Interconnects are classified as static or dynamic. Static networksConsists of point-to-point communication links among processing nodesAlso referred to as direct networksDynamic networksBuilt using switches (switching element) and linksCommunication links are connected to one another dynamically by the switches to establish paths among processing nodes and memory banks.
*Static and DynamicInterconnection Networks Classification of interconnection networks: (a) a static network; and (b) a dynamic network.
*Network TopologiesBus-Based NetworksThe simplest network that consists a shared medium(bus) that is common to all the nodes.The distance between any two nodes in the network is constant (O(1)).Ideal for broadcasting information among nodes.Scalable in terms of cost, but not scalable in terms of performance. The bounded bandwidth of a bus places limitations on the overall performance as the number of nodes increases. Typical bus-based machines are limited to dozens of nodes.Sun Enterprise servers and Intel Pentium based shared-bus multiprocessors are examples of such architectures
*Network TopologiesCrossbar NetworksEmploys a grid of switches or switching nodes to connect p processors to b memory banks.Nonblocking network: the connection of a processing node to a memory bank doesnot block the connection of any other processing nodes to other memory banks.The total number of switching nodes required is (pb). (It is reasonable to assume b>=p)Scalable in terms of performanceNot scalable in terms of cost.Examples of machines that employ crossbars include the Sun Ultra HPC 10000 and the Fujitsu VPP500
*Network Topologies: Multistage NetworkMultistage NetworksIntermediate class of networks between bus-based network and crossbar network Blocking networks: access to a memory bank by a processor may disallow access to another memory bank by another processor.More scalable than the bus-based network in terms of performance, more scalable than crossbar network in terms of cost.The schematic of a typical multistage interconnection network.
*Network Topologies: Multistage Omega NetworkOmega networkConsists of log p stages, p is the number of inputs(processing nodes) and also the number of outputs(memory banks)Each stage consists of an interconnection pattern that connects p inputs and p outputs:Perfect shuffle(left rotation):
Each switch has two conncetion modes:Pass-thought conncetion: the inputs are sent straight through to the outputsCross-over connection: the inputs to the switching node are crossed over and then sent out.Has p/2*log p switching nodes
*Network Topologies: Multistage Omega NetworkA complete omega network connecting eight inputs and eight outputs.
An omega network has p/2 log p switching nodes, and the cost of such a network grows as (p log p).A complete Omega network with the perfect shuffle interconnects and switches can now be illustrated:
*Network Topologies: Multistage Omega Network RoutingLet s be the binary representation of the source and d be that of the destination processor.The data traverses the link to the first switching node. If the most significant bits of s and d are the same, then the data is routed in pass-through mode by the switch else, it switches to crossover.This process is repeated for each of the log p switching stages.Note that this is not a non-blocking switch.
*Network Topologies: Multistage Omega Network RoutingAn example of blocking in omega network: one of the messages (010 to 111 or 110 to 100) is blocked at link AB.
*Network Topologies - Fixed Connection Networks (static)Completely-connection NetworkStar-Connected NetworkLinear array2d-array or 2d-mesh or mesh3d-meshComplete Binary Tree (CBT)2d-Mesh of TreesHypercubeButterfly
*Evaluating Static Interconnection NetworkOne can view an interconnection network as a graph whose nodes correspond to processors and its edges to links connecting neighboring processors. The properties of these interconnection networks can be described in terms of a number of criteria.(1) Set of processor nodes V . The cardinality of V is the number of processors p (also denoted by n).(2) Set of edges E linking the processors. An edge e = (u, v) is represented by a pair (u, v) of nodes. If the graph G = (V,E) is directed, this means that there is a unidirectional link from processor u to v. If the graph is undirected, the link is bidirectional. In almost all networks that will be considered in this course communication links will be bidirectional. The exceptions will be clearly distinguished.(3) The degree du of node u is the number of links containing u as an endpoint. If graph G is directed we distinguish between the out-degree of u (number of pairs (u, v) E, for any v V ) and similarly, the in-degree of u. T he degree d of graph G is the maximum of the degrees of its nodes i.e. d = maxudu.
*Evaluating Static Interconnection Network (cont.)(4) The diameter D of graph G is the maximum of the lengths of the shortest paths linking any two nodes of G. A shortest path between u and v is the path of minimal length linking u and v. We denote the length of this shortest path by duv. Then, D = maxu,vduv. The diameter of a graph G denotes the maximum delay (in terms of number of links traversed) that will be incurred when a packet is transmitted from one node to the other of the pair that contributes to D (i.e. from u to v or the other way around, if D = duv). Of course such a delay would hold if messages follow shortest paths (the case for most routing algorithms).(5) latency is the total time to send a message including software overhead. Message latency is the time to send a zero-length message.(6) bandwidth is the number of bits transmitted in unit time.(7) bisection width is the number of links that need to be removed from G to split the nodes into two sets of about the same size (1).
*Network Topologies - Fixed Connection Networks (static)Completely-connection NetworkEach node has a direct communication link to every other node in the network.Ideal in the sense that a node can send a message to another node in a single step.Static counterpart of crossbar switching networksNonblockingStar-Connected NetworkOne processor acts as the central processor. Every other processor has a communication link connecting it to this central processor.Similar to bus-based network. The central processor is the bottleneck.
*Network Topologies: Completely Connected and Star Connected NetworksExample of an 8-node completely connected network.(a) A completely-connected network of eight nodes; (b) a star connected network of nine nodes.
*Network Topologies - Fixed Connection Networks (static)Linear arrayIn a linear array, each node(except the two nodes at the ends) has two neighbors, one each to its left and right.Extension: ring or 1-D torus(linear array with wraparound).2d-array or 2d-mesh or meshThe processors are ordered to form a 2-dimensional structure (square) so that each processor is connected to its four neighbor (north, south, east, west) except perhaps for the processors of the boundary. Extension of linear array to two-dimensions: Each dimension has p nodes with a node identified by a two-tuple (i,j).3d-meshA generalization of a 2d-mesh in three dimensions. Exercise: Find the characteristics of this network and its generalization in k dimensions (k > 2).Complete Binary Tree (CBT)There is only one path between any pair of two nodesStatic tree network: have a processing element at each node of the tree.Dynamic tree network: nodes at intermediate levels are switching nodes and the leaf nodes are processing elements.Communication bottleneck at higher levels of the tree.Solution: increasing the number of communication links and switching nodes closer to the root.
*Network Topologies: Linear ArraysLinear arrays: (a) with no wraparound links; (b) with wraparound link.
*Network Topologies: Two- and Three Dimensional MeshesTwo and three dimensional meshes: (a) 2-D mesh with no wraparound; (b) 2-D mesh with wraparound link (2-D torus); and (c) a 3-D mesh with no wraparound.
*Network Topologies: Tree-Based NetworksComplete binary tree networks: (a) a static tree network; and (b) a dynamic tree network.
*Network Topologies: Fat TreesA fat tree network of 16 processing nodes.
*Evaluating Network TopologiesLinear array|V | = N, |E| = N 1, d = 2, D = N 1, bw = 1 (bisection width).2d-array or 2d-mesh or meshFor |V | = N, we have a N N mesh structure, with |E| 2N = O(N), d = 4, D = 2N 2, bw = N.3d-meshExercise: Find the characteristics of this network and its generalization in k dimensions (k > 2).Complete Binary Tree (CBT) on N = 2n leavesFor a complete binary tree on N leaves, we define the level of a node to be its distance from the root. The root is of level 0 and the number of nodes of level i is 2i. Then, |V | = 2N 1, |E| 2N 2 = O(N), d = 3, D = 2lgN, bw = 1(c)
*Network Topologies - Fixed Connection Networks (static)2d-Mesh of TreesAn N2-leaf 2d-MOT consists of N2 nodes ordered as in a 2d-array N N (but without the links). The N rows and N columns of the 2d-MOT form N row CBT and N column CBTs respectively. For such a network, |V | = N2+2N(N 1), |E| = O(N2), d = 3, D = 4lgN, bw = N. The 2d-MOT possesses an interesting decomposition property. If the 2N roots of the CBTs are removed we get 4 N/2 N/2 CBT s. The 2d-Mesh of Trees (2d-MOT) combines the advantage