Lecture 22 Interconnection Networks

Computer Architecture

COE 501

Network Media

Copper, 1mm think, twisted to avoidattenna effect (telephone)

Used by cable companies: high BW, good noise immunity

Light: 3 parts are cable, light source, light detector.Multimode light disperse (LED), Single mode sinle wave (laser)

Twisted Pair:

Coaxial Cable:

Copper coreInsulator

Braided outer conductorPlastic Covering

Fiber Optics

Transmitter– L.E.D– Laser Diode

Receiver– Photodiode

lightsource Silica

Total internalreflectionAir

Shared vs. Switched Media

• Shared media: nodes share a single interconnection medium (e.g., Ethernet)– Only one message sent at a time

– Inexpensive : one medium used by all processors

– Limited bandwidth : medium becomes the bottleneck

– Needs arbitration

• Switched media : allow direct communication between source and destination nodes (e.g., ATM)– Multiple users at a time

– More expensive : need to replicated medium

– Higher total bandwidth

– No arbitration

– Added latency to go through the switch

Ethernet Protocol - Shared• Ethernet is an example of a shared media - similar to

a bus

• The ethernet nodes must coordinate so that only one message is sent at a time.– Nodes “listens” to the network to ensure it is not being used.

– If the network is not being used, the node tries to send the message.

– If another node tries to send a message at the same time, a collision occurs

– Both nodes detect the collision and try to resend their messages after a random amount of time

– Subsequent collisons result in exponentially increasing random time between attempts to resend data

– What are advantages and disadvantages of this method?

Switch Topology• Switched media have a topology that indicate how

nodes are connected

• Topology determines– Degree: number of links from a node

– Diameter: max number of links crossed between nodes

– Average distance: number of links to random destination

– Bisection: minimum number of links that separate the network into two halves

– Bisection bandwidth: link bandwidth x bisection

• Warning: Three-dimensional drawings must be mapped onto chips and boards which are essentially two-dimensional media– Elegant when sketched on the blackboard may look awkward

when actually constructed

Common Topologies

Type Degree Diameter Ave Dist Bisection

1D mesh 2 N-1 N/3 1

2D mesh 4 2(N1/2 - 1) 2N1/2 / 3 N1/2

3D mesh 6 3(N1/3 - 1) 3N1/3 / 3 N2/3

nD mesh 2n n(N1/n - 1) nN1/n / 3 N(n-1) / n

Ring 2 N/2 N/4 2

2D torus 4 N1/2 N1/2 / 2 2N1/2

Hypercube Log2N n=Log2N n/2 N/2

2D Tree 3 2Log2N ~2Log2 N 1

Crossbar N-1 1 1 N2/2

N = number of nodes, n = dimension

Butterfly or Omega Network

N/2 Butterfly

°°°

N/2 Butterfly

°°°

• All paths equal length

• Unique path from any input to any output

• Try to avoid conflicts

Multistage Fat Tree

• A multistage fat tree (CM-5) avoids congestion at the root node

• Randomly assign packets to different paths on way up to spread the load

• Increase degree, decrease congestion

Example Networks(Figure 7.19, pg 591)

Name Number Topology Bits Clock Link Bis. BW Year

nCube/ten 1-1024 10-cube 1 10 MHz 1.2 640 1987

iPSC/2 16-128 7-cube 1 16 MHz 2 345 1988

MP-1216 32-512 2D grid 1 25 MHz 3 1,300 1989

Delta 540 2D grid 16 40 MHz 40 640 1991

CM-5 32-2048 fat tree 4 40 MHz 20 10,240 1991

CS-2 32-1024 fat tree 8 70 MHz 50 50,000 1992

Paragon 4-1024 2D grid 16 100 MHz 200 6,400 1992

T3D 16-1024 3D Torus 16 150 MHz 300 19,200 1993

No standard topology!

MBytes/second

Connection-Based vs. Connectionless

• Connection-based system:– Telephone: operator sets up connection between the caller and the

receiver

– Once the connection is established, the line is held until it completes

– Share transmission lines over long distances by using switches to multiplex several conversations on the same lines

» “Time division multiplexing” divide B/W transmission line into a fixed number of slots, with each slot assigned to a conversation

– Often referred to as circuit-swtiching

– Problem: lines busy based on number of conversations, not amount of information sent

– Advantage: reserved bandwidth - call not interrupted once started

Connection-Based vs. Connectionless

• Connectionless system– Postal service : package routed to destination based on

address

– Each message is divided into packets, with an address per packet

– Packets can each take different routes through the network

– Often referred to as packet-switching

– Disadvantage : additional information sent with each message

Store and Forward vs. Cut-Through

• Store-and-forward policy: each switch waits for the full packet to arrive in the switch before it is sent on to the next switch

• Cut-through routing or worm hole routing: switch examines the header, decides where to send the message, and then starts forwarding it immediately – In worm hole routing, when the head of the message is blocked the

message stays strung out over the network, potentially blocking other messages (needs only buffer the size of the packet that is sent between switches).

– Cut through routing lets the tail continue when the head is blocked, storing the whole message into a single switch. (Requires a buffer large enough to hold the largest packet).

Packet Formats

• Fields: Destination, Checksum(C), Length(L), Type(T)

• Data/Header Sizes in bytes: – CM5: (4 to 20)/4, Ethernet: (0 to 1500)/26, ATM: 48/5

Congestion Control• Packet switched networks do not reserve bandwidth;

this leads to contention

• Solution: prevent packets from entering until contention is reduced (e.g., entrance lights on interstate)

• Options:– Packet discarding: If a packet arrives at a switch and there is no room in

the buffer, the packet is discarded, SW detects and resends

– Flow control: between pairs of receivers and senders; use feedback to tell the sender when it is allowed to send the next packet

» Back-pressure: separate wires to tell to stop

» Window: give the original sender the right to send N packets before getting permission to send more (TCP uses)

– Choke packets: Each packet received by busy switch in warning state sent back to the source via choke packet. Source reduces traffic to that destination by a fixed % (ATM Forum)

Example Architectures

Interconnect MPP LAN WANExample CM-5 Ethernet ATM

Topology “Fat” tree Line, Bus Variable, constructed from multistage switches

Connection based? No No Yes

Data Transfer Size Variable: Variable: Fixed: 4 to 20B 0 to 1500B 48B

Store & Forward? No N/A Yes

Congestion control At source: At source: Rate based Flow Listen for via choke control E-net idle packetsvia back pressure

• Deterministic—follows a pre-specified route– mesh: dimension-order routing

» (x1, y1) -> (x2, y2)

» first x = x2 - x1,

» then y = y2 - y1,

– hypercube: edge-cube routing

» X = xox1x2 . . .xn -> Y = yoy1y2 . . .yn

» R = X xor Y

» Traverse dimensions of differing address in order

• Adaptive—route based on network state (e.g., contention)

Routing

001

000

101

100

010 110

111011

Practical IssuesInterconnection MPP LAN WAN

Example CM-5 Ethernet ATM

Standard No Yes Yes

Fault Tolerance? No Yes Yes

Hot Insert? No Yes Yes

• Standards: required for WAN, LAN!

• Fault Tolerance: Can nodes fail and still deliver messages to other nodes? required for WAN, LAN!

• Hot Insert: If the interconnection can survive a failure, can it also continue operation while a new node is added to the interconnection? required for WAN, LAN!

Protocols

• Internetworking: allows computers on independent and incompatible networks to communicate reliably and efficiently;– Enabling technologies: SW standards that allow reliable

communications without reliable networks

– Hierarchy of layers, giving each layer responsibility for portion of overall communications task, called protocol families or protocol suites

• Transmission Control Protocol/Internet Protocol (TCP/IP)– This protocol family is the basis of the Internet

– IP : layer above interconnection network - routes datagram to destination machine

– TCP : layer above IP - guaranees reliable, in-order delivery and prevents corruption of data

TCP/IP packet

• Application sends message

• TCP breaks into 64KB segements, adds 20B header

• IP adds 20B header, sends to network

• If Ethernet, broken into 1500B packets with headers, trailers

• Header, trailers have length field, destination, window number, version, ...

TCP data(< 64KB)

TCP Header

IP Header

IP Data

Ethernet

Protocol

• Key to protocol families is that communication occurs logically at the same level of the protocol, called peer-to-peer, but is implemented via services at the lower level

• Danger is each level increases latency

Connecting Networks

• Bridges: connect LANs together, passing traffic from one side to another depending on the addresses in the packet. – operate at the Ethernet protocol level

– usually simpler and cheaper than routers

• Routers or Gateways: these devices connect LANs to WANs or WANs to WANs and resolve incompatible addressing. – Generally slower than bridges, they operate at the

internetworking protocol (IP) level

– Routers divide the interconnect into separate smaller subnets, which simplifies manageability and improves security

SummaryInterconnection Networks

• Shared vs. Switched media

• Switch topology indicates how nodes are connected

• Switching network can be either connection-based (circuit switched) or connectionless (packet switched)

• In packet-switched networks congestion control is required

• Protocols allow different computers on different networks to communicate reliably