SIMD Models PDA Sp07

8/8/2019 SIMD Models PDA Sp07

http://slidepdf.com/reader/full/simd-models-pda-sp07 1/41

SIMD and Associative

Computational Models

Parallel & Distributed Algorithms



SIMD and Associative

Computational Models

Part I: SIMD Model



Flynn’s Taxonomy

• The best known classification scheme for parallel computers.

• Depends on parallelism they exhibit with

– Instruction streams

– Data streams

• A sequence of instructions (the instruction

stream) manipulates a sequence of operands

(the data stream)• The instruction stream (I) and the data stream

(D) can be either single (S) or multiple (M)

• Four combinations: SISD, SIMD, MISD, MIMD



Flynn’s Taxonomy (cont.)

• SISD

– Single Instruction Stream, Single Data Stream

– Most important member is a sequential computer

– Some argue other models included as well.

• SIMD

– Single Instruction Stream, Multiple Data Streams – One of the two most important in Flynn’s Taxonomy

• MISD

– Multiple Instruction Streams, Single Data Stream

– Relatively unused terminology. Some argue that this includespipeline computing.

• MIMD

– Multiple Instructions, Multiple Data Streams

– An important classification in Flynn’s Taxonomy



The SIMD Computer & Model

Consists of two types of processors:• A front-end or control unit

– Stores a copy of the program

– Has a program control unit to execute program

– Broadcasts parallel program instructions to the array

of processors.

• Array of processors of simplistic processors that

are functionally more like an ALU. – Does not store a copy of the program nor have a

program control unit.

– Executes the commands in parallel sent by the front

end.



SIMD (cont.)

• On a memory access, all activeprocessors must access the same location

in their local memory.

• All active processor executes the sameinstruction synchronously, but on different

data

• The sequence of different data items isoften referred to as a vector.



Alternate Names for SIMDs

• Recall that all active processors of a SIMD

computer must simultaneously access the same

memory location.

• The value in the i-th processor can be viewed asthe i-th component of a vector.

• SIMD machines are sometimes called vector

computers [Jordan,et.al.] or processor arrays

[Quinn 94,04] based on their ability to executevector and matrix operations efficiently.



Alternate Names (cont.)

• In particular, the Quinn Textbook for this

course, Quinn calls a SIMD a processor

array.

• Quinn and a few others also considers a

pipelined vector processor to be a SIMD

– This is a somewhat non-standard use of the

term. – An example is the Cray-1



How to View a SIMD Machine

• Think of soldiers all in a unit.

• A commander selects certain soldiers as

active – for example, every even

numbered row.

• The commander barks out an order that all

the active soldiers should do and they

execute the order synchronously.



SIMD Execution Style

– Collectively, the individual memories of the processingelements (PEs) store the (vector) data that isprocessed in parallel.

– When the front end encounters an instruction whose

operand is a vector, it issues a command to the PEsto perform the instruction in parallel.

– Although the PEs execute in parallel, some units canbe allowed to skip any particular instruction.



SIMD Computers

• SIMD computers that focus on vector

operations – Support some vector and possibly matrix

operations in hardware

– Usually limit or provide less support for non-vector type operations involving data in the“vector components”.

• General purpose SIMD computers

– Support more traditional type operations (e.g.,other than for vector/matrix data types).

– Usually also provide some vector andpossibly matrix operations in hardware.



Possible Architecture for a

Generic SIMD



Interconnection Networks for

SIMDs

• No specific interconnection network is

specified.

• 2D mesh has been used more more

frequently than others.

• Even hybrid networks (e.g., cube

connected cycles) have been used.



Example of a 2-D Processor

Interconnection Network in a SIMD

Each VLSI chip has 16 processing elements.

Each PE can simultaneously send a value to a specific

neighbor (e.g., their left neighbor).

PE =

processor

element




• The traditional (SIMD, vector, processor array)

execution style ([Quinn 94, pg 62], [Quinn 2004, pgs37-43]:

– The sequential processor that broadcasts thecommands to the rest of the processors is calledthe front end or control unit .

– The front end is a general purpose CPU that storesthe program and the data that is not manipulated inparallel.

– The front end normally executes the sequentialportions of the program.

– Each processing element has a local memory thatcan not be directly accessed by the host or other processing elements.




– Collectively, the individual memories of the processingelements (PEs) store the (vector) data that isprocessed in parallel.

– When the front end encounters an instruction whose

operand is a vector, it issues a command to the PEsto perform the instruction in parallel.

– Although the PEs execute in parallel, some units canbe allowed to skip any particular instruction.



Masking on Processor Arrays

• All the processors work in lockstep except thosethat are masked out (by setting mask register).• The parallel if-then-else is frequently used in

SIMDs to set masks, – Every active processor tests to see if its data meets the

negation of the boolean condition. – If it does, it sets its mask bit so those processors will

not participate in the operation initially. – Next the unmasked processors, execute the THEN

part.

– Afterwards, mask bits (for original set of activeprocessors) are flipped and unmasked processorsperform the the ELSE part.

• Note: differs from the sequential version of “If”



if (COND) then A else B









Data Parallelism

(A strength for SIMDs)

• All tasks (or processors) apply the same set of operations todifferent data.

• Example:

• . Accomplished on SIMDs by having all active processorsexecute the operations synchronously

• MIMDs can also handle data parallel execution, but mustsynchronize more frequently.

for i←0 to 99 do

a[i]←

b[i] + c[i]endfor



Functional/Control/Job Parallelism

(A Strictly-MIMD Paradigm)

• Independent tasks apply different operations to

different data elements

• First and second statements execute concurrently

• Third and fourth statements execute concurrently

a←2 b←3

m←(a + b) / 2

s←(a2 + b2) / 2

v←s - m2



SIMD Machines

• An early SIMD computer designed for vector and matrix processing was the Illiac IV

computer

– built at the University of Illinois

– See Jordan et. al., pg 7

• The MPP, DAP, the Connection Machines

CM-1 and CM-2, MasPar MP-1 and MP-2

are examples of SIMD computers

– See Akl pg 8-12 and [Quinn, 94]



SIMD Machines

• Quinn [1994, pg 63-67] discusses the CM-2Connection Machine and a smaller & updated

CM-200.

• Professor Batcher was the chief architect for the

STARAN and the MPP (Massively ParallelProcessor) and an advisor for the ASPRO

– ASPRO is a small second generation STARAN used

by the Navy in the spy planes.

• Professor Batcher is best known architecturally

for the MPP, which is at the Smithsonian

Institute & currently displayed at a D.C. airport.



Today’s SIMDs

• Many SIMDs are being embedded in SISDmachines.

• Others are being build as part of hybrid

architectures.• Others are being build as special purpose

machines, although some of them could

classify as general purpose.• Much of the recent work with SIMD

architectures is proprietary.

C



A Company Building Inexpensive

SIMD

WorldScape is producing a COTS(commodity off the shelf) SIMD

• Not a traditional SIMD as – The PEs are full-fledged CPU’s

– the hardware doesn’t synchronize every step.• Hardware design supports efficientsynchronization

• Their machine is programmed like a SIMD.

• The U.S. Navy has observed that their machinesprocess radar a magnitude faster than others.• There is quite a bit of information about their

work at http://www.wscape.com

http://www.wscape.com/







Hybrid Architecture

High speed Myrinet switchHigh speed Myrinet switch

Systola1024

Systola1024

Systola1024

Systola1024

Systola1024

Systola1024

Systola1024

Systola1024

Systola

1024

Systola

1024

Systola

1024

Systola

1024

Systola

1024

Systola

1024

Systola

1024

Systola

1024

ombines SIMD and MIMD paradigm within a parallel architecture⇒Hybrid Computer Hybrid Computer



Architecture of Systola

1024

Interface processors

ISA

RAM NORTH

host computer bus

Controller

RAM WEST

program memory

• Instruction Systolic Array:

– 32 × 32 mesh of processingelements

– wavefront instruction

execution



SIMDs Embedded in SISDs• Intel's Pentium 4 includes what they call MMX

technology to gain a significant performanceboost

• IBM and Motorola incorporated the technologyinto their G4 PowerPC chip in what they call

their Velocity Engine.• Both MMX technology and the Velocity Engine

are the chip manufacturer's name for their proprietary SIMD processors and parallel

extensions to their operating code.• This same approach is used by NVidia and

Evans & Sutherland to dramatically accelerategraphics rendering.



Special Purpose SIMDs in the

Bioinformatics Arena• Parcel

– Acquired by Celera Genomics in 2000

– Products include the sequencesupercomputer GeneMatcher, which has ahigh throughput sequence analysis capability

• Supported over a million processors earlier

– GeneMatcher was used by Celera in their

race with U.S. government to complete thedescription of the human genome sequencing

• TimeLogic, Inc – Has DeCypher, a reconfigurable SIMD



Advantages of SIMDs

• Reference: [Roosta, pg 10]

• Less hardware than MIMDs as they have only one

control unit.

– Control units are complex.• Less memory needed than MIMD

– Only one copy of the instructions need to be stored

– Allows more data to be stored in memory.

• Less startup time in communicating between PEs.



Advantages of SIMDs

• Single instruction stream and synchronization of PEs make SIMD applications easier to program,

understand, & debug.

– Similar to sequential programming

• Control flow operations and scalar operations

can be executed on the control unit while PEs

are executing other instructions.

• MIMD architectures require explicitsynchronization primitives, which create a

substantial amount of additional overhead.



Advantages of SIMDs

• During a communication operation between PEs, – PEs send data to a neighboring PE in parallel and in

lock step – No need to create a header with routing information as

“routing” is determined by program steps. – the entire communication operation is executedsynchronously

– A tight (worst case) upper bound for the time for thisoperation can be computed.

• Less complex hardware in SIMD since nomessage decoder is needed in PEs – MIMDs need a message decoder in each PE.

SIMD Shortcomings



SIMD Shortcomings(with some rebuttals)

• Claims are from our textbook by Quinn. – Similar statements are found in one of our

“primary reference book” by Grama, et. al [13].

• Claim 1: Not all problems are data-parallel

– While true, most problems seem to have dataparallel solutions.

– In [Fox, et.al.], the observation was made in

their study of large parallel applications thatmost were data parallel by nature, but oftenhad points where significant branchingoccurred.




• Claim 2: Speed drops for conditionally executedbranches – Processors in both MIMD & SIMD normally have to do a

significant amount of ‘condition’ testing

– MIMDs processors can execute multiple branchesconcurrently.

– For an if-then-else statement with execution times for the“then” and “else” parts being roughly equal, about ½ of theSIMD processors are idle during its execution

• With additional branching, the average number of inactiveprocessors can become even higher.

• With SIMDs, only one of these branches can be executedat a time.

• This reason justifies the study of multiple SIMDs (or MSIMDs).

SIMD Shortcomings



SIMD Shortcomings

(with some rebuttals)

• Claim 2 (cont): Speed drops for conditionallyexecuted code

– In [Fox, et.al.], the observation was made that for thereal applications surveyed, the MAXIMUM number of

active branches at any point in time was about 8. – The cost of the extremely simple processors used in

a SIMD are extremely low

• Programmers used to worry about ‘full utilization of

memory’ but stopped this after memory cost becameinsignificant overall.

SIMD Shortcomings




• Claim 3: Don’t adapt to multiple users well.

– This is true to some degree for all parallel computers.

– If usage of a parallel processor is dedicated to a important

problem, it is probably best not to risk compromising its

performance by ‘sharing’ – This reason also justifies the study of multiple SIMDs (or

MSIMD).

– SIMD architecture has not received the attention that MIMD

has received and can greatly benefit from further research.

SIMD Shortcomings




• Claim 4: Do not scale down well to

“starter” systems that are affordable.

– This point is arguable and its ‘truth’ is likely to

vary rapidly over time

– WorldScape/ClearSpeed currently sells a very

economical SIMD board that plugs into a PC.

SIMD Sh i




Claim 5: Requires customized VLSI for processorsand expense of control units has dropped

• Reliance on COTS (Commodity, off-the-shelf parts)

has dropped the price of MIMDS• Expense of PCs (with control units) has dropped

significantly

• However, reliance on COTS has fueled the success

of ‘low level parallelism’ provided by clusters andrestricted new innovative parallel architecture

research for well over a decade.



Documents

SIMD Models PDA Sp07