Parallel Computing Techniques. 1. Introduction 2. Parallel Machines 3. Clusters 4. Computational Grids 5. unGrid 6. Questions & Answers

ParallelParallel ComputiComputing ng TechniquesTechniques

ParallelParallel ComputiComputing ng TechniquesTechniques

1. Introduction1. Introduction2. 2. Parallel Parallel MachinesMachines

3. 3. ClustersClusters4. 4. Computational Computational

GridsGrids5. 5. unGridunGrid

6. 6. Questions & Questions & AnswersAnswers





11. . IntroductIntroduct

ionion


ionion

SuperComputersSuperComputers

http://www.research.ibm.com/bluehttp://www.research.ibm.com/bluegene/gene/

SuperComputersSuperComputers

http://www.research.ibm.com/bluehttp://www.research.ibm.com/bluegene/gene/ http://ungrid.unal.edu.co/ScDeadProjects.html


ionion


ionion

How to reduce processing How to reduce processing time?time?

1. SuperComputers1. SuperComputers (10(101 1 - 10- 1044 µP, µP, 10101 1 - 10- 102 2

m$)m$)2. Clusters2. Clusters

(10(101 1 - 10- 1033 µP, µP, 50.000 - 1050.000 - 101 1

m$)m$)3. Grids3. Grids

(10(101 1 – 10– 106..n6..n µP, µP, ~0$)~0$)

How to reduce processing How to reduce processing time?time?

1. SuperComputers1. SuperComputers (10(101 1 - 10- 1044 µP, µP, 10101 1 - 10- 102 2

m$)m$)2. Clusters2. Clusters

(10(101 1 - 10- 1033 µP, µP, 50.000 - 1050.000 - 101 1

m$)m$)3. Grids3. Grids

(10(101 1 – 10– 106..n6..n µP, µP, ~0$)~0$)


ionion


ionion

Multitasking Multitasking !=!= MultiprocessingMultiprocessing

MultitaskingMultitasking: 1 CPU: 1 CPU


MultitaskingMultitasking: 1 CPU: 1 CPU

t

CPU

~10ms

T1 T2 T3 T1 T2 T3 T1 T2 T3 T1 T2 T3 ...


ionion


ionion


Multiprocessing Multiprocessing : 2+ CPUs: 2+ CPUs


Multiprocessing Multiprocessing : 2+ CPUs: 2+ CPUs

t

CPU1T1 T3 T1 T3 T1 T3 T1 T3 T1 T3 T1 T3

T2 T4 T2 T4 T2 T4 T2 T4 T2 T4 T2 T4CPU2

...

...


ionion


ionion

Parallel ComputParallel Computinging: 2+ : 2+ CPUsCPUs

Parallel ComputParallel Computinging: 2+ : 2+ CPUsCPUs

t

CPU1T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1

T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1CPU2

...

...

.

.

.

T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1CPU3...


ionion


ionion

Distributed ComputDistributed Computinging: : 2+ Machines2+ Machines

Distributed ComputDistributed Computinging: : 2+ Machines2+ Machines

NET(LAN, MAN, WAN)

T1

T1

T2

T1

T2









22. . Parallel Parallel MachinesMachines


Flynn's Taxonomy Flynn's Taxonomy A classification of A classification of computer architectures computer architectures based on the number of based on the number of streams of instructions streams of instructions and data:and data:

SISD / MISD / SIMD / MIMDSISD / MISD / SIMD / MIMD

Flynn's Taxonomy Flynn's Taxonomy A classification of A classification of computer architectures computer architectures based on the number of based on the number of streams of instructions streams of instructions and data:and data:

SISD / MISD / SIMD / MIMDSISD / MISD / SIMD / MIMD



SISDSISDSingle Instruction Stream/Single Data Single Instruction Stream/Single Data StreamStream. sequential computer: PC (CPU+RAM). sequential computer: PC (CPU+RAM)

MISDMISDMultiple Instruction Stream/Single Multiple Instruction Stream/Single Data StreamData Stream. unusual. unusual

SISDSISDSingle Instruction Stream/Single Data Single Instruction Stream/Single Data StreamStream. sequential computer: PC (CPU+RAM). sequential computer: PC (CPU+RAM)

MISDMISDMultiple Instruction Stream/Single Multiple Instruction Stream/Single Data StreamData Stream. unusual. unusual



SIMDSIMDSingle Instruction Stream/Multiple Single Instruction Stream/Multiple Data StreamData Stream. array processor or “vector . array processor or “vector processor” processor” (Westinghouse, (Westinghouse, Fujitsy, NEC, Cray)Fujitsy, NEC, Cray). all CPUs execute same instruction at . all CPUs execute same instruction at same timesame time. parallel done by different data. parallel done by different data

MIMDMIMDMultiple Instruction Stream/Multiple Multiple Instruction Stream/Multiple Data StreamData Stream. . each cpu has own program and dataeach cpu has own program and data. . syncronization by communicationsyncronization by communication

SIMDSIMDSingle Instruction Stream/Multiple Single Instruction Stream/Multiple Data StreamData Stream. array processor or “vector . array processor or “vector processor” processor” (Westinghouse, (Westinghouse, Fujitsy, NEC, Cray)Fujitsy, NEC, Cray). all CPUs execute same instruction at . all CPUs execute same instruction at same timesame time. parallel done by different data. parallel done by different data

MIMDMIMDMultiple Instruction Stream/Multiple Multiple Instruction Stream/Multiple Data StreamData Stream. . each cpu has own program and dataeach cpu has own program and data. . syncronization by communicationsyncronization by communication



Amdahl's LawAmdahl's Law ( (Gene Gene AmdahlAmdahl/IBM)/IBM)

““If it takes one man one If it takes one man one minute to dig a post-hole minute to dig a post-hole then sixty men can dig it then sixty men can dig it in one secondin one second””

Amdahl's LawAmdahl's Law ( (Gene Gene AmdahlAmdahl/IBM)/IBM)

““If it takes one man one If it takes one man one minute to dig a post-hole minute to dig a post-hole then sixty men can dig it then sixty men can dig it in one secondin one second””



S S = =

S S = =

1111

F + (1-F + (1-F)/NF)/N

F + (1-F + (1-F)/NF)/N

S : speedupS : speedupF : sequential factionF : sequential factionN : number of N : number of processorsprocessors

S : speedupS : speedupF : sequential factionF : sequential factionN : number of N : number of processorsprocessors



if N if N infinite infiniteS = 1 / FS = 1 / F

e.g. e.g. if F = 20% if F = 20%

SSmaxmax = 5 = 5

if N if N infinite infiniteS = 1 / FS = 1 / F

e.g. e.g. if F = 20% if F = 20%

SSmaxmax = 5 = 5









33. . ClustersClusters33. . ClustersClusters

adm

LANLAN


Distributed System:Distributed System:

Homogeneus Homogeneus platform (HW/SW)platform (HW/SW)

Dedicated machinesDedicated machines Local area networksLocal area networks Centralized Centralized

administrationadministration


Homogeneus Homogeneus platform (HW/SW)platform (HW/SW)

Dedicated machinesDedicated machines Local area networksLocal area networks Centralized Centralized

administrationadministration


Gigabit ethernet Gigabit ethernet cards/switches cards/switches (60-90 MB/s)(60-90 MB/s)

RAM ~10GB/s (~10RAM ~10GB/s (~1022 faster)faster)

Latency (50-100 µs)Latency (50-100 µs)(~5 µs w/special cards)(~5 µs w/special cards)

Gigabit ethernet Gigabit ethernet cards/switches cards/switches (60-90 MB/s)(60-90 MB/s)

RAM ~10GB/s (~10RAM ~10GB/s (~1022 faster)faster)

Latency (50-100 µs)Latency (50-100 µs)(~5 µs w/special cards)(~5 µs w/special cards)









adm

LAN-MAN-WANLAN-MAN-WAN

44. . Computational Computational GridsGrids





Heterogeneus Heterogeneus platform (HW/SW)platform (HW/SW)

General purpose General purpose machinesmachines

LAN/MAN/WANLAN/MAN/WAN Multiple admin Multiple admin

domainsdomains(e.g. UN Bogotá (e.g. UN Bogotá campus/UN)campus/UN)


Heterogeneus Heterogeneus platform (HW/SW)platform (HW/SW)

General purpose General purpose machinesmachines

LAN/MAN/WANLAN/MAN/WAN Multiple admin Multiple admin

domainsdomains(e.g. UN Bogotá (e.g. UN Bogotá campus/UN)campus/UN)



Specific Purpose Specific Purpose GridsGrids(seti@HOME, ...)(seti@HOME, ...)

General Purpose General Purpose GridsGrids(unGrid, ...)(unGrid, ...)

Specific Purpose Specific Purpose GridsGrids(seti@HOME, ...)(seti@HOME, ...)

General Purpose General Purpose GridsGrids(unGrid, ...)(unGrid, ...)

10-100 ethernet 10-100 ethernet cards/switches cards/switches (1-10 MB/s)(1-10 MB/s)

Latency (10-200 ms)Latency (10-200 ms)

10-100 ethernet 10-100 ethernet cards/switches cards/switches (1-10 MB/s)(1-10 MB/s)

Latency (10-200 ms)Latency (10-200 ms)











55. . unGridunGrid55. . unGridunGrid

National University of ColombiaNational University of Colombia:: comprises several campuses comprises several campuses across the countryacross the country

The Bogotá campus is The Bogotá campus is interconnected by a FDDI based interconnected by a FDDI based net, where several LAN and VLAN net, where several LAN and VLAN coexistcoexist

Around Around 55000000 computers computers interconnected by this netinterconnected by this net

Free Free CPU cycles are in the 60%-CPU cycles are in the 60%-90% range90% range

National University of ColombiaNational University of Colombia:: comprises several campuses comprises several campuses across the countryacross the country

The Bogotá campus is The Bogotá campus is interconnected by a FDDI based interconnected by a FDDI based net, where several LAN and VLAN net, where several LAN and VLAN coexistcoexist

Around Around 55000000 computers computers interconnected by this netinterconnected by this net

Free Free CPU cycles are in the 60%-CPU cycles are in the 60%-90% range90% range

The unGrid Project attempts to The unGrid Project attempts to construct a virtualconstruct a virtual supercomputer supercomputer that takes profit from these CPU that takes profit from these CPU idle cyclesidle cycles

The unGrid Project attempts to The unGrid Project attempts to construct a virtualconstruct a virtual supercomputer supercomputer that takes profit from these CPU that takes profit from these CPU idle cyclesidle cycles

knowledge grid

information grid

computation/data grid


Factors to consider when Factors to consider when participating in a grid:participating in a grid:

. Privacy. Privacy

. Integrity. Integrity

. Transparency. Transparency

. Configuration/Uninstallation. Configuration/Uninstallation

. Advantages. Advantages

Factors to consider when Factors to consider when participating in a grid:participating in a grid:

. Privacy. Privacy

. Integrity. Integrity

. Transparency. Transparency

. Configuration/Uninstallation. Configuration/Uninstallation

. Advantages. Advantages



JavaSpaces


Master-Worker Pattern









66. . Questions & Questions & AnswersAnswers

66. . Questions & Questions & AnswersAnswers

?? ??

ungrid.unal.ungrid.unal.edu.coedu.co

ungrid.unal.ungrid.unal.edu.coedu.co

i n g r e s a ri n g r e s a r i n s t a l e u n G r i n s t a l e u n G r

i di d u t i l i c e u n G r u t i l i c e u n G r

i di d

i n g r e s a ri n g r e s a r i n s t a l e u n G r i n s t a l e u n G r

i di d u t i l i c e u n G r u t i l i c e u n G r

i di d

Documents

Parallel Computing Techniques. 1. Introduction 2. Parallel Machines 3. Clusters 4. Computational Grids 5. unGrid 6. Questions & Answers