Pattern Language for Parallel Programming, 2004

  • Published on
    03-Mar-2015

  • View
    377

  • Download
    1

Embed Size (px)

Transcript

<p>"Ifyoubuildit,theywillcome." Andsowebuiltthem.Multiprocessorworkstations,massivelyparallelsupercomputers,aclusterin everydepartment...andtheyhaven'tcome.Programmershaven'tcometoprogramthesewonderful machines.Oh,afewprogrammersinlovewiththechallengehaveshownthatmosttypesofproblems canbeforcefitontoparallelcomputers,butgeneralprogrammers,especiallyprofessional programmerswho"havelives",ignoreparallelcomputers. Andtheydosoattheirownperil.Parallelcomputersaregoingmainstream.Multithreaded microprocessors,multicoreCPUs,multiprocessorPCs,clusters,parallelgameconsoles...parallel computersaretakingovertheworldofcomputing.Thecomputerindustryisreadytofloodthemarket withhardwarethatwillonlyrunatfullspeedwithparallelprograms.Butwhowillwritethese programs? Thisisanoldproblem.Evenintheearly1980s,whenthe"killermicros"startedtheirassaulton traditionalvectorsupercomputers,weworriedendlesslyabouthowtoattractnormalprogrammers. Wetriedeverythingwecouldthinkof:highlevelhardwareabstractions,implicitlyparallel programminglanguages,parallellanguageextensions,andportablemessagepassinglibraries.But aftermanyyearsofhardwork,thefactofthematteristhat"they"didn'tcome.Theoverwhelming majorityofprogrammerswillnotinvesttheefforttowriteparallelsoftware. Acommonviewisthatyoucan'tteacholdprogrammersnewtricks,sotheproblemwillnotbesolved untiltheoldprogrammersfadeawayandanewgenerationtakesover. Butwedon'tbuyintothatdefeatistattitude.Programmershaveshownaremarkableabilitytoadopt newsoftwaretechnologiesovertheyears.LookathowmanyoldFortranprogrammersarenow writingelegantJavaprogramswithsophisticatedobjectorienteddesigns.Theproblemisn'twithold programmers.Theproblemiswitholdparallelcomputingexpertsandthewaythey'vetriedtocreatea poolofcapableparallelprogrammers. Andthat'swherethisbookcomesin.Wewanttocapturetheessenceofhowexpertparallel programmersthinkaboutparallelalgorithmsandcommunicatethatessentialunderstandinginaway professionalprogrammerscanreadilymaster.Thetechnologywe'veadoptedtoaccomplishthistaskis apatternlanguage.Wemadethischoicenotbecausewestartedtheprojectasdevoteesofdesign patternslookingforanewfieldtoconquer,butbecausepatternshavebeenshowntoworkinwaysthat wouldbeapplicableinparallelprogramming.Forexample,patternshavebeenveryeffectiveinthe fieldofobjectorienteddesign.Theyhaveprovidedacommonlanguageexpertscanusetotalkabout theelementsofdesignandhavebeenextremelyeffectiveathelpingprogrammersmasterobject orienteddesign.</p> <p>Thisbookcontainsourpatternlanguageforparallelprogramming.Thebookopenswithacoupleof chapterstointroducethekeyconceptsinparallelcomputing.Thesechaptersfocusontheparallel computingconceptsandjargonusedinthepatternlanguageasopposedtobeinganexhaustive introductiontothefield. Thepatternlanguageitselfispresentedinfourpartscorrespondingtothefourphasesofcreatinga parallelprogram: * FindingConcurrency.Theprogrammerworksintheproblemdomaintoidentifytheavailable concurrencyandexposeitforuseinthealgorithmdesign. * AlgorithmStructure.Theprogrammerworkswithhighlevelstructuresfororganizingaparallel algorithm. * SupportingStructures.Weshiftfromalgorithmstosourcecodeandconsiderhowtheparallel programwillbeorganizedandthetechniquesusedtomanageshareddata. * ImplementationMechanisms.Thefinalstepistolookatspecificsoftwareconstructsfor implementingaparallelprogram. Thepatternsmakingupthesefourdesignspacesaretightlylinked.Youstartatthetop(Finding Concurrency),workthroughthepatterns,andbythetimeyougettothebottom(Implementation Mechanisms),youwillhaveadetaileddesignforyourparallelprogram. Ifthegoalisaparallelprogram,however,youneedmorethanjustaparallelalgorithm.Youalsoneed aprogrammingenvironmentandanotationforexpressingtheconcurrencywithintheprogram's sourcecode.Programmersusedtobeconfrontedbyalargeandconfusingarrayofparallel programmingenvironments.Fortunately,overtheyearstheparallelprogrammingcommunityhas convergedaroundthreeprogrammingenvironments.</p> <p>* OpenMP.AsimplelanguageextensiontoC,C++,orFortrantowriteparallelprogramsfor sharedmemorycomputers. * MPI.Amessagepassinglibraryusedonclustersandotherdistributedmemorycomputers. * Java.Anobjectorientedprogramminglanguagewithlanguagefeaturessupportingparallel programmingonsharedmemorycomputersandstandardclasslibrariessupportingdistributed computing. Manyreaderswillalreadybefamiliarwithoneormoreoftheseprogrammingnotations,butfor readerscompletelynewtoparallelcomputing,we'veincludedadiscussionoftheseprogramming environmentsintheappendixes. Inclosing,wehavebeenworkingformanyyearsonthispatternlanguage.Presentingitasabookso peoplecanstartusingitisanexcitingdevelopmentforus.Butwedon'tseethisastheendofthis effort.Weexpectthatotherswillhavetheirownideasaboutnewandbetterpatternsforparallel programming.We'veassuredlymissedsomeimportantfeaturesthatreallybelonginthispattern language.Weembracechangeandlookforwardtoengagingwiththelargerparallelcomputing communitytoiterateonthislanguage.Overtime,we'llupdateandimprovethepatternlanguageuntil ittrulyrepresentstheconsensusviewoftheparallelprogrammingcommunity.Thenourrealwork willbeginusingthepatternlanguagetoguidethecreationofbetterparallelprogramming environmentsandhelpingpeopletousethesetechnologiestowriteparallelsoftware.Wewon'trest untilthedaysequentialsoftwareisrare. ACKNOWLEDGMENTS Westartedworkingtogetheronthispatternlanguagein1998.It'sbeenalongandtwistedroad, startingwithavagueideaaboutanewwaytothinkaboutparallelalgorithmsandfinishingwiththis book.Wecouldn'thavedonethiswithoutagreatdealofhelp. ManiChandy,whothoughtwewouldmakeagoodteam,introducedTimtoBeverlyandBerna.The NationalScienceFoundation,IntelCorp.,andTrinityUniversityhavesupportedthisresearchat varioustimesovertheyears.HelpwiththepatternsthemselvescamefromthepeopleatthePattern LanguagesofPrograms(PLoP)workshopsheldinIllinoiseachsummer.Theformatofthese</p> <p>workshopsandtheresultingreviewprocesswaschallengingandsometimesdifficult,butwithout themwewouldhaveneverfinishedthispatternlanguage.Wewouldalsoliketothankthereviewers whocarefullyreadearlymanuscriptsandpointedoutcountlesserrorsandwaystoimprovethebook. Finally,wethankourfamilies.Writingabookishardontheauthors,butthatistobeexpected.What wedidn'tfullyappreciatewashowharditwouldbeonourfamilies.WearegratefultoBeverly's family(DanielandSteve),Tim'sfamily(Noah,August,andMartha),andBerna'sfamily(Billie)for thesacrificesthey'vemadetosupportthisproject.</p> <p>TimMattson,Olympia,Washington,April2004 BeverlySanders,Gainesville,Florida,April2004 BernaMassingill,SanAntonio,Texas,April2004</p> <p> Chapter 1.APatternLanguageforParallelProgramming Section1.1. INTRODUCTION Section1.2. PARALLELPROGRAMMING Section1.3. DESIGNPATTERNSANDPATTERNLANGUAGES Section1.4. APATTERNLANGUAGEFORPARALLELPROGRAMMING Chapter 2.BackgroundandJargonofParallelComputing Section2.1. CONCURRENCYINPARALLELPROGRAMSVERSUSOPERATINGSYSTEMS Section2.2. PARALLELARCHITECTURES:ABRIEFINTRODUCTION Section2.3. PARALLELPROGRAMMINGENVIRONMENTS Section2.4. THEJARGONOFPARALLELCOMPUTING Section2.5. AQUANTITATIVELOOKATPARALLELCOMPUTATION Section2.6. COMMUNICATION Section2.7. SUMMARY Chapter 3.TheFindingConcurrencyDesignSpace Section3.1. ABOUTTHEDESIGNSPACE Section3.2. THETASKDECOMPOSITIONPATTERN Section3.3. THEDATADECOMPOSITIONPATTERN Section3.4. THEGROUPTASKSPATTERN Section3.5. THEORDERTASKSPATTERN Section3.6. THEDATASHARINGPATTERN Section3.7. THEDESIGNEVALUATIONPATTERN Section3.8. SUMMARY Chapter 4.TheAlgorithmStructureDesignSpace Section4.1. INTRODUCTION Section4.2. CHOOSINGANALGORITHMSTRUCTUREPATTERN Section4.3. EXAMPLES </p> <p> Section4.4. THETASKPARALLELISMPATTERN Section4.5. THEDIVIDEANDCONQUERPATTERN Section4.6. THEGEOMETRICDECOMPOSITIONPATTERN Section4.7. THERECURSIVEDATAPATTERN Section4.8. THEPIPELINEPATTERN Section4.9. THEEVENTBASEDCOORDINATIONPATTERN Chapter 5.TheSupportingStructuresDesignSpace Section5.1. INTRODUCTION Section5.2. FORCES Section5.3. CHOOSINGTHEPATTERNS Section5.4. THESPMDPATTERN Section5.5. THEMASTER/WORKERPATTERN Section5.6. THELOOPPARALLELISMPATTERN Section5.7. THEFORK/JOINPATTERN Section5.8. THESHAREDDATAPATTERN Section5.9. THESHAREDQUEUEPATTERN Section5.10. THEDISTRIBUTEDARRAYPATTERN Section5.11. OTHERSUPPORTINGSTRUCTURES Chapter 6.TheImplementationMechanismsDesignSpace Section6.1. OVERVIEW Section6.2. UEMANAGEMENT Section6.3. SYNCHRONIZATION Section6.4. COMMUNICATION Endnotes Appendix A:ABriefIntroductiontoOpenMP SectionA.1. CORECONCEPTS SectionA.2. STRUCTUREDBLOCKSAND IRECTIVEFORMATS D SectionA.3. WORKSHARING SectionA.4. DATAENVIRONMENTCLAUSES SectionA.5. THEOpenMPRUNTIMELIBRARY SectionA.6. SYNCHRONIZATION SectionA.7. THESCHEDULECLAUSE SectionA.8. THERESTOFTHELANGUAGE Appendix B:ABriefIntroductiontoMPI SectionB.1. CONCEPTS SectionB.2. GETTINGSTARTED SectionB.3. BASICPOINTTOPOINTMESSAGEPASSING SectionB.4. COLLECTIVEOPERA TIONS SectionB.5. ADVANCEDPOINTTOPOINTMESSAGEPASSING SectionB.6. MPIANDFORTRAN SectionB.7. CONCLUSION Appendix C:ABriefIntroductiontoConcurrentProgramminginJava SectionC.1. CREATINGTHREADS SectionC.2. ATOMICITY,MEMORYSYNCHRONIZATION,ANDTHEvolatileKEYWORD </p> <p> SectionC.3. SYNCHRONIZEDBLOCKS SectionC.4. WAITANDNOTIFY SectionC.5. LOCKS SectionC.6. OTHERSYNCHRONIZATIONMECHANISMSANDSHAREDDATA STRUCTURES SectionC.7. INTERRUPTS Glossary Bibliography AbouttheAuthors Index</p> <p>APatternLanguageforParallelProgramming&gt;INTRODUCTION</p> <p>Chapter 1. A Pattern Language for Parallel Programming1.1INTRODUCTION 1.2PARALLELPROGRAMMING 1.3DESIGNPATTERNSANDPATTERNLANGUAGES 1.4APATTERNLANGUAGEFORPARALLELPROGRAMMING</p> <p>1.1. INTRODUCTIONComputersareusedtomodelphysicalsystemsinmanyfieldsofscience,medicine,andengineering. Modelers,whethertryingtopredicttheweatherorrenderasceneinthenextblockbustermovie,can usuallyusewhatevercomputingpowerisavailabletomakeevermoredetailedsimulations.Vast amountsofdata,whethercustomershoppingpatterns,telemetrydatafromspace,orDNAsequences, requireanalysis.Todelivertherequiredpower,computerdesignerscombinemultipleprocessing elementsintoasinglelargersystem.Thesesocalledparallelcomputersrunmultipletasks simultaneouslyandsolvebiggerproblemsinlesstime. Traditionally,parallelcomputerswererareandavailableforonlythemostcriticalproblems.Sincethe mid1990s,however,theavailabilityofparallelcomputershaschangeddramatically.With multithreadingsupportbuiltintothelatestmicroprocessorsandtheemergenceofmultipleprocessor coresonasinglesilicondie,parallelcomputersarebecomingubiquitous.Now,almostevery universitycomputersciencedepartmenthasatleastoneparallelcomputer.Virtuallyalloilcompanies, automobilemanufacturers,drugdevelopmentcompanies,andspecialeffectsstudiosuseparallel computing. Forexample,incomputeranimation,renderingisthestepwhereinformationfromtheanimationfiles, suchaslighting,textures,andshading,isappliedto3Dmodelstogeneratethe2Dimagethatmakes upaframeofthefilm.Parallelcomputingisessentialtogeneratetheneedednumberofframes(24 persecond)forafeaturelengthfilm.ToyStory,thefirstcompletelycomputergeneratedfeature lengthfilm,releasedbyPixarin1995,wasprocessedona"renderfarm"consistingof100dual</p> <p>processormachines[PS00].By1999,forToyStory2,Pixarwasusinga1,400processorsystemwith theimprovementinprocessingpowerfullyreflectedintheimproveddetailsintextures,clothing,and atmosphericeffects.Monsters,Inc.(2001)usedasystemof250enterpriseserverseachcontaining14 processorsforatotalof3,500processors.Itisinterestingthattheamountoftimerequiredtogenerate aframehasremainedrelativelyconstantascomputingpower(boththenumberofprocessorsand thespeedofeachprocessor)hasincreased,ithasbeenexploitedtoimprovethequalityofthe animation. ThebiologicalscienceshavetakendramaticleapsforwardwiththeavailabilityofDNAsequence informationfromavarietyoforganisms,includinghumans.Oneapproachtosequencing,championed andusedwithsuccessbyCeleraCorp.,iscalledthewholegenomeshotgunalgorithm.Theideaisto breakthegenomeintosmallsegments,experimentallydeterminetheDNAsequencesofthesegments, andthenuseacomputertoconstructtheentiresequencefromthesegmentsbyfindingoverlapping areas.ThecomputingfacilitiesusedbyCeleratosequencethehumangenomeincluded150fourway serversplusaserverwith16processorsand64GBofmemory.Thecalculationinvolved500million trillionbasetobasecomparisons[Ein00]. TheSETI@homeproject[SET,ACK02 + ]providesafascinatingexampleofthepowerofparallel computing.Theprojectseeksevidenceofextraterrestrialintelligencebyscanningtheskywiththe world'slargestradiotelescope,theAreciboTelescopeinPuertoRico.Thecollecteddataisthen analyzedforcandidatesignalsthatmightindicateanintelligentsource.Thecomputationaltaskis beyondeventhelargestsupercomputer,andcertainlybeyondthecapabilitiesofthefacilitiesavailable totheSETI@homeproject.Theproblemissolvedwithpublicresourcecomputing,whichturnsPCs aroundtheworldintoahugeparallelcomputerconnectedbytheInternet.Dataisbrokenupintowork unitsanddistributedovertheInternettoclientcomputerswhoseownersdonatesparecom putingtime tosupporttheproject.EachclientperiodicallyconnectswiththeSETI@homeserver,downloadsthe datatoanalyze,andthensendstheresultsbacktotheserver.Theclientprogramistypically implementedasascreensaversothatitwilldevoteCPUcyclestotheSETIproblemonlywhenthe computerisotherwiseidle.Aworkunitcurrentlyrequiresanaverageofbetweensevenandeight hoursofCPUtimeonaclient.Morethan205,000,000workunitshavebeenprocessedsincethestart oftheproject.Morerecently,similartechnologytothatdemonstratedbySETI@homehasbeenused foravarietyofpublicresourcecomputingprojectsaswellasinternalprojectswithinlargecompanies utilizingtheiridlePCstosolveproblemsrangingfromdrugscreeningtochipdesignvalidation. Althoughcomputinginlesstimeisbeneficial,andmayenableproblemstobesolvedthatcouldn'tbe otherwise,itcomesatacost.Writingsoftwaretorunonparallelcomputerscanbedifficult.Onlya smallminorityofprogrammershaveexperiencewithparallelprogramming.Ifallthesecomputers designedtoexploitparallelismaregoingtoachievetheirpotential,moreprogrammersneedtolearn howtowriteparallelprograms. Thisbookaddressesthisneedbyshowingcompetentprogrammersofsequentialmachineshowto designprogramsthatcanrunonparallelcomputers.Althoughmanyexcellentbooksshowhowtouse particularparallelprogrammingenvironments,thisbookisuniqueinthatitfocusesonhowtothink aboutanddesignparallelalgor ithms.Toaccomplishthisgoal,wewillbeusingtheconceptofa patternlanguage.Thishighlystructuredrepresentationofexpertdesignexperiencehasbeenheavily usedintheobjectorienteddesigncommunity.</p> <p>Thebookopenswithtwointroductorychapters.Thefirstgivesanoverviewoftheparallelcomputing landscapeandbackgroundneededtounderstandandusethepatternlanguage.Thisisfollowedbya moredetailedchapterinwhichwelayoutthebasicconceptsandjargonusedbyparallel programmers.Thebookthenmovesintothepatternlanguageitself.</p> <p>1.2. PARALLEL PROGRAMMINGThekeytoparallelcomputingisexploitableconcurrency.Concurrencyexistsinacomputational problemwhentheproblemcanbedecomposedintosubproblemsthatcansafelyexecuteatthesame time.Tobeofanyuse,however,itmustbepossibletostructurethecodetoexposeandlaterexploit theconcurrencya...</p>

Recommended

View more >