Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
HIDDENSLIDE– Summary• Theseslidesaremeanttobeused‘asis’togiveanupperlevelviewofperfSONAR foranaudiencethatisnotfamiliarwiththeconcept.• You*ARE*allowedtodeletethingsyoudon’tunderstand,ormodifythemessagetofitaspecificgroup• You*ARE*allowedtoaddmoreiftheaudienceismoretechnical• Presentationstylesvary– pleasestudyupbeforepresenting
HIDDENSLIDE– About• Title:perfSONAR-9-Hardware-Therory• ExpectedPresentationTime:Theseslidesshouldtake30minutestopresent.• ExecSummary:Thisisaverytheoreticaldiscussiononnetworkperformance– startingwithnetworkdevices,hosts,operatingsystems,andapplications.• ExpectedAudience:Engineers(midtohighlevel),notmanagement.• AudienceSkillLevel:Intermediatetoexpertknowledgeofhownetworkswork.• LastReviewed:SSConJuly13th 2017
HIDDENSLIDE– ContentOutline• Outline:
• Overviewofthelayersofnetworking• Discussionaboutwhatisbeingmeasured
• WhatCanbeCut:• Nothingreallyshouldbecut– ifyoudon’tunderstandthematerialthisisnotaslidedeckyouwanttoaimtopresent.
NetworkPerformance-Theory
ThisdocumentisaresultofworkbytheperfSONAR Project(http://www.perfsonar.net)andislicensedunderCCBY-SA4.0(https://creativecommons.org/licenses/by-sa/4.0/).
Event
Presenter,Organization,EmailDate
HIDDENSLIDE– Slide6• Thissetofmaterialsisverytheoretical– andismeanttocomebeforethehosthardwaresection• Thepurposeistomotivatewhynetworkperformancemeasurementis‘hard’andwhyweneedhardwaretobeperformingatpeakefficiency.• Itconcludeswithatiebacktowhatmeasurementswilltellyou.
Introduction
• perfSONAR isatooltomeasure“end-to-end”networkperformance.Whatdoesthisimply:• End-to-end:theentirenetworkpathbetweenperfSONAR Hosts
• ApplicationsSoftware• OperatingSystem• HostHardware• @EachHop:transitionbetweenOSILayersinrouting/switchingdevices(e.g.TransporttoNetworkLayer,etc.),buffering,processingspeed
• Flowthroughsecuritydevices• Noeasywaytoseparateouttheindividualcomponentsbydefault– thenumberthetoolgiveshasthemallcombined
August28,2017©2016,http://www.perfsonar.net 6
HIDDENSLIDE– Slides8- 12• Thenextcoupleofslidesare‘networking101’.Ifyouaren’tcomfortablewiththematerial,schoolup• BasicallyyouneedtounderstandthelayersofanOS,anetwork,andhowtheyallrelateviatheconceptoftheOSIstack(e.g.transitionsandreasonsthereissuchamodel)• Hostsaretalkedaboutfirst.
InitialEnd-to-EndNetwork
August28,2017©2016,http://www.perfsonar.net 8
InitialEnd-to-EndNetwork
August28,2017©2016,http://www.perfsonar.net 9
Src HostDelay:• ApplicationwritingtoOS
(kernel)• Kernelwritingviamemory
tohardware• NICwritingtonetwork
Dst HostDelay:• NICreceivingdata• Kernelallocatingspace,
sendingtoapplication• Applicationreading/acting
onreceiveddata
Src LAN:• Bufferingoningress
interfacequeues• Processingdatafor
destinationinterface• Egressinterfacequeuing• Transmission/Serialization
towire
WAN:• Propagationdelayforlongspans• Ingressqueuing/processing/egressqueuing/serializationforeachhop
Dst LAN:• Bufferingoningress
interfacequeues• Processingdatafor
destinationinterface• Egressinterfacequeuing• Transmission/Serialization
towire
OSIStackReminder
August28,2017©2016,http://www.perfsonar.net 10
• ThedemarcationbetweeneachlayerhasanAPI(e.g.the‘narrowwaist’ofanhourglass)
• Somelayersaremorewelldefinedthanothers:• Withinanapplicationthejobof
presentationandsessionmaybehandled
• TheoperatingsystemhandlesTCPandIP,althoughtheseareseparatelibraries
• Network/DataLinkoccurwithinnetworkdevices(Routers,Switches)
HostBreakout
August28,2017©2016,http://www.perfsonar.net 11
• Mostapplicationsarewrittenin‘userspace’,e.g.specialsectionoftheOSthatisjailedfrom‘kernelspace’.
• RequeststousefunctionslikethenetworkaredonebyusingsystemcallsthroughanAPI(e.g.openasocket soIcancommunicatewitharemotehost)
• TheTCP/IPlibrariesarewithinthekernel,theyreceivetherequestandtakecareoftheheavyliftingofconvertingthedatafromtheapplication(e.g.alargechunkofmemory)intoindividualpacketsforthenetwork
• TheNICwillthenencapsulateintoLinklayerprotocol(e.g.ethernet frames)andsendontothewireforthenexthoptodealwith
HostBreakout
August28,2017©2016,http://www.perfsonar.net 12
• Thereceivesideworkssimilar– justinreverse
• FramescomeoffofthenetworkandintotheNIC.Theonboardprocessorwillextractpackets,andpassthemtothekernel
• Thekernelwillmapthepacketstotheapplicationthatshouldbedealingwiththem
• TheapplicationwillreceivethedataviatheAPI
• Note– theTCP/IPlibrariesmanagethingslikedatacontrol.Theapplicationonlyseesasocket,andknowsthatitwillsendindata,anditwillmakeittotheotherside.Itisthejobofthelibrarytoensurereliabledelivery
HIDDENSLIDE– Slides14-16• Thenextcoupleofslidestalkabouttheinternalsofanetworkdevice– assumeswitchorrouter•Wethenstarttorollthisbackandexplainhowthisrelatedtonetworkperformanceonanendtoendbasis.
NetworkDeviceBreakout
August28,2017©2016,http://www.perfsonar.net 14
NetworkDeviceBreakout
August28,2017©2016,http://www.perfsonar.net 15
• Dataarrivesfrommultiplesources
• Buffershaveafiniteamountofmemory• Somehavethisperinterface• Othersmayhaveaccesstoashared
memoryregionwithotherinterfaces• Theprocessingenginewill:
• Extracteachpacket/framefromthequeues
• Pulloffheaderinformationtoseewherethedestinationshouldbe
• Movethepacket/frametothecorrectoutputqueue
• Additionaldelayispossibleasthequeuesphysicallywritethepackettothetransportmedium(e.g.opticalinterface,copperinterface)
NetworkDeviceBreakout– Delays
August28,2017©2016,http://www.perfsonar.net 16
HIDDENSLIDE– Slides18-22• Nowweputittogether– traversinganetworkmeanstraversingtheOSUstack(partsofitatleast)ineverysingledeviceonthepath.• Thistakestime,anditcausesproblemsifthereareslownessesinpiecesofthenetwork
NetworkDevices&OSI
August28,2017©2016,http://www.perfsonar.net 18
• Noteverydevicewillcareabouteverylayer
• Hostsunderstandthemallviavariouslibraries
• Networkdevicesonlyknowuptoapoint:• ‘Routers’knowuptothe‘NetworkLayer’.
TheywillmakethechoiceofsendingtothenexthopbasedonNetworkLayerheaders.E.g.TCPinformation– IPaddresses
• ‘Switches’knowuptothe‘LinkLayer’.TheywillmakethechoiceofsendingtothenexthopbasedonLinkLayerheaders.E.g.MACaddressingfromtheIP
• Eachhophasthehardware/softwaretopulloftheencapsulateddatatofindwhatitneeds
“End-to-End”
• Anetworkuserinterfaceswiththenetworkviaatool(datamovementapplication,portal).Theyonlygetasinglepieceoffeedback– howlongtheinteractiontakes• Inreality– it’sacomplexseriesofmovestogetthedataendtoend,withlimitedvisibilitybydefault• Delaysonthesourcehost• DelaysinthesourceLAN• DelaysintheWAN• DelaysinthedestinationLAN• Delaysonthedestinationhost
August28,2017©2016,http://www.perfsonar.net 19
“End-to-End”• Theonlywaytogetvisibilityistorelyoninstrumentationatthevariouslayers:• Hostlevelmonitoringofkeycomponents(CPU,memory,network)• LAN/WANlevelmonitoringofindividualnetworkdevices(utilization,drops/discards,errors)• End-to-endsimulations
• Thelateroneistricky– wewantto‘simulate’whatauserwouldseebyhavingourown(welltuned)applicationtellushowitcandoacrossthecommonnetworksubstrate
August28,2017©2016,http://www.perfsonar.net 20
DereferencingIndividualComponents• HostPerformance• SoftwareTools– Ganglia,HostSNMP+Cacti/MRTG
• NetworkDevicePerformance• SNMP/TL1PassivePolling(e.g.interfacecounters,internalbehavior)
• SoftwarePerformance• ???Thisdependsheavilyonhowwellthesoftware(e.g.operatingsystem,application)isinstrumented.
• End-to-end• perfSONAR activetools(iperf3,owamp,etc.)
August28,2017©2016,http://www.perfsonar.net 21
Takeaways• Sincewewant“network”performance– wewanttoremovethehosthardware/operatingsystem/applicationsfromtheequationasmuchaspossible• Thingsthatwecandoonourown,orthatwegetforfreebyusingperfSONAR:• HostHardware:Choosinghardwarematters.Thereneedstobepredictableinteractionsbetweensystemcomponents(NIC,motherboard,memory,processors)
• OperatingSystem:perfSONAR featuresatunedversionofCentOS.Thisversioneliminatesextrasoftwareandhasbeenmodifiedtoallowforhighperformancenetworking
• Applications:perfSONAR applicationsaredesignedtomakeminimalsystemcalls,anddonotinvolvethedisksubsystem.Theperformancetheyreportisdesignedtobeaslow-impactonthehosttoachieverealisticnetworkperformance
August28,2017©2016,http://www.perfsonar.net 22
HIDDENSLIDE– Slides24-27• Nowwegetdownintowhatthisallmeansfornetworkperformance• Toolslikeiperf (andothers)aredesignedtotellus‘whatisgoingonrightnow’givensomeconfig• perfSONAR istunedtoperformwell,soitshouldgiveyouapurenetworkperformanceindication(aslongasthehostsareadequate).
LetsTalkaboutIPERF• Startwithadefinition:
• networkthroughputistherateofsuccessfulmessagedeliveryoveracommunicationchannel
• Easierterms:howmuchdatacanIshovelintothenetworkforsomegivenamountoftime
• Thingsitincludes:theoperatingsystem,thehosthardware,andtheentirenetoworkpath
• Whatdoesthistellus?• Oppositeofutilization(e.g.itshowmuchwecangetatagivenpointintime,minuswhatisutilized)
• Utilizationandthroughputaddedtogetherare“capacity”• Toolsthatmeasurethroughputareasimulationofarealworkusecase(e.g.howwellcouldbulkdatamovementperform)
August28,2017©2016,http://www.perfsonar.net 24
WhatIPERFTellsUs• Letsstartbydescribingthroughput,whichisvague.
• Capacity:linkspeed• NarrowLink:linkwiththelowestcapacityalongapath• Capacityoftheend-to-endpath=capacityofthenarrowlink
• Utilizedbandwidth:currenttrafficload• Availablebandwidth:capacity– utilizedbandwidth
• TightLink:linkwiththeleastavailablebandwidthinapath• Achievablebandwidth:includesprotocolandhostissues(e.g.BDP!)
• Allofthisis“memorytomemory”,e.g.wearenotinvolvingaspinningdisk(morelater)
August28,2017©2016,http://www.perfsonar.net 25
45 Mbps 10 Mbps 100 Mbps 45 Mbps
Narrow Link Tight Link
source sink
(Shaded portion shows background traffic)
BWCTLExample(iperf3)[zurawski@wash-pt1 ~]$ bwctl -T iperf3 -f m -t 10 -i 2 -c sunn-pt1.es.netbwctl: run_tool: tester: iperf3bwctl: run_tool: receiver: 198.129.254.58bwctl: run_tool: sender: 198.124.238.34bwctl: start_tool: 3598657653.219168Test initializedRunning clientConnecting to host 198.129.254.58, port 5001[ 17] local 198.124.238.34 port 34277 connected to 198.129.254.58 port 5001[ ID] Interval Transfer Bandwidth Retransmits[ 17] 0.00-2.00 sec 430 MBytes 1.80 Gbits/sec 2[ 17] 2.00-4.00 sec 680 MBytes 2.85 Gbits/sec 0[ 17] 4.00-6.00 sec 669 MBytes 2.80 Gbits/sec 0[ 17] 6.00-8.00 sec 670 MBytes 2.81 Gbits/sec 0[ 17] 8.00-10.00 sec 680 MBytes 2.85 Gbits/sec 0[ ID] Interval Transfer Bandwidth Retransmits
Sent[ 17] 0.00-10.00 sec 3.06 GBytes 2.62 Gbits/sec 2
Received[ 17] 0.00-10.00 sec 3.06 GBytes 2.63 Gbits/sec
iperf Done.bwctl: stop_tool: 3598657664.995604
SENDER END
August28,2017©2016,http://www.perfsonar.net 26
N.B.ThisiswhatperfSONARGraphs– theaverageofthecompletetest
Summary
•Wehaveestablishedthatourtoolsaredesignedtomeasurethenetwork• Forbetterorforworsethe‘network’isalsoourhosthardware,operatingsystem,andapplication• Togetthemostaccuratemeasurement,weneed:• Hardwarethatperformswell• Operatingsystemsthatperformwell• Applicationsthatperformwell
August28,2017©2016,http://www.perfsonar.net 27
NetworkPerformance-Theory
ThisdocumentisaresultofworkbytheperfSONAR Project(http://www.perfsonar.net)andislicensedunderCCBY-SA4.0(https://creativecommons.org/licenses/by-sa/4.0/).
Event
Presenter,Organization,EmailDate