View
255
Download
0
Category
Preview:
Citation preview
Duncan Rand, Jisc and Imperial College London
perfSONAR: getting telemetry on your network
WLCG/GridPP as an example community
19 Oct 2016
»The Worldwide Large Hadron Collider Computing Grid (WLCG) is a global collaboration of more than 170 computing centres in 42 countries
» Its mission is to provide global computing resources to store, distribute and analyse the ~30 petabytes of data generated per year by the LHC experiments
»GridPP is a collaboration providing data-intensive distributed computing resources for the UK HEP community and the UK contribution to the WLCG
»Hierarchically arranged with four tiers:› Tier-0 at CERN (and Wigner in Hungary)› 13 Tier-1s (mainly national physics laboratories)› 149 Tier-2s (generally university physics laboratories)› Tier-3s
19 Oct 2016
» Initial modelling of LHC computing requirements suggested a hierarchical tier-based data management and transfer model
»Data exported from Tier-0 at CERN to each Tier-1 and then on to Tier-2s
»However better than expected network bandwidth means that the LHC experiments have been able to relax this hierarchy
»Now data is transferred in an all-to-all mesh configuration»Data often transferred across multiple domains
› e.g. a CMS transfer to Imperial College London might come predominately from Fermilab near Chicago along with other CMS sites
»So good network is crucial to the operation of the WLCG and that means good monitoring
19 Oct 2016
perfSONAR
19 Oct 2016
»Network monitoring tool developed by ESnet, GEANT, Indiana University and Internet2
»'perfSONAR is a widely-deployed test and measurement infrastructure that is used by science networks and facilities around the world to monitor and ensure network performance.’
»'perfSONAR’s purpose is to aid in network diagnosis by allowing users to characterize and isolate problems. It provides measurements of network performance metrics over time as well as “on-demand” tests’
»http://www.perfsonar.net/about/what-is-perfsonar/
Worldwide perfSONAR host locations
19 Oct 2016
19 Oct 2016
19 Oct 2016
19 Oct 2016
19 Oct 2016
Latency
Loss
19 Oct 2016
Reverse throughput
Throughput
19 Oct 2016
Reverse throughput
Throughput
19 Oct 2016
Durham University GridPP site
19 Oct 2016
Replaced perfSONAR host motherboard
Lancaster University GridPP site
19 Oct 2016
“a number of major tweaks to our network configuration”
Oxford University GridPP site
19 Oct 2016
Reconfiguration of site core network
MaDDash visualisation dashboard
19 Oct 2016
»With large meshes it is difficult to check all hosts»Centralised dashboards really help visualise overall
performance»MaDDash (Monitoring and Debugging Dashboard)
displays meshes of perfSONAR hosts»Many examples of MaDDash dashboards, e.g.
ICNRG, WLCG»WLCG dashboard has two aspects
› Open Monitoring Distribution (Nagios monitoring)› MaDDash
»http://psmad.grid.iu.edu/maddash-webui/
perfSONAR configuration interface
19 Oct 2016
»A perfSONAR host can participate in multiple meshes
»Configuration interface and auto-URL enables dynamic configuration of entire network
McKee et al.CHEP2015
19 Oct 2016
»Adding and removing hosts from the mesh configuration is very simple
»Makes use of a WLCG database of hosts»Version of GUI developed by OSG to be
included in perfSONAR toolkit
19 Oct 2016
Initial WLCG meshes based around countries, e.g. UK/GridPP
MaDDash
19 Oct 2016
19 Oct 2016
19 Oct 2016
Dual-stack perfSONAR measurements
19 Oct 2016
»IPv6 rollout is slow but steady»Assumption (hope) that future campus
upgrades will include provision of IPv6»perfSONAR supports IPv4 and IPv6
measurements»Can leave perfSONAR hosts to default to using
IPv6 if it exists but then not always clear which is in use
»Otherwise can force with "ipv6_only": "1” parameter
WLCG/HEPiX IPv6 Working Groups
19 Oct 2016
»The WLCG has an ongoing effort to promote the adoption of IPv6
»Aim to be able to allow sites to offer IPv6-only computing resources to the WLCG by April 2017
»HEPiX/WLCG IPv6 working groups looking into issues
»Developed mesh to track roll-out of IPv6 capable perfSONAR hosts within WLCG
»Currently twenty one WLCG perfSONAR dual-stack nodes are in the mesh
Dual-stack bandwidth measurements
19 Oct 2016
Dual-stack Traceroute
19 Oct 2016
Oxford Oct 2015
19 Oct 2016
IPv4 ~ 5Gbps
IPv6 ~ 0.5Gbps
Oxford Sept 2016
19 Oct 2016
IPv4 ~ 1.3Gbps
IPv6 ~1.3Gbps
Small perfSONAR node projects
19 Oct 2016
»Data Transfer Zones need well-specified, dedicated hardware to run perfSONAR hosts
»Requires some investment of time and money»Would be nice to have an easier way to get
any idea of network performance»GÉANT have developed a small perfSONAR
node using Gigabyte Brix devices costing about £150-200 each
»Using these in a short but time-limited small perfSONAR node project
»IPv6 included from the start
GÉANT
19 Oct 2016
19 Oct 2016
Small perfSONAR node projects
19 Oct 2016
»Jisc would like to take this project forward»Will probably use existing image»Send out small perfSONAR node to users who wish to
get a rapid and easy idea of their network performance
»For example a scientist in a UK institute with slow download of data set from e.g. Diamond or Jasmin
»Also plan to produce a UK mesh into which these small nodes could be added more or less temporarily
»Training course on how to set up such a mesh being run by GEANT in Zurich on 4th November 2016› https://eventr.geant.org/events/2496
Improving diagnostics: Pundit
19 Oct 2016
»A large mesh such as those in use by WLCG contains a lot of useful data
»Should be possible to use network tomography to, for example, identify problematic routers by correlating traceroute and performance data
»PUNDIT project in US aimed at this»Additional executable installed on perfSONAR
host
»More details: http://pundit.gatech.edu and https://indico.cern.ch/event/505613/contributions/2227428/
Summary
19 Oct 2016
»perfSONAR is a valuable resource for characterising and diagnosing network performance
»Bandwidth nodes typically record throughput and traceroute data; latency nodes record latency and loss
»Network administrators should consider installing several at pertinent places, e.g. at the border, next to storage etc
»Meshes together with MadDash dashboards allow relatively easy monitoring of groups of hosts
»Future perfSONAR meshes should include IPv6»Development work is ongoing to improve the automatic
notification and diagnosis of network faults
19 Oct 2016
»Thank you
»Duncan.Rand@jisc.ac.uk
Recommended