Roger Jones, Lancaster University

1

Upgrade Computing & the LoIUpgrade Computing & the LoI

RWL JonesRWL Jones

LAPP AnnecyLAPP Annecy

20 April 201220 April 2012

Thanks to Hans Von der Schmitt, Markus Elsing, Rolf Seuster,Simone Campana, Dan Van der Steer, Jeff Tseng and many others

2

• Two related aspects

• S&C for the Upgrade Upgrade of S&C– S&C provides support: simulation, reconstruction and analysis are

adapted to changed detector and running conditions

– S&C is integral part of the overall system: requires evolution itself and can even drive it to some extent

• Physics performance is determined by offline software as well as by the detector

• The trigger is realized in software as well as in hardware

• Distributed Computing on large and growing scale enables physics output

• And it is long-term– Even after the last beams are dumped, S&C must live on

• Data Preservation

Computing & Upgrades

The Drivers

• We can identify at least 6 drivers for upgrade computing 1. consequences of increasing pile-up, event complexity and size

2. consequences of new detectors and triggers

3. consequences of increasing sample size (growing much faster than the livetime)

4. consequences of new architectures

5. consequences of new software technologies

6. requirement for data curation and open data access

Roger Jones, Lancaster University

3

4

• Need to change while the system is operational– Yes - can do some bigger steps during the long shutdowns, starting with

Phase 0 – a big chance!

– But life goes on during the shutdowns, especially true in distributed computing (analyses, simulation, reprocessings, …)

– Part of the nature of S&C is its interface to us physicists – more directly exposed than parts of the detector – which constrains evolution paths

– Software parts are interwoven at least as strongly as the detector parts• From the point of view of data, that’s largely the purpose – starting with DAQ,

detector data coming from individual cables are put together – offline physics objects span all detectors

• Hard to keep the software flexible when things are so interwoven. Danger of polymerization

• Installation of SW is infinitely easier than of detectors…– But our C++ objects much less tangible than detector parts

Characteristic challenges in S&C upgrades

5

The Architecture ChallengeR.Seuster

6

Towards Many Core Architectures

• ... and technology may develop faster than we expect– industry may require us for best performance to go to many/multi core

already before Phase-1 (2014...)

Emerging consensus that to profit in future from performance increase - “free lunch” - code must utilize many core platforms

ATLAS software will have to be modified to fit new CPU platforms

Implications of Architecture Changes

• Architectural changes are already affecting us!– The framework is already having to adapt

• AthenaMP being rolled-out, helps control the memory issues, but has limitations by ~32 cores

• Synergies: AthenaMP being investigated for HLT• Extensive work on IO – new framework planned, see D Malon’s talk in the

AUW Computing session https://indico.cern.ch/contributionDisplay.py?sessionId=70&contribId=95&confId=158038

• A re-write of the Gaudi core is also likely

• Beyond this, the software algorithms must also adapt (reconstruction, simulation) – Concurrency, parallel programming

• The distributed computing must also adapt to make efficient use (e.g. whole-node scheduling)

Roger Jones, Lancaster University

7

Software

• All drivers call for changes in the our algorithms– The simulation is obviously most advanced at adapting to the new

detectors

– Some work has started on adapting reconstruction for upgrade conditions, particularly for the IBL

– Efficiency and architectural work is starting• Optimization of existing code• Exploring GPGPUs, especially for tracking

– It is no benefit to just tweak the existing algorithms to cope with pile-up and complexity if we require new programming methods (e.g. parallelism & concurrency) to cope with efficiency, available architectures and software technologies

– See Markus Elsing’s talk in AUW computing session https://indico.cern.ch/getFile.py/access?contribId=72&sessionId=39&resId=0&materialId=slides&confId=158038

Roger Jones, Lancaster University

8

RecExCommon

9

Pileup and Technical Performance

scaling up toPhase-2 pileupclearly problematic

10

• Software: short, medium and long-term work necessary– memory saving with athenaMP to be able to keep all cores in a

machine busy (the usual, in principle trivial parallelism on event level)

• related: I/O framework, getting rid of POOL, etc - advancing well

– full usage of each core: vectorizing a few algorithms - tracking will be explored first. Likely influence on data model

– parallelism on intermediate levels: on algorithm level, on sub-event data level (RoI-like seeding by FTK?). Both require refined communication mechanisms

– …

– will work together with TDAQ. Common work with PH-SFT strongly considered. Common tracking effort with CMS desirable

Some Software Ideas

11

Example: Tracking

• resource needs scale fast– tracking will be (the) main resource driver

• global optimization– requirements on tracking already evolved with accelerator/physics program

– different luminosity regimes lead to different working points

2009 / early 2010commissioning

Min.Bias

pt > 50 MeVopen cuts, robust

settingsmin. 5 clusters

2010 stable running< ~4 events pileup

low lumi physics program (soft QCD, b-

physics, ...),b-tagging...

pt > 100 MeVmin. 7 clusters

2011 pp running~11 events pileup

focus more on high-pt physics (top, W/Z, Higgs), b-tagging...

pt > 400 MeV, harder cuts in seeding

min. 7 clusters

Phase 1 upgradeincluding IBL

24-50 events pileup

high-pt physics, study new physics (I hope),

b-tagging....

pt > 900 MeV,harder tracking cuts,

min. 9 clusters

SLHCup to 150-200 events

pileup

replace Inner Detector to cover very high luminosity physics

program

further evolve strategy...

R-o-I or z-vertex seeding,

reco. per trigger type, GPUs

12

GPU based Tracking• E.g. first tracking prototypes for

Level-2 track trigger and offline tracking– concentrate on aspects of track

reconstruction chain• z-vertex finder

• track seed finder

• Kalman filter

– early phase, still significant approximations

• very significant timing gains– but: lots of software development

needed to obtain precise tracking– investigate mixed scenario ?

• e.g. combinatorial seed finder on GPUs

• CPUs for serial processing steps to do precision calculations

~150 speed-up seen in other Kalman filter studiesExperience with GPUs can help with many-cores

Event Data Model

• Event sizes and event content will continue to be a delicate trade-off, depending luminosity

• Looking at the 2011 experience several improvements went in: – fewer MET containers, tighter cuts in ID, higher pileup noise for calo, trigger – some things in the pipeline but no silver bullet:

• track covarience compression, drop some JetCollections incl. b-tagging from AOD (can be re-done now)

• We also need to plan for the long-term curation of the data (after data-taking)

• This effort will surely be required to continue well into the upgrade era!

14

• LHC produces precious (and expensive) and probably unique data

• Will want to make long-term use of them

• Preserve the data– Avoid disk ➞ γγ

– Organization needed for repeated copying to new media

• Preserve the software– (parts) must live on

– Solid enough not to decay, yet flexible for new environments

– Danger of sclerosis, work towards continuous rejuvenation: act today!

• Preserve the capability to do analyses– Document the analyses!

– Recast is an important development

Data Preservation

Distributed Computing Evolution

• The ADC & Grid is doing very very well. – 1000’s of users can process petabytes of data with millions (billions?) of jobs

• But at the same time, we are starting to hit some limits:– Scaling up, elastic resource usage, global access to data

• What can we learn from these external innovations? (without disrupting operations!)

• Various R&D Projects and Task Forces were formed one year ago– NoSQL databases R&D– Cloud Computing R&D– XROOTD Federation and File level Caching Task force– Event Level Caching R&D– Tier3 Monitoring Task Force– CVMFS Task Force– Multicores Task Force– Also Network Monitoring…

• https://twiki.cern.ch/twiki/bin/viewauth/Atlas/TaskForcesAndRnD

16.3.2012 R&D for DC -- D. van der Ster 17

We must have periodic check-ups

Generally, ATLAS planning here is well developed

18

• Computing: short and medium-term work well under way– …

– important technologies: Cloud computing, Hadoop basket (my favourites...)

– common work ATLAS/CMS/IT-ES on job management with Panda+Condor

• Long term– future role of middleware? Try consolidate middleware flavours,

possible consequences for systems we have on-top – or rather try to be independent of middleware

– …

Some ADC Observations

19

• With other experiments, with CERN (IT, PH), with other labs

• Many of the TEG areas are chances for commonality

• Concrete progress in a few areas so far

– ATLAS, CMS, IT-ES: common analysis framework based on PanDA+Condor/Glidein

– Helix/Nebula: Cloud computing project involving CERN/ATLAS, EMBL, ESA, and 13 industrial partners

• Obvious additional opportunities: storage federations, network monitoring, data preservation

• Common solutions are necessary when manpower/funding shrink (EGI, EMI deadlines, OSG cuts)

– Some solutions should be in common since long but aren’t - too complex for LGC? Experiments have own solutions - revert...?

Common Solutions?

Virtualization and Cloud R&D

• Active participation, almost 10 persons working part time on various topics– https://twiki.cern.ch/twiki/bin/view/Atlas/CloudcomputingRnD

• Data Processing– Panda Queues in the Cloud– Tier3 Analysis Clusters (Instant cloud site)

• User/Institute Managed, Low/Medium Complexity– Personal Analysis Queue (~One click, run my jobs)

• User Managed, Low Complexity (almost transparent)

• Data Storage (tough!)– Short term data caching to accelerate above data processing

use cases• Transient data

– Long term data storage in the cloud• Integrate with DDM

04/19/23 R&D for DC -- D. van der Ster 20

EF

FIC

IEN

CY

, E

LA

ST

ICIT

Y

Already some successful ATLAS projectsATLAS provides the HEP examples in the Helix/Nebula projects

The ATLAS Computing Model changes

• More Bandwidth, Less Hierarchy!• 4 recurring themes:

– Flat(ter) hierarchy: Any site can replicate data from any other site

– Multi Domain Production• Need to replicate output files to remote Tier-1

– Dynamic data caching: Analysis sites receive datasets from any other site “on demand” based on usage pattern

– Remote data access: local jobs accessing data stored at remote sites

• ATLAS is now heavily relying on multi-domain networks and needs decent e2e network monitoring

• Work Ongoing on global access/Data Federation

04/19/23 R&D for DC -- D. van der Ster 21

US

E T

HE

NE

TW

OR

KG

LO

BA

L D

AT

A

AC

CE

SS

Other Distributed Computing

• Database Scaling– Hadoop environment looks best

• Storage and data management– Maintain stable storage for placed data– Support access from experiment jobs

• Workload management– Pilots and frameworks

• GlideinWMS– Whole node scheduling

• CPUs and IO– Use of CPU affinity and pinning– Handling of CPU-bound and IO-bound jobs

22Exploring Common Solutions with other experiments/WLCG!

From Adhocracy to Project!

• There is already activity, as evident from the session yesterday and in previous Upgrade Weeks– Simulation and Grid Computing are in good shape– Reconstruction is less developed, but must grow– R&D Work on new techniques and architectures is underway

More cohesion will help More effort will help

Roger Jones, Lancaster University

23

24

• Software & Computing is part of the ATLAS Upgrade organisation since the recent Upgrade Week at Stanford– we will have a chapter in the LoI for Phase2 Upgrade, to be ready

by fall 2012

– Internal S&C document before the summer break (Roger Jones is our contact)

– we are part of the Upgrade Steering Committee (Computing Coordinator is in the USC)

– both S&C depend on technology development (hardware, networking, software technology)

– need to organise the detailed planning

– looking for main responsible persons for areas such as software upgrade, computing upgrade

Phase 2 LoI

Planning Document Outline

• Introduction• Frameworks

– IO frameworks– Event and memory management

• Offline• Trigger

• Note: we can benefit throughout by having as much development in common with the trigger as possible through all items

Roger Jones, Lancaster University

25

Outline ii

• Simulation– Simulation software for new detectors and detector

R&D– Simulation algorithm efficiency

• R&D for new architectures

• Reconstruction software – basic object reconstruction

• ID, calorimeters, muons

– Reconstruction algorithm strategy and efficiency• R&D for new architectures

Roger Jones, Lancaster University

26

Outline iii

• Event data model– Data complexity– Data curation– Data openness

• Visualization for the upgrade detectors & analysis• Analysis?

Roger Jones, Lancaster University

27

Outline iv

• Distributed computing evolution– Data management– Event flow– Cloud paradigms and virtualisation– Database scaling

• The evolution of the Computing Model– E.g. Tier 0 usage/event filtering and Tier 0/1/2/3 roles– Resource requirements, sustainable rates

• To the Tier 0

• To the Tier 1s

• To the Tier 2s

Roger Jones, Lancaster University

28

Conclusion

• There is good work going on across ATLAS, but will benefit from more cohesion and explicit inclusion in the Upgrade plans

• Several existing synergies may be typical; the offline and HLT can learn a lot from joint efforts

• As with all the upgrade tasks, a documented plan will help with funding opportunities

Roger Jones, Lancaster University

29