Embed Size (px)
DESCRIPTION
Background The Concurrency Forum has successfully brought the community together ( Pere) - To focus on problems of more optimally using modern computer architectures - Sharing knowledge across HEP - Forum for dissemination The scope of the problem we need to address for the next 10+ years is significant - Needs to draw in available expertise - Needs to provide training and education for next generation of physicists We should not re-invent the wheel – HEP may have lost some skills in this area - Other sciences have not – there is expertise and experience available We need a more formal framework within which to plan our (HEP) activities and which allows us to construct collaborative activities where necessary July 17, 2013
Citation preview
A Hep Software Collaboration- A proposal
Ian BirdCERN, 17th July 2013
July 17, 2013
Background• The Concurrency Forum has successfully brought the community
together ( Pere)- To focus on problems of more optimally using modern computer architectures- Sharing knowledge across HEP- Forum for dissemination
• The scope of the problem we need to address for the next 10+ years is significant- Needs to draw in available expertise- Needs to provide training and education for next generation of physicists
• We should not re-invent the wheel – HEP may have lost some skills in this area- Other sciences have not – there is expertise and experience available
We need a more formal framework within which to plan our (HEP) activities and which allows us to construct collaborative activities where necessary
July 17, 2013
Some of the risks• There are a lot of activities on HEP-HPC• The danger of
- Diverging- Reinventing the wheel (inside and outside HEP)- Spending time in wrong directions
• ...is very real• Relations with other labs & sciences & industry is essential for
items such as- HPC & concurrency programming- Data preservation- Algorithms- Big Data- Future of Grid / Cloud / …- and so on…
July 17, 2013
Resource constraints• Optimistically(!) funding for computing in the next few
years will be ~flat- Worse – it competes with detector upgrades – there is only 1
pot of money, and this sharing is not being managed by the experiments
- we must optimise our use of [CPU, memory/storage, bandwidth, power] & human resources
• HEP cannot go it alone- External funding for computing people will be limited, - HEP must build synergies with other sciences and
organisations - Must be sure to focus our scarce efforts on the real problems
that we have
July 17, 2013
HEP is not (just) LHC• LHC – long term programme with
significantly increasing data rates • ILC/CLIC• TLEP• FAIR• Etc.• Will all be facing the same problem –
significant need for resources in a time of limited funding
July 17, 2013
A HEP software collaboration• Formalise the Concurrency Forum as a collaboration• Provide a framework to
- Increase level of collaboration and involvement- Coordinate and review existing efforts- Recognise and credit the contributions of FAs, institutes and
individuals (shown to be important in the past)- Roadmaps, priorities for investigation and development- Create new projects, make them visible, and provide review- Launch new projects where R&D need is clear- Clarify that the scope is not limited to LHC – but is HEP-wide- Also a mechanism to collaborate and bring in existing
expertise from other sciences and industry- Would very much help in bidding for funds (especially if this
was joined up EU/US etc)July 17, 2013
A Software Collaboration• We should also recognise that HEP is a very
successful community – we are well integrated compared to almost every other science
• BUT: we are fragmented in computing- Notable exceptions – GEANT4, ROOT- All HEP experiments do essentially the same
workflows, but we always (without fail) start over again for the software for every experiment
• We should not miss the huge opportunity to build commonality
July 17, 2013
More commonality?• Common distributed computing models• Common software frameworks for processing and analysis• Common libraries – now that incorporate all the best
practices of high performance computing (vectorised, multi-threaded, parallel, …)
• This would allow us to have a common optimisation team/expertise/… to help experiments optimise their code- Commonality will help bring the best practice to all of HEP codes
(in the same way GEANT4 incorporates physics understanding)• HEP is also missing a huge opportunity to provide our
expertise to the wider scientific community- ROOT to supplement R, Matlab, ???- etc
July 17, 2013
Commonality – why?• Cannot expect to train the entire community to
become experts in HPC/Parallel software; must encode the developments/optimisations in common frameworks and libraries
• Developing concurrent frameworks and parallel algorithms will be a significant effort and we cannot afford to duplicate it
July 17, 2013
R&D Techlab• PH/SFT and IT are establishing a lab to support the
development and benchmarking of parallel software on a variety of architectures- Would be activity as part of the software collaboration- Cluster of machines of interesting architectures- Environment for benchmarking and comparison of
software developments- Provide access to range of code development and
analysis tools to aid in parallelising and vectorising code- Means to collaborate with interested industrial partners,
eg via openlab
July 17, 2013
But what about broader HEP computing?
• Software optimisation is only 1 aspect• How will HEP manage computing in 10-15
years?- How should the models change, with Terabit
networks, computing advances across the board in science, consumer, social, …
- How can HEP make the best/optimal use of developments from other areas?
- For how much longer does HEP run all/part of its own computing?
July 17, 2013
How does HEP computing adapt?
• HEP-wide project on future computing?- Coordinate all the projects in HEP that address future
evolution of computing and software- Governance by the HEP community itself- Launch new projects in areas where there are identified holes- Act as the focal point for collaboration with other sciences and
industry• We have experience in this level of collaboration• We have a bigger problem now than we have had in the
past• HEP computing needs to take a change of direction now
July 17, 2013
Timelines• The software collaboration should be formed
by the end of this year• The broader scope of a project for HEP
future computing needs some community discussion- Aim for a workshop in Q4 2013?
July 17, 2013
