Upload
justin-howard
View
220
Download
0
Tags:
Embed Size (px)
Citation preview
ATLAS and the GridATLAS and the Grid
ACAT02 Moscow June 2002ACAT02 Moscow June 2002RWL Jones Lancaster UniversityRWL Jones Lancaster University
RWL Jones, Lancaster University
The ATLAS Computing ChallengeThe ATLAS Computing Challenge
Running conditions at startup:Running conditions at startup:
0.8x100.8x1099 event sample event sample 1.3 PB/year1.3 PB/year, before data processing, before data processing
““Reconstructed” events, Monte Carlo data Reconstructed” events, Monte Carlo data ~10 PB/year (~3 PB on disk) ~10 PB/year (~3 PB on disk)
CPU: ~1.6M SpecInt95 including analysisCPU: ~1.6M SpecInt95 including analysis
CERN alone can handle only a fraction of these resources
2007Average Luminosity (10^33) 1Trigger Rate (Hz) 160Physics Rate (Hz) 140Running (Equiv. Days) 50Physics Events (10^9) 0.8
RWL Jones, Lancaster University
The Solution: The GridThe Solution: The Grid
Note: Truly
HPC, but requires
more
Not designed for tight-coupled
problems, but spin-offs many
RWL Jones, Lancaster University
ATLAS Needs Grid ApplicationsATLAS Needs Grid Applications
The ATLAS OO software framework is Athena, which co-The ATLAS OO software framework is Athena, which co-evolves with the LHCb Gaudi frameworkevolves with the LHCb Gaudi framework
ATLAS is truly intercontinentalATLAS is truly intercontinental In particular, it is present on both sides of the AtlanticIn particular, it is present on both sides of the Atlantic
Opportunity: the practical convergence between US and European Grid projects will come through the transatlantic applications
Threat: There is an inevitable tendency towards fragmentation/divergence of effort to be resisted
Other relevant talks:Other relevant talks: Nick Brook: co-development with LHCb, especially through
UK GridPP collaboration (or rather, I’ll present this later) Alexandre Vaniachine, describing work for the ATLAS Data
Challenges
RWL Jones, Lancaster University
Data ChallengesData ChallengesTest Bench –Data ChallengesTest Bench –Data Challenges
Prototype IPrototype I May 2002May 2002 Performance and scalability testing of components of the
computing fabric (clusters, disk storage, mass storage system, system installation, system monitoring) using straightforward physics applications. Test job scheduling and data replication software (DataGrid release 1.2)
Prototype IIPrototype II Mar 2003Mar 2003 Prototyping of the integrated local computing fabric, with
emphasis on scaling, reliability and resilience to errors. Performance testing of LHC applications. Distributed application models (DataGrid release 2).
Prototype IIIPrototype III Mar 2004Mar 2004 Full scale testing of the LHC computing model with fabric
management and Grid management software for Tier-0 and Tier-1 centres, with some Tier-2 components (DataGrid release 3).
RWL Jones, Lancaster University
The Hierarchical ViewThe Hierarchical View
Tier2 Centre ~1 TIPS
Online System
Offline Farm~20 TIPS
CERN Computer Centre >20 TIPS
UK Regional Centre (RAL)
US Regional Centre
French Regional Centre
Italian Regional Centre
SheffieldManchesterLiverpoolLancaster ~0.25TIPS
Workstations
~100 MBytes/sec
~100 MBytes/sec
100 - 1000 Mbits/sec
•One bunch crossing per 25 ns
•100 triggers per second
•Each event is ~1 Mbyte
Physicists work on analysis “channels”
Each institute has ~10 physicists working on one or more channels
Data for these channels should be cached by the institute server
Physics data cache
~PBytes/sec
~ Gbits/sec or Air Freight
Tier2 Centre ~1 TIPS
Tier2 Centre ~1 TIPS
~Gbits/sec
Tier Tier 00
Tier Tier 11
Tier Tier 33
Tier Tier 44
1 TIPS = 25,000 SpecInt95
PC (1999) = ~15 SpecInt95
Northern Tier ~1 TIPS
Tier Tier 22
RWL Jones, Lancaster University
A More Grid-like ModelA More Grid-like Model
CERNTier2
Lab a
Lancs
Lab c
Uni n
Lab m
Lab b
Uni bUni y
Uni x
PhysicsDepartment
Desktop
Germany
Tier 1
USAFermiLab
UK
France
Italy
NL
USABrookhaven
……….
The LHC Computing
Facility
RWL Jones, Lancaster University
Features of the Cloud ModelFeatures of the Cloud Model
All regional facilities have 1/3 of the full reconstructed dataAll regional facilities have 1/3 of the full reconstructed data Allows more on disk/fast access space, saves tapeAllows more on disk/fast access space, saves tape Multiple copies mean no need for tape backupMultiple copies mean no need for tape backup All regional facilities have all of the analysis data (AOD)All regional facilities have all of the analysis data (AOD) Resource broker can still keep jobs fairly localResource broker can still keep jobs fairly local Centres are Regional and NOT National Centres are Regional and NOT National
Physicists from other Regions should have also Access to the Computing Resources
Cost sharing is an issue Implications for the Grid middleware on accountingImplications for the Grid middleware on accounting
Between experiments Between regions Between analysis groups
Also, different activities will require different prioritiesAlso, different activities will require different priorities
RWL Jones, Lancaster University
Resource EstimatesResource Estimates
2007Average Luminosity (10^33) 1Trigger Rate (Hz) 160Physics Rate (Hz) 140Running (Equiv. Days) 50Physics Events (10^9) 0.8
1.6 0.5 10 0.1 2 0.5
RWL Jones, Lancaster University
Resource EstimatesResource Estimates
Analysis resources?:Analysis resources?: 20 analysis groups 20 jobs/group/day = 400 jobs/day sample size : 108 events 2.5 SI95s/ev => 1011 SI95 (s/day) = 1.2*106 SI95 Additional 20% for activities on smaller samples
CPU (MSI95) Tape (PB) Disk (PB)
CERN (T0+T1) 0.4 6.7 0.5Each RF 0.2 0.2 0.46 External RFs 1.2 1.2 2.4Total 1.6 7.9 2.9
RWL Jones, Lancaster University
Rough ArchitectureRough Architecture
Installation ofSoftware and Env
Compute + Store Sites
User Interface to Grid+ experiment framework
User
MiddlewareRB, GIS
Data Catalogue
Job Configuration/VDC/metadata
RWL Jones, Lancaster University
Test BedsTest Beds
EDG Test Bed 1EDG Test Bed 1 Common to all LHC experiments Using/testing EDG test bed 1 release code Already running boxed fast simulation and installed full simulation
US ATLAS Test BedUS ATLAS Test Bed Demonstrate success of grid computing model for HEP
in data production in data access in data analysis
Develop & deploy grid middleware and applications wrap layers around apps simplify deployment
Evolve into fully functioning scalable distributed tiered grid NorduGridNorduGrid
Developing a regional test bed Light-weight Grid user interface, working prototypes etc see talk by Aleksandr Konstantinov
RWL Jones, Lancaster University
EDG Release 1.2EDG Release 1.2
EDG has strong emphasis on middleware development; EDG has strong emphasis on middleware development; applications come secondapplications come second
ATLAS has been testing the `stable’ releases of the EDG ATLAS has been testing the `stable’ releases of the EDG software as they become available as part of WP8 (ATLAS software as they become available as part of WP8 (ATLAS key contact Silvia Resconi)key contact Silvia Resconi)
EDG Release EDG Release ((1.21.2)) is under test by Integration Team people is under test by Integration Team people plus Loose Cannons (experiment independent people) on plus Loose Cannons (experiment independent people) on the development testbed at CERN.the development testbed at CERN.
Standard rStandard requirementsequirements must be must be metmet before the before the ATLAS ATLAS Applications Applications peoplepeople test test a a releaserelease::
1. The development testbed “must” consist of at least 3 sites in 3 different countries ( e.g. CERN, CNAF, RAL )
2. There “must” be a long ( > 24 hours) unattended period with a low error rate ( < 1% of jobs failed )
http://pcatl0a.mi.infn.it/~resconi/validation/valid.html
RWL Jones, Lancaster University
EDG TestBed 1 StatusEDG TestBed 1 Status28 May 2002 17:0328 May 2002 17:03
Web interface Web interface showing status of showing status of (~400) servers at (~400) servers at testbed 1 sitestestbed 1 sites
5 Main Production Centres5 Main Production Centres
RWL Jones, Lancaster University
GridPP Sites in Testbed(s)GridPP Sites in Testbed(s)
RWL Jones, Lancaster University
NorduGrid OverviewNorduGrid Overview
Launched in spring 2001, with Launched in spring 2001, with the aim of creating a Grid the aim of creating a Grid infrastructure in the Nordic infrastructure in the Nordic countriescountries
Partners from Denmark, Partners from Denmark, Norway, Sweden, and FinlandNorway, Sweden, and Finland
Initially the Nordic branch of Initially the Nordic branch of the the EU DataGridEU DataGrid (EDG) (EDG) project testbedproject testbed
Independent developmentsIndependent developments Relies on funding from Relies on funding from
NorduNet2NorduNet2
http://www.nordugrid.org
RWL Jones, Lancaster University
Lawrence BerkeleyNational Laboratory
BrookhavenNationalLaboratoryIndiana
University
Boston University
ArgonneNationalLaboratory
U Michigan
University ofTexas atArlington
OklahomaUniversity
US -ATLAS testbed launched February 2001
US Grid Test Bed SitesUS Grid Test Bed Sites
RWL Jones, Lancaster University
US Hardware and DeploymentUS Hardware and Deployment
8 gatekeepers - ANL, BNL, LBNL, BU, IU, UM, OU, UTA8 gatekeepers - ANL, BNL, LBNL, BU, IU, UM, OU, UTA Farms - BNL, LBNL, IU, UTA + Multiple R&D gatekeepersFarms - BNL, LBNL, IU, UTA + Multiple R&D gatekeepers Uniform OS through kickstartUniform OS through kickstart
Running RH 7.2 First stage deploymentFirst stage deployment
Pacman, Globus 2.0b, cernlib (installations) Simple application package
Second stage deploymentSecond stage deployment Magda, Chimera, GDMP… (Grid data management)
Third stageThird stage MC production software + VDC
Many US names mentioned later, thanks also to Craig Tull, Dan Engh, Mark Many US names mentioned later, thanks also to Craig Tull, Dan Engh, Mark SosebeeSosebee
RWL Jones, Lancaster University
Important ComponentsImportant Components
GridView - simple script tool to monitor status of test bed GridView - simple script tool to monitor status of test bed (Java version being developed)(Java version being developed)
Gripe - unified user accountsGripe - unified user accounts Magda - MAnager for Grid DataMagda - MAnager for Grid Data Pacman - package management and distribution toolPacman - package management and distribution tool Grappa - web portal based on active notebook technologyGrappa - web portal based on active notebook technology
RWL Jones, Lancaster University
Grid User InterfaceGrid User Interface
Several prototype interfacesSeveral prototype interfaces GRAPPA EDG Nordugrid
Lightweight
Nothing experiment specific Nothing experiment specific GRAT
Line mode (and we will always need to retain line mode!)
Now defining an ATLAS/LHCb joint user interface, GANGANow defining an ATLAS/LHCb joint user interface, GANGA Co-evolution with Grappa Knowledge of experiment OO architecture needed
(Athena/Gaudi)
RWL Jones, Lancaster University
Interfacing Interfacing Athena/Gaudi Athena/Gaudi to the to the GRIDGRID
Athena/GAUDI Application
GANGA/GrappaGU
IjobOptions/Virtual DataAlgorithms
GRIDServices
HistogramsMonitoringResults
?
?
RWL Jones, Lancaster University
GRAPPAGRAPPA
Based on XCAT Science Portal, framework for building personal Based on XCAT Science Portal, framework for building personal science portals science portals
A A science portalscience portal is an application-specific Grid portal is an application-specific Grid portal Active notebookActive notebook
HTML pages to describe the features of the notebook and how to use it HTML forms which can be used to launch parameterizable scripts
(transformation) Parameters stored in a sub-notebook (derivation)
Very flexibleVery flexible Jython - access to Java classesJython - access to Java classes
Globus Java CoG kit XCAT XMESSAGES
Not every user has to write scripts Not every user has to write scripts Notebooks can be shared among usersNotebooks can be shared among users
Import/export capabilityShava Smallen, Rob GardnerShava Smallen, Rob Gardner
RWL Jones, Lancaster University
GRAPPA/XCAT Science Portal ArchitectureGRAPPA/XCAT Science Portal Architecture
Portal Web Server(tomcat server + java servlets)
JythonIntepreter
NotebookDatabase
GSI Authentication
User’s Web Browser
Grid
•Submit Athena jobs to Grid computing elements
•Manage JobOptions, record sessions
•Staging and output collection supported
•Tested on US ATLAS Grid Testbed
The prototype can:
RWL Jones, Lancaster University
GANGA/Grappa Development GANGA/Grappa Development StrategyStrategy
Completed existing technology + requirement surveyCompleted existing technology + requirement survey Must be Grid aware but not Grid-dependentMust be Grid aware but not Grid-dependent
Still want to be able to `pack and go’ to a standalone laptop
Must be component-basedMust be component-based Interface Technologies (Standards needed Interface Technologies (Standards needed GGF) GGF)
Programmatic API (eg. C, C++, etc) Scripting as Glue ala Stallman (eg. Python) Others eg. SOAP, CORBA, RMI, DCOM, .NET, etc. ………
Defining the experiment software services to capture and Defining the experiment software services to capture and present the functionality of the Grid servicepresent the functionality of the Grid service
RWL Jones, Lancaster University
Possible DesignsPossible Designs
Two ways of implementation: Two ways of implementation: Based on one of the general-purpose grid portals (not tied to a
single application/framework): Alice Environment (AliEn) Grid Enabled Web eNvironment for Site-Independent User Job
Submission (GENIUS) Grid access portal for physics applications (Grappa)
Based on the concept of Python bus (P. Mato): use different modules whichever are required to provide full
functionality of the interface use Python to glue this modules, i.e., allow interaction and
communication between them
RWL Jones, Lancaster University
Workspaces
DB
InternetInternet
GAUDI
Python BusPython Bus
Localuser
GaudiPython
Remote userHTML page
GRIDGRID
JobConfiguration
DB
Bookkeeping
DB
ProductionDB
GU
IJava Module OS Module EDG API PythonROOT
PYTHON SW BUS
GAUDI client
RWL Jones, Lancaster University
Installation ToolsInstallation Tools
To use the Grid, deployable software must be deployed on the Grid fabrics, and the To use the Grid, deployable software must be deployed on the Grid fabrics, and the deployable run-time environment established (Unix and Windows)deployable run-time environment established (Unix and Windows)
Installable code and run-time environment/configurationInstallable code and run-time environment/configuration Both ATLAS and LHCb use CMT for the software management and environment Both ATLAS and LHCb use CMT for the software management and environment
configurationconfiguration CMT knows the package interdependencies and external dependencies CMT knows the package interdependencies and external dependencies this is the this is the
obvious tool to prepare the deployable code and to `expose’ the dependencies to the obvious tool to prepare the deployable code and to `expose’ the dependencies to the deployment tool (Christian Arnault, Chas Loomis)deployment tool (Christian Arnault, Chas Loomis)
Grid aware tool to deploy the aboveGrid aware tool to deploy the above
PACMAN (Saul Youssef) is a candidate which seems fairly easy to interface with CMT PACMAN (Saul Youssef) is a candidate which seems fairly easy to interface with CMT
RWL Jones, Lancaster University
Installation IssuesInstallation Issues
Most Grid projects seem to assume either code is pre-installed or else Most Grid projects seem to assume either code is pre-installed or else can be dumped each time into the input sandboxcan be dumped each time into the input sandbox
The only route for installation of software through the Grid seems to be The only route for installation of software through the Grid seems to be as data in Storage Elementsas data in Storage Elements
In general these are non-local Hard to introduce directory trees etc this way (file based)
How do we advertise installed code?How do we advertise installed code? Check it is installed by a preparation task sent to the remote fabric
before/with the job Advertise the software is installed in your information service for use by the
resource broker Probably need both!Probably need both! The local environment and external packages will always be a problem The local environment and external packages will always be a problem
Points to a virtual machine idea eventually; Java? Options?Options?
DAR – mixed reports, but CMS are interested PACKMAN from AliEn LGCG, OSCAR – not really suitable, more for site management?
RWL Jones, Lancaster University
CMT and deployable code CMT and deployable code
Christian Arnault and Charles Loomis have a beta-release Christian Arnault and Charles Loomis have a beta-release of CMT that will produce package rpms, which is a large of CMT that will produce package rpms, which is a large step along the waystep along the way
Still need to have minimal dependencies/clean code! Need to make the package dependencies explicit Rpm requires root to install in the system database (but not
for a private installation) Developer and binary installations being produced, Developer and binary installations being produced,
probably needs further refinementprobably needs further refinement Work to expose dependencies as PACMAN cache files Work to expose dependencies as PACMAN cache files
ongoingongoing
Note: much work elsewhere in producing rpms of ATLAS Note: much work elsewhere in producing rpms of ATLAS code, notably in Copenhagen; this effort has the advantage code, notably in Copenhagen; this effort has the advantage of the full dependency knowledge in CMT being exposableof the full dependency knowledge in CMT being exposable
RWL Jones, Lancaster University
pacmanpacman
Package manager for the grid in development by Saul Package manager for the grid in development by Saul Youssef (Boston U, GriPhyN/iVDGL)Youssef (Boston U, GriPhyN/iVDGL)
Single tool to easily manage installation and environment Single tool to easily manage installation and environment setup for the setup for the longlong list of ATLAS, grid and other software list of ATLAS, grid and other software components needed to ‘Grid-enable’ a sitecomponents needed to ‘Grid-enable’ a site fetch, install, configure, add to login environment, update
Sits over top of (and is compatible with) the many software Sits over top of (and is compatible with) the many software packaging approaches (rpm, tar.gz, etc.)packaging approaches (rpm, tar.gz, etc.)
Uses dependency hierarchy, so one command can drive the Uses dependency hierarchy, so one command can drive the installation of a complete environment of many packagesinstallation of a complete environment of many packages
Packages organized into Packages organized into cachescaches hosted at various sites hosted at various sites How to fetch can be cached rather than the desired object
Includes a web interface (for each cache) as well as Includes a web interface (for each cache) as well as command line toolscommand line tools
RWL Jones, Lancaster University
# An encryption package needed by Globus# An encryption package needed by Globus##name = ‘SSLeay’name = ‘SSLeay’description = ‘Encryption’description = ‘Encryption’url = ‘http://www.psy.uq.oz.au/~ftp/Crypto/ssleay’url = ‘http://www.psy.uq.oz.au/~ftp/Crypto/ssleay’source = ‘http://www.psy.uq.oz.au/~ftp/Crypto/ssleay’source = ‘http://www.psy.uq.oz.au/~ftp/Crypto/ssleay’systems = { ‘linux-i386’: [‘SSLeay-0.9.0b.tar.gz’,’SSLeay-0.9.0b’],\systems = { ‘linux-i386’: [‘SSLeay-0.9.0b.tar.gz’,’SSLeay-0.9.0b’],\ ‘ ‘linux2’ : [‘SSLeay-0.9.0b.tar.gz’,’SSLeay-0.9.0b’],\linux2’ : [‘SSLeay-0.9.0b.tar.gz’,’SSLeay-0.9.0b’],\ ‘ ‘sunos5’ : [‘SSLeay-0.9.0b.tar.gz’,’SSLeay-0.9.0b’] }sunos5’ : [‘SSLeay-0.9.0b.tar.gz’,’SSLeay-0.9.0b’] }depends = []depends = []exists = [‘/usr/local/bin/perl’]exists = [‘/usr/local/bin/perl’]inpath = [‘gcc’]inpath = [‘gcc’]bins = []bins = []paths = []paths = []enviros = []enviros = []localdoc = ‘README’localdoc = ‘README’daemons = []daemons = []install = { \ install = { \ ‘ ‘root’: [‘./Configure linux-elf’,’make clean’, \root’: [‘./Configure linux-elf’,’make clean’, \ ’ ’make depend’,’make’,’make rehash’,’make test’,’make install’], \make depend’,’make’,’make rehash’,’make test’,’make install’], \ ‘ ‘*’ : [‘./Configure linux-elf’,’make clean’,’make depend’,’make’, \*’ : [‘./Configure linux-elf’,’make clean’,’make depend’,’make’, \ ’ ’make rehash’,’make test’] }make rehash’,’make test’] }
RWL Jones, Lancaster University
Grid Applications ToolkitGrid Applications Toolkit
Horst Severini, Kaushik De, Ed May, Wensheng Deng, Jerry GHorst Severini, Kaushik De, Ed May, Wensheng Deng, Jerry Gieraltowskiieraltowski..… (US Test Bed)… (US Test Bed)
Repackaged Athena-Atlfast (OO fast detector simulation) for grid testbed Repackaged Athena-Atlfast (OO fast detector simulation) for grid testbed (building on Julian Phillips and UK effort)(building on Julian Phillips and UK effort)
Script 1: can run on any globus enabled node (requires transfer of ~17MB source)
Script 2: runs on machine with packaged software preinstalled on grid site
Script 3: runs on afs enabled sites (latest version of software is used)
Other user toolkit contentsOther user toolkit contents to check status of grid nodes submit jobs (without worrying about underlying middleware or ATLAS
software) uses only basic RSL & globus-url-copy
RWL Jones, Lancaster University
Monitoring ToolMonitoring Tool
GridView - a simple visualization tool using Globus ToolkitGridView - a simple visualization tool using Globus Toolkit First native Globus application for ATLAS grid (March 2001) Collects information using Globus tools. Archival information is stored in
MySQL server on a different machine. Data published through web server on a third machine.
Plans:Plans: Java versionJava version Better visualizationBetter visualization
Historical plots Hierarchical MDS information Graphical view of system health
New MDS schemasNew MDS schemas Optimize archived variablesOptimize archived variables Publishing historical information through GIIS servers??Publishing historical information through GIIS servers?? Explore discovery toolsExplore discovery tools Explore scalability to large systemsExplore scalability to large systemsPatrick McGuiganPatrick McGuigan
RWL Jones, Lancaster University
RWL Jones, Lancaster University
MDS InformationMDS Information
Listing of available object classes
RWL Jones, Lancaster University
More DetailsMore Details
RWL Jones, Lancaster University
Data Management Data Management ArchitectureArchitecture
AMI
ATLAS Metatdata Interface
Query LFN
Associated attributes and
values
MAGDA
MAnager MAnager for Grid-for Grid-
based Databased Data
Manage Manage replication, replication,
physical locationphysical location
VDC
Virtual Data
Catalog
Derive and
transform LFNs
RWL Jones, Lancaster University
MAnager for Grid-based Data (essentially the `replica catalogue’ tool)MAnager for Grid-based Data (essentially the `replica catalogue’ tool) Designed for ‘managed production’ Designed for ‘managed production’ andand ‘chaotic end-user’ usage ‘chaotic end-user’ usage Designed for rapid development of components to support users quickly, with components later Designed for rapid development of components to support users quickly, with components later
replaced by Grid Toolkit elementsreplaced by Grid Toolkit elements Deploy as an evolving production tool and as a testing ground for Grid Toolkit components GDMP will be incorporated
Application in DCsApplication in DCs Logical files can optionally be organized into collections File management in production; replication to BNL; CERN, BNL data access GDMP integration, replication and end-user data access in DC1
DevelopmentsDevelopments Interface with AMI (ATLAS Metadata Interface, allows queries on Logical File Name collections by Interface with AMI (ATLAS Metadata Interface, allows queries on Logical File Name collections by
users, Grenoble project)users, Grenoble project) Interfaces to Virtual Data Catalogue, see AV’s talk)Interfaces to Virtual Data Catalogue, see AV’s talk) Interfacing with hybrid ROOT/RDBMS event storeInterfacing with hybrid ROOT/RDBMS event store Athena (ATLAS offline framework) integration; further grid integrationAthena (ATLAS offline framework) integration; further grid integration
Managing Data -MagdaManaging Data -Magda
Info: http://www.usatlas.bnl.gov/magda/info
Engine: http://www.usatlas.bnl.gov/magda/dyShowMain.pl
T Wenaus, W Deng
RWL Jones, Lancaster University
Magda ArchitectureMagda Architecture
DB access via perl, C++, java, cgi (perl) scripts; C++ and Java APIs auto-generated off the MySQL DB schema
User interaction via web interface and command lineUser interaction via web interface and command line
RWL Jones, Lancaster University
ConclusionConclusion
The Grid is the only viable solution to the ATLAS Computing The Grid is the only viable solution to the ATLAS Computing problemproblem
The problems of coherence across the Atlantic are large ATLAS (and CMS etc) are `at the sharp end’, so we will force the
divide to be bridged Many applications have been developed, but need to be Many applications have been developed, but need to be
refined/mergedrefined/merged These revise our requirements – we must use LCG/GGF and any
other forum to ensure the middleware projects satisfy the real needs; this is not a test bed!
The progress so far is impressive and encouragingThe progress so far is impressive and encouraging Good collaborations (especially ATLAS/LHCb)
The real worry is scaling up to the full systemThe real worry is scaling up to the full system Money! Manpower! Diplomacy?!