PASI: Mendoza, Argentina (May 17, 2005)Paul Avery1 University of Florida [email protected] Integrating Universities and Laboratories In National Cyberinfrastructure

PASI: Mendoza, Argentina (May 17, 2005)

Paul Avery 1

Paul AveryUniversity of [email protected]

Integrating Universities and Laboratories In National

Cyberinfrastructure

PASI LectureMendoza, Argentina

May 17, 2005


Paul Avery 2

Outline of Talk Cyberinfrastructure and Grids

Data intensive disciplines and Data Grids

The Trillium Grid collaborationGriPhyN, iVDGL, PPDG

The LHC and its computing challenges Grid3 and the Open Science Grid A bit on networks Education and Outreach Challenges for the future Summary

Presented from a physicist’s perspective!


Paul Avery 3

Cyberinfrastructure (cont)Software programs, services, instruments,

data, information, knowledge, applicable to specific projects, disciplines, and communities.

Cyberinfrastructure layer of enabling hardware, algorithms, software, communications, institutions, and personnel. A platform that empowers researchers to innovate and eventually revolutionize what they do, how they do it, and who participates.

Base technologies: Computation, storage, and communication components that continue to advance in raw capacity at exponential rates.

[Paraphrased from NSF Blue Ribbon Panel report, 2003]

Challenge: Creating and operating advanced cyberinfrastructure andintegrating it in science and engineering applications.


Paul Avery 4

Cyberinfrastructure and Grids Grid: Geographically distributed computing

resources configured for coordinated use

Fabric: Physical resources & networks provide raw capability

Ownership: Resources controlled by owners and shared w/ others

Middleware: Software ties it all together: tools, services, etc.

Enhancing collaboration via transparent resource sharing

US-CMS“Virtual Organization”


Paul Avery 5

Data Grids & Collaborative Research

Team-based 21st century scientific discoveryStrongly dependent on advanced information technologyPeople and resources distributed internationally

Dominant factor: data growth (1 Petabyte = 1000 TB)2000 ~0.5 Petabyte2005 ~10 Petabytes2010 ~100 Petabytes2015-7 ~1000 Petabytes?

Drives need for powerful linked resources: “Data Grids”

Computation Massive, distributed CPUData storage and access Distributed hi-speed disk and

tapeData movement International optical networks

Collaborative research and Data GridsData discovery, resource sharing, distributed analysis, etc.

How to collect, manage,access and interpret thisquantity of data?


Paul Avery 6

Examples of Data Intensive Disciplines

High energy & nuclear physicsBelle, BaBar, Tevatron, RHIC, JLABLarge Hadron Collider (LHC)

AstronomyDigital sky surveys, “Virtual” ObservatoriesVLBI arrays: multiple- Gb/s data streams

Gravity wave searchesLIGO, GEO, VIRGO, TAMA, ACIGA, …

Earth and climate systemsEarth Observation, climate modeling, oceanography, …

Biology, medicine, imagingGenome databasesProteomics (protein structure & interactions, drug delivery,

…)High-resolution brain scans (1-10m, time dependent)

Primary driver


Paul Avery 7

Our Vision & Goals

Develop the technologies & tools needed to exploit a Grid-based cyberinfrastructure

Apply and evaluate those technologies & tools in challenging scientific problems

Develop the technologies & procedures to support a permanent Grid-based cyberinfrastructure

Create and operate a persistent Grid-based cyberinfrastructure in support of discipline-specific research goals

GriPhyN + iVDGL + DOE Particle Physics Data Grid (PPDG) = Trillium

End-to-end


Paul Avery 8

Our Science Drivers Experiments at Large Hadron Collider

New fundamental particles and forces100s of Petabytes 2007 - ?

High Energy & Nuclear Physics exptsTop quark, nuclear matter at extreme

density~1 Petabyte (1000 TB) 1997 – present

LIGO (gravity wave search)Search for gravitational waves100s of Terabytes 2002 – present

Sloan Digital Sky SurveySystematic survey of astronomical objects10s of Terabytes 2001 – present

Data

gro

wth

Com

mu

nit

y g

row

th

2007

2005

2003

2001

2009


Paul Avery 9

Grid Middleware: Virtual Data Toolkit

Sources(CVS)

Patching

GPT srcbundles

NMI

Build & TestCondor pool

22+ Op. Systems

Build

Test

Package

VDT

Build

Many Contributors

Build

Pacman cache

RPMs

Binaries

Binaries

Binaries Test

A unique laboratory for testing, supporting, deploying, packaging, upgrading, & troubleshooting complex sets of software!


Paul Avery 10

VDT Growth Over 3 Years

0

5

10

15

20

25

30

35Ja

n-02

Apr

-02

Jul-0

2

Oct

-02

Jan-

03

Apr

-03

Jul-0

3

Oct

-03

Jan-

04

Apr

-04

Jul-0

4

Oct

-04

Jan-

05

Apr

-05

VDT 1.1.x VDT 1.2.x VDT 1.3.x

# o

f co

mponen

ts

VDT 1.0Globus 2.0bCondor 6.3.1

VDT 1.1.7Switch to Globus 2.2

VDT 1.1.11Grid3

VDT 1.1.8First real use by LCG

www.griphyn.org/vdt/


Paul Avery 11

Components of VDT 1.3.5 Globus 3.2.1 Condor 6.7.6 RLS 3.0 ClassAds 0.9.7 Replica 2.2.4 DOE/EDG CA certs ftsh 2.0.5 EDG mkgridmap EDG CRL Update GLUE Schema 1.0 VDS 1.3.5b Java Netlogger 3.2.4 Gatekeeper-Authz MyProxy1.11 KX509

System Profiler GSI OpenSSH 3.4 Monalisa 1.2.32 PyGlobus 1.0.6 MySQL UberFTP 1.11 DRM 1.2.6a VOMS 1.4.0 VOMS Admin 0.7.5 Tomcat PRIMA 0.2 Certificate Scripts Apache jClarens 0.5.3 New GridFTP Server GUMS 1.0.1


Paul Avery 12

Collaborative Relationships:A CS + VDT Perspective

ComputerScience

Research

VirtualData

Toolkit

Partner science projectsPartner networking projectsPartner outreach projects

LargerScience

Community

Globus, Condor, NMI, iVDGL, PPDGEU DataGrid, LHC Experiments,QuarkNet, CHEPREO, Dig. Divide

ProductionDeployment

Tech

Transfer

Techniques

& software

RequirementsPrototyping

& experiments

Other linkages Work force CS researchers Industry

U.S.GridsInt’l

Outreach


Paul Avery 13

U.S. “Trillium” Grid Partnership Trillium = PPDG + GriPhyN + iVDGL

Particle Physics Data Grid: $12M (DOE) (1999 – 2006)GriPhyN: $12M (NSF) (2000 – 2005) iVDGL: $14M (NSF) (2001 – 2006)

Basic composition (~150 people)PPDG: 4 universities, 6 labsGriPhyN: 12 universities, SDSC, 3 labs iVDGL: 18 universities, SDSC, 4 labs, foreign partnersExpts: BaBar, D0, STAR, Jlab, CMS, ATLAS, LIGO,

SDSS/NVO

Coordinated internally to meet broad goalsGriPhyN: CS research, Virtual Data Toolkit (VDT)

development iVDGL: Grid laboratory deployment using VDT,

applicationsPPDG: “End to end” Grid services, monitoring, analysisCommon use of VDT for underlying Grid middlewareUnified entity when collaborating internationally


Paul Avery 14

Goal: Peta-scale Data Grids forGlobal Science

Virtual Data Tools

Request Planning &Scheduling Tools

Request Execution & Management Tools

Transforms

Distributed resources(code, storage, CPUs,networks)

ResourceManagement

Services

Security andPolicy

Services

Other GridServices

Interactive User Tools

Production TeamSingle Researcher Workgroups

Raw datasource

PetaOps Petabytes Performance


Paul Avery 15

Sloan Digital Sky Survey (SDSS)Using Virtual Data in GriPhyN

1

10

100

1000

10000

100000

1 10 100

Num

ber

of C

lust

ers

Number of Galaxies

Galaxy clustersize distribution

Sloan Data


Paul Avery 16

The LIGO Scientific Collaboration (LSC)and the LIGO Grid

LIGO Grid: 6 US sites

* LHO, LLO: observatory sites* LSC - LIGO Scientific Collaboration - iVDGL supported

iVDGL has enabled LSC to establish a persistent production grid

Cardiff

AEI/Golm •

+ 3 EU sites (Cardiff/UK, AEI/Germany)

Birmingham•


Paul Avery 17

Large Hadron Collider &its Frontier Computing

Challenges


Paul Avery 18

Search for Origin of Mass New fundamental forces Supersymmetry Other new particles 2007 – ?

TOTEM

LHCb

ALICE

27 km Tunnel in Switzerland & France

CMS

ATLAS

Large Hadron Collider (LHC)@ CERN


Paul Avery 19

CMS: “Compact” Muon Solenoid

Inconsequential humans


Paul Avery 20

LHC Data Rates: Detector to Storage

Level 1 Trigger: Special Hardware

40 MHz

75 KHz 75 GB/sec

5 KHz 5 GB/sec

Level 2 Trigger: Commodity CPUs

100 Hz0.15 – 1.5 GB/sec

Level 3 Trigger: Commodity CPUs

Raw Data to storage(+ simulated data)

Physics filtering ~TBytes/sec


Paul Avery 21

All charged tracks with pt > 2 GeV

Reconstructed tracks with pt > 25 GeV

(+30 minimum bias events)

109 collisions/sec, selectivity: 1 in 1013

Complexity: Higgs Decay to 4 Muons


Paul Avery 22

LHC: Petascale Global Science Complexity: Millions of individual detector channels Scale: PetaOps (CPU), 100s of Petabytes (Data) Distribution: Global distribution of people & resources

CMS Example- 20075000+ Physicists 250+ Institutes 60+ Countries

BaBar/D0 Example - 2004700+ Physicists 100+ Institutes 35+ Countries


Paul Avery 23

CMS Experiment

LHC Global Data Grid (2007+)

Online System

CERN Computer Center

USAKorea RussiaUK

Maryland

150 - 1500 MB/s

>10 Gb/s

10-40 Gb/s

2.5-10 Gb/s

Tier 0

Tier 1

Tier 3

Tier 2

Physics caches

PCs

Iowa

UCSDCaltechU Florida

5000 physicists, 60 countries

10s of Petabytes/yr by 2008 1000 Petabytes in < 10 yrs?

FIU

Tier 4


Paul Avery 24

University Tier2 Centers Tier2 facility

Essential university role in extended computing infrastructure

20 – 25% of Tier1 national laboratory, supported by NSFValidated by 3 years of experience (CMS, ATLAS, LIGO)

FunctionsPerform physics analysis, simulationsSupport experiment softwareSupport smaller institutions

Official role in Grid hierarchy (U.S.)Sanctioned by MOU with parent organization (ATLAS, CMS,

LIGO)Selection by collaboration via careful processLocal P.I. with reporting responsibilities


Paul Avery 25

Grids and Globally Distributed Teams

Non-hierarchical: Chaotic analyses + productions Superimpose significant random data flows


Paul Avery 26

Grid3 andOpen Science Grid


Paul Avery 27

Grid3: A National Grid Infrastructure32 sites, 4000 CPUs: Universities + 4 national

labsPart of LHC Grid, Running since October 2003Sites in US, Korea, Brazil, TaiwanApplications in HEP, LIGO, SDSS, Genomics, fMRI,

CS

Brazil

http://www.ivdgl.org/grid3


Paul Avery 28

Grid3 World Map


Paul Avery 29

Grid3 Components Computers & storage at ~30 sites: 4000 CPUs Uniform service environment at each site

Globus Toolkit: Provides basic authentication, execution management, data movement

Pacman: Installs numerous other VDT and application services

Global & virtual organization servicesCertification & registration authorities, VO membership

services, monitoring services

Client-side tools for data access & analysis Virtual data, execution planning, DAG management,

execution management, monitoring

IGOC: iVDGL Grid Operations Center Grid testbed: Grid3dev

Middleware development and testing, new VDT versions, etc.


Paul Avery 30

Grid3 Applications

CMS experiment p-p collision simulations & analysis

ATLAS experiment p-p collision simulations & analysis

BTEV experiment p-p collision simulations & analysis

LIGO Search for gravitational wave sources

SDSS Galaxy cluster findingBio-molecular analysis

Shake n Bake (SnB) (Buffalo)

Genome analysis GADU/GnarefMRI Functional MRI (Dartmouth)CS Demonstrators Job Exerciser, GridFTP,

NetLoggerwww.ivdgl.org/grid3/applications


Paul Avery 31

Grid3 Shared Use Over 6 months

CMS DC04

ATLASDC2

Sep 10

Usa

ge:

CP

Us


Paul Avery 32

Grid3 Production Over 13 Months


Paul Avery 33

U.S. CMS 2003 Production10M p-p collisions; largest ever

2x simulation sample ½ manpower

Multi-VO sharing


Paul Avery 34

Grid3 as CS Research Lab:E.g., Adaptive Scheduling

Adaptive data placementin a realistic environment(K. Ranganathan)

Enables comparisonswith simulations

0

10

20

30

40

50

60

70

80

22-Jan 23-Jan 24-Jan 25-Jan 26-Jan 27-Jan 28-Jan

Aver

age

num

ber o

f Idl

e C

PUs

FNAL_CMS

IU_ATLAS_Tier2

0

500

1000

1500

2000

2500

BN

L_

AT

LA

S

Ca

lTe

ch

-Gri

d3

UFlo

rid

a-P

G

IU_

AT

LA

S_

Tie

r2

UB

uff

alo

-CC

R

UC

Sa

nD

ieg

oP

G

UFlo

rid

a-G

rid

3

UM

_A

TLA

S

Va

nd

erb

ilt

AN

L_

HE

P

Ca

lTe

ch

-PG

Tota

l I/O

tra

ffic

(M

Byte

s)

0

200

400

600

800

Least Loaded At Data Cost Function

Tot

al R

espo

nse

Tim

e (s

econ

ds)


Paul Avery 35

Grid3 Lessons Learned How to operate a Grid as a facility

Tools, services, error recovery, procedures, docs, organization

Delegation of responsibilities (Project, VO, service, site, …)Crucial role of Grid Operations Center (GOC)

How to support people people relationsFace-face meetings, phone cons, 1-1 interactions, mail lists,

etc.

How to test and validate Grid tools and applicationsVital role of testbeds

How to scale algorithms, software, processSome successes, but “interesting” failure modes still occur

How to apply distributed cyberinfrastructureSuccessful production runs for several applications


Paul Avery 36

Grid3 Open Science Grid Iteratively build & extend Grid3

Grid3 OSG-0 OSG-1 OSG-2 …Shared resources, benefiting broad set of disciplinesGrid middleware based on Virtual Data Toolkit (VDT)Emphasis on “end to end” services for applications

OSG collaborationComputer and application scientistsFacility, technology and resource providers (labs,

universities)

Further develop OSGPartnerships and contributions from other sciences,

universities Incorporation of advanced networkingFocus on general services, operations, end-to-end

performance

Aim for Summer 2005 deployment


Paul Avery 37

http://www.opensciencegrid.org


Paul Avery 38

Enterprise

Technical Groups

ResearchGrid Projects

VOs

Researchers

Sites

Service Providers

Universities,Labs

activity1activity

1activity1Activities

Advisory Committee

Core OSG Staff(few FTEs, manager)

OSG Council(all members abovea certain threshold,

Chair, officers)

Executive Board(8-15 representatives

Chair, Officers)

OSG Organization


Paul Avery 39

OSG Technical Groups & Activities

Technical Groups address and coordinate technical areas

Propose and carry out activities related to their given areasLiaise & collaborate with other peer projects (U.S. &

international)Participate in relevant standards organizations.Chairs participate in Blueprint, Integration and Deployment

activities

Activities are well-defined, scoped tasks contributing to OSG

Each Activity has deliverables and a plan… is self-organized and operated… is overseen & sponsored by one or more Technical

GroupsTGs and Activities are where the real work gets done


Paul Avery 40

OSG Technical GroupsGovernance Charter, organization, by-laws,

agreements, formal processes

Policy VO & site policy, authorization, priorities, privilege & access rights

Security Common security principles, security infrastructure

Monitoring and Information Services

Resource monitoring, information services, auditing, troubleshooting

Storage Storage services at remote sites, interfaces, interoperability

Support Centers Infrastructure and services for user support, helpdesk, trouble ticket

Education / Outreach

Training, interface with various E/O projects

Networks (new) Including interfacing with various networking projects


Paul Avery 41

OSG Activities

Blueprint Defining principles and best practices for OSG

Deployment Deployment of resources & servicesProvisioning Connected to deploymentIncidence response

Plans and procedures for responding to security incidents

Integration Testing & validating & integrating new services and technologies

Data Resource Management (DRM)

Deployment of specific Storage Resource Management technology

Documentation Organizing the documentation infrastructure

Accounting Accounting and auditing use of OSG resources

Interoperability Primarily interoperability between Operations Operating Grid-wide services


Paul Avery 42

Connections to European Projects:

LCG and EGEE


Paul Avery 43

The Path to the OSG Operating Grid

OS

G D

eplo

ymen

t A

ctiv

ity

Metrics &Certification

Applicationvalidation

VO Application

SoftwareInstallation

OSG Integration Activity

Release Description

MiddlewareInteroperability

Software &packaging

Functionality & Scalability

Tests

Readiness plan adopted

Service deployment

OS

G O

per

atio

ns-

Pro

visi

on

ing

Act

ivit

y

ReleaseCandidate

Readinessplan

Effort

Resources

feedback


Paul Avery 44

OSG Integration Testbed

Brazil


Paul Avery 45

Status of OSG Deployment OSG infrastructure release accepted for

deployment. US CMS MOP “flood testing” successful D0 simulation & reprocessing jobs running on selected

OSG sites Others in various stages of readying applications &

infrastructure(ATLAS, CMS, STAR, CDF, BaBar, fMRI)

Deployment process underway: End of July? Open OSG and transition resources from Grid3 Applications will use growing ITB & OSG resources

during transition

http://osg.ivdgl.org/twiki/bin/view/Integration/WebHome


Paul Avery 46

Interoperability & Federation Transparent use of Federated Grid infrastructures a

goal There are sites that appear as part of “LCG” as well

as part of OSG/Grid3D0 bringing reprocessing to LCG sites through adaptor nodeCMS and ATLAS can run their jobs on both LCG and OSG

Increasing interaction with TeraGridCMS and ATLAS sample simulation jobs are running on

TeraGridPlans for TeraGrid allocation for jobs running in Grid3

model: with group accounts, binary distributions, external data management, etc


Paul Avery 47

Networks


Paul Avery 48

Evolving Science Requirements for Networks (DOE High Perf. Network

Workshop)

Science Areas

Today End2End

Throughput

5 years End2End

Throughput

5-10 Years End2End

Throughput

Remarks

High Energy Physics

0.5 Gb/s 100 Gb/s 1000 Gb/s High bulk throughput

Climate (Data &

Computation)

0.5 Gb/s 160-200 Gb/s

N x 1000 Gb/s

High bulk throughput

SNS NanoScience

Not yet started

1 Gb/s 1000 Gb/s + QoS for Control Channel

Remote control and time critical throughput

Fusion Energy

0.066 Gb/s(500 MB/s

burst)

0.2 Gb/s(500MB/20 sec. burst)

N x 1000 Gb/s

Time critical throughput

Astrophysics 0.013 Gb/s(1 TB/week)

N*N multicast

1000 Gb/s Computational steering and

collaborations

Genomics Data &

Computation

0.091 Gb/s(1 TB/day)

100s of users

1000 Gb/s + QoS for Control Channel

High throughput

and steering

See http://www.doecollaboratory.org/meetings/hpnpw/


Paul Avery 49

UltraLight: Advanced Networking

in Applications

10 Gb/s+ network• Caltech, UF, FIU, UM, MIT• SLAC, FNAL• Int’l partners• Level(3), Cisco, NLR

Funded by ITR2004


Paul Avery 50

UltraLight: New Information System

A new class of integrated information systems Includes networking as a managed resource for the first

timeUses “Hybrid” packet-switched and circuit-switched

optical network infrastructureMonitor, manage & optimize network and Grid Systems in

realtime

Flagship applications: HEP, eVLBI, “burst” imaging“Terabyte-scale” data transactions in minutesExtend Real-Time eVLBI to the 10 – 100 Gb/s Range

Powerful testbedSignificant storage, optical networks for testing new Grid

services

Strong vendor partnershipsCisco, Calient, NLR, CENIC, Internet2/Abilene


Paul Avery 51

Education and Outreach


Paul Avery 52

iVDGL, GriPhyN Education/Outreach

Basics $200K/yr Led by UT Brownsville Workshops, portals, tutorials New partnerships with

QuarkNet, CHEPREO, LIGO E/O, …


Paul Avery 53

June 2004 Grid Summer School First of its kind in the U.S. (South Padre Island,

Texas)36 students, diverse origins and types (M, F, MSIs, etc)

Marks new direction for U.S. Grid effortsFirst attempt to systematically train people in Grid

technologiesFirst attempt to gather relevant materials in one placeToday: Students in CS and PhysicsNext: Students, postdocs, junior & senior scientists

Reaching a wider audiencePut lectures, exercises, video, on the webMore tutorials, perhaps 2-3/yearDedicated resources for remote tutorialsCreate “Grid Cookbook”, e.g. Georgia Tech

Second workshop: July 11–15, 2005South Padre Island again


Paul Avery 54

QuarkNet/GriPhyN e-Lab Project

http://quarknet.uchicago.edu/elab/cosmic/home.jsp


Paul Avery 55

Student Muon Lifetime Analysis in GriPhyN/QuarkNet

CHEPREO: Center for High Energy Physics Research and Educational OutreachFlorida International University

Physics Learning Center CMS Research iVDGL Grid Activities AMPATH network (S.

America)

Funded September 2003

$4M initially (3 years) MPS, CISE, EHR, INT


Paul Avery 57

NEWS: Bulletin: ONE TWOWELCOME BULLETIN General InformationRegistrationTravel Information Hotel RegistrationParticipant List How to Get UERJ/Hotel Computer AccountsUseful Phone Numbers ProgramContact us: Secretariat Chairmen

Grids and the Digital DivideRio de Janeiro, Feb. 16-20, 2004

Background World Summit on Information Society HEP Standing Committee on Inter-

regional Connectivity (SCIC)

Themes Global collaborations, Grids and

addressing the Digital Divide Focus on poorly connected regionsNext meeting: Daegu, Korea May 23-27, 2005

http://www.lishep.uerj.br/LAFEX.jpg


Paul Avery 58

Partnerships Drive Success Integrating Grids in scientific research

“Lab-centric”: Activities center around large facility“Team-centric”: Resources shared by distributed teams“Knowledge-centric”: Knowledge generated/used by a community

Strengthening the role of universities in frontier researchCouples universities to frontier data intensive researchBrings front-line research and resources to studentsExploits intellectual resources at minority or remote institutions

Driving advances in IT/science/engineeringDomain sciences Computer ScienceUniversities LaboratoriesScientists StudentsNSF projects NSF projectsNSF DOEResearch communities IT industry


Paul Avery 59

Fulfilling the Promise ofNext Generation Science

Supporting permanent, national-scale Grid infrastructure

Large CPU, storage and network capability crucial for scienceSupport personnel, equipment maintenance, replacement,

upgradeTier1 and Tier2 resources a vital part of infrastructureOpen Science Grid a unique national infrastructure for science

Supporting the maintenance, testing and dissemination of advanced middleware

Long-term support of the Virtual Data ToolkitVital for reaching new disciplines & for supporting large

international collaborations

Continuing support for HEP as a frontier challenge driver

Huge challenges posed by LHC global interactive analysisNew challenges posed by remote operation of Global

Accelerator Network


Paul Avery 60

Fulfilling the Promise (2) Creating even more advanced cyberinfrastructure

Integrating databases in large-scale Grid environments Interactive analysis with distributed teamsPartnerships involving CS research with application drivers

Supporting the emerging role of advanced networksReliable, high performance LANs and WANs necessary for

advanced Grid applications

Partnering to enable stronger, more diverse programs

Programs supported by multiple Directorates, a la CHEPREONSF-DOE joint initiativesStrengthen ability of universities and labs to work together

Providing opportunities for cyberinfrastructure training, education & outreach

Grid tutorials, Grid CookbookCollaborative tools for student-led projects & research


Paul Avery 61

Summary Grids enable 21st century collaborative science

Linking research communities and resources for scientific discovery

Needed by global collaborations pursuing “petascale” science

Grid3 was an important first step in developing US Grids

Value of planning, coordination, testbeds, rapid feedbackValue of learning how to operate a Grid as a facilityValue of building & sustaining community relationships

Grids drive need for advanced optical networks Grids impact education and outreach

Providing technologies & resources for training, education, outreach

Addressing the Digital Divide

OSG: a scalable computing infrastructure for science?Strategies needed to cope with increasingly large scale


Paul Avery 62

Grid Project ReferencesOpen Science Grid

www.opensciencegrid.org

Grid3 www.ivdgl.org/grid3

Virtual Data Toolkit www.griphyn.org/vdt

GriPhyN www.griphyn.org

iVDGL www.ivdgl.org

PPDG www.ppdg.net

CHEPREO www.chepreo.org

UltraLight ultralight.cacr.caltech.edu

Globus www.globus.org

Condor www.cs.wisc.edu/condor

LCG www.cern.ch/lcg

EU DataGrid www.eu-datagrid.org

EGEE www.eu-egee.org


Paul Avery 63

Extra Slides


Paul Avery 64

GriPhyN Goals Conduct CS research to achieve vision

Virtual Data as unifying principlePlanning, execution, performance monitoring

Disseminate through Virtual Data ToolkitA “concrete” deliverable

Integrate into GriPhyN science experimentsCommon Grid tools, services

Educate, involve, train students in IT researchUndergrads, grads, postdocs, Underrepresented groups


Paul Avery 65

iVDGL Goals Deploy a Grid laboratory

Support research mission of data intensive experimentsProvide computing and personnel resources at university sitesProvide platform for computer science technology

developmentPrototype and deploy a Grid Operations Center (iGOC)

Integrate Grid software tools Into computing infrastructures of the experiments

Support delivery of Grid technologiesHardening of the Virtual Data Toolkit (VDT) and other

middleware technologies developed by GriPhyN and other Grid projects

Education and OutreachLead and collaborate with Education and Outreach effortsProvide tools and mechanisms for underrepresented groups

and remote regions to participate in international science projects


Paul Avery 66

CMS: Grid Enabled Analysis Architecture

SchedulerCatalogs

Grid ServicesWeb Server

ExecutionPriority

Manager

Grid WideExecutionService

DataManage-

mentFully-ConcretePlanner

Fully-AbstractPlanner

Analysis Client

Virtual Data

Replica

ApplicationsMonitoring

Partially-AbstractPlanner

Metadata

HTTP, SOAP, XML-RPC

Chimera

Sphinx

MonALISA

•ROOT (analysis tool)•Python•Cojac (detector viz)/IGUANA (cms viz)

Clarens

MCRunjob

BOSS

RefDB

POOL

ORCA

ROOT FAMOS

VDT-Server

MOPDB

•Discovery•ACL management•Cert. based access

Clients talk standard protocols to “Grid Services Web Server”

Simple Web service API allows simple or complex analysis clients

Typical clients: ROOT, Web Browser, ….

Clarens portal hides complexity

Key features: Global Scheduler, Catalogs, Monitoring, Grid-wide Execution service

AnalysisClient


Paul Avery 67

“Virtual Data”: Derivation & Provenance

Most scientific data are not simple “measurements”They are computationally corrected/reconstructedThey can be produced by numerical simulation

Science & eng. projects are more CPU and data intensive

Programs are significant community resources (transformations)

So are the executions of those programs (derivations)

Management of dataset dependencies critical!Derivation: Instantiation of a potential data productProvenance: Complete history of any existing data

productPreviously: Manual methodsGriPhyN: Automated, robust

tools


Paul Avery 68

decay = WWWW e Pt > 20

decay = WWWW e

decay = WWWW leptons

mass = 160

decay = WW

decay = ZZ

decay = bb

Other cuts

Scientist adds a new derived data branch & continues analysis

Virtual Data Example: HEP Analysis

Other cuts Other cuts Other cuts


Paul Avery 69

Language: define software environments Interpreter: create, install, configure, update, verify

environmentsVersion 3.0.2 released Jan. 2005 LCG/Scram ATLAS/CMT CMS DPE/tar/make LIGO/tar/make OpenSource/tar/

make

Globus/GPT NPACI/TeraGrid/tar/

make D0/UPS-UPD Commercial/tar/make

Combine and manage software from arbitrary sources.

% pacman –get iVDGL:Grid3

“1 button install”: Reduce burden on administrators

Remote experts define installation/ config/updating for everyone at once

VDT

ATLAS

ATLAS

NPACI

NPACI

D-Zero

D-Zero

iVDGL

iVDGL

UCHEP

UCHEP

% pacman

VDT

CMS/DPE

LIGO

Packaging of Grid Software: Pacman


Paul Avery 70

“I’ve detected a muon calibration error and want to know which derived data products need to be recomputed.”

“I’ve found some interesting data, but I need to know exactly what corrections were applied before I can trust it.”

“I want to search a database for 3 muon events. If a program that does this analysis exists, I won’t have to write one from scratch.”

“I want to apply a forward jet analysis to 100M events. If the results already exist, I’ll save weeks of computation.”

Virtual Data Motivations

VDC

Describe Discover

Reuse Validate


Paul Avery 71

U.S. Funded ProjectsGriPhyN (NSF) iVDGL (NSF)Particle Physics Data Grid (DOE)UltraLightTeraGrid (NSF)DOE Science Grid (DOE)NEESgrid (NSF)NSF Middleware Initiative (NSF)

Background: Data Grid Projects

EU, Asia projectsEGEE (EU)LCG (CERN)DataGridEU national ProjectsDataTAG (EU)CrossGrid (EU)GridLab (EU) Japanese, Korea Projects

Many projects driven/led by HEP + CS Many 10s x $M brought into the field Large impact on other sciences, education

Driven primarily by HEP applications


Paul Avery 72

“Virtual Data”: Derivation & Provenance

Most scientific data are not simple “measurements”They are computationally corrected/reconstructedThey can be produced by numerical simulation

Science & eng. projects are more CPU and data intensive

Programs are significant community resources (transformations)

So are the executions of those programs (derivations)

Management of dataset dependencies critical!Derivation: Instantiation of a potential data productProvenance: Complete history of any existing data

productPreviously: Manual methodsGriPhyN: Automated, robust

tools


Paul Avery 73

Muon Lifetime Analysis Workflow

pythia_input

pythia.exe

cmsim_input

cmsim.exe

writeHits

writeDigis

begin v /usr/local/demo/scripts/cmkin_input.csh file i ntpl_file_path file i template_file file i num_events stdout cmkin_param_fileend

begin v /usr/local/demo/binaries/kine_make_ntpl_pyt_cms121.exe pre cms_env_var stdin cmkin_param_file stdout cmkin_log file o ntpl_fileend

begin v /usr/local/demo/scripts/cmsim_input.csh file i ntpl_file file i fz_file_path file i hbook_file_path file i num_trigs stdout cmsim_param_fileend

begin v /usr/local/demo/binaries/cms121.exe condor copy_to_spool=false condor getenv=true stdin cmsim_param_file stdout cmsim_log file o fz_file file o hbook_fileend

begin v /usr/local/demo/binaries/writeHits.sh condor getenv=true pre orca_hits file i fz_file file i detinput file i condor_writeHits_log file i oo_fd_boot file i datasetname stdout writeHits_log file o hits_dbend

begin v /usr/local/demo/binaries/writeDigis.sh pre orca_digis file i hits_db file i oo_fd_boot file i carf_input_dataset_name file i carf_output_dataset_name file i carf_input_owner file i carf_output_owner file i condor_writeDigis_log stdout writeDigis_log file o digis_dbend

(Early) Virtual Data Language

CMS “Pipeline”


Paul Avery 75

QuarkNet Portal Architecture

Simpler interface for non-experts Builds on Chiron portal


Paul Avery 76

Integration of GriPhyN and IVDGL

Both funded by NSF large ITRs, overlapping periodsGriPhyN: CS Research, Virtual Data Toolkit (9/2000–

9/2005) iVDGL: Grid Laboratory, applications (9/2001–

9/2006) Basic composition

GriPhyN: 12 universities, SDSC, 4 labs (~80 people) iVDGL: 18 institutions, SDSC, 4 labs (~100 people)Expts: CMS, ATLAS, LIGO, SDSS/NVO

GriPhyN (Grid research) vs iVDGL (Grid deployment)GriPhyN: 2/3 “CS” + 1/3 “physics” ( 0% H/W) iVDGL: 1/3 “CS” + 2/3 “physics” (20% H/W)

Many common elementsCommon Directors, Advisory Committee, linked

managementCommon Virtual Data Toolkit (VDT)Common Grid testbedsCommon Outreach effort

ScienceReview

ProductionManager

Researcher

instrument

Applications

storageelement

Grid

Grid Fabric

storageelement

storageelement

data

ServicesServices

discovery

discovery

sharing

VirtualData

Production Analysisparams

exec.

data

composition

VirtualData

planning

Planning

Production Analysisparams

exec.

data

Planning

Execution

planning

Virtual Data

Toolkit

Chimera virtual data

system

Pegasus planner

DAGman

Globus ToolkitCondor

Ganglia, etc.Gri

Ph

yN

O

verv

iew

Execution


Paul Avery 78

Chiron/QuarkNet Architecture


Paul Avery 79

Cyberinfrastructure

“A new age has dawned in scientific & engineering research, pushed by continuing progress in computing, information, and communication technology, & pulled by the expanding complexity, scope, and scale of today’s challenges. The capacity of this technology has crossed thresholds that now make possible a comprehensive “cyberinfrastructure” on which to build new types of scientific & engineering knowledge environments & organizations and to pursue research in new ways & with increased efficacy.”

[NSF Blue Ribbon Panel report, 2003]


Paul Avery 80

Fulfilling the Promise ofNext Generation Science

Our multidisciplinary partnership of physicists, computer scientists, engineers, networking specialists and education experts, from universities and laboratories, has achieved tremendous success in creating and maintaining general purpose cyberinfrastructure supporting leading-edge science.

But these achievements have occurred in the context of overlapping short-term projects. How can we ensure the survival of valuable existing cyber-infrastructure while continuing to address new challenges posed by frontier scientific and engineering endeavors?


Paul Avery 81

Production Simulations on Grid3

Used = 1.5 US-CMS resources

US-CMS Monte Carlo Simulation

USCMS

Non-USCMS

Documents

PASI: Mendoza, Argentina (May 17, 2005)Paul Avery1 University of Florida [email protected] Integrating Universities and Laboratories In National Cyberinfrastructure