Upload
clementine-hutchinson
View
222
Download
1
Tags:
Embed Size (px)
Citation preview
Managing Simulation OutputManaging Simulation Output
Storage Resource BrokerStorage Resource Broker
Reagan W. MooreReagan W. Moore
http://www.sdsc.edu/srbhttp://www.sdsc.edu/srb
TopicsTopics
• Data management evolution• Shared collections• Digital Libraries• Persistent Archives
• Managing simulation output• Project level / National level / International
• Demonstration of shared collections• Access to collections at SDSC
Date
ProjectGBs of
data stored
1000Õs of files
GBs of data
stored
1000Õs of files
Users with
ACLs
GBs of data
stored
1000Õs of files
Users with
ACLsData Grid
NSF / NVO 17,800 5,139 51,380 8,690 80 106,070 14,001 100
NSF / NPACI 1,972 1,083 17,578 4,694 380 35,109 7,240 380
Hayden 6,800 41 7,201 113 178 8,013 161 227
Pzone 438 31 812 47 49 23,475 13,576 68
NSF / LDAS-SALK 239 1 4,562 16 66 143,429 165 67
NSF / SLAC-JCSG 514 77 4,317 563 47 17,595 1,814 55
NSF / TeraGrid 80,354 685 2,962 267,422 6,970 3,267
NIH / BIRN 5,416 3,366 148 17,155 16,116 385
Digital Library
NSF / LTER 158 3 233 6 35 257 41 36
NSF / Portal 33 5 1,745 48 384 2,620 53 460
NIH / AfCS 27 4 462 49 21 733 94 21
NSF / SIO Explorer 19 1 1,734 601 27 2,653 1,159 27
NSF / SCEC 15,246 1,737 52 168,689 3,544 73
Persistent Archive
NARA 7 2 63 81 58 2,999 2,033 58
NSF / NSDL 2,785 20,054 119 5,698 50,600 136
UCSD Libraries 127 202 29 190 208 29
NHPRC / PAT 1,888 521 28
TOTAL 28 TB 6 mil 194 TB 40 mil 4,635 804 TB 118 mil 5,417
5/17/02 6/30/04 7/10/06
Data GridData Grid
Using a Data Grid – Using a Data Grid – in Abstractin Abstract
Ask for d
ata
•User asks for data from the data grid
Data d
elivere
d
•The data is found and returned•Where & how details are hidden
Using a Data Grid - Using a Data Grid - DetailsDetails
Storage Resource Broker
•Data request goes to SRB Server
Storage Resource Broker
Metadata Catalog
DB
•Server looks up data in catalog
•Catalog tells which SRB server has data
•1st server asks 2nd for data
•The data is found and returned
•User asks for data
Using a Data Grid - Using a Data Grid - DetailsDetails
SRB
MCAT
DB
SRB
SRB
SRB
SRB SRB
•Data Grid has arbitrary number of servers•Heterogeneity is hidden from users
Unix Shell
NT Browser,Kepler Actors,HDF5 Library
http,Portlet,WSDL,
OAI-PMH)
DSpace,OpenDAP,GridFTP,Fedora
Archives - Tape,Sam-QFS, DMF,
HPSS, ADSM,UniTree, ADS
Databases -DB2, Oracle,
Sybase, Postgres, mySQL, Informix
File SystemsUnix, NT,Mac OSX
Application
ORB
Storage Repository AbstractionDatabase Abstraction
Databases -DB2, Oracle, Sybase,
Postgres, mySQL,Informix
CLibrary,
Java
Logical Name Space
LatencyManagement
DataTransport
MetadataTransport
Consistency & Metadata Management / Authorization, Authentication, Audit
Linux I/OC++
DLL /Python,
Perl, Windows
Federation Management
Storage Resource Broker 3.3.1Storage Resource Broker 3.3.1
Unix Shell
NT Browser,Kepler Actors,HDF5 Library
http,Portlet,WSDL,
OAI-PMH)
DSpace,OpenDAP,GridFTP,Fedora
Archives - Tape,Sam-QFS, DMF,
HPSS, ADSM,UniTree, ADS
Databases -DB2, Oracle,
Sybase, Postgres, mySQL, Informix
File SystemsUnix, NT,Mac OSX
Application
ORB
Storage Repository AbstractionDatabase Abstraction
Databases -DB2, Oracle, Sybase,
Postgres, mySQL,Informix
C Library,Java
Logical Name Space
LatencyManagement
DataTransport
MetadataTransport
Consistency & Metadata Management / Authorization, Authentication, Audit
Linux I/OC++
DLL /Python,
Perl, Windows
Federation Management
Storage Resource Broker 3.3.1Storage Resource Broker 3.3.1
Logical Name SpacesLogical Name Spaces
Storage Repository
• Storage location
• User name
• File name
• File context (creation date,…)
• Access constraints
Data Access Methods (C library, Unix, Web Browser)
Data access directly between
application and storage
repository using names
required by the local
repository
Logical Name SpacesLogical Name Spaces
Storage Repository
• Storage location
• User name
• File name
• File context (creation date,…)
• Access constraints
Data Grid
• Logical resource name space
• Logical user name space
• Logical file name space
• Logical context (metadata)
• Control/consistency constraints
Data Collection
Data Access Methods (C library, Unix, Web Browser)
Data is organized as a shared collection
Federation Between Data GridsFederation Between Data Grids
Data Grid
• Logical resource name space
• Logical user name space
• Logical file name space
• Logical context (metadata)
• Control/consistency constraints
Data Collection B
Data Access Methods (Web Browser, DSpace, OAI-PMH)
Data Grid
• Logical resource name space
• Logical user name space
• Logical file name space
• Logical context (metadata)
• Control/consistency constraints
Data Collection A
Access controls and consistency constraints on cross registration of digital entities
NARA Persistent ArchiveNARA Persistent Archive
NARA U Md SDSC
MCAT MCAT MCAT
Original data at NARA, data replicated to U Md & SDSC
Replicated copyat U Md for improvedaccess, load balancingand disaster recovery
Active archive atSDSC, user access
Demonstrate preservation environment • Authenticity• Integrity• Management of technology evolution• Mitigation of risk of data loss
• Replication of data• Federation of catalogs
• Management of preservation metadata• Scalability
• Types of data collections• Size of data collections
Federation of Three Independent Data Grids
Shared CollectionsShared Collections
• Data grids support the creation of shared collections that may be distributed across multiple institutions, sites, and storage systems.
• Digital libraries publish data, and provide services for discovery and display
• Persistent archives preserve data, managing the migration to new technology
Shared CollectionsShared Collections
• Purpose of SRB data grid is to enable the creation of a collection that is shared between academic institutions• Register digital entity into the shared collection• Assign owner, access controls• Assign descriptive, provenance metadata• Manage state information
• Audit trails, versions, replicas, backups, locks• Size, checksum, validation date, synchronization date, …
• Manage interactions with storage systems• Unix file systems, Windows file systems, tape archives, …
• Manage interactions with preferred access mechanisms• Web browser, Java, WSDL, C library, …
SRBserver
SRB agent
SRBserver
Federated Server ArchitectureFederated Server Architecture
MCAT
Read Application
SRB agent
1
2
34
6
5
Logical NameOr
Attribute Condition
1.Logical-to-Physical mapping2.Identification of Replicas3.Access & Audit Control
Peer-to-peer
Brokering
Server(s) SpawningData
Access
Parallel Data Access
R1R2
5/6
Generic InfrastructureGeneric Infrastructure
• Digital libraries now build upon data grids to manage distributed collections• DSpace digital library - MIT and Hewlitt Packard• Fedora digitial library - Cornell University and University
of Virginia
• Persistent archives build upon data grids to manage technology evolution• NARA research prototype persistent archive• California Digital Library - Digital Preservation Repository• NSF National Science Digital Library persistent archive
Southern California Earthquake CenterSouthern California Earthquake Center
SCEC Community
Library
Select Receiver (Lat/Lon)
OutputTime HistorySeismograms
Select ScenarioFault Model
Source Model
•Intuitive User Interface–Pull-Down Query Menus –Graphical Selection of Source Model–Clickable LA Basin Map (Olsen)–Seismogram/History extraction (Olsen)
•Access SCEC Digital Library
–Data stored in a data grid–Annotated by modelers–Standard naming convention–Automated extraction of selected data and metadata–Management of visualizations
SCEC Digital LibrarySCEC Digital Library
Terashake Data HandlingTerashake Data Handling
• Simulate 7.7 magnitude earthquake on San Andreas fault• 50 Terabytes in a simulation• Move 10 Terabytes per day
• Post-Processing of wave field• Movies of seismic wave propagation• Seismogram formatting for interactive on-
line analysis• Velocity magnitude• Displacement vector field• Cumulative peak maps• Statistics used in visualizations• Register derived data products into
SCEC digital library
HumidityClimateEcologicalWirelessOceanography
Wind SpeedClimateEcologicalWirelessOceanography
SeismicGeophysics
ROADNet Sensor Network Data Integration
Fire startRain start
Frank Vernon - UCSD/SIOFrank Vernon - UCSD/SIO
National Science Digital LibraryNational Science Digital Library
• URLs for educational material for all grade levels registered into repository at Cornell
• SDSC crawls the URLs, registers the web pages into a SRB data grid, builds a persistent archive• 750,000 URLs• 13 million web pages• About 3 TBs of data
Astronomy Data GridAstronomy Data Grid
• Chile• Tucson, Arizona• NCSA, Illinois
• A functioning international Data Grid for Astronomy Manchester-SDSC mirror
Moved over 400,000 imagesMoved over 400,000 images
BaBar High-energy PhysicsBaBar High-energy Physics
• Stanford Linear Accelerator
• Lyon, France• Rome, Italy• San Diego• RAL, UK
• A functioning international Data Grid for high-energy physics
Manchester-SDSC mirror
Moved over 100 TBs of dataMoved over 100 TBs of data
Worldwide University Network Data GridWorldwide University Network Data Grid
• SDSC• Manchester• Southampton• White Rose• NCSA• U. Bergen
• A functioning, general purpose international Data Grid for academic collaborations
Manchester-SDSC mirror
KEK Data GridKEK Data Grid
• Japan• Taiwan• South Korea• Australia• Poland• US
• A functioning, general purpose international Data Grid for high-energy physics
Manchester-SDSC mirror
SRB ObjectivesSRB Objectives
• Automate all aspects of data discovery, access, management, analysis, preservation• Security paramount• Distributed data
• Provide distributed data support for• Data sharing - data grids• Data publication - digital libraries• Data preservation - persistent archives• Data collections - Real time sensor data
SRB DevelopersSRB DevelopersReagan Moore Reagan Moore - PI- PIMichael Wan Michael Wan - SRB Architect- SRB ArchitectArcot Rajasekar Arcot Rajasekar - SRB Manager- SRB ManagerWayne Schroeder Wayne Schroeder - SRB Productization- SRB ProductizationCharlie CowartCharlie Cowart - inQ- inQLucas Gilbert Lucas Gilbert - Jargon- JargonBing Zhu Bing Zhu - Perl, Python, Windows- Perl, Python, WindowsAntoine de Torcy Antoine de Torcy - mySRB web browser- mySRB web browserSheau-Yen Chen Sheau-Yen Chen - SRB Administration- SRB AdministrationGeorge KremenekGeorge Kremenek - SRB Collections- SRB CollectionsArun Jagatheesan Arun Jagatheesan - Matrix workflow- Matrix workflowMarcio Faerman Marcio Faerman - SCEC Application- SCEC ApplicationSifang Lu Sifang Lu - ROADnet Application- ROADnet ApplicationRichard Marciano Richard Marciano - SALT persistent archives- SALT persistent archives
Contributors from UK e-Science, Academia Sinica, Ohio State University, Aerospace Corporation, …
75 FTE-years of support75 FTE-years of supportAbout 300,000 lines of CAbout 300,000 lines of C
HistoryHistory• 1995 - DARPA Massive Data Analysis Systems• 1997 - DARPA/USPTO Distributed Object Computation Testbed• 1998 - NSF National Partnership for Advanced Computational Infrastructure
• 1998 - DOE Accelerated Strategic Computing Initiative data grid• 1999 - NARA persistent archive• 2000 - NASA Information Power Grid• 2001 - NLM Digital Embryo digital library• 2001 - DOE Particle Physics data grid• 2001 - NSF Grid Physics Network data grid• 2001 - NSF National Virtual Observatory data grid• 2002 - NSF National Science Digital Library persistent archive• 2003 - NSF Southern California Earthquake Center digital library• 2003 - NIH Biomedical Informatics Research Network data grid• 2003 - NSF Real-time Observatories, Applications, and Data management Network
• 2004 - NSF ITR, Constraint based data systems• 2005 - LC Digital Preservation Lifecycle Management• 2005 - LC National Digital Information Infrastructure and Preservation program
DevelopmentDevelopment
• SRB 1.1.8 - December 15, 2000• Basic distributed data management system• Metadata Catalog
• SRB 2.0 - February 18, 2003• Parallel I/O support• Bulk operations
• SRB 3.0 - August 30, 2003• Federation of data grids
• SRB 3.4 - October 31, 2005• Feature requests (extensible schema)
Separation of Access Method Separation of Access Method from Storage Protocolsfrom Storage Protocols
Storage SystemStorage System
Storage ProtocolStorage Protocol
Access MethodAccess Method
Access OperationsAccess Operations
Data GridData Grid
Map from the Map from the
operations used byoperations used by
the access methodthe access method
to a standard set ofto a standard set of
operations used to operations used to
interact with theinteract with the
storage systemstorage system
Storage OperationsStorage Operations
Data Grid OperationsData Grid Operations• File access
• Open, close, read, write, seek, stat, synch, …• Audit, versions, pinning, checksums, synchronize, …• Parallel I/O and firewall interactions• Versions, backups, replicas
• Latency management• Bulk operations
• Register, load, unload, delete, …
• Remote procedures• HDFv5, data filtering, file parsing, replicate, aggregate
• Metadata management• SQL generation, schema extension, XML import and export,
browsing, queries, • GGF, “Operations for Access, Management, and Transport at Remote
Sites”
Examples of ExtensibilityExamples of Extensibility• Storage Repository Driver evolution
• Initially supported Unix file system• Added archival access - UniTree, HPSS• Added FTP/HTTP• Added database blob access• Added database table interface• Added Windows file system• Added project archives - Dcache, Castor, ADS• Added Object Ring Buffer, Datascope• Added GridFTP version 3.3
• Database management evolution• Postgres• DB2• Oracle• Informix• Sybase• mySQL (most difficult port - no locks, no views, limited SQL)
Examples of ExtensibilityExamples of Extensibility• The 3 fundamental APIs are C library, shell commands,
Java• Other access mechanisms are ported on top of these interfaces
• API evolution• Initial access through C library, Unix shell command• Added iNQ Windows browser (C++ library)• Added mySRB Web browser (C library and shell commands)• Added Java (Jargon)• Added Perl/Python load libraries (shell command)• Added WSDL (Java)• Added OAI-PMH, OpenDAP, DSpace digital library (Java)• Added Kepler actors for dataflow access (Java)• Added GridFTP version 3.3 (C library)• Added Fedora
Types of Risk Types of Risk
• Media failure• Replicate data onto multiple media
• Vendor specific systemic errors• Replicate data onto multiple vendor products
• Operational error• Replicate data onto a second administrative domain
• Natural disaster• Replicate data to a geographically remote site
• Malicious user• Replicate data to a deep archive
How Many ReplicasHow Many Replicas
• Three sites minimize risk• Primary site
• Supports interactive user access to data
• Secondary site• Supports interactive user access when first site is
down• Provides 2nd media copy, located at a remote site,
uses different vendor product, independent administrative procedures
• Deep archive• Provides 3rd media copy, staging environment for
data ingestion, no user access
Deep ArchiveDeep Archive
Z2Z2 Z1Z1Z3Z3
Z2:D2:U2Z2:D2:U2
RegisterRegister
Z3:D3:U3Z3:D3:U3
RegisterRegister
PullPull PullPull
FirewallFirewall
Server initiated I/OServer initiated I/O
DeepDeep
ArchiveArchive
StagingStaging
ZoneZone
Remote ZoneRemote Zone
No access byNo access by
Remote zonesRemote zones
PVNPVN
iRODS DesigniRODS Design
Intelligent Rule-Oriented Data System
• Traditional shared collection• Metadata catalog manages state information
• Add rule engine• Allow dynamic rule changes
• Track version of rule, date version was applied and the level of granularity (item, sub-collection)
• Manage state information needed for rule execution
• Manage state information resulting from rule application
• Validate consistency of rules after a change• Validate order of application of rules
• Deadlock detection
Rule EngineRule Engine
• Rule is an Event-Condition-Action• An action can be another rule or a micro-service
or a set of rules and micro-services
• Rules are organized • Named rule sets• Separate list for each rule type
• Rules incorporate:• $ - temporary state information• Recovery procedure for each action
Rule Engine Design IssuesRule Engine Design Issues
• What are the types of rules?• Deferred rules (consistency assertions)• Periodic rules (maintenance assertions)• Atomic rules (micro-service composition)• Dynamic rule definition (state information)
• Rule types also characterized as:• Consistency rules• Procedural rules• Periodic consistency rules• Event triggered consistency rules• Scheduled versus delayed versus triggered
ImplementationImplementation
• Open-source software• Requires re-write of SRB software system• 30,000 lines of “C” written to date
• Rebuilt:• Data management micro-services for remote data
manipulation (M. Wan)• Metadata catalog management system (W. Schroeder)• Generic client API generation (standardized options)
• Adding• Rule engine (A. Rajasekar)• Rule consistency checking (UCSD CSE)
iRODS - intelligent Rule-Oriented Data iRODS - intelligent Rule-Oriented Data SystemSystem
Resources
Client Interface Admin Interface
MetadataModifierModule
ConfigModifierModule
RuleModifierModule
ConsistencyCheck
Module
Confs
RuleBase
Meta DataBase
Engine
Rule
Current State
Rule Invoker
MicroService
Modules
Resource-based Services
MicroService
Modules
Metadata-based Services
ServiceManager
ConsistencyCheck
Module
ConsistencyCheck
Module
Rule ListRule List 0 ON register_data
IF $objPath like /home/collections.nvo/2mass/fits-images/*DO cut [nop]AND check_data_type(fits image) [nop]AND get_resource(nvo-image-resource) [nop]AND registerData [recover_registerData]AND addACLForDataToUser(2massusers.nvo,write)
[recover_addACLForDataToUser]
AND extractMetadataForFitsImage [recover_extractMetadataForFitsImage]
1 ON register_dataIF $objPath like /home/collections.nvo/2mass/*DO get_resource(2mass-other-resource) [nop]AND registerData [recover_registerData]AND addACLForDataToUser(2massusers.nvo,write)
[recover_addACLForDataToUser]
2 ON register_dataDO get_resource(null) [nop]AND registerData [recover_registerData]
DemonstrationDemonstration
• URL • https://srb.npaci.edu/iRods/ruleAdmin.html• https://srb.npaci.edu/iRods/ruleAdminRaja.html
• Rule example:
0 ON register_data
IF $dataType like *image*DO get_resource(null) [nop]AND registerData
[recover_registerData]AND extract_metadata_for_image [nop]
For More InformationFor More Information
Reagan W. Moore
San Diego Supercomputer Center
http://www.sdsc.edu/srb/