Grid Technology
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
DBCFCFGT
dpm-xrootd v3
Creating Federated Data Stores for the LHC David Smith, on behalf of IT-GT, CERN
14 Sep 2012
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT Introduction
• An existing dpm-xrootd written 2006– A pair of plugins for OFS, XMI– Only ALICE token based access– Some performance issues with dispatch of
requests to dpm
dpm-xrootd - 2
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT Introduction dpm-xrootd v3
Aim to:•Provide xroot file I/O for DPM for all VOs•Allow participation in xrootd federations•Allow ALICE token based access•Other VOs to use GSI for authentication
– Support VOMS extensions– Authorization is done by DPM system use of an
identity (user, [/vo,/vo/group,…]).
dpm-xrootd - 3
Grid Technology Reminder of DPM
dpm-xrootd - 4
/vo
/dpm
/domain
/home
DPMhead node file
(uid, gid1, …)
DPMdisk servers
DPM Name Server– Namespace– Authorization– Physical files location
Disk Servers– Physical files
Direct data transfer from/to disk server
External transfers via gridFTP
CLI, C API, SRM-enabled
client, etc. data transfer
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT Relevant features of DPM
• DPM has a central service– A get request for a SURL
(/dpm/example.com/home/dteam/file) gives an SFN (disk001.example.com:/data1/dteam/2012-09-14/file.1353.0) to read
– A put request for a SURL returns an SFN to write to
– Put concluded with put_done
– Authorization check during get or put
– Files can not be modified once written
– DPM instance is often run for multiple VOs
• Disk servers providing I/O– Clients have connectivity to disk servers
dpm-xrootd - 5
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT dpm-xrootd
Now:•dpm-xrootd v3 is a set of plugins to xrootd server, using plugin interfaces (as for previous version)•But different interfaces:
– Use XrdOss, XrdCmsClient and XrdAccAuthorize plugins
•VOMS extraction disabled for now
dpm-xrootd - 6
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT
dpm-xrootd - 7
Example: file access
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT dpm-xrootd basic file access
dpm-xrootd - 8
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT dpm-xrootd and deployment
dpm-xrootd - 9
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xroot client: openxroot://dpm.example.com//dpm/example.com/dteam/file
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT dpm-xrootd and deployment
dpm-xrootd - 10
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xroot client: openxroot://dpm.example.com//dpm/example.com/dteam/file
dpm_put or dpm_get
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT dpm-xrootd and deployment
dpm-xrootd - 11
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xroot client: openxroot://dpm.example.com//dpm/example.com/dteam/file
Redirect: with host/port and opaque info&dpm.sfn=/data1/dteam/2012-09-14/file.1353.0
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT dpm-xrootd and deployment
dpm-xrootd - 12
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xroot client: openxroot://dpm.example.com//dpm/example.com/dteam/file
Open: with original filename in the request and opaque information
disk001.example.com
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT dpm-xrootd and deployment
dpm-xrootd - 13
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xroot client: closexroot://dpm.example.com//dpm/example.com/dteam/file
Open: with original filename in the request and opaque information
disk001.example.com
dpm_putdone (for put only)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT Redirection to disk server
• Redirection xrootd interacts with DPM– Configured to offer XrdSecunix and XrdSecgsi to
the xroot client– If a valid ALICE token is present in the opaque
data from the client the identity is a preset. (Check if authz or signature in opaque data)
– Otherwise an identity must be derivable from the XrdSecEntity authentication data (at disk too)
– Goes through get or put sequence with DPM system (currently using classic dpns/dpm api)
– May need to return a wait time to the client if the get or put sequence takes more than ~1 second
dpm-xrootd - 14
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT Redirection to disk server II
• If get or put is successful, redirect is returned directing to the target disk server.
• Opaque data is added:
dpm-xrootd - 15
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT Disk access keyed hash
• Disk servers require keyed hash (with limited duration) in opaque data
• Use HMAC-SHA256-128 for keyed hash
• Key stored in disk and memory of xrootd processes (required to be 32 to 64 bytes)
dpm-xrootd - 16
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT At the disk server
• Hash and dpm.dhost checked• dpm.time must be within validity window
(300 seconds default) of current time• Access mode of request must be consistent
with dpm.put• Then disk I/O via native XrdOss• ofs.persist auto hold 0 is set, to remove files
not closed at the end of writing
dpm-xrootd - 17
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT At the disk server II
• For Put– On success call dpm_putdone with dpm token
and surl.– On fail (no close) call dpm_abortfiles with dpm
token and surl.
• For Get– No interactions with central DPM
dpm-xrootd - 18
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT dpm-xrootd and federation
• With the basic xroot file access vo/users could devise federations.. but
• Want to provide integrated method (as for a native xroot site)
• Next: a couple of example sequences
dpm-xrootd - 19
FEDERATION of XROOT access
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT dpm-xrootd federating for VO
dpm-xrootd - 20
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT dpm-xrootd federating for VO
dpm-xrootd - 21
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xrootd:11000(fedredir_vo)
cmsd(fedredir_vo)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT dpm-xrootd federating for VO
dpm-xrootd - 22
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xrootd:11000(fedredir_vo)
cmsd(fedredir_vo)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT dpm-xrootd federating for VO
dpm-xrootd - 23
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xrootd:11000(fedredir_vo)
cmsd(fedredir_vo)
regional redirector
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT federation: client example 1
dpm-xrootd - 24
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xrootd:11000(fedredir_vo)
cmsd(fedredir_vo)
regional redirector
xroot client: openxroot://regional.example.org//vo/example.dat
/vo/example.dat ?
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT federation: client example 1
dpm-xrootd - 25
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xrootd:11000(fedredir_vo)
cmsd(fedredir_vo)
regional redirector
xroot client: openxroot://regional.example.org//vo/example.dat
/vo/example.dat ?
XrdPss: stat /vo/example.dat(specially trusted at xrootd)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT federation: client example 1
dpm-xrootd - 26
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xrootd:11000(fedredir_vo)
cmsd(fedredir_vo)
regional redirector
xroot client: openxroot://regional.example.org//vo/example.dat
/vo/example.dat ?
dpns_stat
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT federation: client example 1
dpm-xrootd - 27
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xrootd:11000(fedredir_vo)
cmsd(fedredir_vo)
regional redirector
xroot client: openxroot://regional.example.org//vo/example.dat
...if site has example.datclient may be directed there
Redirect to dpm.example.com:11000
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT federation: client example 1
dpm-xrootd - 28
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xrootd:11000(fedredir_vo)
cmsd(fedredir_vo)
regional redirector
xroot client: openxroot://regional.example.org//vo/example.dat
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT federation: client example 1
dpm-xrootd - 29
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xrootd:11000(fedredir_vo)
cmsd(fedredir_vo)
regional redirector
xroot client: openxroot://regional.example.org//vo/example.dat
Not Involved
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT
dpm-xrootd - 30
Client example 2
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT federation: client example 2
dpm-xrootd - 31
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xrootd:11000(fedredir_vo)
cmsd(fedredir_vo)
regional redirector
xroot client: openxroot://dpm.example.com//vo/example2.dat
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT federation: client example 2
dpm-xrootd - 32
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xrootd:11000(fedredir_vo)
cmsd(fedredir_vo)
regional redirector
xroot client: openxroot://dpm.example.com//vo/example2.dat
Redirect: to dpm.example.com:11000
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT federation: client example 2
dpm-xrootd - 33
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xrootd:11000(fedredir_vo)
cmsd(fedredir_vo)
regional redirector
xroot client: openxroot://dpm.example.com//vo/example2.dat
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT federation: client example 2
dpm-xrootd - 34
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xrootd:11000(fedredir_vo)
cmsd(fedredir_vo)
regional redirector
xroot client: openxroot://dpm.example.com//vo/example2.dat
In case of no file:Redirect to regional.example.orgdpm_get
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT
dpm-xrootd - 35
Few other details
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT
xrootd:1095(disk server)
Name2name libs
dpm-xrootd - 36
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xrootd:11000(fedredir_vo)
cmsd(fedredir_vo)
• May use a standard name2name lib• But loaded by our cmslib not OSS• Calls lfn2pfn method, e.g. for global
namespace to surl
xrootd:11001(fedredir_vo2)
cmsd(fedredir_vo2)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT Plugins provided
• xrootd redirector– libXrdDPMFinder (XrdCmsClient)– libXrdDPMOss (XrdOss)– libXrdDPMRedirAcc (XrdAccAuthorize)
• xrootd disk– libXrdDPMDiskAcc (XrdAccAuthorize)– libXrdDPMOss
dpm-xrootd - 37
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT Options
dpm-xrootd - 38
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT Options: dpm
dpm-xrootd - 39
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT Options: alice token handling
dpm-xrootd - 40
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT Options: name translation
dpm-xrootd - 41
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT Status summary
• Available via dedicated repository– including xrootd packages (EPEL packaging)
• Expect xrootd 3.2.x in the EMI third party repo first– Eventually recent version of xrootd to go into
EPEL
• dpm-xrootd setup instructions on the web– YAIM module written for yaim generation of the
config files
• Deployments at four sites– May be others
dpm-xrootd - 42
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT To do
• Eventually use dmlite rather than classic dpm/dpns api for dpm service access.
• VOMS• dpm-xootd uses few xrootd interfaces for
which ABI compatibility across minor releases isn’t guaranteed. Remove when possible.
• Monitoring– dpm site probably multi-vo
dpm-xrootd - 43
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT Monitoring sources
dpm-xrootd - 44
xrootd:1095(disk server)
xrootd:1095(disk server)
xrootd(redirector)
DPM service
xrootd:11000(fedredir_vo)
cmsd(fedredir_vo)
xrootd:11001(fedredir_vo2)
cmsd(fedredir_vo2)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
GT Contacts
• DPM Support: [email protected]
• dpm-xrootd setup wiki:– https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm/Xroo
t/Setup
dpm-xrootd - 45