A. Forti A. McNab

Cluster distributed dynamic storage

BACKGROUND

• Manchester University 1000 nodes new cluster (2x1000 processors)

• 2x250 GB disks equivalent to ~400 TB of available disk space excluding the space reserved for OS and scratch directories.

• There is no tape storage behind.• Cluster internal connectivity is 1 Gb/s and

is connected to the external world by the 2.5 Gb/s university production network. It will be replaced by a 10Gb/s dedicated link.

• High rate data transfers from Tier1 is possible possible. 350 Mb/s achieved on the university production network.

• The aim is to use the space as a cache.

CONFIGURATION

• dcache has been chosen as software• There will be 2 types of nodes: external and

internal– External nodes will be front end nodes

with an active gridftp door for transfer from the outside world. They’ll have a dedicated cpu to handle the transfers.

– Internal will accessible only from other nodes

• On each node:– 2 dcache partitions 1 on each disk– 2 pools, 1 for each partition– /pnfs mounted– Gsidcapdoor active

• No raid arrays will be created it spoils the independence of the discs. In this way

– Disk 2 can be assigned to a particular VO– Disk 1 can be used for resilience. – Disk 1 can be wiped out during installation

if needed– Disk 2 can be left untouched.

• Data will have 2 copies on the system using resilience.

Single external node internals

gridftpDoor

Externalworld

Disk 2250GB

Disk 1150GB

pool

OS/Home100GB

gsidcapDoor

Pnfs

pool

Headnode

PnfsSingle external node internals

gridftpDoor

Externalworld

Disk 2250GB

Disk 1150GB

pool

OS/Home100GB

gsidcapDoor

Pnfs

pool

Disk 2250GB

Disk 1150GB

pool

OS/Home100GB

gsidcapDoor

Pnfs

pool

Headnode

Pnfs

Internal nodes access.

Disk 2250GB

Disk 1150GB

pool

OS/Home100GB

gsidcapDoor

Pnfs

pool

Headnode

Disk 2250GB

Disk 1150GB

pool

OS/Home100GB

gsidcapDoor

Pnfs

pool

applicationapplication

application

Internal nodes access.

Disk 2250GB

Disk 1150GB

pool

OS/Home100GB

gsidcapDoor

Pnfs

pool

Disk 2250GB

Disk 1150GB

pool

OS/Home100GB

gsidcapDoor

Pnfs

pool

Headnode

Disk 2250GB

Disk 1150GB

pool

OS/Home100GB

gsidcapDoor

Pnfs

pool

Disk 2250GB

Disk 1150GB

pool

OS/Home100GB

gsidcapDoor

Pnfs

pool

applicationapplication

application

APPLICATIONS

• Applications will access the data through /pnfs or gsidap doors

– Gsidcap doors might be turned off in the future

OTHER SOLUTIONS

• Solution based on http technology is being developed in Manchester:

• Uses HTTP or HTTPS for data transfer– Access control via XML policies and

X.509 or VOMS certificates– File location using Hypertext Cache

Protocol and multicast queries– Avoids need for central database, by

enforcing atomic file operations and using status of files on disk to respond to queries

• Experiments using root have asked to support Xrootd as a possible alternative solution

– Plain root APIs to access dcache are also being evaluated. This turns out useful in the final stages of analysis when the experiment framework becomes irrelevant.