INFN regional cache federation - Indico...Ansible K8s (soon also in helm) Mesos/Marathon 6 D. Ciangottini - DOMA Access - INFN cache federation for CMS Testing with CMS workflows Real

D. Ciangottini - DOMA Access - INFN cache federation for CMS

INFN regional cache federation for CMS workflows

DOMA Access Meeting4/12/2018

Diego Ciangottinion behalf of CMS Collaboration and INFN-Cache team

1


Outline

● Introduction○ cache on ephemeral storage for opportunistic resources○ INFN geo-distributed cache with unmanaged storage

● Tests and results● Ongoing activities● Plans

2


In synergy with the activity on smart caches for eXtreme DataCloud

Introduction

● Integration and functionality scouting of XCache @INFN for CMS workflows● Two main scenarios of application:

○ Automatic procedures for the creation of an XCache cluster on-demand■ ready-to-use recipe for both bare metal and cloud environment/orchestration

● complete high level infrastructure description → QoS configuration■ mitigate latency for dynamic resources (cloud and HPC)

○ Creation of an INFN (regional) cache federation■ enabling “storage-less” T2s and efficient scale out over opportunistic resources ■ propaedeutic to distributed cache layer for data-lake environment

3


XCache for dynamic resource provisioning

Computing resources are opportunistically deployed on cloud/HPC resources

● storage not necessarily available○ remote read latency○ I/O inefficient

The cache introduction may offer:● ephemeral storage for hot data near the

computing provider● optimized wan access, only for data not

already on the cache

Cache resources

HPC

Scenario 1

4


INFN regional XCache layer

Few world-wide custodial centers with data replica managed by the experiment

● Computing Tiers access data directly from closest custodial center

Using cache for a Content Delivery Network approach:

● geo-distributed network of unmanaged storages

● common namespace (no data replication)● request mitigation to custodial sites

Custodial data

Regional cache

HPC

Tier2

Tier3

Scenario 2

5


Cache for opportunistic resources

Cache Redirector

Disk proxy cache

Disk proxy cache

Sto

rage

fede

ratio

n

xrootd proxy cache

RAM Disk

WN

STORAGE

STORAGE

STORAGE

STORAGE

Opportunistic resources

Remote CMS AAA Federation

In case of computing on opportunistic resources the remote data access pattern can be improved providing:

● an on-demand cache layer near cpu resources (same cloud provider) STORAGE

Scenario 1

Automated deployment through:● Ansible● K8s (soon also in helm)● Mesos/Marathon

6


Testing with CMS workflows

● Real CMS analysis workflows on cloud resources (2 volunteer users)

○ 2k jobs @OpenTelekomCloud (OTC)○ ~150k of users jobs completed reading from

standalone cache cluster deployed at OTC

● DODAS (*) have been used for:■ same configuration for setup on different

cloud providers ■ automated deployment through:

● Ansible for infrastructure● K8s or Mesos/Marathon for container

orchestration

Cloud resource provider

Opportunistic CMS startd Service

Opportunistic Cache Service

Redirector

XcacheXcache Xcache

WNWN WN

Opportunistic Storage Service

Ceph/HDFS/IOVolumes/?

WLCG XRootD

Federation

(*) https://dodas-ts.github.io/dodas-doc/Scenario 1

7

https://dodas-ts.github.io/dodas-doc/

D. Ciangottini - DOMA Access - INFN cache federation for CMS 8

Cache performance: Net Out - Net In

2.8Gbps outbound network

2.8Gbps network inbound

8

Cache Network IN

Not negligible gain with significant spike of direct cache disk/memory usage

Cache Network OUT


Success rates

9

Two categories of tasks by input data latency:

● 200-300ms latency links

● <100ms latency links

~100% failed for remote reading at first try (cold cache)~80% succeeded at second CRAB retry (warm cache)

No significant discrepancy w.r.t. results with standard CRAB jobs


CPU efficiency overhead

10

No cache overhead observed

Local read reference

Cache hit - Avg CPU efficiency

No cache overhead observed● local-like performances when a cache hit occurs○ No cache overhead observed

with this workflow samples○ slight improvement due to good

performances of the low latency storage flavor provided


Distributed testbed deployment

● Deployment a geo-distributed cache:○ Clients contact the cache redirector○ Redirector steers client to

■ the cache that actually have file on disk■ If no cache has the requested file, a round

robin selection of cache server is used

● Network of unmanaged storages for hot data

XCacheT2_IT_Bari

XCacheCNAF

WLCG XRootD

Federation

Clients

Cache redirector

Scenario 2

11

XCacheT2_IT_Legnaro


Distributed testbed deployment: testbed

Current functional test setup:

● CNAF XCache redirector federating 3 servers:○ CNAF XCache server (5TB)○ T2 Bari XCache server (10TB)○ T2 Legnaro XCache server (22TB)

● Redirecting part of the CMS analysis workflows to contact National redirector○ based on dataset name requested

● Tier2 at Pisa is joining the testbed

12

Scenario 2


INFN XCache: functional checks 1/2

● Test tasks submitted to T2_IT_Bari with empty cache● Comparing jobs running at Bari (pointing to cache) with “Ignore locality” ones on

other sites

Bari → Cache

Pisa → No-Cache

Legnaro → No-Cache

Avg Job CPU Eff.

No penalty in CPU eff in case of empty cache

Performances of jobs reading from empty cache is comparable with remote reading.

Scenario 2 LOW LATENCY LINKS


INFN XCache: functional checks 2/2

14

Analysis task with XCache @INFN

Input dataset ZeroBias-RunB2017/AODSIM - task with 300 jobsStarting from empty cache vs run with remote read on T2_IT_*

Avg CPU eff

80%60%

35%Same task with remote read

Latency hiding + same file reused in different jobs

Scenario 2 HIGH LATENCY LINKS


Work in progress: cache simulation

● Starting from regional CMS job statistics of the latest 2-3 months○ time series of CMS jobs @ItalianTiers2○ LRU cache eviction simulation

● Regional cache effect evaluation varying:○ cache size○ Datasets to be cached

● Metrics:○ hit rate vs cache size○ cpu hrs miss vs hit per cache size

Base ground for decision on contributions on the optimization of cache algorithms and cache population (e.g. different cache algos w.r.t. LRU and inter cluster load balancing - round robin so far-)

15

N.B. at a beginning phase: hoping for inputs and coordination to avoid work duplication


Work in progress: AI investigation

● starting from data-popularity ML studies already performed (CMS, ATLAS, ALICE)● tuning to adapt in the context of regional caches

○ access time information○ per region studies

● two main algorithms subgroups:○ put data ahead → simulate smart data fetching effect○ optimize what stays on disk → simulate smart eviction effect

Possible evolution to studies on client-cache routing based on real-time metrics (networks, recent failures etc..)

16

N.B. at a beginning phase: hoping for inputs and coordination to avoid work duplication

Activity synergy with a PhD thesis in Perugia


Conclusions

1717

● Working on 2 scenarios: ○ cache for dynamic resources○ regional cache layer for a future data-lake model

● Performance evaluation motivates further activities○ on-demand deployment ○ latency hiding and local-like performances on hits

● Work in progress:

Production-like regional deployment

Cache simulation with historical dataAI-based caches management

External decision agents interface


BACKUP

18

Documents

INFN regional cache federation - Indico...Ansible K8s (soon also in helm) Mesos/Marathon 6 D. Ciangottini - DOMA Access - INFN cache federation for CMS Testing with CMS workflows Real