26
ATLAS Rucio DB characteristics ( based on the Oracle RDBMS ) Rucio community workshop, Oslo, Feb - March 2019 Gancho Dimitrov (CERN)

ATLAS Rucio DB characteristics

  • Upload
    others

  • View
    3

  • Download
    0

Embed Size (px)

Citation preview

Page 1: ATLAS Rucio DB characteristics

ATLASRucioDBcharacteristics(basedontheOracleRDBMS)

Ruciocommunityworkshop,Oslo,Feb-March2019GanchoDimitrov(CERN)

Page 2: ATLAS Rucio DB characteristics

Aboutme

1

• MemberoftheATLASdatabasegroupsince2006

• Acquired(some)DBrelatedknowledgethroughoutthelast15+years

• Mainfocuson:Ø datamanagementØ databaseschemadesignØ databaseperformancetuning

• CertifiedinOracleRDBMS

• RegularspeakeratOracleUserGroupcommunityconferencesinUK,BulgariaandGermany

RucioCommunityWorkshop,Oslo,Feb2019

Page 3: ATLAS Rucio DB characteristics

Rucioarchitecture

2RucioCommunityWorkshop,Oslo,Feb2019

Page 4: ATLAS Rucio DB characteristics
Page 5: ATLAS Rucio DB characteristics

Ruciodatabaseschema

4

Designedin2013

RucioCommunityWorkshop,Oslo,Feb2019

Page 6: ATLAS Rucio DB characteristics

Ruciodatabaseschema

5

61tables

RucioCommunityWorkshop,Oslo,Feb2019

Page 7: ATLAS Rucio DB characteristics

Ruciodatabaseschema

6

74indices

RucioCommunityWorkshop,Oslo,Feb2019

Page 8: ATLAS Rucio DB characteristics

Ruciodatabaseschema

7

5triggers

RucioCommunityWorkshop,Oslo,Feb2019

Page 9: ATLAS Rucio DB characteristics

Ruciodatabaseschema

8

3functions

RucioCommunityWorkshop,Oslo,Feb2019

Page 10: ATLAS Rucio DB characteristics

Ruciodatabaseschema

9

12procedures

RucioCommunityWorkshop,Oslo,Feb2019

Page 11: ATLAS Rucio DB characteristics

Ruciodatabaseschema

10

17jobs

RucioCommunityWorkshop,Oslo,Feb2019

Page 12: ATLAS Rucio DB characteristics

Tablesegments

11

Physicallyorganised basedonRucio’s scopeunit

scopedata=tablepartition**separatesegment(providesflexibilityinsetofoperations)

RucioCommunityWorkshop,Oslo,Feb2019

Page 13: ATLAS Rucio DB characteristics

Rucioscopegeneration

12

In2014:5140scopes(systemstartup)

2015-2018=~400peryear

RucioCommunityWorkshop,Oslo,Feb2019

Page 14: ATLAS Rucio DB characteristics

Currently

13

About7000scopesaremappedto34500table

partitions

RucioCommunityWorkshop,Oslo,Feb2019

Page 15: ATLAS Rucio DB characteristics

DIDS(DataIDentifierS)*

14

1014million(AVGrowsize:220bytes)

*Theprimaryaddressableunitofdata(files,datasets,containers)

RucioCommunityWorkshop,Oslo,Feb2019

Page 16: ATLAS Rucio DB characteristics

DIDSreplicasontheGrid

15

1106million(AVGrowsize:150bytes)

RucioCommunityWorkshop,Oslo,Feb2019

Page 17: ATLAS Rucio DB characteristics

Rucio<-->DBinteraction

16

SQLcommandsgeneratedby

SQLAlchemy ORM*

*Object-RelationalMapperRucioCommunityWorkshop,Oslo,Feb2019

Page 18: ATLAS Rucio DB characteristics

DBsessions

17

1100-1400DBsessions15-25activesessions

RucioCommunityWorkshop,Oslo,Feb2019

Page 19: ATLAS Rucio DB characteristics

Rucioinsert/update/deleterate

18

~2-4Kexec/sec

RucioCommunityWorkshop,Oslo,Feb2019

Page 20: ATLAS Rucio DB characteristics

RucioDBqueriesrate

19

~4-8Kexec/sec2Mblockreads(15GB)/sec

RucioCommunityWorkshop,Oslo,Feb2019

Page 21: ATLAS Rucio DB characteristics

DBserverCPUusage

20

~30-40%

RucioCommunityWorkshop,Oslo,Feb2019

Page 22: ATLAS Rucio DB characteristics

RucioDBvolumegrowth2016-2019

21RucioCommunityWorkshop,Oslo,Feb2019

Page 23: ATLAS Rucio DB characteristics

April-June2019

22

Excitingperiod:

TestandvalidationonOracle18c

DBserverupgradefrom11gto18c

RucioCommunityWorkshop,Oslo,Feb2019

Page 24: ATLAS Rucio DB characteristics

Spring2020

23

ExpectedtohavenewDBhardware

RucioCommunityWorkshop,Oslo,Feb2019

Page 25: ATLAS Rucio DB characteristics

RucioDBschemaextension

24

EnrichRucioDIDSwithmetadatastoredinJSON*

*CERNcomputingseminaronDBbuilt-inJSONfeaturesplannedfor15th May2019

RucioCommunityWorkshop,Oslo,Feb2019

Page 26: ATLAS Rucio DB characteristics

Take-homemessages• RucioDBsetuponOracleRDBMSkeepsonperformingwellseveralconsecutiveyears.•WorkforsmoothDBserverupgradeisnecessary• PlannedextensionofthecurrentDBschematosupportDIDSmetadatainJSON

SpecialthankstotheRucioteamforthecollaborativeworkintheyears.

WelltunedDB=goodnight’ssleepforall!25RucioCommunityWorkshop,Oslo,Feb2019