Embed Size (px)
Citation preview
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 134
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 234
2
Randy Johnson
bull Principal Consultant Enkitec LP
bull 20 or so years in the IT industry
bull Began working with Oracle RDBMS in 1992 at the launch of Oracle 7
bull Main areas of interest are HA Backup amp Recovery
bull Working exclusively with Exadata for over a year now
bull Numerous Exadata implementations
bull Numerous POCrsquos amp workshops
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 334
3
1 What is Exadata
2 What Is ASM
3 Storage Cell Disks
4 Creating and Allocating Storage at the Cell
5 Demo ndash Creating amp Presenting Grid Disks to ASM
6 QampA
983105983143983141983150983140983137
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 434
4
Database Servers
2 - 8 Servers per rack
bull 24 - 96 CPU Cores
bull 96 - 768 Gigabytes Memory
Storage Cells
3 - 14 Storage Cells per rack
72 - 336 CPU Cores
72 - 336 Gigabytes Memory12 - 53 Terabytes Flash Cache
72 - 336 Terabytes Raw Disk Storage
Infiniband Switches
2 Per rack + 1 spine switch for expansion
40Gbs - 5 gigabytes per database server storage cell
Expandable up to 8 racks - more with additional spine switches
983127983144983137983156 983113983155 983109983160983137983140983137983156983137983103
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 534
`
983105 983108983137983156983137 983107983141983150983156983141983154 983113983150 983105 983107983137983150
Database Servers
bull Sun Fire 5670 1U
bull 2 Hex-Core CPUrsquos 293GHz
bull 96G Memory
bull OEL5 64 bit
Infiniband Network
bull Redundant 40Gb Switches
bull Unified Internal Networkbull Used by both DB and Storage
bull Can connect externally
StorageCell Servers
bull SunFire X4270 2Ubull 2 Hex-core CPUrsquos 293GHz
bull 24G Memory
bull 384GB Flash Cache
bull OEL5 64 bit
bull 12 Drives per Storage Server
bull 2TB SAS - High Cap
bull 600GB SAS - High Perf
bull 54GB Sec Transfer Rate
5
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 634
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 734
983112983151983159 983140983151 983156983144983141 983107983151983149983152983151983150983141983150983156983155 983127983151983154983147983103
7
Exadata Storage ServersExadata Database Servers
IO Intensive Processing
bull Index Table Scansbull HCC Uncompress
bull Functions
bull Primitive Joins
bull Predicate Filtering
bull Column Projectionsbull Storage Index Processing
Compute Intensive Processing
bull Complex Joins
bull Aggregation
bull Functions
bull Data Conversion
bull IO Over Infiniband Using iDB Protocol
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 934
9
983123983105983117983109
ASM is the Implementation of SAME
The goal of SAME is to optimize storage IO utilizationbull Bandwidth
bull Sequential and random access
bull Workload types
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1034
10
983113983149983152983148983141983149983141983150983156983145983150983143 983108983137983156983137983138983137983155983141 983123983156983151983154983137983143983141 983127983145983156983144983151983157983156 983105983123983117
Design DB layout strategy
bull How many file systems data files
bull Where data files reside
Inventory of files
bull 2 control files 2 log files multiple datafiles per tablespace
backup files temp files etchellip
bull 100rsquos or 1000rsquos of files to create name manage and not make
mistakes
bull Multiplied by n number of Databases
What about tuning expandingcontracting your DB
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1134
11
983110983145983148983141 983116983137983161983151983157983156983155 983111983141983156983156983145983150983143 983137 983116983145983156983156983148983141 983107983151983149983152983148983145983139983137983156983141983140983103983103
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1234
12
Current data is what end users are usually interested in So as space is
added IO tends to follow the new disks creating hot spots
983120983154983151983158983145983155983145983151983150983145983150983143 983159983145983156983144 983110983145983148983141 983123983161983155983156983141983149983155
u01marketing_01dbf
u01marketing_02dbf
hellip
u01marketing_03dbf u01marketing_04dbf
u01
u02marketing_05dbf
u02marketing_06dbf
hellip
u02marketing_07dbf u02marketing_08dbf
u02 u03
2009 2010 2011
Most Recently
Added Storage
u03marketing_09dbf
u03marketing_10dbf
hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1434
14
983108983137983156983137 983122983141983140983157983150983140983137983150983139983161
bull 983105983123983117 983152983154983151983158983145983140983141983155 983154983141983140983157983150983140983137983150983139983161 983157983155983145983150983143 983142983137983145983148983157983154983141 983143983154983151983157983152983155
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983137983154983141 983157983155983141983154 983140983141983142983145983150983141983140 983157983150983145983156983155 983151983142 983142983137983145983148983157983154983141
ndash 983123983145983150983143983157983148983137983154 983140983145983155983147 ndash 983111983154983151983157983152 983151983142 983140983145983155983147983155 (983123983105983118 983151983154 983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983141983148983148)
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983141983150983155983157983154983141 983149983145983154983154983151983154 983139983151983152983145983141983155 983151983142 983140983137983156983137 983137983154983141 983150983151983156
983155983156983151983154983141983140 983151983150 983156983144983141 983155983137983149983141 983140983145983155983147983155 (983151983154 983143983154983151983157983152983155 983151983142 983140983145983155983147983155)
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1534
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
15
How Exadata Storage Server Redundancy Works
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 234
2
Randy Johnson
bull Principal Consultant Enkitec LP
bull 20 or so years in the IT industry
bull Began working with Oracle RDBMS in 1992 at the launch of Oracle 7
bull Main areas of interest are HA Backup amp Recovery
bull Working exclusively with Exadata for over a year now
bull Numerous Exadata implementations
bull Numerous POCrsquos amp workshops
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 334
3
1 What is Exadata
2 What Is ASM
3 Storage Cell Disks
4 Creating and Allocating Storage at the Cell
5 Demo ndash Creating amp Presenting Grid Disks to ASM
6 QampA
983105983143983141983150983140983137
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 434
4
Database Servers
2 - 8 Servers per rack
bull 24 - 96 CPU Cores
bull 96 - 768 Gigabytes Memory
Storage Cells
3 - 14 Storage Cells per rack
72 - 336 CPU Cores
72 - 336 Gigabytes Memory12 - 53 Terabytes Flash Cache
72 - 336 Terabytes Raw Disk Storage
Infiniband Switches
2 Per rack + 1 spine switch for expansion
40Gbs - 5 gigabytes per database server storage cell
Expandable up to 8 racks - more with additional spine switches
983127983144983137983156 983113983155 983109983160983137983140983137983156983137983103
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 534
`
983105 983108983137983156983137 983107983141983150983156983141983154 983113983150 983105 983107983137983150
Database Servers
bull Sun Fire 5670 1U
bull 2 Hex-Core CPUrsquos 293GHz
bull 96G Memory
bull OEL5 64 bit
Infiniband Network
bull Redundant 40Gb Switches
bull Unified Internal Networkbull Used by both DB and Storage
bull Can connect externally
StorageCell Servers
bull SunFire X4270 2Ubull 2 Hex-core CPUrsquos 293GHz
bull 24G Memory
bull 384GB Flash Cache
bull OEL5 64 bit
bull 12 Drives per Storage Server
bull 2TB SAS - High Cap
bull 600GB SAS - High Perf
bull 54GB Sec Transfer Rate
5
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 634
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 734
983112983151983159 983140983151 983156983144983141 983107983151983149983152983151983150983141983150983156983155 983127983151983154983147983103
7
Exadata Storage ServersExadata Database Servers
IO Intensive Processing
bull Index Table Scansbull HCC Uncompress
bull Functions
bull Primitive Joins
bull Predicate Filtering
bull Column Projectionsbull Storage Index Processing
Compute Intensive Processing
bull Complex Joins
bull Aggregation
bull Functions
bull Data Conversion
bull IO Over Infiniband Using iDB Protocol
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 934
9
983123983105983117983109
ASM is the Implementation of SAME
The goal of SAME is to optimize storage IO utilizationbull Bandwidth
bull Sequential and random access
bull Workload types
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1034
10
983113983149983152983148983141983149983141983150983156983145983150983143 983108983137983156983137983138983137983155983141 983123983156983151983154983137983143983141 983127983145983156983144983151983157983156 983105983123983117
Design DB layout strategy
bull How many file systems data files
bull Where data files reside
Inventory of files
bull 2 control files 2 log files multiple datafiles per tablespace
backup files temp files etchellip
bull 100rsquos or 1000rsquos of files to create name manage and not make
mistakes
bull Multiplied by n number of Databases
What about tuning expandingcontracting your DB
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1134
11
983110983145983148983141 983116983137983161983151983157983156983155 983111983141983156983156983145983150983143 983137 983116983145983156983156983148983141 983107983151983149983152983148983145983139983137983156983141983140983103983103
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1234
12
Current data is what end users are usually interested in So as space is
added IO tends to follow the new disks creating hot spots
983120983154983151983158983145983155983145983151983150983145983150983143 983159983145983156983144 983110983145983148983141 983123983161983155983156983141983149983155
u01marketing_01dbf
u01marketing_02dbf
hellip
u01marketing_03dbf u01marketing_04dbf
u01
u02marketing_05dbf
u02marketing_06dbf
hellip
u02marketing_07dbf u02marketing_08dbf
u02 u03
2009 2010 2011
Most Recently
Added Storage
u03marketing_09dbf
u03marketing_10dbf
hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1434
14
983108983137983156983137 983122983141983140983157983150983140983137983150983139983161
bull 983105983123983117 983152983154983151983158983145983140983141983155 983154983141983140983157983150983140983137983150983139983161 983157983155983145983150983143 983142983137983145983148983157983154983141 983143983154983151983157983152983155
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983137983154983141 983157983155983141983154 983140983141983142983145983150983141983140 983157983150983145983156983155 983151983142 983142983137983145983148983157983154983141
ndash 983123983145983150983143983157983148983137983154 983140983145983155983147 ndash 983111983154983151983157983152 983151983142 983140983145983155983147983155 (983123983105983118 983151983154 983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983141983148983148)
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983141983150983155983157983154983141 983149983145983154983154983151983154 983139983151983152983145983141983155 983151983142 983140983137983156983137 983137983154983141 983150983151983156
983155983156983151983154983141983140 983151983150 983156983144983141 983155983137983149983141 983140983145983155983147983155 (983151983154 983143983154983151983157983152983155 983151983142 983140983145983155983147983155)
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1534
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
15
How Exadata Storage Server Redundancy Works
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 334
3
1 What is Exadata
2 What Is ASM
3 Storage Cell Disks
4 Creating and Allocating Storage at the Cell
5 Demo ndash Creating amp Presenting Grid Disks to ASM
6 QampA
983105983143983141983150983140983137
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 434
4
Database Servers
2 - 8 Servers per rack
bull 24 - 96 CPU Cores
bull 96 - 768 Gigabytes Memory
Storage Cells
3 - 14 Storage Cells per rack
72 - 336 CPU Cores
72 - 336 Gigabytes Memory12 - 53 Terabytes Flash Cache
72 - 336 Terabytes Raw Disk Storage
Infiniband Switches
2 Per rack + 1 spine switch for expansion
40Gbs - 5 gigabytes per database server storage cell
Expandable up to 8 racks - more with additional spine switches
983127983144983137983156 983113983155 983109983160983137983140983137983156983137983103
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 534
`
983105 983108983137983156983137 983107983141983150983156983141983154 983113983150 983105 983107983137983150
Database Servers
bull Sun Fire 5670 1U
bull 2 Hex-Core CPUrsquos 293GHz
bull 96G Memory
bull OEL5 64 bit
Infiniband Network
bull Redundant 40Gb Switches
bull Unified Internal Networkbull Used by both DB and Storage
bull Can connect externally
StorageCell Servers
bull SunFire X4270 2Ubull 2 Hex-core CPUrsquos 293GHz
bull 24G Memory
bull 384GB Flash Cache
bull OEL5 64 bit
bull 12 Drives per Storage Server
bull 2TB SAS - High Cap
bull 600GB SAS - High Perf
bull 54GB Sec Transfer Rate
5
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 634
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 734
983112983151983159 983140983151 983156983144983141 983107983151983149983152983151983150983141983150983156983155 983127983151983154983147983103
7
Exadata Storage ServersExadata Database Servers
IO Intensive Processing
bull Index Table Scansbull HCC Uncompress
bull Functions
bull Primitive Joins
bull Predicate Filtering
bull Column Projectionsbull Storage Index Processing
Compute Intensive Processing
bull Complex Joins
bull Aggregation
bull Functions
bull Data Conversion
bull IO Over Infiniband Using iDB Protocol
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 934
9
983123983105983117983109
ASM is the Implementation of SAME
The goal of SAME is to optimize storage IO utilizationbull Bandwidth
bull Sequential and random access
bull Workload types
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1034
10
983113983149983152983148983141983149983141983150983156983145983150983143 983108983137983156983137983138983137983155983141 983123983156983151983154983137983143983141 983127983145983156983144983151983157983156 983105983123983117
Design DB layout strategy
bull How many file systems data files
bull Where data files reside
Inventory of files
bull 2 control files 2 log files multiple datafiles per tablespace
backup files temp files etchellip
bull 100rsquos or 1000rsquos of files to create name manage and not make
mistakes
bull Multiplied by n number of Databases
What about tuning expandingcontracting your DB
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1134
11
983110983145983148983141 983116983137983161983151983157983156983155 983111983141983156983156983145983150983143 983137 983116983145983156983156983148983141 983107983151983149983152983148983145983139983137983156983141983140983103983103
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1234
12
Current data is what end users are usually interested in So as space is
added IO tends to follow the new disks creating hot spots
983120983154983151983158983145983155983145983151983150983145983150983143 983159983145983156983144 983110983145983148983141 983123983161983155983156983141983149983155
u01marketing_01dbf
u01marketing_02dbf
hellip
u01marketing_03dbf u01marketing_04dbf
u01
u02marketing_05dbf
u02marketing_06dbf
hellip
u02marketing_07dbf u02marketing_08dbf
u02 u03
2009 2010 2011
Most Recently
Added Storage
u03marketing_09dbf
u03marketing_10dbf
hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1434
14
983108983137983156983137 983122983141983140983157983150983140983137983150983139983161
bull 983105983123983117 983152983154983151983158983145983140983141983155 983154983141983140983157983150983140983137983150983139983161 983157983155983145983150983143 983142983137983145983148983157983154983141 983143983154983151983157983152983155
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983137983154983141 983157983155983141983154 983140983141983142983145983150983141983140 983157983150983145983156983155 983151983142 983142983137983145983148983157983154983141
ndash 983123983145983150983143983157983148983137983154 983140983145983155983147 ndash 983111983154983151983157983152 983151983142 983140983145983155983147983155 (983123983105983118 983151983154 983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983141983148983148)
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983141983150983155983157983154983141 983149983145983154983154983151983154 983139983151983152983145983141983155 983151983142 983140983137983156983137 983137983154983141 983150983151983156
983155983156983151983154983141983140 983151983150 983156983144983141 983155983137983149983141 983140983145983155983147983155 (983151983154 983143983154983151983157983152983155 983151983142 983140983145983155983147983155)
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1534
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
15
How Exadata Storage Server Redundancy Works
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 434
4
Database Servers
2 - 8 Servers per rack
bull 24 - 96 CPU Cores
bull 96 - 768 Gigabytes Memory
Storage Cells
3 - 14 Storage Cells per rack
72 - 336 CPU Cores
72 - 336 Gigabytes Memory12 - 53 Terabytes Flash Cache
72 - 336 Terabytes Raw Disk Storage
Infiniband Switches
2 Per rack + 1 spine switch for expansion
40Gbs - 5 gigabytes per database server storage cell
Expandable up to 8 racks - more with additional spine switches
983127983144983137983156 983113983155 983109983160983137983140983137983156983137983103
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 534
`
983105 983108983137983156983137 983107983141983150983156983141983154 983113983150 983105 983107983137983150
Database Servers
bull Sun Fire 5670 1U
bull 2 Hex-Core CPUrsquos 293GHz
bull 96G Memory
bull OEL5 64 bit
Infiniband Network
bull Redundant 40Gb Switches
bull Unified Internal Networkbull Used by both DB and Storage
bull Can connect externally
StorageCell Servers
bull SunFire X4270 2Ubull 2 Hex-core CPUrsquos 293GHz
bull 24G Memory
bull 384GB Flash Cache
bull OEL5 64 bit
bull 12 Drives per Storage Server
bull 2TB SAS - High Cap
bull 600GB SAS - High Perf
bull 54GB Sec Transfer Rate
5
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 634
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 734
983112983151983159 983140983151 983156983144983141 983107983151983149983152983151983150983141983150983156983155 983127983151983154983147983103
7
Exadata Storage ServersExadata Database Servers
IO Intensive Processing
bull Index Table Scansbull HCC Uncompress
bull Functions
bull Primitive Joins
bull Predicate Filtering
bull Column Projectionsbull Storage Index Processing
Compute Intensive Processing
bull Complex Joins
bull Aggregation
bull Functions
bull Data Conversion
bull IO Over Infiniband Using iDB Protocol
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 934
9
983123983105983117983109
ASM is the Implementation of SAME
The goal of SAME is to optimize storage IO utilizationbull Bandwidth
bull Sequential and random access
bull Workload types
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1034
10
983113983149983152983148983141983149983141983150983156983145983150983143 983108983137983156983137983138983137983155983141 983123983156983151983154983137983143983141 983127983145983156983144983151983157983156 983105983123983117
Design DB layout strategy
bull How many file systems data files
bull Where data files reside
Inventory of files
bull 2 control files 2 log files multiple datafiles per tablespace
backup files temp files etchellip
bull 100rsquos or 1000rsquos of files to create name manage and not make
mistakes
bull Multiplied by n number of Databases
What about tuning expandingcontracting your DB
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1134
11
983110983145983148983141 983116983137983161983151983157983156983155 983111983141983156983156983145983150983143 983137 983116983145983156983156983148983141 983107983151983149983152983148983145983139983137983156983141983140983103983103
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1234
12
Current data is what end users are usually interested in So as space is
added IO tends to follow the new disks creating hot spots
983120983154983151983158983145983155983145983151983150983145983150983143 983159983145983156983144 983110983145983148983141 983123983161983155983156983141983149983155
u01marketing_01dbf
u01marketing_02dbf
hellip
u01marketing_03dbf u01marketing_04dbf
u01
u02marketing_05dbf
u02marketing_06dbf
hellip
u02marketing_07dbf u02marketing_08dbf
u02 u03
2009 2010 2011
Most Recently
Added Storage
u03marketing_09dbf
u03marketing_10dbf
hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1434
14
983108983137983156983137 983122983141983140983157983150983140983137983150983139983161
bull 983105983123983117 983152983154983151983158983145983140983141983155 983154983141983140983157983150983140983137983150983139983161 983157983155983145983150983143 983142983137983145983148983157983154983141 983143983154983151983157983152983155
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983137983154983141 983157983155983141983154 983140983141983142983145983150983141983140 983157983150983145983156983155 983151983142 983142983137983145983148983157983154983141
ndash 983123983145983150983143983157983148983137983154 983140983145983155983147 ndash 983111983154983151983157983152 983151983142 983140983145983155983147983155 (983123983105983118 983151983154 983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983141983148983148)
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983141983150983155983157983154983141 983149983145983154983154983151983154 983139983151983152983145983141983155 983151983142 983140983137983156983137 983137983154983141 983150983151983156
983155983156983151983154983141983140 983151983150 983156983144983141 983155983137983149983141 983140983145983155983147983155 (983151983154 983143983154983151983157983152983155 983151983142 983140983145983155983147983155)
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1534
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
15
How Exadata Storage Server Redundancy Works
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 534
`
983105 983108983137983156983137 983107983141983150983156983141983154 983113983150 983105 983107983137983150
Database Servers
bull Sun Fire 5670 1U
bull 2 Hex-Core CPUrsquos 293GHz
bull 96G Memory
bull OEL5 64 bit
Infiniband Network
bull Redundant 40Gb Switches
bull Unified Internal Networkbull Used by both DB and Storage
bull Can connect externally
StorageCell Servers
bull SunFire X4270 2Ubull 2 Hex-core CPUrsquos 293GHz
bull 24G Memory
bull 384GB Flash Cache
bull OEL5 64 bit
bull 12 Drives per Storage Server
bull 2TB SAS - High Cap
bull 600GB SAS - High Perf
bull 54GB Sec Transfer Rate
5
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 634
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 734
983112983151983159 983140983151 983156983144983141 983107983151983149983152983151983150983141983150983156983155 983127983151983154983147983103
7
Exadata Storage ServersExadata Database Servers
IO Intensive Processing
bull Index Table Scansbull HCC Uncompress
bull Functions
bull Primitive Joins
bull Predicate Filtering
bull Column Projectionsbull Storage Index Processing
Compute Intensive Processing
bull Complex Joins
bull Aggregation
bull Functions
bull Data Conversion
bull IO Over Infiniband Using iDB Protocol
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 934
9
983123983105983117983109
ASM is the Implementation of SAME
The goal of SAME is to optimize storage IO utilizationbull Bandwidth
bull Sequential and random access
bull Workload types
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1034
10
983113983149983152983148983141983149983141983150983156983145983150983143 983108983137983156983137983138983137983155983141 983123983156983151983154983137983143983141 983127983145983156983144983151983157983156 983105983123983117
Design DB layout strategy
bull How many file systems data files
bull Where data files reside
Inventory of files
bull 2 control files 2 log files multiple datafiles per tablespace
backup files temp files etchellip
bull 100rsquos or 1000rsquos of files to create name manage and not make
mistakes
bull Multiplied by n number of Databases
What about tuning expandingcontracting your DB
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1134
11
983110983145983148983141 983116983137983161983151983157983156983155 983111983141983156983156983145983150983143 983137 983116983145983156983156983148983141 983107983151983149983152983148983145983139983137983156983141983140983103983103
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1234
12
Current data is what end users are usually interested in So as space is
added IO tends to follow the new disks creating hot spots
983120983154983151983158983145983155983145983151983150983145983150983143 983159983145983156983144 983110983145983148983141 983123983161983155983156983141983149983155
u01marketing_01dbf
u01marketing_02dbf
hellip
u01marketing_03dbf u01marketing_04dbf
u01
u02marketing_05dbf
u02marketing_06dbf
hellip
u02marketing_07dbf u02marketing_08dbf
u02 u03
2009 2010 2011
Most Recently
Added Storage
u03marketing_09dbf
u03marketing_10dbf
hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1434
14
983108983137983156983137 983122983141983140983157983150983140983137983150983139983161
bull 983105983123983117 983152983154983151983158983145983140983141983155 983154983141983140983157983150983140983137983150983139983161 983157983155983145983150983143 983142983137983145983148983157983154983141 983143983154983151983157983152983155
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983137983154983141 983157983155983141983154 983140983141983142983145983150983141983140 983157983150983145983156983155 983151983142 983142983137983145983148983157983154983141
ndash 983123983145983150983143983157983148983137983154 983140983145983155983147 ndash 983111983154983151983157983152 983151983142 983140983145983155983147983155 (983123983105983118 983151983154 983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983141983148983148)
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983141983150983155983157983154983141 983149983145983154983154983151983154 983139983151983152983145983141983155 983151983142 983140983137983156983137 983137983154983141 983150983151983156
983155983156983151983154983141983140 983151983150 983156983144983141 983155983137983149983141 983140983145983155983147983155 (983151983154 983143983154983151983157983152983155 983151983142 983140983145983155983147983155)
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1534
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
15
How Exadata Storage Server Redundancy Works
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 634
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 734
983112983151983159 983140983151 983156983144983141 983107983151983149983152983151983150983141983150983156983155 983127983151983154983147983103
7
Exadata Storage ServersExadata Database Servers
IO Intensive Processing
bull Index Table Scansbull HCC Uncompress
bull Functions
bull Primitive Joins
bull Predicate Filtering
bull Column Projectionsbull Storage Index Processing
Compute Intensive Processing
bull Complex Joins
bull Aggregation
bull Functions
bull Data Conversion
bull IO Over Infiniband Using iDB Protocol
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 934
9
983123983105983117983109
ASM is the Implementation of SAME
The goal of SAME is to optimize storage IO utilizationbull Bandwidth
bull Sequential and random access
bull Workload types
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1034
10
983113983149983152983148983141983149983141983150983156983145983150983143 983108983137983156983137983138983137983155983141 983123983156983151983154983137983143983141 983127983145983156983144983151983157983156 983105983123983117
Design DB layout strategy
bull How many file systems data files
bull Where data files reside
Inventory of files
bull 2 control files 2 log files multiple datafiles per tablespace
backup files temp files etchellip
bull 100rsquos or 1000rsquos of files to create name manage and not make
mistakes
bull Multiplied by n number of Databases
What about tuning expandingcontracting your DB
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1134
11
983110983145983148983141 983116983137983161983151983157983156983155 983111983141983156983156983145983150983143 983137 983116983145983156983156983148983141 983107983151983149983152983148983145983139983137983156983141983140983103983103
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1234
12
Current data is what end users are usually interested in So as space is
added IO tends to follow the new disks creating hot spots
983120983154983151983158983145983155983145983151983150983145983150983143 983159983145983156983144 983110983145983148983141 983123983161983155983156983141983149983155
u01marketing_01dbf
u01marketing_02dbf
hellip
u01marketing_03dbf u01marketing_04dbf
u01
u02marketing_05dbf
u02marketing_06dbf
hellip
u02marketing_07dbf u02marketing_08dbf
u02 u03
2009 2010 2011
Most Recently
Added Storage
u03marketing_09dbf
u03marketing_10dbf
hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1434
14
983108983137983156983137 983122983141983140983157983150983140983137983150983139983161
bull 983105983123983117 983152983154983151983158983145983140983141983155 983154983141983140983157983150983140983137983150983139983161 983157983155983145983150983143 983142983137983145983148983157983154983141 983143983154983151983157983152983155
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983137983154983141 983157983155983141983154 983140983141983142983145983150983141983140 983157983150983145983156983155 983151983142 983142983137983145983148983157983154983141
ndash 983123983145983150983143983157983148983137983154 983140983145983155983147 ndash 983111983154983151983157983152 983151983142 983140983145983155983147983155 (983123983105983118 983151983154 983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983141983148983148)
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983141983150983155983157983154983141 983149983145983154983154983151983154 983139983151983152983145983141983155 983151983142 983140983137983156983137 983137983154983141 983150983151983156
983155983156983151983154983141983140 983151983150 983156983144983141 983155983137983149983141 983140983145983155983147983155 (983151983154 983143983154983151983157983152983155 983151983142 983140983145983155983147983155)
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1534
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
15
How Exadata Storage Server Redundancy Works
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 734
983112983151983159 983140983151 983156983144983141 983107983151983149983152983151983150983141983150983156983155 983127983151983154983147983103
7
Exadata Storage ServersExadata Database Servers
IO Intensive Processing
bull Index Table Scansbull HCC Uncompress
bull Functions
bull Primitive Joins
bull Predicate Filtering
bull Column Projectionsbull Storage Index Processing
Compute Intensive Processing
bull Complex Joins
bull Aggregation
bull Functions
bull Data Conversion
bull IO Over Infiniband Using iDB Protocol
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 934
9
983123983105983117983109
ASM is the Implementation of SAME
The goal of SAME is to optimize storage IO utilizationbull Bandwidth
bull Sequential and random access
bull Workload types
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1034
10
983113983149983152983148983141983149983141983150983156983145983150983143 983108983137983156983137983138983137983155983141 983123983156983151983154983137983143983141 983127983145983156983144983151983157983156 983105983123983117
Design DB layout strategy
bull How many file systems data files
bull Where data files reside
Inventory of files
bull 2 control files 2 log files multiple datafiles per tablespace
backup files temp files etchellip
bull 100rsquos or 1000rsquos of files to create name manage and not make
mistakes
bull Multiplied by n number of Databases
What about tuning expandingcontracting your DB
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1134
11
983110983145983148983141 983116983137983161983151983157983156983155 983111983141983156983156983145983150983143 983137 983116983145983156983156983148983141 983107983151983149983152983148983145983139983137983156983141983140983103983103
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1234
12
Current data is what end users are usually interested in So as space is
added IO tends to follow the new disks creating hot spots
983120983154983151983158983145983155983145983151983150983145983150983143 983159983145983156983144 983110983145983148983141 983123983161983155983156983141983149983155
u01marketing_01dbf
u01marketing_02dbf
hellip
u01marketing_03dbf u01marketing_04dbf
u01
u02marketing_05dbf
u02marketing_06dbf
hellip
u02marketing_07dbf u02marketing_08dbf
u02 u03
2009 2010 2011
Most Recently
Added Storage
u03marketing_09dbf
u03marketing_10dbf
hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1434
14
983108983137983156983137 983122983141983140983157983150983140983137983150983139983161
bull 983105983123983117 983152983154983151983158983145983140983141983155 983154983141983140983157983150983140983137983150983139983161 983157983155983145983150983143 983142983137983145983148983157983154983141 983143983154983151983157983152983155
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983137983154983141 983157983155983141983154 983140983141983142983145983150983141983140 983157983150983145983156983155 983151983142 983142983137983145983148983157983154983141
ndash 983123983145983150983143983157983148983137983154 983140983145983155983147 ndash 983111983154983151983157983152 983151983142 983140983145983155983147983155 (983123983105983118 983151983154 983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983141983148983148)
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983141983150983155983157983154983141 983149983145983154983154983151983154 983139983151983152983145983141983155 983151983142 983140983137983156983137 983137983154983141 983150983151983156
983155983156983151983154983141983140 983151983150 983156983144983141 983155983137983149983141 983140983145983155983147983155 (983151983154 983143983154983151983157983152983155 983151983142 983140983145983155983147983155)
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1534
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
15
How Exadata Storage Server Redundancy Works
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 934
9
983123983105983117983109
ASM is the Implementation of SAME
The goal of SAME is to optimize storage IO utilizationbull Bandwidth
bull Sequential and random access
bull Workload types
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1034
10
983113983149983152983148983141983149983141983150983156983145983150983143 983108983137983156983137983138983137983155983141 983123983156983151983154983137983143983141 983127983145983156983144983151983157983156 983105983123983117
Design DB layout strategy
bull How many file systems data files
bull Where data files reside
Inventory of files
bull 2 control files 2 log files multiple datafiles per tablespace
backup files temp files etchellip
bull 100rsquos or 1000rsquos of files to create name manage and not make
mistakes
bull Multiplied by n number of Databases
What about tuning expandingcontracting your DB
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1134
11
983110983145983148983141 983116983137983161983151983157983156983155 983111983141983156983156983145983150983143 983137 983116983145983156983156983148983141 983107983151983149983152983148983145983139983137983156983141983140983103983103
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1234
12
Current data is what end users are usually interested in So as space is
added IO tends to follow the new disks creating hot spots
983120983154983151983158983145983155983145983151983150983145983150983143 983159983145983156983144 983110983145983148983141 983123983161983155983156983141983149983155
u01marketing_01dbf
u01marketing_02dbf
hellip
u01marketing_03dbf u01marketing_04dbf
u01
u02marketing_05dbf
u02marketing_06dbf
hellip
u02marketing_07dbf u02marketing_08dbf
u02 u03
2009 2010 2011
Most Recently
Added Storage
u03marketing_09dbf
u03marketing_10dbf
hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1434
14
983108983137983156983137 983122983141983140983157983150983140983137983150983139983161
bull 983105983123983117 983152983154983151983158983145983140983141983155 983154983141983140983157983150983140983137983150983139983161 983157983155983145983150983143 983142983137983145983148983157983154983141 983143983154983151983157983152983155
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983137983154983141 983157983155983141983154 983140983141983142983145983150983141983140 983157983150983145983156983155 983151983142 983142983137983145983148983157983154983141
ndash 983123983145983150983143983157983148983137983154 983140983145983155983147 ndash 983111983154983151983157983152 983151983142 983140983145983155983147983155 (983123983105983118 983151983154 983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983141983148983148)
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983141983150983155983157983154983141 983149983145983154983154983151983154 983139983151983152983145983141983155 983151983142 983140983137983156983137 983137983154983141 983150983151983156
983155983156983151983154983141983140 983151983150 983156983144983141 983155983137983149983141 983140983145983155983147983155 (983151983154 983143983154983151983157983152983155 983151983142 983140983145983155983147983155)
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1534
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
15
How Exadata Storage Server Redundancy Works
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 934
9
983123983105983117983109
ASM is the Implementation of SAME
The goal of SAME is to optimize storage IO utilizationbull Bandwidth
bull Sequential and random access
bull Workload types
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1034
10
983113983149983152983148983141983149983141983150983156983145983150983143 983108983137983156983137983138983137983155983141 983123983156983151983154983137983143983141 983127983145983156983144983151983157983156 983105983123983117
Design DB layout strategy
bull How many file systems data files
bull Where data files reside
Inventory of files
bull 2 control files 2 log files multiple datafiles per tablespace
backup files temp files etchellip
bull 100rsquos or 1000rsquos of files to create name manage and not make
mistakes
bull Multiplied by n number of Databases
What about tuning expandingcontracting your DB
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1134
11
983110983145983148983141 983116983137983161983151983157983156983155 983111983141983156983156983145983150983143 983137 983116983145983156983156983148983141 983107983151983149983152983148983145983139983137983156983141983140983103983103
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1234
12
Current data is what end users are usually interested in So as space is
added IO tends to follow the new disks creating hot spots
983120983154983151983158983145983155983145983151983150983145983150983143 983159983145983156983144 983110983145983148983141 983123983161983155983156983141983149983155
u01marketing_01dbf
u01marketing_02dbf
hellip
u01marketing_03dbf u01marketing_04dbf
u01
u02marketing_05dbf
u02marketing_06dbf
hellip
u02marketing_07dbf u02marketing_08dbf
u02 u03
2009 2010 2011
Most Recently
Added Storage
u03marketing_09dbf
u03marketing_10dbf
hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1434
14
983108983137983156983137 983122983141983140983157983150983140983137983150983139983161
bull 983105983123983117 983152983154983151983158983145983140983141983155 983154983141983140983157983150983140983137983150983139983161 983157983155983145983150983143 983142983137983145983148983157983154983141 983143983154983151983157983152983155
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983137983154983141 983157983155983141983154 983140983141983142983145983150983141983140 983157983150983145983156983155 983151983142 983142983137983145983148983157983154983141
ndash 983123983145983150983143983157983148983137983154 983140983145983155983147 ndash 983111983154983151983157983152 983151983142 983140983145983155983147983155 (983123983105983118 983151983154 983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983141983148983148)
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983141983150983155983157983154983141 983149983145983154983154983151983154 983139983151983152983145983141983155 983151983142 983140983137983156983137 983137983154983141 983150983151983156
983155983156983151983154983141983140 983151983150 983156983144983141 983155983137983149983141 983140983145983155983147983155 (983151983154 983143983154983151983157983152983155 983151983142 983140983145983155983147983155)
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1534
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
15
How Exadata Storage Server Redundancy Works
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1034
10
983113983149983152983148983141983149983141983150983156983145983150983143 983108983137983156983137983138983137983155983141 983123983156983151983154983137983143983141 983127983145983156983144983151983157983156 983105983123983117
Design DB layout strategy
bull How many file systems data files
bull Where data files reside
Inventory of files
bull 2 control files 2 log files multiple datafiles per tablespace
backup files temp files etchellip
bull 100rsquos or 1000rsquos of files to create name manage and not make
mistakes
bull Multiplied by n number of Databases
What about tuning expandingcontracting your DB
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1134
11
983110983145983148983141 983116983137983161983151983157983156983155 983111983141983156983156983145983150983143 983137 983116983145983156983156983148983141 983107983151983149983152983148983145983139983137983156983141983140983103983103
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1234
12
Current data is what end users are usually interested in So as space is
added IO tends to follow the new disks creating hot spots
983120983154983151983158983145983155983145983151983150983145983150983143 983159983145983156983144 983110983145983148983141 983123983161983155983156983141983149983155
u01marketing_01dbf
u01marketing_02dbf
hellip
u01marketing_03dbf u01marketing_04dbf
u01
u02marketing_05dbf
u02marketing_06dbf
hellip
u02marketing_07dbf u02marketing_08dbf
u02 u03
2009 2010 2011
Most Recently
Added Storage
u03marketing_09dbf
u03marketing_10dbf
hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1434
14
983108983137983156983137 983122983141983140983157983150983140983137983150983139983161
bull 983105983123983117 983152983154983151983158983145983140983141983155 983154983141983140983157983150983140983137983150983139983161 983157983155983145983150983143 983142983137983145983148983157983154983141 983143983154983151983157983152983155
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983137983154983141 983157983155983141983154 983140983141983142983145983150983141983140 983157983150983145983156983155 983151983142 983142983137983145983148983157983154983141
ndash 983123983145983150983143983157983148983137983154 983140983145983155983147 ndash 983111983154983151983157983152 983151983142 983140983145983155983147983155 (983123983105983118 983151983154 983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983141983148983148)
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983141983150983155983157983154983141 983149983145983154983154983151983154 983139983151983152983145983141983155 983151983142 983140983137983156983137 983137983154983141 983150983151983156
983155983156983151983154983141983140 983151983150 983156983144983141 983155983137983149983141 983140983145983155983147983155 (983151983154 983143983154983151983157983152983155 983151983142 983140983145983155983147983155)
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1534
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
15
How Exadata Storage Server Redundancy Works
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1134
11
983110983145983148983141 983116983137983161983151983157983156983155 983111983141983156983156983145983150983143 983137 983116983145983156983156983148983141 983107983151983149983152983148983145983139983137983156983141983140983103983103
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1234
12
Current data is what end users are usually interested in So as space is
added IO tends to follow the new disks creating hot spots
983120983154983151983158983145983155983145983151983150983145983150983143 983159983145983156983144 983110983145983148983141 983123983161983155983156983141983149983155
u01marketing_01dbf
u01marketing_02dbf
hellip
u01marketing_03dbf u01marketing_04dbf
u01
u02marketing_05dbf
u02marketing_06dbf
hellip
u02marketing_07dbf u02marketing_08dbf
u02 u03
2009 2010 2011
Most Recently
Added Storage
u03marketing_09dbf
u03marketing_10dbf
hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1434
14
983108983137983156983137 983122983141983140983157983150983140983137983150983139983161
bull 983105983123983117 983152983154983151983158983145983140983141983155 983154983141983140983157983150983140983137983150983139983161 983157983155983145983150983143 983142983137983145983148983157983154983141 983143983154983151983157983152983155
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983137983154983141 983157983155983141983154 983140983141983142983145983150983141983140 983157983150983145983156983155 983151983142 983142983137983145983148983157983154983141
ndash 983123983145983150983143983157983148983137983154 983140983145983155983147 ndash 983111983154983151983157983152 983151983142 983140983145983155983147983155 (983123983105983118 983151983154 983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983141983148983148)
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983141983150983155983157983154983141 983149983145983154983154983151983154 983139983151983152983145983141983155 983151983142 983140983137983156983137 983137983154983141 983150983151983156
983155983156983151983154983141983140 983151983150 983156983144983141 983155983137983149983141 983140983145983155983147983155 (983151983154 983143983154983151983157983152983155 983151983142 983140983145983155983147983155)
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1534
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
15
How Exadata Storage Server Redundancy Works
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1234
12
Current data is what end users are usually interested in So as space is
added IO tends to follow the new disks creating hot spots
983120983154983151983158983145983155983145983151983150983145983150983143 983159983145983156983144 983110983145983148983141 983123983161983155983156983141983149983155
u01marketing_01dbf
u01marketing_02dbf
hellip
u01marketing_03dbf u01marketing_04dbf
u01
u02marketing_05dbf
u02marketing_06dbf
hellip
u02marketing_07dbf u02marketing_08dbf
u02 u03
2009 2010 2011
Most Recently
Added Storage
u03marketing_09dbf
u03marketing_10dbf
hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1434
14
983108983137983156983137 983122983141983140983157983150983140983137983150983139983161
bull 983105983123983117 983152983154983151983158983145983140983141983155 983154983141983140983157983150983140983137983150983139983161 983157983155983145983150983143 983142983137983145983148983157983154983141 983143983154983151983157983152983155
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983137983154983141 983157983155983141983154 983140983141983142983145983150983141983140 983157983150983145983156983155 983151983142 983142983137983145983148983157983154983141
ndash 983123983145983150983143983157983148983137983154 983140983145983155983147 ndash 983111983154983151983157983152 983151983142 983140983145983155983147983155 (983123983105983118 983151983154 983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983141983148983148)
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983141983150983155983157983154983141 983149983145983154983154983151983154 983139983151983152983145983141983155 983151983142 983140983137983156983137 983137983154983141 983150983151983156
983155983156983151983154983141983140 983151983150 983156983144983141 983155983137983149983141 983140983145983155983147983155 (983151983154 983143983154983151983157983152983155 983151983142 983140983145983155983147983155)
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1534
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
15
How Exadata Storage Server Redundancy Works
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1434
14
983108983137983156983137 983122983141983140983157983150983140983137983150983139983161
bull 983105983123983117 983152983154983151983158983145983140983141983155 983154983141983140983157983150983140983137983150983139983161 983157983155983145983150983143 983142983137983145983148983157983154983141 983143983154983151983157983152983155
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983137983154983141 983157983155983141983154 983140983141983142983145983150983141983140 983157983150983145983156983155 983151983142 983142983137983145983148983157983154983141
ndash 983123983145983150983143983157983148983137983154 983140983145983155983147 ndash 983111983154983151983157983152 983151983142 983140983145983155983147983155 (983123983105983118 983151983154 983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983141983148983148)
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983141983150983155983157983154983141 983149983145983154983154983151983154 983139983151983152983145983141983155 983151983142 983140983137983156983137 983137983154983141 983150983151983156
983155983156983151983154983141983140 983151983150 983156983144983141 983155983137983149983141 983140983145983155983147983155 (983151983154 983143983154983151983157983152983155 983151983142 983140983145983155983147983155)
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1534
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
15
How Exadata Storage Server Redundancy Works
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1434
14
983108983137983156983137 983122983141983140983157983150983140983137983150983139983161
bull 983105983123983117 983152983154983151983158983145983140983141983155 983154983141983140983157983150983140983137983150983139983161 983157983155983145983150983143 983142983137983145983148983157983154983141 983143983154983151983157983152983155
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983137983154983141 983157983155983141983154 983140983141983142983145983150983141983140 983157983150983145983156983155 983151983142 983142983137983145983148983157983154983141
ndash 983123983145983150983143983157983148983137983154 983140983145983155983147 ndash 983111983154983151983157983152 983151983142 983140983145983155983147983155 (983123983105983118 983151983154 983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983141983148983148)
bull 983110983137983145983148983157983154983141 983143983154983151983157983152983155 983141983150983155983157983154983141 983149983145983154983154983151983154 983139983151983152983145983141983155 983151983142 983140983137983156983137 983137983154983141 983150983151983156
983155983156983151983154983141983140 983151983150 983156983144983141 983155983137983149983141 983140983145983155983147983155 (983151983154 983143983154983151983157983152983155 983151983142 983140983145983155983147983155)
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1534
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
15
How Exadata Storage Server Redundancy Works
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1534
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
15
How Exadata Storage Server Redundancy Works
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1634
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983107983151983150983142983145983143983157983154983137983156983145983151983150
16
Disk Configuration Overview
The CellCLI program is used on the Storage Servers to configure disks
for presentation to ASM
bull Physical - Physical Disk
bull LUN - OS Block Device
bull Cell Disk - Used by Cell Server to manage disksbull Grid Disk - Presented to the DB as ASM disks
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1734
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143
17
ASM View of Grid Disks
SYS+ASM1gt select path total_mb failgroup from v$asm_disk
order by failgroup path
Non-Exadata
PATH TOTAL_MB FAILGROUP--------------- ---------- ----------
devrdskhellip 11444 DATA01devrdskhellip 11444 DATA02hellip
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01o192168123DATA_CD_01_cell01 1313600 CELL01o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1834
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 1934
983109983160983137983140983137983156983137 983123983156983151983154983137983143983141 983123983141983154983158983141983154 983107983151983150983142983145983143983157983154983137983156983145983151983150
19
Typical Exadata Storage Configuration
Grid Disks
Cell Disk 12Cell Disk 11Cell Disk 10
SYSTEM_DG
Cell Disk 4Cell Disk 3Cell Disk 2Cell Disk 1
Storage Cell 1
Failure Group CELL01
DATA_DG
RECO_DG
DATA_DG
CELL01
CELL02
CELL03
RECO_DG
CELL01
CELL02
CELL03
SYSTEM_DG
CELL01
CELL02
CELL03
ASM Disk Groups
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2034
983111983154983145983140 983108983145983155983147 983118983137983149983141983155
20
Grid Disk Names
Exadata
PATH TOTAL_MB FAILGROUP-------------------------------- -------- ---------o192168123DATA_CD_00_cell01 1313600 CELL01
o192168123DATA_CD_01_cell01 1313600 CELL01
o192168123DATA_CD_02_cell01 1313600 CELL01hellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2134
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
21
There are two ways to create grid disks
CellCLIgt create griddisk TEST_DG_CD_00_cell03 ndashcelldisk=CD_00_cell03 size=100M
GridDisk TEST_DG_CD_00_cell03 successfully created
CellCLIgt list griddisk attributes name celldisk size ndashwhere name=TEST_DG_CD_00_cell03
TEST_DG_CD_00_cell03 CD_00_cell03 96M
1 Create a single grid disk by fully qualifying the name
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2234
983107983154983141983137983156983145983150983143 983111983154983145983140 983108983145983155983147983155
22
There are two ways to create grid disks
2 Create a collection of grid disks
bull using the lsquoallrsquo parameterbull specify a name lsquoprefixrsquo
bull Grid disks are named as follows
prefix_celldisk_name
CellCLIgt create griddisk all harddisk prefix=TEST size=100MGridDisk TEST_CD_00_cell03 successfully createdGridDisk TEST_CD_01_cell03 successfully createdhellip
GridDisk TEST_CD_10_cell03 successfully createdGridDisk TEST_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size -where name like TEST_
TEST_CD_00_cell03 FD_00_cell03 HardDisk 96MhellipTEST_CD_10_cell03 FD_10_cell03 HardDisk 96MTEST_CD_11_cell03 FD_11_cell03 HardDisk 96M
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2334
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
23
Dealing with the lsquoshort disksrsquo
[enkcel03root] rootgt fdisk -lu devsdahellip
Device Boot Start End Blocks Id Systemdevsda1 63 240974 120456 fd Linux raid autodetectdevsda2 240975 257039 8032+ 83 Linux
devsda3 257040 3843486989 1921614975 83 Linuxdevsda4 3843486990 3904293014 30403012+ f W95 Extd (LBA)devsda5 3843487053 3864451814 10482381 fd Linux raid autodetecthellipdevsda10 3897995598 3899425319 714861 fd Linux raid autodetectdevsda11 3899425383 3904293014 2433816 fd Linux raid autodetect
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2434
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
24
Dealing with the lsquoshort disksrsquo
CellCLIgt list celldisk attributes name devicePartition size ndashwhere diskType = HardDisk
CD_00_cell03 devsda3 183259375GCD_01_cell03 devsdb3 183259375GCD_02_cell03 devsdc 1861703125G
CD_03_cell03 devsdd 1861703125GCD_07_cell03 devsdh 1861703125GCD_08_cell03 devsdi 1861703125GCD_09_cell03 devsdj 1861703125GCD_10_cell03 devsdk 1861703125GCD_11_cell03 devsdl 1861703125G
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2534
983111983154983145983140 983108983145983155983147 983123983145983162983145983150983143
25
bull The lsquosizersquo parameter determines the size of each grid
disk
bull If no size is specified then all remaining cell disk
space is used for the grid disk
CellCLIgt create griddisk all prefix=RECO_DGGridDisk RECO_DG_CD_00_cell03 successfully created
GridDisk RECO_DG_CD_01_cell03 successfully createdhellip
GridDisk RECO_DG_CD_10_cell03 successfully createdGridDisk RECO_DG_CD_11_cell03 successfully created
CellCLIgt list griddisk attributes name cellDisk diskType size where namelike RECO
RECO_CD_00_cell01 CD_00_cell01 HardDisk 91265625G
RECO_CD_01_cell01 CD_01_cell01 HardDisk 91265625G
RECO_CD_02_cell01 CD_02_cell01 HardDisk 120375GRECO_CD_03_cell01 CD_03_cell01 HardDisk 120375GRECO_CD_04_cell01 CD_04_cell01 HardDisk 120375Ghellip
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2634
26
983123983156983151983154983137983143983141 983107983141983148983148 983120983154983151983158983145983155983145983151983150983145983150983143
bull Storage Cells may be allocated to specific database
servers creating multiple storage grids
bull Multiple RAC Clusters Production Cluster
Test Cluster
Dev Cluster
bull Caution ndash This reduces IO bandwidth
Storage Cells need not be carved up to support multiple
database servers
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2734
27
983124983144983141 983107983141983148983148983145983150983145983156983151983154983137 983110983145983148983141
optoraclecell112220_LINUXX64_1012062cellsrvdeployconfigcellinitora
CELL Initialization Parameters
version=00
DEPLOYED=TRUESSL_PORT=23943
JMS_PORT=9127
ipaddress1=19216812924 -- Storage Grid address ofHTTP_PORT=8888 the storage cellBMC_SNMP_PORT=162 (on the Infiniband switch)RMI_PORT=23791
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2834
28
983124983144983141 983107983141983148983148983145983152983151983154983137 983110983145983148983141
dm01cel12 1921681220
dm01cel13 1921681221
dm01cel14 1921681222
cell=1921681220
cell=1921681221
cell=1921681222
Test Storage Cells 12-14Test Database Servers 7-8
dm01cel01 192168129
dm01cel02 1921681210dm01cel03 1921681211
dm01cel04 1921681212
dm01cel05 1921681213
dm01cel06 1921681214
dm01cel07 1921681215
dm01cel08 1921681216dm01cel09 1921681217
dm01cel10 1921681218
dm01cel11 1921681219
cell=192168129
cell=1921681210
cell=1921681211
cell=1921681212
cell=1921681213
cell=1921681214
cell=1921681215
cell=1921681216cell=1921681217
cell=1921681218
cell=1921681219
Production Storage Cells 1-11Production Database Servers 1-6
etcoraclecellnetwork-configcellipora
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 2934
29
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 983108983141983142983137983157983148983156 983107983151983150983142983145983143983157983154983137983156983145983151983150
Oracle Cluster PROD_CLUSTER
14 Storage Cells 168 Cell
disks 3 ASM Disk Groups
DATA_DG
RECO_DG
SYSTEM_DG
dbm2
dbm3
dbm4
dbm5
dbm6
dbm7
dbm8
dbm1
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3034
30
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 8 983150983151983150983085983122983105983107 983108983137983156983137983138983137983155983141983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3134
31
983109983160983137983140983137983156983137 983110983157983148983148 983122983137983139983147983098 3 983122983105983107 983107983148983157983155983156983141983154983155
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3234
32
983122983141983137983148 983127983151983154983148983140 983109983160983137983149983152983148983141983098 2 983109983160983137983140983137983156983137 983112983137983148983142 983122983137983139983147983155
Exadata Rack 1 Exadata Rack 2
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3334
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
8122019 Presentation - Exadata Storage Layout
httpslidepdfcomreaderfullpresentation-exadata-storage-layout 3434
34
randyjohnsonenkiteccom
