Upload
others
View
17
Download
0
Embed Size (px)
Citation preview
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle Database NaBve Sharding
Nagesh BaEula [email protected] Oracle Sharding Product Development Feb 16, 2017
Technical Overview
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement The following is intended to outline our general product direcBon. It is intended for informaBon purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or funcBonality, and should not be relied upon in making purchasing decisions. The development, release, and Bming of any features or funcBonality described for Oracle’s products remains at the sole discreBon of Oracle.
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Sharding Overview
Sharded Database and Schema CreaBon
Data Dependent RouBng
MAA Benchmark Results -‐ Sharding on Bare Metal Cloud
Customers’ PerspecBve
CompeBBve
1
2
3
4
3
5
6
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Sharding Overview
Sharded Database and Schema CreaBon
Data Dependent RouBng
MAA Benchmark Results -‐ Sharding on Bare Metal Cloud
Customers’ PerspecBve
CompeBBve
1
2
3
4
4
5
6
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Horizontal parBBoning of data across up to 1000 independent Oracle Databases (shards)
• Shared-‐nothing hardware architecture – Each shard runs on commodity server – No shared storage – No clusterware
• Data is parBBoned using a sharding key (i.e. account_id)
5
Oracle Sharding – An ElasBc Database Architecture
A single logical DB sharded into N physical Databases
Shard01: Table 1
ParBBons
B A
D C
ParBBons
F E
H G
Shard02: Table 1
Shardnn: Table 1
ParBBons
… …
… …
Server A
Server B
Server N
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 6
Oracle Database Sharding – Benefits
Linear Scalability
Add shards online to increase database size and throughput. Online split and rebalance.
Extreme Availability
Shared-‐nothing hardware architecture. Fault of one shard
has no impact on others.
Geographic DistribuJon
User defined data placement for performance, availability, DR or to meet regulatory requirements.
…
…
…
…
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Shard Catalog/Coordinator
App Tier
RouBng Tier
Data Tier
7
CapabiliBes of Oracle NaBve Sharding
ConnecBon Pool
Shard Directors
Sharding key
…
…
• Auto deployment of up to 1000 shards – Supports Data Guard and Oracle GoldenGate
• MulBple sharding methods – System Managed (Consistent Hash)
• Uniform data distribuBon across shards à linear scalability
– Composite (Range/List-‐Consistent Hash) • Geo-‐distribuBon plus linear scalability
• Centralized schema maintenance – NaBve SQL for sharded and duplicated tables
• RouBng – Direct rouBng from ConnecBon pools – Proxy rouBng for MulB-‐shard queries
• Geo-‐distribuBon for data proximity and sovereignty • Supports on-‐premises, cloud or hybrid deployments • Online scale-‐out with automaBc rebalance or scale-‐back • Shards may run different Oracle releases
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle Database 12c Release 2 Offers Choice
Oracle Sharding
…
• Single logical database • Supports OLTP applicaBons designed to shard • Single physical database
• Supports any applicaBon
N Physical Databases
Oracle RAC
…
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Sharding – a Different Way to Scale FricJonless linear scaling due to zero shared hardware or soPware
# of Shards
TPS
9
0
2,000,000
4,000,000
6,000,000
8,000,000
10,000,000
12,000,000
50 100 150 200
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Fault IsolaJon Sharding for Extreme Data Availability
The porBon of users who undergo brown-‐out at one Bme by an unplanned outage or database upgrade in a
sharded database with 100 shards
1% 10
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Sharding – a Superior Way to Distribute Data
11
Data Sovereignty and Data Proximity
Customers Americas Customers Asia Customers Europe
Sharded Database Linear Scalability & GeodistribuBon
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Sharding with Exadata
12
Data Sovereignty Sharded Database
GeodistribuBon for Data Sovereignty & Privacy Compliance
Ø Fast, direct access from within each region
Ø Queries access data from all regions as a single logical database
Customers Americas Customers Asia Customers Europe
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Sharding – a Superior Way to Deploy Database
13
Flexible On-‐Premises or Cloud Deployment
On-‐Premises Cloud Hybrid
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Automated Data Availability ConfiguraBon
14
AcBve Data Guard with Fast-‐Start Failover
GoldenGate ‘chunk-‐level’ acBve-‐acBve replicaBon with automaBc conflict detecBon/resoluBon (OGG 12.3)
ParBBon_F
ParBBon_F
ParBBon_A ParBBon_C
OpBonally – complement replicaBon with Oracle RAC for server HA
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
AcBve-‐AcBve Sharded Database – Oracle GoldenGate 12.3
15
• AutomaBc GoldenGate deployment – Chunk-‐level replicaBon within and across shardgroups
• AutomaBc CDR (conflict detecBon & resoluBon) to simplify acBve-‐acBve replicaBon – Including insert-‐delete conflicts, JSON columns, LOB updates
• Shardgroups can have different number of shards, replicaBon factor, database versions and H/W plaqorms
#Shards = 50; ReplicaBon Factor = 3 Shardgroup 1: Availability_Domain1
#Shards = 20; ReplicaBon Factor = 2
......
Shardgroup 2: Availability_Domain2
......
Oracle GoldenGate replicaJon within and across Shardgroups
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Data Modeling Requirements
• Hierarchical table family that consists of a root table and many child and grandchild tables
• Every table in the table family must contain the sharding _key – Sharding _key must be the leading column of the primary key of the root table
• The sharding method and sharding_key are determined based upon applicaBon and business requirements
• IdenBfy tables that need to be duplicated on all shards – Common reference data (e.g. Products)
16
Table Family and Duplicated Tables
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Direct RouBng • OLTP workloads that require linear scale, high performance, and fault isolaBon must specify a sharding_key during connect – All data required for a transacBon is contained within a single shard
Proxy RouBng • The following workloads must be able to accept lesser SLAs for linear scale, performance, and fault isolaBon than OLTP transacBons – ReporBng workloads – MulB-‐shard queries – ApplicaBons unable to specify a sharding_key
Database ConnecBons • Must use Oracle integrated connecBon pools (UCP, OCI, ODP.NET, JDBC)
• Must be able to separate workloads that use Direct RouBng from those that use Proxy RouBng – Each uses separate connecBon pools
ApplicaBon Requirements
17
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Sharding Overview
Sharded Database and Schema CreaBon
Data Dependent RouBng
MAA Benchmark Results -‐ Sharding on Bare Metal Cloud
Customers’ PerspecBve
CompeBBve
1
2
3
4
18
5
6
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Deployment of a System-‐Managed SDB with Data Guard
19
…
…
Clients
Data Guard Fast-‐Start Failover
Region Availabilty_Domain1
Shard Catalog shardcat_stdby
Shard Director shdir3,4
Region Availability_Domain2
Shardgroup shgrp2
Shardgroup shgrp1 Shard Director
shdir1,2 Shard Catalog
shardcat
ConnecBon Pools
ConnecBon Pools
Primaries
HA Standbys
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
DeclaraBve SpecificaBon of SDB ConfiguraBon
20
Primary components create shardcatalog -‐database <shardcat host>:1521:shardcat -‐user sdb_admin/passwd_sdb_admin add gsm –gsm sharddirector1 -‐listener 1571 -‐pwd passwd_gsmcatuser -‐catalog <shardcat host>:1521:shardcat .. add credenBal -‐credenBal oracle_cred -‐osaccount oracle -‐ospassword < > add shardgroup -‐shardgroup shgrp1 -‐deploy_as primary -‐region Availability_Domain1 create shard -‐shardgroup shgrp1 -‐desJnaJon <host1> -‐credenJal oracle_cred …
Duplicated components for HA deployed in separate availability domain add gsm –gsm sharddirector2 -‐listener 1571 -‐pwd passwd_gsmcatuser -‐catalog <shardcat host>:1521:shardcat .. add shardgroup -‐shardgroup shgrp2 -‐deploy_as acBve_standby -‐region Availability_Domain2 create shard -‐shardgroup shgrp2 -‐desJnaJon <host11> -‐credenJal oracle_cred … deploy add service -‐service oltp_rw_srvc -‐role primary
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Sharded Tables
Shard 1 Shard 2 Shard N
… … …
21
Schema CreaBon – Sharded and Duplicated Tables
Customers Orders Line Items
Products
Database Tables
Sets of related parBBons of customers, orders & line items placed on a given tablespace[s]
Duplicated Tables
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Map Related ParBBons to a Chunk
• In simplest form, each chunk is a tablespace containing related parBBons of the sharded tables, i.e. Chunk#1 contains Customers_P1, Orders_P1, etc – A chunk can also be a set of tablespaces that store related parBBons – a separate tablespace for each parBBon of the table
• All data perBnent to a given sharding key resides in a single chunk
22
Chunk is the Unit of Data Movement in a Sharded Database
Customers_P1 Orders_P1 Lineitems_P1 Chunk #1 Related ParBBons of Sharded Tables
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Each Shard Contains MulBple Chunks
Lineitems_P6
23
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
System Managed Sharding • Based on parBBoning by CONSISTENT HASH • Range of hash values assigned to each chunk • Data is uniformly sharded / re-‐sharded automaBcally • Many relaBvely small equally sized chunks
+ Linear Scalability + AutomaBc balanced data distribuBon -‐ User has no control on locaBon of data
24
Ø Customer facing applicaBons sharded by customer_id
Ø For linear scalability
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Tablespace Set
25
Shard1
Tablespace tbs1-‐120
Tablespace tbs1-‐1
Shard2
Tablespace tbs1-‐240
Tablespace tbs1-‐121 …. ….
Tablespace Set tbs1
….
Tablespace tbs1-‐360
Tablespace tbs1-‐241
Shard3
• CREATE TABLESPACE SET tbs1;!– Automates creaBon of tablespaces and distributes them among shards – Number of tablespaces = Number of chunks
• Supports all tablespace clauses
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 26
CreaBng a Sharded Table Family with ReferenBal Integrity
CREATE DUPLICATED TABLE Products ( !ProductId INTEGER PRIMARY KEY, !Name VARCHAR2(128), !LastPrice NUMBER(19,4),!… !)!TABLESPACE products_tsp ;!
Execute DDLs on the Shard Catalog !CREATE TABLESPACE SET tbs1 ;!!CREATE SHARDED TABLE Customers!( CustId VARCHAR2(60) NOT NULL, ! FirstName VARCHAR2(60), ! LastName VARCHAR2(60),! …! CONSTRAINT pk_customers ! PRIMARY KEY(CustId)!)!PARTITION BY CONSISTENT HASH (CustId) !PARTITIONS AUTO !TABLESPACE SET tbs1 ;!
CREATE SHARDED TABLE Orders ( ! OrderId INTEGER,! CustId VARCHAR2(60),! OrderDate TIMESTAMP,! …! CONSTRAINT pk_orders ! PRIMARY KEY (CustId, OrderId),! CONSTRAINT fk_orders_parent ! FOREIGN KEY (CustId) REFERENCES Customers(CustId) !)!PARTITION BY REFERENCE (fk_orders_parent) ;!
AutomaBcally maps parBBons to the tablespaces that were created
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle Sharding for Document Store ApplicaBons
27
Supports Schemaless ApplicaJon Development with JSON
CREATE TABLESPACE SET tbs1 ;!!CREATE SHARDED TABLE Customers!( CustId VARCHAR2(60) NOT NULL, ! CustProfile CLOB, ! CONSTRAINT pk_customers PRIMARY KEY(CustId),! CONSTRAINT cons_json CHECK (CustProfile IS JSON)!)!PARTITION BY CONSISTENT HASH (CustId) !PARTITIONS AUTO !TABLESPACE SET tbs1 ;!
• Full power of SQL to JSON document stores plus linear scalability of SDB
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Composite Sharding
• Provides two-‐levels of sharding • The sharded database is divided into N sets of shards called shardspaces – Data is parBBoned across shardspaces by LIST or RANGE on super-‐sharding key (e.g. geography)
– Within each shardspace, data is parBBoned across shards by CONSISTENT HASH using sharding key (e.g. customer id)
+ Best pracBce for geo-‐distribuBon or hybrid clouds with linear scalability
-‐ Requires two sharding keys: super_sharding_key and sharding_key
28
Ø Billing system sharded by geo then by account_id
Ø For geo-‐distribuBon and linear scaling
Customers Asia
Sharded Database Linear Scalability in each Geography
Customers Americas
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Concept: Shardspace • A set of shards that stores data corresponding to a range or list of key values – The super_sharding key is a column other than the sharding_key (e.g. Geo), that defines the number of sets
– The sharding_key is used to maintain automaBc data distribuBon across shards within a group using consistent hash
• Each shardspace includes 1 or more replicated shardgroups for HA/DR
29
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Composite Sharding – Example
30
sh3
Tablespace tbs2-‐60 P_60
Tablespace tbs2-‐1 P_1
…. ….
sh4
Tablespace tbs2-‐120 P_120
Tablespace tbs2-‐61 P_61
sh5
Tablespace tbs2-‐180 P_180
Tablespace tbs2-‐121 P_121
…. ….
sh1
Tablespace tbs1-‐120 P_60
Tablespace tbs1-‐1 P_1
Tablespace tbs1-‐120 P_120
Tablespace tbs1-‐61 P_61
…. ….
sh2 ADD SHARDSPACE –SHARDSPACE Americas –CHUNKS 120; CREATE SHARD –SHARDSPACE Americas -DESTINATION <host1> -CREDENTIAL oracle_cred; CREATE SHARD –SHARDSPACE Americas -DESTINATION <host2> -CREDENTIAL oracle_cred; DEPLOY; CREATE TABLESPACE SET tbs1 IN SHARDSPACE Americas; ADD SHARDSPACE –SHARDSPACE Asia –CHUNKS 180; !CREATE SHARD –SHARDSPACE Asia -DESTINATION <host3> -CREDENTIAL oracle_cred; CREATE SHARD –SHARDSPACE Asia -DESTINATION <host4> -CREDENTIAL oracle_cred; CREATE SHARD –SHARDSPACE Asia -DESTINATION <host5> -CREDENTIAL oracle_cred; DEPLOY; CREATE TABLESPACE SET tbs2 IN SHARDSPACE Asia;
Tablespace Set tbs1
Tablespace Set tbs2
Shardspace Americas
Shardspace Asia
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 31
Map Ranges of Sharding Keys to ParJJons Composite Sharding -‐ Example
sh3
Tablespace tbs2-‐60 P_60
Tablespace tbs2-‐1 P_1
…. ….
sh4
Tablespace tbs2-‐120 P_120
Tablespace tbs2-‐61 P_61
sh5
Tablespace tbs2-‐180 P_180
Tablespace tbs2-‐121 P_121
…. ….
sh1
Tablespace tbs1-‐120 P_60
Tablespace tbs1-‐1 P_1
Tablespace tbs1-‐120 P_120
Tablespace tbs1-‐61 P_61
…. ….
sh2
Tablespace Set tbs1
Tablespace Set tbs2
Shardspace Americas
Shardspace Asia
CREATE SHARDED TABLE customers ( CustId VARCHAR2(60) NOT NULL, .. Geo VARCHAR2(8), CONSTRAINT cust_pk PRIMARY KEY(CustId) ) PARTITIONSET BY LIST (Geo) PARTITION BY CONSISTENT HASH (CustId) PARTITIONS AUTO (PARTITIONSET Americas VALUES (‘Americas’) TABLESPACE SET tbs1, PARTITIONSET Asia VALUES (‘Asia’) TABLESPACE SET tbs2) ;
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Sharding Overview
Sharded Database and Schema CreaBon
Data Dependent RouBng
MAA Benchmark Results -‐ Sharding on Bare Metal Cloud
Customers’ PerspecBve
CompeBBve
1
2
3
4
32
5
6
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
RouBng in an Oracle Sharded Environment • Direct RouBng based on sharding_key
– For OLTP workloads that specify sharding_key (e.g. customer_id) during connect • A transacBon happens on single shard
– Enabled by enhancements to mid-‐Ber connecBon pools and drivers • JDBC/UCP, OCI, and ODP.NET recognize sharding keys
– PHP, Python, Perl, and Node.js leverage OCI support for sharding
• Proxy RouBng via coordinator (shard catalog) – For workloads that cannot specify sharding_key (as part of connecBon) – Applies to reporBng, batch jobs – Queries can span 1 or more or all shards
• Performed in parallel across shards (e.g. aggregates on sales data)
33
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
UCP: ConnecBon Request with a Shard Key
• Pool creaBon: no Java applicaBon code change • Shard-‐aware applicaBon gets a connecBon, using UCP Sharding APIs OracleShardingKey keyMaryEmail =
! ! ! pds.createShardingKeyBuilder() ! ! ! .subkey("[email protected]", OracleType.VARCHAR2)
.build(); !
Connection connection = ! ! ! pds.createConnectionBuilder()
.shardingKey(keyMaryEmail) .build(); !
34
An Example of New Sharding APIs
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 35
Session-‐Based RouBng via Sharding Key
• Sharding key is provided by the applicaBons at connecBon checkout
• Shard Director looks up the sharding key and redirects client to the correct shard containing the data
• Client executes SQL directly at shard
• A{er all transacBons for the given sharding key are executed, applicaBon must check-‐in the connecBon to the pool and check-‐out a connecBon for another key
ApplicaBon Server
Shard Directors
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Fast Path for key-‐based access App Tier
RouBng Tier
Data Tier
Direct RouBng via Sharding Key
• ConnecBon pool maintains the shard topology cache • Upon first connecBon to a shard
– ConnecBon pool retrieves all sharding key ranges in the shard
– ConnecBon pool caches the key range mappings
• DB request for a key that is in any of the cached key ranges goes directly to the shard (i.e., bypasses shard director)
36
ApplicaBon Server
Shard Directors
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Coordinator (shard catalog)
Proxy RouBng
• ApplicaBons connect to Query coordinator/Shard Catalog • Single Shard Queries
– SELECT CUSTID, FIRSTNAME, LASTNAME FROM CUSTOMERS WHERE CUSTID=1001;!
– Coordinator parses SQL, extracts sharding key and routes query to the correct shard
37
For Workloads that cannot pass sharding key during connecJon check-‐out
ApplicaBon Server
Shard Directors
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
UCP Sample Code
38
1. Create a connecBon pool by passing DB_URL, DB_USER, DB_PASSWORD for a sharded database.
2. Build and Pass the sharding key for a connecBon
3. Perform any Query/DML on the customer record
UCP transparently handles failover and runBme load balancing
Make sure to include ojdbc8.jar, ucp.jar and ons.jar in the classpath
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
ConnecBon URL for a sharded database
39
jdbc:oracle:thin:@ (DESCRIPTION=(FAILOVER=on)(CONNECT_TIMEOUT=5) (TRANSPORT_CONNECT_TIMEOUT=3)(RETRY_COUNT=3) (ADDRESS_LIST=(LOAD_BALANCE=ON) (ADDRESS=(HOST=orclgsm1.oracle.com)(PORT=1571)(PROTOCOL=tcp)) (ADDRESS=(HOST=orclgsm2.oracle.com)(PORT=1571)(PROTOCOL=tcp))) (ADDRESS_LIST=(LOAD_BALANCE=ON) (ADDRESS=(HOST=orclgsm3.oracle.com)(PORT=1572)(PROTOCOL=tcp)) (ADDRESS=(HOST=orclgsm4.oracle.com)(PORT=1572)(PROTOCOL=tcp))) (CONNECT_DATA=(SERVICE_NAME=oltp_rw_srvc.cust_sdb.oradbcloud)(REGION=region1)))
In the URL, specify the Shard Director listener endpoints
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Step 1: Create a connecBon pool
40
// Get the PoolDataSource for UCP PoolDataSource pds = PoolDataSourceFactory.getPoolDataSource(); Connection connection; // Set the connection factory first before all other properties pds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource"); pds.setURL(DB_URL); pds.setUser(DB_USER); pds.setPassword(DB_PASSWORD); pds.setConnectionPoolName("UCP_SHARDING_POOL"); // Default is 0. Set the initial number of connections created when UCP is started. pds.setInitialPoolSize(5); // Default is 0. Set the minimum number of connections maintained by UCP at runtime. pds.setMinPoolSize(5); // Default is Integer.MAX_VALUE (2147483647). Set the maximum number of connections allowed on the connection pool. pds.setMaxPoolSize(20); // set max connection per shard pds.setMaxConnectionsPerShard(10);!
Set the ConnecBon Pool parameters
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Step 2: Build Sharding Key for a connecBon
41
try { // Loop through this for 5 customers for (int i=0 ; i < 5 ; i++ ) { String email = emailIds[i]; // Build the Sharding key by passing the email id, which is the sharding key OracleShardingKey shardKey = pds.createShardingKeyBuilder() .subkey(email, OracleType.VARCHAR2) .build(); // Get the connection by passing the sharding key connection = pds.createConnectionBuilder()
! ! ! !.shardingKey(shardKey) .build(); // Perform a database operation doSQLWork(connection, email); } } catch (SQLException e) { System.out.println("UCPShardingSample - SQLException occurred : " + e.getMessage()); }!
Build the Sharding key
Check out a connecBon to a given shard by specifying the sharding key
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Step 3: Perform Query/DML on the customer record
42
/* This method shows the customer id, first name and last name */ public static void doSQLWork(Connection conn, String email) { try { // Prepare a statement to execute the SQL Queries. PreparedStatement prepStatement = conn.prepareStatement("SELECT CUSTID, FIRSTNAME, LASTNAME FROM CUSTOMERS WHERE CUSTID=?"); prepStatement.setString(1, email); ResultSet rsCust = prepStatement.executeQuery(); while (rsCust.next()) { System.out.print("Customer Id" + rsCust.getString("CUSTID")); System.out.print("First Name:" + rsCust.getString("FIRSTNAME")); System.out.println("Last Name:" + rsCust.getString("LASTNAME")); } } catch (SQLException e) { System.out.println("UCPShardingSample - “ + "doSQLWork()- SQLException occurred : " + e.getMessage()); } }!
Once connected, session scope is the shard
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Coordinator (shard catalog)
Proxy RouBng
• ApplicaBons connect to Query coordinator/Shard Catalog • Single Shard Queries
– E.g. SELECT CUSTID, FIRSTNAME, LASTNAME FROM CUSTOMERS WHERE CUSTID=1001;!
– Coordinator parses SQL, extracts sharding key and routes query to the correct shard
• MulB-‐Shard Queries – E.g. SELECT GEO, CLASS, COUNT(*) FROM CUSTOMERS GROUP BY GEO, ROLLUP(CLASS);
– Coordinator rewrites the query to do most processing on the shards
– Supports shard pruning and scaEer-‐gather – Final aggregaBon performed on the coordinator
43
For Workloads that cannot pass sharding key during connecJon check-‐out
ApplicaBon Server
Shard Directors
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 44
Example : MulB-‐shard Query • ROLLUP query to get region-‐wise number of customers in each class
• SELECT geo, class, COUNT(*) FROM Customers GROUP BY geo, ROLLUP(class); !• Query is executed on all shards
– First level aggregaBon and group by is pushed on all the shards – Coordinator does the second and final level of aggregaBon on the rows returned by all shards before returning results to client
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Shard Director
Centralized Schema Management
45
Shard 1
Shard 2
Shard n
Shard Catalog
connect to GDS$CATALOG service alter session enable shard ddl; create tablespace set … create user ... create sharded table … .. Create duplicated table …
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 46
Duplicated Table Maintenance
Read-‐Only Materialized Views Master Table
Shard 1
Shard 2
Shard 3
Shard Catalog
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Online AddiBon and Rebalancing of Shards
47
• A chunk is the unit of data movement when resharding
• Chunks are automaBcally rebalanced when a new shard is added
• Uses RMAN Incremental Backup & Transportable Tablespace
Add shard
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 48
Online AddiBon and Rebalancing of Shards Auto
Rebalance • A chunk is the unit of data movement when resharding
• Chunks are automaBcally rebalanced when a new shard is added
• Uses RMAN Incremental Backup & Transportable Tablespace
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Lifecycle Management of SDB • DBA can manually move or split a chunk from one shard to another • When a new shard is added, chunks are automaBcally rebalanced
– Uses RMAN Incremental Backup & Transportable Tablespace
• ConnecBon pools get noBfied (via ONS) about split, move, add/remove shards, auto-‐resharding, read-‐only – ApplicaBon can either reconnect or access read-‐only
• All MAA pracBces apply for BUR of shards and shard catalog with special consideraBons
• Can patch all shards with one command via opatchauto
• EM supports monitoring & management of SDB
49
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Sharding Overview
Sharded Database and Schema CreaBon
Data Dependent RouBng
MAA Benchmark Results -‐ Sharding on Bare Metal Cloud
Customers’ PerspecBve
CompeBBve
1
2
3
4
50
5
6
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• ObjecBve – ElasBcally scale-‐out the Shared Database
• Demonstrate linear scalability of RelaBonal transacBons
– Highest Availability with MAA on Oracle Bare Metal Cloud
• Infrastructure – Each shard hosted on dedicated server
51
Oracle Sharding Demo on Oracle Bare Metal Cloud
Oracle Bare Metal Cloud
Availability Domain 1 (100 Primary Shards)
Availability Domain 2 (100 HA Standby Shards)
Shard-‐level AcBve Data Guard (Max Availability) Component Resources per shard
CPU 36 Cores
Memory 512 GB
Flash 12.8 TB NVMe
Network 10 GbE
Demo Topology
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Sharding – a Different Way to Scale FricJonless linear scaling due to zero shared hardware or soPware
# of Shards
TPS
52
0
2,000,000
4,000,000
6,000,000
8,000,000
10,000,000
12,000,000
50 100 150 200
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle Sharding Demo on Oracle Bare Metal Cloud
• ElasBcally scaled-‐out to 200 shards on Oracle Bare Metal Cloud – Demonstrated linear scalability of RelaBonal transacBons – Demonstrated 11 Million transacBons per sec that includes:
• 4.5 Million Read-‐Write TransacBons per sec across all 100 Primary shards • 6.5 Million Read-‐Only TransacBons per sec across all 100 AcBve Standby shards
• MAA Sharding provides highest availability – Each shard is protected by Data Guard Fast-‐Start Failover across Availability Domains – Single Shard Failure resulted in 100% availability for 99% of the applicaBon
• 1% of the applicaBon experienced only 15 seconds blackout
Conclusions
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Shard Catalog Outage TesBng
54
Shard Catalog Outage Has Zero Impact on Availability for OLTP
• Outage of shard catalog has no effect on applicaBon performance (Direct RouBng)
• Ranges of sharding keys are cached within the connecBon pools
• OLTP TransacBons use direct rouBng, completely bypassing the shard catalog
• MAA Best PracBce is to protect catalog with Data Guard Maximum Availability
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Shard Directors Outage TesBng
55
Shard Director Outage Has Zero Impact on Availability
• Outage of shard directors does not affect a running connecBon pool
• ConnecBon pool caches ranges of sharding keys / shards
• MAA best pracBce to have 3 shard directors per region
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Each Shard is a Physically Separate Oracle Database Shard Outage has Zero Impact on Surviving Shards
ApplicaBon Throughput
ApplicaBon remains available on other shards
Shard Outage
AutomaBc failover quickly restores service for the shard impacted by outage
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Order Entry Service runs on Primary
• ReporBng Service runs on Standby
Role Based Global Services for AcBve Data Guard
57
AD 1 AD 2
Standby ReporJng Service
Primary Order Entry Service
AcBve Data Guard
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Order Entry Service runs on Primary
• ReporBng Service runs on Standby
• Upon Data Guard role change, Shard Directors failover services based on Role
Role Based Global Services for AcBve Data Guard
Data Center 2
58
AD 1 AD 2
Primary Order Entry Service
Standby ReporJng Service
AcBve Data Guard
GDSCTL>add service -service order_entry_service –preferred_all –role PRIMARY GDSCTL>add service -service reporting_service –preferred_all –role PHYSICAL_STANDBY
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MiBgate Unplanned Shard Outage
59
AutomaJc Failover of Client Workload to another Availability Domain
ApplicaJons using …
Oracle pools or drivers : UCP, ODP.NET, OCI, WebLogic AcJve GridLink
3rd party App Servers using UCP: IBM WebSphere, Apache Tomcat
ApplicaBon config
• Subscribe to FAN events (published by Shard Directors via ONS) by enabling Fast ConnecBon Failover (FCF)
• TNS entry to include RETRY_COUNT, CONNECT_TIMEOUT and TRANSPORT_CONNECT_TIMEOUT
Unplanned events For AcBve Data Guard: Data Guard role change
Sessions Drain
FAN posts unplanned downBme event and FCF ensues.
• New work is redirected by Shard Directors immediately
• Idle sessions are released immediately
• Checked out connecBons receive invalid connecBon; ApplicaBon closes the connecBon and gets new one from another database in the pool
AD 2 AD 1
Oracle Integrated Clients
FAN Events (RLB, Service Up/Down) via ONS
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Sharding Overview
Sharded Database and Schema CreaBon
Data Dependent RouBng
MAA Benchmark Results -‐ Sharding on Bare Metal Cloud
Customers’ PerspecBve
CompeBBve
1
2
3
4
60
5
6
Oracle Database Native Sharding: a Customer Perspective
©2016 PayPal Inc. Confidential and proprietary.
John Kanagaraj, Sr. Member of Technical Staff, PayPal Core Data Platform
61
©2016 PayPal Inc. Confidential and proprietary. 62
Sharding Principles
• Sharding is mostly a Scalability play • ”Key is Key” – Choose one STRONG key and align. E.g. Account/User ID • Data model changes (drastically!) – Normalization is relaxed • Cross-key transaction boundaries are broken • Non-shard key access can be expensive; forces “scatter-gather” pattern • CAP Theorem: 2 of 3 for (C)onsistency, (A)vailability and Network (P)artition • “Lookup” requirement for Common data elements • Scheme for mapping logical to physical is critical for future scale-out • Joins and ACID principles usually not available in Sharded systems (e.g. NoSQL)
• Oracle Sharding supports the sharding principles while providing consistency, transactions, and relational capabilities such as Joins
©2016 PayPal Inc. Confidential and proprietary. 63
Oracle Sharding – Feedback from Beta software
• What we liked • Auto-configuration (Create and Manage Shards and Routing) • Out of box support for Shard Catalog, Routing layer, Duplicated Tables • Cross-shared query support (Not available with custom sharding) • Familiar SQL for sharding management • Two level sharding (not available in other technologies such as NoSQL) • Consistent hashing to prevent large data movement as new shards are added • Dynamically changing the number of shards
• Things to be careful of • Sharding requires major rework in Data Model and Data Design • Duplicated Table – Make sure not to over-use; Not for write-heavy use cases • Enable connection pool limits per Shard
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Sharding Use Case – LinkedIn
• Target applicaBon – LinkedIn uses home-‐grown sharding with Oracle Database to scale customer facing applicaBons
• Challenges faced/key business driver – Manual re-‐configuraBon during rebalancing and new table addiBons – No mulB-‐shard queries or composite sharding capabiliBes
• How does Oracle Sharding addresses LinkedIn’s requirement? – Online addiBon of shards and automaBc rebalance with no impact to applicaBon – Architectural simplicity – Composite sharding and mulB-‐shard queries – Efficient use of CAPEX
64
Business-‐oriented Social Networking with 380 Million Members in Over 200 Countries
ORACLE SHARDING
Home Grown Sharding Architecture: LinkedIn
©2015 LinkedIn Corporation. All Rights Reserved. 65
SHARD1 SHARD2 SHARD3 SHARD4 SHARD5
ORACLE - DB01
ORACLE - DB03
ORACLE - DB02
……..
1-2,000,000 2 MM – 4 MM 4 MM – 6 MM 6 MM – 8 MM 8 MM – 10 MM
…..
txMgr txMgr txMgr txMgr txMgr
APPLICATION
ShardLocator shard.config
ORACLE SHARDING
Where LinkedIn Wants To Be - Oracle 12c Sharding
©2015 LinkedIn Corporation. All Rights Reserved. 66
….. …..
Shard Group 1 - Rep factor - 1
Data Guard Standby
…..
Shard Group 2 - Rep factor - 1
Shard Group 3 - Rep factor - 1
APP
APP
APP
…..
…..
Oracle Database Native Sharding
Sudhi Vijayakumar Database Systems Architect, Facebook Inc
Target Sharded Architecture
Various Data Sources
Shard Group 1 Shard Group 2
Primary DC
Shard Group 3
Secondary DC
Benefits of Oracle Sharding • Full power of Oracle RDBMS
• Deployments can be made automatic
• Open Compute Platform can be used for deployments
• Horizontal scaling based on demand/growth
• Reduce SPOF
• Reduce tech debt (apply patches in rolling manner, etc.)
• Uses familiar technology (Data Guard, GoldenGate)
Key Metrics for Oracle Sharding POC CPU
REG
ION
• Support Facebook’s existing and growing ingestion rate
• Response times for queries
• Ease of Deployment
• Ease of Maintenance
• Iterative design changes
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Sharding Overview
Sharded Database and Schema CreaBon
Data Dependent RouBng
MAA Benchmark Results -‐ Sharding on Bare Metal Cloud
Customers’ PerspecBve
CompeBBve
1
2
3
4
71
5
6
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle NaBve Database Sharding
• MulB-‐modal database – RelaBonal, JSON, XML, etc. • SQL and programmaBc interfaces -‐ PL-‐SQL, OCI, JDBC, etc. • Schema flexibility & developer agility with JSON within Oracle Database • BeEer consistency than NoSQL databases • Easier applicaBon maintenance – schema in database instead of applicaBon
• Enterprise-‐class features of a mature DBMS • Leverage in-‐house and world-‐wide Oracle DBA skillset • Plus superlinear scalability & extreme availability
72
Combines the best of RDBMS and NoSQL
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 73
Scalability and Fault IsolaJon with Enterprise QualiJes of Oracle RDBMS Oracle Sharding vs. NoSQL Data Stores
NoSQLs Oracle Sharding
Linear scalability and fault isolaBon
Deploy on commodity hardware on premises or on Cloud
Developer agility (e.g. flexible schema, JSON)
ReplicaBon
RelaBonal/SQL – joins, transacBons, no pre-‐defined queries, mulB-‐shard queries
ACID within shard, MVCC & strong consistency
Enterprise class features of mature DBMS (parBBoning, compression, security)
Supports MulB-‐core over SMP (Shard-‐level)
Leverage in-‐house and world-‐wide Oracle DBA skillset
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 74
Scalability and Fault IsolaJon with Enterprise QualiJes of Oracle RDBMS
Oracle Sharding vs MySQL Fabric
MySQL Fabric Oracle Sharding
System Managed Sharding
Range based Sharding
List Based Sharding
Composite Sharding
ACID within shard
MulB-‐shard queries
AcBve-‐AcBve replicaBon between shards (OGG 12.3)
GUI support for Management
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 75
NaJve Sharding Capability vs Client Library and External Tools Oracle Sharding vs Azure ElasBc Scale
Azure Oracle Sharding
System Managed Sharding
User Defined Sharding
Composite Sharding
Deployment on-‐premises
NaBve integraBon with database automates all DDLs operaBons
ACID within shard
MulB-‐shard queries support parBal results
Shards remain online while chunks are being moved
Cloud tooling for shard deployment
AcBve-‐AcBve replicaBon between shards (OGG 12.3)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle Sharding 12.2.0.1 Core Features
76
Support Matrix
Sharding Method
Supported in 12.2.0.1?
Direct RouJng Proxy RouJng AcJve Data Guard
Oracle GoldenGate
Enterprise Manager
System Managed
Yes Yes Yes Yes Yes (Requires Oracle GoldenGate 12.3)
Yes (EM 13.2.2.0 Database Plug-‐in)
Composite Yes Yes No Yes No (Planned for Oracle Database 12.2.0.2)
Yes (EM 13.2.2.0 Database Plug-‐in)
User-‐defined No
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle Sharding 12.2.0.1 Licensing -‐ Summary
77
Licensing Availability
On-‐Premises For a sharded database (SDB) with 3 or fewer primary shards, Sharding is included with EE (includes Data Guard). No limit on number of standby shards For an SDB with more than 3 primary shards, in addiBon to EE, all shards must be licensed for either AcBve Data Guard, GoldenGate or RAC
March 15, 2017 Ref: External link
Oracle Cloud (PaaS) DBCS EE and DBCS EE-‐HP: Use is limited to three primary shards; No limit on number of standby shards DBCS EE-‐EP and ECS: No limit on number of primary shards or standby shards Ref: Licensing InformaBon Manual
Automated Deployment -‐ 2nd Half of CY2017 Till then – PaaS customers will be able to deploy Sharding on DBCS instances using a White Paper
Oracle Cloud (IaaS) BYOL March 15, 2017
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Summary • Complete plaqorm for sharding an Oracle database • Ideal for mission-‐criBcal transacBon processing that require extreme level of scalability and availability than possible with a single physical database
78
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 79