Effiziente Abfragen mit Oracle Database In-Memory · Oracle Database In-Memory Architecture Enterprise Edition option available as of 12.1.0.2 Row Format vs. Column Format –Row

Effiziente Abfragen mitOracle Database In-MemoryDOAG Webinar, 8. Februar 2019Dani Schnider, Trivadis AG

@dani_schnider DOAG2018@dani_schnider danischnider.wordpress.com

Dani Schnider

2

Working for Trivadis in Glattbrugg/Zurich– Senior Principal Consultant– Data Warehouse Lead Architect– Trainer of several CoursesCo-Author of the Books– Data Warehousing mit Oracle– Data Warehouse BlueprintsCertified Data Vault Data ModelerOracle ACE

08.02.2019 DOAG Webinar: Oracle In-Memory

@dani_schnider danischnider.wordpress.com

Oracle Database In-Memory Architecture

DOAG Webinar: Oracle In-Memory3 08.02.2019


Enterprise Edition option available as of 12.1.0.2

Row Format vs. Column Format

– Row format is optimal for OLTP

– Column format is optimal for analytical processing

Data is populated in both formats in memory

Source: Oracle Whitepaper twp-oracle-database-in-memory-2245633.pdf

08.02.2019 DOAG Webinar: Oracle In-Memory4



SGA

Buffer Cache IM Column Store TX Journal

DBWn User IMCO Wnnn

Row Format

C5C1 C2 C3 C4

SELECTUPDATE

Advantages of Oracle Database In-Memory


Support of mixed workloads (OLTP / DWH)

Optimized for analytical queries and real-time reporting

Fast queries and aggregations on large data sets

Column-wise compression of data in memory

Only required columns are read

Less effort needed for performance optimization

No application changes necessary

Basic Configuration – Very Simple

INMEMORY_SIZE initialization parameter controls the In-Memory area

INMEMORY_SIZE ≥ 100MB

– As of 12.2, the size can be increased without restarting the database

COMPATIBLE ≥ 12.1.0.0.0

SQL> ALTER SYSTEM SET inmemory_size=16G SCOPE=spfile;

SQL> SELECT name, value FROM v$sga;

NAME VALUE------------------- ---------------Fixed Size 5304248Variable Size 11274289224Database Buffers 108716359680Redo Buffers 263131136In-Memory Area 17179869184

08.02.2019 DOAG Webinar: Oracle In-Memory7

In-Memory FastStart


SGA

Buffer Cache IM Column Store TX Journal

DBWn User IMCO Wnnn

Row Format

C5C1 C2 C3 C4

FastStart Area

BEGINDBMS_INMEMRY_ADMIN.FASTSTART_ENABLE (<tablespace>);

END;

Many Enhancements in Oracle 12.2 and 18c


In-Memory Expressions

In-Memory Virtual Columns

In-Memory FastStart

Automatic Data Optimization Support for In-Memory Column Store

Join Groups

Expression Tracking

Oracle Database In-Memory Support on Oracle Active Data Guard

In-Memory Column Store Dynamic Resizing

Dynamic Capture Window for In-Memory Expressions

Automatic In-Memory

Database In-Memory Support for External Tables

Flexible Parallelization Using In-Memory Dynamic Scans

In-Memory Optimized Arithmetic

18c

Typical Use Case: Star Schema


Typical Use Case: Star Schema


Reasons:

Query optimization offrequently used queries

Star Schema is an ideal candidate for In-Memory Column Store

Only required columns must be read and aggregated

Data WarehouseMartsCleansing Area CoreStaging Area

Metadata

Physical Design of Data Marts (classically)


Dimension Tables:– Primary key constraint

– ev. Bitmap indexes on filter columns

Fact Tables:– Partitioning (usually by date)

– Foreign key constraints to dimension tables

– Bitmap index for each dimension key

– ev. Additional bitmap join indexes

Optimizer Statistics– Refresh of statistics after each ETL run

Materialized Views– Materialized Views for frequently used aggregation

levels

– Dimension objects for hierarchies on dimensiontables

– ev. Materialized View Logs on Core tables

– ev. Indexes on Materialized Views

– ev. Partitioning of Materialized Views

Physical Design of Data Marts (with In-Memory)


Dimension Tables:– Primary key constraint

Fact Tables:– Partitioning (usually by date)

– Foreign key constraints to dimension tables

– ev. Partial bitmap indexes

Optimizer Statistics– Refresh of statistics after each ETL run

In-Memory Column Store– All dimension tables

– Small fact tables (complete)

– For large fact tables ev. only frequently usedmeasures / dimension keys

– For partitioned fact tables ev. only current (orfrequently used) partitions

In-Memory Compression Type– Typically MEMCOMPRESS FOR QUERY

(LOW or HIGH)

In-Memory Compression Type


In-Memory Compression Type


Compression Level Description

NO MEMCOMPRESS Data is populated without any compression

MEMCOMPRESS FOR DML Minimal compression optimized for DML performance

MEMCOMPRESS FOR QUERY LOW Optimized for query performance (default)

MEMCOMPRESS FOR QUERY HIGH Optimized for query performance as well as space saving

MEMCOMPRESS FOR CAPACITY LOW Balanced with a greater bias towards space saving

MEMCOMPRESS FOR CAPACITY HIGH Optimized for space saving


Demo 1

a) Populate In-Memory Column Storeb) In-Memory Compression Types

Query Performance with In-Memory


In-Memory Performance Features


In-Memory Scan– Column-wise reading of data from IM Column Store

– Aggregation to required granularity

In-Memory Join (= Join Filter Pruning)

– Efficient method to join tables

– Use of „Bloom Filters“

In-Memory Aggregation (= Vector Transformation)

– Comparable with Star Transformation

– Use of „Key Vectors“ instead of bitmap indexes

In-Memory Scans


Only required columns are scanned

– Internal In-Memory Storage Index

In-Memory Expressions

– Automatical detected expressionsvia Expression Statistics Store (ESS)

In-Memory Virtual Columns

– Must explicitly be loaded in IMCS

See blog post “Derived Measures and Virtual Columns”https://danischnider.wordpress.com/2017/04/10/derived-measures-and-virtual-columns/

In-Memory Joins


Based on Hash Join

Bloom Filter

– Bit array on join column

– Result is false positive

Create Bloom Filter for build input table

Use Bloom Filter as additional filter on probe input

Read build input

Read probe input

Hash Table

Bloo

m F

ilter

In-Memory Aggregation (Vector Transformation)


Phase 1 (for each dimension with filter criteria)

1. Scan on dimension table (including data filtering)

2. Build key vector

3. Aggregation of data (In-Memory Accumulator)

4. Create temporary table

Phase 25. Full table scan on fact table, filtering using key vectors

6. Aggregation using HASH GROUP BY / VECTOR GROUP BY

7. Join with temporary tables (Join Back)

8. Ev. join with additional dimensions (without filter criteria)



FACTS11 22 100011 24 120012 21 30012 22 320012 24 70013 22 110014 21 200014 24 80014 25 160014 26 70015 23 110015 24 120015 26 50016 22 240016 23 80017 22 130017 25 110018 21 90018 24 210018 26 600

DIM111 Alpha12 Alpha13 Beta14 Beta15 Beta16 Gamma17 Delta18 Delta

DIM221 X green22 X blue23 Y green24 Y blue25 Y red26 Z red

SELECT D1, D21, D22, SUM(FACTS.F)

FROM FACTSJOIN DIM1 ON (...)JOIN DIM2 ON (...)WHERE D1 IN ('Beta',

'Gamma')AND D21 = 'Y'

GROUP BY D1, D21, D22



FACTS11 22 100011 24 120012 21 30012 22 320012 24 70013 22 110014 21 200014 24 80014 25 160014 26 70015 23 110015 24 120015 26 50016 22 240016 23 80017 22 130017 25 110018 21 90018 24 210018 26 600

DIM111 Alpha12 Alpha13 Beta14 Beta15 Beta16 Gamma17 Delta18 Delta

DIM221 X green22 X blue23 Y green24 Y blue25 Y red26 Z red

KV100111200

KV2001230

TMP11 Beta2 Gamma

TMP21 Y green2 Y blue3 Y red

0 00 20 00 00 21 01 01 21 31 01 11 21 02 02 10 00 30 00 20 0

Beta Y green 1100Beta Y blue 2000Beta Y red 1600Gamma Y green 800



In-Memory Aggregation is used when

– All tables are populated into In-Memory Column Store

– Dimension tables are small (< 10% of fact table)

– Fact table contains > 10 million rows

Enforce In-Memory Aggregation

– Hint /*+ vector_transform */

– “_always_vector_transformation” = TRUE

Analytic Views always use vector transformation– See blog post “Analytic Views: A Performance Deep Dive Analysis”

https://danischnider.wordpress.com/2017/09/09/analytic-views-a-performance-deep-dive-analysis/


Demo 2

a) In-Memory Scanb) In-Memory Joinc) In-Memory Aggregation

In-Memory Column Store is not unlimited


In-Memory Configuration for Individual Tables


Complete Table

ALTER TABLE salesINMEMORY

ALTER TABLE salesMODIFYPARTITION p_2016INMEMORY

ALTER TABLE salesINMEMORY NO INMEMORY(cust_id,time_id)

Individual Columns

Individual Partitions

In-Memory Configuration for a Star Schema


Typical Configuration:

Dimension Tables

– Completely loaded inIn-Memory Column Store

Fact Table

– Partitioned by date column

– Current (or frequently used) partitions in In-Memory Column Store

– History Partitioned not inIn-Memory Colum Store

In-Memory & Partial Local Indexes


TablePartition P1

IndexPartition P1

TablePartition P2

IndexPartition P2

TablePartition P3

IndexPartition P3

TablePartition P4

IndexPartition P4

TablePartition P5INMEMORY

IndexPartition P5

TablePartition P6INMEMORY

IndexPartition P6

INDEXING ON INDEXING OFF


Demo 3

a) In-Memory for Individual Columnsb) In-Memory & Partial Local Indexes

Oracle Database In-Memory & Data Warehouse


Oracle Database In-Memory:Powerful Performance Features for Query Performance

Perfect for Analytical Queries in Data Warehouses

No Impact on Architecture of Data Warehouse

To Be Considered:Physical Database Design must be Adapted

Important to Understand how In-Memory Works

Many Enhancements in Oracle 12.2 and 18c

Documents

Effiziente Abfragen mit Oracle Database In-Memory · Oracle Database In-Memory Architecture Enterprise Edition option available as of 12.1.0.2 Row Format vs. Column Format –Row