Oracle Optimizer: 12c New Capabilities

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.1

Oracle safe harbor statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle.


3 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Insert Information Protection Policy Classification from Slide 8

Oracle Optimizer: 12c New CapabilitiesThomas Kytehttp://asktom.oracle.com/

ORACLEPRODUCT

LOGO


Optimizer evolution

In the beginning there were rules

RULES

Optimizer evolved to be cost based

CBO

CBO

Optimizer proactively adapts to become self-learning

Rule are not enough

Databases became more feature rich

Reactive tuning with the use of advisors and auto jobs

As environment changes Potential for plan changes

Databases become more real-time, ad-hoc environments

Reactive tuning not enough


Agenda Adaptive Query Optimization Optimizer Statistics enhancements SQL Plan Management enhancements


Adaptive PlansJoin methodsParallel distribution

methods

Adaptive Statistics Discovered at compile

timeDiscovered at run time

Adaptive Query OptimizationOverview

Adaptive Query Optimization

Adaptive Plans Adaptive Statistics

Join Methods

Parallel distribution

Methods

At compile time

At run time


Query: Find all of the products with a unit price of 15 that we have sold more that 1 of

Two possible join methods for this query

Nested Loops

Hash Join

Adaptive Execution PlansAdapt join methods


Alternative sub-plans are pre-computed

Sub-plans stored in the cursor

Stats collect inserted before join

Rows buffered until final decision is made


Table scanOrder _items

NESTED LOOPS

Index Scan Prod_info_ind

Statistics Collector

Default Plan is a nested loops join

HASH JOIN

Table scanProd_info

Rows coming out of order_items table are buffered up to a point If row count is less than the threshold use nested Loops otherwise switch to hash join


Number of rows seen in statistics collector exceeds threshold

Plan switches to hash join

Statistics collect disabled

Plan resolved on first execution & remains the same for subsequent executions


Final Plan is a hash join

Statistics collector disabled after decision is made and becomes a pass through operation

Table scanOrder _items

NESTED LOOPS

Index Scan Prod_info_ind

HASH JOIN

Table scanProd_info

Statistics Collector


Explain plan command always shows default plan

Example shows a nested loops join as default plan

No statistics collector shown in plan

Adaptive Execution PlansDisplaying the default plan


After the statement has completed use DBMS_XPLAN.DISPLAY_CURSOR to see the final plan selected

Example shows that hash join picked at execution time

Again the statistics collector is not visible in the plan

Adaptive Execution PlansDisplaying the final plan


Full adaptive plan displayed when format parameter ‘+adaptive’ is set

Example shows both the nested loops and hash join in the plan

Adaptive Execution PlansDisplaying the full adaptive plan


Additional information displayed on why operations are inactive can be seen with format parameter ‘+report’

Adaptive Execution PlansDisplaying plan with +adaptive & +report formats


New column in V$SQL IS_RESOLVED_ADAPTIVE_PLAN

Indicates statement had an adaptive plan which was resolved on first execution

Resolved plan is used for subsequent executions

Statistics collectors and buffering is disabled

Adaptive Execution PlansIndicator in V$SQL


Adaptive plans are enabled by default

Can be put in reporting mode OPTIMIZER_ADAPTIVE_REPORTING_ONLY

Reporting mode shows what would have happened during execution in the plan

Adaptive Execution PlansReporting mode



methods

Adaptive Statistics Discovered at compile timeDiscovered at run time




Join Methods


Methods

At compile

time

At run time


Dynamic Statistics

During compilation optimizer decides if statistics are sufficient to generate a good plan or not

Dynamic statistics are used to compensate for missing, stale, or incomplete statistics

They can be used for table scans, index access, joins and group bys One type of dynamic statistics is dynamic sampling


Dynamic sampling has a new level 11(AUTO)

Decision to use dynamic sampling depends on the complexity of predicate, existing statistics and total execution time

Dynamic statistics shared among queries

Dynamic Statistics Dynamic Sampling


Dynamic Statistics SGA

SHARED_POOL

Select * From product_informationWhere list_price-min_price=29And category_id not in (11,22)And prod_name like ‘Smart%’;

Optimizer checksfor existing statistics in data dictionary

2

SHARED DYNAMIC STATISTICS

DATA DICTIONARY

SQL statement is submitted

1

Optimizer determines plan using dynamic statistics

5

Cursor 0: Select * from product_information …

Dynamic sampling occurs on small number of blocks from table Resulting dynamic statistics are stored in cache

4

Statistics found but need to be augmented due to complex predicates

3

Table cardinality estimate for prod_info


Dynamic Statistics SGA

SHARED_POOL

Select supplier_id, prod_nameFrom product_informationWhere list_price-min_price=29And category_id not in (11,22)And prod_name like ‘Smart%’;

Optimizer checksfor existing statistics in data dictionary

7

SHARED DYNAMIC STATISTICS

DATA DICTIONARY

Different SQL statement is submitted with same predicates

6

Optimizer determines plan using dynamic statistics

10

Cursor 0: Select supplier_id, prod_name …

Necessary Dynamic statistics found in shared cache

9

Statistics found but need to be augmented due to complex predicates

8

Table cardinality estimate for prod_info


Adaptive Statistics

During execution optimizer estimates are compared to execution statistics

If statistics vary significantly then a new plan will be chosen for subsequent executions based on execution statistics

Re-optimization uses statistics gathered from previous executions First introduced as Cardinality Feedback in 11.2

Re-optimization


Adaptive Statistics

Statistics gathered about data volume and data type seen during execution If execution statistics vary significantly statement will be hard parsed on

the next execution using the execution statistics instead Statements are only monitored once if they don’t show significant

differences initially they won’t change in the future Only individual table cardinalities and group by estimates examined Information is stored in the cursor only and is lost if cursor ages out

Cardinality Feedback pre 12c


Initial execution of a query is monitored

Example shows initial plan is hash join between sales and customers

Cardinality estimates 8X off

Adaptive Statistics Cardinality Feedback pre 12c

Initial Cardinality estimates are more than 8X off


Execution statistics used to reparse the statement on the second execution

New plan shows correct cardinality estimates and a new join order

Information learnt is stored in the cursor only and is lost if cursor ages out

Adaptive StatisticsCardinality Feedback pre 12c

Execution Plan after the second execution

Estimates are now correct


Adaptive Statistics

Join statistics are also monitored Works with adaptive cursor sharing for statement with binds New Column in V$SQL IS_REOPTIMIZABLE Information found at execution time is persisted as SQL Plan Directives

Re-optimization in 12c


New column in V$SQL IS_REOPTIMIZABLE

Indicates that the statement will be re-parsed on the next execution

Adaptive StatisticsRe-optimization – indicator in V$SQL


Adapt join method example showed the join change from nested loops to hash join

But cursor is also marked IS_REOPTIMIZABLE

Why?

Re-optimization will occur on second execution because cardinality estimates off

Adaptive StatisticsAdaptive Plans & Re-optimization working together

Initial Cardinality estimates are off


On second execution statement is re-parsed using execution statistics

New child cursor created with new plan

New plan changes the join order

Adaptive StatisticsAdaptive Plans & Re-optimization working together


Adaptive Statistics

Directives are additional information used during optimization to generate a better plan

– For example, when table T1 is joined to T2 use dynamic statistics to get accurate cardinality estimate

– Directives are collected on query expressions not at a statement level Allows for directives to be used for multiple statements

– Persisted on disk in the SYSAUX tablespace– Directives will be automatically maintained– Managed using the new package DBMS_SPD

SQL Plan Directives


SQL Plan Directives SGA

SHARED_POOL

Select * FROM customerswhere state =‘CA’AND country=‘USA’;

SQL statement is submitted

Cursor 0: Select * from customers ……..

Optimizer determines plan2

DIRECTIVE CACHE

During execution cardinality estimate discovered to be wrong and directive created

3IS_REOPTIMIZABLE = Y

Directive: Use DS for customers table when column city, country are used

1

Sysaux Tablespace


SQL Plan DirectivesSelect * FROM customerswhere state=‘CA’AND country=‘USA’;

SQL Directives

Sysaux Tablespace

SGA

This execution and subsequent execution use child cursor 1

6

Directive: Use DS for customer table when column city, country are used

SHARED_POOLCursor 0: Select * from customers …..

IS_REOPTIMIZABLE = Y

Optimizer determines new plan using execution statistics

5

Cursor 1: Select * from customers …..IS_REOPTIMIZABLE = N

Directives periodically persisted to disk in the sysaux tablespace

Extension Binds Stats Sqlid

SQL Directives

DIRECTIVE CACHEDIRECTIVE CACHE

Same SQL statement is submitted again

4


SQL Plan DirectivesSelect emailFROM customerswhere state=‘MA’AND country=‘USA’;

New SQL statement is submitted on customers table

7

Optimizer checks for directives on customers table and finds one on the columns state and country

8

Cursor 0: Select email from customers

Optimizer determines plan with help of directive

9

Optimizer adds column group creation for state & country columns to next statistics gather on customer table

10

SGADIRECTIVE CACHE

SHARED_POOL

Sysaux Tablespace

Extension Binds Stats Sqlid

SQL Directives

DIRECTIVE CACHE


Number of SQL Plan Directives used for a statement is displayed in the notes section of the execution plan

In this case one SQL Plan Directive was used

SQL Plan DirectivesIn use


SQL Plan DirectivesMonitoring

Auto flushed every 15 minutes



Auto flushed every 15 minutes


SQL Plan Directive state was “MISSING_STATS”

Indicates Oracle can automatically gather statistics to improve the original mis-estimation

In this case a column group on STATE and COUNTRY will address the problem

SQL Plan DirectivesGather statistics to allow automatic creation of missing statistics




Correct statistics in place

Optimizer uses statistics instead of directives

Accurate cardinality estimate seen in the plan

Statement not marked for re-optimization

SQL Plan DirectivesFinal plan no longer needs the directive



methods

Adaptive Statistics Discovered at compile timeDiscovered at run time




Join Methods


Methods

At compile

time

At run time




Statistics Enhancements

New types of histograms Online statistics gathering Session level statistics for GGTs Enhanced incremental statistics Concurrent statistics gathering Automatic detection of column groups Statistic gathering reporting


Histograms

Histograms tell Optimizer about the data distribution in a Column Creation controlled by METHOD_OPT parameter Default create histogram on any column that has been used in the

WHERE clause or GROUP BY of a statement AND has a data skew Relies on column usage information gathered at compilation time and

stored in SYS.COL_USAGE$ Four types of histograms

– Frequency – Top-Frequency

Oracle Confidential

– Height balanced– Hybrid


Histograms

A frequency histogram is only created if the number of distinct values in a column (NDV) is less than 254 values

Frequency Histograms (FREQUENCY)

Frequency histogram

Oracle Confidential


Histograms

Traditionally a frequency histogram is only created if NDV < 254 But if a small number of values occupies most of the rows (>99%

rows) Creating a frequency histograms on that small set of values is very

useful even though NDV is greater than 254• Ignores the unpopular values to create a better quality histogram for

popular values Built using the same technique used for frequency histograms Only created with AUTO_SAMPLE_SIZE

Top Frequency (TOP-FREQUENCY)


Histograms

A height balanced histogram is created if the number of distinct values in a column (NDV) is greater than 254 values

Height Balanced Histograms (HEIGHT BALANCED)

Height balanced histogram


Histograms

A height balanced histogram is created if the number of distinct values in a column (NDV) is greater than 254 values

Hybrid Histograms (HYBRID)

Hybrid histogram


Histograms

Similar to height balanced histogram as created if the NDV >254 Store the actual frequencies of bucket endpoints in histograms No values are allowed to spill over multiple buckets More endpoint values can be squeezed in a histogram Achieves the same effect as increasing the # of buckets Only created with AUTO_SAMPLE_SIZE

Hybrid Histograms (HYBRID)



New types of histograms *Online statistics gathering Session level statistics for GGTs Enhanced incremental statistics Concurrent statistics gathering Automatic detection of column groups Statistic gathering reporting



New types of histograms Online statistics gathering *Session level statistics for GGTs Enhanced incremental statistics Concurrent statistics gathering Automatic detection of column groups Statistic gathering reporting





Enhanced Incremental Statistics

Incremental statistics allows global level statistics to accurately generated from partition level statistics

NDV statistics can now be accurately aggregated by the introduction of the synopsis

The synopses are stored in the Sysaux tablespace In 12c reduced the space required to store synopses on disk


NDV statistics can now be accurately aggregated

The synopses are stored in the Sysaux tablespace

12c reduces the space required to store synopses on disk

Enhanced Incremental StatisticsSales Table

May 22nd 2012

May 23rd 2012

May 18th 2012

May 19th 2012

May 20th 2012

May 21st 2012

Sysaux Tablespace

S1

S2

S3

S4

S5

S6

1. Partition level stats are gathered & synopsis created

Global Statistic

2. Global stats generated by aggregating partition level statistics and synopsis


Enhanced Incremental Statistics for Partition Exchange

DBA1. Create external table for flat files

3. Set INCREMENTAL to true & INCREMENTAL_LEVEL to TABLE

4. Gather Statistics

Sales Table

May 22nd 2012

May 23rd 2012

May 18th 2012

May 19th 2012

May 20th 2012

May 21st 20122. Use CTAS command to create non-partitioned table TMP_SALES

TMP_SALES

May 24th 2012

Sysaux Tablespace

Global Statistic

6. Global stats generated by aggregating partition level statistics for existing partition with stats on new partition

S1

S2

S3

S4

S5

S6

S75. Alter table Sales exchange partition May_24_2012 with table tmp_sales

May 24th 2012


During data load some rows going to the older partitions

In 11g any DML on older partitions triggered partition statistics to be re-gathered

New DBMS_STATS preference INCREMENTAL_STALENESS

When set to USE_STALE_PERCENT DML on less than 10% of rows in older partitions will not trigger re-gather

Enhanced Incremental Statistics Staleness Tolerance

Sales Table

May 22nd 2012

May 23rd 2012

May 18th 2012

May 19th 2012

May 20th 2012

May 21st 2012

Sysaux Tablespace

S1

S2

S3

S4

S5

S6

1. Partition level stats are gathered & synopsis created

Global Statistic

2. Global stats generated by aggregating partition level statistics and synopsis



New types of histograms Online statistics gathering Session level statistics for GGTs Enhanced incremental statistics *Concurrent statistics gathering Automatic detection of column groups Statistic gathering reporting



New types of histograms Online statistics gathering Session level statistics for GGTs Enhanced incremental statistics Concurrent statistics gathering *Automatic detection of column groups Statistic gathering reporting





DBMS_STATS.REPORT_STATS_OPERATIONS


DBMS_STATS.REPORT_GATHER_SCHEMA_STATS




SQL Plan ManagementHow it works

Parse

HJ

HJ

GB

Plan history

Plan baseline

ExecutePlan Acceptable

HJ

HJ

GB

Users

Actual execution plans stored in SQL plan baseline in Oracle Database 12c



Parse

NL

NL

GB

Plan history

Plan baseline

HJ

HJ

GB

Users

NL

NL

GB

Note you do not need to be in auto-capture mode to have a new plan added to an existing SQL plan baseline

Note additional fields such as fetches, row processed etc. are not populated because new plan never executed



Parse

Plan history

Plan baseline

ExecutePlan Acceptable

HJ

HJ

GB

Users

NL

NL

GB

HJ

HJ

GB


Adaptive SQL Plan Management

New evolve auto task running in the maintenance window– Ranks all non-accepted plans and runs evolve process for them

Newly found plans are ranked the highest– If new plan performs better than existing plan it is automatically accepted– If new plan performs worse than existing plan it will remain unaccepted– Poor performing plans will not be retried for 30 days and then only if the

statement is active

SPM Evolve Advisor


Automatic Plan Evolution

Plan history

Plan baseline

NL

NL

GB

HJ

HJ

GB

Nightly verification task checks for unaccepted plans

It then test execute to check if new plan is as good as or better than old plan

1

2

Plan history

HJ

HJ

GB

Plan baseline

NL

NL

GB

Plans which perform better than original plan are recommended to be added to the plan baseline

3L

Plans which don’t perform as good as the original plan stay in the plan history & are marked unaccepted but last_verified updated

NL

NL

GB

4


SPM Evolve AdvisorNew nightly auto task called SYS_AUTO_SPM_EVOLVE_TASK

Information on task found in DBA_ADVISOR_TASKS


You can review what happening during the task using DBMS_SPM. REPORT_AUTO_EVOLVE_TASK


Reports are persisted and can be review any time

Clearer findings section summaries the results of the evolve task

New recommendation section that clearly states how to implement the recommendations proposed

Reviewing the evolve report

Same comparison criteria as in 11g


Manually Invoking evolve task

Each task is named to allow it to be re-executed any time

Each execution is names to allow the results to be persisted

Results can be viewed at any time using the task_id and exec_id

Insert Chart Here



Summary

Optimizer begins to learn from its mistakes New types of statistics More automatic statistics gathering techniques Faster statistics gathering Multi-Table Left Outer Join Partial Join Evaluation (PJE) New Transformations

– Null-Accepting Semi Join– Scalar Subquery Unnesting


More Information White papers

– Optimizer with Oracle Database 12c– SQL Plan Management with Oracle Database 12c– Understanding Optimizer Statistics with Oracle Database 12c

Optimizer Blog– http://blogs.oracle.com/optimizer

Oracle.com– http://www.oracle.com/technetwork/database/focus-areas/bi-dat

awarehousing/dbbi-tech-info-optmztn-092214.html

http://www.oracle.com/technetwork/database/bi-datawarehousing/twp-optimizer-with-oracledb-12c-1963236.pdf

http://www.oracle.com/technetwork/database/bi-datawarehousing/twp-sql-plan-mgmt-12c-1963237.pdf

http://www.oracle.com/technetwork/database/bi-datawarehousing/twp-statistics-concepts-12c-1963871.pdf

http://blogs.oracle.com/optimizer

http://www.oracle.com/technetwork/database/focus-areas/bi-datawarehousing/dbbi-tech-info-optmztn-092214.html

http://www.oracle.com/technetwork/database/focus-areas/bi-datawarehousing/dbbi-tech-info-optmztn-092214.html


Technology

Oracle Optimizer: 12c New Capabilities