8/3/2019 2004 Presentation 549
1/42
1
www.SageLogix.Com
International Oracle Users GroupLive 2004
Technical Session #549:
Understanding Indexes
Tim Gorman
Principal - SageLogix, Inc.
Email: [email protected]
8/3/2019 2004 Presentation 549
2/42
2
B*Tree index architecture
Issues with B*Tree indexes in Oracle
Real issues
Sparsely-populated indexes
Contention on INSERT
Uneven data distribution
Low data cardinality (a.k.a. selectivity)
Imagined or mythical issues
Indexes needing rebalancing
???
Some interesting options for indexing
Function-based, descending, compression
Agenda
8/3/2019 2004 Presentation 549
3/42
3
B*Tree index architecture
Oracle implements balanced B*Treeindexesas its primary indexing method
Tree structure consisting of root, branch, and leafnodes
each nodeis one database block in Oracle
rootnode is the top-level branchnode
starting point for searches into the index
branchnodes are connectors point to other branchnodes or to leafnodes
leafnodes
contain data values and pointers to rows
8/3/2019 2004 Presentation 549
4/42
4
B*Tree index architecture
root
branch branch branch branch
leaf leaf leaf leafleaf leaf leaf
Branchentries contain:
Max data value piece DBA of next level branch or leafLeafentries contain:
Data value
ROWID of table row
Index: PK_X
Table: X
8/3/2019 2004 Presentation 549
5/42
5
B*Tree index architecture
Index read I/O is sequential in nature Oracle wait-event for single-block I/O requests used during
indexed access
db file sequential read
Used with RANGE, UNIQUE, and FULL scans
FAST FULL index-scans are an exception
Uses sequential multiblock I/O similar to FULL table-scans
Conventional wisdomthat tables and indexes must be
separated to different tablespaces to enable some form ofparallel I/O is a myth
Think of how a treasure hunt is performed.
8/3/2019 2004 Presentation 549
6/42
6
B*Tree index architecture
When indexes are built on populatedtables using CREATE INDEX or ALTERINDEX xxx REBUILD
1. Table is scanned and column values aresorted
2. Leaf blocks of index are populated first
3. Then supporting branchlevels are built to
support those leaves Allblocks are filled up to PCTFREE
threshold
8/3/2019 2004 Presentation 549
7/427
B*Tree index architecture
When indexes are built transactionallyusing conventional-path INSERT orUPDATE statements
How the indexes grow is dependent upontwo possible design decisions
Optimize either for:
random data values
Or:
sequentially ascending data values
8/3/2019 2004 Presentation 549
8/428
B*Tree index architecture
If optimizing for randomdata values As blocks fill, split them in half, move upper half to
new block, leave lower half behind
Leaves behind half-full blocks
Allows back-fillingas lower data values occur
If optimizing for sequentialdata values
As blocks fill, simply overflow into a new block
Leaves behind full blocks
No need to worry about back-fillingas lowerdata values could never occur
8/3/2019 2004 Presentation 549
9/429
B*Tree index architecture
Overflow: Grows like this
Split: Grows like this
Splitting anticipates back-filling in a sorteddatastructure Therefore uses space more efficiently when
optimizingrandom data values
Overflowing uses space more efficiently in a heap(non-sorted) data structure Also uses space more efficiently when optimizingsequentialdata values
8/3/2019 2004 Presentation 549
10/4210
B*Tree index architecture
If the last index entry inserted is the highestdata value in the block
Then data will overflow into the next block on thesame level
If the last index entry inserted is not thehighest data value in the block
Then the current block will split, with the lower setof values remaining in place and the higher set ofvalues moving to the next block on the same level
8/3/2019 2004 Presentation 549
11/4211
B*Tree index architecture
If the branch block above cannotaccommodate another child
Then this algorithm is applied recursively first
If this algorithm iterates recursively all theway back to the root block, and the rootbecomes full
Then the root splits/overflows to generate a newbranch level (BLEVEL)
Root block stays in place, generates two new blocks at anew 1st BLEVEL below
8/3/2019 2004 Presentation 549
12/4212
B*Tree index architecture
Balanced B*Tree indexes grow balanced bydesign, automatically
New levels are added to the tree structure above,dynamically, as needed
Indexes do not shrinkautomatically whendeletions occur
Instead, empty index entries are left in place
For possible reuse by newly inserted data
ROWID (pointer to data) portion is set to NULL
Deletions may cause an index to become sparseand thus less efficient over time
8/3/2019 2004 Presentation 549
13/4213
Issues with B*Tree indexes
B*Tree indexes optimize:
Both the randomand sequentialdata values!
Even distribution of distinct data values
High data cardinality or selectivity of values
As a result, issues with B*Tree indexes are:
Sparseness
Resulting from data deletions
Block contention when inserting sequential data Uneven distribution of data values
Popularand unpopulardata values
Low data cardinality
8/3/2019 2004 Presentation 549
14/4214
Sparsely-populated indexes
Indexes never become unbalancedover time
If so, then why does performance sometimesdeteriorateover time?
Instead, indexes can become sparselypopulateddue to:
Deletions of row data
Unfortunate patterns of inserted random data
Sparsenesssimply means that more I/O isrequired to perform the same work
8/3/2019 2004 Presentation 549
15/4215
Sparsely-populated indexes
How is this condition detected? ANALYZE INDEX VALIDATE STRUCTURE
command
Populate session-private view named
INDEX_STATS with only one row Column PCT_USED is the average percentage of
space utilized in the blocks belonging to the index
Derived from the ratio of the value of the
columns USED_SPACE and BTREE_SPACE Expect PCT_USED to be 90 by default
Lesser values may indicate sparsenessdeveloping
Best to watch values over time
8/3/2019 2004 Presentation 549
16/4216
Sparsely-populated indexes
How is this condition detected? If PCT_USED is much less than 90
Non-zero values in the column
DEL_LF_ROWSCause of sparseness is probably row
deletions, if the value in the columnDEL_LF_ROWS is a large percentage of
value in column LF_ROWS
8/3/2019 2004 Presentation 549
17/4217
Sparsely-populated indexes
Rebuild or coalesce are two possiblesolutions for index sparseness
Dont guess!!!
Make certain using ANALYZE INDEX
VALIDATE STRUCTURE
Be aware that the ANALYZE command locksthe index, however
To rebuild an index: Use ALTER INDEX REBUILD
To coalesce space within an index:
Use ALTER INDEX COALESCE
8/3/2019 2004 Presentation 549
18/4218
Sparsely-populated indexes
ALTER INDEX REBUILD is the most commonly-used solution for sparsely-populated indexes
Unlike CREATE INDEX, uses the existing index as thesource, which is faster because:
Index is usually smaller than the table (less I/O) Index is already sorted (some sorting still needed)
Other features of ALTER INDEX REBUILD include:
Parallel execution (throw more resources at task)
Direct-path operations (no rollback/undo generated)
NOLOGGING (no redo generated)
COMPUTE STATISTICS (better CBO stats cheaply)
ONLINE (allows full usage of index during task)
8/3/2019 2004 Presentation 549
19/4219
Sparsely-populated indexes
ALTER INDEX COALESCE is anotheralternative to rebuilding
Merges unused space within indexes to free blocksfor reuse
Slower than ALTER INDEX REBUILD
because it is a transactionaloperation, not abulkoperation
No direct-path, parallel, nologging, etc But COALESCE is implicitly an ONLINE
operation
8/3/2019 2004 Presentation 549
20/4220
Contention on INSERT
Multiple concurrently-executing INSERTs intoan index on column(s) with sequentialdatavalues can also bottle-neck on buffer busywaits (a.k.a. block-level contention)
Pseudo-randomizing such data with REVERSEkey indexes relieves this performance problem
Instead of many processes attempting to insertindex entries into the right-most leaf block
Insertions are evenly scattered over all of theavailable leaf blocks
8/3/2019 2004 Presentation 549
21/4221
Contention on INSERT
[ CREATE | ALTER ] INDEX REVERSE Pseudo-randomizes non-random data
By flipping or reversing the physical orderof the data value during storage
Causes sequentialdata values to become morerandomby simply reversing data
123456 becomes 654321
123457 becomes 754321, etc...
Index data values are transparentlyconverted(flipped) and unconverted (unflipped) uponinsert and retrieval, respectively
8/3/2019 2004 Presentation 549
22/4222
Contention on INSERT
Adverse impacts of using REVERSE indexes: only equivalence operations will use the index
=, !=, , IN, and NOT IN
range-scans will not use the index
>, >=,
8/3/2019 2004 Presentation 549
23/42
23
Uneven distribution of data values
Basic rule of thumb for using indexes: Use indexes when selecting unpopulardata
Otherwise, use FULL table-scan
Or a different index? Or partitioning? Or
By default, the Oracle cost-based optimizer(CBO) assumes even data distribution
Only LOVAL, HIVAL, and DISTINCT_KEYSgathered during ANALYZE or DBMS_STATS
Real-life often invalidates this assumption
Selectivelygathering column-level statisticsprovides the CBO with information on data valueswhich are popularand which are unpopular
8/3/2019 2004 Presentation 549
24/42
24
Uneven distribution of data values
There are two commands for gathering CBOstatistics
ANALYZE TABLE FOR [ ALL | INDEXED ]COLUMNS
DBMS_STATS.GATHER_COLUMN_STATS Populates data dictionary views
DBA_TAB_HISTOGRAMS andDBA_PART_HISTOGRAMS with rows
representing buckets of data values All buckets assumed to have the same number of
rows
Each bucket is defined by its highest value
By default, only one row populated
8/3/2019 2004 Presentation 549
25/42
25
Please dont gather column-level statisticsunless the problem is proven
Gathering column-level statistics is not a goodchoice as a default operation
Perform SQL tuning with SQL Trace/TKPROF orSTATSPACK to provide proof of a problem
If the CBO fails to choose an index
it could be because it has detected low cardinality
on average If the CBO chooses an index
it could perform poorly if the data value is overlypopular
Uneven distribution of data values
8/3/2019 2004 Presentation 549
26/42
26
Illustrating the importance of data
SQL> create table t1
2 ( c1 varchar2(30),
3 c2 number,
4 c3 number
5 ) tablespace tools;
Table created.
SQL> begin
2 for i in 1..100000 loop
3 insert into t1
4 values(to_char(mod(i,187)), i, mod(i,187));5 end loop;
6 end;
7 /
PL/SQL procedure successfully completed.
8/3/2019 2004 Presentation 549
27/42
27
Illustrating the importance of data
SQL> create index i1 on t1(c1) tablespace tools;
Index created.
SQL> analyze table t1 compute statistics;
Table analyzed.
SQL> set autotrace on
SQL> select c1 from t1 where c1 = '10000';
no rows selected
Execution Plan
-----------------------------------------------------0 SELECT STATEMENT Optimizer=CHOOSE (Cost=37 Card=535
Bytes=1605)
1 0 TABLE ACCESS (FULL) OF 'T1' (Cost=37 Card=535 Bytes=1605)
8/3/2019 2004 Presentation 549
28/42
28
Illustrating the importance of data
SQL> select num_rows, blocks
2 from user_tables3 where table_name = 'T1';
NUM_ROWS BLOCKS
------------- ---------
100,000 232
SQL> select num_rows, distinct_keys,
2 avg_leaf_blocks_per_key, avg_data_blocks_per_key
3 from user_indexes where index_name = 'I1';
Avg Leaf Avg Data
Distinct Blocks Blocks
Nbr Rows Key Per Key Per Key
-------- -------- ---------- ---------
100,000 187 1 231
8/3/2019 2004 Presentation 549
29/42
29
Illustrating the importance of data
SQL> truncate table t1
Table truncated.
SQL> begin
2 for i in 1..100000 loop3 insert into t1
4 values(to_char(round(i/187,0)), i,
5 round(i/187,0));
6 end loop;
7 end;
8 /
PL/SQL procedure successfully completed.
8/3/2019 2004 Presentation 549
30/42
30
Illustrating the importance of data
SQL> create index i1 on t1(c1) tablespace tools;
Index created.
SQL> analyze table t1 compute statistics;
Table analyzed.
SQL> set autotrace on
SQL> select c1 from t1 where c1 = '10000';
no rows selected
Execution Plan
-----------------------------------------------------0 SELECT STATEMENT Optimizer=CHOOSE (Cost=2 Card=187 Bytes=561)
1 0 TABLE ACCESS (BY INDEX ROWID) OF 'T1' (Cost=2 Card=187Bytes=561)
2 1 INDEX (RANGE SCAN) OF 'I1' (NON-UNIQUE) (Cost=1 Card=187)
8/3/2019 2004 Presentation 549
31/42
31
SQL> select num_rows, blocks
2 from user_tables3 where table_name = 'T1';
NUM_ROWS BLOCKS
------------- ---------
100000 242
SQL> select num_rows, distinct_keys,
2 avg_leaf_blocks_per_key, avg_data_blocks_per_key
3 from user_indexes where index_name = 'I1';
Avg Leaf Avg Data
Distinct Blocks Blocks
Nbr Rows Key Per Key Per Key
-------- -------- ---------- ---------
100000 536 1 1
Illustrating the importance of data
8/3/2019 2004 Presentation 549
32/42
32
Low data cardinality
B*Tree indexes work best with highcardinality
Think of the example of a telephone book
Would you use the standard indexing method
to find all occurances of a name comprising25% of the book?
Or, would it be better to just scan through thebook?
8/3/2019 2004 Presentation 549
33/42
33
Low data cardinality
Bitmap indexes are extremely compact That is their primary advantage
No real magic involved
Less I/O is faster, plain and simple
Still, no matter how compact they are a FULL table scan is still likely faster than a
single bitmap index on low cardinality data
However, merging two or more bitmap indexes
brings out the real power of the mechanism
Scanning of merged bitmaps using fast low-level bitmasking operations
Sifts through huge numbers of rows quickly
8/3/2019 2004 Presentation 549
34/42
34
Low data cardinality
Bitmap indexes are designed for quickly scanning low
cardinalitydata each database block is comprised of two bitmap segments
each bitmap segmentis comprised of a bitmap, a ROWID list,and a list of distinct data values
ROWID listis a forward-compressedlist of ROWIDs, positioned according tobits in bitmap
Bitmapis organized into columnsfordistinct data values and rowsof bitsrepresenting rows
More distinct data values means lessroom for rows within each data value
ROWID list
Bitmap
Distinct values list
ROWID list
Bitmap
Distinct values list
8/3/2019 2004 Presentation 549
35/42
35
Low data cardinality
Bitmap indexes extremely slow during DML INSERT, UPDATE, and DELETE
For each change, entries within the bitmap segmentsmust be: Expanded/decoded
Manipulated Re-encoded
For multiple concurrently-executing DML changes Slower individual changes cause queueing
Only two bitmap segments per block increases contention
Because of this, bitmap indexes are most feasible onpartitioned tables in non-transactional applications Data changes should be performed in bulk using some
variation on the technique of EXCHANGE PARTITION
8/3/2019 2004 Presentation 549
36/42
36
Function-based indexes
Indexes can be based on functions andexpressions
create index xxx_ix1 on xxx(upper(c1));
create index xxx_ix2 on
xxx(upper(c1)||yadda_yadda(c2));
8/3/2019 2004 Presentation 549
37/42
37
Function-based indexes
User-defined functions usable along with standardbuilt-in functions
Restrictions for using user-definedfunctions
But functions and expressions must be deterministic
repeatable: given the same inputs, the function orexpression must always provide the same results
user-defined functions cannot accesstables or package variables
PRAGMA RESTRICT_REFERENCES(, RNDS, WNDS,RNPS, WNPS)
CREATE FUNCTION DETERMINISTIC
8/3/2019 2004 Presentation 549
38/42
38
Function-based indexes must have statisticsgathered before they can be used
System permissions
[ GLOBAL ] QUERY REWRITE needed to CREATE or ALTER
REBUILD
Parameter QUERY_REWRITE_ENABLED = TRUE to
use
Function-based indexes
8/3/2019 2004 Presentation 549
39/42
39
Descending indexes
Implemented as a form of function-basedindexes
CREATE INDEX DESC
data values in index leaf blocks are sorted in
descending order must have statistics before it will be utilized
not usable with rule-based optimizer
system permission QUERY REWRITE or GLOBAL
QUERY REWRITE needed to CREATE or ALTER REBUILD
parameter QUERY_REWRITE_ENABLED notnecessary to use
8/3/2019 2004 Presentation 549
40/42
40
Index compression
Since Oracle8i, compression has been available onB*Tree indexes:
Reduces repeated storage of prefix columndata values
Index compression
[ COMPRESS [ #-prefix-cols ] | NOCOMPRESS ]
UNIQUE: default #-prefix-colsis (#-cols)-1
NONUNIQUE: default #-prefix-colsis #-cols
Same syntax for IOT in USING clause as for indexcompression
DML is supported on compressed indexes, but it becomesmuch slower
Index compression is best used for read-mostlyor read-onlysituations
Mixing of compressed and uncompressed indexpartitions is possible
8/3/2019 2004 Presentation 549
41/42
41
Solutions for B*Tree indexes
Dont assume that there is anything wrong with a
B*Tree index Prove it first!!!
To address sparse index structures
ALTER INDEX [ REBUILD | COALESCE ]
To address block contention on INSERTs ALTER INDEX REVERSE
To address uneven data distribution
Gather column-level statistics or histograms
To address low cardinality Use bitmap indexes or no indexes at all
Additional groovystuff
Function-based, descending, and compressed indexes
8/3/2019 2004 Presentation 549
42/42
Technical Session #549
Slides, paper, and scripts downloadable from
http://www.SageLogix.com
and
http://www.EvDBT.com/papers.htm
Email: [email protected]
http://www.sagelogix.com/http://www.evdbt.com/papers.htmhttp://www.evdbt.com/papers.htmhttp://www.sagelogix.com/