Upload
elaina-paley
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
© IBM Corporation 2007
1
Patrick Bossman
IBM Silicon Valley Lab
Tuning with Optimization Service Center
Part II
®
Columbia, MDSeptember 12, 2007
© IBM Corporation 2007
2
Agenda
• Query tuning with OSC– Statistics Advisor
– Query annotation
– Query report
– Access plan graph
– Visual plan hint
• Demo as time permits
© IBM Corporation 2007
3
Tuning a query
• Steps to tuning a query– Identify statement
• Collect information regarding SQL– Review SQL
– Physical design of referenced objects
– Statistics
– Access path
– Mentally compute, is access path efficient?
– Could we do better?
© IBM Corporation 2007
4
Identify query
• Choose a query from an existing workload– Selecting query from a workload
• Use Tune Query input options– Copy / paste / manually type within “Tune a query”
option
– Grab static SQL from the catalog
– Choose from statement cache
– Retrieve from QMF
© IBM Corporation 2007
5
Tune query from workload
• Right click on selected query
OR
• Select query Query Tools
© IBM Corporation 2007
6
Select query from workload…
© IBM Corporation 2007
7
Press Query Tools button
© IBM Corporation 2007
8
Tune query input options
• Manual input (type, copy / paste)• Use catalog search options (static SQL)• Select SQL from snap statement cache• Select from QMF
© IBM Corporation 2007
9
Welcome page
© IBM Corporation 2007
10
Tune single query
© IBM Corporation 2007
11
Choose identify target query…
© IBM Corporation 2007
12
Query options
• Top of window, select query source– Top of window varies based on selected source
• Bottom of window – Show selected SQL
– Tuning options
© IBM Corporation 2007
13
Query selection window
© IBM Corporation 2007
14
Query source
© IBM Corporation 2007
15
Query text
© IBM Corporation 2007
16
Query tuning options
© IBM Corporation 2007
17
Query menu
© IBM Corporation 2007
18
Query menu
© IBM Corporation 2007
19
Query menu
© IBM Corporation 2007
20
Choose query source…
© IBM Corporation 2007
21
Text input…
© IBM Corporation 2007
22
Source: Statement cache…
• Improved statement cache interface!– Use predefined cache views sorted in popular use
sequence– Define and save your own display of the cache
© IBM Corporation 2007
23
Other cache improvements
• Design your own cache display view– Choose filter criteria
– Choose how to sort rows retrieved from cache
– Choose which columns to display
– After rows retrieved, you can re-sort the rows displayed on the screen also
• Vastly improved statement cache interface– One-time setup, repeating use
© IBM Corporation 2007
24Cache new view optionsFilter…
© IBM Corporation 2007
25Cache new view optionssort on…
© IBM Corporation 2007
26Cache new view optionsdisplay columns…
© IBM Corporation 2007
27
Source section of window
• Double left click to see SQL text• Left click on row to select SQL• Left-click on column header to sort retrieved
records by column
© IBM Corporation 2007
28
Source: Catalog
• Build catalog search criteria as a view
© IBM Corporation 2007
29
Package filter options• Package filter options
– Collection id, name, owner, …
– Can use equals, like, in, etc.
© IBM Corporation 2007
30
Package filter options
© IBM Corporation 2007
31
Plan filter options• Plan filter options
– Plan name, plan creator, etc.
– Can use equals, like, in, etc.
© IBM Corporation 2007
32
Plan filter options
© IBM Corporation 2007
33
Cost & Object filter options• Cost and Object filters
– Filters SQL within package / plan filter– Requires the static SQL be bound with explain yes.– DSN_STATEMNT_TABLE must already be populated.– Show only SQL with PROCSU > …– Show SQL which uses table…– Show SQL which uses index…
• Choice to return rows which qualify for ANY or ALL of the cost / object filter conditions
© IBM Corporation 2007
34
Cost & Object filter options
© IBM Corporation 2007
35
Access path filters• Access path filters
– Filters SQL within package / plan filter
– Requires the static SQL be bound with explain yes.
– PLAN_TABLE must already be populated.
• Show SQL which performs…– Tablespace scan
– Sort
– Non-matching index scan
– List prefetch
– Outer join
– ….
© IBM Corporation 2007
36
Access path filter options
© IBM Corporation 2007
37
Catalog SQL list
• Left click on row to select…
© IBM Corporation 2007
38
Source QMF / QMF HPO
© IBM Corporation 2007
39
QMF search options
• Searching for Bob’s QMF queries…
© IBM Corporation 2007
40
Source: File
© IBM Corporation 2007
41
Source: Categorized SQL
© IBM Corporation 2007
42
Source: Plan Table
© IBM Corporation 2007
43
Plan table sort options
© IBM Corporation 2007
44
Plan table display options
© IBM Corporation 2007
45
Plan table display options
• Plan table comments– OSC will map plan_table to SYSPACKSTMT / SYSSTMT– Explain row selected must map to currently bound version of
SQL to retrieve SQL statement text– Ease of use TIP: Make sure you also sort by BIND_TIME
DESC if you have history
© IBM Corporation 2007
46
After you’ve selected a query…
• Query options– Run SQL
– Categorize SQL
• Advisors– Run statistics advisor
• Tools– Query annotation
– Query graph
– Visual Plan Hint
– Query report
© IBM Corporation 2007
47
Query menu
© IBM Corporation 2007
48
Categorize query
• Categorize and store SQL you work with
© IBM Corporation 2007
49
Categorize query
© IBM Corporation 2007
50
Run query(just click run…)
© IBM Corporation 2007
51
Run query output
© IBM Corporation 2007
52
Statistics Advisor
• “Why” statistics advisor outside scope of this session – Covered somewhat in part I
– Covered as session in and of itself.
• Run Statistics Advisor• Statistics Advisor Options
© IBM Corporation 2007
53
FYI - Switching SQL…
SELECT X.DISCOUNT
, SUM(X.QUANTITY) AS SUMQ
, AVG(X.QUANTITY * X.PRICE) AS AVGP
FROM LITEM_UIV_NOGBY AS X
WHERE X.QUANTITY <= 5
AND X.DISCOUNT > 0.08
AND ORDERKEY >30000000
GROUP BY X.DISCOUNT
© IBM Corporation 2007
54
Generate targeted statistics…
• Identify interesting objects– Tables– Indexes
• Identify interesting columns, column groups– Predicate analysis
• Which column (groups) have predicates?
• Which column (groups) are indexed?
• Which statistics should I collect?– Column statistics– Correlation statistics– Frequencies– Histograms
– Statistic needed depends on predicate types (range, equal)
© IBM Corporation 2007
55
Run Statistics Advisor
© IBM Corporation 2007
56
Run Statistics Advisor
© IBM Corporation 2007
57
Statistics Advisor Options
• General options– Enable / disable certain analysis options
– Histogram predicate types
• Typical default list– In addition to DDL defined defaults
• Conflict thresholds
© IBM Corporation 2007
58
Statistics Advisor options
© IBM Corporation 2007
59
SA general options
© IBM Corporation 2007
60
SA general options
© IBM Corporation 2007
61
RUNSTATS collection parameters
© IBM Corporation 2007
62
Report type
© IBM Corporation 2007
63
Defaults and conflicts
• Typical default list– Default values typically skewed– If SA observes a “typical default” or a DDL defined default
value within a predicate, it will trigger frequent value collection
• Conflict thresholds– Statistics Advisor looks for inconsistent statistics– Could be inconsistent collection strategy
• Collect table / index statistics at different intervals• Collect statistics on indexes on same columns at different times
– Minor inconsistencies OK and can naturally occur even with consistent collection strategies
• Index cardinalities more accurate than column collected cardinalities (simulation)
© IBM Corporation 2007
64
Predefined defaults
© IBM Corporation 2007
65
Conflict threshold (1)
© IBM Corporation 2007
66
Conflict threshold (2)
© IBM Corporation 2007
67
Query Annotation
© IBM Corporation 2007
68
Query annotation
• Original SQL– Format SQL
– Annotate SQL with statistics
– Helpful highlighting
– Expand / collapse query sections
© IBM Corporation 2007
69
Original SQL
SELECT S_SUPPKEY, S_NAME , SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)) AS REVENUE FROM SYSADM.ORDER , SYSADM.LINEITEM , SYSADM.SUPPLIER WHERE L_ORDERKEY = O_ORDERKEY AND S_SUPPKEY = L_SUPPKEY AND O_ORDERDATE >= DATE('1993-10-01') AND O_ORDERDATE < DATE('1993-10-01') + 3 MONTH AND L_RETURNFLAG = 'R' AND S_NATIONKEY IN (1,2,3,4,5) AND S_SUPPKEY < 10000 GROUP BY S_SUPPKEY, S_NAME HAVING SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)) > 1000000 ORDER BY REVENUE DESC QUERYNO 3
© IBM Corporation 2007
70
FormattedSELECT SYSADM.SUPPLIER.S_SUPPKEY , SYSADM.SUPPLIER.S_NAME , SUM( SYSADM.LINEITEM.L_EXTENDEDPRICE * ( 1 -
SYSADM.LINEITEM.L_DISCOUNT ) ) AS REVENUEFROM SYSADM.SUPPLIER , SYSADM.ORDER , SYSADM.LINEITEMWHERE ( SYSADM.LINEITEM.L_ORDERKEY = SYSADM.ORDER.O_ORDERKEY AND SYSADM.SUPPLIER.S_SUPPKEY = SYSADM.LINEITEM.L_SUPPKEY AND SYSADM.LINEITEM.L_RETURNFLAG = 'R' AND SYSADM.ORDER.O_ORDERDATE < ( DATE( '1993-10-01' ) + 3 MONTHS ) AND SYSADM.ORDER.O_ORDERDATE >= DATE( '1993-10-01' ) AND SYSADM.SUPPLIER.S_NATIONKEY IN ( 1, 2, 3, 4, 5 ) AND SYSADM.SUPPLIER.S_SUPPKEY < 10000 )GROUP BY SYSADM.SUPPLIER.S_SUPPKEY , SYSADM.SUPPLIER.S_NAMEHAVING SUM( SYSADM.LINEITEM.L_EXTENDEDPRICE * ( 1 -
SYSADM.LINEITEM.L_DISCOUNT ) ) > 1000000ORDER BY REVENUE DESC
© IBM Corporation 2007
71
AnnotatedSELECT SYSADM.SUPPLIER.S_SUPPKEY , SYSADM.SUPPLIER.S_NAME , SUM( SYSADM.LINEITEM.L_EXTENDEDPRICE * ( 1 - SYSADM.LINEITEM.L_DISCOUNT ) ) AS REVENUEFROM SYSADM.SUPPLIER CARDF=10,000 QUALIFIED_ROWS=1,526.2466 NPAGESF=402 , SYSADM.ORDER CARDF=1,500,000 QUALIFIED_ROWS=57,114.59 NPAGESF=42,417 , SYSADM.LINEITEM CARDF=4,254,339 QUALIFIED_ROWS=1,048,737.0 NPAGESF=141,743WHERE ( SYSADM.LINEITEM.L_ORDERKEY = SYSADM.ORDER.O_ORDERKEY COLCARDF=1,063,455/1,500,00
MAX_FREQ=0.0001645%/0.000066666666FF=6.666666649834951E-7
AND SYSADM.SUPPLIER.S_SUPPKEY = SYSADM.LINEITEM.L_SUPPKEY COLCARDF=10,000/10,000 MAX_FREQ=99.99%/
0.011917244958617542% FF=9.99999901978299E-5
AND SYSADM.LINEITEM.L_RETURNFLAG = 'R' COLCARDF=4MAX_FREQ=50.75460606218732%
FF=0.24650996923446655
AND SYSADM.ORDER.O_ORDERDATE < ( DATE( '1993-10-01' ) + 3 MONTHS ) COLCARDF=2,304 MAX_FREQ=0.0414% FF=0.30288922786712646 LOW2KEY=1992-01-02 HIGH2KEY=1998-
08-01 AND SYSADM.ORDER.O_ORDERDATE >= DATE( '1993-10-01' ) COLCARDF=2,304
MAX_FREQ=0.0414% FF=0.7351873517036438 LOW2KEY=1992-01-02 HIGH2KEY=1998-
08-01 AND SYSADM.SUPPLIER.S_NATIONKEY IN ( 1, 2, 3, 4, 5 ) COLCARDF=25
MAX_FREQ=4.21%FF=0.1526399850845337
AND SYSADM.SUPPLIER.S_SUPPKEY < 10000 COLCARDF=10,000 MAX_FREQ=99.99% FF=0.9998999834060669 LOW2KEY=2 HIGH2KEY=9999
)GROUP BY SYSADM.SUPPLIER.S_SUPPKEY , SYSADM.SUPPLIER.S_NAMEHAVING SUM( SYSADM.LINEITEM.L_EXTENDEDPRICE * ( 1 - SYSADM.LINEITEM.L_DISCOUNT ) ) > 1000000 FF=0.3333333134651184ORDER BY REVENUE DESC
© IBM Corporation 2007
72
Annotated within OSC
© IBM Corporation 2007
73
Annotation table details
• Table annotations– CARDF – number of rows in the table
– NPAGESF – number of pages
– Qualified rows – optimizer estimated number of rows after local filtering (as though table is the outer table)
• Value– Poor join sequence often due to inaccurate filtering estimate at
the single table level
– Qualified rows shows optimizer estimate, is it accurate?
– CARDF, NPAGESF – useful to have this embedded as you’re looking at SQL and predicates
© IBM Corporation 2007
74
Annotation table close-upsSELECT cols…
FROM SYSADM.SUPPLIER
CARDF=10,000
QUAL_ROWS=1,526.2466
NPAGESF=402
, SYSADM.ORDER
CARDF=1,500,000
QUAL_ROWS=57,114.59
NPAGESF=42,417
, SYSADM.LINEITEM
CARDF=4,254,339
QUAL_ROWS=1,048,737.0
NPAGESF=141,743
© IBM Corporation 2007
75
Annotation predicate details
• Column information– COLCARDF – number of distinct values for a
column. For join predicate, both COLCARDF’s shown
• Show selectivity of each predicate
– Max Freq – Maximum frequency (if collected)• Skew statistics collected?
• Show most skewed frequency (indicator of degree of skew)
© IBM Corporation 2007
76
Annotation predicate details
• Column information (cont.)– FF (filter factor) – optimizer single predicate filter
factor estimate• Show optimizer filtering estimate• Is it accurate? (Compare to actual filtering)
– LOW2KEY / HIGH2KEY – shown for range predicates.
• Observe literal value with “domain” of actual values• Can spot stale columns statistics…• Eg. HIGH2KEY = 2006-11-01, you’ve got 2007 data in the
table…• WHERE SALES_DT BETWEEN 2007-01-01 AND 2007-
01-31
© IBM Corporation 2007
77
Annotation predicate close-up( SYSADM.LINEITEM.L_ORDERKEY = SYSADM.ORDER.O_ORDERKEY
COLCARDF=1,063,455/1,500,00MAX_FREQ=0.0001645%/0.000066666666FF=6.666666649834951E-7
AND SYSADM.SUPPLIER.S_SUPPKEY = SYSADM.LINEITEM.L_SUPPKEY COLCARDF=10,000/10,000 MAX_FREQ=99.99%/0.0119% FF=9.99999901978299E-5
AND SYSADM.LINEITEM.L_RETURNFLAG = 'R' COLCARDF=4MAX_FREQ=50.75460606218732%
FF=0.24650996923446655
AND SYSADM.ORDER.O_ORDERDATE < ( DATE( '1993-10-01' ) + 3 MONTHS ) COLCARDF=2,304 MAX_FREQ=0.0414%FF=0.30288922786712646 LKEY=1992-01-02 HKEY=1998-08-01
AND SYSADM.ORDER.O_ORDERDATE >= DATE( '1993-10-01' ) COLCARDF=2,304 MAX_FREQ=0.0414%FF=0.7351873517036438 LOW2KEY=1992-01-02HIGH2KEY=1998-08-01
AND SYSADM.SUPPLIER.S_NATIONKEY IN ( 1, 2, 3, 4, 5 ) COLCARDF=25 MAX_FREQ=4.21%FF=0.1526399850845337
AND SYSADM.SUPPLIER.S_SUPPKEY < 10000 COLCARDF=10,000 MAX_FREQ=99.99% FF=0.9998999834060669 LOW2KEY=2 HIGH2KEY=9999
© IBM Corporation 2007
78
General annotation comments
• SQL analysis– Start at the table level
• Identify any tables where qualified rows estimate looks off
– Examine predicates on that table• Any individual predicates look too
– Optimistic?– Pessimistic?– Frequencies, histograms?
• After individual predicates are OK, any unaddressed correlation?
– KEYCARD, COLGROUP?
– Difficult to estimate predicate?• WHERE SUBSTR(C1,1,4) = ‘ABCD’
© IBM Corporation 2007
79
Query annotation transformed SQL
• Transformed SQL– All original SQL annotation benefits plus
– Show optimizer rewritten SQL• Predicate transitive closure
• Predicate transformations
• Distribution and pruning
• Merge / materialization
© IBM Corporation 2007
80
Query annotation transformed SQL
• Transformed SQL– Show SQL before / after distribution
– Before / after pruning
© IBM Corporation 2007
81
Untransformed UIV SQL
– Where predicates referencing the view
© IBM Corporation 2007
82
Transformed UIV SQL
Hey, where are the predicates?!?
© IBM Corporation 2007
83
Transformed UIV SQL
– Predicates distributed into each leg of the union all view
– Wonder what transformations optimizer did with your query?
– Wonder no more…
© IBM Corporation 2007
84Untransformed UIV5 union all legs
1
2
3
4
5
© IBM Corporation 2007
85Transformed UIV SQL3 union all legs
1
2
3
© IBM Corporation 2007
86
Query report
• Easy to read report which shows– Database / table space
– Table
– Index
– Indexed columns
– Column group cardinality
• Quick reference– Statistics on objects
– Statistics on indexed columns
– KEYCARD statistics mapped into index
© IBM Corporation 2007
87
Query Report
© IBM Corporation 2007
88
New SQL
SELECT <cols>FROM SYSIBM.SYSPACKSTMT AS A , SYSIBM.SYSPACKAGE AS CWHERE ( A.QUERYNO > -1 AND C.COLLID = 'TEST' AND C.NAME = 'PGM1' AND A.COLLID = C.COLLID AND A.LOCATION = C.LOCATION AND A.NAME = C.NAME AND A.VERSION = C.VERSION )ORDER BY <cols>
© IBM Corporation 2007
89
Query report table 1
© IBM Corporation 2007
90
Query report table 2
• Looks like someone forgot to run RUNSTATS…
• Hey… this is MY database!
© IBM Corporation 2007
91
Statistics advisor says…
RUNSTATS TABLESPACE DSNDB06.SYSPKAGE
TABLE(SYSIBM.SYSPACKSTMT)
COLUMN(NAME,QUERYNO,VERSION,COLLID)
TABLE(SYSIBM.SYSPACKAGE)
COLUMN(NAME,VERSION,COLLID)
INDEX(SYSIBM.DSNKSX01 KEYCARD
,SYSIBM.DSNKKX02 KEYCARD
,SYSIBM.DSNKKX01 KEYCARD)
SHRLEVEL CHANGE REPORT YES
© IBM Corporation 2007
92
Query report table 1
• Nice to see cascading MCARD’s– Only matching on 2 or 3 columns?
– Easy to see the KEYCARD bound…
© IBM Corporation 2007
93
Query report table 2
• Easy to read index, indexed columns report.• Observations…
– Clusterratiof of DSNKKX01? 0.795 (reorg?)
– COLCARDF of CONTOKEN?• Not used as a predicate!
© IBM Corporation 2007
94
Access Plan Graph
• Visual explain– Ported from Visual Explain product
– Graphically display access path
© IBM Corporation 2007
95
Access plan graph
© IBM Corporation 2007
96
Access plan graph
© IBM Corporation 2007
97
Access plan graph
© IBM Corporation 2007
98
Query graph overview
• Reading query graph– Read from bottom up
– Left to right
© IBM Corporation 2007
99
Access plan graph
© IBM Corporation 2007
100
Access plan graph zoom
© IBM Corporation 2007
101
Query graph zoom reading
• Access index– Plan_table_IDX1
– Qualifies 2 rows
• Fetch from table– Plan table has 930 rows
– Approximately 1.6 survive
– Inner table is accessed approximately 1.6 times
© IBM Corporation 2007
102
Access plan graph zoom
© IBM Corporation 2007
103
Descriptor
© IBM Corporation 2007
104
Descriptor
© IBM Corporation 2007
105
Descriptor
• Displays detailed information– Descriptor changes as you navigate plan graph
– Click on index node for index information
– Click on table node, table information shown
– Double click node will pop-up independent descriptor window for node
© IBM Corporation 2007
106
Index descriptor
• Index node information– Index creator, name
– Statistics on index
– Uniqueness
– Clustering attribute
– …
© IBM Corporation 2007
107
Index scan descriptor
• Index scan information– Index cardinality
– Index leaf pages
– Matching predicates• Individual matching predicate filter factor
• Bounded matching predicate filter factor
– Total index filtering (matching and screening)
– Estimated scanned leaf pages
– Estimated index qualified rows
© IBM Corporation 2007
108
There’s more…
• Other information…– Identify matching, screening, S1, S2 predicates– Filter factors for predicates– Qualified partitions– Parallelism
• Key range • Page Range• Degree of parallelism
– Sort keys, sort key length– Estimated number of records at
• Index matching, total index filtering• Data access• Sort input / output• Join
© IBM Corporation 2007
109
Demo query graph as time permits
© IBM Corporation 2007
110
Textual explain view…
© IBM Corporation 2007
111
Textual explain view…
• OSC shows plan_table rows• Short / long format• … this is obviously a truncated view…
© IBM Corporation 2007
112
Visual Plan Hint
• Visual Plan Hint– Graphically alter the explain table
• Will show an overview– More details included as part of
– Optimization hints presentation• http://www.bwdb2ug.org/PDF/Optimization%20Hints.pdf
© IBM Corporation 2007
113Visual Plan HintVisual Plan Hint
© IBM Corporation 2007
114Change join sequenceChange join sequence
© IBM Corporation 2007
115Join connectors removedJoin connectors removed
© IBM Corporation 2007
116Move the tables aroundMove the tables around
© IBM Corporation 2007
117Create, connect nodesCreate, connect nodes
© IBM Corporation 2007
118Validate hintValidate hint
Hint
© IBM Corporation 2007
119Validate hint(will show screen shot)
© IBM Corporation 2007
120
Deploy hint
Input query level settings
© IBM Corporation 2007
121
Visual Plan Hint Summary
• Visual Plan Hint– Graphically alter the explain table
– Use of drop down lists reduces errors
– Validation of optimization hint built in
– Colorful compare of before / after access path, very nice
© IBM Corporation 2007
122
OSC Summary
• Optimization Service Center overview– Support for workloads, workload advisors– Monitoring allows for profiling of applications and exception
generated workloads– Consolidated inputs / outputs– Improved statement cache display (allow customized views)
• Improved query tuning tools– Statistics advisor– Annotation– Query report– Access plan graph– Visual plan hint
© IBM Corporation 2007
123
Patrick Bossman
Tuning with Optimization Service Center II
®