Maximizing Database Tuning in SAP SQL Anywhere

(c) 2015 Independent SAP Technical User Group Annual Conference, 2015

SQL Anywhere Performance

and Tuning Jason Hinsperger

Sr. Product Manager

SAP


Agenda

Performance Analysis

Example of Analysis

Graphical Plan Viewer

Performance Timing Utilities

Types of Performance Problems

Communication Patterns

I/O Bound Applications

CPU-Bound Applications

Concurrency Bound Applications

Performance Tips


PERFORMANCE ANALYSIS


Primary SQL Anywhere Performance Factors

• Application patterns

• Server cache size

• Current cache contents (cold, warm, hot)

• Server multiprogramming level (-gn/gnh command line switch)

• Number of CPUs available for server thread execution (license dependent)

• Database working set size

• Speed and configuration of the server’s disk subsystem

• Workload characteristics: transaction inter-arrival rate, number of connections, workload mix

• Database page size

• There can be other secondary factors

4


Detecting Performance Problems Systematic approach to the problem: “My application is slow – what do I do!?”

Poor performance happens because some resource is maxed out

o Limiting resource at machine-level:

I/O bandwidth

CPU cycles

o Machine might have more I/O or CPU available in parallel

But server might not be able to use it in parallel concurrency-bound

Several types of bottlenecks:

o Client Application Issues

o IO-Bound

o CPU-Bound

o Concurrency-Bound


SQL Anywhere Performance Analysis Tools

SQL Anywhere Profiling tool

Combines functionality of several separate tools (which can also be used individually)

o Request logging

o Procedure profiling

o Graphical plan capturing

o Index consultant

o Statistics monitoring

Can help determine whether problem is occurring at server or client

Event tracing can be more targeted

Can trace specific events of interest (including graphical plans)

Can use sp_read_etd() from within the server


Agenda


Example of Analysis








Performance Tips


Every Sunday at 09:00, everything gets slow!

Identify what statements are executing during the problem period

Compare min/avg/max times for “good” periods with “problem” period

Use sa_conn_info and sa_conn_activity, sa_performance_diagnostics

Look for statements, ReqTimeBlockIO, ReqCountBlockLock,

ReqCountBlockContention

Find procedure / statement with the biggest difference from good to bad

Look for locking issues

Consider other activity on server or network (backups, reconfiguring routers, …)

Compare execution plans

Why would the optimizer ever choose a sequential scan?

Look at predicate selectivity, cache contents

Look at index clustering statistics in sysphysidx (seq_transitions, rand_transitions)


Agenda


Example of Analysis








Performance Tips




Graphical Query Plans

Displays, in graphical format, a detailed execution plan for a query

An execution plan appears as a tree of physical operators

• For example, hash joins, table scans, sorts

• Edge thickness represents number of rows generated by the child operator

• Box colour represents relative cost of operator from “fast” to “slow”

• Can click on any operator in the tree to get detailed information



Some things to look for:

• Predicate selectivities

o Are the selectivities reasonable?

o What is the source for the reported selectivity?

• Join selectivities: PK-FK constraints, etc.

o Always use Foreign Keys when you can

• Check optimization goal

• Index scan for order-by when optimizing for response (‘first-row’)

o Best to use an index, if one exists, unless small sort

o Consider adding one otherwise



More things to look for:

• Check data currently in cache

o Sequential scans usually better than index scans when cache is cold

o Index scans will almost always do well with hot cache

o (Index probes often better than either seq scan or index scan)

If retrieving a lot of rows, seq scan may be better (40:1 on table pages)

• Look for expensive sub-queries

• Estimated vs. actual values (if plan built “with statistics”)

o Row counts

o Execution costs


Comparing Graphical Plans

Choose two saved

plans

Manually match some

nodes

Highlights differences

Consider cached pages

Look for row count /

selectivities


Compare Graphical Plans (cont)

Consider differences

Row counts are usually

important

Selectivity

differences

Look for option / config

diff


Agenda


Example of Analysis








Performance Tips



What?

• Some utilities available for testing performance

• Provide an indication of “laws of physics” given the server and db configurations

When?

• Use these tools, not graphical plan with statistics, for accurate timings

How?

• Available in %SQLANYSAMP16%

• Complete documentation in Readme.txt in the same folder as the utility


The dbping System Utility

The dbping utility can be used to estimate the latency of the connection

With –s (and –st) it will also measure more detailed statistics Statistic Description

DBLib connect and disconnect The time to perform one DBLib connect and disconnect

Round trip simple request The time it takes to send a request from the client to the server

plus the time it takes to send a response from the server back to

the client. The round trip time is twice the average latency.

Send throughput Throughput of 100KB/iteration.

Receive throughput Throughput of 100KB/iteration.


Testing Query Performance

Fetchtst (or odbcfet) • \Samples\SQLAnywhere\PerformanceFetch

• Measures fetch rates for an arbitrary query

• Put arbitrary queries in a file (test.sql, by default) separated by “go”

o Often SELECT statements are used but update and insert works too

Useful switches • -ga – measures several useful statistics

• -j nnn – repeat the file several times (or –js for each statement)

• -p – print the plan (can show plan changes for one query)

• -i – think time

Try to match your configuration when testing • Isolation, cursor type, prefetch, connection


Example fetchtst Results

SQL Step count seconds min.s/i ------------ ----- -------- -------- plan 10 0.002706 0.000262 PREPARE 10 0.000420 0.000039 DESCRIBE 10 0.000566 0.000054 OPEN 10 0.000293 0.000027 FETCH first row 10 0.000562 0.000052 FETCH remaining rows 10 6.451974 0.581380 CLOSE 10 0.000564 0.000055 DROP 10 0.000041 0.000004 EXECUTE (described non-query) 0 0.000000 EXECUTE IMMEDIATE (non-query) 0 0.000000 ------------ ----- -------- -------- Total 10 6.457126 0.581962 Total elapsed for whole run 1 6.515817 6.515817 Engine CPU usage (tot) 10 5.288434 0.436803 Engine CPU usage (usr) 10 5.288434 0.436802 Engine CPU usage (sys) 10 0.000000 0.000000 ------------ ----- -------- -------- select pk 10 6.457126 0.581962 ------------ ----- -------- --------


Testing Insert Performance

Instest

• \Samples\SQLAnywhere\PerformanceInsert

• Measures insert performance for a table

• Reads query from a file

• Uses PUT to insert rows

Try to match your configuration

• Commit frequency

• Insert width (rows to insert per request)

• Pre-grow database file and ensure it is contiguous



Trantest

• \Samples\SQLAnywhere\PerformanceTransaction

• Measures the load that can be handled by a given server configuration given a database design and a set of

transactions.

• Simulates a number of client machines running transactions against the server

• Define what transactions to execute

• Can run on multiple client machines: master/slave


TYPES OF PERFORMANCE

PROBLEMS


Broad Classes of Performance Problems

Communication patterns / cheap requests that add up to too much time

Time for client to form request and interpret the result

Network latency

Build / open time

Expensive requests

Queries that are inherently expensive

Configurations that make it expensive to process queries

Query plan quality

Slow concurrent requests

Mutual exclusion due to locking or other blocking

Concurrent use of shared resource can be slower than serial


Rough Categorization of a Workload

Estimate (total time / # requests)

• If average request time is < 100ms, probably an issue with cheap requests

Use –zt server option and dbo.sa_performance_diagnostics()

• If ReqTimeActive or ReqTimeBlockIO high – expensive requests

• If ReqTimeBlockLock or ReqTimeBlockContention high – Slow concurrent requests

If performance of one query varies substantially with identical parameters, it may be

query plan differences

• Make sure to rule out differences in cache contents, concurrent activity, …

• If the period is about 11 executions, could be related to plan caching


Agenda


Example of Analysis








Performance Tips


Using Many Cheap Requests Repeats Overheads

Server overheads

Parsing SQL

Building plan

Inter-process communication

Latency

Client overheads

Building up statements

Starting to process results


Total Request Time

Request(value-params)

Results

Think

Think

Execute Wait

Wait Request(value-params)

Execute

Total Time


Millisecond Workloads Think

Execute Wait

Wait

Total Time < 1ms

Execute


Server Operations in Detail

Build

Execute

Close

Pre-Optimization

Scan SQL, Parms

Parse

Semantic Transformations

Prepare Parse Tree

Cursor

Join Enumeration

DFO Build

Open

Execute

Close

Describe

Post-Optimization


Effect of Overhead in a Cheap Request

0.2

0.5

26.1

42.4

13.3

0.5

Drop

Close

Fetch

Open

Describe

Prepare

Time (micro-seconds)

R1: SELECT * FROM T WHERE T.pk = 100


Expensive Statements Over Versions 14.600

7.700

4.200

2.800

0.510 0.477

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

7.0.4 8.0.0 9.0.1 10.0.0 12.0.1 16.0.0

Average TPC/H SF10 Query Time (minutes)


Performance of Simple Statements Over Versions

0

50

100

150

200

250

5.5

.5

6.0

.0

6.0

.1

6.0

.2

6.0

.3

6.0

.4

7.0

.0

7.0

.1

7.0

.2

7.0

.3

7.0

.4

8.0

.0

8.0

.1

8.0

.3

9.0

.0

9.0

.1

9.0

.2

10

.0.0

10

.0.1

11

.0.0

11

.0.1

12

.0.0

12

.0.1

16

.0.0

Mic

roseco

nd

s

R1

R2

R1: SELECT * FROM T WHERE T.pk = 100

R2: SELECT * FROM T WHERE T.pk = 100 and row_num+1=4

1995 2000 2003 ‘06 2013 ‘08

17 Plan Cache


Categorizing Requests

Simple Complex

Cheap

Expensive

select count(*) from T

select * from T1,T2,T2,…,TN where T.x < :a (many rows match)

select * from T where pk=?

select * from T1,T2,T2,…,TN where T.x < :a (very few rows match)


Agenda


Example of Analysis








Performance Tips


I/O-Bound applications

How to detect?

• Server is slow but not CPU-bound

• In Windows Task Manager, see lots of reads and writes by database server process

• In perfmon, look at %Idle Time counter for Physical Disk objects – if below 1%, server is likely I/O-bound

• Consider Average sec/Transfer and ReqTimeBlockIO

Sanity-check – hard drive making lots of noise, lit up

I/O-bound applications may require addition of extra disk hardware

• SSDs can be used and even combined in RAID


Windows perfmon


Cache Size Too Small

If server can’t keep frequently-used database pages in buffer pool, thrashing occurs

Can detect using SQL Anywhere counters in perfmon or tracing:

• CacheReadTable vs. DiskReadTable

• CacheReadIndex vs. DiskReadIndex

These counters are absolute values, so look at growth over a fixed period of time

• Should see CacheReadTable grow more than 10 times faster than DiskReadTable, CacheReadIndex more

than 100 times faster

• If not, indication that cache size may be too small


Missing Indexes

Properly tuned indexes can greatly reduce I/O requirements

• Read only the rows needed to satisfy a query, rather than all rows

Best way to investigate is with Index Consultant

• Available from SC Profiling Mode or DBISQL

• Can also experiment manually with Index Consultant using CREATE VIRTUAL INDEX statement

• Inspect application queries

Clustering of indexes is also important

• Compare the clustering statistics in sys.sysphysidx to the clustered_index_id of systab


Query Processing Memory

Server may not have enough memory to process queries using regular methods,

causing it to use expensive low-memory strategies

• Special case of cache that is too small

• Likely cause – very high -gn value (above 100) and small cache size

o Make sure you really need such a high value if you use it

If this is the problem you may see in profiling mode that queries identified as

expensive don’t use hash operators

• Or, if they do use them, the operator details indicate reversion to low memory strategies, or multiple passes

• Compare QueryMemMaxUseful to QueryMemLikelyGrant

Check max_statement_count/max_cursor_count options for handle leaks


Fragmentation

Check OS file fragmentation

• Server window on startup, DBFileFragments property

• Use tools provided by the OS to fix

o Eg. contig.exe, available free from www.sysinternals.com

Check table and index fragmentation

• sa_table_fragmentation(), sa_index_density() or SQL Central Fragmentation tab

• Correction

o REORGANIZE TABLE

o Unload/reload database

• Avoidance

o PCTFREE


Fragmentation View in Sybase Central


Suboptimal File Placement

Placement of different database files (system dbspace, secondary dbspace,

temporary dbspace, transaction log) may be suboptimal

• Often good to put transaction log on separate disk system

• Good to place whatever files you can on an SSD

In applications that update or delete large quantities of data:

• can help to push user tables into secondary dbspace on a different physical disk leaves checkpoint log with

more bandwidth

• Make sure transaction log is on its own physical disk

• Avoid RAID-5 for disks for all types of log!

Check table/index fragmentation internally

Check database file fragmentation externally


Agenda


Example of Analysis








Performance Tips



Computation and memory access dominating

Good news – easy to detect!

Bad news – many possible causes…

How to detect?

• CPU above 98% use for all CPUs assigned to server process

• Task Manager shows server process consuming all available CPUs assigned to it

• Make sure no other applications are competing for large amounts of CPU

• Watch out for a system that has 1/N % of CPU in use (no parallelism)


Suboptimal Query Plans

Optimizer is choosing one or more access plans that are substantially poorer than

the optimal plan

Most common causes:

• Outdated optimizer statistics

• Incorrect setting of OPTIMIZATION_GOAL option

• Missing indexes (with large enough buffer pool)

To analyze:

• Capture plans with statistics, either manually with DBISQL, or from a tracing session

• Look for slow queries

• Look for query operators where estimated number of rows or cost vary drastically from actual values


Suboptimal Queries

Optimizer does the best it can, but sometimes is faced with queries that make it

hard to do its job

Basic cause: server is being asked to compute more data than you really need

Some common types:

• Asking for extra columns in a result set that you don’t use

• Frequent calling of user-defined functions (in a predicate, for example) or that have queries in them that tie the

hands of the optimizer

• Failing to specify READ ONLY access for queries that are not used for update

• nested correlated subqueries -- consider the use of WINDOW functions


Optimization Goal

First Row vs All Rows

• Application displays first few rows of a query

• Reporting type application that will always fetch all rows of a query

• Certain application interface patterns:

o TOP <N> STARTAT <S>

o API layer fetches all rows of query even if application doesn’t use them

Solution

• Optimization_goal option

• OPTION clause on SQL Query

• TABLE Hints in the FROM clause


Agenda


Example of Analysis








Performance Tips


Concurrency-Bound Applications

If application does not seem to be either CPU or I/O bound on the server, it may be

concurrency bound

• Special case of CPU or I/O-boundedness – application can’t take advantage of extra resources

• ActiveReq performance counter: if high (10 to 20 or higher), an indication of being concurrency bound

• If it is low, and CPU and I/O are also low, problem is that application isn’t giving server enough work to keep

busy

Concurrency problem may be internal to server, or may be in server-side

application code

• Determine using DBCONSOLE or sa_connection_info() if connections are blocked on others, application-

based concurrency problem; otherwise could be internal server-based


Application Concurrency

User connections hold locks on rows, preventing other connections from getting their work done

Most common causes: • Hot row – detected by application profiling (blocking analysis)

• Long term holding of read locks – detected by looking for cursors that stay open for a long time, long transactions at isolation levels > 0

• Long term holding of write locks keep update transactions as short as possible

Check isolation level of all transactions • 0 (default) - no locking; a latch ensures that the entire row is consistent when retrieved from the disk page

• 1,2 - lock rows in the query’s result, but with level 1 the lock is held only while the cursor is on that row

• 3 - lock every row read and every insertion point crossed during query execution

• Snapshot isolation – writers don’t block readers, achieved by maintaining copies of modified rows


Avoid Designs Where Many Transactions

Update A Row

This can come up with “one row tables”

• For example, key generation; consider using sequences

When a foreign row is inserted, lock primary row

• In version 12 and above, only the primary key is locked, other columns open


Internal Server Concurrency

Server maintains internal locks to protect server structures

Contention for these resources may happen:

• Transaction log solution: move to new disk, pregrow

• Checkpoint log solution: secondary dbspace

• Lock table – only possible when servicing hundreds of connections


Agenda


Example of Analysis








Performance Tips


Avoid Extra Work – Client Issues Make client-server communication more efficient by reducing the number of

requests to the server

• TURN OFF AUTOCOMMIT

• Avoid client-side joins

• Utilize wide fetches or wide inserts from your application

• Make use of PREFETCHing for large result sets

• Locally cache information in your application

• Combine a set of statements into a batch, or embed the statements within a stored procedure so

that only one CALL statement needs to be sent from the application

• PREPARE/DESCRIBE once during initialization (or on first use)

o Client statement caching will avoid multiple DROP/PREPARE sequences for identical SQL statements

• Bind columns whenever possible

o Eg. use SQLBindCol() instead of SQLGetData()


Avoid Extra Work – Turn Off Autocommit

AutoCommit – huge penalty

• Several interfaces default to AutoCommit ON

o OLEDB, ODBC, ADO.NET, JDBC, …

• Every Commit operation causes at least 1 physical IO to the transaction log and the server won’t

respond to Commit until IO is complete!

o Turning off the transaction log is worse because every commit causes a checkpoint – all dirty pages are

written

• Note that any DDL causes implicit COMMITs and sometimes CHECKPOINTs

Solution:

• Turn off AutoCommit, use Commits and Rollbacks in your application as appropriate

• Always use a transaction log

• Avoid DDL – use temp tables, create permanent objects once


Avoid Extra Work – Client Side Joins

Client-side join

• Application fetches from one table and for each row, fetches from one or more other tables all with separate

queries

• Simple variant: same query repeatedly issued for the same values

Solution

• Look for opportunities to cache values that don’t change on the server

• Look for ways to combine sets of statements using joins in the server, possibly more complex procedures


Avoid Extra Work - Prefetch

Prefetch reduces communication by transferring sets of rows to the client in

advance of a FETCH request

Prefetch is ON by default

• To disable use the DisableMultiRowFetch connection parameter or set the Prefetch option to OFF

• Prefetch is turned off on cursors declared with sensitive value semantics

Adaptive prefetching

• Increases or decreases depending on application behaviour

• Maximum number of rows that will be prefetched is 1000

• Incluenced by number of rows the application can FETCH in one elapsed second


Avoid Extra Work – Wide Fetches/Inserts

For relatively large result sets, use wide fetches

• Each API call obtains several rows; explicitly set by the application

o Prefetching may or may not also occur

• Number of rows wide fetched is configurable

With wide (multi-row) inserts (and updates, deletes in V17):

• Supported by ESQL, ODBC, JDBC

• Consider LOAD TABLE where appropriate

• COMMIT at regular intervals to reduce lock contention, limit size of rollback log

Updates: use set based UPDATE … FROM … WHERE operations


Network Latency and Performance

• Latency: time it takes for a packet to be received at a different machine once sent

• Throughput: number of bits (bytes) that can be transferred in a given period of time

• LAN: typically 1ms (perhaps less) latency, at least 1MB/sec throughput

• WAN: 5-500 ms latency, 4-200KB/sec throughput

o These are ballpark estimates

• DBPing: can be used to determine round trip time to the server

• reducing requests to server reduces impact of network latency


Reducing Network Latency

Increase the database server’s packet size

• Default in Version 11 has increased from 1460 to 7300; even larger sizes can be beneficial for large result

sets

• Can improve the performance of large FETCHes and multi-row fetches, or BLOB operations (both retrieval

and insertion)

Use the CommBufferSize connection parameter

• Alter the packet size only for connections that would benefit from a larger packet size.

Consider altering the ReceiveBufferSize and SendBufferSize TCP/IP parameters

• Preallocate the amount of memory used by the TCP/IP protocol stack to receive and send packets over the

wire

• Defaults for these values are machine-dependent (OS, driver, card manufacturer)


Improving Network Throughput

Communication compression may improve throughput between client and server

over a modem or WAN:

• Packets are compressed before encryption

• Compressed data can be less than 10% of original size, but depends completely on data and the application

• Increase packet size for greater compression, fewer packets

• Compression requires additional ~46K per connection

• You must analyze your application's performance and verify results

o Compression requires additional CPU; on LANs, compression costs typically outweigh

savings in bandwidth


SUMMARY OF PERFORMANCE

TIPS


Performance TIPS – Application Patterns

• Turn off autocommit mode

• Avoid client-side joins

• Set optimizer goal appropriately

• Use appropriate data types

• Use multi-row operations whenever possible

• Reduce requests between client and server

• Specify the correct cursor type


Performance TIPS

• Always use a transaction log • Check for concurrency issues • Choose the optimizer goal • Collect statistics on small tables • Declare constraints • Minimize cascading referential actions • Monitor query performance • Normalize your table structure • Place different files on different devices • Rebuild your database • Reduce fragmentation • Reduce file fragmentation • Reduce table fragmentation • Reduce index fragmentation and skew • Reduce primary key width • Reduce table widths • Reduce requests between client and server • Reduce expensive user-defined functions • Replace expensive triggers • Review the order of columns in tables

• Strategic sorting of query results • Specify the correct cursor type • Supply explicit selectivity estimates sparingly • Turn off autocommit mode • Use an appropriate page size • Use appropriate data types • Use AUTOINCREMENT to create primary keys • Use bulk operations methods • Use delayed commits • Use in-memory mode • Use indexes and keys effectively • Use the cache to improve performance • Monitoring cache size • Use cache warming • Use compression carefully • Use WITH EXPRESS CHECK when validating • Use work tables in query processing (use All-rows optimization

goal)


Agenda


Example of Analysis








Performance Tips

Conclusion and Resources


Conclusion

SQLA Continues to evolve to exploit new hardware / usage scenarios

Trends in increasing memory, core counts, and new storage technologies are very relevant

Dealing better with larger data volumes is important

But: speed of simple operations is also critical to some application patterns

Performance analysis and tuning is an interesting and idiosyncratic domain

It can be a lot of fun to trace down what is really going on

It is best if it can be done before there is a problem with unhappy customers


Resources

White papers:

• Capacity Planning with SQL Anywhere

o http://scn.sap.com/docs/DOC-35069

• Diagnosing Application Performance Issues with SQL Anywhere

o http://scn.sap.com/docs/DOC-35601

SQL Anywhere Documentation

o http://dcx.sap.com

SQL Anywhere Q&A forum

o http://sqlanywhere-forum.sap.com

68


Thank you

Jason Hinsperger

Sr Product Manager

SAP Labs Canada, 445 Wes Graham Way, Waterloo, ON N2L 6R, Canada

[email protected]

519-883-6492

mailto:[email protected]


© 2014 SAP AG or an SAP affiliate company. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG or an SAP affiliate company.

SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG (or an SAP affiliate

company) in Germany and other countries. Please see http://global12.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices.

Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

National product specifications may vary.

These materials are provided by SAP AG or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP AG or its

affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP AG or SAP affiliate company products and services

are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an

additional warranty.

In particular, SAP AG or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or

release any functionality mentioned therein. This document, or any related presentation, and SAP AG’s or its affiliated companies’ strategy and possible future

developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP AG or its affiliated companies at any time for

any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-

looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place

undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.

http://global12.sap.com/corporate-en/legal/copyright/index.epx



Technology

Maximizing Database Tuning in SAP SQL Anywhere