Ing. Eduardo Castro, PhD ecastro@grupoasesor.net http://ComunidadWindows.org

Session Objectives:

Get familiar with database performance tuning tools, especially what SQL Server 2008 offers

Understand methodology of database performance bottleneck analysis

Learn features/improvements of SQL Server 2008 that help improve performance

Target the most challenging and innovative SQL Server applications

10+ TB DW, 3k/tran/s OLTP, Large 500GB+ Cubes, Competitive migrations, Complex deployments, Server Consolidation (1000+)

Invest in large scale, referenceable SQL Server projects across the world

Provide SQLCAT technical & project experience

Conduct architecture and design reviews covering performance, operation, scalability and availability

Offer use of HW lab in Redmond with direct access to SQL Server development team

Performance Tuning Tools

I/O Bottleneck

CPU Bottleneck

Memory Bottleneck

System Performance Monitor (Perfmon)

SQL Profiler Trace

Dynamic Management Views/Functions

DBCC commands

SQLDiag (Shipped with SQL 2005/2008)

SQLIO, SQLIOSim

KernRate

Debugging tools (Windbg, DebugDiag,…)

More…

Data Collector & Management Data Warehouse (MDW)

Extended Events

Activity Monitor

New DMVs/DMFs

SQL Agent

dcexec.exe

Cache Files

SSIS Pkgs

Job Metadata

Collection Set Metadata

MDW

Snapshot Data

Management Studio

MDW

Reports

msdb

MDW Host

Target Server Client Workstation

Time line

Navigation control

Resource usage

Highly scalable with low overhead

Future of SQL tracing

Built-in system_health session select event_session_id, name, startup_state from

sys.server_event_sessions

Session definition example create event session session_error on server ADD EVENT sqlserver.error_reported (action (sqlserver.sql_text, sqlserver.tsql_stack,

sqlserver.client_app_name) where error = 50001) ADD target package0.ring_buffer with (max_dispatch_latency = 1 seconds) go alter event session session_error on server state = start; go

Memory allocation tracking sys.dm_os_memory_brokers sys.dm_os_memory_nodes

Reqource Governor sys.dm_resource_governor_configuration sys.dm_resource_governor_resource_pools sys.dm_resource_governor_workload_groups

Query performance (hash value for query and plan) sys.dm_exec_query_stats

Sys.dm_exec_requests sys.dm_exec_procedure_stats sys.dm_exec_trigger_stats

More…

Bottleneck – A point or component of a process where throughput is the slowest.

Understanding where time is spent

Elapsed Time = Running + Waiting

Key is the waiting - what type of wait(s)?

Monitor and identify I/O bottleneck

Resource Monitor & Performance Monitor

MDW

DMV & Extended Events

What is driving high I/O load?

Inefficient query plan

Too much data to read/write

Slow I/O subsystem

File level I/O throughput

and disk response time

Runtime Performance monitoring

System Reports

What counters to look at?

Physical Disk Object: Avg. Disk Queue Length Avg. Disk Sec/Read Avg. Disk Sec/Write %Disk Time Disk Transfers/Sec Avg. Disk Bytes/Read Avg. Disk Bytes/Write Disk Read Bytes/Sec Disk Write Bytes/Sec

SQL Server Buffer Manager Object: (memory pressure could cause I/O pressure)

Buffer Cache hit ratio Page Life Expectancy Checkpoint pages/sec Lazywrites/sec

WriteLog and LogBuffer

waits

Page I/O waits in Buffer pool

High I/O Latch Wait Time

pending/stalled I/O

ERRORLOG warnings of I/O

delays

sys.dm_os_wait_stats

sys.dm_exec_requests

sys.dm_io_pending_io_requests

sys.dm_io_virtual_file_stats

15 second delay in IO

completion

This is

way too

slow

sqlos package

• async_io_requested

• async_io_completed

• wait_info

• wait_info_external

sqlserver package

• physical_page_read

• physical_page_write

• file_read

• file_write

• long_io_detected

2009-04-13 18:30:10.170 spid5s SQL Server has encountered 1

occurrence(s) of I/O requests taking longer than 15 seconds to complete on file

[T:\tempdb\templog.ldf] in database [tempdb] (2). The OS file handle is

0x0000000000000670. The offset of the latest long I/O is: 0x000007eecac000

king longer than 15 seconds to complete on file [e:\myslowdb.mdf] in database

[myslowdb] (1). The OS file handle is

0x00000438. The offset of the latest long IO is: 0x000000513ce000

Look if there is memory pressure

Further narrow down to what queries are generating the I/O workload. Tune these queries.

Explore data compression to reduce I/O load

Upgrade storage hardware

Compress tables/indexes to reduce database size

Compression ratio depends on schema & data distribution (avg 50-60%)

Reduce query runtime I/O (increase cpu)

Identify compression candidates

sp_estimate_data_compression_savings

sys.dm_db_index_operational_stats

Sustained %Processor Time > 80%

Performance Monitor/Task Manager

MDW Server Activity Report

DMV & Extended Events

What is driving high CPU?

Recompilations

Cursors

Inefficient query plans

More…

• Monitor it Live Resource Monitor/Task Manager

• Watch Live or Log counters you like

Performance Monitor Counters

• Formatted Perfmon counter based report

System Data Collector Reports

• Dig deeper using ETW Windows Performance Toolkit (XPerf)

Processor object: %Privileged Time %Processor Time %User Time

Process object (SQL Server): %Privileged Time %Processor Time %User Time

SQL Server:Workload Group Stats CPU usage % Max request cpu time

SQL Server:Resource Pool Stats CPU usage % CPU usage target %

New in SQL Server 2008 for Resource Governor feature

SQL Server:SQL Statistics

Batch requests/sec

SQL Compilations/sec

SQL Re-Compilations/sec

SQL Server:Cursor Manager by Type

SQL Server:Cursor Manager Total

Concurrent processing of multiple cursors could be cpu

intensive

Wait in runnable

queue

Total CPU waits

Thread voluntarily

yields

CPU cycles that did the real work

Sys.dm_exec_sessions

cpu_time

total_scheduled_time

total_elapsed_time

sys.dm_exec_requests

cpu_time

total_elapsed_time

query_hash

sys.dm_exec_query_stats

sys.dm_exec_procedure_stats

sys.dm_exec_trigger_stats

total_worker_time

total_elapsed_time

Sql_handle/query_hash

Sys.sysprocesses

cpu

Backward compatibility view

sqlos package

•scheduler_monitor_non_yielding_ring_buffer_recorded

•spinlock_backoff

•Wait_info

•wait_info_external

sqlserver package

•sql_statement_completed

•sp_statement_completed

•cursor_manager_cursor_end

•checkpoint_end

Extended Event can track individual statement level events like Profiler does

with minimal impact -

ADD EVENT sqlserver.sql_statement_completed

(action (sqlserver.sql_text)

WHERE duration > 0),

ADD EVENT sqlserver.sp_statement_completed

(action (sqlserver.sql_text)

WHERE duration > 0)

select top 10

CONVERT(xml, event_data).value('(/event/data/value)[4]','int') as 'cpu'

High cpu doesn’t necessarily mean it’s a problem.

Drill down to specific workload and queries that drive cpu usage. Tune them.

Use Resource Governor to protect high priority workloads

Upgrade hardware

Resource Governor

allocate cpu/memory to prioritized workloads based on connection string parameters

Mixed workloads: OLTP, Reporting, Maintenance

Soft Numa

Node configuration (soft numa node and cpu affinity)

Network Configuration (soft numa node and tcp port)

Monitor and identify memory pressure

Performance Monitor/Task Manager

MDW Server Activity Report

DMV & Extended Events

What is eating up my memory?

External or internal memory pressure

Inefficient query plan (hashing, sorting,…)

AWE

32-bit uses AWE to cache data/index pages

64-bit uses AWE to implement “Locked pages in memory”

Environment Virtual Address

Space

Max physical

memory

32-bit on 32-bit OS 2 GB 64 GB

32-bit on 64-bit OS (Wow) 4 GB 64 GB

64-bit on 64-bit OS 8 TB 2 TB

Perfmon is golden tool to identify memory pressure

Memory object: Available K/M bytes

Commit Limit

Paging File %Usage (Peak)

Page reads/sec

Process object: Private bytes

Virtual bytes

Working set

SQL Server Memory Manager object: Connection Memory Lock Memory Optimizer Memory Memory Grants Pending …

SQL Server Buffer Manager object: Buffer cache hit ratio Checkpoint pages/sec Free list stalls/sec Free pages Lazy writes/sec Page life expectancy Stolen pages Target pages Total pages

SQL Server Plan Cache object SQL Server Resource Pool Stats object

Somebody is waiting on memory

How long will the page live?

Query execution waits on memory for sorting,

hashing,…

DBCC MEMORYSTATUS

sys.dm_os_sys_memory

system_memory_state_desc a good indicator of available physical memory

sys.dm_os_process_memory

process_physical_memory_low = 1 indicates low physical memory.

memory_utilization_percentage value below 100% indicates external memory pressure.

sys.dm_os_memory_clerks

sys.dm_os_ring_buffers

RING_BUFFER_SCHEDULER_MONITOR

RING_BUFFER_RESOURCE_MONITOR

RING_BUFFER_OOM

RING_BUFFER_MEMORY_BROKER

RING_BUFFER_BUFFER_POOL

sqlos package

• page_heap_memory_allocated

• page_heap_memory_freed

• memory_node_oom_ring_buffer_recorded

sqlserver package

• sort_memory_grant_adjustment

• plan_cache_cache_hit

• plan_cache_cache_attempt

• cursor_manager_cursor_memory_usage

2009-04-14 09:27:15.43 spid62 Failed allocate

pages: FAIL_PAGE_ALLOCATION 1

Check memory configuration – sp_configure

External memory pressure – Identify what other processes are competing

Internal memory pressure – Identify which SQL component is main consumer

Workload/query tuning (sorting, hash join…)

Resource Governor

Hardware upgrade

Focus on methodology – no guess

Correlate data points from multiple tools

Performance tuning is iterative process – bottleneck could move

Resource contention impacts each other

Lack of memory -> high cpu & I/O

Slow I/O -> tempdb contention

…

Ing. Eduardo Castro, PhD ecastro@grupoasesor.net

“Troubleshooting Performance Problems in SQL Server 2008” http://msdn.microsoft.com/en-us/library/dd672789.aspx