SQL Server Performance Tuning Reference Document

Manikandan Rajagopalan

Abstract Our mission is to optimize the performance of a SQL Server database. This document

focuses on refactoring and applying TSQL best practices to improve performance. Ref Link: https://www.mssqltips.com/sql-server-tip-category/9/performance-tuning/

https://sqlserverperformance.wordpress.com/category/diagnostic-queries/

Contents

Scope: 3

Step 1: Apply proper indexing in the table columns in the database ................................................................... 3

Make sure that every table in your database has a primary key. ............................................................... 3

Create non-clustered indexes on columns which are:................................................................................ 3

Step 2: Create the appropriate covering indexes ................................................................................................. 4

Step 3: Defragment indexes if fragmentation occurs ........................................................................................... 5

What is index fragmentation? ................................................................................................................... 5

Internal Fragmentation: ............................................................................................................................ 5

External Fragmentation: ........................................................................................................................... 5

How to know whether index fragmentation has occurred or not? ............................................................. 5

How to defragment indexes? .................................................................................................................... 6

Step 4: Identify inefficient TSQL, re-factor, and apply best practices ................................................................... 7

Some TSQL Best Practices ......................................................................................................................... 7

Don't use "SELECT*" in a SQL query .......................................................................................................... 7

Avoid unnecessary columns in the SELECT list and unnecessary tables in join conditions .......................... 7

Do not use the COUNT() aggregate in a subquery to do an existence check .............................................. 7

Try to avoid joining between two types of columns .................................................................................. 7

Try to avoid deadlocks .............................................................................................................................. 8

Write TSQL using "Set based approach" rather than "Procedural approach" ............................................. 8

Try not to use COUNT(*) to obtain the record count in a table .................................................................. 8

Try to avoid dynamic SQL .......................................................................................................................... 8

Try to avoid the use of temporary tables................................................................................................... 8

Instead of LIKE search, use full text search for searching textual data ....................................................... 8

Try to use UNION to implement an "OR" operation .................................................................................. 9

Implement a lazy loading strategy for large objects................................................................................... 9

Use VARCHAR(MAX), VARBINARY(MAX), and NVARCHAR(MAX) ................................................................ 9

Implement the following good practices in User Defined Functions .......................................................... 9

Implement the following good practices in Stored Procedures .................................................................. 9

Implement the following good practices in Triggers .................................................................................. 9

Implement the following good practices in Views ....................................................................................10

Implement the following good practices in Transactions ..........................................................................10

Understanding the query execution plan ..................................................................................................10

Estimated execution plan in Management Studio ....................................................................................11

What information do we get by viewing the execution plans? ..................................................................12

Step 5: Diagnose performance problems, use SQL Profiler and the Performance Monitoring Tool effectively ...12

Basic use of the SQL Profiler tool ..............................................................................................................12

Effective use of SQL Profiler to troubleshot performance related problems ..............................................14

Use the Performance Monitoring Tool (Perfmon) to diagnose performance problems ..............................15

Step 6: Organize file groups and files in the database .........................................................................................18

Step 7 (The bonus step): Better-manage DBMS objects, Use TSQL templates ....................................................19

How to create templates .........................................................................................................................19

STEP 8 : Performance tuning tips for database developers ..................................................................................20

STEP 9 LIST of 15 things to do a matter of tune performance ......................................................................................... 21

STEP 10 PerfMon Objects and Counters for SQL Server Benchmark ............................................................................... 22

Scope: When a database based application performs slowly, there is a 90% probability that the data access routines of that

application are not optimized, or not written in the best possible way. So, you need to review and optimize your data

access/manipulation routines for improving the overall application performance.

Step 1: Apply proper indexing in the table columns in the database Well, some could argue whether implementing proper indexing should be the first step in the performance optimization

process for a database. But I would prefer applying indexing properly in the database in the first place, because of the following

two reasons:

1. This will allow you to achieve the best possible performance in the quickest amount of time in a production system.

2. Applying/creating indexes in the database will not require you to do any application modification, and thus will not require any

build and deployment. Of course, this quick performance improvement can be achieved if you find that indexing is not properly

done in the current database. However, if indexing is already done, I would still recommend you to go through this step.

Make sure that every table in your database has a primary key. This will ensure that every table has a clustered index created (and hence, the corresponding pages of the table are physically sorted

in the disk according to the primary key field). So, any data retrieval operation from the table using the primary key, or any sorting

operation on the primary key field or any range of primary key values specified in the where clause will retrieve data from the

table very fast.

Create non-clustered indexes on columns which are: Frequently used in the search criteria

Used to join other tables

Used as foreign key fields

Of having high selectivity (column which returns a low percentage (0-5%) of rows from a total number of rows on a particular

value)

Used in the ORDER BY clause

Of type XML (primary and secondary indexes need to be created; more on this in the coming articles)

Following is an example of an index creation command on a table:

-----------------------------------------------------------------------------------------

CREATE INDEX

NCLIX_OrderDetails_ProductID ON

dbo.OrderDetails(ProductID)

-----------------------------------------------------------

Alternatively, you can use SQL Server Management Studio to create an index on the desired table

Step 2: Create the appropriate covering indexes So you have created all the appropriate indexes in your database, right? Suppose, in this process, you have created an

index on a foreign key column (ProductID) in the Sales(SelesID,SalesDate,SalesPersonID,ProductID,Qty) table. Now, assuming

that the ProductID column is a "highly selective" column (selects less than 5% of the total number of rows rows using any

ProductID value in the search criteria), any SELECT query that reads data from this table using the indexed column (ProductID)

in the where clause should run fast, right?

Yes, it does, compared to the situation where no index is created on the foreign key column (ProductID), in which case, a full table

scan is done (scanning all related pages in the table to retrieve desired data). But still, there is further scope to improve this

query.Let's assume that the Sales table contains 10,000 rows, and the following SQL selects 400 rows (4% of the total rows):

SELECT SalesDate, SalesPersonID FROM Sales WHERE ProductID = 112

Fortunately, there is a way to implement this feature. This is what is called "covered index". You create "covered indexes" in table

columns to specify what additional column values the index page should store along with the clustered index key values (primary

keys). Following is an example of creating a covered index on the ProductID column in the Sales table:

CREATE INDEX NCLIX_Sales_ProductID--Index name

ON dbo.Sales(ProductID)--Column on which index is to be created

INCLUDE(SalesDate, SalesPersonID)--Additional column values to include

Please note that covered index should be created including a few columns that are frequently used in the Select queries. Including

too many columns in the covered indexes would not give you too much benefit. Rather, doing this would require too much

memory to store all the covered index column values, resulting in over consumption of memory and slow performance.

Step 3: Defragment indexes if fragmentation occurs OK, you created all the appropriate indexes in your tables. Or, may be, indexes are already there in your database tables. But you

might not still get the desired good performance according to your expectations. There is a strong chance that index fragmentation

has occurred.

What is index fragmentation? Index fragmentation is a situation where index pages split due to heavy insert, update, and delete operations on the tables in the

database. If indexes have high fragmentation, either scanning/seeking the indexes takes much time, or the indexes are not used at

all (resulting in table scan) while executing queries. Thus, data retrieval operations perform slow.

Two types of fragmentation can occur:

Internal Fragmentation: Occurs due to data deletion/update operations in the index pages which end up in the distribution

of data as a sparse matrix in the index/data pages (creates lots of empty rows in the pages). Also results in an increase of

index/data pages that increases query execution time.

External Fragmentation: Occurs due to data insert/update operations in the index/data pages which end up in page

splitting and allocation of new index/data pages that are not contiguous in the file system. That reduces performance in

determining the query result where ranges are specified in the "where" clauses. Also, the database server cannot take advantage

of the read-ahead operations as the next related data pages are not guaranteed to be contiguous, rather these next pages could

be anywhere in the data file.

How to know whether index fragmentation has occurred or not? Execute the following SQL in your database (the following SQL will work in SQL Server 2005 or later databases). Replace the

database name 'AdventureWorks' with the target database name in the following query:

------------------------------------------------------------------------------------------------------ SELECT object_name(dt.object_id) Tablename,si.name,IndexName,dt.avg_fragmentation_in_percent AS

ExternalFragmentation,dt.avg_page_space_used_in_percent AS

InternalFragmentation

FROM

(

SELECT object_id,index_id,avg_fragmentation_in_percent,avg_page_space_used_in_percent

FROM sys.dm_db_index_physical_stats (db_id('AdventureWorks'),null,null,null,'DETAILED'

)

WHERE index_id <> 0) AS dt INNER JOIN sys.indexes si ON si.object_id=dt.object_id

AND si.index_id=dt.index_id AND dt.avg_fragmentation_in_percent>10

AND dt.avg_page_space_used_in_percent<75 ORDER BY avg_fragmentation_in_percent DESC

------------------------------------------------------------------------------------------------------------------

The above query shows index fragmentation information for the 'AdventureWorks' database as follows:

How to defragment indexes? You can do this in two ways:

Reorganize the fragmented indexes: execute the following command to do this:

ALTER INDEX ALL ON TableName REORGANIZE

Rebuild indexes: execute the following command to do this:

ALTER INDEX ALL ON TableName REBUILD WITH (FILLFACTOR=90,ONLINE=ON)

You can also rebuild or reorganize individual indexes in the tables by using the index name instead of the 'ALL' keyword in the

above queries. Alternatively, you can also use SQL Server Management Studio to do index defragmentation.

When to reorganize and when to rebuild indexes?

You should "reorganize" indexes when the External Fragmentation value for the corresponding index is between 10-15 and the

Internal Fragmentation value is between 60-75. Otherwise, you should rebuild indexes.

Step 4: Identify inefficient TSQL, re-factor, and apply best practices No matter how good indexing you apply to your database, if you use poorly written data retrieval/access logic, you are bound to

get slow performance.

Time to re-factor the code now. Time to implement the best practices in your code.

I have some SQL best practices for you that you can follow. But I am sure that you already know most of them. Problem is, in

reality, you just don't implement these good stuff in your code (of course, you always have some good reasons for not doing so).

But what happens at the end of the day? Your code runs slowly, and your client becomes unhappy.

While you should know that best practices alone is not enough, you have to make sure that you follow the best practices while

writing TSQL. This is the most important thing to remember.

Some TSQL Best Practices Don't use "SELECT*" in a SQL query

Unnecessary columns may get fetched that will add expense to the data retrieval time.

The database engine cannot utilize the benefit of "Covered Index" (discussed in the previous article), and hence the query

performs slowly.

Avoid unnecessary columns in the SELECT list and unnecessary tables in join conditions Selecting unnecessary columns in a Select query adds overhead to the actual query, specially if the unnecessary columns are of

LOB types.

Including unnecessary tables in join conditions forces the database engine to retrieve and fetch unnecessary data and increases

the query execution time.

Do not use the COUNT() aggregate in a subquery to do an existence check Do not use:

SELECT column_list FROM table WHERE 0 < (SELECT count(*) FROM table2 WHERE ..)

Instead, use: SELECT column_list FROM table WHERE EXISTS (SELECT * FROM table2 WHERE ...)

When you use COUNT(), SQL Server does not know that you are doing an existence check. It counts all matching values, either

by doing a table scan or by scanning the smallest non-clustered index.

When you use EXISTS, SQL Server knows you are doing an existence check. When it finds the first matching value, it returns

TRUE and stops looking. The same applies to using COUNT() instead of IN or ANY.

Try to avoid joining between two types of columns When joining between two columns of different data types, one of the columns must be converted to the type of the other. The

column whose type is lower is the one that is converted.

If you are joining tables with incompatible types, one of them can use an index, but the query optimizer cannot choose an index

on the column that it converts. For example:

SELECT column_list FROM small_table, large_table WHERE

smalltable.float_column = large_table.int_column

In this case, SQL Server converts the integer column to float, because int is lower in the hierarchy than float. It cannot use an

index on large_table.int_column, although it can use an index on smalltable.float_column.

Try to avoid deadlocks Always access tables in the same order in all your Stored Procedures and triggers consistently.

Keep your transactions as short as possible. Touch as few data as possible during a transaction.

Never, ever wait for user input in the middle of a transaction.

Write TSQL using "Set based approach" rather than "Procedural approach" The database engine is optimized for Set based SQL. Hence, Procedural approach (use of Cursor or UDF to process rows in a result

set) should be avoided when large result sets (more than 1000) have to be processed.

How can we get rid of "Procedural SQL"? Follow these simple tricks:

o Use inline sub queries to replace User Defined Functions.

o Use correlated sub queries to replace Cursor based code.

o If procedural coding is really necessary, at least, use a table variable instead of a cursor to navigate and process the

result set.

For more info on "set" and "procedural" SQL, see Understanding "Set based" and "Procedural" approaches in SQL.

Try not to use COUNT(*) to obtain the record count in a table To get the total row count in a table, we usually use the following Select statement:

SELECT COUNT(*) FROM dbo.orders

This query will perform a full table scan to get the row count.

The following query would not require a full table scan. (Please note that this might not give you 100% perfect results always, but this

is handy only if you don't need a perfect count.)

SELECT rows FROM sysindexes

WHERE id = OBJECT_ID('dbo.Orders') AND indid < 2

Try to avoid dynamic SQL Unless really required, try to avoid the use of dynamic SQL because:

Dynamic SQL is hard to debug and troubleshoot.

If the user provides the input to the dynamic SQL, then there is possibility of SQL injection attacks.

Try to avoid the use of temporary tables Unless really required, try to avoid the use of temporary tables. Rather use table variables.

In 99% of cases, table variables reside in memory, hence it is a lot faster. Temporary tables reside in the TempDb database. So

operating on temporary tables require inter database communication and hence will be slower.

Instead of LIKE search, use full text search for searching textual data Full text searches always outperform LIKE searches.

Full text searches will enable you to implement complex search criteria that can't be implemented using a LIKE search, such as

searching on a single word or phrase (and optionally, ranking the result set), searching on a word or phrase close to another

word or phrase, or searching on synonymous forms of a specific word.

Implementing full text search is easier to implement than LIKE search (especially in the case of complex search requirements).

For more info on full text search, see http://msdn.microsoft.com/en-us/library/ms142571(SQL.90).aspx

Try to use UNION to implement an "OR" operation Try not to use "OR" in a query. Instead use "UNION" to combine the result set of two distinguished queries. This will improve

query performance.

Better use UNION ALL if a distinguished result is not required. UNION ALL is faster than UNION as it does not have to sort

the result set to find out the distinguished values.

Implement a lazy loading strategy for large objects Store Large Object columns (like VARCHAR(MAX), Image, Text etc.) in a different table than the main table, and put a

reference to the large object in the main table.

Retrieve all the main table data in a query, and if a large object is required to be loaded, retrieve the large object data from

the large object table only when it is required.

Use VARCHAR(MAX), VARBINARY(MAX), and NVARCHAR(MAX) In SQL Server 2000, a row cannot exceed 8000 bytes in size. This limitation is due to the 8 KB internal page size of SQL Server.

So to store more data in a single column, you need to use TEXT, NTEXT, or IMAGE data types (BLOBs) which are stored in a

collection of 8 KB data pages.

These are unlike the data pages that store other data in the same table. These pages are arranged in a B-tree structure. These

data cannot be used as variables in a procedure or a function, and they cannot be used inside string functions such as

REPLACE, CHARINDEX, or SUBSTRING. In most cases, you have to use READTEXT, WRITETEXT, and UPDATETEXT.

To solve this problem, use VARCHAR(MAX), NVARCHAR(MAX), and VARBINARY(MAX) in SQL Server 2005. These data

types can hold the same amount of data BLOBs can hold (2 GB), and they are stored in the same type of data pages used for

other data types.

When data in a MAX data type exceeds 8 KB, an over-flow page is used (in the ROW_OVERFLOW allocation unit), and a pointer

to the page is left in the original data page in the IN_ROW allocation unit.

Implement the following good practices in User Defined Functions Do not call functions repeatedly within your Stored Procedures, triggers, functions, and batches. For example, you might need

the length of a string variable in many places of your procedure, but don't call the LEN function whenever it's needed; instead,

call the LEN function once, and store the result in a variable for later use.

Implement the following good practices in Stored Procedures Do not use "SP_XXX" as a naming convention. It causes additional searches and added I/O (because the system Stored

Procedure names start with "SP_"). Using "SP_XXX" as the naming convention also increases the possibility of conflicting with

an existing system Stored Procedure.

Use "Set Nocount On" to eliminate extra network trip.

Use the WITH RECOMPILE clause in the EXECUTE statement (first time) when the index structure changes (so that the

compiled version of the Stored Procedure can take advantage of the newly created indexes).

Use default parameter values for easy testing.

Implement the following good practices in Triggers Try to avoid the use of triggers. Firing a trigger and executing the triggering event is an expensive process.

Never use triggers that can be implemented using constraints.

Do not use the same trigger for different triggering events (Insert, Update, Delete).

Do not use transactional code inside a trigger. The trigger always runs within the transactional scope of the code that fires the

trigger.

Implement the following good practices in Views Use views for re-using complex TSQL blocks, and to enable it for indexed views (Will be discussed later).

Use views with the SCHEMABINDING option if you do not want to let users modify the table schema accidentally.

Do not use views that retrieve data from a single table only (that will be an unnecessary overhead). Use views for writing

queries that access columns from multiple tables.

Implement the following good practices in Transactions Prior to SQL Server 2005, after BEGIN TRANSACTION and each subsequent modification statement, the value of @@ERROR

had to be checked. If its value was non-zero, then the last statement caused an error, and if an error occurred, the transaction

had to be rolled back and an error had to be raised (for the application). In SQL Server 2005 and onwards, the Try...Catch

block can be used to handle transactions in TSQL. So try to use Try...Catch based transactional code.

Try to avoid nested transactions. Use the @@TRANCOUNT variable to determine whether a transaction needs to be started (to

avoid nested transactions).

Start a transaction as late as possible and commit/rollback the transaction as fast as possible to reduce the time period of resource

locking.

And, that's not the end. There are lots of best practices out there! Try finding some of them from the following URL: MSDN.

Remember, you need to implement the good things that you know; otherwise, your knowledge will not add any value to the system

that you are going to build. Also, you need to have a process for reviewing and monitoring the code (that is written by your team)

to see whether the data access code is being written following the standards and best practices.

Understanding the query execution plan Whenever you issue a SQL statement in the SQL Server engine, SQL Server first has to determine the best possible way to execute

it. In order to carry this out, the Query Optimizer (a system that generates the optimal query execution plan before executing the

query) uses several information like the data distribution statistics, index structure, metadata, and other information to analyze

several possible execution plans and finally select one that is likely to be the best execution plan most of the time.

Did you know? You can use SQL Server Management Studio to preview and analyze the estimated execution plan for the query

that you are going to issue. After writing the SQL in SQL Server Management Studio, click on the estimated execution plan icon

(see below) to see the execution plan before actually executing the query.

(Note: Alternatively, you can switch the actual execution plan option "on" before executing the query. If you do this, Management

Studio will include the actual execution plan that is being executed along with the result set in the result window.)

Estimated execution plan in Management Studio Understanding the query execution plan in detail

Each icon in the execution plan graph represents an action item (Operator) in the plan. The execution plan has to be read from right

to left, and each action item has a percentage of cost relative to the total execution cost of the query (100%).

In the above execution plan graph, the first icon in the right most part represents a "Clustered Index Scan" operation (reading all

primary key index values in the table) in the HumanResources table (that requires 100% of the total query execution cost), and the

left most icon in the graph represents a SELECT operation (that requires only 0% of the total query execution cost).

Following are the important icons and their corresponding operators you are going to see frequently in the graphical query execution

plans:

(Each icon in the graphical execution plan represents a particular action item in the query. For a complete list of the icons and their

corresponding action items, go to http://technet.microsoft.com/en-us/library/ms175913.aspx.)

Note the "Query cost" in the execution plan given above. It has 100% cost relative to the batch. That means, this particular query

has 100% cost among all queries in the batch as there is only one query in the batch. If there were multiple queries simultaneously

executed in the query window, each query would have its own percentage of cost (less than 100%).

To know more details for each particular action item in the query plan, move the mouse pointer on each item/icon. You will see a

window that looks like the following:

This window provides detailed estimated information about a particular query item in the execution plan. The above window shows

the estimated detailed information for the clustered index scan and it looks for the row(s) which have/has Gender = 'M' in the

Employee table in HumanResources schema in the AdventureWorks database. The window also shows the estimated IO, CPU,

number of rows, with the size of each row, and other costs that is used to compare with other possible execution plans to select the

optimal plan.

I found an article that can help you further understand and analyze TSQL execution plans in detail. You can take a look at it here:

http://www.simple-talk.com/sql/performance/execution-plan-basics/.

What information do we get by viewing the execution plans? Whenever any of your query performs slowly, you can view the estimated (and, actual if required) execution plan and can identify

the item that is taking the most amount of time (in terms of percentage) in the query. When you start reviewing any TSQL for

optimization, most of the time, the first thing you would like to do is view the execution plan. You will most likely quickly identify

the area in the SQL that is creating the bottleneck in the overall SQL.

Keep watching for the following costly operators in the execution plan of your query. If you find one of these, you are likely to

have problems in your TSQL and you need to re-factor the TSQL to improve performance.

Table Scan: Occurs when the corresponding table does not have a clustered index. Most likely, creating a clustered index or defragmenting index will enable you to get rid of it. Clustered Index Scan: Sometimes considered equivalent to Table Scan. Takes place when a non-clustered index on an eligible column is not available. Most of the time, creating a non-clustered index will enable you to get rid of it. Hash Join: The most expensive joining methodology. This takes place when the joining columns between two tables are not indexed. Creating indexes on those columns will enable you to get rid of it. Nested Loops: Most cases, this happens when a non-clustered index does not include (Cover) a column that is used in the SELECT column list. In this case, for each member in the non-clustered index column, the database server has to seek into the clustered index to retrieve the other column value specified in the SELECT list. Creating a covered index will enable you to get rid of it. RID Lookup: Takes place when you have a non-clustered index but the same table does not have any clustered index. In this case, the database engine has to look up the actual row using the row ID, which is an expensive operation. Creating a clustered index on the corresponding table would enable you to get rid of it.

Step 5: Diagnose performance problems, and use SQL Profiler and the Performance Monitoring Tool effectively The SQL Profiler tool is perhaps the most well-known performance troubleshooting tool in the SQL Server arena. In most cases,

when a performance problem is reported, this is the first tool that you are going to launch to investigate the problem.

Basic use of the SQL Profiler tool There is a 90% chance that you already know how to use it. But I assume a lot of newbies out there who are reading this

article might feel good if there is a section on the basic usage of the SQL Profiler (if you know this tool already, just feel

free to skip this section). So, here is a brief section:

Start working with the SQL Profiler in the following way:

Launch SQL Profiler (Tools -> SQL Server Profiler in Management Studio) and connect it to the desired SQL Server

instance. Select a new trace to be created (File -> New Trace) and select a trace template (a trace template is a template

where some pre-selected events and columns are selected to be traced).

Trace template

Optionally, select particular events (which should be captured in the trace output) and select/deselect columns (to specify the information you want to see in the trace output).

Optionally, organize the columns (click the "Organize Columns" button) to specify the order of their appearance in the

trace. Also, specify the column filter values to filter the event data which you are interested in. For example, click on "Column

Filters" and specify the database name value (in the "Like" text box) to trace events only for the specified database. Please note

that filtering is important because SQL Profiler would otherwise capture unnecessary events and trace too many information that

you might find difficult to deal with.

Filter column values

Run the profiler (by clicking the green Play button) and wait for the events to be captured on the trace.

Running the profiler

When enough information is traced, stop the profiler (b pressing the red Stop icon) and save the trace either into a trace file or into

a SQL Server table (you have to specify a table name and the SQL Server Profiler would create the table with necessary fields and

store all the tracing records inside it).

Storing profiler trace data into a table

If the trace is saved on a table, issue a query to retrieve the expensive TSQLs using a query like the following:

SELECT TextData,Duration,..., FROM Table_Name ORDER BY

Duration DESC

Querying for the most expensive TSQL/Stored Procedure

Voila! You just identified the most expensive TSQLs in your application in quick time.

Effective use of SQL Profiler to troubleshot performance related problems

Use existing templates, but create your own templates when in need.

In this situation, you can create a customized template using File -> Templates -> New Template, and specifying the template

name and events and columns. Also, you can select an existing template and modify it according to your needs.

Creating a new template

Specifying events and columns for the new template

Capture TableScan or DeadLock events.

Detecting table scan

Detecting deadlocks

Create Replay trace.

To learn more about TSQL Replay trace, see http://msdn.microsoft.com/en-us/library/ms189604.aspx.

Create Tuning trace.

You would like to learn and use the database tuning advisor to get tuning suggestions while you try to troubleshoot performance

issues in SQL Server. Take a look at this article to learn this interesting tool: http://msdn.microsoft.com/en-

us/library/ms166575.aspx.

Capture ShowPlan to include SQL execution plans in the profiler.

Use the Performance Monitoring Tool (Perfmon) to diagnose performance problems Basic use of Performance Monitor

Windows has lots of built-in objects with their corresponding performance counters. These are installed when you install Windows. While SQL Server gets installed, performance counters for SQL Server also get installed. Hence, these counters are available when you define a performance counter log. Follow these steps to create a performance counter log:

Launch the Performance Monitor Tool from Tools->Performance Monitor in the SQL Profiler tool.

Create a new performance counter log by clicking on Counter Logs->New Log Settings.

Specify the log file name and press OK.

Specify the name for the performance counter log

Click on the "Add Counters" button to select the preferred counters in the newly created counter log.

Add the preferred counters by selecting the desired objects and their corresponding counters from the list. Click on "Close" when

done.

Specify the objects and the corresponding counters

The selected counters will be displayed in the form.

Specify the counters

Click on the Log Files tab and click on the "Configure" tab to specify the log file location and modify the log file name if

required. Click "OK" when done.

Click on the "Schedule" tab to specify a schedule for reading the counter information and write in the log file. Optionally, you can

also select "Manually" for the "Start log" and "Stop log" options, in which case, the counter data will be logged after you start the

performance counter log.

Click on the "General" tab and specify the interval for gathering counter data.

Click on the "General" tab and specify the interval for gathering counter data.

For viewing log data, close and open the Performance Monitor Tool again. Click on the View Log icon (the icon in the red box) to

view the counter log. Click on the "Source" tab and select the "Log files" radio button, and add the log file to view by clicking on

the "Add" button.

By default, only three default counters are selected to be shown in the counter log output. Specify other counters (that were included

while creating the counter log) by clicking on the "Data" tab and selecting the desired counters by clicking on the "Add" button.

Click the "OK" button to view the performance counter log output in a graphical view.

Correlate the performance counter log and SQL Profiler trace for better investigation

The SQL Profiler can give you information about long running queries, but it cannot provide you with the context information to

explain the reason for the long query execution time.

Correlating these two tool outputs can help you in identifying the exact root cause of the performance problem. For example, you

might find that each time the query takes 10 seconds to execute in the production server, the CPU utilization reaches up to 100%.

So, instead of trying to tune the SQL, you should investigate the reason why the CPU utilization rises up to 100%, to optimize the

query performance.

Follow these steps to correlate the SQL Profiler trace with the performance counter log:

Create a performance counter log by incorporating the following common performance counters. Specify "Manual" option for

starting and stopping the counter log.

o Network Interface\Output Queue length

o Processor\%Processor Time

o SQL Server: Buffer Manager\Buffer Cache Hit Ratio

o SQL Server: Buffer Manager\Page Life Expectancy

o SQL Server: SQL Statistics\Batch Requests/sec

o SQL Server: SQL Statistics\SQL Compilations

o SQL Server: SQL Statistics\SQL Re-compilations/sec

Create the performance counter log, but don't start it.

Correlating these two tool outputs can help you in identifying the exact root cause of the performance problem. For example, you

might find that each time the query takes 10 seconds to execute in the production server, the CPU utilization reaches up to 100%.

So, instead of trying to tune the SQL, you should investigate the reason why the CPU utilization rises up to 100%, to optimize the

query performance.

Follow these steps to correlate the SQL Profiler trace with the performance counter log:

Create a performance counter log by incorporating the following common performance counters. Specify "Manual"

option for starting and stopping the counter log.

o Network Interface\Output Queue length

o Processor\%Processor Time

o SQL Server: Buffer Manager\Buffer Cache Hit Ratio

o SQL Server: Buffer Manager\Page Life Expectancy

o SQL Server: SQL Statistics\Batch Requests/sec

o SQL Server: SQL Statistics\SQL Compilations

o SQL Server: SQL Statistics\SQL Re-compilations/sec

Create the performance counter log, but don't start it.

Click on a particular TSQL in the profiler trace output (in the upper part of the window). You'll see that a red vertical bar will be

set in the performance counter log output to indicate the particular counter statistics when that particular query was being executed.

Similarly, click on the performance counter log output any where you see a certain performance counter's value is high (or, above

the normal value). You'll see that the corresponding TSQL that was being executed on the database server will be highlighted in

the SQL Profiler trace output.

Step 6: Organize file groups and files in the database When a SQL Server database is created, the database server internally creates a number of files in the file system. Every database

related object that gets created later in the database is actually stored inside these files.

A SQL Server database has the following three kinds of files:

.mdf file: This is the primary data file. There can be only one primary data file for each database. All system objects

reside in the primary data file, and if a secondary data file is not created, all user objects (user created database objects)

also take their place in the primary data file.

.ndf file: These are the secondary data files, which are optional. These files also contain user created objects.

o .ldf file: These are the Transaction log files. These files could be one or many in number. They contain Transaction logs.

o The primary file group must be totally separate, and should be left to have only system objects, and no user defined object should

be created on the primary file group. Also, the primary file group should not be set as the default file group. Separating the system

objects from other user objects will increase performance and enhance the ability to access tables in cases of serious data failures.

o If there are N physical disk drives available in the system, then try to create N files per file group and put each one in a separate

disk. This will allow distributing disk I/O load over multiple disks, and will increase performance.

o For frequently accessed tables containing indexes, put the tables and the indexes in separate file groups. This would enable reading

the index and table data faster.

o For frequently accessed table containing Text or Image columns, create a separate file group and put the text, next, and image

columns in that file group on different physical disks, and put the tables in a different file group. This would enable faster data

retrieval from the table with queries that don't contain text or image columns.

o Put the transaction log file on a different physical disk that is not used by the data files. The logging operation (Transaction log

writing operation) is more write-intensive, and hence it is important to have the log on the disk that has good I/O performance.

o Consider assigning "Read only" tables into a file group that is marked as "Read only". This would enable faster data retrieval from

these read only tables. Similarly, assign "Write only" tables in a different file group to allow for faster updates.

o Do not let SQL Server fire the "Auto grow" feature too often because it is a costly operation. Set an "Auto grow" increment value

so that the database size is increased less frequently (say, once per week). Similarly, do not use the "Auto shrink" feature for the

same reason. Disable it, and either shrink the database size manually, or use a scheduled operation that runs in a timed interval

(say, once a month).

Step 7 (The bonus step): Better-manage DBMS objects, Use TSQL templates To help you in this regard, I have developed some TSQL templates that you can use to develop

DBMS objects using a consistent structure, within a short amount of time.

I've attached some sample templates of various DBMS objects in SQL Server. These are:

Template_StoredProcedure.txt: Template for developing Stored Procedures

Template_View.txt: Template for developing Views

Template_Trigger.txt: Template for developing Triggers

Template_ScalarFunction.txt: Template for developing Scalar functions

Template_TableValuedFunction.txt: Template for developing Table valued functions

How to create templates At first, you need to create templates in your SQL Server Management Studio. To do this, you

need to download the attached templates and follow the steps given below. I've used

Template_StoredProcedure.txt for creating the Stored Procedure template. You can follow the

same procedure to create the other templates.

Open SQL Server Management Studio and go to View->Template Explorer.

Go to Template Explorer, "Stored Procedure" node, and expand it:

Rename the newly created blank template as follows:

Right click on the newly created template and open it in Edit mode, as follows:

SQL Server Management Studio will ask for credentials. Provide valid credentials, and press

"Connect" to access the database instance and to edit the template.

Open the attached Template_StoredProcedure.txt in an editor, copy all the contents, and paste onto

the template that is being edited in Management Studio:

Save the template by pressing the Save button in Management Studio:

STEP 8 : Performance tuning tips for database developers

Basically, the goal is to index the major searching and ordering columns.

Note that if your tables are constantly hammered by INSERT, UPDATE, and DELETE, you should be

careful when indexing—you could end up decreasing performance as all indexes need to be

modified after these operations.

STEP 9 LIST of 15 things to do a matter of tune performance

Below is list of the top 15 things I believe developers should do as a matter of course to tune performance when

coding. These are the low hanging fruit of SQL Server performance – they are easy to do and often have a substantial

impact. Doing these won’t guarantee lightening fast performance, but it won’t be slow either.

1. Create a primary key on each table you create and unless you are really knowledgeable enough to figure out a better

plan, make it the clustered index (note that if you set the primary key in Enterprise Manager it will cluster it by

default).

2. Create an index on any column that is a foreign key. If you know it will be unique, set the flag to force the index to

be unique.

3. Don’t index anything else (yet).

4. Unless you need a different behaviour, always owner qualify your objects when you reference them in TSQL. Use

dbo.sysdatabases instead of just sysdatabases.

5. Use set nocount on at the top of each stored procedure (and set nocount off) at the bottom.

6. Think hard about locking. If you’re not writing banking software, would it matter that you take a chance on a dirty

read? You can use the NOLOCK hint, but it’s often easier to use SET TRANSACTION ISOLATION LEVEL

READ UNCOMMITTED at the top of the procedure, then reset to READ COMMITTED at the bottom.

7. I know you’ve heard it a million times, but only return the columns and the rows you need.

8. Use transactions when appropriate, but allow zero user interaction while the transaction is in progress. I try to do

all my transactions inside a stored procedure.

9. Avoid temp tables as much as you can, but if you need a temp table, create it explicitly using Create Table #temp.

10. Avoid NOT IN, instead use a left outer join – even though it’s often easier to visualize the NOT IN.

11. If you insist on using dynamic sql (executing a concatenated string), use named parameters and sp_executesql

(rather than EXEC) so you have a chance of reusing the query plan. While it’s simplistic to say that stored

procedures are always the right answer, it’s also close enough that you won’t go wrong using them.

12. Get in the habit of profiling your code before and after each change. While you should keep in mind the depth of

the change, if you see more than a 10-15% increase in CPU, Reads, or Writes it probably needs to be reviewed.

13. Look for every possible way to reduce the number of round trips to the server. Returning multiple resultsets is one

way to do this.

14. Avoid index and join hints.

15. When you’re done coding, set Profiler to monitor statements from your machine only, then run through the

application from start to finish once. Take a look at the number of reads and writes, and the number of calls to the

server. See anything that looks unusual? It’s not uncommon to see calls to procedures that are no longer used, or to

see duplicate calls. Impress your DBA by asking him to review those results with you.

STEP:10 PerfMon Objects and Counters for SQL Server Benchmark

PerfMon Objects and Counters Useful for SQL Server Baselining and Benchmarking

When configuring PerfMon for a baseline or benchmark test, consider using the following objects and counters. By default, set the PerfMon session to record the log file to a location other than the system under test. Set the polling frequency to be as low as is acceptable keeping in mind the added overhead created by a frequent polling interval. A typical polling interval might be anywhere from every 3 seconds to 30 seconds, or even every minute, under some baselines or benchmark tests.

Object Counter Rule of Thumb Notes

Paging File % Usage < 70% The amount of page file currently in use. High page file usage

indicates memory shortages that are so severe they must be spilled to the I/O subsystem, resulting in a serious performance bottleneck.

Processor % Processor Time

<= 80% The amount of total CPU in use across all system activities. The higher this metric is, the more likely users will be delayed.

Processor % Privilege Time

< 40% of % Processor Time

The amount of time spent executing kernel commands (i.e. real work) like SQL Server IO requests. This value should seldom rise above 40% of total %Processor Time.

Process(sqlservr) Process(msmdsrv)

% Processor Time

< 80% This is the amount of CPU used solely by SQL Server (sqlservr) and/or Analysis Services (msmdsrv) process threads. It’s even more important to monitor both of these metrics if the server is running both services simultaneously. In dedicated servers, you may monitor the single service in use on that machine.

System Processor Queue Length

< 2 This is the single best indicator of CPU pressure on the Windows Server. It shows how many threads are currently waiting on an answer by a CPU on the server. Less than 12 per CPU is good/fair, less than 8 is better, and less than 2 is best

System Context Switches/sec

< 1500 This metric shows the number of execution contexts switched in the last second. High-end hardware can now handle very high amounts of context switching, but in the absence of any other indicators greater than 6000 is a busy system, less than 6000 is fair, less than 3000 is good, and less than 1500 is excellent.

Physical Disk Avg ms/read < 8 This is metric indicates the time, in milliseconds, spent waiting to complete a read operation on average. It is also known as read latency. The most common physical bottleneck is usually the I/O subsystem. Microsoft’s guidance for HDD-based systems is that read latency of greater than 20 is poor, less than 20 is good/fair, less than 12 is better, and less than 8 is best.

Physical Disk Avg ms/write <1

This is metric indicates the time, in milliseconds, spent waiting to complete a write operation on average. It is also known as write latency. As with the previous metric, Microsoft’s guidance for HDD-based systems is that write latencies of greater than 4 is poor, less than 4 is fair, less than 2 is better, and 1 or less is best – on systems with a write-cache. On I/O subsystems without a write-cache, a write latency greater than 20 is poor, less than is 20 fair, less than 12 is better, and 8 or less is best.

Memory Available Mbytes

=>300 An indicator showing the amount of physical memory, in megabytes, available to run processes on the machine. You generally want a few hundred MB available to handle the random file copy or similar activity needed for system administration.

SQL Server: Memory Manager

Memory Grants Pending

~0 This counter shows the current number of processes waiting for a workspace memory grant within SQL Server. You want this value to be close to zero most of the time. If it is frequently greater than zero, then SQL Server needs more memory or is running T-SQL code that is wasteful of memory.

SQL Server: Buffer Manager

Free List Stalls/sec

< 2 This counter measures the frequency that requests for data buffer pages are suspended (a.k.a “stalled”) because there are no data cache buffers. If this metric is greater than 2, then SQL Server needs more memory or is running T-SQL code that is wasteful of memory.

SQL Server: Databases

Log Flush Waits/sec

~0 This counter shows, effectively, the number of times per second that SQL Server must wait for pages to be written to the transaction log. Only highly-stressed systems should have a sustained value of greater than zero.

SQL Server: General Statistics

User Connections

Proportionate to Batch Requests /sec

This metric shows the number of users currently connected to the SQL Server. For most workloads, it should roughly track with Batch Requests/Sec, rising and falling together. Taken with Batch Requests/Sec, it is a good indicator of overall busyness of a SQL Server.

SQL Server: Buffer Manager

Stolen Pages/sec

Proportionate to Batch Requests /sec

This metric shows the number of 8k pages that were “stolen” from the data cache to satisfy other memory needs, like plan cache and workspace memory. It’s a good indicator of the rate of data flow among SQL Server caches and should remain proportionate to Batch Requests/sec.

SQL Server: SQL Statistics

Batch Requests/Sec

Proportionate to workload

This shows the overall busyness of a SQL Server, that is, the number of batch requests the database engine receives per second. High-end servers running modern versions of SQL Server can sustain 10’s or even 100’s of thousands of batch requests per second on a well-tuned workload. (Note - this counter does not track all Hekaton transactions).

SQL Server: SQL Statistics

SQL Server: SQL Statistics

< 10% of Batch Requests/Sec

This indicator shows the number of times that T-SQL compilations occurred, per second, including recompiles. In general for OLTP workloads, this metric should remain proportionate to the number of Batch Requests/sec. High values may indicate excessive ad hoc querying due to poorly coded T-SQL.

SQL Server: SQL Statistics

SQL Re-Compilations/sec

< 10% of SQL Compil-ations/sec

This indicator shows the number of times, per second, that T-SQL compiled code attempted to execute but had to be recompiled before completion. This number is preferably at or near zero, since recompiles can cause deadlocks and exclusive compile locks. For most workloads, it should remain in proportion to Batch Requests/sec and SQL Compilations/ sec.

https://www.sqlsentry.com/help/ug/webframe.html?Performance_Metrics.html#Performance_Metrics.html

https://www.sqlskills.com/blogs/jonathan/the-accidental-dba-day-21-of-30-essential-perfmon-counters/