DAT382 Troubleshooting Deadlocks in SQL Server 2000 Ron Talmage Prôspice, L.L.C. [email protected]

DAT382DAT382

Troubleshooting Deadlocks in Troubleshooting Deadlocks in SQL Server 2000SQL Server 2000Ron TalmageRon Talmage

PrPrôôspice, L.L.C.spice, L.L.C.

[email protected]@prospice.com

Agenda: Troubleshooting Agenda: Troubleshooting Deadlocks in SQL Server 2000Deadlocks in SQL Server 2000• I. Deadlocking ConceptsI. Deadlocking Concepts• II. Types of Lock DeadlocksII. Types of Lock Deadlocks• III. Gathering Deadlock InformationIII. Gathering Deadlock Information• IV. Finding CausesIV. Finding Causes• V. Resolution Techniques V. Resolution Techniques

• Focus of this session will be on deadlocks due Focus of this session will be on deadlocks due to lock contentionto lock contention

Troubleshooting Deadlocks in Troubleshooting Deadlocks in SQL Server 2000SQL Server 2000

• I. Deadlocking ConceptsI. Deadlocking Concepts• II. Types of DeadlocksII. Types of Deadlocks• III. Gathering Deadlock InformationIII. Gathering Deadlock Information• IV. Finding CausesIV. Finding Causes• V. Resolution Techniques V. Resolution Techniques

• A SQL Server deadlock involves two or A SQL Server deadlock involves two or more server threads or processes (spids)more server threads or processes (spids)

• Each must wait to acquire some resource Each must wait to acquire some resource that the other thread already hasthat the other thread already has

• Neither thread can finish, resulting in a Neither thread can finish, resulting in a deadly embracedeadly embrace

Deadlocking DefinedDeadlocking Defined

Deadlocking IllustratedDeadlocking Illustrated

ResourceResourceResourceResourceThread 1Thread 1 Thread 2Thread 2

GrantGrant

WaitWait

BlockedBlockedrequestrequest

SQL Server Resource WaitsSQL Server Resource Waits• According to BOL, SQL Server threads can wait onAccording to BOL, SQL Server threads can wait on– LocksLocks– Parallel query resourcesParallel query resources– Other threads Other threads – Memory resourceMemory resource– Application eventsApplication events

• Deadlocking can occur with locks, parallelism, threads, and Deadlocking can occur with locks, parallelism, threads, and application events.application events.– Memory waits are resolved by query time-out.Memory waits are resolved by query time-out.– Sp_getapplock deadlocks are treated differentlySp_getapplock deadlocks are treated differently

• The most frequent source of SQL Server deadlocking is The most frequent source of SQL Server deadlocking is resource lockingresource locking

Deadlocks Involving LocksDeadlocks Involving Locks

• Involve two or more spids, usually each in a Involve two or more spids, usually each in a transaction, and one or more resourcestransaction, and one or more resources

• The deadlock takes place in two stages:The deadlock takes place in two stages:– Grant stageGrant stage

• Each spid is granted a lock on its resourceEach spid is granted a lock on its resource

– Blocked Request stageBlocked Request stage• Each spid requests an incompatible lock on the other thread's Each spid requests an incompatible lock on the other thread's

resourceresource• Each spid waits on the other to release its locks before it can Each spid waits on the other to release its locks before it can

complete, entering a WAIT statecomplete, entering a WAIT state

• SQL Server detects the deadlocked state and chooses SQL Server detects the deadlocked state and chooses one of the spids as the victimone of the spids as the victim

Sample Deadlock Transaction Sample Deadlock Transaction HistoryHistoryTimeTime Spid 1Spid 1 Spid 2Spid 2

T1T1 Begin TranBegin Tran Begin TranBegin Tran

T2T2

GrantedGranted

Lock Request Lock Request

T3T3

GrantedGranted


T4T4

BlockedBlocked


(WAIT on Tran2)(WAIT on Tran2)

T5T5

BlockedBlocked

Lock RequestLock Request

(WAIT on Tran1)(WAIT on Tran1)

T6T6 Deadlock VictimDeadlock Victim (blocking removed)(blocking removed)

T7T7 CommitCommit

Note: DeadlockingNote: Deadlockingis more than Blockingis more than Blocking

• Blocking occurs when one spid is waiting on Blocking occurs when one spid is waiting on anotheranother

• Some blocking is normal Some blocking is normal – Expected in an RDBMS that uses locking to maintain Expected in an RDBMS that uses locking to maintain

transaction isolation, and not versioningtransaction isolation, and not versioning– Only blocks with long durations should be considered a Only blocks with long durations should be considered a

problemproblem

• Lock-based deadlocking is a special type of Lock-based deadlocking is a special type of blockingblocking– Two or more spids mutually block each otherTwo or more spids mutually block each other

How SQL ServerHow SQL Serverhandles a Deadlockhandles a Deadlock• The Lock Monitor thread (spid 4) detects the deadlockThe Lock Monitor thread (spid 4) detects the deadlock– Uses periodic detection (<5 seconds)Uses periodic detection (<5 seconds)

• Automatically chooses one spid as the deadlock Automatically chooses one spid as the deadlock victimvictim– Rolls back its transactionRolls back its transaction– Returns error 1205 to the clientReturns error 1205 to the client– Cancels its current queryCancels its current query

• Generally chooses the least expensive transaction to Generally chooses the least expensive transaction to roll backroll back– You can override this somewhat using SET You can override this somewhat using SET

DEADLOCK_PRIORITY to LOW for a sessionDEADLOCK_PRIORITY to LOW for a session

Error 1205 NotesError 1205 Notes• Text of the message:Text of the message:

"Server: Msg 1205, Level 13, State 50, Line 1"Server: Msg 1205, Level 13, State 50, Line 1Transaction (Process ID 54) was deadlocked on lock resources with another Transaction (Process ID 54) was deadlocked on lock resources with another

process and has been chosen as the deadlock victim. Rerun the process and has been chosen as the deadlock victim. Rerun the transaction."transaction."

• Error 1205 is non-fatal - connection not terminatedError 1205 is non-fatal - connection not terminated– SQL Server aborts the batch and cancels the querySQL Server aborts the batch and cancels the query– Error 1205 cannot be detected by @@ERROR within the batch or from Error 1205 cannot be detected by @@ERROR within the batch or from

calling a stored procedurecalling a stored procedure– Severity level 13 – correctable by the userSeverity level 13 – correctable by the user

• Does not revealDoes not reveal– Statements at the Grant stageStatements at the Grant stage– Statements at the Blocked Request stageStatements at the Blocked Request stage– The identity of the other participating spid or spidsThe identity of the other participating spid or spids

SQL Server Deadlocking FactorsSQL Server Deadlocking Factors

1.1. Lock compatibilityLock compatibility

2.2. Transaction timingTransaction timing

3.3. Order of lock requestsOrder of lock requests

4.4. Isolation levelIsolation level

1. Lock Compatibility (lock modes)1. Lock Compatibility (lock modes)Lock Mode Abbreviation Description

Schema-Stability

Sch-S Used when compiling queries

Schema Modification

Sch-M Used for DDL operations (ALTER or DROP) on a table schema

Shared S Used for reading (read lock)

Update U Used to evaluate prior to writing (may become exclusive)

Exclusive X Used for writing (insert, update, delete)

Intent Shared IS Have or intend to request shared lock(s) at a finer level

Intent Update IU Have or intend to request update lock(s) at a finer level

Intent Exclusive IX Have or intend to request exclusive lock(s) at a finer level

Shared Intent Update

SIU Have shared lock with intent to acquire update lock at a finer level

Shared Intent Exclusive

SIX Have shared lock with intent to acquire exclusive lock at a finer level

Update Intent Exclusive

UIX Have update lock with intent to acquire exclusive lock at a finer level

Bulk Update BU Used with bulk copy into a table with either TABLOCK hint or table lock option is set

Lock Compatibility MatrixLock Compatibility Matrix(default isolation level)(default isolation level)

Lock Mode Requested

SS UU XX ISIS IUIU IXIX SIXSIX Sch-SSch-S Sch-MSch-M BUBU

SS YesYes YesYes No YesYes Yes No No YesYes No No

UU YesYes No No YesYes No No No YesYes No No

XX No No No No No No No YesYes No No

ISIS YesYes YesYes No YesYes YesYes YesYes YesYes YesYes No No

IUIU YesYes YesYes No YesYes YesYes YesYes ? YesYes No No

IXIX NoNo NoNo No YesYes YesYes YesYes No YesYes No No

SIXSIX NoNo NoNo No YesYes ?? NoNo No YesYes No No

Sch-SSch-S YesYes YesYes YesYes YesYes YesYes YesYes YesYes YesYes No YesYes

Sch-MSch-M No No No No No No No No No No

BUBU No No No No No No No YesYes No YesYes

Lock Mode Already Granted

Lock Compatibility ExceptionLock Compatibility Exception

• Exclusively-locked rows can be read on pages with Exclusively-locked rows can be read on pages with no data changes:no data changes:– Xlock row-locking hint (reported by Fernando Guerrero)Xlock row-locking hint (reported by Fernando Guerrero)

Select *Select * From titleauthor (With xlock)From titleauthor (With xlock) Where au_id = '172-32-1176'Where au_id = '172-32-1176'

– A no-op Update statement:A no-op Update statement:Update titleauthor Update titleauthor Set royaltyper = royaltyperSet royaltyper = royaltyper Where au_id = '172-32-1176'Where au_id = '172-32-1176'

• If any data on the page is changed, the exclusive If any data on the page is changed, the exclusive lock appears to be honored.lock appears to be honored.

2. Transaction Timing2. Transaction Timing

• Transactions must overlap in time for Transactions must overlap in time for blocking and deadlocking to occurblocking and deadlocking to occur

• Transactions must have sufficiently long Transactions must have sufficiently long duration to make blocking and duration to make blocking and deadlocking likelydeadlocking likely

• Some deadlocks require a large number Some deadlocks require a large number of concurrent transactions to surfaceof concurrent transactions to surface

3. Order of Lock Requests3. Order of Lock Requests

• Initial locks must be granted before Initial locks must be granted before incompatible locks are requestedincompatible locks are requested

• Changing the order of lock requests may Changing the order of lock requests may release locks early or remove release locks early or remove incompatible locksincompatible locks

4. Isolation Level4. Isolation Level

• Higher isolation levels make deadlocks more Higher isolation levels make deadlocks more likelylikely

• Deadlocks involving all exclusive locks are Deadlocks involving all exclusive locks are independent of isolation levelindependent of isolation level

• Some deadlocks involving shared locks can Some deadlocks involving shared locks can occur with the READ COMMITTED isolation leveloccur with the READ COMMITTED isolation level

• Other deadlocks involving shared locks require Other deadlocks involving shared locks require isolation levels higher than READ COMMITTEDisolation levels higher than READ COMMITTED

Isolation Level and Lock AccumulationIsolation Level and Lock Accumulation

Lock Lock ModeMode

Read Un-Read Un-committedcommitted

Read Read CommittedCommitted

Repeatable Repeatable ReadRead

SerializableSerializable

Shared Held until data read and processed

Held until data read and processed

Held until end of transaction


Update Held until end of transaction unless promoted to exclusive or released

Held until end of transaction unless promoted to exclusive or released

Held until end of transaction unless promoted to exclusive

Held until end of transaction unless promoted to exclusive

Exclusive Held until end of transaction




Troubleshooting DeadlocksTroubleshooting Deadlocksin SQL Server 2000in SQL Server 2000• I. Deadlocking ConceptsI. Deadlocking Concepts• II. Types of Lock DeadlocksII. Types of Lock Deadlocks• III. Gathering Deadlock InformationIII. Gathering Deadlock Information• IV. Finding CausesIV. Finding Causes• V. Resolution Techniques V. Resolution Techniques

Types of Lock DeadlocksTypes of Lock Deadlocks

• 1. Single-Resource Deadlock1. Single-Resource Deadlock– Sometimes called 'conversion' deadlock but also is a cycleSometimes called 'conversion' deadlock but also is a cycle– Involves a single resourceInvolves a single resource– Deadlock occurs when attempting a conversion from compatible Deadlock occurs when attempting a conversion from compatible

to incompatible locksto incompatible locks

• 2. Multi-Resource Deadlock2. Multi-Resource Deadlock– Sometimes called ‘cycle’ deadlockSometimes called ‘cycle’ deadlock– Involves distinct resources for each threadInvolves distinct resources for each thread– Deadlock occurs when each spid requests an incompatible lock Deadlock occurs when each spid requests an incompatible lock

on the other’s resourceon the other’s resource

• 3. Application resource Deadlocks3. Application resource Deadlocks– Use sp_getapplockUse sp_getapplock

1.1. Single-Resource Deadlock Single-Resource Deadlock OverviewOverview

• Involves only one resourceInvolves only one resource• Initial Grant stage requires shared locks, Initial Grant stage requires shared locks,

because both spids must obtain a lock on a because both spids must obtain a lock on a single resourcesingle resource– Requires an isolation level higher than READ Requires an isolation level higher than READ

COMMITTED to hold share locksCOMMITTED to hold share locks

• Blocked Request stage requires exclusive Blocked Request stage requires exclusive lockslocks

Single-ResourceSingle-Resource Deadlock Deadlock Illustrated Illustrated

ResourceResourceSpid 1Spid 1 Spid 2Spid 2

GrantGrant

WaitWait


Single-Resource Deadlock ExampleSingle-Resource Deadlock Example TimeTime Spid 1Spid 1 Spid 2Spid 2


T2T2

GrantedGranted

Select * Select *

From Authors With (HOLDLOCK) From Authors With (HOLDLOCK)

Where au_id = '172-32-1176'Where au_id = '172-32-1176'

T3T3

GrantedGranted

Select * Select *

From Authors With (HOLDLOCK) From Authors With (HOLDLOCK)


T4T4

BlockedBlocked

Update Authors Update Authors

Set contract = 0 Set contract = 0


T5T5

BlockedBlocked





T7T7 CommitCommit

Single-Resource Deadlock NotesSingle-Resource Deadlock Notes

• Often called 'conversion' deadlocksOften called 'conversion' deadlocks– Attempt is made to 'convert' a shared lock to an exclusive Attempt is made to 'convert' a shared lock to an exclusive

locklock– Update lock mode prevents a type of conversion deadlockUpdate lock mode prevents a type of conversion deadlock

• Require shared locks at the Grant stageRequire shared locks at the Grant stage• Are comparatively rare because they require a higher Are comparatively rare because they require a higher

than default isolation level:than default isolation level:• SQL Server’s default isolation level is READ COMMITTEDSQL Server’s default isolation level is READ COMMITTED• Conversion deadlocks require REPEATABLE READ or Conversion deadlocks require REPEATABLE READ or

SERIALIZABLE isolation levelsSERIALIZABLE isolation levels• HOLDLOCK hint is effectively the SERIALIZABLE isolation levelHOLDLOCK hint is effectively the SERIALIZABLE isolation level

• Require exclusive locks at the Blocked Request stageRequire exclusive locks at the Blocked Request stage– Requests must be blocked by existing share locksRequests must be blocked by existing share locks

2. Multi-Resource Deadlocks 2. Multi-Resource Deadlocks Overview Overview• Involve more than one resourceInvolve more than one resource• Grant stage normally requires exclusive locksGrant stage normally requires exclusive locks• There are many variations; here are two:There are many variations; here are two:– Exclusive only (X-only): spids only make exclusive lock Exclusive only (X-only): spids only make exclusive lock

requestsrequests– Mixed exclusive and shared (X-S): Mixed exclusive and shared (X-S):

• First, spids are granted exclusive locksFirst, spids are granted exclusive locks• Then they request shared locks and are blockedThen they request shared locks and are blocked• Mixed X-S deadlocks can be more subtle and challengingMixed X-S deadlocks can be more subtle and challenging

MRMR Deadlock Illustrated Deadlock Illustrated

Res 1Res 1Spid 1Spid 1 Spid 2Spid 2Res 2Res 2

GrantGrant

WaitWait


X-only MR Deadlock VariationX-only MR Deadlock Variation

• Involves only exclusive locksInvolves only exclusive locks• Simplest type of MR deadlockSimplest type of MR deadlock• Works at any isolation levelWorks at any isolation level– Exclusive locks are always held to the end of Exclusive locks are always held to the end of

a transactiona transaction

X-only MR Deadlock ExampleX-only MR Deadlock Example

TimeTime Spid 1Spid 1 Spid 2Spid 2


T2T2

GrantedGranted




T3T3

GrantedGranted

Update Titles Update Titles

Set ytd_sales = 0 Set ytd_sales = 0

Where title_id = ‘BU1032'Where title_id = ‘BU1032'

T4T4

BlockedBlocked




T5T5

BlockedBlocked





T7T7 CommitCommit

X-only MR Deadlock NotesX-only MR Deadlock Notes

• Relatively easy to diagnoseRelatively easy to diagnose– Involves two spids, each in a transaction, accessing the Involves two spids, each in a transaction, accessing the

same resources in inverse ordersame resources in inverse order

• Could use Update locks in place of Exclusive locksCould use Update locks in place of Exclusive locks– Like an exclusive lock, only one update lock can be Like an exclusive lock, only one update lock can be

placed on a resource at one timeplaced on a resource at one time

• Often used as an example of 'cycle' deadlockOften used as an example of 'cycle' deadlock• Easy to prevent if all resources are accessed in Easy to prevent if all resources are accessed in

the same orderthe same order

Mixed X-S MR Deadlock VariationMixed X-S MR Deadlock Variation

• Can be a more complex and subtle type of Can be a more complex and subtle type of deadlockdeadlock

• Many variations:Many variations:– X-X:S-S, X-X:S-S, – X-X-X:S-S-S,X-X-X:S-S-S,– X-IS:IS-X, X-IS:IS-X, – S-IX:IX-S,S-IX:IX-S,– X-X:U-U, etc.X-X:U-U, etc.

• Can occur with the READ COMMITTED isolation Can occur with the READ COMMITTED isolation levellevel

Mixed X-S MR Example #1Mixed X-S MR Example #1



T2T2

GrantedGranted




T3T3

GrantedGranted




T4T4

RequestRequest

Select *Select *

From Titles From Titles


T5T5

RequestRequest

Select *Select *

From Authors From Authors



T7T7 CommitCommit

Mixed X-S MR Example #2Mixed X-S MR Example #2



T2T2

GrantedGranted

Insert Authors Values Insert Authors Values

('111-11-1111', 'test1', '', '', '', '', ('111-11-1111', 'test1', '', '', '', '', '', '11111', 0)'', '11111', 0)

T3T3

GrantedGranted

Insert Authors Values Insert Authors Values

('111-11-1112', 'test2', '', '', '', '', '', ('111-11-1112', 'test2', '', '', '', '', '', '11111', 0)'11111', 0)

T4T4

RequestRequest

Select * Select *


T5T5

RequestRequest

Select *Select *



T7T7 CommitCommit

Mixed X-S MR Deadlock NotesMixed X-S MR Deadlock Notes

• Can be much more difficult to diagnoseCan be much more difficult to diagnose• Can take place with the default isolation level of Can take place with the default isolation level of

READ COMMITTEDREAD COMMITTED• SELECT statements can also participate with SELECT statements can also participate with

READ COMMITTED or higher isolation levelREAD COMMITTED or higher isolation level• Requires exclusive locks at the initial stageRequires exclusive locks at the initial stage– An update lock is compatible with the subsequent An update lock is compatible with the subsequent

shared locksshared locks

3. A Note About sp_getapplock3. A Note About sp_getapplock

• Used for locking an application resource by nameUsed for locking an application resource by name– Locks a resource nameLocks a resource name

• Shared, Update, Exclusive, IntentExclusive, IntentSharedShared, Update, Exclusive, IntentExclusive, IntentShared

– Procedure is a wrapper around xp_userlockProcedure is a wrapper around xp_userlock– Can be used in a transaction with Commit and RollbackCan be used in a transaction with Commit and Rollback– Managed by the lock managerManaged by the lock manager– Application locks (type APP) are reported by sp_lockApplication locks (type APP) are reported by sp_lock

• DeadlocksDeadlocks– Reported in ProfilerReported in Profiler– Not reported by the –1204 trace flagNot reported by the –1204 trace flag– Stored procedure returns a –3, not 1205 errorStored procedure returns a –3, not 1205 error

Troubleshooting DeadlocksTroubleshooting Deadlocksin SQL Server 2000in SQL Server 2000

• I. Deadlocking ConceptsI. Deadlocking Concepts• II. Types of Lock DeadlocksII. Types of Lock Deadlocks• III. Gathering Deadlock InformationIII. Gathering Deadlock Information• IV. Finding CausesIV. Finding Causes• V. Resolution Techniques V. Resolution Techniques

Gathering Deadlock InformationGathering Deadlock Information

• 1. Use the Perfmon Deadlocks/sec counter1. Use the Perfmon Deadlocks/sec counter• 2. Use the 1204 trace flag2. Use the 1204 trace flag– Redirect deadlock event information to the SQL Redirect deadlock event information to the SQL

Server Error Log Server Error Log

• 3. Use SQL Trace/Profiler 3. Use SQL Trace/Profiler

1. Use the Deadlocks/sec Sysmon 1. Use the Deadlocks/sec Sysmon (Perfmon) counter(Perfmon) counter

• Useful in measuring frequent deadlockingUseful in measuring frequent deadlocking

• Useful when combined with multiuser Useful when combined with multiuser testing to determine the minimum number of testing to determine the minimum number of users required to reproduce deadlocksusers required to reproduce deadlocks

Adding the Deadlock CounterAdding the Deadlock Counter

2. Use the 1204 Trace Flag2. Use the 1204 Trace Flag

• Used in combination with the 3605 flag:Used in combination with the 3605 flag:DBCC TRACEON(3605)DBCC TRACEON(3605)

DBCC TRACEON(1204)DBCC TRACEON(1204)

• Output will be sent to the SQL Server error logOutput will be sent to the SQL Server error log• It can be useful to make a SQL Agent job execute It can be useful to make a SQL Agent job execute

these statements at startupthese statements at startup• Can be used in combination with a SQL Server Can be used in combination with a SQL Server

Agent Alert defined on Error 1205.Agent Alert defined on Error 1205.

Retrieving 1204 OutputRetrieving 1204 Output

• When using Enterprise Manager to read the When using Enterprise Manager to read the 1204 output, statements occur in unusual 1204 output, statements occur in unusual orderorder– Do not export the logDo not export the log

• Instead, use a text editor like Notepad to Instead, use a text editor like Notepad to read the statements in correct orderread the statements in correct order– You can strip out the datetime and spid data to You can strip out the datetime and spid data to

give more readable outputgive more readable output

Sample Trace Flag 1204 Output Sample Trace Flag 1204 Output (using MR Example #2)(using MR Example #2)Deadlock encountered .... Printing deadlock informationDeadlock encountered .... Printing deadlock information

Wait-for graphWait-for graph

Node:1Node:1KEY: 7:1977058079:1 (f600d547433a) CleanCnt:1 Mode: X Flags: 0x0KEY: 7:1977058079:1 (f600d547433a) CleanCnt:1 Mode: X Flags: 0x0 Grant List::Grant List:: Owner:0x19090e20 Mode: X Flg:0x0 Ref:0 Life:02000000 SPID:54 ECID:0Owner:0x19090e20 Mode: X Flg:0x0 Ref:0 Life:02000000 SPID:54 ECID:0 SPID: 54 ECID: 0 Statement Type: SELECT Line #: 1SPID: 54 ECID: 0 Statement Type: SELECT Line #: 1 Input Buf: Language Event: Select * From Authors Input Buf: Language Event: Select * From Authors

Requested By: Requested By: ResType:LockOwner Stype:'OR' Mode: S SPID:53 ECID:0 Ec:(0x19c5d530) Value:0x19090dc0 Cost:(0/114)ResType:LockOwner Stype:'OR' Mode: S SPID:53 ECID:0 Ec:(0x19c5d530) Value:0x19090dc0 Cost:(0/114)

Node:2Node:2KEY: 7:1977058079:1 (f5006f164aa3) CleanCnt:1 Mode: X Flags: 0x0KEY: 7:1977058079:1 (f5006f164aa3) CleanCnt:1 Mode: X Flags: 0x0 Grant List::Grant List:: Owner:0x19091000 Mode: X Flg:0x0 Ref:0 Life:02000000 SPID:53 ECID:0Owner:0x19091000 Mode: X Flg:0x0 Ref:0 Life:02000000 SPID:53 ECID:0 SPID: 53 ECID: 0 Statement Type: SELECT Line #: 1SPID: 53 ECID: 0 Statement Type: SELECT Line #: 1 Input Buf: Language Event: Select * From Authors Input Buf: Language Event: Select * From Authors

Requested By: Requested By: ResType:LockOwner Stype:'OR' Mode: S SPID:54 ECID:0 Ec:(0x19dc3530) Value:0x19091180 Cost:(0/114)ResType:LockOwner Stype:'OR' Mode: S SPID:54 ECID:0 Ec:(0x19dc3530) Value:0x19091180 Cost:(0/114)Victim Resource Owner:Victim Resource Owner: ResType:LockOwner Stype:'OR' Mode: S SPID:54 ECID:0 Ec:(0x19dc3530) Value:0x19091180 Cost:(0/114)ResType:LockOwner Stype:'OR' Mode: S SPID:54 ECID:0 Ec:(0x19dc3530) Value:0x19091180 Cost:(0/114)

Locked resource (leaf node Locked resource (leaf node of clustered key, I.e. row)of clustered key, I.e. row)

Spid owning the lockSpid owning the lock

Current input bufferCurrent input buffer

Type of lock grantedType of lock granted

Requesting spid and Requesting spid and lock type requestedlock type requested

Deadlock victim spidDeadlock victim spid

Hash of key valueHash of key value

InterpretingInterpretingTrace Flag 1204 OutputTrace Flag 1204 Output• KEY: 7:1977058079:1 (f600d547433a)KEY: 7:1977058079:1 (f600d547433a)– Database 7, Object 1977058079, Indid 1Database 7, Object 1977058079, Indid 1• Reflects a row lock on a clustered indexReflects a row lock on a clustered index

– Hash of key value: f600d547433aHash of key value: f600d547433a

• Spid numbers are important when cross-Spid numbers are important when cross-referencing to SQL Trace datareferencing to SQL Trace data– See 'Displaying Locking Information' and See 'Displaying Locking Information' and

'Troubleshooting Deadlocks' in BOL for more 'Troubleshooting Deadlocks' in BOL for more information information

Trace Flag 1204 Output NotesTrace Flag 1204 Output Notes

• What 1204 shows:What 1204 shows:– The input buffer of the Blocked Request statementsThe input buffer of the Blocked Request statements– The locks held at the Grant stageThe locks held at the Grant stage

• What it does not show:What it does not show:– The statements initiating the Grant locksThe statements initiating the Grant locks– The transaction history for each spidThe transaction history for each spid

• Input buffer may be incompleteInput buffer may be incomplete– Some system deadlocks provide incomplete informationSome system deadlocks provide incomplete information

3. Use SQL Trace with Profiler3. Use SQL Trace with Profiler

• Add Deadlock eventsAdd Deadlock events• Save trace data to a tableSave trace data to a table• Reproduce the transaction history of the Reproduce the transaction history of the

deadlockdeadlock

Configuring SQL Trace/ProfilerConfiguring SQL Trace/Profiler

• When tracing specifically for deadlocks, includeWhen tracing specifically for deadlocks, include– EventsEvents

• Cursors: CursorExecute, CursorPrepareCursors: CursorExecute, CursorPrepare• Errors and Warnings: Attentions and ExceptionsErrors and Warnings: Attentions and Exceptions• Locks: Lock:Deadlock and Lock:Deadlock ChainLocks: Lock:Deadlock and Lock:Deadlock Chain• Stored Procedures: RPC:Completed, RPC:Starting, Stored Procedures: RPC:Completed, RPC:Starting,

SP:StmtCompleted, SP:StmtStartedSP:StmtCompleted, SP:StmtStarted• Transactions: SQL TransactionTransactions: SQL Transaction• TSQL: SQL:BatchCompleted, SQL:BatchStarting, TSQL: SQL:BatchCompleted, SQL:BatchStarting,

SQL:StmtCompleted, SQL:StmtStartingSQL:StmtCompleted, SQL:StmtStarting

– Data Columns:Data Columns:• EventClass, EventSubclass, Spid, IntegerData, Mode, StartTime, EventClass, EventSubclass, Spid, IntegerData, Mode, StartTime,

TextData, DatabaseID, IndexID, HostName, ObjectID, TextData, DatabaseID, IndexID, HostName, ObjectID, TransactionID, SeverityTransactionID, Severity

SQL Trace and SP2SQL Trace and SP2

• Make sure you apply SQL Server 2000 Make sure you apply SQL Server 2000 Service Pack 2 for deadlock tracingService Pack 2 for deadlock tracing

• See "FIX: Deadlock Information Reported See "FIX: Deadlock Information Reported with SQL Server 2000 Profiler Is Incorrect" with SQL Server 2000 Profiler Is Incorrect" (Q282749)(Q282749)

Viewing SQL Trace Output in ProfilerViewing SQL Trace Output in Profiler

Deadlock spids now Deadlock spids now cross-referenced in cross-referenced in SP2SP2

SQL Trace Output NotesSQL Trace Output Notes

• What SQL Trace shows: What SQL Trace shows: – The statements involved in the deadlockThe statements involved in the deadlock

• The deadlock victim will have a statement starting but not The deadlock victim will have a statement starting but not endingending

– Shows the beginning and end of each spid's Shows the beginning and end of each spid's transactiontransaction

– Can be used to reproduce each deadlocking spid's Can be used to reproduce each deadlocking spid's transaction historytransaction history

• What it does not show:What it does not show:– The statements gaining the initial locks at the Grant The statements gaining the initial locks at the Grant

stagestage

Agenda: Troubleshooting Agenda: Troubleshooting Deadlocks in SQL Server 2000Deadlocks in SQL Server 2000• I. Deadlocking ConceptsI. Deadlocking Concepts• II. Types of Lock DeadlocksII. Types of Lock Deadlocks• III. Gathering Deadlock InformationIII. Gathering Deadlock Information• IV. Finding CausesIV. Finding Causes• V. Resolution Techniques V. Resolution Techniques

Isolating the CauseIsolating the Cause

• Find a method to reproduce the deadlockFind a method to reproduce the deadlock– Stress test, load testStress test, load test– Minimal conditions possibleMinimal conditions possible

• Reproduce with both 1204 Trace Flag and SQL Trace Reproduce with both 1204 Trace Flag and SQL Trace activeactive

• Examine Trace Flag 1204 outputExamine Trace Flag 1204 output– Determine which spids are involvedDetermine which spids are involved– Determine what types of locks are involvedDetermine what types of locks are involved

• Get each spid’s transaction historyGet each spid’s transaction history– Save SQL Trace data output to SQL Server tableSave SQL Trace data output to SQL Server table– Reproduce the deadlock under controlled conditionsReproduce the deadlock under controlled conditions

Steps for Reproducing a Steps for Reproducing a Complex DeadlockComplex Deadlock• Save the Trace output to a tableSave the Trace output to a table• Find the beginning and ending row numbers of Find the beginning and ending row numbers of

each spid's transactioneach spid's transaction• Select the StartTime and TextData for a given spid Select the StartTime and TextData for a given spid

the transaction in questionthe transaction in question• Arrange the transaction histories according to Arrange the transaction histories according to

StartTimeStartTime• Remove extraneous commandsRemove extraneous commands• Observe locks held using sp_lockObserve locks held using sp_lock

Extracting a Transaction HistoryExtracting a Transaction History

Reproducing the DeadlockReproducing the Deadlock

Observing LocksObserving Locks

• Use sp_lock to detect locks per spidUse sp_lock to detect locks per spid• Additional tools:Additional tools:

• Both sp_lock2 and sp_blockinglocks from Kalen Both sp_lock2 and sp_blockinglocks from Kalen Delaney, Delaney, Inside SQL Server 2000Inside SQL Server 2000

• Identify the statements resulting in the Grant Identify the statements resulting in the Grant stage locksstage locks• Find the statement causing the lock found in the Find the statement causing the lock found in the

1204 output1204 output

Identifying the Cause (query)Identifying the Cause (query)

Identifying the Cause (locks)Identifying the Cause (locks)

KEY: 7:1977058079:1 (f600d547433a)KEY: 7:1977058079:1 (f600d547433a)(from trace flag 1204 output)(from trace flag 1204 output)

Troubleshooting DeadlocksTroubleshooting Deadlocksin SQL Server 2000in SQL Server 2000• I. Deadlocking ConceptsI. Deadlocking Concepts• II. Types of Lock DeadlocksII. Types of Lock Deadlocks• III. Gathering Deadlock InformationIII. Gathering Deadlock Information• IV. Finding CausesIV. Finding Causes• V. Resolution TechniquesV. Resolution Techniques

Resolution Techniques Resolution Techniques

• 1. Remove incompatible lock requests1. Remove incompatible lock requests• 2. Change the timing of transactions2. Change the timing of transactions• 3. Change the order of resource requests3. Change the order of resource requests• 4. Lower the isolation level 4. Lower the isolation level

1. Remove incompatible lock 1. Remove incompatible lock requestsrequests

• Queries may be obtaining too many locksQueries may be obtaining too many locks– WHERE clauses may be too broadWHERE clauses may be too broad• I.e., reading a whole table or range of rows where I.e., reading a whole table or range of rows where

only one row's data is actually desiredonly one row's data is actually desired

• Queries may be obtaining the wrong kinds Queries may be obtaining the wrong kinds of locksof locks– Using hints, such as HOLDLOCK or UPDLOCK, Using hints, such as HOLDLOCK or UPDLOCK,

unnecessarilyunnecessarily

2. Change the timing of 2. Change the timing of transactionstransactions

• Make transactions shorterMake transactions shorter– Long transactions increase the likelihood of Long transactions increase the likelihood of

deadlocksdeadlocks

• Transactions may not need to overlapTransactions may not need to overlap– Run batch processes sequentially rather than in Run batch processes sequentially rather than in

parallelparallel

3. Change the order of resource 3. Change the order of resource requestsrequests

• For X-only Multi-Resource deadlocks, For X-only Multi-Resource deadlocks, ensure that resources are accessed in the ensure that resources are accessed in the same ordersame order

• May not work for Mixed X-S MR deadlocksMay not work for Mixed X-S MR deadlocks

4. Lower the Isolation Level of the 4. Lower the Isolation Level of the Transaction or SessionTransaction or Session• Use SET TRANSACTION ISOLATION LEVELUse SET TRANSACTION ISOLATION LEVEL• For single-resource 'conversion' deadlocksFor single-resource 'conversion' deadlocks– lower the isolation level to READ COMMITTED or lower the isolation level to READ COMMITTED or

READ UNCOMMITTEDREAD UNCOMMITTED

• For Mixed X-S MR deadlocks, with Share locks For Mixed X-S MR deadlocks, with Share locks involved, lower the isolation level to READ involved, lower the isolation level to READ UNCOMMITTEDUNCOMMITTED

• For X-only MR deadlocks, lowering the isolation For X-only MR deadlocks, lowering the isolation level will not worklevel will not work– Exclusive locks are held until the end of the transactionExclusive locks are held until the end of the transaction

Lower the Isolation Level of a Lower the Isolation Level of a QueryQuery• For Mixed X-S MR deadlocks, use a locking For Mixed X-S MR deadlocks, use a locking

hint on the Share lock queries:hint on the Share lock queries:– Use READPAST when it is not important to read Use READPAST when it is not important to read

uncommitted datauncommitted data– Use NOLOCK when uncommited data must Use NOLOCK when uncommited data must

also be readalso be read

• Most effective for low-impact resolutionMost effective for low-impact resolution

Some Sample Deadlocks: Trace Some Sample Deadlocks: Trace Flag 1204 OutputFlag 1204 Output

• Deadlock involving ParallelismDeadlock involving Parallelism• Cascading deadlockCascading deadlock• Page and IX deadlockPage and IX deadlock• As time permits…As time permits…

Additional ResourcesAdditional Resources

• SQL Server Locking InternalsSQL Server Locking Internals– Inside SQL Server 2000Inside SQL Server 2000, Kalen Delaney (MS Press), Kalen Delaney (MS Press)

• Transaction Isolation Level ConceptsTransaction Isolation Level Concepts– Transaction Processing: Concepts Transaction Processing: Concepts

and Techniques,and Techniques, Gray and Reuter (Morgan Kaufmann) Gray and Reuter (Morgan Kaufmann)– http://http://research.microsoft.com/~gray/Isolation.docresearch.microsoft.com/~gray/Isolation.doc

“A Critique of ANSI SQL Isolation Levels”– DAT380, Fernando Guerrero, “Concurrency Problems and Locking DAT380, Fernando Guerrero, “Concurrency Problems and Locking

Techniques in SQL Server 2000 and Visual Basic .NET”Techniques in SQL Server 2000 and Visual Basic .NET”

• Relevant Knowledge Base ArticlesRelevant Knowledge Base Articles– Q224453 - INF: Understanding and Resolving SQL Server 7.0 Q224453 - INF: Understanding and Resolving SQL Server 7.0

or 2000 Blocking Problems or 2000 Blocking Problems – Q271509 - INF: How to Monitor SQL Server 2000 BlockingQ271509 - INF: How to Monitor SQL Server 2000 Blocking– Q282749 - FIX: Deadlock Information Reported with SQL Q282749 - FIX: Deadlock Information Reported with SQL

Server 2000 Profiler Is Incorrect Server 2000 Profiler Is Incorrect

http://research.microsoft.com/~gray/Isolation.doc

http://research.microsoft.com/~gray/Isolation.doc

Thank YouThank You

• Please fill out and turn in your evaluation!Please fill out and turn in your evaluation!

• If you have an interesting deadlock, please If you have an interesting deadlock, please send the 1204 output to:send the 1204 output to:– [email protected]@prospice.com

Don’t forget to complete the Don’t forget to complete the on-line Session Feedback form on-line Session Feedback form on the Attendee Web siteon the Attendee Web site

https://web.mseventseurope.com/teched/https://web.mseventseurope.com/teched/

https://web.mseventseurope.com/teched/

Documents

DAT382 Troubleshooting Deadlocks in SQL Server 2000 Ron Talmage Prôspice, L.L.C. [email protected]