8/11/2019 Wait Events in Rac
1/25
Wait Events in RAC
Session 362
Arup Nanda
Longtime Oracle DBAarup.blogspot.com
8/11/2019 Wait Events in Rac
2/25
What is This?
RAC Performance Tuning
I teach a coursePerformance Tuning in RAC
Wait events are useful for understanding the bottlenecks
All single instance wait events are applicable to RAC
RAC has some special cases This session talks about those RAC-specific wait events
This is just a subset of the events; not a comprehensive
list
2Wait Events in RAC
8/11/2019 Wait Events in Rac
3/25
Whats a Wait?
A process in Oracle can only be in three states
Doing something Useful (consuming CPU) ..U
Idle, waiting for some work to be assigned .. I
Waiting for something, e.g. .. W
a block from disk
a lock
a latch (could be waiting on CPU)
Response time = U + I + W
We must accurately measure each component time
before we decide what and how to tune
3Wait Events in RAC
8/11/2019 Wait Events in Rac
4/25
Node 1
Cluster Coordination
RAC for Beginners4
Buffer Cache Buffer Cache
DBWR DBWR
LMS LMS
SCN1
DBWR must get a lock on the database block before writing to
the disk. This is called a Block Lock.
Node 2
Database
SCN2
Checkpoint!
Checkpoint!
8/11/2019 Wait Events in Rac
5/25
Cache Fusion
5Wait Events in RAC
Instance 1 Instance 2
5
session1 session2
5
Has
modifiedit
Wants to
modify it
Will get it via
interconnect
8/11/2019 Wait Events in Rac
6/25
Checking for Buffers
How exactly is this checkperformed?
By checking for a lock on the
block
The request comes to the GrantQueue of the block
Someone checks that no other
instance has any lock
Instance 1 can read from the
disk
i.e. Instance 1 is grantedthe
lock
6
Block
SID1
SID2
SID3
Grant
Queue
Convert
Queue
SID5
SID6
SID7
Wait Events in RAC
8/11/2019 Wait Events in Rac
7/25
Master Instance
Only one instance holds the grant and
convert queues of a specific block This instance is called Master Instance
of that block
Master instance varies for each block
The memory structure that shows themaster instance of a buffer is calledGlobal Resource Directory (GRD)
That is replicated across all instances
The requesting instance must check theGRD to find the master instance
Then make a request to the masterinstance for the lock
7Wait Events in RAC
Block
SID1
SID2
SID3
Grant
Queue
Convert
Queue
SID5
SID6
SID7
8/11/2019 Wait Events in Rac
8/25
Scenario 1
Session connected to Instance 1 wants to select a blockfrom the table
Activities by Instance 1
1. Check its own buffer cache to see if the block exists1. If it is found, can it just use it?
2. If it not found, can it select from the disk?
2. If not, then check the other instances
How will it know which copy of the block is the bestsource?
8Wait Events in RAC
Instance 1 Instance 2Session
8/11/2019 Wait Events in Rac
9/25
Node 2Node 1
Cache Fusion
9Wait Events in RAC
Buffer Cache Buffer Cache
SMON SMON
LMS LMS
When node 2 wants a buffer, it sends a message to the
other instance. The message is sent to the LMS (Lock
Management Server) of the other instance. LMS then
sends the buffer to the other instance. LMS is also called
Global Cache Server (GCS).
message
buffer
8/11/2019 Wait Events in Rac
10/25
Grant Scenario 2
1. Check its buffer cache to see if the block exists
2. And the buffer is found. Can Instance1 use it?
Not really. The buffer may be old; it may have been changed
3. LMS of node1 sends message to master of the buffer
3. Master checks the GES and doesnt sees any lock
4. Instance 1 is grantedthe global block lock
5. No buffer actually gets transferred
10Wait Events in RAC
8/11/2019 Wait Events in Rac
11/25
Grant Scenario 3
Instance 1 is the master
Then it doesnt have to make a request for the grant
In summary, here are the possible scenarios when
Instance1 requests a buffer
Instance1 is the master; so no more processing is
required
No one has the lock on the buffer, the grant is made
by the master immediately
Another instance has the buffer in an incompatible
mode. It has to be changed.
11Wait Events in RAC
8/11/2019 Wait Events in Rac
12/25
Buffer States and Locks
Buffers can be gotten in two states
Currentwhen the intention is to modify Shared Currentmost recent copy. One copy per
instance. Same as disk
Exclusive Currentonly one copy in the entire
cluster. No shared current present
CRwhen the intention is to only select
Locks facilitate the state enforcement
XCUR for Exclusive Current
SCUR for Shared Current
No locking for CR
12Wait Events in RAC
8/11/2019 Wait Events in Rac
13/25
Placeholder Event
When the buffer is first requested, the session does not
know which of the three paths it will go on to
Therefore it is assigned aplaceholderevent
This event is known as gc cr block request (for Consistent
Read requests)
If the request is made for the buffer in Current mode, the
event is gc current block request
When one of the three options is chosen, the appropriate
event replaces the placeholder event
13Wait Events in RAC
8/11/2019 Wait Events in Rac
14/25
Grant Event
If the session merely requests a grant from the master:
It waits with the gc cr|current grant 2-way event
gc cr grant 2-way, for requests for buffers in Consistent
Read mode
gc current grant 2-way, for requests for buffers in
Currentmode
14Wait Events in RAC
8/11/2019 Wait Events in Rac
15/25
15
gc current|cr grant 2-way
Instance 1
Instance 2
Session
Database
LMSLMS
gc current block requestgc current grant 2-way
GRD GES
db file scattered read
Wait Events in RAC
Block
SID1
SID2
SID3
Grant
Queue
Conver
t Queue
SID5
SID6
SID7
request
granted
8/11/2019 Wait Events in Rac
16/25
Block Event
After the request is made
and assuming that the buffer is not in the local cache
the buffer may be found in another instance
Requestor requests the buffer from the holding
instance
Possibilities:
Holder is also the master of that buffer
Holder is not the master
If the holder is the master, the requesting session waitswith the event gc cr|current block 2-way
16Wait Events in RAC
8/11/2019 Wait Events in Rac
17/25
17
gc current|cr block 2-way
Wait Events in RAC
Instance 1
Instance 2
Session
Database
LMSLMS
gc current block requestgc current block 2-way
GRD GES
Block
SID1
SID2
SID3
Grant
Queue
Conver
t Queue
SID5
SID6
SID7
8/11/2019 Wait Events in Rac
18/25
18
gc current|cr block 3-way
Instance 1
Instance 2
Instance 3
Session
Requestor
Master
Holder
gc current block 3-way
Wait Events in RAC
Block
SID1
SID2
SID3
GrantQueue
Convert Queue
SID5
SID6
SID7
8/11/2019 Wait Events in Rac
19/25
Grant vs- Get Events
Action Wait Event is
Block is requested by a session Placeholder eventgc cr|current block
request
Buffer lock is requested from the master gc cr|current grant 2-way
Buffer lock is granted, get from the disk db file scattered|sequential read
Buffer is requested from the holder which
is the same as the master
gc cr|current block 2-way
Buffer is requested from the holder which
is not the master
gc cr|current block 3-way
19
1. There is no 3-way grant event, since the request is made to the master
2. There is no 4-way block event; since there will a maximum of 2 hops:
requestormasterholder
Wait Events in RAC
8/11/2019 Wait Events in Rac
20/25
Interpretation
Scenario Interpretation Tuning Implication
The grant waits are very high
compared to block waits
The requested blocks are in
the current instance; but the
master is another instance
Manually remaster the
object?
The 2-way block waits are
very high compared to 3-way
The cache fusion is taking
place as expected, and most
of the blocks are held and
mastered at the same
instance. But most of the
blocks are not found in the
local instance.
Faster interconnect
Shorten the run queue
The 3-way block waits arehigh compared to 2-way
Cache fusion is taking place;but the master and the
holder are usually different.
Manually remaster theobject
Application partitioning
20Wait Events in RAC
8/11/2019 Wait Events in Rac
21/25
gc current/cr block lost
Lost blocks while being transferred to
the remote instance [in theinterconnect]
Cause: may not be the network itself
Cause #1 Network
Network traffic drops the packets
Confirm from the ifconfig -a output If packets are dropping, this couldbe a cause
Why?
Bad network configuration
CPU used for networkprocessing
Cause #2 CPU
LMS process is CPU starved
21Wait Events in RAC
Instance 1
Instance 2
Session
Database
LMSLMS
gc current block requestgc current block 2-way
GRD GES
Block
S
I
D
1S
I
D
2S
I
D
3
G
r
a
n
t
Q
ue
u
e
C
o
n
v
e
r
t
Qu
e
u
eS
I
D
5S
I
D
6S
I
D
7
8/11/2019 Wait Events in Rac
22/25
gc current/cr block busy
What it Means
Session wants a block from the remote instance
Remote instance delays preparing the block to
fulfill the request [CR or current] read
Cause:
Local delay on the remote instance
Most likely: an I/O bottleneck on the remote
instance
block is being accessed by some session which is CPU
starved
LGWR has not written a buffer to redo yet
Less likely: CPU starvation
22Wait Events in RAC
8/11/2019 Wait Events in Rac
23/25
gc current/cr block congested
Meaning
The instance has requested block from remote instance.
Remote instance has prepared the block and shipped it
but it has not reached the requesting instance within 1 ms.
What could be causenetwork bottleneck?
Not necessarily. Causes:
Long run queues, causing the LMS process to be
delayed in processing the incoming block
Heavy paging due to memory deficiency. This causes
the blocks to be paged in before being processed.
23Wait Events in RAC
ll h
8/11/2019 Wait Events in Rac
24/25
Putting it All Together
Every Oracle process is either
Doing some productive work Waiting for some work to be given (idle)
Waiting for some resource
Understand the reason for the wait
Devise a plan accordingly RAC related wait events are manifestations of these
issues mostly
Caused by
Network issues LMS being overloaded
Blocks busy
24Wait Events in RAC
8/11/2019 Wait Events in Rac
25/25
Thank You!Session 362
More Information:
Blog: arup.blogspot.com
100 Things You Probably Didn't Know About OracleDatabase http://bit.ly/evr05e
Twitter: arupnanda
25Wait Events in RAC
http://bit.ly/evr05ehttp://bit.ly/evr05e