Upload
ajaypulavarti
View
305
Download
11
Embed Size (px)
Citation preview
ABC of Teradata System Performance Analysis
Shaheryar [email protected]
GCC Pakistan.Date created: Feb 20, 2009.Last Updated: Oct 12, 2009.
2 > 10 April 2023
Agenda
1) Co-existing systems parallel efficiency
2) CPU Reports1) CPU Utilization2) CPU Node Hours available3) Percent Shift Busy4) OS % of CPU5) CPU Utilization among Vprocs6) Nodes Parallel Efficiency7) AMPs Parallel Efficiency
3) Memory Reports1) Memory availbilty2) Mem allocation Failures
4) Disk I/O Reports1) Disk busy %2) I/O Wait 3) Measuring I/O4) Disk Read Writes5) Full cylinder Reads6) Logical vs. Physical Reads7) Mini Cyl Packs
5) Buddy Backup Effectiveness
6) AWT Usage
7) Host Utilities Report
8) Charting Teradata Manager Reports
3 > 10 April 2023
Feed back
______________________________________________ From: Clark, Dave Sent: Saturday, March 07, 2009 12:14 AMTo: Iqbal, ShaheryarSubject: ABC of Teradata System Performance Analysis
Shaheryar-
I have approved your presentation for general viewing. Thank-you very much for your effort in developing this. The information is very useful.
-dave.clark(858)485-2177
• http://pc01.teradata.com/CKS
4 > 10 April 2023
1) Co-existing systems parallel efficiency
5 > 10 April 2023
Co-existing systems parallel efficiencyCalculation & Reconfiguration
Before Reconfiguration:
Unusable Node Capacity
14.23 – 13.25 = 0.98;
Parallel Efficiency = 93.09
After Reconfiguration:
Unusable Node Capacity
14.23 -14.20 = 0.03;
Parallel Efficiency = 99.80
Gain in Co-exiting systems parallel efficiency is the source of the biggest gain in performance.
6 > 10 April 2023
• 2) CPU Reports
7 > 10 April 2023
CPU Utilization Chart
Generated with Higa Macro ResPMATotal
Higa Macro ResPmaTotal
8 > 10 April 2023
Absolute CPU Node Hours available
•ABS Avail Hours: Time when CPU was available to process a job but there was nothing to process•This graph gives a ratio between total CPU available to Max CPU used on daily basis
Higa Macro ResPmaThreeShifts
9 > 10 April 2023
CPU Utilization: Percent Shift Busy
Shift Time 00-07Mon-Fri
Shift Time 07-17Mon-Fri
Shift Time 17-24Mon-Fri
Shift Time 00-24Sat-Sun
•From previous slide same graph shown in Percent shift busy
Higa Macro ResPmaThreeShifts
10 > 10 April 2023
OS % of CPU Higa Macro ResPmasec
Extremely degraded system performance
•CPU 100 % Busy•OS% of CPU less than 20%•For more than 30 minutes
In-Efficient Use of CPU
11 > 10 April 2023
CPU Utilization among Vprocs Example #1 Higa Macro ResSvprCPUs
A normal breakdown of the total CPU utilization among the AMP, PE and node vprocs
12 > 10 April 2023
CPU Utilization among Vprocs Example #2 Higa Macro ResSvprCPUs
A very high CPU usage by PE. (Non optimized ) TPUMP sessions one of the reasons
PE consuming 40% of Node CPU, works at their max capacity. And have tendency to become bottleneck.
13 > 10 April 2023
Nodes Parallel Efficiency – Example 1 Higa Macro ResPmaTotal
Extreme case of Node skew
•Node Skew = Max Node CPU – Avg Node CPU
14 > 10 April 2023
Nodes Parallel Efficiency – Example 2 Higa Macro ResPmaTotal
Node Skew = Max Node CPU – Avg Node CPU
High Parallel Efficiency among Nodes
15 > 10 April 2023
AMPs Parallel Efficiency – Example 1
High parallel Efficiency is present among AMPs
Higa Macro ResSvprCPUs
16 > 10 April 2023
AMPs Parallel Efficiency – Example 2 Higa Macro ResSvprCPUs
Parallel Efficiency not good among AMPs at times
17 > 10 April 2023
AMPs Parallel Efficiency – Example 3 Higa Macro ResSvprCPUs
Extreme case of skew among AMPs
18 > 10 April 2023
• 3) Memory Reports
19 > 10 April 2023
Memory Utilization
Average & Minimum Free Memory Available
0
200
400
600
800
1 000
1 200
12
/18
Sa
t 0
0:1
0
12
/20
Mo
n 2
3:1
0
12
/23
Th
u 2
1:5
0
12
/26
Su
n 2
0:3
0
12
/29
We
d 1
9:1
0
01
/01
Sa
t 1
7:5
0
01
/04
Tu
e 1
6:3
0
01
/07
Fri
15
:10
01
/10
Mo
n 1
3:5
0
01
/13
Th
u 1
2:3
0
01
/16
Su
n 1
1:1
0
12
/19
Su
n 1
7:2
0
12
/22
We
d 1
6:0
0
12
/25
Sa
t 1
4:4
0
12
/28
Tu
e 1
3:2
0
12
/31
Fri
12
:00
01
/03
Mo
n 1
0:4
0
01
/06
Th
u 0
9:2
0
01
/09
Su
n 0
8:0
0
01
/12
We
d 0
6:4
0
01
/15
Sa
t 0
5:2
0
12
/18
Sa
t 1
1:1
0
12
/21
Tu
e 1
0:1
0
12
/24
Fri
08
:50
12
/27
Mo
n 0
7:3
0
12
/30
Th
u 0
6:1
0
01
/02
Su
n 0
4:5
0
01
/05
We
d 0
3:3
0
01
/08
Sa
t 0
2:1
0
01
/11
Tu
e 0
0:5
0
01
/13
Th
u 2
3:3
0
01
/16
Su
n 2
2:1
0
Mem Free ------
Mem Free ------
Coexistence View
One of the node groupsis experiencing
low memory condition
• Can FSGcache adjustment help?
Higa Macro ResPma
20 > 10 April 2023
Memory Utilization
There are low memory or depletions present, as marked with circles. But there frequency of occurrence is not alarming.
•Available Free memory less than 100 MB and 40 MB are termed as “Memory depletion” and “system panic” states respectively.
Higa Macro ResPma
21 > 10 April 2023
Paging & Memory Allocation Fails
Mem Alloc Fails occurs when FREE MEM reaches to zero.
Negligible Mem Alloc Failures occurred in last month.
Higa Macro ResPma
22 > 10 April 2023
• 4) Disk IO Reports
23 > 10 April 2023
DISK IO: Disk % Busy
System is not IO bound: Disk doesn't remain 100% busy.
Higa Macro ResSldvNode
•Disk Busy time is the amount of time in which there is at least one I/O request outstanding
24 > 10 April 2023
I/O Wait% Example #1
Average Cpu Busy vs. I/O Wait
0
20
40
60
80
100
120
03
/01
We
d 0
0:1
0
03
/01
We
d 0
9:5
0
03
/01
We
d 1
9:3
0
03
/02
Th
u 0
5:1
0
03
/02
Th
u 1
4:5
0
03
/03
Fri
00
:30
03
/03
Fri
10
:10
03
/03
Fri
19
:50
03
/04
Sa
t 0
5:3
0
03
/04
Sa
t 1
5:1
0
03
/05
Su
n 0
0:5
0
03
/05
Su
n 1
0:3
0
03
/05
Su
n 2
0:1
0
03
/06
Mo
n 0
5:5
0
03
/06
Mo
n 1
5:3
0
03
/07
Tu
e 0
1:1
0
03
/07
Tu
e 1
0:5
0
03
/07
Tu
e 2
0:3
0
03
/08
We
d 0
6:1
0
03
/08
We
d 1
5:5
0
03
/09
Th
u 0
1:3
0
03
/09
Th
u 1
1:1
0
03
/09
Th
u 2
0:5
0
03
/10
Fri
06
:30
03
/10
Fri
16
:10
03
/11
Sa
t 0
1:5
0
03
/11
Sa
t 1
1:5
0
03
/11
Sa
t 2
1:3
0
03
/12
Su
n 0
7:1
0
03
/12
Su
n 1
6:5
0
03
/13
Mo
n 0
2:3
0
Chart Type: Stacked Area
Wait % ----
CPU bsy ---
• Average CPU + I/O Wait = System Busy• If Avg CPU + I/O Wait = 100%, desirable ratio is >= 90% avg CPU to >= 10% I/O Wait
Ratio is well within target range
Higa Macro ResPmaTotal
25 > 10 April 2023
I/O Wait% Example #2
Average Cpu Busy vs. I/O Wait
0
20
40
60
80
100
120
1/9
Fri
0:00
1/9
Fri
8:50
1/9
Fri
17:4
0
1/10
Sat
2:3
0
1/10
Sat
11:
20
1/10
Sat
20:
10
1/11
Sun
6:2
0
1/11
Sun
15:
10
1/12
Mon
0:0
0
1/12
Mon
8:5
0
1/12
Mon
17:
40
1/13
Tue
2:3
0
1/13
Tue
11:
20
1/13
Tue
20:
10
1/14
Wed
5:0
0
1/14
Wed
13:
50
1/14
Wed
22:
40
1/15
Thu
7:3
0
1/15
Thu
16:
20
1/16
Fri
1:10
1/16
Fri
10:0
0
1/16
Fri
18:5
0
1/17
Sat
3:4
0
1/17
Sat
12:
30
1/17
Sat
21:
20
1/18
Sun
6:1
0
1/18
Sun
15:
00
1/19
Mon
0:1
0
1/19
Mon
12:
30
1/19
Mon
21:
30
1/20
Tue
6:2
0
1/20
Tue
15:
10
1/21
Wed
0:0
0
1/21
Wed
8:5
0
1/21
Wed
17:
40
1/22
Thu
2:3
0
1/22
Thu
11:
20
1/22
Thu
20:
10
1/23
Fri
5:00
1/23
Fri
13:5
0
Chart Type: Stacked Area
I/O Wait %
Avg CPU bsy
• Wait I/O should be 10% or less for configuration optimally balanced for power and throughput
#1 40%-60% Wait I/O shown here
Higa Macro ResPmaTotal
26 > 10 April 2023
Disk Read KByte and Write KByte
0
50 000
100 000
150 000
200 000
250 000
300 00006
/15
Thu
06:
20
06/1
5 T
hu 0
6:20
06/1
5 T
hu 0
6:20
06/1
5 T
hu 0
6:20
06/1
5 T
hu 0
6:20
06/1
5 T
hu 0
6:30
06/1
5 T
hu 0
6:30
06/1
5 T
hu 0
6:30
06/1
5 T
hu 0
6:30
06/1
5 T
hu 0
6:30
06/1
5 T
hu 0
6:40
06/1
5 T
hu 0
6:40
06/1
5 T
hu 0
6:40
06/1
5 T
hu 0
6:40
06/1
5 T
hu 0
6:40
06/1
5 T
hu 0
6:50
06/1
5 T
hu 0
6:50
06/1
5 T
hu 0
6:50
06/1
5 T
hu 0
6:50
06/1
5 T
hu 0
6:50
Chart Type: Stacked Area
Disk WrKB /Sec
Disk RdKB /Sec
Measuring I/O
On the system, differences in throughputwere viewable with disk read/writes Actual rated bandwidth for the configuration
Higa Macro ResPmabyNode
27 > 10 April 2023
Disk Position Reads, Pre-Reads and Writes
0
2 000 000
4 000 000
6 000 000
8 000 000
10 000 000
12 000 000
09/0
3 S
at 00:0
0
09/0
4 S
un 0
1:1
0
09/0
5 M
on 0
2:2
0
09/0
6 T
ue 0
3:3
0
09/0
7 W
ed 0
4:4
0
09/0
8 T
hu 0
5:5
0
09/0
9 F
ri 0
7:0
0
09/1
0 S
at 08:1
0
09/1
1 S
un 0
9:2
0
09/1
2 M
on 1
0:3
0
09/1
3 T
ue 1
3:0
0
09/1
4 W
ed 1
4:1
0
09/1
5 T
hu 1
5:2
0
09/1
6 F
ri 1
6:5
0
09/1
7 S
at 18:0
0
09/1
8 S
un 1
9:1
0
09/1
9 M
on 2
0:2
0
09/2
0 T
ue 2
1:3
0
09/2
1 W
ed 2
3:3
0
09/2
3 F
ri 0
0:4
0
09/2
4 S
at 01:5
0
09/2
5 S
un 0
3:0
0
09/2
6 M
on 0
4:1
0
09/2
7 T
ue 0
5:2
0
09/2
8 W
ed 0
6:3
0
09/2
9 T
hu 0
7:4
0
09/3
0 F
ri 0
8:5
0
10/0
1 S
at 10:0
0
10/0
2 S
un 1
1:1
0
Chart Type: Stacked Area
Total DB Wrts
Total Pre Rds
Total Position Rds
Total Disk Reads/Writes
Total Pre-Reads are large proportion of total reads, so this system may be a good candidate for raising
Full Cylinder Read slots
Next step is to look at FCR denied cache. If this is high, and there is wait I/O often,
then this system is a candidate for higher FCR slots.
Higa Macro ResPmaTotal
28 > 10 April 2023
Full Cylinder Reads Example #1 Higa Macro ResFullCylReadTotal
Less FCR requests with high
successful rate
29 > 10 April 2023
Full Cylinder Reads Example #2 Higa Macro ResFullCylReadTotal
High FCR requests with moderate successful rate
30 > 10 April 2023
Logical vs. Physical Reads Higa Macro SvprReadTotal
The more that the logical reads exceeds physical reads, the better use of memory as a cache
31 > 10 April 2023
Mini-cyl-packs Higa Macro ResCylPackTotal
Mini Cyl Pack occurs when there are only ten Free cylinders left on any AMP resulting in degraded system performance.
32 > 10 April 2023
5) AWT Usage Higa Macro ResSvprQLenAvg
ByVproc
Potential indicator of “FLOW Control” state
Message queue length > 20
33 > 10 April 2023
6) Buddy Backup Effectiveness Higa Macro ResPmaBkupHour
Total
When there are many complete and few partial blocks, sent to the buddy, then the buddy backup should be turned off
34 > 10 April 2023
7) Host Utilities Traffic Higa Macro reshostTotalHour
Total read/write traffic per Hour, physical MBytes transferred between Host and Node.
how much data is read or written by utility
35 > 10 April 2023
• 8) Charting Teradata Manager Reports
Use Teradata Manager to record
Active Session detail data
Generate Custom reports using pivot
tables
36 > 10 April 2023
Charting Teradata Manager Reports:CPU utilized per User
CPU utilized per User
37 > 10 April 2023
Charting Teradata Manager Reports:CPU utilization Per Group
CPU utilized per Performance Group
38 > 10 April 2023
Questions
The only bad questionis the questionnever asked