Upload
avis-page
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
Page 1© 2001 Hewlett-Packard Company
Tools for Measuring System
and
Application Performance • Introduction
• GlancePlus• Introduction
• Glance Motif
• Glance Character Mode
• Glance Command-line
• Xverbosegc
• HPjmeter and –Xeprof
• Other Tools
• gdb
Page 2© 2001 Hewlett-Packard Company
GlancePlus Motif
/opt/perf/bin/gpm
Page 3
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus Motif
• Open multiple windows to simultaneously view:
• Machine’s behavior
• Process’s behavior
• Thread behavior
• We will learn how to use it with Java applications
Page 4
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus Motif Main Window – Overview of System
Page 5
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifMain Window – Overview of System
CPU
I/O
Memory
Network
Page 6
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifConfigure Sampling Time
Page 7
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifConfigure Sampling Time
Configure:Measurement
Sample Interval:Minutes: 0Seconds: 1
Graph Points: 50
Page 8
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifMain Window – CPU Usage
Active Button: CPU Graph
Page 9
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifCPU Graph
Page 10
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifCPU Graph
CPU Queue Length
8 CPUs
CPU Usage Categories:• Real• Negative Nice (-20 to 0)• Nice (greater than 0)• Normal User• System
Page 11
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifCPU Graph – Garbage Collection
1 Thread Active during GC
Page 12
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifMain Window - Reports
Page 13
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifMain Window - Reports:Process List
Reports:Process List
Page 14
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess List
Page 15
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess List
8 CPU System: 800% Maximum CPU%
Page 16
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess List – Arrange Columns
Page 17
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess List – Arrange Columns
Configure:Arrange Columns
Page 18
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess List – Arrange Columns
Page 19
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess List – Arrange Columns
Interesting values:• CPU%• User CPU%• System CPU%• Virtual Memory• Res(ident) Memory• Physical I/O• Configure:Choose Metrics:• PROC_THREAD_COUNT
Page 20
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess List - Reports
Page 21
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess List - Reports
Process ResourcesProcess Open FilesProcess Memory RegionsProcess System CallsProcess Thread List
Page 22
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess System Calls
Page 23
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess System Calls
Sorted on: SysCall Rate
Page 24
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess System Calls - Sort
Configure:Sort Fields
Page 25
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess System Calls - Sort
Page 26
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess System Calls - Sort
Page 27
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess System Calls
System Calls for Idling JVM
Page 28
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess List - Reports
Process ResourcesProcess Open FilesProcess Memory RegionsProcess System CallsProcess Thread List
Page 29
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Memory Regions
Page 30
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Memory Regions
VSS – Virtual Set Size
Total Reserved Space
RSS – Resident Set Size
Space Actively in Use
Total Number of Memory Regions
Page 31
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Memory Regions
Data == C Heap
Text == Executable
Other == Java Heap Shared Libraries Thread Stacks
Stack == For 1st Thread
Page 32
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Memory Regions - Sort
Page 33
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Memory Regions - Sort
Page 34
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Memory Regions - Sort
Sort on VSS
Page 35
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Memory Regions - Metrics
Add Page Size Metrics
Page 36
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Memory Regions - Metrics
Page 37
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Memory Regions - Metrics
Select All
PROC_REGION_PAGE_COUNT*
Page 38
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Memory Regions - Metrics
For Address: Select
PROC_REGION_VIRT_ADDRS
Page 39
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Memory Regions – Pages
Page Sizes AllocatedIn Java Heap
All: 4 KB
Too ManyTotal!!
Page 40
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Memory Regions – Pages
Page Sizes AllocatedIn Java Heap
All: 4 MB
Very FewTotal!!
Page 41
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess List - Reports
Process ResourcesProcess Open FilesProcess Memory RegionsProcess System CallsProcess Thread List
Page 42
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Thread List
Page 43
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Thread List
Active Threads
JVM Threads
Total Number of Threads
Thread Id
Page 44
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess List - Reports
Process ResourcesProcess Open FilesProcess Memory RegionsProcess System CallsProcess Thread List
Page 45
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Open Files
Page 46
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifProcess Open Files
File Type
Sockets – Monitor Bytes Read and Written!
File Name
File Offset
Total Number of Open Files
Page 47
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifMain Window - Reports
System Info: System Tables Graph
Page 48
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifSystem Tables Graph
Process TableFile TableShared Memory TableSemaphore TableFile LocksSwap Space
Page 49
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifIdentifying Performance Problems
• Patterns to recognize• High System CPU time
– Often associated with monitor contention
– Check by looking at System Calls– High sched_yield, ksleep, kwakeup call rate confirms
• Increasing memory usage– Thread stacks
– C Heap
– Percentage of the Java Heap in active use
Page 50
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
HotSpot JVMObject Model
• Efficient and low overhead
header
non-staticfields
methodtable ptr
staticfields
GC maps
header
C++ vtable
Class
Object
Page 51
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
HotSpot JVMObject Header
• Age (7 bits)
• Hash (23 bits)
• Lock (2 bits)• lock
• unlock
• pthread mutex used (inflated)
• used by mark/sweep
LockHashAge
Page 52
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
HotSpot JVMEfficient, Low-Overhead Locking
Stack
header
non-staticfields Object
ULheaderL
If another thread tries to obtain the same
monitor: Lock inflation (uses a pthread mutex)
Page 53
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
M onitor/M utex
thread 1 thread 3
M onitor/M utex
thread 1 thread 2thread 3
M utexLock
M onitor/M utex
thread 1 thread 3
M onitor/M utex
thread 1thread 3
M utexLock
M onitor/M utex
thread 1
How Does a pthread mutex Work?
• Try to get lock immediately
• If fail, be optimistic:– Enter queue and wait
– Repeat: spin, sched_yield
– Next waiter given lock
• Until: not optimistic– ksleep
Has 2 WaitersHas 2 Waiters
Page 54
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
Contention: System Impact
• Monitors • Implemented using a single word in memory that serves as
the “lock word”
• On a system with multiple CPUs:• All of the caches on the individual CPUs must be updated
during the spin-sched_yield() cycle
• Consequences:• Threads time is spent accessing one word – spinning
– Result: High CPU usage with little application work
• Machine spends all of its time maintaining cache coherency!!
• More threads yield lower performance!
Page 55
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
M onitor/M utex
thread 1 thread 3
M onitor/M utex
thread 1 thread 2thread 3
M utexLock
M onitor/M utex
thread 1 thread 3
M onitor/M utex
thread 1thread 3
M utexLock
M onitor/M utex
thread 1
Contention: System Impact
CPU 1
Cache
CPU 2
Cache
CPU 3
Cache
CPU 4
Cache
Memory
thread 1
thread 3
thread 4
thread 5
thread 7
thread 6
thread 8
Cache Coherency
Cache CoherencyLDCW
Page 56
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifExample of High Contention
Page 57
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifExample of High Contention
HIGH SYSTEM CPU TIME
LOW USER CPU TIME
Page 58
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifExample of High Contention
Page 59
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifExample of High Contention
HIGH sched_yield() ksleep() kwakeup()CALL RATES
Page 60
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifExample of High Contention
User
System
Page 61
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifExample of High Contention - Fixed
Page 62
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifExample of High Contention
Page 63
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifExample of High Contention - Fixed
2 Fold Improvement!
5 Fold Improvement!
100x Reduction!
Page 64
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifMonitoring Memory Usage
• Select process ID for process in Process window and select Report:Memory Regions
• Data VSS (Virtual Set Size - allocated) represents the C heap • Text VSS represents memory for the executable• Other VSS represents the memory used by shared libraries,
thread stacks, Java Heap, Code cache, …• Private VSS (part of Other) contains the Java Heap and
Thread stacks
• Corresponding RSS (Resident Set Size) for each area can also be monitored
• Watch for significant growth in any of these regions• Use the Glance adviser syntax for long term
monitoring
Page 65
Tools for Measuring
Performance
© 2001 Hewlett-Packard Company
GlancePlus MotifConclusions
• Motif version is a powerful tool
• Exposes problems with Java performance
• Easy to do systematic analysis using displayed information