Diagnostics Guide from IBM

7/29/2019 Diagnostics Guide from IBM

1/487

IBM Developer Kit and Runtime Environment, Java Technology Edition, Version 5.0

Diagnostics Guide

Version 5 Release 0


2/487


3/487

IBM Developer Kit and Runtime Environment, Java Technology Edition, Version 5.0

Diagnostics Guide

Version 5 Release 0


4/487

NoteBefore using this information and the product it supports, read the information in Notices on page 451.

Eleventh Edition (June 2009)

This edition applies to all the platforms that are included in the IBM Developer Kit and Runtime Environment, Java2 Technology Edition, Version 5.0 and to all subsequent releases and modifications until otherwise indicated in neweditions. Technical changes made for all the editions of this book are indicated by vertical bars to the left of thechanges.

Copyright International Business Machines Corporation 2003, 2009.US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.


5/487

Contents

F i g u r e s . . . . . . . . . . . . . . . ix

Tables . . . . . . . . . . . . . . . xi

About the Diagnostics Guide . . . . . xiiiWhat does the Java Virtual Machine (JVM) mean? xiiiWho should read this book . . . . . . . . . xivUsing this book. . . . . . . . . . . . . xivOther sources of information . . . . . . . . xivReporting problems in the JVM . . . . . . . xvConventions and terminology . . . . . . . . xvHow to send your comments . . . . . . . . xvContributors . . . . . . . . . . . . . . xviSummary of changes . . . . . . . . . . . xvi

Part 1. Understanding the IBM SDKfor Java . . . . . . . . . . . . . . 1

Chapter 1. The building blocks of theIBM Virtual Machine for Java. . . . . . 3

Java application stack . . . . . . . . . . . 4Components of the IBM Virtual Machine for Java . . 4

JVM API . . . . . . . . . . . . . . . 5Diagnostics component . . . . . . . . . . 5Memory management . . . . . . . . . . 5Class loader . . . . . . . . . . . . . 6Interpreter . . . . . . . . . . . . . . 6Platform port layer . . . . . . . . . . . 6

Chapter 2. Memory management . . . . 7Overview of memory management . . . . . . . 7

Object allocation . . . . . . . . . . . . 7Reachable objects . . . . . . . . . . . . 8Garbage collection . . . . . . . . . . . 8Heap sizing problems . . . . . . . . . . 8

Allocation . . . . . . . . . . . . . . . 9Heap lock allocation . . . . . . . . . . . 9Cache allocation . . . . . . . . . . . . 9Large Object Area . . . . . . . . . . . 10

Detailed description of garbage collection . . . . 11Mark phase . . . . . . . . . . . . . 11Sweep phase . . . . . . . . . . . . . 13

Compaction phase . . . . . . . . . . . 15Subpool (AIX, Linux PPC and zSeries, z/OS andi5/OS only) . . . . . . . . . . . . . 15Reference objects . . . . . . . . . . . 16Final reference processing . . . . . . . . 17

JNI weak reference . . . . . . . . . . . 17Heap expansion . . . . . . . . . . . . 17Heap shrinkage . . . . . . . . . . . . 18

Generational Concurrent Garbage Collector. . . . 19Tenure age. . . . . . . . . . . . . . 20Tilt ratio . . . . . . . . . . . . . . 20

How to do heap sizing . . . . . . . . . . 20Initial and maximum heap sizes . . . . . . 20

Using verbose:gc . . . . . . . . . . . 21Using fine tuning options. . . . . . . . . 21Interaction of the Garbage Collector withapplications . . . . . . . . . . . . . . 22How to coexist with the Garbage Collector . . . . 22

Root set . . . . . . . . . . . . . . 23Thread local heap . . . . . . . . . . . 23Bug reports . . . . . . . . . . . . . 23Finalizers . . . . . . . . . . . . . . 23Manually starting the Garbage Collector. . . . 25

Frequently asked questions about the GarbageCollector . . . . . . . . . . . . . . . 25

Chapter 3. Class loading . . . . . . . 29

The parent-delegation model . . . . . . . . 29Namespaces and the runtime package . . . . . 30Custom class loaders . . . . . . . . . . . 31

Chapter 4. Class data sharing . . . . . 33

Chapter 5. The JIT compiler. . . . . . 35JIT compiler overview . . . . . . . . . . . 35How the JIT compiler optimizes code. . . . . . 36

Phase 1 - inlining . . . . . . . . . . . 36Phase 2 - local optimizations . . . . . . . 36Phase 3 - control flow optimizations . . . . . 37Phase 4 - global optimizations . . . . . . . 37

Phase 5 - native code generation . . . . . . 37Frequently asked questions about the JIT compliler 37

Chapter 6. Java Remote MethodInvocation . . . . . . . . . . . . . 39The RMI implementation . . . . . . . . . . 39Thread pooling for RMI connection handlers . . . 40Understanding distributed garbage collection (DGC) 40Debugging applications involving RMI . . . . . 41

Chapter 7. The ORB . . . . . . . . . 43CORBA. . . . . . . . . . . . . . . . 43RMI and RMI-IIOP . . . . . . . . . . . . 43

Java IDL or RMI-IIOP? . . . . . . . . . . 44RMI-IIOP limitations . . . . . . . . . . . 44Further reading . . . . . . . . . . . . . 44Examples of clientserver applications . . . . . 45

Interfaces . . . . . . . . . . . . . . 45Remote object implementation (or servant) . . . 45Stubs and ties generation . . . . . . . . . 46Server code . . . . . . . . . . . . . 46Summary of major differences between RMI(JRMP) and RMI-IIOP . . . . . . . . . . 49

Using the ORB . . . . . . . . . . . . . 50How the ORB works . . . . . . . . . . . 54

Copyright IBM Corp. 2003, 2009 iii


6/487

The client side . . . . . . . . . . . . 54The server side . . . . . . . . . . . . 58

Additional features of the ORB . . . . . . . . 60Portable object adapter . . . . . . . . . 60Fragmentation . . . . . . . . . . . . 62Portable interceptors . . . . . . . . . . 62Interoperable Naming Service (INS) . . . . . 65

Chapter 8. The Java Native Interface(JNI) . . . . . . . . . . . . . . . . 67Overview of JNI . . . . . . . . . . . . . 67The JNI and the Garbage Collector . . . . . . 68

Overview of JNI object references . . . . . . 68JNI transitions . . . . . . . . . . . . 71

Copying and pinning . . . . . . . . . . . 72Using the isCopy flag . . . . . . . . . . 73Using the mode flag . . . . . . . . . . 73A generic way to use the isCopy and mode flags 74

Handling exceptions . . . . . . . . . . . 74Synchronization . . . . . . . . . . . . . 74Debugging the JNI . . . . . . . . . . . . 75

JNI checklist . . . . . . . . . . . . . . 77

Part 2. Submitting problem reports 79

Chapter 9. Overview of problemsubmission . . . . . . . . . . . . . 81How does the IBM service team service Java? . . . 81Submitting Java problem reports to IBM service . . 81Problem severity ratings . . . . . . . . . . 82Escalating problem severity . . . . . . . . . 82

Java duty manager . . . . . . . . . . . . 83

Chapter 10. MustGather: collecting the

correct data to solve problems . . . . 85Before you submit a problem report . . . . . . 85Data to include . . . . . . . . . . . . . 85Test cases . . . . . . . . . . . . . . . 86Performance problems . . . . . . . . . . . 86

Factors that affect JVM performance . . . . . 87Questions to ask about performance problems . . 87

Chapter 11. Submitting data with aproblem report . . . . . . . . . . . 89Sending files to IBM service . . . . . . . . . 89Obtaining files from IBM service . . . . . . . 90Compressing files . . . . . . . . . . . . 91

How you receive your fix . . . . . . . . . 91

Part 3. Problem determination . . . 93

Chapter 12. First steps in problemdetermination. . . . . . . . . . . . 95

Chapter 13. AIX problem determination 97Setting up and checking your AIX environment . . 97

Enabling full AIX core files . . . . . . . . 98

General debugging techniques . . . . . . . . 99AIX debugging commands . . . . . . . . 100DBX Plug-in . . . . . . . . . . . . . 109

Diagnosing crashes . . . . . . . . . . . 110Documents to gather . . . . . . . . . . 110Locating the point of failure . . . . . . . 111

Debugging hangs . . . . . . . . . . . . 112AIX deadlocks . . . . . . . . . . . . 112AIX busy hangs . . . . . . . . . . . 112Poor performance on AIX . . . . . . . . 114

Understanding memory usage . . . . . . . . 11532- and 64-bit JVMs . . . . . . . . . . 115The 32-bit AIX Virtual Memory Model . . . . 115The 64-bit AIX Virtual Memory Model . . . . 116Changing the Memory Model (32-bit JVM) . . 117The native and Java heaps . . . . . . . . 117The AIX 32-bit JVM default memory models . . 118Monitoring the native heap. . . . . . . . 118Native heap usage . . . . . . . . . . . 119Specifying MALLOCTYPE . . . . . . . . 120Monitoring the Java heap . . . . . . . . 120Receiving OutOfMemoryError exceptions . . . 120Is the Java or native heap exhausted? . . . . 121

Java heap exhaustion . . . . . . . . . . 121Native heap exhaustion . . . . . . . . . 121AIX fragmentation problems . . . . . . . 122Submitting a bug report . . . . . . . . . 123

Debugging performance problems . . . . . . 123Finding the bottleneck . . . . . . . . . 123CPU bottlenecks . . . . . . . . . . . 124Memory bottlenecks . . . . . . . . . . 128I/O bottlenecks. . . . . . . . . . . . 128

JVM heap sizing . . . . . . . . . . . 129JIT compilation and performance . . . . . . 129Application profiling . . . . . . . . . . 129

MustGather information for AIX . . . . . . . 129

Chapter 14. Linux problemdetermination . . . . . . . . . . . 131Setting up and checking your Linux environment 131General debugging techniques. . . . . . . . 133

Using the dump extractor . . . . . . . . 133Using system dump tools . . . . . . . . 134Examining process information . . . . . . 134ldd . . . . . . . . . . . . . . . . 136Tracing tools . . . . . . . . . . . . 136Debugging with gdb . . . . . . . . . . 137

Diagnosing crashes . . . . . . . . . . . 139Debugging hangs . . . . . . . . . . . . 140

Debugging memory leaks . . . . . . . . . 141Debugging performance problems . . . . . . 141

Finding the bottleneck . . . . . . . . . 141CPU usage . . . . . . . . . . . . . 142Memory usage . . . . . . . . . . . . 142Network problems . . . . . . . . . . 142


MustGather information for Linux . . . . . . 144Known limitations on Linux . . . . . . . . 146

iv IBM SDK for Java: Diagnostics Guide


7/487

Chapter 15. Windows problemdetermination . . . . . . . . . . . 149Setting up and checking your Windowsenvironment. . . . . . . . . . . . . . 149

Windows 32-bit large address aware support 150General debugging techniques. . . . . . . . 151

Using the Cross-Platform Dump Viewer . . . 151

System dump . . . . . . . . . . . . 151Diagnosing crashes in Windows . . . . . . . 152

Data to send to IBM . . . . . . . . . . 152Debugging hangs . . . . . . . . . . . . 153

Getting a dump from a hung JVM . . . . . 153Analyzing deadlocks . . . . . . . . . . 153

Debugging memory leaks . . . . . . . . . 153The Windows memory model . . . . . . . 154Classifying leaks . . . . . . . . . . . 154Tracing leaks . . . . . . . . . . . . 154Using Heapdump to debug memory leaks. . . 155

OutOfMemoryError creating a thread . . . . . 155Debugging performance problems . . . . . . 155

Finding the bottleneck . . . . . . . . . 156

Windows systems resource usage. . . . . . 156JVM heap sizing . . . . . . . . . . . 156JIT compilation and performance . . . . . . 156Application profiling . . . . . . . . . . 156

MustGather information for Windows . . . . . 157

Chapter 16. z/OS problemdetermination . . . . . . . . . . . 159Setting up and checking your z/OS environment 159

Maintenance. . . . . . . . . . . . . 159LE settings . . . . . . . . . . . . . 159Environment variables . . . . . . . . . 159Private storage usage . . . . . . . . . . 159Setting up dumps . . . . . . . . . . . 160

General debugging techniques. . . . . . . . 161Using IPCS commands . . . . . . . . . 161Using dbx . . . . . . . . . . . . . 162Interpreting error message IDs . . . . . . 162

Diagnosing crashes . . . . . . . . . . . 163Documents to gather . . . . . . . . . . 163Determining the failing function . . . . . . 164Working with TDUMPs using IPCS . . . . . 166

Debugging hangs . . . . . . . . . . . . 170The process is deadlocked . . . . . . . . 170The process is looping . . . . . . . . . 170The process is performing badly . . . . . . 171

Understanding Memory Usage . . . . . . . 171Allocations to LE HEAP . . . . . . . . . 171z/OS virtual storage . . . . . . . . . . 172OutOfMemoryError exceptions . . . . . . 172

Debugging performance problems . . . . . . 173Finding the bottleneck . . . . . . . . . 174z/OS systems resource usage . . . . . . . 174


MustGather information for z/OS . . . . . . 175

Chapter 17. i5/OS problemdetermination . . . . . . . . . . . 177Determining which VM is in use . . . . . . . 177Setting up your IBM Technology for JavaEnvironment . . . . . . . . . . . . . 177

Required Software and Licensing. . . . . . 177Configuring JAVA_HOME . . . . . . . . 178

Enabling i5/OS PASE core files . . . . . . 179Setting environment variables for i5/OS PASEor QShell . . . . . . . . . . . . . . 180Determining a users home directory . . . . 181Setting default Java command-line options . . 181

General debugging techniques. . . . . . . . 183Diagnosing problems at the command line . . . 183i5/OS debugging commands . . . . . . . . 184

Work with Active Jobs (WRKACTJOB) . . . . 184Work with Job (WRKJOB) . . . . . . . . 184Work with System Status (WRKSYSSTS) . . . 184Work with Disk Status (WRKDSKSTS) . . . . 184Process Status (ps). . . . . . . . . . . 184Debugger (dbx). . . . . . . . . . . . 185

Debugging performance problems . . . . . . 185Analyzing CPU bottlenecks. . . . . . . . 185Analyzing memory problems . . . . . . . 187Analyzing I/O problems . . . . . . . . 188

Diagnosing crashes . . . . . . . . . . . 192Checking the system environment . . . . . 192Finding out about the Java environment . . . 192Detailed crash diagnosis . . . . . . . . . 192

Diagnosing hangs . . . . . . . . . . . . 192i5/OS deadlocks . . . . . . . . . . . 192i5/OS busy hangs . . . . . . . . . . . 192

Understanding memory usage . . . . . . . . 193The 32-bit i5/OS PASE Virtual memory model 193The process and garbage-collected heaps . . . 194

Monitoring the garbage-collected heap . . . . 194Process heap usage . . . . . . . . . . 194OutOfMemoryError exceptions . . . . . . 195Garbage-collected heap exhaustion . . . . . 195Submitting a bug report . . . . . . . . . 195

Using dbx . . . . . . . . . . . . . . 196Using the DBX Plug-in for Java . . . . . . 196Important dbx usage notes and warnings . . . 197Using dbx to investigate a Java system dump 197Starting dbx on a system dump(core.{date}.{time}.{pid}.dmp) . . . . . . . 198

Chapter 18. Sun Solaris problemdetermination . . . . . . . . . . . 199

Chapter 19. Hewlett-Packard SDKproblem determination . . . . . . . 201

Chapter 20. ORB problemdetermination . . . . . . . . . . . 203Identifying an ORB problem . . . . . . . . 203Debug properties . . . . . . . . . . . . 204ORB exceptions . . . . . . . . . . . . 205Completion status and minor codes . . . . . . 206

Contents v


8/487

Java security permissions for the ORB . . . . . 207Interpreting the stack trace . . . . . . . . . 208

Description string . . . . . . . . . . . 209Interpreting ORB traces . . . . . . . . . . 209

Message trace . . . . . . . . . . . . 209Comm traces . . . . . . . . . . . . 210Client or server . . . . . . . . . . . . 211Service contexts . . . . . . . . . . . 212

Common problems . . . . . . . . . . . 212ORB application hangs . . . . . . . . . 212Running the client without the server running

before the client is started . . . . . . . . 213Client and server are running, but not namingservice. . . . . . . . . . . . . . . 214Running the client with MACHINE2 (client)unplugged from the network . . . . . . . 214

IBM ORB service: collecting data . . . . . . . 215Preliminary tests . . . . . . . . . . . 215

Chapter 21. NLS problemdetermination . . . . . . . . . . . 217

Overview of fonts . . . . . . . . . . . . 217Font utilities. . . . . . . . . . . . . . 218Common NLS problem and possible causes . . . 218

Part 4. Using diagnostic tools . . . 221

Chapter 22. Overview of the availablediagnostics . . . . . . . . . . . . 223Categorizing the problem . . . . . . . . . 223Summary of diagnostic information . . . . . . 223Summary of cross-platform tooling . . . . . . 225

Heapdump analysis tooling . . . . . . . 225Cross-platform dump viewer . . . . . . . 225

JVMTI tools . . . . . . . . . . . . . 225JVMPI tools . . . . . . . . . . . . . 226JPDA tools . . . . . . . . . . . . . 226DTFJ . . . . . . . . . . . . . . . 226Trace formatting . . . . . . . . . . . 226

JVMRI. . . . . . . . . . . . . . . 227

Chapter 23. Using dump agents . . . 229Using the -Xdump option . . . . . . . . . 229

Merging -Xdump agents . . . . . . . . . 231Dump agents . . . . . . . . . . . . . 233

Console dumps. . . . . . . . . . . . 234System dumps . . . . . . . . . . . . 234Stack dumps . . . . . . . . . . . . 235LE CEEDUMPs. . . . . . . . . . . . 235Tool option . . . . . . . . . . . . . 236

Javadumps . . . . . . . . . . . . . 237Heapdumps . . . . . . . . . . . . . 237Snap traces . . . . . . . . . . . . . 238

Dump events . . . . . . . . . . . . . 238Advanced control of dump agents . . . . . . 239

exec option . . . . . . . . . . . . . 239file option . . . . . . . . . . . . . 240filter option . . . . . . . . . . . . . 240opts option . . . . . . . . . . . . . 241

Priority option . . . . . . . . . . . . 241range option. . . . . . . . . . . . . 242request option . . . . . . . . . . . . 242defaults option . . . . . . . . . . . . 243

Dump agent tokens . . . . . . . . . . . 243Default dump agents . . . . . . . . . . . 244Removing dump agents . . . . . . . . . . 245Dump agent environment variables . . . . . . 245Signal mappings . . . . . . . . . . . . 246Windows, Linux, AIX, and i5/OS specifics . . . 247z/OS specifics . . . . . . . . . . . . . 247

Chapter 24. Using Javadump. . . . . 251Enabling a Javadump. . . . . . . . . . . 251Triggering a Javadump . . . . . . . . . . 251Interpreting a Javadump . . . . . . . . . 252

Javadump tags . . . . . . . . . . . . 253TITLE, GPINFO, and ENVINFO sections . . . 254Storage Management (MEMINFO) . . . . . 256Locks, monitors, and deadlocks (LOCKS) . . . 257Threads and stack trace (THREADS) . . . . 258

Classloaders and Classes (CLASSES) . . . . 259Environment variables and Javadump . . . . . 260

Chapter 25. Using Heapdump . . . . 263Information for users of previous releases ofHeapdump . . . . . . . . . . . . . . 263Getting Heapdumps . . . . . . . . . . . 263

Enabling text formatted (classic) Heapdumps 264Available tools for processing Heapdumps . . . 264Using -Xverbose:gc to obtain heap information . . 264Environment variables and Heapdump. . . . . 264Text (classic) Heapdump file format . . . . . . 265

Chapter 26. Using system dumps andthe dump viewer . . . . . . . . . . 269Overview of system dumps . . . . . . . . 269System dump defaults . . . . . . . . . . 270Overview of the dump viewer. . . . . . . . 270

Using the dump extractor, jextract . . . . . 270Using the dump viewer, jdmpview . . . . . 272

Dump viewer commands . . . . . . . . . 273General commands . . . . . . . . . . 273Working with locks . . . . . . . . . . 275Showing details of a dump . . . . . . . . 275Analysing the memory . . . . . . . . . 275Working with classes . . . . . . . . . . 276Working with objects . . . . . . . . . . 277

Generating Heapdumps . . . . . . . . . 277Working with trace . . . . . . . . . . 278Example session . . . . . . . . . . . . 279

Chapter 27. Tracing Java applicationsand the JVM . . . . . . . . . . . . 289What can be traced? . . . . . . . . . . . 289Types of tracepoint . . . . . . . . . . . 290Default tracing . . . . . . . . . . . . . 290Where does the data go? . . . . . . . . . 291

Writing trace data to memory buffers . . . . 292Writing trace data to a file . . . . . . . . 292

vi IBM SDK for Java: Diagnostics Guide

||

||


9/487

External tracing . . . . . . . . . . . 293Tracing to stderr . . . . . . . . . . . 293Trace combinations . . . . . . . . . . 293

Controlling the trace . . . . . . . . . . . 293Specifying trace options . . . . . . . . . 294Detailed descriptions of trace options . . . . 294Using the Java API . . . . . . . . . . 308

Using the trace formatter . . . . . . . . . 309Determining the tracepoint ID of a tracepoint . . 310Application trace . . . . . . . . . . . . 311

Implementing application trace . . . . . . 311Using application trace at runtime . . . . . 313

Using method trace . . . . . . . . . . . 314Running with method trace . . . . . . . 314Untraceable methods . . . . . . . . . . 315Examples of use . . . . . . . . . . . 317Example of method trace . . . . . . . . 318

Chapter 28. JIT problem determination 321Diagnosing a JIT problem . . . . . . . . . 321

Disabling the JIT compiler . . . . . . . . 321

Selectively disabling the JIT compiler . . . . 322Locating the failing method . . . . . . . 323Identifying JIT compilation failures . . . . . 324

Performance of short-running applications . . . 325JVM behavior during idle periods . . . . . . 325

Chapter 29. Garbage Collectordiagnostics . . . . . . . . . . . . 327How do the garbage collectors work? . . . . . 327Common causes of perceived leaks . . . . . . 327

Listeners . . . . . . . . . . . . . . 327Hash tables . . . . . . . . . . . . . 328Static class data . . . . . . . . . . . 328

JNI references . . . . . . . . . . . . 328

Objects with finalizers . . . . . . . . . 328-verbose:gc logging . . . . . . . . . . . 328

Global collections . . . . . . . . . . . 329Garbage collection triggered by System.gc() . . 330Allocation failures . . . . . . . . . . . 331Scavenger collections . . . . . . . . . . 332Concurrent garbage collection . . . . . . . 333Timing problems during garbage collection . . 337

-Xtgc tracing . . . . . . . . . . . . . 338-Xtgc:backtrace . . . . . . . . . . . . 338-Xtgc:compaction . . . . . . . . . . . 339-Xtgc:concurrent . . . . . . . . . . . 339-Xtgc:dump . . . . . . . . . . . . . 339-Xtgc:excessiveGC . . . . . . . . . . . 340-Xtgc:freelist . . . . . . . . . . . . . 340-Xtgc:parallel . . . . . . . . . . . . 341-Xtgc:references. . . . . . . . . . . . 341-Xtgc:scavenger. . . . . . . . . . . . 341-Xtgc:terse . . . . . . . . . . . . . 342

Finding which methods allocated large objects . . 342

Chapter 30. Class-loader diagnostics 345Class-loader command-line options . . . . . . 345Class-loader runtime diagnostics . . . . . . . 345Loading from native code . . . . . . . . . 346

Chapter 31. Shared classesdiagnostics . . . . . . . . . . . . 349Deploying shared classes . . . . . . . . . 349

Cache naming . . . . . . . . . . . . 349Cache access . . . . . . . . . . . . 350Cache housekeeping . . . . . . . . . . 350Cache performance . . . . . . . . . . 351

Compatibility between service releases . . . . 352Nonpersistent shared cache cleanup . . . . . 353

Dealing with runtime bytecode modification . . . 354Potential problems with runtime bytecodemodification . . . . . . . . . . . . . 354Modification contexts . . . . . . . . . . 355SharedClassHelper partitions . . . . . . . 355Using the safemode option . . . . . . . . 356Further considerations for runtime bytecodemodification . . . . . . . . . . . . . 356

Understanding dynamic updates . . . . . . . 357Using the Java Helper API . . . . . . . . . 359

SharedClassHelper api . . . . . . . . . 360Understanding shared classes diagnostics output 361

Verbose output . . . . . . . . . . . . 361VerboseIO output . . . . . . . . . . . 361VerboseHelper output . . . . . . . . . 362printStats utility . . . . . . . . . . . 362printAllStats utility . . . . . . . . . . 363

Debugging problems with shared classes . . . . 364Using shared classes trace . . . . . . . . 364Why classes in the cache might not be found orstored . . . . . . . . . . . . . . . 364Dealing with initialization problems. . . . . 365Dealing with verification problems . . . . . 367Dealing with cache problems . . . . . . . 367

Class sharing with OSGi ClassLoading framework 368

Chapter 32. Using the Reliability,Availability, and ServiceabilityInterface . . . . . . . . . . . . . 369Preparing to use JVMRI . . . . . . . . . . 369

Writing an agent . . . . . . . . . . . 369Registering a trace listener . . . . . . . . 370Changing trace options . . . . . . . . . 371Starting the agent . . . . . . . . . . . 371Building the agent. . . . . . . . . . . 372Agent design . . . . . . . . . . . . 372

JVMRI functions . . . . . . . . . . . . 372API calls provided by JVMRI . . . . . . . . 373

CreateThread . . . . . . . . . . . . 373

DumpDeregister . . . . . . . . . . . 373DumpRegister . . . . . . . . . . . . 373DynamicVerbosegc . . . . . . . . . . 374GenerateHeapdump . . . . . . . . . . 374GenerateJavacore . . . . . . . . . . . 374GetComponentDataArea. . . . . . . . . 374GetRasInfo . . . . . . . . . . . . . 375InitiateSystemDump . . . . . . . . . . 375InjectOutOfMemory . . . . . . . . . . 375InjectSigSegv . . . . . . . . . . . . 375NotifySignal . . . . . . . . . . . . . 376ReleaseRasInfo . . . . . . . . . . . . 376

Contents vii


10/487

RunDumpRoutine . . . . . . . . . . . 376SetOutOfMemoryHook . . . . . . . . . 377TraceDeregister . . . . . . . . . . . . 377TraceDeregister50 . . . . . . . . . . . 377TraceRegister . . . . . . . . . . . . 377TraceRegister50 . . . . . . . . . . . . 378TraceResume . . . . . . . . . . . . 378TraceResumeThis . . . . . . . . . . . 378TraceSet . . . . . . . . . . . . . . 379TraceSnap . . . . . . . . . . . . . 379TraceSuspend . . . . . . . . . . . . 379TraceSuspendThis . . . . . . . . . . . 379

RasInfo structure . . . . . . . . . . . . 380RasInfo request types. . . . . . . . . . . 380Intercepting trace data . . . . . . . . . . 380

The -Xtrace:external=. . . . . . . 380Calling external trace . . . . . . . . . . 381

Formatting . . . . . . . . . . . . . . 381

Chapter 33. Using the HPROF Profiler 383Explanation of the HPROF output file . . . . . 384

Chapter 34. Using the JVMTI . . . . . 389

Chapter 35. Using the Diagnostic ToolFramework for Java . . . . . . . . 391Using the DTFJ interface . . . . . . . . . 391DTFJ example application . . . . . . . . . 395

Chapter 36. Using JConsole . . . . . 399

Part 5. Appendixes . . . . . . . . 403

Appendix A. CORBA minor codes . . 405

Appendix B. Environment variables 407Displaying the current environment . . . . . . 407Setting an environment variable . . . . . . . 407Separating values in a list . . . . . . . . . 408

JVM environment settings . . . . . . . . . 408

z/OS environment variables . . . . . . . . 412

Appendix C. Messages . . . . . . . 413DUMP messages . . . . . . . . . . . . 413

J9VM messages. . . . . . . . . . . . . 415SHRC messages . . . . . . . . . . . . 417

Appendix D. Command-line options 429Specifying command-line options. . . . . . . 429General command-line options . . . . . . . 430System property command-line options . . . . 432

JVM command-line options. . . . . . . . . 433JIT command-line options . . . . . . . . . 440Garbage Collector command-line options . . . . 441

Appendix E. Default settings for theJVM . . . . . . . . . . . . . . . 449

Notices . . . . . . . . . . . . . . 451Trademarks . . . . . . . . . . . . . . 452

Index . . . . . . . . . . . . . . . 455

viii IBM SDK for Java: Diagnostics Guide


11/487

Figures

1. Screenshot of ReportEnv tool . . . . . . 150 2. DTFJ interface diagram . . . . . . . . 394

Copyright IBM Corp. 2003, 2009 ix


12/487

x IBM SDK for Java: Diagnostics Guide


13/487

Tables

Copyright IBM Corp. 2003, 2009 xi


14/487

xii IBM SDK for Java: Diagnostics Guide


15/487

About the Diagnostics Guide

The Diagnostics Guide tells you about how the IBM Virtual Machine for Java

works, debugging techniques, and the diagnostic tools that are available to help

you solve problems with JVMs. It also gives guidance on how to submit problemsto IBM.

What does the Java Virtual Machine (JVM) mean?

The Java Virtual machine (JVM) is the application that executes a Java programand it is included in the Java package.

The installable Java package supplied by IBM comes in two versions on Linux

and Windows platforms:

v The Java Runtime Environment (JRE)

v The Java Software Development Kit (SDK)

The AIX, z/OS, and i5/OS platforms ship only the SDK.

The JRE provides runtime support for Java applications. The SDK provides theJava compiler and other development tools. The SDK includes the JRE.

The JRE (and, therefore, the SDK) includes a JVM. This is the application thatexecutes a Java program. A Java program requires a JVM to run on a particularplatform, such as Linux, z/OS, or Windows.

The IBM SDK, Version 5.0 contains a different implementation of the JVM and theJust-In-Time compiler (JIT) from most earlier releases of the IBM SDK, apart fromthe version 1.4.2 implementation on z/OS 64-bit and on AMD64/EM64T platforms.

You can identify this implementation in the output from the java -versioncommand, which gives these strings for the different implementations:

Implementation Output

5.0 IBM J9 VM (build 2.3, J2RE 1.5.0 IBM...

6 IBM J9 VM (build 2.4, J2RE 1.6.0 IBM...

1.4.2 classic Classic VM (build 1.4.2, J2RE 1.4.2 IBM...

1.4.2 on z/OS 64-bit andAMD64/EM64T platforms

IBM J9SE VM (build 2.2, J2RE 1.4.2 IBM...

i5/OS classic 1.3.1 java version "1.3.1"Java(TM) 2 Runtime Environment, Standard Edition (build ...)Classic VM (build 1.3, build JDK-1.3, native threads, jitc_de)



i5/OS classic 1.6.0 java version "1.6.0"Java(TM) SE Runtime Environment (build ...)Classic VM (build 1.6, build JDK-1.6, native threads, jitc_de)

Copyright IBM Corp. 2003, 2009 xiii


16/487

For Diagnostics Guides that describe earlier IBM SDKs, see http://www.ibm.com/developerworks/java/jdk/diagnosis/. For earlier versions of the i5/OS Classic

JVM, see the iSeries Information Center at http://publib.boulder.ibm.com/infocenter/iseries/v5r4/.

Who should read this book

This book is for anyone who is responsible for solving problems with Java.

Using this book

Before you can use this book, you must have a good understanding of SoftwareDeveloper Kits and the Runtime Environment.

This book is to be used with the IBM SDK and Runtime Environment Version 5.0.Check the full version of your installed JVM. If you do not know how to do this,see Chapter 12, First steps in problem determination, on page 95. Some of thediagnostic tools described in this book do not apply to earlier versions.

You can use this book in three ways:

v As an overview of how the IBM Virtual Machine for Java operates, withemphasis on the interaction with Java. Part 1, Understanding the IBM SDK for

Java, on page 1 of the book provides this information. You might find thisinformation helpful when you are designing your application.

v As straightforward guide to determining a problem type, collecting thenecessary diagnostic data, and sending it to IBM. Part 2, Submitting problemreports, on page 79 and Part 3, Problem determination, on page 93 of the

book provide this information.

v As the reference guide to all the diagnostic tools that are available in the IBMVirtual Machine for Java. This information is given in Part 4, Using diagnostictools, on page 221 of the book.

The parts overlap in some ways. For example, Part 3, Problem determination, onpage 93 refers to chapters that are in Part 4, Using diagnostic tools, on page 221when those chapters describe the diagnostics data that is required. You will be ableto more easily understand some of the diagnostics that are in Part 4, Usingdiagnostic tools, on page 221 if you read the appropriate chapter in Part 1,Understanding the IBM SDK for Java, on page 1.

The appendixes provide supporting reference information that is gathered intoconvenient tables and lists.

Other sources of information

You can obtain additional information about the latest tools and documentation,Java documentation and the IBM SDKs by following the links.

v For the latest tools and documentation, see IBM developerWorks at:

http://www.ibm.com/developerworks/java/

v For Java documentation, see:

http://java.sun.com/reference/docs/index.html

v For the IBM SDKs, see the downloads at:

xiv IBM SDK for Java: Diagnostics Guide
http://www.ibm.com/developerworks/java/jdk/diagnosis/http://www.ibm.com/developerworks/java/jdk/diagnosis/http://publib.boulder.ibm.com/infocenter/iseries/v5r4/http://publib.boulder.ibm.com/infocenter/iseries/v5r4/http://www.ibm.com/developerworks/java/http://java.sun.com/reference/docs/index.htmlhttp://java.sun.com/reference/docs/index.htmlhttp://www.ibm.com/developerworks/java/http://publib.boulder.ibm.com/infocenter/iseries/v5r4/http://publib.boulder.ibm.com/infocenter/iseries/v5r4/http://www.ibm.com/developerworks/java/jdk/diagnosis/http://www.ibm.com/developerworks/java/jdk/diagnosis/


17/487

http://www.ibm.com/developerworks/java/jdk/index.html

Reporting problems in the JVM

If you want to send a problem report, you should understand which sectionsdescribe how to do so.

If you want to use this Information Center only to determine your problem and tosend a problem report to IBM, go to Part 3, Problem determination, on page 93,and to the section that relates to your platform. Go to the section that describes thetype of problem that you are having. This section might offer advice about how tocorrect the problem, and might also offer workarounds. The section will also tellyou what data IBM service needs you to collect to diagnose the problem. Collectthe data and send a problem report and associated data to IBM service, asdescribed in Part 2, Submitting problem reports, on page 79.

Conventions and terminology

Specific conventions are used to describe methods and classes, and command-lineoptions.

Methods and classes are shown in normal font:

v The serviceCall() method

v The StreamRemoteCall class

Command-line options are shown in bold. For example:

v -Xgcthreads

Options shown with values in braces signify that one of the values must bechosen. For example:

-Xverify:{remote | all | none}

with the default underscored.

Options shown with values in brackets signify that the values are optional. Forexample:

-Xrunhprof[:help][=...]

In this information, any reference to Sun is intended as a reference to SunMicrosystems, Inc.

How to send your comments

Your feedback is important in helping to provide accurate and useful information.

If you have any comments about this Diagnostics Guide, you can send them bye-mail to [email protected]. Include the name of the Diagnostics Guide, theplatform you are using, the version of your JVM, and, if applicable, the specificlocation of the text you are commenting on (for example, the title of the page).

Do not use this method for sending in bug reports on the JVM. For these, use theusual methods, as described in Part 2, Submitting problem reports, on page 79.

About the Diagnostics Guide xv
http://www.ibm.com/developerworks/java/jdk/index.htmlhttp://www.ibm.com/developerworks/java/jdk/index.html


18/487

Contributors

This Diagnostics Guide has been put together by members of the Java TechnologyCenter IBM development and service departments in Hursley, Bangalore, Austin,Toronto, Ottawa, and Rochester.

Summary of changesThis topic introduces whats new for this Version 5.0 Diagnostics Guide. ThisVersion 5.0 Diagnostics Guide is based on the Diagnostics Guide for z/OS64 and

AMD64 platforms for the IBM Developer Kit and Runtime Environment, Java 2Technology Edition, Version 1.4.2, SC34-6359-02.

For Version 5, additional platforms and new diagnostics features have been added.

To help people migrating from Version 1.4.2 or earlier, technical changes made forthe first edition are still indicated by revision bars to the left of the changes.

For the tenth edition

The significant changes in this edition are:

v Description of a new dump agent event, called allocation that can be triggeredwhen a Java object is allocated with a specified size. For more information aboutthis event, see Dump events on page 238.

v Description of-Dcom.ibm.nio.DirectByteBuffer.AggressiveMemoryManagement=true inSystem property command-line options on page 432.

v Logging of data now occurs both to the console and to stderr or stdout.

For the ninth edition


v Addition of the JavaReference class to the DTFJ overview diagram, Using theDTFJ interface on page 391.

For the eighth edition


v Description of-Xmxcl in JVM command-line options on page 433 and of theproblem it can resolve in OutOfMemoryError exception when using delegatedclass loaders on page 30.

v Description of-Dsun.timezone.ids.oldmapping in System propertycommand-line options on page 432.

For the seventh edition


v Clarification of the use of jextract and jdmpview on z/OS in Using the dumpviewer, jdmpview on page 272

v Correction to the state flags in Threads and stack trace (THREADS) on page258

xvi IBM SDK for Java: Diagnostics Guide

|

|

|

|

|

|

|

|

|


19/487

For the sixth edition

The significant changes for this edition are:

v The addition of -Xdump:events=systhrow in Dump events on page 238.

v A clarification of the JVM behavior during idle periods in JVM behavior duringidle periods on page 325.

v

A clarification of the results of setting the JIT parameter count=0 in Selectivelydisabling the JIT compiler on page 322.

For the fifth edition

The significant changes for this edition are:

v Error information about va_list for z/OS, in Debugging the JNI on page 75.

v A new section, Text (classic) Heapdump file format on page 265.

For the fourth edition

The fourth edition was based on the third edition, the Diagnostics Guide for Java 2

Technology Edition Version 5.0, SC34-6650-02. Technical changes made for thisedition are indicated by revision bars to the left of the changes.

The significant changes for the fourth edition are:

v Reorganization of chapters in Part 4, Using diagnostic tools, on page 221.

For the third edition

The significant changes for the third edition are:

v A new chapter, Chapter 17, i5/OS problem determination, on page 177.

For the second edition

The significant changes for the second edition are:

v Improved information about debugging performance problems

v A new section, Generational Concurrent Garbage Collector on page 19

v Reviews of the AIX, Linux, Windows, and z/OS chapters in Part 3, Problemdetermination, on page 93

v Inclusion of Chapter 18, Sun Solaris problem determination, on page 199

v Inclusion of Chapter 19, Hewlett-Packard SDK problem determination, onpage 201

v A new section, Deploying shared classes on page 349

v A new chapter, Chapter 35, Using the Diagnostic Tool Framework for Java, onpage 391

For the first edition

The most significant changes are:

v New chapters:

Chapter 4, Class data sharing, on page 33

Chapter 13, AIX problem determination, on page 97, based on the AIX(R)chapter in the Diagnostics Guide for the IBM Developer Kit and RuntimeEnvironment, Java 2 Technology Edition, Version 1.4.2, SC34-6358-02. Revision

bars indicate changes from that chapter.

About the Diagnostics Guide xvii


20/487

Chapter 31, Shared classes diagnostics, on page 349

Chapter 33, Using the HPROF Profiler, on page 383

Chapter 34, Using the JVMTI, on page 389

Appendix C, Messages, on page 413

v Much changed chapters:

Chapter 2, Memory management, on page 7

Part 4, Using diagnostic tools, on page 221

Chapter 24, Using Javadump, on page 251

Chapter 29, Garbage Collector diagnostics, on page 327

xviii IBM SDK for Java: Diagnostics Guide


21/487


22/487

2 IBM SDK for Java: Diagnostics Guide


23/487

Chapter 1. The building blocks of the IBM Virtual Machine forJava

The IBM Virtual Machine for Java (JVM) is a core component of the Java RuntimeEnvironment (JRE) from IBM. The JVM is a virtualized computing machine thatfollows a well-defined specification for the runtime requirements of the Javaprogramming language.

The JVM is called virtual because it provides a machine interface that does notdepend on the underlying operating system and machine hardware architecture.This independence from hardware and operating system is a cornerstone of thewrite-once run-anywhere value of Java programs. Java programs are compiled intobytecodes that target the abstract virtual machine; the JVM is responsible forexecuting the bytecodes on the specific operating system and hardwarecombinations.

The JVM specification also defines several other runtime characteristics. All JVMs:v Execute code that is defined by a standard known as the class file format

v Provide fundamental runtime security such as bytecode verification

v Provide intrinsic operations such as performing arithmetic and allocating newobjects

JVMs that implement the specification completely and correctly are calledcompliant. The IBM Virtual Machine for Java is certified as compliant. Not allcompliant JVMs are identical. JVM implementers have a wide degree of freedom todefine characteristics that are beyond the scope of the specification. For example,implementers might choose to favour performance or memory footprint; theymight design the JVM for rapid deployment on new platforms or for variousdegrees of serviceability.

All the JVMs that are currently used commercially come with a supplementarycompiler that takes bytecodes and produces platform-dependent machine code.This compiler works with the JVM to select parts of the Java program that could

benefit from the compilation of bytecode, and replaces the JVMs virtualizedinterpretation of these areas of bytecode with concrete code. This is called

just-in-time (JIT) compilation. IBMs JIT compiler is described in Chapter 5, TheJIT compiler, on page 35.

The IBM Virtual Machine for Java contains a number of private and proprietarytechnologies that distinguish it from other implementations of the JVM. In thisrelease, IBM has made a significant change to the JVM and JIT compiler that were

provided in earlier releases, while retaining full Java compliance. When you readthis Diagnostics Guide, bear in mind that the particular unspecified behavior of thisrelease of the JVM might be different to the behavior that you experienced inprevious releases. Java programmers should not rely on the unspecified behaviorof a particular JRE for this reason.

This Diagnostics Guide is not a JVM specification; it discusses the characteristics ofthe IBM JRE that might affect the non-functional behavior of your Java program.This guide also provides information to assist you with tracking down problemsand offers advice, from the point of view of the JVM implementer, on how you cantune your applications. There are many other sources for good advice about Java

Copyright IBM Corp. 2003, 2009 3


24/487

performance, descriptions of the semantics of the Java runtime libraries, and toolsto profile and analyze in detail the execution of applications.

Java application stack

A Java application uses the Java class libraries that are provided by the JRE toimplement the application-specific logic. The class libraries, in turn, are

implemented in terms of other class libraries and, eventually, in terms of primitivenative operations that are provided directly by the JVM. In addition, someapplications must access native code directly.

The following diagram shows the components of a typical Java Application Stackand the IBM JRE.

The JVM facilitates the invocation of native functions by Java applications and anumber of well-defined Java Native Interface functions for manipulating Java fromnative code (for more information, see Chapter 8, The Java Native Interface (JNI),on page 67).

Components of the IBM Virtual Machine for JavaThe IBM Virtual Machine for Java technology comprises a set of components.

The following diagram shows component structure of the IBM Virtual Machine forJava:

Java Application

Java Application Stack

JavaCo

de

Native

Code

Java ClassExtensions

Class Libraries ORB

Platform

IBM JVM

Native Libraries

Others

NativeOpt.Packages

UserNativeExts.



25/487

JVM APIThe JVM API encapsulates all the interaction between external programs and the

JVM.

Examples of this interaction include:

v Creation and initialization of the JVM through the invocation APIs.

v Interaction with the standard Java launchers, including handling command-linedirectives.

v Presentation of public JVM APIs such as JNI and JVMTI.

v Presentation and implementation of private JVM APIs used by core Java classes.

Diagnostics componentThe diagnostics component provides Reliability, Availability, and Serviceability(RAS) facilities to the JVM.

The IBM Virtual Machine for Java is distinguished by its extensive RAScapabilities. The IBM Virtual Machine for Java is designed to be deployed in

business-critical operations and includes several trace and debug utilities to assistwith problem determination.

If a problem occurs in the field, it is possible to use the capabilities of thediagnostics component to trace the runtime function of the JVM and help toidentify the cause of the problem. The diagnostics component can produce output

selectively from various parts of the JVM and the JIT. Part 4, Using diagnostictools, on page 221 describes various uses of the diagnostics component.

Memory managementThe memory management component is responsible for the efficient use of systemmemory by a Java application.

Java programs run in a managed execution environment. When a Java programrequires storage, the memory management component allocates the application adiscrete region of unused memory. After the application no longer refers to the

JVM API

DiagnosticsMemory

management

Class loader Interpreter

Platform port layer

Chapter 1. The building blocks of the IBM Virtual Machine for Java 5


26/487

storage, the memory management component must recognize that the storage isunused and reclaim the memory for subsequent reuse by the application or returnit to the operating system.

The memory management component has several policy options that you canspecify when you deploy the application. Chapter 2, Memory management, onpage 7 discusses memory management in the IBM Virtual Machine for Java.

Class loaderThe class loader component is responsible for supporting Javas dynamic codeloading facilities.

The dynamic code loading facilities include:

v Reading standard Java .class files.

v Resolving class definitions in the context of the current runtime environment.

v Verifying the bytecodes defined by the class file to determine whether thebytecodes are language-legal.

v Initializing the class definition after it is accepted into the managed runtime

environment.v Various reflection APIs for introspection on the class and its defined members.

InterpreterThe interpreter is the implementation of the stack-based bytecode machine that isdefined in the JVM specification. Each bytecode affects the state of the machineand, as a whole, the bytecodes define the logic of the application.

The interpreter executes bytecodes on the operand stack, calls native functions,contains and defines the interface to the JIT compiler, and provides support forintrinsic operations such as arithmetic and the creation of new instances of Javaclasses.

The interpreter is designed to execute bytecodes very efficiently. It can switchbetween running bytecodes and handing control to the platform-specificmachine-code produced by the JIT compiler. The JIT compiler is described inChapter 5, The JIT compiler, on page 35.

Platform port layerThe ability to reuse the code for the JVM for numerous operating systems andprocessor architectures is made possible by the platform port layer.

The platform port layer is an abstraction of the native platform functions that arerequired by the JVM. Other components of the JVM are written in terms of the

platform-neutral platform port layer functions. Further porting of the JVM requiresthe provision of implementations of the platform port layer facilities.



27/487

Chapter 2. Memory management

This description of the Garbage Collector and Allocator provides backgroundinformation to help you diagnose problems with memory management.

Memory management is explained under these headings:

v Overview of memory management

v Allocation on page 9

v Detailed description of garbage collection on page 11

v Generational Concurrent Garbage Collector on page 19

v How to do heap sizing on page 20

v Interaction of the Garbage Collector with applications on page 22

v How to coexist with the Garbage Collector on page 22

v Frequently asked questions about the Garbage Collector on page 25

For detailed information about diagnosing Garbage Collector problems, seeChapter 29, Garbage Collector diagnostics, on page 327.

See also the reference information in Garbage Collector command-line options onpage 441.

Overview of memory management

Memory management contains the Garbage Collector and the Allocator. It isresponsible for allocating memory in addition to collecting garbage. Because thetask of memory allocation is small, compared to that of garbage collection, theterm garbage collection usually also means memory management.

This section includes:

v A summary of some of the diagnostic techniques related to memorymanagement.

v An understanding of the way that the Garbage Collector works, so that you candesign applications accordingly.

The Garbage Collector allocates areas of storage in the heap. These areas of storagedefine Java objects. When allocated, an object continues to be live while a reference(pointer) to it exists somewhere in the JVM; therefore the object is reachable. Whenan object ceases to be referenced from the active state, it becomes garbage and can

be reclaimed for reuse. When this reclamation occurs, the Garbage Collector must

process a possible finalizer and also ensure that any internal JVM resources thatare associated with the object are returned to the pool of such resources.

Object allocationObject allocation is driven by requests by applications, class libraries, and the JVMfor storage of Java objects, which can vary in size and require different handling.

Every allocation requires a heap lock to be acquired to prevent concurrent threadaccess. To optimize this allocation, particular areas of the heap are dedicated to athread, known as the TLH (thread local heap), and that thread can allocate from itsTLH without having to lock out other threads. This technique delivers the best

Copyright IBM Corp. 2003, 2009 7


28/487

possible allocation performance for small objects. Objects are allocated directlyfrom a thread local heap. A new object is allocated from this cache withoutneeding to grab the heap lock. All objects less than 512 bytes (768 bytes on 64-bit

JVMs) are allocated from the cache. Larger objects are allocated from the cache ifthey can be contained in the existing cache. This cache is often referred to as thethread local heap or TLH.

Reachable objectsReachable objects are found using frames on the thread stack, roots and references.

The active state of the JVM is made up of the set of stacks that represents thethreads, the statics that are inside Java classes, and the set of local and global JNIreferences. All functions that are called inside the JVM itself cause a frame on thethread stack. This information is used to find the roots. A root is an object whichhas a reference to it from outside the heap. These roots are then used to findreferences to other objects. This process is repeated until all reachable objects arefound.

Garbage collectionWhen the JVM cannot allocate an object from the heap because of lack ofcontiguous space, a memory allocation fault occurs, and the Garbage Collector iscalled.

The first task of the Garbage Collector is to collect all the garbage that is in theheap. This process starts when any thread calls the Garbage Collector eitherindirectly as a result of allocation failure, or directly by a specific call toSystem.gc(). The first step is to acquire exclusive control on the virtual machine toprevent any further Java operations. Garbage collection can then begin.

Heap sizing problemsIf the operation of the heap, using the default settings, does not give the best

results for your application, there are actions that you can take.

For the majority of applications, the default settings work well. The heap expandsuntil it reaches a steady state, then remains in that state, which should give a heapoccupancy (the amount of live data on the heap at any given time) of 70%. At thislevel, the frequency and pause time of garbage collection should be acceptable.

For some applications, the default settings might not give the best results. Listedhere are some problems that might occur, and some suggested actions that you cantake. Use verbose:gc to help you monitor the heap.

The frequency of garbage collections is too high until the heap reaches a steadystate.

Use verbose:gc to determine the size of the heap at a steady state and set -Xmsto this value.

The heap is fully expanded and the occupancy level is greater than 70%.Increase the -Xmx value so that the heap is not more than 70% occupied, butfor best performance try to ensure that the heap never pages. The maximumheap size should, if possible, be able to be contained in physical memory toavoid paging.

At 70% occupancy the frequency of garbage collections is too great.Change the setting of -Xminf. The default is 0.3, which tries to maintain 30%



29/487

free space by expanding the heap. A setting of 0.4, for example, increases thisfree space target to 40%, and reduces the frequency of garbage collections.

Pause times are too long.Try using -Xgcpolicy:optavgpause. This reduces the pause times and makesthem more consistent when the heap occupancy rises. It does, however, reducethroughput by approximately 5%, although this value varies with different

applications.

Here are some useful tips:

v Ensure that the heap never pages; that is, the maximum heap size must be ableto be contained in physical memory.

v Avoid finalizers. You cannot guarantee when a finalizer will run, and often theycause problems. If you do use finalizers, try to avoid allocating objects in thefinalizer method. A verbose:gc trace shows whether finalizers are being called.

v Avoid compaction. A verbose:gc trace shows whether compaction is occurring.Compaction is usually caused by requests for large memory allocations. Analyzerequests for large memory allocations and avoid them if possible. If they arelarge arrays, for example, try to split them into smaller arrays.

Allocation

The memory management component contains the Garbage Collector and theAllocator. The task of memory allocation is small, compared to that of garbagecollection.

Heap lock allocationHeap lock allocation occurs when the allocation request cannot be satisfied in theexisting cache.

For a description of cache allocation, when the request cannot be satisfied, seeCache allocation. As its name implies, heap lock allocation requires a lock and istherefore avoided, if possible, by using the cache.

If the Garbage Collector cannot find a big enough chunk of free storage, allocationfails and the Garbage Collector must perform a garbage collection. After a garbagecollection cycle, if the Garbage Collector created enough free storage, it searchesthe freelist again and picks up a free chunk. The heap lock is released either afterthe object has been allocated, or if not enough free space is found. If the GarbageCollector does not find enough free storage, it returns OutOfMemoryError.

Cache allocationCache allocation is specifically designed to deliver the best possible allocationperformance for small objects. Objects are allocated directly from a thread local

allocation buffer that the thread has previously allocated from the heap. A newobject is allocated from this cache without the need to grab the heap lock;therefore, cache allocation is very efficient.

All objects less than 512 bytes (768 bytes on 64-bit JVMs) are allocated from thecache. Larger objects are allocated from the cache if they can be contained in theexisting cache; if not a locked heap allocation is performed.

Chapter 2. Memory management 9


30/487

The cache block is sometimes called a thread local heap (TLH). The size of theTLH varies from 2 KB to 128 KB, depending on the allocation rate of the thread.Threads which allocate lots of objects are given larger TLHs to further reducecontention on the heap lock.

Large Object Area

As objects are allocated and freed, the heap can become fragmented in such a waythat allocation can be met only by time-consuming compactions. This problem ismore pronounced if an application allocates large objects. In an attempt to alleviatethis problem, the large object area (LOA) is allocated.

The LOA is an area of the tenure area of the heap set used solely to satisfyallocations for large objects that cannot be satisfied in the main area of the tenureheap, which is referred to in the rest of this section as the small object area (SOA).A large object in this context is considered to be any object 64 KB or greater in size;allocations for new TLH objects are not considered to be large objects. The largeobject area is allocated by default for all GC polices except -Xgcpolicy:subpool (forAIX, Linux PPC and zSeries, z/OS, and i5/OS) but, if it is not used, it is shrunkto zero after a few collections. It can be disabled explicitly by specifying the

-Xnoloa command-line option.

Initialization and the LOAThe LOA boundary is calculated when the heap is initialized, and recalculatedafter every garbage collection. The size of the LOA can be controlled usingcommand-line options: -Xloainitial and -Xloamaximum.

The options take values between 0 and 0.95 (0% thru 95% of the current tenureheap size). The defaults are:

v -Xloainitial0.05 (5%)

v -Xloamaximum0.5 (50%)

Expansion and shrinkage of the LOAThe Garbage Collector expands or shrinks the LOA, depending on usage.

The Garbage Collector uses the following algorithm:

v If an allocation failure occurs in the SOA:

If the current size of the LOA is greater than its initial size and if the amountof free space in the LOA is greater than 70%, reduce by 1% the percentage ofspace that is allocated to the LOA.

If the current size of the LOA is equal to or less than its initial size, and if theamount of free space in the LOA is greater than 90%:

- If the current size of the LOA is greater than 1% of the heap, reduce by 1%the percentage of space that is allocated to the LOA.

- If the current size of the LOA is 1% or less of the heap, reduce by 0.1%, thepercentage of space that is allocated to the LOA.

v If an allocation failure occurs on the LOA:

If the size of the allocation request is greater than 20% of the current size ofthe LOA, increase the LOA by 1%.

If the current size of the LOA is less than its initial size, and if the amount offree space in the LOA is less than 50%, increase the LOA by 1%.

If the current size of the LOA is equal to or greater than its initial size, and ifthe amount of free space in the LOA is less than 30%, increase the LOA by1%.



31/487

Allocation in the LOAWhen allocating an object, the allocation is first attempted in the SOA. If it is notpossible to find a free entry of sufficient size to satisfy the allocation, and the sizeof the request is equal to or greater than 64 KB, the allocation is tried in the LOAagain. If the size of the request is less than 64 KB or insufficient contiguous spaceexists in the LOA, an allocation failure is triggered.

Detailed description of garbage collection

Garbage collection is performed when an allocation failure occurs in heap lockallocation, or if a specific call to System.gc() occurs. The thread that has theallocation failure or the System.gc() call takes control and performs the garbagecollection.

The first step in garbage collection is to acquire exclusive control on the Virtualmachine to prevent any further Java operations. Garbage collection then goesthrough the three phases: mark, sweep, and, if required, compaction. The IBMGarbage Collector is a stop-the-world (STW) operation, because all applicationthreads are stopped while the garbage is collected.

Mark phaseIn mark phase, all the live objects are marked. Because unreachable objects cannot

be identified singly, all the reachable objects must be identified. Therefore,everything else must be garbage. The process of marking all reachable objects isalso known as tracing.

The mark phase uses:

v A pool of structures called work packets. Each work packet contains a mark stack.A mark stack contains references to live objects that have not yet been traced.Each marking thread refers to two work packets; an input packet from whichreferences are popped and an output packet to which unmarked objects that

have just been discovered are pushed. References are marked when they arepushed onto the output packet. When the input packet becomes empty, it isadded to a list of empty packets and replaced by a non-empty packet. When theoutput packet becomes full it is added to a list of non-empty packets andreplaced by a packet from the empty list.

v A bit vector called the mark bit array identifies the objects that are reachable andhave been visited. This bit array is allocated by the JVM at startup based on themaximum heap size (-Xmx).The mark bit array contains one bit for each 8 bytesof heap space. The bit that corresponds to the start address for each reachableobject is set when it is first visited.

The first stage of tracing is the identification of root objects. The active state of theJVM is made up of the saved registers for each thread, the set of stacks that

represent the threads, the statics that are in Java classes, and the set of local andglobal JNI references. All functions that are called in the JVM itself cause a frameon the C stack. This frame might contain references to objects as a result of eitheran assignment to a local variable, or a parameter that is sent from the caller. Allthese references are treated equally by the tracing routines.

All the mark bits for all root objects are set and references to the roots pushed tothe output work packet. Tracing then proceeds by iteratively popping a referenceoff the marking threads input work packet and then scanning the referenced objectfor references to other objects. If there are any references to unmarked objects, thatis, mark bit is off, the object is marked by setting the appropriate bit in the mark



32/487

bit array and the reference is pushed to the marking threads output work packet.This process continues until all the work packets are on the empty list, at whichpoint all the reachable objects have been identified.

Mark stack overflowBecause the set of work packets has a finite size, it can overflow and the GarbageCollector then performs a series of actions.

If an overflow occurs, the Garbage Collector empties one of the work packets bypopping its references one at a time, and chaining the referenced objects off theirowning class by using the class pointer slot in the object header. All classes withoverflow objects are also chained together. Tracing can then continue as before. If afurther mark stack overflow occurs, more packets are emptied in the same way.

When a marking thread asks for a new non-empty packet and all work packets areempty, the GC checks the list of overflow classes. If the list is not empty, the GCtraverses this list and repopulates a work packet with the references to the objectson the overflow lists. These packets are then processed as described above. Tracingis complete when all the work packets are empty and the overflow list is empty.

Parallel markThe goal of parallel mark is to not degrade mark performance on a uniprocessor,and to increase typical mark performance on a multiprocessor system. Objectmarking is increased through the addition of helper threads that share the use ofthe pool of work packets; for example, full output packets that are returned to thepool by one thread can be picked up as new input packets by another thread.

Parallel mark still requires the participation of one application thread that is usedas the master coordinating agent. This thread performs very much as it always did,

but the helper threads assist both in the identification of the root pointers for thecollection and in the tracing of these roots. Mark bits are updated by using hostmachine atomic primitives that require no additional lock.

By default, a platform with n processors also has n-1 new helper threads, thatwork with the master thread to complete the marking phase of garbage collection.You can override the default number of threads by using the -Xgcthreads option. Ifyou specify a value of 1, there will be no helper threads. The -Xgcthreads optionaccepts any value greater than 0, but clearly you gain nothing by setting it to morethan N-1.

Concurrent markConcurrent mark gives reduced and consistent garbage collection pause timeswhen heap sizes increase. It starts a concurrent marking phase before the heap isfull. In the concurrent phase, the Garbage Collector scans the roots, i.e. stacks, JNIreferences, class statics, and so on. The stacks are scanned by asking each thread to

scan its own stack. These roots are then used to trace live objects concurrently.Tracing is done by a low-priority background thread and by each applicationthread when it does a heap lock allocation.

While the Garbage Collector is marking live objects concurrently with applicationthreads running, it has to record any changes to objects that are already traced. Ituses a write barrier that is run every time a reference in an object is updated. Thewrite barrier flags when an object reference update has occurred, to force a rescanof part of the heap. The heap is divided into 512-byte sections and each section isallocated a one-byte card in the card table. Whenever a reference to an object isupdated, the card that corresponds to the start address of the object that has been



33/487

updated with the new object reference is marked with 0x01. A byte is used insteadof a bit to eliminate contention; it allows marking of the cards using non-atomicoperations. An STW collection is started when one of the following occurs:

v An allocation failure

v A System.gc

v Concurrent mark completes all the marking that it can do

The Garbage Collector tries to start the concurrent mark phase so that it completesat the same time as the heap is exhausted. The Garbage Collector does this byconstant tuning of the parameters that govern the concurrent mark time. In theSTW phase, the Garbage Collector rescans all roots and uses the marked cards tosee what else must be retraced, and then sweeps as normal. It is guaranteed thatall objects that were unreachable at the start of the concurrent phase are collected.It is not guaranteed that objects that become unreachable during the concurrentphase are collected. Objects which become unreachable during the concurrentphase are referred to as floating garbage.

Reduced and consistent pause times are the benefits of concurrent mark, but theycome at a cost. Application threads must do some tracing when they are requesting

a heap lock allocation. The processor usage needed varies depending on howmuch idle CPU time is available for the background thread. Also, the write barrierrequires additional processor usage.

The -Xgcpolicy command-line parameter is used to enable and disable concurrentmark:

-Xgcpolicy:

The -Xgcpolicy options have these effects:

optthruputDisables concurrent mark. If you do not have pause time problems (as seen

by erratic application response times), you get the best throughput withthis option. Optthruput is the default setting.

optavgpauseEnables concurrent mark with its default values. If you are havingproblems with erratic application response times that are caused by normalgarbage collections, you can reduce those problems at the cost of somethroughput, by using the optavgpause option.

genconRequests the combined use of concurrent and generational GC to helpminimize the time that is spent in any garbage collection pause.

subpool

Disables concurrent mark. It uses an improved object allocation algorithmto achieve better performance when allocating objects on the heap. Thisoption might improve performance on SMP systems with 16 or moreprocessors. The subpool option is available only on AIX, Linux PPC andzSeries, z/OS, and i5/OS.

Sweep phaseOn completion of the mark phase the mark bit vector identifies the location of allthe live objects in the heap. The sweep phase uses this to identify those chunks ofheap storage that can be reclaimed for future allocations; these chunks are addedto the pool of free space. To avoid filling this free space pool with lots of small



34/487

chunks of storage, only chunks of at least a certain size are reclaimed and added tothe free pool. The minimum size for a free chunk is currently defined as 512 bytes(768 bytes on 64-bit platforms).

A free chunk is identified by examining the mark bit vector looking for sequencesof zeros, which identify possible free space. GC ignores any sequences of zeros thatcorrespond to a length less than the minimum free size. When a sequence of

sufficient length is found, the Garbage Collector checks the length of the object atthe start of the sequence to determine the actual amount of free space that can bereclaimed. If this amount is greater than or equal to the minimum size for a freechunk, it is reclaimed and added to the free space pool.

The small areas of storage that are not on the freelist are known as dark matter,and they are recovered when the objects that are next to them become free, orwhen the heap is compacted. It is not necessary to free the individual objects in thefree chunk, because it is known that the whole chunk is free storage. When achunk is freed, the Garbage Collector has no knowledge of the objects that were init.

Parallel bitwise sweepParallel bitwise sweep improves the sweep time by using available processors. Inparallel bitwise sweep, the Garbage Collector uses the same helper threads that areused in parallel mark, so the default number of helper threads is also the same andcan be changed with the -Xgcthreads option.

The heap is divided into sections of 256 KB and each thread (helper or master)takes a section at a time and scans it, performing a modified bitwise sweep. Theresults of this scan are stored for each section. When all sections have beenscanned, the freelist is built.

Concurrent sweepLike concurrent mark, concurrent sweep gives reduced garbage collection pausetimes when heap sizes increase. Concurrent sweep starts immediately after astop-the-world (STW) collection, and must at least complete a certain subset of itswork before concurrent mark is allowed to kick off, because the mark map usedfor concurrent mark is also used for sweeping, as described previously.

The concurrent sweep process is split into two types of operations:

v Sweep analysis, which processes sections of data in the mark map anddetermines ranges of free or potentially free memory to be added to the free list

v Connection, which takes analyzed sections of the heap and connects them intothe free list

Heap sections are calculated in the same way as for parallel bitwise sweep.

An STW collection initially performs a minimal sweep operation that searches forand finds a free entry large enough to satisfy the current allocation failure. Theremaining unprocessed portion of the heap and mark map are left to concurrentsweep to be both analyzed and connected. This work is accomplished by Javathreads through the allocation process. For a successful allocation, an amount ofheap relative to the size of the allocation is analyzed, and is performed outside theallocation lock. In an allocation, if the current free list cannot satisfy the request,sections of analyzed heap are found and connected into the free list. If sectionsexist but are not analyzed, the allocating thread must also analyze them beforeconnecting.



35/487

Because the sweep is incomplete at the end of the STW collection, the amount offree memory reported (through verbose GC or the API) is an estimate based onpast heap occupancy and the ratio of unprocessed heap size against total heap size.In addition, the mechanics of compaction require that a sweep be completed beforea compaction can occur. Consequently, an STW collection that compacts will nothave concurrent sweep active during the next round of execution.

To enable concurrent sweep, use the -Xgcpolicy: parameter optavgpause. It isactive in conjunction with concurrent mark. The modes optthruput, gencon, andsubpool do not support concurrent sweep.

Compaction phaseWhen the garbage has been removed from the heap, the Garbage Collector canconsider compacting the resulting set of objects to remove the spaces that are

between them. The process of compaction is complicated because, if any object ismoved, the Garbage Collector must change all the references that exist to it. Thedefault is not to compact.

The following analogy might help you understand the compaction process. Think

of the heap as a warehouse that is partly full of pieces of furniture of differentsizes. The free space is the gaps between the furniture. The free list contains onlygaps that are above a particular size. Compaction pushes everything in onedirection and closes all the gaps. It starts with the object that is closest to the wall,and puts that object against the wall. Then it takes the second object in line andputs that against the first. Then it takes the third and puts it against the second,and so on. At the end, all the furniture is at one end of the warehouse and all thefree space is at the other.

To keep compaction times to a minimum, the helper threads are used again.

Compaction occurs if any one of the following is true and -Xnocompactgc has notbeen specified:

v -Xcompactgc has been specified.

v Following the sweep phase, not enough free space is available to satisfy theallocation request.

v A System.gc() has been requested and the last allocation failure garbagecollection did not compact or -Xcompactexplicitgc has been specified.

v At least half the previously available memory has been consumed by TLHallocations (ensuring an accurate sample) and the average TLH size falls below1024 bytes

v Less than 5% of the active heap is free.

v Less than 128 KB of the heap is free.

Subpool (AIX, Linux PPC and zSeries, z/OS and i5/OS only)Subpool is an improved GC policy for object allocation that is specifically targetedat improving the performance of object allocation on SMP systems with 16 or moreprocessors. You start it with the -Xgcpolicy:subpool command-line option.

The subpool algorithm uses multiple free lists rather than the single free list usedby optavgpause and optthruput. It tries to predict the size of future allocationrequests based on earlier allocation requests. It recreates free lists at the end ofeach GC based on these predictions. While allocating objects on the heap, freechunks are chosen using a best fit method, as against the first fit method used



36/487

in other algorithms. It also tries to minimize the amount of time for which a lock isheld on the Java heap, thus reducing contention among allocator threads.Concurrent mark is disabled when subpool policy is used. Also, subpool policyuses a new algorithm for managing the Large Object Area (LOA). Hence, thesubpool option might provide additional throughput optimization for someapplications.

Reference objectsWhen a reference object is created, it is added to a list of reference objects of thesame type. The referent is the object to which the reference object points. Instancesof SoftReference, WeakReference, and PhantomReference are created by the userand cannot be changed; they cannot be made to refer to objects other than theobject that they referenced on creation. Objects with a class that defines a finalizemethod result in a pointer to that object being placed on a list of objects thatrequire finalization.

During garbage collection, immediately following the mark phase, these lists areprocessed in a specific order:

1. Soft

2. Weak

3. Final

4. Phantom

Soft, weak, and phantom reference processingThe Garbage Collector determines if a reference object is a candidate for collectionand, if so, performs a collection process that differs for each reference type. Softreferences are collected if their referent has not been marked for the previous 32garbage collection cycles. Weak and phantom references are always collected iftheir referent is not marked.

For each element on a list, GC determines if the reference object is eligible for

processing and then if it is eligible for collection.

An element is eligible for processing if it is marked and has a non-null referentfield. If this is not the case, the reference object is removed from the reference list,resulting in it being freed during the sweep phase.

If an element is determined to be eligible for processing, GC must determine if it iseligible for collection. The first criterion here is simple. Is the referent marked? If itis marked, the reference object is not eligible for collection and GC moves onto thenext element of the list.

If the referent is not marked, GC has a candidate for collection. At this point theprocess differs for each reference type. Soft references are collected if their referent

has not been marked for the previous 32 garbage collection cycles. You adjust thefrequency of collection with the -Xsoftrefthreshold option. If there is a shortage ofavailable storage, all soft references are cleared. All soft references are guaranteedto have been cleared before the OutOfMemoryError is thrown.

Weak and phantom references are always collected if their referent is not marked.When a phantom reference is processed, its referent is marked so it will persistuntil the following garbage collection cycle or until the phantom reference isprocessed if it is associated with a reference queue. When it is determined that areference is eligible for collection, it is either queued to its associated referencequeue or removed from the reference list.



37/487

Final reference processingThe processing of objects that require finalization is more straightforward.

1. The list of objects is processed. Any element that is not marked is processed by:

a. Marking and tracing the object

b. Creating an entry on the finalizable object list for the object

2. The GC removes the element from the unfinalized object list.3. The final method for the object is run at an undetermined point in the future

by the reference handler thread.

JNI weak referenceJNI weak references provide the same capability as that of WeakReference objects,but the processing is very different. A JNI routine can create a JNI Weak referenceto an object and later delete that reference.