| J avaO ne 2003 | S ess ion 2690

Performance Monitoring for the Java™ Virtual Machine (JVM™)

Brian Doherty, Ben Kim,David StoutamireJ ava P erformance E ngineeringS un Microsystems, Inc.

| J avaO ne 2003 | S ess ion 26902

Presentation Goal 

L earn about non- intrus ive performance monitoring and the tools available for monitoring production systems utiliz ing the J ava ™ software s tack.

| J avaO ne 2003 | S ess ion 26903

Speaker’s Qualifications

• Brian, Ben, and D avid are all members ofthe J ava ™ P latform P erformance E ngineeringgroup at S un Microsystems

• We represent expertise in performance measurement, analys is , and tuning of industry s tandard benchmarks, customer applications, the J ava ™ class libraries , and the J avaHotS pot™ virtual machine

| J avaO ne 2003 | S ess ion 26904

Performance Measurement—A Product Lifecycle Process

D on’t s top monitoring your application just because it’s been deployed!

O bserve your production applications us ing minimally invas ive tools that are becoming better integrated across the software s tack.

| J avaO ne 2003 | S ess ion 26905

Agenda 

• What is performance monitoring

• Monitoring J ava™ applications─ J ava™ software s tack─ D istributed applications

• D emonstration─ T ools─ A nalys is─ T uning

| J avaO ne 2003 | S ess ion 26906

Monitoring Goals

• Maintain and manage system health– D etection of performance events– D etection of application progress or liveness

• P erformance tuning– D rilling down to find causes– Identifying solutions

| J avaO ne 2003 | S ess ion 26907

Performance Management Process

Monitor

Start

Analysis

Design

Code

Deploy

ProfileNo

Yes

PerformanceOK

Benchmark

| J avaO ne 2003 | S ess ion 26908

Profiling Vs. Monitoring

• P roduction vs . D evelopment– P roduction systems have more restrictive

measurement needs– P roduction loads vary over time– P roduction loads difficult to s imulate

| J avaO ne 2003 | S ess ion 26909

Profiling Vs. Monitoring

Profiling MonitoringIntrus ion High Low

D etail High LowA ccuracy High LowR equired K nowledge High Low

| J avaO ne 2003 | S ess ion 269010

Complexities

Hardware

Operating System

Virtual Machine

Application Server

Application

T he J ava ™ S oftware S tack

| J avaO ne 2003 | S ess ion 269011

Complexities

C lientT ier

WebT ier

J 2E E ™ A pplicationS erver T ier

D atabaseT ier

T he D eployment A rchitecture

| J avaO ne 2003 | S ess ion 269012

Monitoring Tools

• Network management systems

• O S monitoring tools

• J V M monitoring tools

• J 2E E™ monitoring tools

Demo

| J avaO ne 2003 | S ess ion 269014

What We Saw

• S ymptoms─ E xcess ive memory utilization─ R estart

• T ools seen here─ R untime.freeMemory()─ gcore─ S erviceability A gent

• Identified source of memory consumption

| J avaO ne 2003 | S ess ion 269015

Other Tools

• P roduction─ - verbose:gc

• D evelopment─ - verbose:gc─ J V MP I/J V MD I profiling and debugging tools

Demo

| J avaO ne 2003 | S ess ion 269017

What We Saw

• S ymptoms─ Incons is tent response times

• T ools seen here─ V isualG C

• Identified explicit S ystem.gc() calls

| J avaO ne 2003 | S ess ion 269018

Other Tools

• P roduction─ - verbose:gc─ mpstat—identify s ingle threaded G C─ pstack—print O S thread dump─ S IG INT —print J ava thread dump

• D evelopment─ -X prof─ J V MP I/J V MD I profiling and debugging tools─ S erviceability A gent

Demo

| J avaO ne 2003 | S ess ion 269020

What We Saw

• S ymptoms─ P oor system utilization─ L ow throughput

• T ools used─ Y A X (perfmeter)

• C P U only ~50% utilized─ S IG INT (thread dump)

• T hreads waiting on locks• Identified source of lock contention

| J avaO ne 2003 | S ess ion 269021

Other Tools

• P roduction─ B ytecode instrumentation─ prstat / top─ pstack

• D evelopment─ -X prof─ J V MP I/J V MD I profiling and debugging tools

| J avaO ne 2003 | S ess ion 269022

What We Covered

Hardware

Operating System

Virtual Machine

Application Server

Application

Monitoring the J ava™ S oftware S tack

| J avaO ne 2003 | S ess ion 269023

What We Covered

• UNIX stat tools– vmstat, ios tat, mpstat,

nets tat ks tat• UNIX proc tools– pstack, pmap, prs tat/top,

truss/strace• Hardware counters– cpustat, busstat•Misc. tools– S E T oolkit

Monitoring the J ava™ S oftware S tack

Hardware

Operating System

Virtual Machine

Application Server

Application

| J avaO ne 2003 | S ess ion 269024

What We Covered

• jvmstat tools• C ommand line options– - verbose:[gc|class |jni]– - X prof– - X X options• S erviceability A gent• J S R 174 interfaces

Hardware

Operating System

Virtual Machine

Application Server

Application

Monitoring the J ava™ S oftware S tack

| J avaO ne 2003 | S ess ion 269025

What We Covered

• J MX interfaces– S NMP , C IM/WBE M• Bytecode instrumentation– V arious commercial tools• J V MP I, J V MD I tools– C ommercial and free

Hardware

Operating System

Virtual Machine

Application Server

Application

Monitoring the J ava™ S oftware S tack

| J avaO ne 2003 | S ess ion 269026

Summary

• P erformance measurement needs to be a product lifecycle process─ Not just a development process

• Non- intrus ive tools are critical for measurement of production applications

• T ools integrated across the software s tack ease the measurement and analys is process

| J avaO ne 2003 | S ess ion 269027

Resources

• HotS pot performance documentationhttp://java.sun.com/docs/performance

• jvmstathttp://www.sun.com/developers/coolstuff

• S olaris ™ performance documentationhttp://docs .sun.com

• J S R 174: Monitoring and Management S pecification for the J ava™ virtual machine

Q&A