WebLogic Diagnostic Framework Dr. Frank Munz / munz & more WLS11g

1

munz & moreDr. Frank Munz

Insync, Aug2010

WebLogic Diagnostic

Framework For the Busy IT Professional

welcome!

about me Frank Munz

10 years

Java EEandEAI

with some BPM

= SOA

2

3years ago

munz & morehttp://www.munzandmore.com

working globally

based @ Manly & Munich

3

Oracle Fusion Middleware

4

Weblogic Server

Version 4 to 11gOracle Service Bus

from the very beginning

Architecture

Operation

Development

Training / Coaching

high end curriculum

WLS / OSB / SOA

arch / features / operation

monitoring / sizing / performance tuning

Weblogic Server CourseArchitecture, Features and Operation

Monitoring, Sizing and Performance

Tuning

Oracle Service Bus Course Architecture, Development and

Operation

5

“Best of WLDF”reasons

1.

2. failure becomes news

6

some

45.000.000passengers per year

(max)

30 821in 6 hours

30 821passengers

7

looking at:

worse: this went on for hours…

monitoring + diagnostic

is important important

8

“So far, this was not necessary” 3.

toolshow is

your most important

application doing right now?

?

9

Free Heap? Size of http session?

Time to

replicate http session

in cluster?

Free DB connections

in connection pool?

What SQL is sent

to the DB?

At which LOC?

How long is it executing?

positioning

10

Operations

EMNagios

Developer

ProfilerAOP

WLDF

features

monitoring + diagnostics

server + applications

persists data

is already installed

is free

only minutes to activate

JMX MBeans encapsulate

11

Configuration Data

+

Runtime ValuesMBeans have

attributes

operations

children

notifications

WLST

start WLST with:

java weblogic.WLST [Script.py]

-> domainRuntime()

WLST Quick Reference

ls() show current mbean / cmo = current managed objectls('a') shows attributes , 'c' child mbeans, 'o' operations

cd() navigate

serverConfig() change to config mbeansserverRuntime() change to server runtimedomainRuntime() change to domain runtimejndi() change to jndi tree

edit() change to edit treestartEdit() start an edit session, then use set('pn','v') / get('pn')save() persist changesactivate() activate changesundo() undo changeseaseSyntaxeaseSyntaxeaseSyntaxeaseSyntax()()()() switch off python syntax, do not use for scriptingswitch off python syntax, do not use for scriptingswitch off python syntax, do not use for scriptingswitch off python syntax, do not use for scripting

WLDF basics

12

Diagnostic System Module

Create a system modul,target it to any number of servers

• You can have many modules configured• Set target to particular server when needed• You can change module target witout restart

collected metrics

Collected Metrics

... replace your WLST script that contains a loop with sleep() e.g. to monitor free heap.

Remember:Collected metrics = harvested data

Collected MetricType and Attributes

Diagnostic Archives

• File (default)

• DB archive

both come with retirement options

watches

&

notifications

13

Watches

A watch consists of:

• expression / rule

• one or more notifications

• alarm setting

Watches

Watches can based on:

• collected metrics

• server logs

• event data (= instrumentation)

WatchRule Expression

Notification

Triggered based on watch rules.

Define all underlying transport protocol configurations first

Notification Types Supported

• JMX • JMS• SMTP • SNMP• Diagnostic Images

instrumentation

14

Instrumentation

Monitor= Unit of diagnostic code

Action= what happens when monitor is triggered

Instrumentation Scope

Server scope= fixed location

Application scope= custom location

Delegate Monitor

Selection of predefined monitors

Actions

Actions create events:• DisplayArgumentsAction (captures args and return value)

• StackDumpAction

• ThreadDumpAction

• TraceAction (generatesTrace event)

• TraceElapsedTimeAction

• TraceMemoryAllocationAction (WLS 10.3.3)

• MethodInvocationStatisticsAction (WLS10.3)

MethodInvocationStatisticsAction

Computes method invocation statistics in memory without persisting.

Access this information through InstrumentationRuntimeMBean by using collected metrics or a watch with notifcation.

MethodInvocationStatisticsAction

MethodInvocationStatisticsMethodInvocationStatisticsMethodInvocationStatisticsMethodInvocationStatistics::=

Map<className, MethodMap>

MethodMapMethodMapMethodMapMethodMap::=

Map<methodName, MethodParamsSignatureMap>

MethodParamsSignatureMapMethodParamsSignatureMapMethodParamsSignatureMapMethodParamsSignatureMap::= Map<MethodParamsSignature,

MethodDataMap>

MethodDataMapMethodDataMapMethodDataMapMethodDataMap::=

<MetricName, Statistic>

MetricNameMetricNameMetricNameMetricName:= min | max | ... std_deviation

15

MethodInvocationStatisticsAction

MethodInvocationStatistics

{java.sql.Statement={executeQuery={java.lang.Stri

ng={count=4, sum=1.0015808E7,

std_deviation=466173.85858443414, avg=2503952.0,

sum_of_squares=2.594

8374738926E13, max=3307465, min=2174694}}}}

MethodStats Application

http://www.oracle.com/technetwork/articles/

cico-wldf-091073.html

Custom MonitorExample<wldf-resource ...

<instrumentation>

<enabled>true</enabled>

<wldf-instrumentation-monitor>

<name>FranksMonitor</name>

<enabled>true</enabled>

<action>TraceElapsedTimeAction</action>

<location-type>around</location-type>

<pointcut>execute ( * com.surfandconsulting.*

bookFligh* (...));</pointcut>

</wldf-instrumentation-monitor>

Instrumentation

always:

Deploy / Redeploy / Update to apply

Instrumentation uses AspectJ

Hot Swap

Without hot swap -> redeployWith hot swap -> update in place

Enable hot swap with:

-javaagent:$WL_HOME\server\lib\diagnostics-agent.jar

dye injection

16

incoming requests

user1=sharon

addr1=192.168.0.7

dye injectionmonitor:

time

Dye Injection Monitor

Monitor sets/unsets bits in

64 bit dye vectorto mark requests

to be able to filter them later

Dye Injection Monitor Dye Injection

Throttle Dye further reduces number of samples:

• THROTTLE_INTERVAL (milliseconds)

• THROTTLE_RATE (n-th sample)

diagnostic images

Capture Image

17

You cannot open these .img files with photoshop

;-)

WLDFconsole extension

plot mbean attributedata over time

example:

connection pool with

5 connections

under load:

18

with “PinnedToThread”enabled:

/dashboard

is the new WLDF console extension

Conclusionwell designed

well docuemented

quick to apply

19

needs learning periodWhat should be

improved:

WLSruntime values

not resettable

one sampling interval

per module but

one module per

server only

hot swap

needs in-place update

motivation/introduction

in Oracle

documentation

20

more information: www.munzandmore.com/blog

WebLogic on Amazon andRackspace

Cloudwww.munzandmore.com/book

Mike Cico‘s article:http://www.oracle.com/technetwork/articles/cico-wldf-091073.html

Oracle WLDF docs:http://download.oracle.com/docs/cd/E14571_01/web.1111/e13714/toc.htm

the end

21

questions?