Performance Tuning Oracle Weblogic Server 12c

Follow: @UKOUG

UKOUG Applications Conference & Exhibition 2015

7th – 9th December, ICC Birmingham

Follow: @UKOUG

Introduction

A standard tuning framework

OS tuning

Monitor & tune JVM

Monitor & tune server

Agenda

Follow: @UKOUG

Ajith Narayanan

11+ Years of work experience as Oracle [APPS] DBA

Speaker @ Oracle conferences AIOUG-SANGAM, NZOUG, AIOUG-

TECHDAY, DOAG, OTNYathra, UKOUG, OTN APAC tour.

Blogger since 2008 http://oracledbascriptsfromajith.blogspot.com

Oracle RACSIG http://www.oracleracsig.org

Website Chair (2011 - 2013) :

Web Seminar Chair (From 2015)

Introduction

Follow: @UKOUG

Terminology

Performance Testing

Benchmarking

Why Is Testing Important

Performance Tuning Methodology

Testing Tools

WL Performance Tuning

Follow: @UKOUG

Performance of a system is majorly characterized by

Response Time(The time taken for the server to deliver a webpage)

Latency(The time between the issuing of a request and the actual work beginning on that request.)

Throughput(Request per second)

Resource utilization(e.g. CPU usage %)

Terminology

Follow: @UKOUG

Functional Test – Specific application or application unit is performing

Stress Test –Type of load test to determine system limits

Load Test – Number of concurrent requests

Soak test – A load test running for long period of time.

Performance Testing

Follow: @UKOUG

A benchmark is taken for:-

Measuring system performance

Comparing the system performance at different scenarios

Application on which you are testing

Create a baseline performance measurement

Tip:- To determine performance objects gather information about the levels of

workload on the application

Number of concurrent users

Number & size of requests

Amount of data & concurrency

Target

Benchmarking

Follow: @UKOUG

More rigor on the testing process reveals the potential issues and outages due to the changes

People with the tough testing process in place are seen the happiest people

Good performance depends on good design, good implementation, defined performance objectives and performance tuning.

Why Is Testing Important

Follow: @UKOUG

Perfomance Testing Methodology

Follow: @UKOUG

Major Performance Testing Concerns are:-

Testing without real network would give us false measures.

Low user simulations can be different from high user simulations.

Network throughput may be larger than the deployed environment.

Non-persistent message performance is dependent on CPU & RAM.

Persistent message performance is dependent on the disk speed.

Performance Testing Methodology

Follow: @UKOUG

Bottlenecks:-


Follow: @UKOUG


CPU Bound:-

Follow: @UKOUG


I/O Bound:-

� The I/O bound case has the following properties:

� CPU Utilization is not at maximum and still has headroom for throwing away CPU cycles to processes.

� Performance remains same regardless the change or increase in workload

� The disk, network or database is the main culprit.

Follow: @UKOUG


Disk bound:-

Follow: @UKOUG


Network bound:-

Follow: @UKOUG


Database bound:-

Follow: @UKOUG

Testing Tools

Many commercial and open-source load testing tools are available in market, few of them as listed below.

• Oracle Application Testing Suite (OATS)

• The Grinder

• Apache Jmeter

• HP Loadrunner

• RadView WebLOAD

Follow: @UKOUG

Testing Tools

How to choose a testing tool?

• Installation & Configuration

• Deployment for distributed testing

• Reporting

• Accuracy of results

• Cost

• Learning

• Other features

Follow: @UKOUG

Testing Tools

The Grinder:-

• Grinder is a Java load-testing framework

• Easy to run a distributed test using load-injector machines

• Freely available under BSD, open source license

• Based on open source technologies like Jython, HTTPClient, XMLBeans, and PicoContainer.

• Grinder makes it easy to coordinate and monitor activity of processes acoss a network of many load-injection machines from a central console.

Follow: @UKOUG

Testing Tools

The Grinder Architecture:-

• Worker processes interpret Jython test scripts and perform tests using a number of worker threads.

• Agent processes manage worker processes.

• The console coordinated the other processes and collates and displays resulting statistics.

• Grinder is written Java and each process is a JVM.

Follow: @UKOUG

Testing Tools

The Grinder Console:-

Follow: @UKOUG

Testing Tools

The Weblogic Admin Console:-

Follow: @UKOUG

Performance metrics for CPU, IO, Memory & Network Subsystem

Monitoring Tools provided by OS.

Parameters to improve performance

OS Tuning

Follow: @UKOUG

Linux/Unix Performance Metrics

CPU metrics

Memory metrics

Network interface metrics

I/O device metrics

Performance monitoring tools

-top, sar, vmstat, ps, iostat, netstat, System Monitor

OS parameters that affect WebLogic performance


Follow: @UKOUG


Processor Metrics

Follow: @UKOUG

Memory Metrics


Follow: @UKOUG

Network Interface Metrics


Follow: @UKOUG


I/O Device Metrics

Follow: @UKOUG

OS Tools


Follow: @UKOUG

OS Tools


Follow: @UKOUG

OS Parameters affecting WebLogic performance

• The file-max parameter – Max number of file handles that the Linux kernel will allocate.

• Parameter = fs.file-max=65535 (in /etc/sysctl.conf)

• To support large number of network connections tune the TCP/IP parameters

• Find default wait time (Usually high), using netstat –a | grep TIME_WAIT |wc –l

• To check the current valuecat /proc/sys/net/ipv4/tcp_keepalive_time and

cat /proc/sys/net/ipv4/tcp_keepalive_intv1


Follow: @UKOUG

OS Parameters affecting WebLogic performance

• Heavily loaded WebLogic or many clients with bad connections with high latency results in an increase half-open connections.

net.ipv4.tcp_max_syn_backlog=4096. Default is 1024

• Servers that needs many connections at the same time, the TIME-WAIT sockets for new connections should be reused.

Net.ipv4.tcp_tw_reuse=1

Net.ipv4.tcp_tw_recycle=1


Follow: @UKOUG

JVM & Java Garbage Collection

Command-Line JVM Tools

GUI JVM Tools

HotSpot JVM Monitoring Tools

Follow: @UKOUG



Follow: @UKOUG



Follow: @UKOUG

GC Tuning Tips

Heap Size influences:

� GC Frequency and the pause duration

� Number of short & long-term objects

� Fragmentation

Tips� An undersized heap woth the concurrent collector leads to full GCs with an increase

in load and fragmentation problems

� An oversized heap leads to increased collection times

� Size the heap to handle peak and burst loads

� Increase memory as you increase the number of CPUs

� Sizing the permanent generation is important for applications that dynamically

generate and load many classes.’

� Setting PermSize and Max PermSize to the same value is recommended.


Follow: @UKOUG

GC Tuning Tips

Tips contd..

� Avoid using finalizations in application, it causes lags in GC activity

� The concurrent collector advantages increase with the number of CPUs

� Use intimate Shared memory(ISM) and variable page sizes to reduce smear problem

where available -XX: +UseISM

Note:- This is a Solaris only feature (ISM)

� For larger applications use throughput collector -XX:+UseParallelGC


Follow: @UKOUG

GC Tuning Tips

Tips contd..

� Setting PRODUCTION_MODE

to true causes weblogic to

start using –server mode.

On server-class machines

o Server JIT compiler

o Throughout garbage collector

o Heap sizes

o Initial heap size of 1/64 of physical

o Memory upto 1GB

o Maximum heap size of ¼ of physical

o memory upto 1 GB


Follow: @UKOUG

Command line JVM tools

� jps:- Command line utility to find running JVM processes� Included in the HotSpot JDK

� Capable of local and remote monitoring.

� Very similar to ps command in unix.

$jps [-q] [mlvV] [<hostid> where <hostid> = <hostname>[::<port>]

� jcmd:- Utility to send diagnostics command to running JVM$jcmd <pid> command

� jinfo:- prints java configuration info for a given java process$jinfo <pid>

� jstat:- Runs in local or remote JVM & included in HotSpot JDK$jstat -<option> [-t] [-h<lines>] <vmid> [<internal> [<count>]]


Follow: @UKOUG

Command line JVM tools

� jstack:- Helps in printing a stack trace of all the threads currently running in a VM

� Prints a thread stack for a VM running remotely

� Prints a thread stack from trace file

$ jstack [option] pid |executable core|[sever-id@]remote-hostname-or-ip

GUI JVM tools

� Java VisualVM

� VisualGC

� jconsole

� GCHisto

� Mission Control

� JVM Browser

� Jrockit Flight Recorder


Follow: @UKOUG

Monitor & Tune Server Performance

Configure domain & server parameters to improve performance

Configure the server to handle struck threads

Setup connection backlog buffering

Follow: @UKOUG

connect()

edit()

startEdit()

cmo.setInternalAppsDeployOnDemandEnabled(true)

activate()


Configure domain & server parameters to improve performance� Set the weblogic server internal applications to start up when first accessed (on demand)

Configure the server to handle struck threads� Threads gets struck in case of infinite computational loop or database calls resulting in a deadlock

situation. We should be able handle the struck threads for improving performance.

Navigate to admin console -> Configuration -> Tuning tab

� Struck Thread Max Time

� Struck Thread Count

� Struck Thread Timer Interval

Follow: @UKOUG

Setup connection backlog buffering

Navigate to admin console -> Configuration -> Tuning tab

� Accept Backlog

� Login Timeout

� SSL Login Timeout

To address connection backlog, We should tune TCP parameters at the OS level

In Linux

/sbin/ifconfig lo mtu

kernel.msgmni

kernel.sem

kernel.shmmax

fs.file-max

net.ipv4.tcp_max_syn_backlog


Follow: @UKOUG

Thank You!

Contact Me:-

[email protected]

About UKOUG

UKOUG exists to serve the Oracle

community in the UK and Ireland. We act

as a focal point for sharing knowledge

about Oracle applications, technologies,

tools and developments and, as an

independent, not for profit membership

organisation, represent the views of

users and partners to Oracle.

© Copyright 2015 UKOUG

Contact

User Group House

591-593 Kingston Road

Wimbledon

London

SW20 8SA

T: +44 (0)20 8545 9670

W: www.ukoug.org

Documents

Performance Tuning Oracle Weblogic Server 12c