www.macro4.com

Application Performance Tuning What’s Happening Under The Hood?

Morrie Meyer

Macro 4, Inc.

www.macro4.com

Agenda - Application Performance

• Why applications matter

• Why application Performance matters

• How’s it different from general system tuning?

• Who’s job is it?

• Tooling up− Application Performance Measurement (APM) software.

• How to go about it …

www.macro4.com

Why applications matter• “Today’s enterprise is dependent on the superlative operation of any

number of applications across its entire integrated IT infrastructure. Non-availability or poor application performance has significant consequences in today’s competitive 24x7 global market place, impacting efficient transaction management and mission-critical operations”

Butler Group, May 2005

• “In the network economy, the website becomes a company's primary interface to the customer. Indeed, for e-commerce companies the site is the company”

• [Applications make the website work]Jakob Nielsen(Leading Usability Researcher)

www.macro4.com

Why application performance matters

• There may simply be hard and fast response time requirements for critical applications − e.g. production line control application

• More commonly, response time requirements are less rigid.

• However ...− "Every usability study I have conducted shows the same thing: users beg us to speed up

response times ... fast (and predictable) response times are the most important design criteria." - Jakob Nielsen

− "Because users prefer sites with high usability, sheer Design Darwinism will ultimately lead to the survival of the most usable sites. High usability means more users and more sales. [This is backed-up by research]" - Jakob Nielson

www.macro4.com

Why application performance matters

• Performance is also about resource usage

• Inefficient, resource hungry applications make excessive demands on hardware and IT staff− Extra MIPS

− Extra upgrades

− Extra servers

− Extra memory

− Extra disk

www.macro4.com

Poorly performing applications cost money

• Lost revenue and customers− Retail Bank: “Every hour of downtime [on our website] impacts $19 million in sales”− Online Insurance Company: “Our competition is just a mouse click away – we can’t make our

customers wait”

• Lower staff productivity− Call Center Rep: “If the application performed more quickly, I could handle more calls (and our

customers would not have to wait so long on the phone)”

• Increased IT costs− Excessive hardware costs – and the costs of staff to manage it− Associated energy costs

» “Greening of IT set to become a major issue” - Gartner» IT shops are often limited by local supply capacity

• Damage to reputation and brand− Immeasurable!!!!!!!!!

www.macro4.com

• What do managers and senior leaders view as the primary cause of downtime?

61% - Applications

» Increased to 80% in organizations with higher use of home-grown applications

21% - Hardware

19% - Don’t know

Survey of more than 200 U.S. IT managers and senior leaders conducted by Managed Objects

Causes of downtime

www.macro4.com

Application performance pain points

• “Detection of application performance problems is too slow (and worse, done by our customers)”− Retail Bank: “Our hardware is fine, we have plenty of tools to monitor that, but the application

performance still sucks”− “We test everything in development and preproduction but it still misbehaves in production”

• “Our problem resolution times are too long”− Pinpointing the problem is slow and involves many people− Retail Bank: "On average it takes 2.5 weeks to diagnose performance issues in [our] multi-tier

application environments“

• “Our Server farm and power bills are growing rapidly”

www.macro4.com

Save Resources Improve Performance

• Use less CPU− Avoid CPU upgrades

» potentially large savings but irregular and difficult to quantify

− Reduce Software Costs through WLC» smaller, but immediate and continuous savings

− Reduce energy usage» Potentially lower number of servers

• Speed things up− Relieve operational stresses/constraints

» Fit shrinking ‘overnight’ batch windows

− Improve response times» Increase throughput» Meet service level agreements

www.macro4.com

Application vs. System TuningHow’s It Different?

• You’ll need to take an application centric view− What service does the application get from the system …− ... rather than what an application does to the system as it executes

• System monitors, SMF records etc.− Can help you identify application processes in need of tuning …

... but can’t show you what’s happening (in any detail) as they execute

• You need to look ‘under the hood’ of the application to focus your performance tuning efforts

www.macro4.com

Whose job is it to do the ‘tuning’ ?

• Systems/Operations point of view− ‘It’s not our job; we’re responsible for tuning the system. We do a good job

of that. Applications are the responsibility of the developers.’

• System Developers point of view− ‘The team responsible for developing that application doesn’t exist anymore.

Many of the developers have left and the others are now busy working on the next thing. There’s no time allocated for application tuning. The system works OK anyway’

• Problems?− Different management styles – process versus project

− Fear and uncertainty

www.macro4.com

Behind the Systems Point of View

• We can (generally) tell which applications are behaving badly (and causing us problems operationally) but …

• We can’t do anything about application performance issues− no application knowledge, no support from development, some apps are packages− no way of seeing what is happening (from an application point of view) as the application

runs− … and it’s not our job anyway

• We do nothing about application performance and adopt this defensive position because of− Fear of being accountable for a job we can’t do (and often goes unrecognized)− Frustration at not being able to deal with application performance issues− We don’t get any recognition for fixing application problems

www.macro4.com

Behind the Development Point of View

• We’re working on other projects now.− No resources for application tuning

• The people who wrote that are no longer around.− Only the person who wrote it will be able to fix it.

• It works according to spec. What’s wrong with it?− Performance isn’t part of the development process. Is it?

• When you say performance problem, can you give me some idea what you mean?− I’m going to need some help if we’re going to give this a try

www.macro4.com

Bringing the Parties Together

• Recognize the role− a joint responsibility where both systems and development have a role to play

− there are benefits to be had and money to be saved

− allocate responsibilities and set some targets

• Allocate some resources (and money)− only need to be minimal to start

− try to encourage co-operation (rather than confrontation) as far as application tuning is concerned

• Support the role− follow progress

− report on, publicize and praise successes (give credit where credit is due)

www.macro4.com

But How?

www.macro4.com

Tooling Up

An Application Performance Measurement (APM) tool provides myriad benefits such as:

• A view from the application’s point of view− of what is going on as it runs

− without an APM the hood stays shut!

• Helps target your application tuning efforts− to make best use of limited resources

• Can be used to judge progress− using before and after ‘images’

• The only way to effectively ‘audit’ your tuning efforts− otherwise the patient either gets better or dies!

www.macro4.com

APM Tools

• Application Performance Measurement (APM) Tools − what do they do?

− how do they do it?

• Sampling is the key− measurement (profiling) of jobs, transactions, etc.

− minimal intrusion and impact on what is being observed

− minimal overhead

− all available Application Performance Measurement tools use a sampling approach

www.macro4.com

What Information is Collected – z/OS

• PSW information− Shows whether job is active (using CPU) or not (WAITing)

− Provides current processing location (next instruction address)

• Load module and ESD map− Used to convert addresses to program names and displacements

• Other basic information− File/DASD I/O information, memory usage, etc.

• Extra subsystem specific information− For CICS, IMS, DB2, MQSeries, Etc.

www.macro4.com

What Information is Collected – Cross Platform

• Application level− Resource usage via MBeans and JMX

− JDBC related data

− Performance data for servlets, JSP’s and EJB’s

• Method level− CPU usage and elapsed time

− Call Stack data

• DB2 on z/OS− SQL activity

− Correlation to JDBC calls

• z/OS− SMF record 120 (Webspehere Application Server)

www.macro4.com

What Happens To This Data?

• It’s analyzed and formatted− statistical− beware of the normal dangers of statistics!

• You get reports and views− giving profiles of CPU usage and WAIT time distribution− possibly with source program mapping (z/OS)− and other reports

» of core system component - DASD, memory usage, etc.» for ‘subsystem’ activity - DB2, IMS, CICS, MQSeries, etc.

• All from an application rather than a system point of view

• A dynamic view− A true ‘dynamic’ picture of where CPU is used, and time lost, when your

application runs. − Focuses on things that can be changed to improve application performance.

www.macro4.com

Moving Forward

• No organization has ‘lots’ of time for application tuning − not budgeted, not a recognized role, a small ongoing commitment if you are lucky

• Every organization will have to do application tuning− User pressure, operational constraints, to meet service levels

• Make sure you target your tuning efforts …… and use your limited tuning resources efficiently

How?

www.macro4.com

How to Use APM Software?

• Don’t fix it if it isn’t broken− don’t start tuning until you’ve isolated the cause of a performance problem

− use APM software to show where CPU is really used and where WAIT times occur

• Profile with an APM tool to get a dynamic view − bad code won’t cause a performance problem unless it’s used (a lot)

− good code may be enhanced to further improve performance

− expect the unexpected

• Profile again after tuning− to check for ‘expected’ results …

− ... and establish a new baseline

− and claim your praise!

www.macro4.com

Look for Quick Wins

• Fixing a problem is going to take much longer if it involves source code changes− and some products are packages; with limited access to the source

• What are Quick Wins?− Changes that can be made without changing the source code

• Examples− Buffering Values

− Some DB2 changes, such as adding an index

www.macro4.com

• Questions

• Comments

• Thank you