Bottom-up Approach to DB2 Performance Reporting and Tuning

Bottom-up Approach to DB2 Performance Reporting and Tuning

John MaenpaaSenior [email protected]

© YL&A 1999-2003 Bottom Up Approach to DB2 Performance2

Bottom Up Approach to DB2 Performance© Copyright 2003 Yevich, Lawson & Associates, Inc. All Rights Reserved

Yevich, Lawson & Associates. Inc.3900 Wood Duck Dr. Suite DSpringfield, IL 62707

TEL: 1-888-246-5049INT’L: 1-217-698-8142FAX : 1-217-698-8143

[email protected]

Developed by:Dan Luksetich


Disclaimer PLEASE READ THE FOLLOWING NOTICE

The information contained in this presentation is based on techniques, algorithms, and documentation published by the several authors and companies, and in addition is the result of research. It is therefore subject to change at any time without notice or warning. The information contained in this presentation has not been submitted to any formal tests or review and is distributed on an “As is” basis without any warranty, either expressed or implied.

The use of this information or the implementation of any of these techniques is a client responsibility and depends on the client’s ability to evaluate and integrate them into the client’s operational environment.

While each item may have been reviewed for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere.

Clients attempting to adapt these techniques to their own environments do so at their own risks.Foils, handouts, and additional materials distributed as part of this presentation or seminar should be reviewed in their entirety.

This presentation may contain information about software releases that are not yet Generally Available. Material is subject to change. YL&A is not responsible for the accuracy of the information.IBM is not responsible for the accuracy of the information or delivery of unreleased features discussed in this presentation.


Presentation Abstract

This paper will focus on a technique to measure and document the costs of DB2 for z/OS applications, addressing the performance of an application after implementation in a management friendly manner so that performance tuning can be properly "sold" to management. Tuning opportunities are appropriately weighed, and effective breakdown of the problem to specific targets accomplished. The result is an approach that works for management and technicians, and saves millions of dollars in operating costs.

Bottom Up Approach to DB2 Performance© YL&A 1999-2003 5

Introduction – Why is this “Bottom Up”?

■ This is Beyond “Typical” DB2 Performance Tuning■ Flips this “Typical” Model Upside Down

– Not looking at Individual Statements First– System Wide – Application Wide

■ Then We Drill Down■ Identifies Performance Opportunities from DB2 Accounting Trace

Data Instead of SQL Explains■ Provides a Systematic Approach to Organization of Performance

Information■ Communicates Performance Information to Managers in a

Language They can Understand ($$$$$$)■ Automation Relieves the DBA and System Programmer from

Additional Burden


Typical DB2 Application Tuning – Development & Testing

■ Performed During Development– Explain Facility Used to Look for Inefficient Access Paths

■ Tablespace Scans■ Non-Matching Index Scans■ Sorts

– Program Reviews■ Ensure Proper Access to the Database■ Avoid Programmatic Joins■ Avoid Repetitive Database Accesses

■ Performed During Testing– DB2 Online Monitors Utilized

■ Excessive I/O Waits■ Locking Contention■ High CPU Utilization

– May Not Involve Production Sized Volumes– May Not Involve Production Transaction Frequencies


Typical DB2 Application Tuning - Production

■ Tuning Performed After Implementation■ Consists Mostly of “Fire Fighting”■ Problems Not Discovered During

– SQL walk-throughs– Monitoring of Test Runs

■ Once “Fire Fighting” Finished Tuning Ends!


So – What’s Wrong With The Typical Approach

■ Most Managers and Application Developers are Not Interested in Fine Tuning– We Tuned the SQL– We Fought the Fires– Now We’re Done?– But We Could Save Millions ($$$)

■ The Traditional Approach Has Failed■ We Need An Alternative

– Post-Implementation Approach– Need to Measure and Communicate Application Performance

■ To Managers■ To Developers

– We need a “Buy-In” From These People■ Enter the “Bottom-Up” Approach■ We’ll Communicate Application Savings in Dollars!


The Bottom-Up Reporting and Tuning Approach

■ Not a New Technique– Something We May be Doing Every Day

■ Very Effective in Various Situations– When the Perception is “Everything has been Tuned”– There has been No Tuning– A Purchased Product has been Installed– Tuning is Performed Using “Traditional” Methods

■ Explain■ Testing■ Subsystem Parameters

■ This Technique Breaks That Mold– Turns Things “Upside Down”– Statements that are Perceived as “Well Behaved” May Now Show Up as

a Bottleneck


The Approach

■ Define the Applications Running in the System■ Determine the “Cost” of Each Application■ What’s More Expensive?

– The Query the Performs a Tablespace Scan Once Per Day?– The Query That Performs a Matching Scan 50 Million Times a Day?

■ If We Use the Typical Approach– The Statement Doing the Tablespace Scan

■ If We Use This Approach– The Statement That Costs More

■ The Difference– Traditional “Top-Down” Looks at the Worst Things From a Technical

Perspective– “Bottom-Up” Looks at Things From a System Wide Measurement

Driven Perspective


What is This Approach?

■ Addresses System Wide Application Performance■ Utilizes a Batch Reporting System

– We Need DB2 Package Level Reporting– DB2 Accounting Traces 1,2,3,7,8

■ Reports Are Inputs to Spreadsheets and Charts■ We Take Application Summaries■ We Track the Applications

– Daily or Weekly Performance– Changes Over Time

■ We Measure Performance in a Management Friendly Way– $$$$$$$$$$$$$$$


DB2 Accounting Performance Impact

■ The Impact is Minimal, the Benefits Substantial■ DB2 Accounting Traces 1,2,3,7,8 to SMF Increases CPU by 4.3%

Monitor Product Percent Overhead

SMF 4.30%

TMON DB2 -0.42%

Insight for DB2 -1.36%

Omegamon for DB2 10.51%


The Bottom-Up Framework

■ Need to Use Your Monitor or Capacity Planning Product– Produce Package Level Reports– Accounting Trace Long or Summary Reports

■ Information From the Reports is Customized– Some Reporting Products can do This– Or We Do It Ourselves (Like with REXX)– Or a Talented SysProg can do it Alone

■ What We Need From the Accounting Report– DB2 Correlation Name– DB2 Package Name– In-DB2 Elapsed Time at Package Level– In-DB2 CPU Time at Package Level


Application Information is Needed

■ We Have to Know Something About Our Applications■ Naming Standards Help

– Program Name Headers– Packages Map to Programs

What Programs Belong to the Payroll Application?• PAYR001• PAYR002• PAYR003


Now We can Categorize Work by Applications

■ Payroll Application– We’ll Call it Application “A”– It Contains Packages That Begin With “PAYR”

■ PAYR001, PAYR002, PAYR003■ Daily Batch Sales Report Application

– We’ll Call it Application “T”– It Contains Packages

■ PKG001, PKG002, PKG003, PKG004, PKG005■ We Can Now Summarize DB2 Accounting Information

– We Produce a Daily DB2 Accounting Package Summary Report– We Feed That Report into A REXX Routine (Lots of Ways to do This,

However)■ Rules can be Built in a Database

– Any Package That Starts with “PAYR” is Application A– Packages PKG001, PKG002, PKG003, PKG004, PKG005 are Application T

■ Then the Package Level Information is Aggregated by Application


Summarized Accounting Information – First Step

■ Summarized DB2 Accounting Information– Daily Reports are Produced– Reports are Formatted Such That They Can be Fed into Spreadsheet

Software– Daily Spreadsheets and Graphs Published

■ Daily Spreadsheet Data– Combined into Weekly and Monthly Reports– Graphs and Charts Produced– Trend Reports Produced


Summarized Accounting Information

■ The First Step is to Provide Application Level Accounting Information– We Summarize Package Level Information– We Aggregate According to Naming Standards

■ Application “T” (Batch Sales Report Application) is Interesting!– If Our Concern is CPU Consumption!


Interesting Things on Our Summary Report

■ What are the Really Interesting Applications in Our Report?

■ Daily CPU Consumption– T Consumes over 7 hours per day– W Consumes over 3 hours per day– N consumes over 2 hours per day

■ No other application consumes nearly this much CPU■ If we want to save CPU, we focus on these three applications


■ What are the Really Interesting Applications in Our Report?

■ CPU Consumed per SQL Statement– Application T Consumes about .5ms CPU per Statement– Application W Consumes less than .4ms CPU per Statement– Application N Consumes about .5ms CPU per Statement

■ Application T provides the best tuning opportunity– Total CPU (by far) and CPU per statement

Interesting Things on Our Summary Report


Drill Down to Application of Interest – Second Step

■ Application “T” is of Further Interest■ We Can Now Drill Down to the Packages (Programs) That

Application “T” Utilizes

■ Packages PKG002 and PKG003 are interesting!– Largest consumers of CPU


Program Analysis – Third Step

■ The Obvious High Consumer of Resources is Application “T”– Within Application “T”, the High Consumer is Package PKG002– This is Where We Initially Focus Our Attention

■ This is Where We Need Application Development “Buy-In”– We Need A Team Approach

■ We Need to Drill Down to the Program– Which DB2 Statements are Executing?– How Often Do They Execute?– How Much CPU Does Each Statement Consume?

■ There Are Tools To Help US– DB2 Performance Trace– Online Monitor Tools– Batch Analysis Tools– DB2 Explain– Bottom Line; We need Statement Level Information at This Point


Program Analysis – The Process

■ You Can First Try EXPLAINing All the Statements in a High-Consuming Package– For Example: REBIND PKG002 EXPLAIN(YES)– Provides a Non-Intrusive Way of Evaluating Each Statement– You May Catch Some Bad Access Paths That Were Previously Missed,

or Changed Without Notice■ A Short Execution of a DB2 Performance Trace May Provide Ample

Statement Level Information– Determine When PKG002 is Executed– Start a Performance Trace During That Time– Run the Trace for a Short Time– Gather the Statement Level Information

■ Address the Highest Consuming Statements■ Or the Most Frequent Statements ■ Tune Those Statements First■ Then Repeat the Reporting/Tuning Process


Continuing The Reporting

■ We Need to Maintain a System Perspective■ Once We Make A Change

– Did it Work?– Did it Affect Anything Else

■ Continual Reporting is a Must– We Measure Changes by Application

Performance Report

To Determine What to Tune

Performance Report

To Monitor the Impact to the Change Across All

Applications

Produce a Report

Implement a Tuning Change

Repeat


Delivering Reports That “Matter”

■ It’s Critical to Deliver Reports That are Understood!– Difference Between Success and Failure

■ We Need to Deliver High Level Information to Managers– Summarized “Cost” Information– CPU Time– Number of Statements

■ We Import the Reports Into Spreadsheets– Graphs for Visual Presentations– Cost Factors are Always Important– We Show Data in Dollars


Reporting CPU Costs in Dollars

■ Most Shops Have a Formula for Converting CPU Seconds into Dollars

■ Build This Information into The Reports– Add the Formula to the Spreadsheets– Convert the Spreadsheets into Charts– The Charts Display Consumption of Resources in Dollars

■ Each Time a Tuning Change is Made– Create a Delta (Before/After) Report– Show the Difference in a Chart– The Measurements in the Chart are in Dollars

■ Why Use This Technique– It Gets Managers Attention– It attaches Money to Costs and Tuning Efforts– It Helps Build Support for Proper Performance Analysis and Tuning

■ The “Savings” Support the “Costs” of the Effort


Delivering Reports That “Matter”

■ Example: We Take Our Summarized Performance Reports– Translated into Dollars– In This Example; $120 per CPU Hour

C D E M N T W$0

$50,000

$100,000

$150,000

$200,000

$250,000

$300,000

$350,000

Yearly Cost


Summary

■ Typical DB2 Performance Tuning begins (and Ends) During Design– It’s On a Statement-by-Statement Basis– This is Great, But…

■ DB2 Performance Tuning Needs an Enterprise Approach– We Look at Application Performance– It’s A System Wide Perspective– We Track Performance Over Time

■ Proper Reporting is Key– Money Base Reports Sell Tuning to Management– Charts are Better Than Stats

■ This Technique Works for any Application– In-house– Purchased– BECAUSE – We Look at System Level Stats


Trademarks

■ IBM– is a registered trademark of International Business Machines

Corporation.■ DB2

– is a trademark of IBM Corp.


Useful IBM Websites

■ DB2 Electronic Support– http://www.ibm.com/software/db2zos

■ FAQs– http://www.ibm.com/software/data/db2/os390/index.html

■ Presentations:– http://www.ibm.com/software/data/db2/os390/presentations.html

■ Data Sharing– http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/

Flash10194■ News: DB2 for z/OS

– http://www.ibm.com/software/data/db2/os390/techdocs2/z01.pdf


Useful IBM Websites

■ Overview of V8– http://www.ibm.com/software/data/db2/os390/techdocs/DMDB12.pdf– ftp://ftp.software.ibm.com/software/db2storedprocedure/db2zos390/

techdocs/db2vnext.pdf■ Introduction to V8 Catalog Changes

– ftp://ftp.software.ibm.com/software/db2storedprocedure/db2zos390/techdocs/vnextcatalog.pdf

■ DB2 Developer Domain articles– http://www7b.boulder.ibm.com/dmdd/– http://www7b.boulder.ibm.com/dmdd/library/techarticle/0209cotner/

0209cotner.html– http://www7b.boulder.ibm.com/dmdd/library/techarticle/0206miller/

0206miller.html


DB2 Books by YL&A

DB2 High PerformanceDesign and Tuning

Prentice Hall, Aug, 2000Richard Yevich and Susan Lawson


DB2 Books by YL&A

DB2 for OS/390 Version 7DBA Certification Guide

Prentice-Hall, June 2001Richard Yevich andSusan Lawson


DB2 Books by YL&A■DB2 UDB for OS/390 V7.1Application Certification Guide

Prentice-Hall Dec 2002Susan Lawson


Courses by YL&A

■ Transition– V6 to V7, V5 to V7 (z/OS), V7 to V8 (LUW)– For applications only, for DBA only, or both– V8 Introduction (1 or 2 day)

■ SQL (z/OS and LUW)– Basic SQL– Advanced and Complex SQL– SQL Performance Tuning and Optimization

We customize all classes based upon customer requirements


Courses by YL&A

■ Application Development– Stored Procedure Development and Implementation– UDFs and Triggers Development

■ Database Design– Physical Design, Logical Design

■ Data Sharing– Implementation, Performance, Recovery (or all topics)

■ High Performance Design and Tuning– Application, Database, Systems

Taught by skilled instructors world-wide!


Cost Avoidance Through Performance Audits

■ DB2 Performance Audits– Existing or new database designs and applications– Certification of design and implementation acceptance– Evaluation of all the performance ‘points’ in a DB2 environment

■ Physical Design■ Subsystem■ Application Code and SQL

Specific to your needs!


Cost Avoidance Through Performance Audits

■ DB2 Performance Audits – Help with bringing legacy application to an e-business environment –

the rules have changed! ■ What was acceptable performance in the past is NOT acceptable in an e-

business environment– Experienced in ‘fighting fires’ – many performance problems do not

become reality until production– Results: problems identified, solutions proposed (many implemented

immediately), continual knowledge transfer during the process

38© YL&A 1999-2003 Bottom Up Approach to DB2 Performance

Thank You

Documents

Bottom-up Approach to DB2 Performance Reporting and Tuning