Embed Size (px)
Citation preview
High Speed Apply Engine Unleashed!Steve SchulzState Farm Insurance CompaniesSession Code: E12May 5, 2010 2:45 pm | Platform: DB2 for z/OS
Agenda:
o High-speed Apply Engine Basics: What is it? Features, options, and parameters
o Leveraging High-speed Apply Engine: Suggested applications Three case studies
o Lessons Learned and Best Practices Design and tuning of the prototype job SQL Generation techniques Considerations for multiple parallel agents
High-speed Apply Engine Basics
What is it?
o Part of BMC Software’s utility solutions: Database Administration for DB2 Recovery Management for DB2
o Runs on DB2 for Z/OS and other platforms
o Formerly known as Apply Plus
High-speed Apply Engine Basics
When used as a component of other BMC utilities:
o Provides LOADPLUS for DB2 the ability to use highly concurrent inserts to populate tables – known as SQLAPPLY Mode
o Provides Log Master for DB2 the ability to execute generated SQL’s for MIGRATE, UNDO, or REDO processing
High-speed Apply Engine Basics
When used as standalone utility:
o An alternative to writing programs for batch applications needed to modify data in tables (inserts, updates, or deletes)
o An alternative to using DSNTEP2 or similar methods to handle mass inserts, updates, or deletes in batch mode
High-speed Apply Engine Basics
Features include:
o High availability – target tables remain online (read/write) while changes are applied
o Multi-threaded distribution of work – multiple agents can process balanced workload in parallel
o Control over commit frequency – Based on time limit, statement count, or both
o Restart processing – continue from point of failure
High-speed Apply Engine Basics
Targets for an Apply request can include:
o An individual table
o A group of tables which may or may not have RI defined on them
o A series of partitions (subset) of a table – can be logical or physical partitions
High-speed Apply Engine Basics
Accepts the following inputs:
o SQL Statements (generated from any source): DML: INSERT, UPDATE, DELETE DDL: CREATE, ALTER, DROP etc. but not SELECT statements
o Logical log files (created by Log Master for DB2): MIGRATE, UNDO or REDO options
High-speed Apply Engine Basics
SQL statements used as input may be:
o Manually coded – when few statements or rows affected
o System generated – when there are a large volume of SQL’s or rows impacted. Some techniques include: Unloading keys from target or staging table SYNCSORT, SAS, or REXX Routine to generate or clone
SQL for multiple target tables Custom program when above methods are not an option
High-speed Apply Engine Basics
Rules for SQL statements used as input:
o Statement must be syntactically valid
o Only one statement per line
o A statement may span multiple input lines
o Large statements are supported (up to 320K)
o Commits do not have to be included in every case
High-speed Apply Engine Basics
DB2 V9 features supported:
o XML Objectso TRUNCATE TABLE statemento Clone tableso New data types (ie BIGINT, BINARY, VARBINARY, DECFLOAT)o Partition by growth
Other recent enhancements:
o Multi-row insert operationso Large Object (LOB) support
High-speed Apply Engine Basics
Apply requests for DB2 on Z/OS execute via batch JCL
//APPLY10 EXEC PGM=APTBMAIN,PARM='SSID=xxxx' //*
//SQLIN DD DSN=HLQUAL.MLQUAL.SQLIN(+1),DISP=SHR SQL Statements here//APTERR DD SYSOUT=*
//APTPRINT DD SYSOUT=*
//APTDUMP DD SYSOUT=*
//APTLOG DD SYSOUT=*,LRECL=550,RECFM=VB
//SYSPRINT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//SYSTERM DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//*
//SYSIN DD * Configuration parameters follow
High-speed Apply Engine BasicsHow an apply request is processed:
Request Processor
Agent #2
Agent #1
Agent #3
Agent #10
Input Source
Input Processor
:Summary Report
Target DB2 Table(s):
T1
T2T3
ConfigParms
Conflict Manager
High-speed Apply Engine Basics
Custom Settings:
o Provide control over distribution of work
o Control degree of static versus dynamic SQL
o Program response to specific SQL return codes
o Allow logical partitioning of non-partitioned tables
o Allow translation of table or column names
High-speed Apply Engine Basics
Options for Distribution of Work:
o Distribution by Object
Uses same agent to handle all SQL’s affecting the same table or partition, regardless of UR Provides integrity for overall request but not
guaranteed during execution Determines when commits should happen Allows optimal use of multiple parallel agents for
faster performance
High-speed Apply Engine Basics
Options for Distribution of Work:
o Distribution by Unit of Recovery (UR)
Uses same agent to handle all SQL’s in an original transaction, regardless of which tables are affected Preserves integrity throughout the Apply request Honors original COMMIT or ROLLBACK statements Performance can be slower when dependencies
exist between transactions – causes agents to wait
High-speed Apply Engine BasicsConfiguration Parameters
/COMMITTRIGGERS/ COMMITONDEMAND=NO Utility ignores commits in the scriptTIME=0010 Commit after 10 seconds orSTATEMENTCOUNT=1000 Commit after 1000 statements/AGENT/ MAXAGENTS=5 Maximum of five parallel agents /DISTRIBUTIONTUNING/ RICLUSTERING=NO Don’t restrict agents to preserve RIURCHECKRI=NO Don’t perform RI checks/BINDTUNING/STATEMENTCOUNT=1 Use statically bound SQL throughout# ; -- These are all valid comment delimiters
High-speed Apply Engine Basics
Options for Conflict Resolution:
The following actions can be taken if an exception occurs:
o Continue - Ignores the conflict and moves ono Defer the statement until the process is completeo Retry the SQL statement a set number of timeso Terminate the process until conflict is resolved
Statements resulting in Defer or Terminate actions will result in an implied commit
High-speed Apply Engine BasicsConflict Rules
Config. parms can be used to set rules on how the utility reacts to warning or error conditions (referred to as “conflicts”)
/CONFLICT/ RETRYLIMIT=COUNT Limit based on number of attemptsRETRYVALUE=5 RETRYFAIL=TERMINATE Abend job if the limit is reached/ANYCONFLICT/CODE=NEGATIVE Handle negative SQL Return Codes by writing the failed ACTION=DEFERSTATEMENT statement to a conflict file and proceeding to next statementCODE=TIMEOUTACTION=RETRY Handle timeouts using the retry rules above/CONFLICTFILE/ Defines a data set to save failed statementsFILENAMEMODEL=HLQUAL.MLQUAL.ERROR.DEFR(+1)SINGLEFILE=YES One file for all Apply Agents
High-speed Apply Engine BasicsRestart Parameters
o In the event of a failure during the process, information is logged in a Restart Table to allow an automated restart
o Input SQL file and config parameters should not be altered for a restart. Only add the Restart parameters:
//SYSIN DD * /RESTART/ TABLENAME=BMCAPT_APTRSCL Restart table nameRESTARTTYPE=RESTART Default is NEWRESTARTID=BRJN141090925816 From APTPRINT of failed run
High-speed Apply Engine BasicsOutput in APTPRINT File15:04:54 001 BMCAPT0181I Database Administration for DB2 Solution V04.03.00 15:04:54 001 BMCAPT0238I Successful connect to IC3A (Release 910(NFM) of DB2) 15:04:54 001 BMCAPT0097I Request ID: ABGX Restart ID: ABGX551111502404 15:04:55 006 BMCAPT0079I High-speed Apply processing started - Agent: 115:04:55 007 BMCAPT0079I High-speed Apply processing started - Agent: 215:04:55 008 BMCAPT0079I High-speed Apply processing started - Agent: 3 15:04:56 009 BMCAPT0079I High-speed Apply processing started - Agent: 4 15:04:56 010 BMCAPT0079I High-speed Apply processing started - Agent: 5 15:04:59 001 BMCAPT0185I Statements committed: 6000 15:05:02 001 BMCAPT0185I Statements committed: 1100015:05:05 001 BMCAPT0118I Input statements processed: 131751 Statement ID:15:05:06 001 BMCAPT0185I Statements committed: 17000 15:05:10 001 BMCAPT0185I Statements committed: 25000 15:05:12 001 BMCAPT0210I Updating Checkpoint Statement ID: 5000
High-speed Apply Engine BasicsOutput in APTPRINT FileBMCAPT0234I ROW STATISTICS
AGENT INSERT UPDATE DELETE MASSDEL TOTAL 1 0 0 245087 0 245087 2 0 0 245097 0 245097 3 0 0 241924 0 241924 4 0 0 245500 0 245500 5 0 0 245850 0 245850
----- --------- --------- --------- --------- ----------TOTAL 0 0 1223458 0 1223458
BMCAPT0206I PACKAGE STATISTICS NAME STMTCNT USAGECNT RETRIES BIND ELAPSED TIME
ABGX0000 101 0 0 00:00:00.753410ABGX0001 1 1223458 0 00:00:00.417018--------- --------- --------- --------- -----------------
TOTAL 102 1223458 0 00:00:01.170429BMCAPT0183I Request elapsed time: 00:06:20.757732
Leveraging High-speed Apply Engine
o Suggested applications
o Three case studies
Leveraging High-speed Apply Engine
An option for batch processes making high volume changesto DB2 tables, such as purge, archival, and mass corrections
These processes:
o Execute large volumes of SQL’s (inserts, updates, or deletes) – OR –
o Might otherwise be handled by one or more SQL’s with broad selection criteria (Update, Delete, or Insert from Select)
Leveraging High-speed Apply Engine
The statement below is an example of the “big bang” type, which affects a large percentage of rows in a table
DELETE FROM OWNR.PLCY_DETL_TB WHERE DATE(INSRT_TSTMP) <= LAST_DAY(CURRENT DATE - 24 MONTHS)
OR (DLTE_TSTMP = ‘Y’AND DATE(UPDT_TSTMP) <= CURRENT DATE – 7 DAYS)
;Commit;
If the table is large enough, it may result in Uncommitted UR or lock escalation warnings, due to a lack of commit frequency
Statements like this are frequently executed through batch SPUFI
Leveraging High-speed Apply Engine
An option for data conversions or migrations:
o Can load target tables with mass volumes of data without need for shadow copies or an outage
o Table and column names may be translated from source to target
o Data types may not be transformed o Not an ETL tool, but may be part of ETL solution
Leveraging High-speed Apply Engine
Most common application uses at State Farm :
o Regular purges of data from “live” or “work” tables (nightly, weekly, monthly …)
o Ad hoc data corrections where large numbers of rows need to be updated or deleted
o Major conversions of data requiring SQL updates
Leveraging High-speed Apply Engine
Three Case Studies:
#1: Refresh of daily snapshot table
#2: Monthly update of a large table
#3: Test environment data clean-up
Case study #1: Refresh of daily snapshot table
Needed to Delete expired rows from snapshot tables each day to be replaced with current rows
Initial Approach was to develop custom COBOL programs:
o First COBOL program averaged 48,000 deletes per minuteo At least 50 hours of Developer time needed per programo Similar programs were needed for 4 other sets of tables
The team asked their DBA if there was better way. . .
Case study #1: Refresh of daily snapshot table
New solution for deleting expired rows:
o We became aware of the High-speed Apply Engine
o Researched this option and developed a prototype job
o Benchmarks in development environment were promising
o Developers easily cloned the prototype JCL and parameters
o Eliminated the need to write four application programs
Case study #1: Refresh of daily snapshot tableComparison benchmarks for delete process, after tuning:
2.8
56
4.47
50
2.414
3.77
165
0
20
40
60
80
100
120
140
160
180
M Rows Deleted Elapsed Minutes CPU Minutes K Rows / Minute
Batch COBOL High-speed Apply Engine
Case study #2: Mass update of a large table
o A column was added to a larger Customer table for queries
o The column value needed to be refreshed monthly for every row on the table
o A PL/1 program was implemented to update the table
o Batch job took nearly 14 hours to update 103 million rows
Case study #2: Mass update of a large tableComparison benchmarks for mass updates:
103
689
68123103
212
98
484
0
100
200
300
400
500
600
700
800
M Rows Updated Elapsed Minutes CPU Minutes K Rows / Minute
Batch PL/I High-speed Apply Engine
Case study #3: Test environment data cleanup
o In preparation for a performance test, a large percent of rows needed to be end-dated (“marked complete”) on some tables
o The Test Analyst used a Silk script to execute the mass updates
o That process completed only 22,400 updates per hour
o Projected total elapsed time was 13 hours
o Updates could only be made during a larger window when no other tests were executing
Case study #3: Test environment data cleanupComparison benchmarks for mass updates:
300
780
0.374
716
4
173
0
100
200
300
400
500
600
700
800
900
K Rows Updated Elapsed Minutes K Rows / Minute
Silk Script High-speed Apply Engine
Lessons Learned and Best Practices
o Design and tuning of the prototype job
o SQL Generation techniques
o Considerations for multiple parallel agents
Lessons Learned and Best Practices
Design of the prototype job:
Step 1: Used a “staging” table and a Load utility to pre-populate keys of the rows to be deleted
Step 2: Generated our DELETE SQL’s using a BMC UNLOAD utility (DIRECT NO option) to unload from the staging table
Step 3: Replicate SQL scripts using SORTD for any child tables needing same deletes
Step 4: Execute High-speed Apply Engine with generated SQL’s
Lessons Learned and Best Practices
How we tuned the prototype job:
o Sorted the transactions in clustering sequence for less random and more synchronous I/O
o Increased commit frequency to every 1,000 statementsto reduce contention between parallel agents (prior to that we were committing after every 10,000)
o Disabled RI protection to allow more than one parallel agent per table having Referential Integrity
Lessons Learned and Best Practices
How we tuned the prototype job (continued)
o Use Conflict Rules to override warning messages written to SYSPRINT each time multiple rows are affected by one statement or row is not found
/UPDATECONFLICT/ - or -/DELETECONFLICT/ - or -/ANYCONFLICT/ CODE=MULTIPLEROWS Suppress warnings if multiple rows qualifyACTION=CONTINUE CODE=NOROWS Suppress warnings if zero rows qualifyACTION=CONTINUE
Lessons Learned and Best Practices
SQL Generation techniques - Unload
You can use an Unload utility to generate SQL from the target table itself
UNLOAD ORDER NO DIRECT NO ON MESSAGE 50253 CONTINUE UTILITY SELECT ' DELETE FROM OWNR.PLCY_DETL_TB WHERE AGENT_ID = ' CONCAT '''' CONCAT A.AGENT_ID CONCAT '''' CONCAT ' AND PLCY_ID = ' CONCAT CHAR(A.CUST_ID_NUM) CONCAT ' ; ' INTO SQL_TEXT CHAR(100) FROM OWNR.PLCY_DETL_TBWHERE DATE(INSRT_TSTMP) <= LAST_DAY(CURRENT DATE - 24 MONTHS)
OR (DLTE_TSTMP = ‘Y’ AND DATE(UPDT_TSTMP) WITH UR <= CURRENT DATE – 7 DAYS)
ORDER BY A.AGENT_ID, A.PLCY_ID WITH UR;
Lessons Learned and Best Practices
SQL Generation techniques - Unload
o Generated statements can be as granular as needed to leverage the benefits of the Apply Engine (ie. parallelism and commit frequency)
o Maximum length of the character string built this way from Unload is 255 bytes - Longer SQL statements can be split between multiple lines
o The INTO clause is needed to produce a Fixed Block (FB) output file, otherwise a Variable Block (VB) file is produced
Lessons Learned and Best Practices
SQL Generation techniques – Unload
Resulting statements:
DELETE FROM OWNR.PLCY_DETL_TB WHERE AGENT_ID = 'AG000123' AND PLCY_ID = 044444.;DELETE FROM OWNR.PLCY_DETL_TB WHERE AGENT_ID = 'AG000123' AND PLCY_ID = 047582.;DELETE FROM OWNR.PLCY_DETL_TB WHERE AGENT_ID = 'AG000123' AND PLCY_ID = 185022.;DELETE FROM OWNR.PLCY_DETL_TB WHERE AGENT_ID = 'AG000123' AND PLCY_ID = 185099.;DELETE FROM OWNR.PLCY_DETL_TB WHERE AGENT_ID = 'AG000123' AND PLCY_ID = 185666.;DELETE FROM OWNR.PLCY_DETL_TB WHERE AGENT_ID = 'AG000123' AND PLCY_ID = 185749.;DELETE FROM OWNR.PLCY_DETL_TB WHERE AGENT_ID = 'AG000123' AND PLCY_ID = 185772.;DELETE FROM OWNR.PLCY_DETL_TB WHERE AGENT_ID = 'AG000123' AND PLCY_ID = 372095.;
: : :DELETE FROM OWNR.PLCY_DETL_TB WHERE AGENT_ID = 'AG000199' AND PLCY_ID = 5747838.;
Lessons Learned and Best Practices
SQL Generation techniques - Sort
This example uses a SORTD utility to format a file with keys of rows to be updated into SQL UPDATE statements
SORT FIELDS=(1,11,CH,A,12,11,CH,A,23,02,CH,A) SUM FIELDS=NONE OUTFIL FILES=01,REMOVECC, OUTREC=(1:C'UPDATE OWNR.CLAIM_SUM_TB SET CLAIM_PD_YTD = ',
44:25,8,PD,EDIT=(TTTTTTTTTTTTT.TT), 61:C'WHERE CUST_ID = ', 77:C'''', 78:1,12, 90:C''' ', 139:C''';')
Lessons Learned and Best Practices
SQL Generation techniques - Sort
Resulting statements:UPDATE OWNR.CLAIM_SUM_TB SET PREM_PD_YTD = 123.45 WHERE CUST_ID = 'C00000010101’; UPDATE OWNR.CLAIM_SUM_TB SET PREM_PD_YTD = 999.99 WHERE CUST_ID = 'C00000010102’; UPDATE OWNR.CLAIM_SUM_TB SET PREM_PD_YTD = 678.99 WHERE CUST_ID = 'C00000010103’;UPDATE OWNR.CLAIM_SUM_TB SET PREM_PD_YTD = 222.33 WHERE CUST_ID = 'C00000010104’;UPDATE OWNR.CLAIM_SUM_TB SET PREM_PD_YTD = 444.55 WHERE CUST_ID = 'C00000010105’;UPDATE OWNR.CLAIM_SUM_TB SET PREM_PD_YTD = 777.44 WHERE CUST_ID = 'C00000010106’;UPDATE OWNR.CLAIM_SUM_TB SET PREM_PD_YTD = 987.65 WHERE CUST_ID = 'C00000010108’;UPDATE OWNR.CLAIM_SUM_TB SET PREM_PD_YTD = 119.11 WHERE CUST_ID = 'C00000010110’;
: : :UPDATE OWNR.CLAIM_SUM_TB SET PREM_PD_YTD = 999.99 WHERE CUST_ID = 'C00000020100’;
Lessons Learned and Best Practices
Considerations for multiple Parallel Agents
o Defaults are 5 for standalone mode or 10 for an SQLAPPLY load
o RI defined on tables, whether enforced or non (informational) will limit the process to one parallel agent - this safeguard can be overridden with a parameter: RICLUSTERING=NO
o Multiple parallel agents may increase total CPU cost of the process
o We’ve seen these jobs can cause stress to the local bufferpool –fewer parallel agents or more frequent commits have helped
Lessons Learned and Best Practices
Considerations for multiple Parallel Agents (continued)
o To assure multiple agents on a partitioned table, include a predicate on the partitioning column in the WHERE clause
o To get multiple agents on a table which is not partitioned, logical partitions must be defined:
/OBJECTPARTITION/ TABLE=OWNR.NEW_POLICY_TB COLUMN=POLICY_NUM PARTITION=PART1 VALUE='0' PARTITION=PART2 VALUE='1' ...
Lessons Learned and Best Practices
Read the manual! We referred to it often while developing the prototype job and many more that followed
Details on configuration parameters and options shown in this presentation are given in the following documents on the vendor’s website:
o High-speed Apply Engine Reference Manualo Release Notes (latest version is 9.2.00)
Any questions
Thank you for attending!
Steve SchulzState Farm Insurance [email protected]
Session Code: E12High Speed Apply Engine Unleashed!
