IDUG NA 2010 Paul Turpin: Unlocking the Mysteries of the

Unlocking the Mysteries of the DB2 Log

Paul TurpinS1 [email protected]

Session Code: C12

Thursday, May 13, 2010 02:45 PM – 03:45 PMPlatform: DB2 for Linux, UNIX, Windows

Logging is a critical part of DB2, but often not understood. This presentation will help you understand the log and how toconfigure it for good performance and recoverability. We will also explore log record layouts and how to use log APIs. Inorder to use the log APIs, you need a C program, so we will cover what is required to setup a C environment.

1

Agenda• What is a log used for? • Understanding log configuration

parameters• What's in a log - log file layouts• Log APIs and how to use them• How to setup a C environment to use

log APIs

ObjectivesObjective 1: What is a log used for? Objective 2: Understanding log configuration parametersObjective 3: What's in a log - log file layoutsObjective 4: Log APIs and how to use themObjective 5: How to setup a C environment to use log APIs

2

3

Sources of Information

• Information Center for DB2 9.5

• Administrative API Reference (SC23-5842-02)• Appendix B. DB2 log records

• Developing Embedded SQL Applications (SC23-5852-02)

Product manualshttp://www-01.ibm.com/support/docview.wss?rs=71&uid=swg27009727 (DB2 9.5 for Linux, UNIX, Windows Product Manuals)

DB2 9.5 Information Centerhttp://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp

Administrative API Reference (SC23-5842-02)http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=/com.ibm.db2.luw.common.doc/doc/r0008264.html

What is a log used for?• To record changes within the database

• Used to replay changes in the event of certain types of system failures

Chris Eaton has an excellent section on logging in his book “High Availability Guide for DB2”. Algorithm Chapter 2 has a lot of detail about how DB2 logging works and the algorithm DB2 uses for logging.

4

Understanding Log Configuration Parameters

20 + Configuration parameters related to logging

Log Configuration Parameters

db2 get db cfg for <dbname> | grep –i log

For instance,

db2 get db cfg for testdb | grep –i log

6

Default Configuration ParametersLog sizes

Log buffer size (4KB) (LOGBUFSZ) = 51Log file size (4KB) (LOGFILSIZ) = 1024

Number of primary log files (LOGPRIMARY) = 13Number of secondary log files (LOGSECOND) = 4

The output has been re-formatted a bit to make it easier to read.

7

How much log space do you need?

Depends on application characteristics

How many updates – inserts, updates, deletes does the application do?

Always need more than you guessed

BAD things happen if you run out

Information on calculating log spacehttp://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.dbobj.doc/doc/c0004930.html

8

Adjust space allocationsAdjust with:

Log file size (LOGFILSIZ)Number of primary log files (LOGPRIMARY)Number of secondary log files (LOGSECOND)


9

Log File SizeLog file size (LOGFILSIZ) = 1024

In 4 KB unitsDefault is ONLY 1024

Change withdb2 update db cfg for <dbname> using LOGFILSZ <For exampledb2 update db cfg for <dbname> using LOGFILSZ 25000

Approximately 4 MB

Approximately 100 MB

Default is only 4,194,304 bytes! Approximately 4 MB

25000 * 4KB = 100,000 KB or 100 MB

10

Log File SizeRange is 1,000 – 1,048,572 (1,024 GB) 9.5 FP3

Needs to be larger if there are many insert, delete and/or update operations against the database

Too small - affects performance due to frequent archiving

Too large – might not fit on archive media

50,000 reasonable

50000 * 4KB = 200,000 KB / 1024 is approximately 200 MB

11

Number of log filesHow many logs?

Primary vs secondary

Total (primary + secondary) <= 256, unless logsecond set to -1 (infinite logging)

Could have logprimary = 100 and logsecond = 156, for instance

Logprimary = 150 and logsecond = 156 would not be possible, though.

12

Primary log files

Maximum number controlled by LOGPRIMARY

When allocated is determined by archive mode

Will affect database “start up” time

The number of primary and secondary (logprimary + logsecond) log files must be <= 256, unless logsecond = -1.

13

Secondary log files

Maximum number controlled by LOGSECOND

When allocated is determined by archive mode and behavior of application

The number of primary and secondary (logprimary + logsecond) log files must be <= 256, unless logsecond = -1.

14

Change number of log files

Primary logs are pre-allocated at database creation and when the database is “started”

Can affect database “start up” timedb2 update db cfg for <dbname> using LOGPRIMARY 150

Secondary logs are allocated when neededdb2 update db cfg for <dbname> using LOGSECOND 106

The number of primary and secondary (logprimary + logsecond) log files must be <= 256, unless logsecond = -1.logprimaryhttp://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.config.doc/doc/r0000240.html

15

Maximum number of log files

Number of primary and secondary (logprimary + logsecond) log files <= 256,

unless logsecond = -1.

If you exceed 256 you get:

SQL5153N The update cannot be completed because the following relationship would be violated: "logprimary + logsecond <= 256".

The number of primary and secondary (logprimary + logsecond) log files must be <= 256, unless logsecond = -1.logprimaryhttp://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.config.doc/doc/r0000240.html

SQL5153N http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.messages.sql.doc/doc/msql05153n.html

16

Infinite LoggingLOGSECOND = -1

Is this a good or bad idea?

Roll back and recovery can be quite slow

If the database is enabled for infinite logging , archive logging MUST be enabled.

During recovery it may be necessary to retrieve archive logs and bring them back to the active log path.

17

Check log space usageSELECTsubstr(DB_NAME,1,10) as DB_NAME,LOG_UTILIZATION_PERCENT,TOTAL_LOG_USED_KB,TOTAL_LOG_AVAILABLE_KB,TOTAL_LOG_USED_TOP_KB FROM SYSIBMADM.LOG_UTILIZATION

Using the Log_Utilization Administrative view to retrieve log utilization information

http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.sql.rtn.doc/doc/r0022019.html

If using this with DPF select the DBPARTITIONNUM to indicate the partition that the log utilization applies to. The example above is for a single partition database.

18

Log Space Usage

Small log files in log directory?logs truncated when all applications disconnect from

database

Activate database to prevent this behavior

Prevent small files & speed up connections to database

http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=/com.ibm.db2.luw.sql.ref.doc/doc/r0024842.html

20

Default Configuration ParametersLog Paths

Changed path to log files (NEWLOGPATH) =Path to log files =

/local/db/db2inst1/db2inst1/NODE0000/SQL00002/SQLOGDIR/

Overflow log path (OVERFLOWLOGPATH) =Mirror log path (MIRRORLOGPATH) =First active log file =


21

Log pathsVery common to change put logs in their own file

systemThat generally would involve changing the path

to the log files.Path to log files =

/local/db/db2inst1/db2inst1/NODE0000/SQL00002/SQLOGDIR

db2 update db cfg for <dbname> using NEWLOGPATH /local/db2inst1/db2inst1/logs


22

Default Configuration ParametersLog Paths

Overflow log path (OVERFLOWLOGPATH)location for DB2 to find log files needed for a rollforward

operation. Prevents having to specify it in Rollforward command

Mirror log path (MIRRORLOGPATH)Contains a duplicate of the active log files

http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.config.doc/doc/r0005826.htmlOverflow log paths can also be used with the db2ReadLog API to indicate where log files are stored.

Mirrorlogpathhttp://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.config.doc/doc/r0005781.html

23

Default Configuration ParametersMiscellaneous

First active log file =Block log on disk full (BLK_LOG_DSK_FUL) = NOPercent max primary log space by transaction (MAX_LOG) = 0Num. of active log files for 1 active UOW(NUM_LOG_SPAN) = 0Percent log file reclaimed before soft chckpt (SOFTMAX) = 520HADR log write synchronization mode (HADR_SYNCMODE) =

NEARSYNCLog pages during index build (LOGINDEXBUILD) = OFF


24

Block log on disk fullBLK_LOG_DSK_FUL=YES

Configurable online (immediate)

Prevent log full condition – retry every 5 minutes to create new log

Will cause application to hang on log full condition – can resolve issue

http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.config.doc/doc/r0005787.htmlhttp://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.ha.doc/doc/c0006131.html

If set to NO, then the transaction will be rolled back and sometimes will cause a DB2 crash

25

Maximum log per transactionMAX_LOG=0 to 100


0 means no limit

1-100 tell how much log space a transaction is allowed to use

http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.admin.config.doc/doc/r0006018.html

26

Soft checkpoint frequencySOFTMAX=1 to 100 * logprimary

Configurable

Default is 100

Percentage of the size of one primary log file


27

Maximum log per transactionMAX_LOG=0 to 100


0 means no limit

1-100 tell how much log space a transaction is allowed to use


28

LOGINDEXBUILDIf set to YESthen

index creation, index recreation and reorg operations are logged.

The logs can be used during rollforward or HADR replay procedures.


29

Default Configuration ParametersArchiving parameters

Log retain for recovery status = NOUser exit for logging status = NOFirst log archive method (LOGARCHMETH1) = OFFOptions for logarchmeth1 (LOGARCHOPT1) =Second log archive method (LOGARCHMETH2) = OFFOptions for logarchmeth2 (LOGARCHOPT2) =Failover log archive path (FAILARCHPATH) =Number of log archive retries on error (NUMARCHRETRY) = 5Log archive retry Delay (secs) (ARCHRETRYDELAY) = 20

Deprecated in 9.5

The output has been re-formatted a bit to make it easier to read.The Log Retain and User exit information are deprecated for DB2 9.5 servers, but is still used for pre 9.5 servers.

NUMARCHRETRY – The number of times that DB2 will try to archive a log file to the primary or secondary archive directory, before it attempts to archive to failover directory.

ARCHRETRYDELAY – number of seconds to wait after a failed archive attempt, before retrying the archive operation

30

Log Archive Modes• Circular• Archive logging

If logarchmeth1 and logarchmeth2 are set to OFF, then circular logging will be used. Rollforward recovery is NOT supported with circular logging, while crash recovery is.

31

Circular logging

• Rollforward logging is NOT supported

1 2 3 4

32

Archive logging

• Rollforward logging IS supported

Logarchmeth1 and Logarchmeth2 are set to OFF Rollforward recovery is supported with archive logging, as well as, crash recovery.

33

Default Configuration ParametersArchiving parameters

Log retain for recovery status = NOUser exit for logging status = NOFirst log archive method (LOGARCHMETH1) = OFFOptions for logarchmeth1 (LOGARCHOPT1) =Second log archive method (LOGARCHMETH2) = OFFOptions for logarchmeth2 (LOGARCHOPT2) =Failover log archive path (FAILARCHPATH) =Number of log archive retries on error (NUMARCHRETRY) = 5Log archive retry Delay (secs) (ARCHRETRYDELAY) = 20

Deprecated in 9.5

The Log Retain and User exit information are deprecated for DB2 9.5 servers, but is still used for pre 9.5 servers.

34

LOGARCHMETHnLOGARCHMETH1 and LOGARCHMETH2

What Archive Mode Am I Using?db2 get db cfg for <dbname> | grep –i logarchmeth

In pre 9.5 servers you could also look at LOGRETAIN and USEREXIT parameters

LOGRETAIN and USEREXIT–deprecated in 9.5, but still listed in db cfg

Logarchmeth1 and Logarchmeth2 are set to OFF Circular logging will be used. Rollforward recovery is not supported with circular logging, while crash recovery is.

36

What Archive Mode Am I Using?db2 get db cfg for <dbname> | grep –i logarchmeth

If Logarchmeth1 and Logarchmeth2 are set to OFF,then

Circular logging is used

Rollforward recovery is not supported with circular logging, only crash recovery.

37

Change Archive Log Modedb2 update db cfg for <dbname> using logarchmeth1

for example

db2 update db cfg for <dbname> using logarchmeth1DISK:/logs/archive/

The Log Retain and User exit information are deprecated for DB2 9.5 servers, but is still used for pre 9.5 servers.

db2 update db cfg for <dbname> using logarchmeth1TSM

38

Turn Archiving OffIn DB2 9.5db2 update db cfg for <dbname> using logarchmeth1OFF

If you getSQL5153N The update cannot be completed because the

following relationship would be violated: "logsecond can only be set to -1 if log archiving is enabled".

db2 update db cfg for <dbname> using logsecond 10

You cannot have infinite logging (logsecond = -1) if you are using circular logging, so you must set logsecond to a number before you can turn off archive logging.

http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.messages.sql.doc/doc/msql05153n.html

39

Where’s My Userexit?

Deprecated in 9.5

Still in pre-9.5 servers

If setarchive logging ON regardless of LOGRETAIN

TSM logging• db2 update db cfg for <dbname> using logarchmeth1

TSM

Carolyn Sanders “TSM is not just a black boxﾔ - Commands a DBA Should Know” gives a wealth of information about using TSM with DB2. Her presentation can be found at:https://www.idug.org/conferences/NA2008/data/NA08I03.pdf

41

Log layouts (What is in a log)

Log record types

• Log manager header• Long field manager log records• Transaction manager log records• Utility manager log records• Data manager log records

http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/topic/com.ibm.db2.luw.apdv.api.doc/doc/r0001910.html

43

Log record types

• Log manager header

• All log records begin with log manager header• Contains information like length of log, type

of log record, log sequence number (LSN)


44

Log record types

• Long field manager log records

• Written whenever long field data is inserted, deleted, or updated

• Only written is archive logging


45

Log record types

• Transaction manager log records

• Written when transaction complete• Timestamps are Coordinated Universal Time

(CUT)


46

Log record types

• Utility manager log records• Log records are associated with :

• Migration• Load• Backup• Table space rollforward


47

Log record types

• Data manager log records

http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=/com.ibm.db2.luw.sql.ref.doc/doc/r0024842.html

Log manager header layouthttp://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=/com.ibm.db2.luw.sql.ref.doc/doc/r0024842.html

48

Log APIs and how to use them

49

Log APIs• db2ReadLogNoConnInit - Initialize reading

the database logs without a database connection

• db2ReadLogNoConn - Read the database logs without a database connection

• db2ReadLog - Extracts log records

50

Log APIs• db2ReadLog - Extracts log records

• db2ReadLogNoConnTerm - Terminate reading the database logs without a database connection

• db2ArchiveLog - Archive the active log file

51

SQLCA data structureSQL Communication area (SQLCA)

• Used by database manager to return error information

• Updated after every API call and SQL statement issued

http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=/com.ibm.db2.luw.apdv.api.doc/doc/r0001673.html

52

db2ReadLogNoConnInit • Initialize reading the database logs without a

database connection

• Allocate memory needed by db2ReadLogNoConn


53

db2ReadLogNoConn• Read the database logs without a database

connection


54

db2ReadLogExtracts log records orqueries Log Manager for current log state

information

Works with recoverable database logs ONLY


55

db2ReadLogNoConnTermTerminate reading database logs without a

database connection

Deallocate memory used by db2ReadLogNoConn


56

db2ArchiveLogArchive the active log file

• Closes and truncates the active log file

• If archiving enabled it also issues archive request


ARCHIVE command (CLP)db2 archive log for database <dbname> for single partition databasesOrdb2 archive log for database <dbname> on dbpartitionnum <partition_number> for DPF databases

57

Compiling C programsCopy sample C programs

cd <instance owner home directory>/sqllib/samples/c/tar -cvf /tmp/myfile.tar *

To expand tar file

tar –xvf myfile.tar

It is safest to make a copy of the sample programs before starting to work with them.

By cd’ing to directory first, the tar file created will NOT try to expand several directories

tar -tf /tmp/myfile.tar Will list all the files in the tar file name myfile.tar

C++<instance owner home directory>/sqllib/samples/cpp directory

58

Setup the environment• To experiment with the samples it is best to create the

SAMPLE database

• The majority of the sample programs work with the SAMPLE database by default

Use make to prepare programs• For example

make tsinfo

• Creates an executable called tsinfo

• Execute (if in directory where executable stored)• ./tsinfo

“Compile” dblognoconn• make dblognoconn• Creates an executable called dblognoconn

DB2 APIs USED:db2CfgGet -- Get Configurationdb2CfgSet -- Set Configurationdb2ReadLog -- Asynchronous Read Logdb2ReadLogNoConnInit -- Read log without a db

connectiondb2ReadLogNoConnTerm -- Terminate read log without a

database connection

61

dblognoconn setup• make dblognoconn

• out of make you get • dblognoconn.o• dblognoconn.bnd• Dblognoconn

dblognoconn setup

• Make uses a file called makefile to control process• with dblognoconn, it runs

• bldapp dblognoconn sample

• because in the makefile ALIAS is the database name and sample is the default value assigned to ALIAS

• If you do• make dblognoconn.sqc

Agenda• What is a log used for? • Understanding log configuration

parameters• What's in a log - log file layouts• Log APIs and how to use them• How to setup a C environment to use

log APIs

ObjectivesObjective 1: What is a log used for? Objective 2: Understanding log configuration parametersObjective 3: What's in a log - log file layoutsObjective 4: Log APIs and how to use themObjective 5: How to setup a C environment to use log APIs

64

Paul [email protected]

Session Code: C12

Paul Turpin is a database architect and database administrator at a software development company in Charlotte, NC. He specializes in DB2 for LUW on large systems. He enjoys exploring new features and functionality in DB2. He has spoken at IDUG North America, IDUG Europe, SHARE, IBM’s Information on Demand conference, and several RUGs. Paul currently serves as chairperson on the IDUG 2010 North America Conference Planning Committee.

65

Extra Material

Setting up CygwinCompiling DB2 UDB Stored Procedures with GCC

on Windows

http://www.ibm.com/developerworks/data/library/techarticle/0306haungs/0306haungs.html

67

Documents

IDUG NA 2010 Paul Turpin: Unlocking the Mysteries of the