Bringing Internals to the Surface Internals Concepts for Highly Effective Engines Mark Scranton Principal Consultant/Trainer Informix Denver

Bringing Internals to Bringing Internals to the Surface the Surface

Internals Concepts for Highly Effective

Engines

Mark ScrantonPrincipal Consultant/Trainer

Informix Denverwww.markscranton.com

2

useruser .conference.conferenceInformixInformix

Who Am I?Who Am I?

• Principal Consultant/Trainer, Informix Denver

• I do engines - XPS and IDS

• 5+ yrs with Informix

• website - www.markscranton.com• tips/tricks• sql/scripts• documents, presentations

• Principal Consultant/Trainer, Informix Denver

• I do engines - XPS and IDS

• 5+ yrs with Informix

• website - www.markscranton.com• tips/tricks• sql/scripts• documents, presentations

3


Presentation OverviewPresentation Overview

• This presentation will identify “internals” concepts that appear to be trivia

• But they could be useful daily for maintenance and support of IDS

• A strong working knowledge of IDS is assumed

• More info available in the “IDS Internal Architecture” class

• This presentation will identify “internals” concepts that appear to be trivia

• But they could be useful daily for maintenance and support of IDS

• A strong working knowledge of IDS is assumed

• More info available in the “IDS Internal Architecture” class

4


TopicsTopics

• Physical pages vs. logical pages? • Discussion about the value of knowing

the difference

• The dog has swallowed it’s tail • ...or the tale of physical log overflow

• PARTNUMs - and the story of getting to data• We’ve all seen them…what does it really mean?

• Physical pages vs. logical pages? • Discussion about the value of knowing

the difference

• The dog has swallowed it’s tail • ...or the tale of physical log overflow

• PARTNUMs - and the story of getting to data• We’ve all seen them…what does it really mean?

5


TopicsTopics

• The maximum number of extents for a table? • Is there REALLY a max?

• Where did my row go?• The long tale of the forward pointer

• The last topic

• The maximum number of extents for a table? • Is there REALLY a max?

• Where did my row go?• The long tale of the forward pointer

• The last topic

6


Topic 1: Logical Pages vs. Physical PagesTopic 1: Logical Pages vs. Physical Pages

• Understanding the difference can help in:• Interpretation of msg log assertion failures• Interpretation of the extent list of a partition page• Lock level identification with respect to rowid(s)• Oncheck usage• A bunch of other stuff

• Understanding the difference can help in:• Interpretation of msg log assertion failures• Interpretation of the extent list of a partition page• Lock level identification with respect to rowid(s)• Oncheck usage• A bunch of other stuff

7


Physical PagesPhysical Pages

• Always with respect to a chunk

• Numbering starts with 0

• Format:

0xCCCPPPPP

ccc - chunk numberppppp - page offset into the chunk

• Always with respect to a chunk


• Format:

0xCCCPPPPP

ccc - chunk numberppppp - page offset into the chunk

8


Physical PagesPhysical Pages

30023c 30023d

300243300242300241300240

30023f30023e

300247300246300245300244

Chunk 3

0xCCCPPPPPc - chunk #p - page offset

9


Logical PagesLogical Pages

• Always with respect to a tablespace


• Format simply 0 through x

• Always with respect to a tablespace


• Format simply 0 through x

Logical PagesLogical Pages

30023c 30023d

300243300242300241300240

30023f30023e

300247300246300245300244

extent 1 for sparky

0

54

21

3

logicalpage num

create tablesparky...

physicalpage num

11


Usage Examples - Extent ListUsage Examples - Extent List

extent list on a partition page (oncheck -pt skippy:sparky)

chunk 0x4 page 0x33

12


Usage Examples - Bad PageUsage Examples - Bad Page

13


Usage Examples - Lock LevelUsage Examples - Lock Level

onstat -k

rowid format: 0xLLLLLLSSL - logical page in the tableS - slot/row # on the page

oncheck -pp 0x100016 2

14


Topic 2: Physical Log OverflowTopic 2: Physical Log Overflow

Physical Log File

1st before images of modified cache pages

75% full: checkpoint request is raised

what happens now?

back to the beginning, and keep writing!

15


Physical Log OverflowPhysical Log Overflow

Physical Log File

75% full: checkpoint request is raised

• Two potential scenarios:• the chkpt completes, and we give the physical log a

new logical beginning. In that case, we don’t care about the overwritten pages

• if the engine falls over, you’ll most likely get stuck in fast recovery when trying to come back up

• Two potential scenarios:• the chkpt completes, and we give the physical log a

new logical beginning. In that case, we don’t care about the overwritten pages

• if the engine falls over, you’ll most likely get stuck in fast recovery when trying to come back up

16


Topic 3: PartnumsTopic 3: Partnums

• Three areas focused on here:• What is a partnum? • How do we use it to get to stuff• What is a tablespace tablespace?

• Three areas focused on here:• What is a partnum? • How do we use it to get to stuff• What is a tablespace tablespace?

17


DefinitionsDefinitions

• Tablespace tablespace• Keeps track of tablespaces, or partitions,

in a dbspace• Each dbspace has one• Typical allocation is 50 pages for each extent

• Partition page• A page in a tablespace tablespace• Typically one page per tablespace• It “describes” the tablespace

• Tablespace tablespace• Keeps track of tablespaces, or partitions,

in a dbspace• Each dbspace has one• Typical allocation is 50 pages for each extent

• Partition page• A page in a tablespace tablespace• Typically one page per tablespace• It “describes” the tablespace

18


partnumpartnum

D D D L L L L L

dbspace number(starting at 1)

Logical page numwithin the tablespace tablespace (starting at 0)

1 1/2 bytes(3 nibbles, or 3 hex digits)

2 1/2 bytes(5 nibbles, or5 hex digits)

4 bytes

0x

19


dbspace4

rootdbs

systablesname partnumdatabase

sparky

0020000311-11-1999informixsparky

database tablespace

partnumflagscreatedownername

0020003cmudtable mud

extentsfor mud

tablespace tablespace page 3c

CREATE DATABASE sparky IN dbspace4;CREATE TABLE mud;

DATABASE sparky;SELECT * FROM mud;

20


Topic 4: Max Number of ExtentsTopic 4: Max Number of Extents

• What’s the maximum number of extents for a tablespace?

• Consider the previous topic, and the use of the partition page for a tablespace

• What’s the maximum number of extents for a tablespace?

• Consider the previous topic, and the use of the partition page for a tablespace

21


The Partition PageThe Partition Page

22


The Partition PageThe Partition Page

oncheck -pt skippy:sparky

23


Oh by the Way...Oh by the Way...

• Do you really want to know the max?

• Fragmentation changes the max for a logical table

• 4K vs. 2K pages

• Do you really want to know the max?

• Fragmentation changes the max for a logical table

• 4K vs. 2K pages

24


Topic 5: Forward PointersTopic 5: Forward Pointers

• Forward pointer - a 4-byte number that “points” to the remainder pieces of a row that spans pages

• Used with rows longer than a page

• Format is same as a rowid, although it’s NOT the rowid for the row

• Used with varchars or really long rows (rows longer than a page typically)

• We’ll focus on varchars ...

• Forward pointer - a 4-byte number that “points” to the remainder pieces of a row that spans pages

• Used with rows longer than a page

• Format is same as a rowid, although it’s NOT the rowid for the row

• Used with varchars or really long rows (rows longer than a page typically)

• We’ll focus on varchars ...

25


Forward Pointers w/ varcharsForward Pointers w/ varchars

• Problem - you can easily end up with a “chain” of forward pointer/remainder portions of rows

• This can happen when you’ve updated a varchar column and increased it’s size

• The new row size won’t fit into it’s old slot, and so we:• Move the row - all or some of it - to another page• Leave back a 4-byte forward pointer

on the home page

• Problem - you can easily end up with a “chain” of forward pointer/remainder portions of rows

• This can happen when you’ve updated a varchar column and increased it’s size

• The new row size won’t fit into it’s old slot, and so we:• Move the row - all or some of it - to another page• Leave back a 4-byte forward pointer

on the home page

26


Forward Pointers w/ varchars Forward Pointers w/ varchars

• This can happen as many times as it needs

• Every move adds an additional 8 bytes to the row length• 4 bytes for the forward pointer in the home page• 4 bytes for the slot table entry on the

new/remainder page

• This can happen as many times as it needs

• Every move adds an additional 8 bytes to the row length• 4 bytes for the forward pointer in the home page• 4 bytes for the slot table entry on the

new/remainder page

27


Forward PointersForward Pointers

28


What Can I Do? What Can I Do?

• To rid yourself of the “chaining” effect:• Unload/reload the table

• If the row will fit onto a single page, then there is no need for the forward pointer

• If it’s grown to be larger than a page (actual row length), then you’ll have at least one. This can’t be avoided - the row is now longer than a page

• Alter index to cluster• Same caveats as above

• To rid yourself of the “chaining” effect:• Unload/reload the table

• If the row will fit onto a single page, then there is no need for the forward pointer

• If it’s grown to be larger than a page (actual row length), then you’ll have at least one. This can’t be avoided - the row is now longer than a page

• Alter index to cluster• Same caveats as above

29


Topic 7: How Do You Find Out More? Topic 7: How Do You Find Out More?

• Attend the “IDS Internal Architecture” class• 5 days - very intense use of oncheck, and topics

covered in this presentation• Taught in Denver, Chicago, Menlo Park most often

• Watch TechNotes for a full white paper on this topic

• Send me email - [email protected]

• Checkout the website - www.markscranton.com

• Attend the “IDS Internal Architecture” class• 5 days - very intense use of oncheck, and topics

covered in this presentation• Taught in Denver, Chicago, Menlo Park most often

• Watch TechNotes for a full white paper on this topic

• Send me email - [email protected]

• Checkout the website - www.markscranton.com

30


Thanks...Thanks...

• For listening. Hope you learned something

• See you in class

• For listening. Hope you learned something

• See you in class

Documents

Bringing Internals to the Surface Internals Concepts for Highly Effective Engines Mark Scranton Principal Consultant/Trainer Informix Denver