Data Recovery and

Fixing Database CorruptionsWhen changing a byte or a bit really makes a difference.

Few steps on knowing what to change.Dan Andrei STEFANSenior Database Administrator SCC Services Romania

@dba-lounge Iasi 27.08.2015

Database corruptions?!

Or possible data loss?

If a DBA did not yet encounter a database corruption issue, he/she is not lucky. Sooner or later, corruptions will strike!

A corruption is not necessary due to infrastructure issues.

Practice your knowledge on the “storage engine”, corruptions or data loss. Be prepared!

a bit of theory, internals, details

you in the next 15 minutes ?!

Physical Architecture

https://technet.microsoft.com/en-us/library/bb497064.aspx?f=255&MSPPError=-2147217396

File Anatomy

Page 0: HeaderPage 1: First PFS (Page Free Space)Page 2: First GAM (Global Allocation Map)Page 3: First SGAM (Shared Global Allocation Map)Page 4: UnusedPage 5: UnusedPage 6: First DCM (Differential Change Map) (DIFF)Page 7: First BCM (Bulk Changed Map) (ML)Page 8: Data PagePage 9: Boot PagePFS: covers 8088 pages / approx. 64 MB

GAM, SGAM, DCM, BCM: covers 64000 extents / approx. 4 GB

Anatomy of a SQL Server Page

• page size is always 8192 bytes• header is always 96 bytes• 8096 bytes available for in-row-data• 8060 bytes max row size

“Viewing” a page

dbcc page ( {'dbname' | dbid}, filenum, pagenum [, printopt={0|1|2|3} ])

filenum – file number for specified databasepagenum – page number for specified databaseprintopt -

0 - print just the page header1 - page header plus per-row hex dumps and a dump of the page slot array (unless its a page that doesn't have one, like allocation bitmaps)2 - page header plus whole page hex dump3 - page header plus detailed per-row interpretation

How to use DBCC PAGE: http://blogs.msdn.com/b/sqlserverstorageengine/archive/2006/06/10/625659.aspxDBCC TRACEON(3604, -1) to enable messages to current connection (default to errorlog)

DBCC PAGE & page header

PAGE HEADER:Page @0x00000001C56AA000m_pageId = (3:33) m_headerVersion = 1 m_type = 1m_typeFlagBits = 0x0 m_level = 0 m_flagBits = 0x204m_objId (AllocUnitId.idObj) = 98 m_indexId (AllocUnitId.idInd) = 256 Metadata: AllocUnitId = 72057594044350464 Metadata: PartitionId = 72057594038386688 Metadata: IndexId = 1Metadata: ObjectId = 2073058421 m_prevPage = (3:32) m_nextPage = (3:34)pminlen = 20 m_slotCnt = 323 m_freeCnt = 21m_freeData = 7525 m_reservedCnt = 0 m_lsn = (30:79:89)m_xactReserved = 0 m_xdesId = (0:0) m_ghostRecCnt = 0m_tornBits = 1384457226 DB Frag ID = 1

Allocation StatusGAM (3:2) = ALLOCATED SGAM (3:3) = NOT ALLOCATED PFS (3:1) = 0x40 ALLOCATED 0_PCT_FULLDIFF (3:6) = NOT CHANGED ML (3:7) = NOT MIN_LOGGED

The 96 bytes header (1)Byte 0 m_headerVersion (tinyint)This is the page header version. Since version 7.0 this value has always been 1.

Byte 1 m_type (tinyint)1 – data page2 – index page3 – text mix page4 – text tree page7 – sort page8 – GAM page9 – SGAM page10 – IAM page11 – PFS page13 – boot page15 – file header page16 – diff map page17 – ML map page18 – a page that’s be deallocated by DBCC CHECKDB during a repair

operation.19 – the temporary page that ALTER INDEX … REORGANIZE (or DBCC

INDEXDEFRAG) uses when working on an index.20 – a page pre-allocated as part of a bulk load operation, which will

eventually be formatted as a ‘real’ page.

Byte 2 m_typeFlagBits (tinyint)This is mostly unused. For data and index pages it will always be 4. For all other pages it will always be 0 – except PFS pages. If a PFS page has m_typeFlagBits of 1, that means that at least one of the pages in the PFS interval mapped by the PFS page has at least one ghost record.

Byte 3 m_level (tinyint)This is the level that the page is part of in the b-tree.Levels are numbered from 0 at the leaf-level and increase to the single-page root level (i.e. the top of the b-tree).For all page types apart from index pages, the level is always 0.

Bytes 4-5 m_flagBits (smallint)This stores a number of different flags that describe the page. For example,0x200 means that the page has a page checksum on it (as our example page does) and 0x100 means the page has torn-page protection on it.Some bits are no longer used in SQL Server 2005.

Bytes 6-7 m_indexId (smallint)In SQL Server 2000, these identified the actual relational object and index IDs to which the page is allocated. In SQL Server 2005 this is no longer the case. The allocation metadata totally changed so these instead identify what’s called the allocation unit that the page belongs to. This post explains how an allocation unit ID is calculated. http://www.sqlskills.com/blogs/paul/inside-the-storage-engine-anatomy-of-a-page/

DECLARE @alloc BIGINT = 72057594044284928;DECLARE @index BIGINT;

SELECT @index = CONVERT (BIGINT, CONVERT (FLOAT, @alloc) * (1 / POWER (2.0, 48)) ); /* right shift, reciprocal of left shift */

SELECT CONVERT (BIGINT, CONVERT (FLOAT, @alloc - (@index * CONVERT (BIGINT, POWER (2.0, 48)))) * (1 / POWER (2.0, 16)) /* right shift, reciprocal of left shift */) AS [m_objId]

, @index AS [m_indexId];

The 96 bytes header (2)Bytes 8-11 m_prevPage (int) (page number)Pointer to the previous page at this level of the b-tree.The pages on the left-hand side of a b-tree level will have the m_prevPage pointer be NULL.In a heap, or if an index only has a single page, these pointers will both be NULL for all pages.This value stands for the page number (file number : page number). Reserved byte order.

Bytes 12-13 m_prevPage (smallint) (file number)This value stands for the file number of the previous page (file number : page number) .Reserved byte order.

Bytes 14-15 pminlen (smallint)This is the size of the fixed-length portion of the records on the page.

Bytes 16-19 m_nextPage (int) (page number)Pointer to the next page at this level of the b-tree.The pages on the right-hand side will have the m_nextPage be NULL.In a heap, or if an index only has a single page, these pointers will both be NULL for all pagesThis value stands for the page number (file number : page number). Reserved byte order.

Bytes 20-21 m_nextPage (smallint) (file number)This value stands for the file number of the next page (file number : page number)Reserved byte order.

Bytes 22-23 m_slotCnt (smallint)This is the count of records on the page.

Bytes 24-27 m_objId (int)

Bytes 28-29 m_freeCnt (smallint)This is the number of bytes of free space in the page.

Bytes 30-31 m_freeData (smallint)This is the offset from the start of the page to the first byte after the end of the last record on the page. It doesn’t matter if there is free space nearer to the start of the page.

Bytes 32-35 m_pageId (int) (page number)

Bytes 36-37 m_pageId (smallint) (file number)

Bytes 38-39 m_reservedCnt (smallint)This is the number of bytes of free space that has been reserved by active transactions that freed up space on the page. It prevents the free space from being used up and allows the transactions to roll-back correctly. There’s a very complicated algorithm for changing this value.

Bytes 40-43 m_lsn (1) (int) The VLF sequence number.

Bytes 44-47 m_lsn (2) (int)The offset to the log block

Bytes 48-49 m_lsn (3) (smallint)The slot number inside the log block

Bytes 50-51 m_xactReserved (smallint)This is the amount that was last added to the m_reservedCnt field

Bytes 52-55 m_xdesId (2) (int)

Bytes 56-57 m_xdesId (1) (smallint)This is the internal ID of the most recent transaction that added to the m_reservedCnt field.

Bytes 58-59 m_ghostRecCnt (smallint)The is the count of ghost records on the page.

Bytes 60-63 m_tornBits (int)Contains torn bits value for checksum value. Reserved byte order.

Bytes 64-95 ?!

Online Resources

Anatomy of a page (Paul Randal)http://www.sqlskills.com/blogs/paul/inside-the-storage-engine-anatomy-of-a-page/

Anatomy of an extent (Paul Randal)http://www.sqlskills.com/blogs/paul/inside-the-storage-engine-anatomy-of-an-extent/

Anatomy of a record (Paul Randal)http://www.sqlskills.com/blogs/paul/inside-the-storage-engine-anatomy-of-a-record/

IAM pages, IAM chains, and allocation units (Paul Randal)

http://www.sqlskills.com/blogs/paul/inside-the-storage-engine-iam-pages-iam-chains-and-allocation-units/

GAM, SGAM, PFS and other allocation maps (Paul Randal)http://www.sqlskills.com/blogs/paul/inside-the-storage-engine-gam-sgam-pfs-and-other-allocation-maps/

Reverse Engineering SQL Server Page Headers (Mark S. Rasmussen)http://improve.dk/reverse-engineering-sql-server-page-headers/

What is an LSN: Log Sequence Number (Remus Rusanu)http://rusanu.com/2012/01/17/what-is-an-lsn-log-sequence-number/

Moment of truth

The Demo

During the demo:- fix page header- fix page linkage- fix page checksum- fix record header- “undo” a truncate table

How?- use HxD hex editor

http://mh-nexus.de/en/hxd/- DBCC WRITEPAGE

http://www.sqlskills.com/blogs/paul/dbcc-writepage/

http://stevestedman.com/server-health/database-corruption-challenge/◉ a 10 weeks challenge◉ 100% data recovery scenarios

Thanks!dbaTDPMon - Troubleshoot Database Performance and Monitoringhttp://dbatdpmon.codeplex.com

• out on 05.08.2015, under GNU GPL v3• work with SQL Server versions from 2000 onwards• custom database maintenance plan (consistency checks; backups; indexes, heaps & statistics

maintenance, etc.)• daily health checks• fully customizable