Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar

8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar

1/19

Change Data Capture

Srikant JahagirdarDevelopment LeadMSIT - Microsoft


2/19

Session Objectives And Takeaways

Session Objective(s):

Describe Change Data Capture architecture

Build a data tracking solution using Change DataCapture

Key Takeaways

Best practices and recommendations for change datacapture solutions


3/19

Tracking Methods

The traditional tracking methods provides differentlevels of change information and can be intrusive

Trigger based trackingTimestamp columns

Join queriesAdditional table to track deletes

Scenario requirements vary and one size fits allmay not be the best approach when buildingtracking solutions


4/19

Typical Scenario

Right time BI is becoming more critical for businesses

Data volume is increasing and batch windows aredecreasing

Efficient ETL through incremental data load is keyto reducing the overall ETL time

Upfront information about what changed helpsbuild efficient ETL solutions


5/19

SQL Server 2008 TrackingFeatures

Change Data CaptureProvide rich change information harvested fromdatabase log

Change TrackingLight weight tracking providing real-time changedetection

SQL AuditingLow impact tracking providing auditing information


6/19

Value Proposition

Change Data Capture provides valuable changeinformation about DML changes on a tableefficiently and close to real-time

Eliminates expensive techniques:

user triggers, timestamp columns, expensive join queries

Can provide answers to a variety of key questionsWhat are ALL the changes that happened between 12:00AM and12:00PM?I want to get a transactional consistent set of all changes thathappened in Order table and Order_Detail since 12:00PMWhich column changed?Was the change an insert or an update?


7/19

Functional Overview

Configuration

Tracking Mechanism

Querying for changes


8/19

Configuration

Change Data Capture enabled at two levelsDatabase - sp_cdc_enable_dbIndividual tables - sp_cdc_enable_table

sysadmin privilege required to enable CDC at thedatabase levelRequires dbo privilege to enable CDC at the table level

System metadata available to track configuration

Supported only on enterprise edition


9/19

Tracking MechanismAsynchronous log reading technology populatechange tables

Provide the flexibility to stop capture to minimize impact onsourceBuild on top of the replication log reader technology

Provides guarantees on transactionalconsistency in case of failures

Tracking infrastructure allows time-based rangequeries

All changes are trackedCaptures the before and after image of updates from the log


10/19

Store_Num Order_Status Store_desc

1001 A Mystore1

1002 B Mystore2

STORE TABLE

Store_Id Order_Status Store_desc Operation Column Info TransactionTimestamp

1001 H Mystore1 Insert 0x00 1

1002 D Mystore2 Insert 0x00 2

1001 A Mystore1 Update 0x10 3

1002 B Mstore2 Update 0x10 4

Lo gBased

CaptureJob

Db Log

Store_Id Order_Status Store_desc Operation Column Info TransactionTimestamp

C DC

A P I s

Select * fromfn_cdc_get_net_changes_*(starttime,endtime)

DW

CDC ConsumerSSIS Package

CleanupJob

STORE Change Table

Insert 1001 ,HInsert 1002, D

Update 1001, AUpdate 1002, B

Store_Num Order_Status Store_desc


11/19

Capturing Changes


12/19

Querying for ChangesTwo table valued functions (TVF) for eachchange tableAllows bounded range based queries

Row filter option provides filtering of result setConsistent result set guaranteed when using time-based ranges

All changes TVFUseful in extracting all changes very efficientlyOptionally provide before and after image of update

Net changes TVFAccurately identifies net operation to a row in a rangeProvides information about which column(s) changed for the netchange rowMore expensive query depending on filter options


13/19

Querying for Changes

TVF uses LSN to specify the rangeselect * from cdc.fn_cdc_get_all_changes_Customer(@from_lsn, @to_lsn, 'all')

Log sequence number (LSN) is a binary(10) typeYuk!!!......Why cant I use date time to specify the range?

Time to LSN mapping function enables range tobe specified using time

Wrapper function generator provides an easy wayto deploy time based enumeration functions


14/19

Querying for ChangesAdvanced system functions provide mapping andother functionality

Easily determine the lowest and the highest watermark forchanges available in the change tablesDetermine which column changed given a bitmask

Security ModelAccess to change data controlled through TVFRequire access to all tracked columns in the base table at aminimumCDC role provide additional layer of security


15/19


16/19

Best Practices and RecommendationsChange tracking Solution

Use the change tracking that suits your applicationrequirement

PerformanceUse separate file-groups for change tracking tableTrack only the required columnsWrites to the change table are logged

Querying change dataMake sure that the fromLSN and toLSN are in the lowend and high end of change tracking windowsys.sp_cdc_generate_wrapper_function


17/19

Resources

http://www.databasejournal.com/features/ms

sql/article.php/3720361

http://www.databasejournal.com/features/ms

sql/article.php/3725476
http://www.databasejournal.com/features/mssql/article.php/3725476http://www.databasejournal.com/features/mssql/article.php/3725476http://www.databasejournal.com/features/mssql/article.php/3720361http://www.databasejournal.com/features/mssql/article.php/3720361


18/19

Contact

[email protected]
mailto:[email protected]


19/19

2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market

conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.MICROSOFT MAKES NO W ARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Documents

Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar