Replacing Oracle CDC with Oracle GoldenGate

www.RedPillAnalytics.com [email protected] @RedPillA © 2014 RED PILL Analytics

Is Oracle GoldenGate a Replacement for Oracle Change

Data Capture?

1

Stewart Bryson

medium.com/@stewartbryson @stewartbryson linkedin.com/in/stewartbryson

http://www.redpillanalytics.com

mailto:[email protected]?subject=

http://twitter.com/RedPillA

© 2014 RED PILL Analytics

Who is Red Pill Analytics?


Red Pill Analytics: Leadership

3

Stewart Bryson

Oracle ACE Director

Previously From: Rittman Mead

Informix Software

Data Warehousing & Business Intelligence

since 1996

Kevin McGinley

Oracle ACE Director

Previously From: Accenture

BI Consulting Group

Data Warehousing & Business Intelligence

since 1997


mailto:[email protected]



About Red Pill Analytics

Other Services

• Expert Retainer • Support • Training

4

•DevOps for BI and DI •Hosted or On-premise •Used in Capacity Analytics

Capacity Analytics

•Development-as-a-service •Agile methodology • Faster/cheaper than traditional consulting • Subscription billing





“Change” Has Different Meanings


ODI Journalization Framework

A B

Oracle GoldenGate (Traditionally)

A B

Oracle CDC (Deprecated)

A B





A BODI Journalization

Framework

A B





A BODI Journalization

Framework

A B






A B

A





A

ODI Journalization Framework

A B





Is the A-Team Right About This?

13









Online Retailing Example


Publishing


Oracle CDC: Publish and Subscribe

17

EDW

R Sandbox

Partner Feed





Oracle CDC: Publishing

18

-- prepare source tablealter table sugarcrm.accounts ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;

BEGIN DBMS_CAPTURE_ADM.PREPARE_TABLE_INSTANTIATION(TABLE_NAME => 'sugarcrm.accounts');END;/

-- create change setBEGIN DBMS_CDC_PUBLISH.CREATE_CHANGE_SET ( change_set_name => 'CRM_ACCOUNTS', description => 'Change set for the CRM Accounts Data', change_source_name => 'HOTLOG_SOURCE', stop_on_ddl => 'y', begin_date => sysdate );END;






19




-- create change tableBEGIN DBMS_CDC_PUBLISH.CREATE_CHANGE_TABLE ( owner => 'cdcpub', change_table_name => 'accounts_ct', change_set_name => 'CRM_ACCOUNTS', source_schema => 'SUGARCRM', source_table => 'ACCOUNTS', column_type_list => 'ID CHAR(36), NAME VARCHAR2(150), DATE_ENTERED DATE,DATE_MODIFIED DATE, ACCOUNT_TYPE VARCHAR2(50), INDUSTRY VARCHAR2(50), PARENT_ID CHAR(36),

ANNUAL_REVENUE VARCHAR2(100), BILLING_ADDRESS_STREET VARCHAR2(150), BILLING_ADDRESS_CITY VARCHAR2(100), BILLING_ADDRESS_STATE VARCHAR2(100),

BILLING_ADDRESS_POSTALCODE VARCHAR2(20), OWNERSHIP VARCHAR2(100), BILLING_ADDRESS_COUNTRY VARCHAR2(255), RATING VARCHAR2(100),

WEBSITE VARCHAR2(255),TICKER_SYMBOL VARCHAR2(10), SIC_CODE VARCHAR2(10), CAMPAIGN_ID CHAR(36)',

capture_values => 'new', rs_id => 'y', row_id => 'n', user_id => 'n', timestamp => 'n', object_id => 'n', source_colmap => 'n', target_colmap => 'y', options_string => 'TABLESPACE UTILS');END;





-- create change tableBEGIN DBMS_CDC_PUBLISH.CREATE_CHANGE_TABLE ( owner => 'cdcpub', change_table_name => 'accounts_ct', change_set_name => 'CRM_ACCOUNTS', source_schema => 'SUGARCRM', source_table => 'ACCOUNTS', column_type_list => 'ID CHAR(36), NAME VARCHAR2(150), DATE_ENTERED DATE,DATE_MODIFIED DATE, ACCOUNT_TYPE VARCHAR2(50), INDUSTRY VARCHAR2(50), PARENT_ID CHAR(36),

ANNUAL_REVENUE VARCHAR2(100), BILLING_ADDRESS_STREET VARCHAR2(150), BILLING_ADDRESS_CITY VARCHAR2(100), BILLING_ADDRESS_STATE VARCHAR2(100),

BILLING_ADDRESS_POSTALCODE VARCHAR2(20), OWNERSHIP VARCHAR2(100), BILLING_ADDRESS_COUNTRY VARCHAR2(255), RATING VARCHAR2(100),

WEBSITE VARCHAR2(255),TICKER_SYMBOL VARCHAR2(10), SIC_CODE VARCHAR2(10), CAMPAIGN_ID CHAR(36)',

capture_values => 'new', rs_id => 'y', row_id => 'n', user_id => 'n', timestamp => 'n', object_id => 'n', source_colmap => 'n', target_colmap => 'y', options_string => 'TABLESPACE UTILS');END;


20




-- enable change setBEGIN DBMS_CDC_PUBLISH.ALTER_CHANGE_SET( change_set_name => 'CRM_ACCOUNTS', enable_capture => 'y');END;




Oracle CDC Subscription Views

www.redpillanalytics.com [email protected] @RedPillA © 2014 RED PILL Analytics

Oracle CDC Subscription View

22

NAME TICKER_SYMBOL OWNERSHIP CSCN$ COMMIT_TIMESTAMP$

OPERATION$

Red Pill Analytics 2992758 06/01/2014 12:00 AM INSERT

NAME TICKER_SYMBOL OWNERSHIP

Red Pill Analytics, LLC Private






23


OPERATION$


Red Pill Analytics, LLC Private 2992760 07/14/2014 12:00 AM UPDATE


Red Pill Analytics, LLC Private






24


OPERATION$


Red Pill Analytics, LLC Private 2992760 07/14/2014 12:00 AM UPDATE

Red Pill Analytics, Inc. RPAI Public 2992762 01/20/2015 12:00 AM UPDATE


Red Pill Analytics, Inc. RPAI Public






25

NVL (LEAD ( fnd_commit_date ) over ( PARTITION BY id ORDER BY fnd_commit_date ),to_date( '12/31/9999','mm/dd/yyyy' ))

NAME CSCN$ COMMIT_TIMESTAMP$ EXPIRE_TIMESTAMP$ OPERATION$ CURRENT_IND

Red Pill Analytics 2992758 06/01/2014 12:00 AM 07/14/2014 12:00 AM INSERT N

Red Pill Analytics, LLC 2992760 07/14/2014 12:00 AM 01/20/2015 12:00 AM UPDATE N

Red Pill Analytics, Inc. 2992762 01/20/2015 12:00 AM 12/31/9999 12:00 AM UPDATE Y






26

NAME CSCN$ COMMIT_TIMESTAMP$ EXPIRE_TIMESTAMP$ OPERATION$ CURRENT_IND

Red Pill Analytics 2992758 06/01/2014 12:00 AM 07/14/2014 12:00 AM INSERT N

Red Pill Analytics, LLC 2992760 07/14/2014 12:00 AM 01/20/2015 12:00 AM UPDATE N

Red Pill Analytics, Inc. 2992762 01/20/2015 12:00 AM 12/31/9999 12:00 AM UPDATE Y

CASEWHEN ( RANK() over (PARTITION BY id ORDER BY fnd_commit_date desc )) = 1THEN 'Y'ELSE 'N'END




DEMO: Viewing Change Data with GoldenGate


Am I in Love With a Deprecated Product?


What’s the Problem with Oracle CDC?

29

11g





11g


30






31

11g

12c

11g






32

11g

12c

11g






33

11g

12c

11g





Pragmatism


Text Here


Werner Heisenberg


Text Here


Text Here


Don’t Use Particles to Measure Particles


Don’t Use Streams to Capture Change


CDC Uses Streams (Capture and Queue)

41


A B





CDC Uses Streams (Capture and Queue)

42


A B

Oracle Streams (Deprecated)

A B





Oracle GoldenGate Architecture

43





Installing GoldenGate

44






45






46





Configuring GoldenGate EXTRACT

47

EXTRACT EXTUSERID oggcore_1, PASSWORD ogg SETENV (ORACLE_HOME="/app/oracle/product/11.2.0/dbhome_1")SETENV (ORACLE_SID="orcl")WARNLONGTRANS 1h, CHECKINTERVAL 30mEXTTRAIL ./dirdat/ltWILDCARDRESOLVE IMMEDIATETABLE SUGARCRM.ACCOUNTS,TOKENS( FND_SCN = @GETENV('TRANSACTION','CSN'), FND_ROW_RANK = @GETENV('RECORD','RSN'), FND_COMMIT_DATE=@GETENV('GGHEADER','COMMITTIMESTAMP'), FND_DML_TYPE=@GETENV('GGHEADER','OPTYPE'));

System change number (SCN) for the transaction

USEDEFAULTS defines column for column mapping

Transaction Commit Date (with Time)

Operation Type: Delete, Update, Insert, PK Update, and others





Configuring GoldenGate EXTRACT

48

EXTRACT EXTUSERID oggcore_1, PASSWORD ogg SETENV (ORACLE_HOME="/app/oracle/product/11.2.0/dbhome_1")SETENV (ORACLE_SID="orcl")WARNLONGTRANS 1h, CHECKINTERVAL 30mEXTTRAIL ./dirdat/ltWILDCARDRESOLVE IMMEDIATETABLE SUGARCRM.ACCOUNTS,TOKENS( FND_SCN = @GETENV('TRANSACTION','CSN'), FND_ROW_RANK = @GETENV('RECORD','RSN'), FND_COMMIT_DATE=@GETENV('GGHEADER','COMMITTIMESTAMP'), FND_DML_TYPE=@GETENV('GGHEADER','OPTYPE'));

GGSCI (oracle.localdomain) 1> add extract ext, integrated tranlog, begin nowEXTRACT added.

GGSCI (oracle.localdomain) 2> add exttrail ./dirdat/lt, extract EXT, megabytes 200EXTTRAIL added.

GGSCI (oracle.localdomain) 3> dblogin userid oggcore_1, password oggSuccessfully logged into database.

GGSCI (oracle.localdomain as oggcore_1@orcl) 4> register extract ext databaseExtract EXT successfully registered with database at SCN 2239208.

GGSCI (oracle.localdomain as oggcore_1@orcl) 5> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER RUNNING EXTRACT RUNNING EXT 00:00:10 00:00:04





Configuring GoldenGate DATA PUMP

49

EXTRACT PMPPASSTHRURMTHOST oracle.localdomain MGRPORT 7810, COMPRESSRMTTRAIL ./dirdat/rtTABLE SUGARCRM.*;





Configuring GoldenGate DATA PUMP

50

EXTRACT PMPPASSTHRURMTHOST linux-db MGRPORT 15000, COMPRESSRMTTRAIL ./dirdat/rtTABLE SUGARCRM.*;

GGSCI (oracle.localdomain) 1> add extract PMP, exttrailsource ./dirdat/ltEXTRACT added.

GGSCI (oracle.localdomain) 2> add rmttrail ./dirdat/rt, extract PMP, megabytes 200RMTTRAIL added.

GGSCI (oracle.localdomain) 3> start extract PMP

Sending START request to MANAGER ...EXTRACT PMP starting

GGSCI (oracle.localdomain) 4> info all


MANAGER RUNNING EXTRACT RUNNING EXT 00:00:02 00:00:07 EXTRACT RUNNING PMP 00:00:00 00:04:15





Configuring GoldenGate REPLICAT

51

REPLICAT REPSETENV (ORACLE_HOME="/app/oracle/product/11.2.0/dbhome_1")SETENV (ORACLE_SID="orcl")USERID oggcore_2, PASSWORD oggDISCARDFILE ./dirrpt/rep.dsc, append, megabytes 500SOURCEDEFS ./dirdef/sugarcrm.defMAP SUGARCRM.ACCOUNTS,TARGET SUGARSTG.ACCOUNTS,KEYCOLS(ID, FND_SCN, FND_ROW_RANK), INSERTALLRECORDS,COLMAP( USEDEFAULTS, FND_SCN=@token('FND_SCN'), FND_ROW_RANK=@token('FND_ROW_RANK'), FND_COMMIT_DATE=@token('FND_COMMIT_DATE'), FND_DML_TYPE=@token('FND_DML_TYPE'));

Map SOURCE and TARGET

KEYCOLS defines matching criteria


Our “Tokens” mapped to columns






52

REPLICAT REPSETENV (ORACLE_HOME="/app/oracle/product/11.2.0/dbhome_1")SETENV (ORACLE_SID="orcl")USERID oggcore_2, PASSWORD oggDISCARDFILE ./dirrpt/rep.dsc, append, megabytes 500SOURCEDEFS ./dirdef/sugarcrm.defMAP SUGARCRM.ACCOUNTS,TARGET SUGARSTG.ACCOUNTS,KEYCOLS(ID, FND_SCN, FND_ROW_RANK), INSERTALLRECORDS,COLMAP( USEDEFAULTS, FND_SCN=@token('FND_SCN'), FND_ROW_RANK=@token('FND_ROW_RANK'), FND_COMMIT_DATE=@token('FND_COMMIT_DATE'), FND_DML_TYPE=@token('FND_DML_TYPE'));

Map SOURCE and TARGET

KEYCOLS defines matching criteria


Our “Tokens” mapped to columns

INSERTALLRECORDS is the secret sauce for CDC






53

REPLICAT REPSETENV (ORACLE_HOME="/app/oracle/product/11.2.0/dbhome_1")SETENV (ORACLE_SID="orcl")USERID oggcore_2, PASSWORD oggDISCARDFILE ./dirrpt/rep.dsc, append, megabytes 500MAP SUGARCRM.ACCOUNTS,TARGET SUGARFND.ACCOUNTS,KEYCOLS(ID, FND_SCN), INSERTALLRECORDS,COLMAP(USEDEFAULTS,FND_SCN=@GETENV("TRANSACTION" , "CSN"),FND_ROW_RANK=@TOKEN("TK_ROW_RANK"),FND_COMMIT_DATE=@GETENV("GGHEADER" , "COMMITTIMESTAMP"),FND_DML_TYPE=@GETENV("GGHEADER" , "OPTYPE"));

GGSCI (oracle.localdomain) 1> dblogin userid oggcore_2, password oggSuccessfully logged into database.

GGSCI (oracle.localdomain as oggcore_2@orcl) 2> add checkpointtable oggcore_2.checkpoint

Successfully created checkpoint table oggcore_2.checkpoint.

GGSCI (oracle.localdomain as oggcore_2@orcl) 3> add replicat REP integrated, exttrail ./dirdat/rtREPLICAT (Integrated) added.

GGSCI (oracle.localdomain as oggcore_2@orcl) 4> start replicat REP

Sending START request to MANAGER ...REPLICAT REP starting

GGSCI (oracle.localdomain as oggcore_2@orcl) 5> info all


MANAGER RUNNING REPLICAT RUNNING REP 00:00:00 00:20:54




DEMO: Change Data in GoldenGate

What Are We Missing?

What Are We Missing?

Subscriptions


Publish and Subscribe

57

EDW

R Sandbox

Partner Feed





create table ogg_subscription ( sub_name varchar2(30) not null enable, sub_type varchar2(6), server_name varchar2(30), effective_scn number, expiration_scn number );

Create the Subscription

Purge the Window

Extend the Window

SCN Window

Subscription Name





create table ogg_subscription ( sub_name varchar2(30) not null enable, sub_type varchar2(6), server_name varchar2(30), effective_scn number, expiration_scn number );


create or replace view ogg_accounts_viewas select id, name, ticker_symbol, ownership, fnd_scn, fnd_dml_type, fnd_commit_date, NVL ( LEAD ( fnd_commit_date ) over ( PARTITION BY id ORDER BY fnd_commit_date ),

to_date( '12/31/9999','mm/dd/yyyy' )) fnd_expire_dt, CASE WHEN ( RANK() over (PARTITION BY id ORDER BY fnd_commit_date desc )) = 1 THEN 'Y' ELSE 'N' END current_ind from sugarstg.accounts join ogg_subscription on sub_name='FND_ACCOUNTS' and fnd_scn >=effective_scn and fnd_scn < expiration_scn order by fnd_commit_date;

SCN Window

Subscription Name

Purge the Window

Extend the Window






MERGE INTO ogg_subscription tUSING ( SELECT to_number( applied_low_position ) complete_scn,

server_name FROM all_gg_inbound_progress

) s ON ( s.server_name = t.server_name ) WHEN MATCHED THEN UPDATE SET t.expiration_scn = s.complete_scn, t.effective_scn = t.expiration_scn WHEN NOT MATCHED THEN INSERT

( t.effective_scn, t.expiration_scn, t.sub_name, t.server_name ) values ( '0', s.complete_scn, 'FND_ACCOUNTS', s.server_name );

Slide our window over if the subscription exists

Last SCN applied by GoldenGate

Purge the Window

Extend the Window

Create our subscription if it doesn’t already exist






delete sugarstg.accounts where fnd_scn < ( select min(expiration_scn)

from ogg_subscription );

Delete all the records that have been read by all subscriptions

Purge the Window

Extend the Window


Is Oracle GoldenGate a Replacement for Oracle Change

Data Capture?

62

Stewart Bryson

medium.com/@stewartbryson @stewartbryson linkedin.com/in/stewartbryson

Technology

Replacing Oracle CDC with Oracle GoldenGate