Informix Enterprise Replication in 5 easy Steps WAIUG Forum December 8-9, 2006

© 2006 Cisco Systems, Inc. All rights reserved. Cisco PublicPresentation_ID 1

Informix Enterprise Replication in 5 easy StepsWAIUG Forum December 8-9, 2006

Nancy Balsbaugh


WAIUG Forum December 8-9, 2006 Introduction to IDS Enterprise Replication

What are we going to cover?

Types of Enterprise Replication

Requirements – Linux, 10.00.UC5

How to set it up

How to verify/check replication

Troubleshooting

Pitfalls and Gotchas

Magic Cookies

Overview



Types of Enterprise Replication

Primary – Target (tree)

Simple

No Conflict

Works or Not



Types of Enterprise Replication cont’d

Update Anywhere

Complex

Conflict resolution

May Partially Work



Requirements Connectivity

Sqlhostsg_hdr group - - i=1

g_server1 group - - i=2

server1 onsoctcp hostname1 server1 g=g_server1


server2 onsoctcp hostname2 server2 g=g_server2

.rhosts

/etc/services

/etc/hosts

Template File <filename>



Requirements Cont’d

Onconfig

Dbserveralias – used in the sqlhosts file



How To Set It Up

Step 1 - Define the Servers

Step 2 - Define the replicates

Step 3 - Change the state of the replicate (start replicating)

Step 4 - Synchronize the Data (copy pre-replication data)

Step 5 - Verify



Step 1 - Define the servers

Participants, master, conflict resolution rules and scope, frequency, error logging, full rows or changed columns, triggers, etc.

Primary

cdr define server --connect=server1 --idle=0 --ats=/errorlog/ats --ris=/errorlog/ris --init g_server1

Secondary

cdr define server -c server1 --idle=0 --init --leaf --sync=g_server1 server2

ATS – aborted transaction spooling, RIS – row information spooling, Init - option initializes the server, Idle – idle timeout, Leaf means tree hierarchy – leave that out if you want update anywhere



After the first server is definedcdr list server

SERVER ID STATE STATUS QUEUE CONNECTION CHANGED

-----------------------------------------------------------------------

g_server1 3 Active Local 0



After the 2nd server is definedcdr list server


-----------------------------------------------------------------------


g_server2 7 Active Connected 0 Nov 14 13:47:09



Cdr list of tree hierarchyListing g_server1


-----------------------------------------------------------------------




**********

Listing g_server2


-----------------------------------------------------------------------



**********

Listing g_server3


-----------------------------------------------------------------------





Update Anywhere cdr list Listing g_server1


-----------------------------------------------------------------------




**********

Listing g_server2


-----------------------------------------------------------------------




**********

Listing g_server3


-----------------------------------------------------------------------






Step 2 – Option 1 - Define the replicates

cdr define replicate --conflict=timestamp --scope=row --ats --ris --master=g_server1 <replicate_name>



Step 2 – Option 1 - Define the replicatescdr list repl

CURRENTLY DEFINED REPLICATES

-------------------------------

REPLICATE: r_test1

STATE: Active ON:g_a1

CONFLICT: Always Apply

FREQUENCY: immediate

QUEUE SIZE: 0

PARTICIPANT: testdb:informix.test1

OPTIONS: transaction,ris,ats,fullrow

REPLID: 65537 / 0x10001

REPLMODE: PRIMARY ON:g_a1

APPLY-AS: INFORMIX ON:g_a1

REPLTYPE: Master

REPLICATE: r_test2

STATE: Active ON:g_a1

CONFLICT: Always Apply

FREQUENCY: immediate

QUEUE SIZE: 0

PARTICIPANT: testdb:informix.test2

OPTIONS: transaction,ris,ats,fullrow

REPLID: 65538 / 0x10002

REPLMODE: PRIMARY ON:g_a1

APPLY-AS: INFORMIX ON:g_a1

REPLTYPE: Master



Step 2 - Option 2 – Define a Template

cdr define template <templatename>

--conflict=timestamp

--scope=row --ats --ris --firetrigger

--master=g_server1

--database=dbname --file=<filename>

The file contains a list of tables to be replicated – easiest way.



Step 2 - Option 2 – Define a Template

cdr list templateTEMPLATE DATABASE TABLES

==============================================

Template_name dbname informix.table1

dbname informix.table2

dbname informix.table3



Step 3 - Option 1 – Change the replicate

cdr start replicate <replicate_name> g_server1 g_server2



Step 3 - Option 2 - Realize the template

cdr realize template <templatename> g_server1

cdr realize template <templatename> g_server2 g_server3

This copies the replicate to the other servers and changes it’s state – “starts it”

Do the primary first!

Then broadcast the rest



Synchronize the data

cdr sync replset -m g_server1 -s <templatename>

--firetrigger g_server2 g_server3



How to Verify/Check Replication

cdr check replicateset -c server1 -m g_server1 -s <templatename> -v -a



Troubleshooting

To Repair Replication

cdr check replicateset -c server1 -m g_server1 -s <templatename> -v –a –R

Cdr list of all the servers (with the connect option)

Cdr err of all the servers (with the connect option)

Check all the servers message logs



cdr err –a Without Errors

SERVER:SEQNO REVIEW TIME ERROR



cdr err –a With Errors

SERVER:SEQNO REVIEW TIME ERROR

g_server2:164 N 2006-11-02 19:39:01 31 GC operation participate replicate 'sync_17797_131103_1162517923' failed: undefined replicate

g_server3:165 N 2006-11-02 19:39:01 31 GC operation participate replicate 'sync_17797_131103_1162517923' failed: undefined replicate



Online Log when replication is being set-up12:10:39 Building 'syscdr' database ...

12:10:40 'syscdr' database built successfully.

12:10:42 CDR queuer initialization complete

12:10:42 CDR NIF listening on asf://g_server1

12:13:10 Checkpoint Completed: duration was 1 seconds.

12:13:10 Checkpoint loguniq 298, logpos 0x132a7018, timestamp: 0xf5f9165

.

12:20:22 ER checkpoint started

.

12:20:25 DDR Log Snooping - Snooping started in log 299

12:25:41 Checkpoint Completed: duration was 0 seconds.

12:25:41 Checkpoint loguniq 299, logpos 0x3bb9018, timestamp: 0xf63be8a



Pitfalls and Gotchas cdr err will mark errors and you’ll need to make sure

you use cdr err –a to see all errors that haven’t been zapped.

If Update-Anywhere, then use the –c option to connect to the other servers and look from their perspective.

Use the –c (connect) option to make control codes come from one central server.

Define can only be done serially – same with all the commands. Do not run them in parallel. To speed things up – Use broadcast mode.



Pitfalls and Gotchas cont’d

You must list all servers for cdr list otherwise 2 servers can have problems between them that doesn’t affect the primary. If you are looking only at the primary, you won’t see it.



Magic Cookies Check the return code

Check the output of cdr err –a

Use cdr err --zap to clear out errors

Always check the list of all the servers and the errors on all the servers.

Use cdr finderr to lookup errors

Scripts Included! Very Basic! Need improvement. Mostly so you can see the cdr commands. Feel free to improve upon them and send them back to me.



cdr finderr12:11:10 CDR GC synchronization error 35

finderr 35

-35 Operation would block.

An operating-system error code with the meaning shown was unexpectedly returned to the database server. If the error recurs, note all circumstances and contact Technical Support at [email protected].


cdr finderr cont’d

cdr finderr 35

35 server not defined in sqlhosts Sqlhosts on leaf

g_hdr group - - i=1


server1 onsoctcp SERV1 server1 g=g_server1


server2 onsoctcp SERV2 server2 g=g_server2


Source: Placeholder for Notes is 14 points

Q & A

Any Questions?


Thank You

Nancy Balsbaugh

[email protected]


To list all the replication servers (cdr list)

#!/bin/ksh echo "Building custom scripts for this Cluster" echo "#!/bin/ksh" > cdrlist.new chmod 700 cdrlist.new echo "echo \"***********************\"" >> cdrlist.new echo "date" >> cdrlist.new echo "echo \"Starting Listing Servers\"" >> cdrlist.new

for i in `cat pub` do echo "Adding in the publisher $i" echo "echo \"Listing g_$i\"" >> cdrlist.new echo "cdr list server -c $i" >> cdrlist.new echo "echo \"Return value:\$?\"" >> cdrlist.new echo "echo \"\"" >> cdrlist.new echo "echo \"\"" >> cdrlist.new echo "echo \"**********\"" >> cdrlist.new done

export pub=$i echo "pub = $pub"

for i in `cat servers` do echo "Adding in the subscriber $i" echo "echo \"Listing g_$i\"" >> cdrlist.new echo "cdr list server -c $i" >> cdrlist.new echo "echo \"Return value:\$?\"" >> cdrlist.new echo "echo \"\"" >> cdrlist.new echo "echo \"\"" >> cdrlist.new echo "echo \"**********\"" >> cdrlist.new done


To see errors (cdr err)

echo "#!/bin/ksh" > cdrerr.new chmod 700 cdrerr.new echo "echo \"***********************\"" >> cdrerr.new echo "date" >> cdrerr.new echo "echo \"Starting Listing Servers\"" >> cdrerr.new

for i in `cat pub` do echo "Adding in the publisher $i" echo "echo \"Showing Errors from g_$i\"" >> cdrerr.new echo "cdr err -a -c $i" >> cdrerr.new echo "echo \"Return value:\$?\"" >> cdrerr.new echo "echo \"\"" >> cdrerr.new echo "echo \"\"" >> cdrerr.new echo "echo \"**********\"" >> cdrerr.new done


for i in `cat servers` do echo "Adding in the subscriber $i" echo "echo \"Showing Errors from g_$i\"" >> cdrerr.new echo "cdr err -a -c $i" >> cdrerr.new echo "echo \"Return value:\$?\"" >> cdrerr.new echo "echo \"\"" >> cdrerr.new echo "echo \"\"" >> cdrerr.new echo "echo \"**********\"" >> cdrerr.new done


To clear errors in cdr err

echo "#!/bin/ksh" > cdrzap.new chmod 700 cdrzap.new echo "echo \"***********************\"" >> cdrzap.new echo "date" >> cdrzap.new echo "echo \"Starting Listing Servers\"" >> cdrzap.new

for i in `cat pub` do echo "Adding in the publisher $i" echo "echo \"Showing Errors from g_$i\"" >> cdrzap.new echo "cdr err -z -c $i" >> cdrzap.new echo "echo \"Return value:\$?\"" >> cdrzap.new echo "echo \"\"" >> cdrzap.new echo "echo \"\"" >> cdrzap.new echo "echo \"**********\"" >> cdrzap.new done


for i in `cat servers` do echo "Adding in the subscriber $i" echo "echo \"Showing Errors from g_$i\"" >> cdrzap.new echo "cdr err -z -c $i" >> cdrzap.new echo "echo \"Return value:\$?\"" >> cdrzap.new echo "echo \"\"" >> cdrzap.new echo "echo \"\"" >> cdrzap.new echo "echo \"**********\"" >> cdrzap.new done


To Run Cdr Check

echo "#!/bin/ksh" > cdrcheck.new chmod 700 cdrcheck.new echo "date" >> cdrcheck.new echo "echo \"Running cdr check for status of Cluster\"" >> cdrcheck.new

echo "time cdr check replicateset -c $pub -m g_$pub -s <templatename> -v -a >> /home/informix/cdr_check_output_all_\`date +%Y%m%d%H%M\` 2>&1" >> cdrcheck.new

echo "echo \"\"" >> cdrcheck.new echo "echo \"Showing Errors from g_$i\"" >> cdrcheck.new for i in `cat servers` do echo "Adding in the subscriber $i" echo "$i" >> cdrcheck.new done echo "-v -a >> /home/informix/cdr_check_output_all_\`date +%Y%m%d%H%M\` 2>&1" >> cdrcheck.new


To Delete a server

echo "#!/bin/ksh" > deleteserv.new chmod 700 deleteserv.new echo "date" >> deleteserv.new

for i in `cat servers` do echo "Adding in $i" echo "echo \"Deleting g_$i\"" >> deleteserv.new echo "cdr delete server g_$i --connect=$i" >> deleteserv.new echo "echo \"Return value:\$?\"" >> deleteserv.new done

for i in `cat pub` do echo "Adding in the publisher $i" echo "echo \"Deleting g_$i\"" >> deleteserv.new echo "cdr delete server g_$i" >> deleteserv.new echo "echo \"Return value:\$?\"" >> deleteserv.new done


To Define a server

echo "#!/bin/ksh" > defserv.new chmod 700 defserv.new echo "date" >> defserv.new echo "echo \"Starting Define Servers\"" >> defserv.new

for i in `cat pub` do echo "Adding in the publisher $i" echo "echo \"Defining g_$i\"" >> defserv.new echo "cdr define server --connect=$i --idle=0 --init g_$i" >> defserv.new echo "echo \"Return value:\$?\"" >> defserv.new done


for i in `cat servers` do echo "Adding in the subscriber $i" echo "echo \"Defining g_$i\"" >> defserv.new echo "cdr define server -c $i --idle=0 --init --leaf --sync=g_$pub g_$i" >> defserv.new echo "echo \"Return value:\$?\"" >> defserv.new done


To Define a template

echo "#!/bin/ksh" > deftemp.new chmod 700 deftemp.new echo "date" >> deftemp.new echo "echo \"Starting Define Template\"" >> deftemp.new

export pub=`cat pub` echo "pub = $pub"

export database=`cat database` echo "database = $database"

echo "cdr define template <templatename> --conflict=always --scope=row --master=g_$pub --database=$database --file=<filename>" >> deftemp.new

echo "echo \"Return value:\$?\"" >> deftemp.new


To Realize the template (copy the template to the other servers and turn on replication)

echo "#!/bin/ksh" > realizetemp.new chmod 700 realizetemp.new echo "date" >> realizetemp.new echo "echo \"Starting Realize Template\"" >> realizetemp.new

for i in `cat pub` do echo "Adding in the publisher $i" echo "echo \"Realizing template on g_$i\"" >> realizetemp.new echo "cdr realize template <templatename> g_$i" >> realizetemp.new echo "echo \"Return value:\$?\"" >> realizetemp.new done

export pub=$i echo "pub is $pub"

echo "Adding in the subscribers" echo "echo \"Realizing template on subscribers\"" >> realizetemp.new echo "cdr realize template <templatename>" >> realizetemp.new for i in `cat servers` do echo " g_$i" >> realizetemp.new done echo "echo \"Return value:\$?\"" >> realizetemp.new echo "YOU NEED TO MODIFY REALIZETEMP"


To Synchronize the servers

echo "#!/bin/ksh" > syncsubs.new chmod 700 syncsubs.new echo "date" >> syncsubs.new echo "echo \"Starting to Sync\"" >> syncsubs.new

for i in `cat pub` do echo "cdr sync replset -m g_$i -s <templatename> --firetrigger" >> syncsubs.new for i in `cat servers` do echo g_$i >> syncsubs.new done echo "echo \"Return value:\$?\"" >> syncsubs.new Done

echo "YOU NEED TO MODIFY SYNCSUBS"

Documents

Informix Enterprise Replication in 5 easy Steps WAIUG Forum December 8-9, 2006