Replication Server Adapter Configuration

Replication Server Adapter Configuration For Sybase Event Stream Processor By Geetha Chittaranjan And Neal Stack

Replication Server Adapter Configuration

2

TABLE OF CONTENTS

INTRODUCTION ............................................................................................................................................... 3

REPLICATION SERVER CONFIGURATION ................................................................................................... 4

ESP CONFIGURATION .................................................................................................................................... 5

TROUBLESHOOTING ...................................................................................................................................... 7 1) Discovery Failure ......................................................................................................................................... 7 2) Additional Discovery Failure ...................................................................................................................... 7 3) Unable to connect to the Replication Server Adapter ............................................................................. 7 4) Authentication Failure ............................................................................................................................... 8 5) Insufficient logging ..................................................................................................................................... 8 6) Transactions may be replayed upon restart of the project ..................................................................... 8 7) Re-materialize source data if desired ........................................................................................................ 8 8) Improve startup time for Replication Server connection to the Adapter .............................................. 8 9) Unsupported authentication methods ...................................................................................................... 9 10) No data is flowing to the RS Adapter stream ......................................................................................... 9 11) Only one row is inserted into a stored procedure stream .................................................................... 9 12) Long delays occur before a long running transaction is applied to the RS Adapter stream ............ 9 13 ) Can’t update a primary key ..................................................................................................................... 9 14) Only the first SQL statement in a batch is processed ........................................................................... 9 15) Dematerialization shuts down the RS connection to the RS Adapter ................................................. 9 16) No errors are reported upon a failed SQL command within a batch process .................................... 9 17) ESP supports a limited range of ASE ‘unsigned bigint’........................................................................ 9 18) Unsupported data types ......................................................................................................................... 10 If all else fails… .............................................................................................................................................. 11


3

INTRODUCTION

Replication Server pushes database transactions to ESP‟s Replication Server Adapter (RSA) which publishes the data to streams in the ESP server. Some of the benefits of this architecture include:

• No performance impact on the ASE database, as approach is log based

• Solution is not intrusive to source databases

• Can correlate data enterprise-wide, across multiple database instances

• Solutions are more accurate, as reference data is most current

• Less intrusive to both ESP and the database, due to avoidance of polling

The RSA is a type of server. It accepts connections from the Replication Server on a port and also understands a limited set of commands. ESP Studio can discover replication definitions and save them as schemas, streams or windows by querying the Replication Server System Database (which can be an RSSD residing in ASE or ERSSD residing in a SQL Anywhere server). In order for all of the pieces in the picture to communicate, there are some configuration steps required in the Replication Server as well as in ESP.


4

REPLICATION SERVER CONFIGURATION

1. Install Replication Server and add the database to which it needs to replicate from. For information on this please refer to the Replication Server installation document http://infocenter/help/topic/com.sybase.infocenter.dc32237.1571/doc/html/title.html 2. Add an entry to the Replication Server‟s interfaces (or sql.ini) file identifying the Replication Server Adapter. Two key things to note:

a) The name of the entry (ESP_RSadapter in this example) will be used later in the Replication

Server‟s “create connection ESP_RSadapter.ESP_Rsadapter …” command.

b) When the RSA is started, it will listen to a port identified by the RSA‟s “tdsListenerPort” parameter.

The Replication Server needs to know which server and port the RSA is running on. The interfaces

file is how the Replication Server finds the RSA:

Example interfaces entry: 3. There are 3 SQL scripts that come with ESP under the folder $ESP_HOME/adapters/repserver/config:

srsa_errors.sql

srsa_funcstrings.sql

srsa_rs_errors.sql

These 3 scripts need to be run against the Replication Server to configure it for ESP. 4. Create a connection, replication definition and subscription as per the following link: http://infocenter/help/topic/com.sybase.infocenter.dc01615.0500/doc/html/cgo1326922394780.html

NOTE: As of ESP 5.1 SP01, the “username” and “password” specified in the Replication Server “create connection” command correspond to the RSA’s “repConnUser” and “repConnPasswd” parameters. They represent the user name and password expected when receiving an incoming connection request from Replication Server. If the values do not match those specified when the connection to the adapter is defined on the Replication Server, the adapter will reject the connection. Prior to ESP 5.1 SP01, the RSA did not authenticate incoming connection requests. 5. Monitor the Replication Server log file to see if it is able to connect to the RSA.

ESP_RSadapter master tcp ether archer.company.com 7444 query tcp ether archer.company.com 7444

http://infocenter/help/topic/com.sybase.infocenter.dc32237.1571/doc/html/title.html

http://infocenter/help/topic/com.sybase.infocenter.dc01615.0500/doc/html/cgo1326922394780.html


5

ESP CONFIGURATION

NOTE: Prior to ESP 5.1 SP01, the Replication Server Adapter did not support SSL. However, the default „Typical‟ installation of ESP enables SSL. When using older versions of ESP, be sure to disable SSL. If the product has already been installed, edit the cluster configuration file (node1.xml, localnode.xml, etc.) and disable SSL. 1. Configuration of ESP‟s Replication Server Adapter is described in detail here: http://infocenter/help/topic/com.sybase.infocenter.dc01615.0510/doc/html/cgo1326922470095.html NOTES:

a) Prior to ESP 5.1 SP01, there exists a bug in the Visual Authoring mode‟s adapter configuration

whereby it will drop certain parameters and replace passwords with asterisks. Therefore, it is

recommended that you configure the adapter in Text Authoring mode in older releases of ESP.

b) Prior to ESP 5.1 SP01, the adapter did not authenticate incoming connections. As of ESP 5.1 SP01,

the adapter authenticates new connections by comparing the incoming username and password to

the RSA‟s “repConnUser” and “repConnPasswd” parameters.

2. Here is a sample configuration of the Replication Server Adapter: CREATE INPUT WINDOW InputStream1 SCHEMA ( c1 integer , c2 string , c3 timestamp , c4 float ) PRIMARY KEY (c1); ATTACH INPUT ADAPTER Adapter1 TYPE repserverplugin TO InputStream1 PROPERTIES //The RSSD (or ERSSD) database name, user and password can be found from //analyzing the <repserver>.cfg file under $SYBASE/$SYBASE_REP/install. //If the RSSD parameters are not configured correctly then schema discovery will not work. rssdHost = 'nstack-sun.company.com' , rssdPort = 5055 , rssdDatabaseName = 'nstack_repserver_RSSD' , rssdUser = 'sa' , // Comment out rssdPasswd if user's password is NULL //rssdPasswd = 'zzz' , repSubscriptionServer = 'ESP_RSadapter' , projectUri = 'esp://archer.company.com:51011/default/repserver_example' , // espUser & espPassword must match the authentication scheme // being used in the ESP Server cluster configuration file. espUser = 'nstack' , espPassword = 'xxxxxx' , // authType can be 'none' for ESP 5.0 but not for ESP 5.1 or later // authType must match the security type used in the cluster configuration file. // 'user_password' was introduced in ESP 5.1 with native OS authentication authType = 'user_password' , // tdsListenerPort matches the port in the Replication Server’s interfaces file. tdsListenerPort = 7444 , // The documentation says adminUser can be anything except do not use the // same user that was used during the create connection command in Rep Server!!! adminUser = 'saxophone' , // As of ESP 5.1 SP01, the following two parameters are introduced and required. // These match the "username" and "password" used in the Replication Server’s // "create connection ..." command: // repConnUser = 'abc' , // repConnPasswd= 'zzz' , storedProcStreamOp = 'insert' , isTransactional = FALSE , isAsync = FALSE , isBatched = TRUE , batchSize = 1000 , publishWhenBatchFull = FALSE , errorOnMissingStreamColumn = FALSE , tableName = 'AARS_example' ;

3. The RSA generates its‟ own log file which can be found in a subdirectory of the project‟s running directory. The log file can be configured to suit your needs. For details, please refer to: http://infocenter/help/topic/com.sybase.infocenter.dc01615.0510/doc/html/hag1340818796405.html

http://infocenter/help/topic/com.sybase.infocenter.dc01615.0510/doc/html/cgo1326922470095.html

http://infocenter/help/topic/com.sybase.infocenter.dc01615.0510/doc/html/hag1340818796405.html


6

(The RSA log file messages also get appended to the stdstreams.log ). 4. ESP 5.1 supports JAVA 1.7. Make sure to set the JAVA_HOME environment variable to point to $ESP_HOME/stduio/jre or to the directory of any other installation of JAVA 1.7. If JAVA_HOME is not defined properly, the RSA will not work. 5. Once configuration is complete, you may switch into Visual Authoring mode and discover tables or procedures marked for replication.


7

TROUBLESHOOTING

1) Discovery Failure If discovery fails due to an invalid configuration, an exception is raised in a pop-up window and repeated in Studio‟s Console View. The exception does not describe the reason why discovery failed but it does point to a temporary file that contains the actual reason. On Windows, the file will be written to your temporary directory with a name similar to this:

C:\Users\I825186>type c:\temp\discoveryFile_a08140

<discover><error>java.sql.SQLException: JZ006: Caught IOException:

java.net.ConnectException: Connection refused: connect</error></discover>

On UNIX, it will also be written to the temporary directory in a similar format: % cat /tmp/discoveryFile_Icvtw0

<discover><error>java.sql.SQLException: JZ00L: Login failed. Examine the SQLWarnings

chained to this exception for the reason(s).</error></discover>archer ESP-5_1/studio> 2) Additional Discovery Failure Discovery may fail if the old Replication Server syntax “create function replication definition” was used. This format is NOT supported within RS Adapter. Drop the existing subscription and replication definition and redefine the replication definition using the new Replication Server syntax “create applied function replication definition”. 3) Unable to connect to the Replication Server Adapter If the Replication Server is unable to connect to the Replication Server Adapter:

a) Make sure that the adapter is running and listening to the port specified by the adapter‟s

“tdsListenerPort” parameter. On the server where the project was started: % ps -e -o pid,args | grep java

7119 /work/nstack/nstack_ESP51/ESP-5_1/lib/jre/bin/java -classpath

/work/nstack/nstack_ESP51/ESP-5_1/adapters/repserver/libj/esp_adapter_repserve

% netstat -an | grep 7444

tcp 0 0 :::7444 :::* LISTEN

b) On the server where the Replication Server is running, try connecting to the adapter with isql and

issuing a command. Remember that prior to ESP 5.1 SP01, the adapter did not authenticate users

but starting with this release, you must use the “repConnUser” and “repConnPasswd” parameters

specified in your project: C:\Users\I825186>isql -UrepConnUser -PrepConnPasswd -S aleri_adapter -w 132 1> select @@version

2> go

@@version

------------------------------------------------------------------------------------

----------------------

Adaptive Server Enterprise/15.5/EBF 17788 SMP ESD#1--2.0.0 Beta Sybase ESP Adapter

for Replication Server

c) If the isql connection fails, this may indicate the Replication Server‟s interfaces (or sql.ini) file is

incorrect. Confirm that the entry in the interfaces file:

Matches the host name of the server where the adapter is running.

Port matches the “tdsListenerPort” specified in the CCL.

d) If the interfaces (or sql.ini) file is correct, there could be a firewall between the Replication Server

and the Replication Server Adapter. One additional test to confirm if connectivity exists between the

two is the “telnet” test:

% telnet 10.168.32.186 7444 Trying 10.168.32.186... telnet: connect to address 10.168.32.186: Connection refused

If the connection fails but you have already confirmed that it is listening to the correct port, contact your network administrator for further assistance.

e) In the Replication Server issue the “admin who” or “admin who_is_down” command to verify that the

connection status is “Awaiting Command” or “Awaiting Message”: 91 DSI EXEC Awaiting Command 107(1) ESP_adapter.ESP_adapter

90 DSI Awaiting Message 107 ESP_adapter.ESP_adapter


8

30 SQM Awaiting Message 107:0 ESP_adapter.ESP_adapter

If the connection failed to resume, review the Replication Server log file for more information.

f) Confirm that all of the username parameters in ESP are correct. See #4 for more details.

4) Authentication Failure Prior to ESP 5.1 SP1 when a connection to the Replication Server Adapter failed due to a user authentication failure, a very generic message would be logged to the “repserveradapter.log” file. ESP 5.1 SP1 enhances the authentication failure messages into three categories:

a) If espUser or espPassword is wrong and it authenticates with correct repConnUser/repConnPasswd: INFO - 11-02-2012 08:37:01.592 ERROR [Thread-1] (TdsSrv.handleLogin) Error code:400048,

Severity : 3 (Error)

INFO - Error message:Error occurred establishing connection to ESP

INFO - Error description:Please check stacktrace below for more information

INFO - javax.security.auth.login.LoginException: Authentication failure:Invalid login

credentials

b) If both espUser & espPassword are right but it authenticates with the wrong repConnUser or

repConnPasswd: INFO - 11-02-2012 08:43:56.191 ERROR [Thread-1] (TdsSrv.handleLogin) Error code:400072,


INFO - Error message:Incorrect Replication Server connection user/password specified

c) If both espUser & espPassword are right but it authenticates with the wrong adminuser or

adminpasswd: INFO - 11-02-2012 09:34:21.238 ERROR [Thread-3] (TdsSrv.handleLogin) Error code:400074,


INFO - Error message:Incorrect Adapter admin user/password specified

INFO - Error description:Incorrect Adapter admin user/password specified, login denied

5) Insufficient logging If the default logging level of the various components does not provide enough detail to assist in the resolution of the problem, you can increase the debug log level for each component:

a) Edit the project‟s CCR file and set the “debug-level” to 4.

b) Edit $ESP_HOME/adapters/fix/config/log4j.properties and set “log4j.rootLogger=DEBUG, stdout, R”.

c) Enable RS tracing to include DSI_BUF_DUMP:

trace “ON”, DSI, DSI_BUF_DUMP 6) Transactions may be replayed upon restart of the project

Non-persistent rs_lastcommit will always replay transactions.

Persistent rs_lastcommit will minimize replay of transactions.

o Persist the rs_lastcommit using a log store.

o Verify the isTransactional CCL parameter is set to “True”.

Recommend purging the RS Adapter queue (queue number) prior to a project restart. Log into the

RS and execute the following:

sysadmin hibernate_on go sysadmin sqm_purge_queue, 107,0 go sysadmin hibernate_off go

NOTE: The queue number (107) should correspond with the queue number of your ESP RS Adapter

connection on the Replication Server. Execute an “admin who” to determine the queue number.

7) Re-materialize source data if desired Drop and recreate subscriptions

8) Improve startup time for Replication Server connection to the Adapter Suspend the RS connection to the RS Adapter prior to stopping the model.

A suspend forces an update to the persistent rs_lastcommit.


9

Improves startup time for RS connecting to the RS Adapter.

9) Unsupported authentication methods Use of unsupported authentication methods (Kerberos, RSA) will result in failure of the adapter to connect to the ESP server. Please see CR 720700. 10) No data is flowing to the RS Adapter stream If no data flows following a full configuration of the RS replication definition, subscription and RS Adapter streams.

Verify the ASE source table has been marked for replication using “sp_setreptable”. On the ASE

source table, use the following example to verify the table has been marked successfully:

sp_setreptable „TESTTABLE‟ The following message will be returned:

The replication status for „TESTTABLE‟ is currently true, owner_off. (return status = 0)

11) Only one row is inserted into a stored procedure stream

Verify the storedProcStreamOp parameter for the RS Adapter has been defined as insert.

Verify the autogen parameter for ra_pkey column has been set to True for the stored procedure

stream.

12) Long delays occur before a long running transaction is applied to the RS Adapter stream When using the RS Adaper “isBatching” parameter is true, long running transactions may see

delays while the RS Adapter waits for the RS to send the final commit. This is an expected behavior.

13 ) Can’t update a primary key Currently there is not solution. CR 652066 has been opened with engineering to address the issue.

14) Only the first SQL statement in a batch is processed The dsi_command_separator must be defined as “;”. If it isn‟t, all statements beyond the first SQL

will be interpreted as garbage and thrown out.

15) Dematerialization shuts down the RS connection to the RS Adapter Dematerialization is not supported. To clear the dematerialization queue, perform the following:

o Identify the dematerialization queue using an “admin who” on the RS.

o Execute a drop_queue similar to the following:

sysadmin drop_queue, 107, -2147483468 go

16) No errors are reported upon a failed SQL command within a batch process

This is an expected behavior and a result of the ESP publisher API. The ESP publisher API and RS

Adapter do not wait for a confirmation on each transaction, thus a performance gain.

17) ESP supports a limited range of ASE ‘unsigned bigint’ The ESP Replication Server Adapter can map to the ASE „unsigned bigint‟ data type but it can only handle numbers in the range of a normal signed bigint. If ESP tries to read outside of this range, the following error is raised: Message from server: Message: 32000, State 0, Severity 20 -- 'java.lang.NumberFormatException: For

input string: "18446744073709551615"'.

I. 2012/10/08 11:41:05. Database 'aleri_adapter.aleri_adapter' returns messages that are mapped to

IGNORE or WARN by error action mapping. See logged data server messages for more information.

10-08-2012 17:18:07.877 DEBUG [Thread-2] (TdsSrv.handleLanguage) statement: begin transaction ;

insert into RSA_big_example (c1, c2, c3) values (9223372036854775807, 18446744073709551615,

'20121008 11:41:04.577801')


10

10-08-2012 17:18:07.877 DEBUG [Thread-2] (TdsSrv.handleLanguage) statementNum: 2

10-08-2012 17:18:07.877 DEBUG [Thread-2] (TdsSrv.handleLanguage) statementNum: 2

10-08-2012 17:18:07.878 ERROR [Thread-2] (StreamPublication.setStringColumnByIndex) Error

code:400060, Severity : 3 (Error)

Error message:Error parsing LONG value for column 1

Error description:Error parsing LONG value for column 1

java.lang.NumberFormatException: For input string: "18446744073709551615"

at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)

at java.lang.Long.parseLong(Long.java:444)

at java.lang.Long.parseLong(Long.java:483)

at

com.sybase.esp.adapter.repserver.StreamPublication.setStringColumnByIndex(StreamPublication.java:364

)

at

com.sybase.esp.adapter.repserver.StreamPublication.setStringColumnByName(StreamPublication.java:557)

at com.sybase.esp.adapter.repserver.TdsSrv.handleLanguage(TdsSrv.java:402)

at com.sybase.jdbc4.tds.SrvReceiver.language(Unknown Source)

at com.sybase.jdbc4.tds.SrvSession.fireEvent(Unknown Source)

at com.sybase.jdbc4.tds.SrvSession.processRequests(Unknown Source)

at com.sybase.jdbc4.tds.SrvSession.run(Unknown Source)

at java.lang.Thread.run(Thread.java:722)

18) Unsupported data types At run time the RSA will throw a parsing exception on unsupported data types. For example, Oracle timestamp(9) runs into this issue because the adapter expects bigdatetime data in yyyymmddd hh:mm:ss format (not like Oracle‟s internal format that ESP receives from Replication Server). To get around the parsing issue we need to configure Replication Server in such a way that it sends timestamp data in a format which ESP can understand. For example, ESP understands ASE‟s bigdatetime format so you can simply map Oracle‟s timestamp(9) to ASE‟s bigdatetime format. You can do this in two ways: on a per table basis or at the database level (all the tables). Here are the instructions to do this change at the database level: /*

** Query the RSSD to get the ESP function string class ID

** You must do this AFTER the 'aars_function_class' has been created

*/

select classid from rs_classes where classname = 'aars_function_class'

go

classid

------------------

0x0100006501000066

/*

**************************************************************************************

* Replace 0x0100006501000066 with the results of the class ID from your environment

* to add the translations from

* rs_oracle_timestamp9 to bigdatetime

**************************************************************************************

*/

delete from rs_translation where classid = 0x0100006501000066 and source_dtid = 0x000000000001020f

insert into rs_translation (prsid, classid, type, source_dtid, target_dtid, target_length,

target_status, rowtype)

values(

0, /* primary rep server ID */

0x0100006501000066, /* function string class (aars_function_class)*/

'D', /* type */

0x000000000001020f, /* source data type ID ( rs_oracle_timestamp9 ) */

0x000000000000011d, /* target data type ID ( rs_rs_bigdatetime ) */

29, /* max target_length */

0, /* target status */

0)

Go

NOTE: Rep Server must be restarted for this change to take affect. You can manually see what the result of the translations should look like (what will be sent to CEP) by using the Rep Server admin translate command: 1> admin translate, '02/25/2006 13.35.41.123456789', rs_oracle_timestamp9, rs_rs_bigdatetime

2> go

Delimiter Prefix Translated Value Delimiter Postfix


11

------------------------------- ------------------------- -------------------------------

' 20060225 13:35:41.123456 '

If all else fails… Be prepared to send the following files to technical support:

a) The Replication Server log file typically located in $SYBASE/$SYBASE_REP/install

b) The Replication Server Adapter log file normally located in $ESP_HOME/cluster/<base-

directory>/<workspace_name>.<project_name>.<instance>/logs/repserveradapter.log

c) The log files for the ESP project ESP_HOME/cluster/<base-

directory>/<workspace_name>.<project_name>.<instance>/esp_server.log and

ESP_HOME/cluster/<base-

directory>/<workspace_name>.<project_name>.<instance>/stdstreams.log

© 2013 SAP AG. All rights reserved.

SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP

BusinessObjects Explorer, StreamWork, SAP HANA, and other SAP

products and services mentioned herein as well as their respective

logos are trademarks or registered trademarks of SAP AG in Germany

and other countries.

Business Objects and the Business Objects logo, BusinessObjects,

Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and

other Business Objects products and services mentioned herein as

well as their respective logos are trademarks or registered trademarks

of Business Objects Software Ltd. Business Objects is an SAP

company.

Sybase and Adaptive Server, iAnywhere, Sybase 365, SQL

Anywhere, and other Sybase products and services mentioned herein

as well as their respective logos are trademarks or registered

trademarks of Sybase Inc. Sybase is an SAP company.

Crossgate, m@gic EDDY, B2B 360°, and B2B 360° Services are

registered trademarks of Crossgate AG in Germany and other

countries. Crossgate is an SAP company.

All other product and service names mentioned are the trademarks of

their respective companies. Data contained in this document serves

informational purposes only. National product specifications may vary.

These materials are subject to change without notice. These materials

are provided by SAP AG and its affiliated companies ("SAP Group")

for informational purposes only, without representation or warranty of

any kind, and SAP Group shall not be liable for errors or omissions

with respect to the materials. The only warranties for SAP Group

products and services are those that are set forth in the express

warranty statements accompanying such products and services, if

any. Nothing herein should be construed as constituting an additional

warranty.

www.sap.com

Documents

Replication Server Adapter Configuration