TDV Technical Note - Capabilities - TIBCO Software...• Using System Adapters and Custom Adapters to alter data source behavior ... • A valid JDBC/ODBC driver and associated JAR

1

Professional

Services

www.tibco.com

Global Headquarters 3303 Hillview

Avenue Palo Alto, CA 94304

Tel: +1 650-846-1000 +1 800-420-8450 Fax: +1 650-846-1005

TIBCO Software empowers

executives, developers, and

business users with Fast Data

solutions that make the right data

available in real time for faster

answers, better decisions, and

smarter action. Over the past 15

years, thousands of businesses

across the globe have relied on

TIBCO technology to integrate their

applications and ecosystems,

analyze their data, and create real-

time solutions. Learn how TIBCO

turns data—big or small—into

differentiation at www.tibco.com.

TIBCO Data Virtualization Technical Note Capabilities - Customizing Data Source Properties

Project Name

Release 2.0

Date April 2018

Primary Author

Manny Bhatia

Document Owner

Tony Young

Client

Document Location

Purpose Demonstrates how capabilities and

behavior of a data sources can be changed in TIBCO Data Virtualization.

TDV Technical Note - Capabilities

© Copyright TIBCO Software Inc. 2 of 19



Revision History

Version Date Author Comments

1.0 May 2010 Manny Bhatia Initial revision

2.0 April 2018 Deane Harding Updated with TIBCO branding

Approvals This document requires the following approvals. Signed approval forms are filed in the project files.

Name Signature Title Company Date of Issue Version

Distribution This document has been distributed to:

Name Title Company Date of Issue Version

Related Documents This document is related to:

Document File Name Author

TDV User’s Guide TIB_tdv_7.0.6_UsersGuide.pdf TIBCO DV Engineering



Copyright Notice COPYRIGHT© TIBCO Software Inc. This document is unpublished and the foregoing notice is affixed to protect TIBCO Software Inc. in the event of inadvertent publication. All rights reserved. No part of this document may be reproduced in any form, including photocopying or transmission electronically to any computer, without prior written consent of TIBCO Software Inc. The information contained in this document is confidential and proprietary to TIBCO Software Inc. and may not be used or disclosed except as expressly authorized in writing by TIBCO Software Inc. Copyright protection includes material generated from our software programs displayed on the screen, such as icons, screen displays, and the like.

Trademarks All brand and product names are trademarks or registered trademarks of their respective holders and are hereby acknowledged. Technologies described herein are either covered by existing patents or patent applications are in progress.

Confidentiality The information in this document is subject to change without notice. This document contains information that is confidential and proprietary to TIBCO Software Inc. and its affiliates and may not be copied, published, or disclosed to others, or used for any purposes other than review, without written authorization of an officer of TIBCO Software Inc. Submission of this document does not represent a commitment to implement any portion of this specification in the products of the submitters.

Content Warranty The information in this document is subject to change without notice. THIS DOCUMENT IS PROVIDED "AS IS" AND TIBCO MAKES NO WARRANTY, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING BUT NOT LIMITED TO ALL

WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. TIBCO Software Inc. shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance or use of this material.

Export This document and related technical data, are subject to U.S. export control laws, including without limitation the U.S. Export Administration Act and its associated regulations, and may be subject to export or import regulations of other countries. You agree not to export or re-export this document in any form in violation of the applicable export or import laws of the United States or any foreign jurisdiction.

For more information, please contact:

TIBCO Software Inc. 3303 Hillview Avenue Palo Alto, CA 94304

USA



Table of Contents 1 Introduction ............................................................................................................... 6

1.1 Purpose ..................................................................................................... 6

1.2 Audience ................................................................................................... 6

2 TDV Query Engine – Query Optimizer Process ........................................................ 7

3 Data Sources and Their Capabilities ......................................................................... 9

3.1 What is a Capabilities File? ....................................................................... 9

3.2 Sample Contents of a Capabilities File.................................................... 10

3.2.1 Driver Capabilities ........................................................................................................ 11 3.2.2 Introspection Capabilities ............................................................................................. 11 3.2.3 JDBC Data Source Capabilities .................................................................................... 12 3.2.4 Native-to-TDV and TDV-to-Native Data Type Mapping ................................................. 13 3.2.5 Expression Capabilities ................................................................................................ 13 3.2.6 Comparison Capabilities............................................................................................... 14 3.2.7 Supported Function Patterns ........................................................................................ 14 3.2.8 Operator Capabilities.................................................................................................... 15

3.3 How Capabilities Files Are Used ............................................................. 15

3.4 Modifying Default Capabilities of a Data Source ..................................... 16

3.5 Effect of Patches on Capabilities File ...................................................... 17

3.6 Data Source Adapters – System and Custom Adapters.......................... 17

3.7 Best Practices Regarding Modification of Data Source Capabilities ....... 18



1 Introduction

1.1 Purpose The purpose of this document is to understand how capabilities and behavior of a data sources can be changed in TIBCO Data Virtualization (TDV). The need to alter the default behavior may be either to implement a specific use case, for example, where case sensitivity needs to be altered, or to add new functionality to a data source capability.

This document discusses topics regarding:

• TDV Query Engine process

• TSV Query Engine use of capabilities files.

• Various Sections in TSV Capabilities files

• Process of modification of capabilities files.

• Effects of modification of the capabilities files

• Effects of patches and upgrades to capabilities files

• Using System Adapters and Custom Adapters to alter data source behavior

• Best practices to modify data source behavior

1.2 Audience This document is intended to provide guidance to the following users:

• TIBCO Professional Services

• TDV Architects

• TDV Developers



2 TDV Query Engine – Query Optimizer Process

TDV Query Optimizer is the brain behind the TDV Server with its main purpose to analyze, rewrite and execute queries on the underlying data sources. The TDV query optimizer includes a Rules Based and Cost based optimizer, which once invoked by an incoming query performs the following functions:

• Checks the query SQL to validate the syntax

• Identifies resources that are used in the query

o Checks if resources are valid

o Checks if resources is cached and cache is valid

• Identifies the data sources used by the resources

o Checks if data source is enabled and

o Checks if statistics have been enabled and they are valid

• In case of a federated query, rewrites the query for each data source with native syntax.

• Submits the query on the data source connection

• Retrieves the result sets from individual data sources

• Aggregates the results returned from disparate data sources and apply required joins, groupings, sorting or transformations.

• Streams the data back to the calling client.

Figure 1: TDV Query Life Cycle



For TDV query optimizer to perform the above functions, it needs to know capabilities and behavior of the data sources. This is accomplished by reading capabilities files for each data source. Following sections discuss these capabilities files, how TDV server utilizes these capabilities file and finally, how data source behavior can be modified through the capabilities files.



3 Data Sources and Their Capabilities

Before a data source can be used in TDV server, it must be set up and introspected. Data source setup and introspection is described in detail within the TDV User’s Guide. In summary, setting up a relational data source in TDV requires the following steps before it can be used:

• A valid JDBC/ODBC driver and associated JAR files. Some drivers for data sources such as Oracle, Sybase and MySQL come pre-packaged with TDV, while others such as DB2, SQL Server need to be installed.

• Capabilities file, which is a text file that tells the TDV server various capabilities of the data source and the features supported by the data source.

• In the case when a relational data source is not natively supported in TDV and data source does not show up in Data Source creation dialog box, a custom data source may be created based on a generic data source

adapter as long as it supports JDBC and has a JDBC driver.

3.1 What is a Capabilities File? Capabilities files are text files, and as the name suggests they describe the capabilities of a specific data source. These capabilities files determine how a data source will behave during introspection and query execution. We discuss the workings of capabilities file in the following sections.

Every data source used by TDV needs to have its own capabilities file. For example, on a Windows server, capabilities file for Oracle data sources would be located under the C:/Program Files/TIBCO/TDV Server 7.0/apps/dlm/cis_ds_oracle/conf folder, where C:/Program Files/TIBCO/TDV Server 7.0 is the TDV install folder.

Depending on number of data source versions supported by TDV, there could be multiple capabilities files under the ‘conf’ folder. Further, if patches have been applied to TDV server and capabilities files have been modified, multiple versions of these files would be present in the folder. Each of these files follow a naming standard that includes <data source driver version> and <TDV patch number>. The figure below shows a screen shot of capabilities file for Oracle

data source.



The above example shows the following details about capabilities:

• Base Oracle capabilities file (oracle-<patch version>.capabilities).

• Capabilities files available for Oracle thin (JDBC) driver for Oracle versions 8i, 91, 10g, 11g and 12c. (oracle_<oracle version>_thin_driver.oracle-<patch version>capabilities)

• Multiple versions of capabilities files for each patch installed on the system (Patch 7.0.5.00.05; 7.0.6.00.00).

• XML version of driver configuration files for each Oracle version (oracle_<oracle version>_thin_driver_config_def.xml).

3.2 Sample Contents of a Capabilities File As noted above, entries in capabilities file determine the behavior of the data source during introspection and also during query execution. This section discusses the contents of capabilities file and how each section of capabilities file is used. For simplicity purposes, we use Oracle data source related capabilities files.

A capabilities file is divided into logical sections, each focusing on certain behavior. Depending on type of data source, these sections may be unnecessary. Examples of these sections are as follows (Note: All capabilities mapping shown are partial sets):



3.2.1 Driver Capabilities As name states, the entries in this section are related to the JDBC/ODBC driver capabilities. Driver properties such as URL pattern, Class name etc. are listed here and when a TDV data source is created, the data source properties are read from this section

#Driver capabilities

dsdriver.classname.oracle_8i_(thin_driver): oracle. jdbc.OracleDriver

dsdriver.url_pattern.oracle_8i_(thin_driver):jdbc:o racle:thin:@<HOST>:<PORT>:<DATABASE_NAME>

Depending on the type of data source, the URL pattern may be different. The table below shows some of the patterns used in TDV while connecting using a JDBC Driver

Data Source Name URL Pattern

Oracle Thin (8i, 9i, 10g, 11g) jdbc:oracle:thin@://<host>:<port>/<service name>

Oracle Thin with Load Balancer jdbc:oracle:thin@://<TNS Entry>

Oracle OCI (8i, 9i, 10g, 11g) jdbc:oracle:thin@://<TNS SID>

DB2 version 8, 9 (Type 4),

DB2 z/OS

jdbc:db2://<host>:<port>/<database name>

DB2 (type 2) jdbc:db2:< database name >

MySQL (4.1, 5) jdbc:mysql://<host>:<port>/<database name>

SQL Server (2000, 05, 08) jdbc:microsoft:sqlserver://<host>:<port>;databaseName=

<DATABASE_NAME>;SelectMethod=<SELECT_MODE>

Sybase jdbc:sybase:Tds::/

Netezza jdbc:netezza://<host>:<port>/<database name>

Teradata jdbc:teradata:///DBS_PORT=/DATABASE= <DATABASE_NAME>/CHARSET=UTF8,COMPAT_DBS=true

3.2.2 Introspection Capabilities This section has entries that the TDV server will use during introspection and re-introspection of a data source. Values in this section will tell the TDV server if the data source supports catalogs and schemas, and if to introspect tables, procedures, primary keys, indexes etc.

#Introspection capabilities

introspect.relational.include_catalog: false



introspect.relational.include_schema: true

introspect.relational.include_table: true

introspect.relational.include_table.primary_keys: t rue

introspect.relational.include_table.indexes: true

introspect.relational.include_table.foreign_keys: t rue

introspect.relational.include_table.foreign_keys.im ported: true

introspect.relational.include_procedure: true

introspect.parameter.length: data_length

introspect.parameter.precision: data_precision

introspect.parameter.scale: data_scale

3.2.3 JDBC Data Source Capabilities Entries in this section tell the TDV query optimizer as how the JDBC driver is expecting the SQL statement, what SQL syntax is supported.

jdbcds.alias_generation_pattern: expr$1

jdbcds.catalogs_in_tables_for_updates: false

jdbcds.delimited_identifier_characters_pattern: (?i )(DATE|.*#.*)

jdbcds.delimited_identifier_closing_char: "

jdbcds.delimited_identifier_opening_char: "

jdbcds.schema_delimited_identifier_opening_char: "

jdbcds.schema_delimited_identifier_closing_char: "

jdbcds.table_delimited_identifier_opening_char: "

jdbcds.table_delimited_identifier_closing_char: "

jdbcds.column_delimited_identifier_opening_char: "

jdbcds.column_delimited_identifier_closing_char: "

jdbcds.procedure_delimited_identifier_opening_char: "

jdbcds.procedure_delimited_identifier_closing_char: "

jdbcds.insert_default_string: ''

jdbcds.max_number_of_in_operator_values: 1000

jdbcds.no_cross_join_with_long: true

jdbcds.no_support_for_null_projections_in_set_opera tion: true

jdbcds.promotes_integer_division_to_float: true

jdbcds.schemas_in_tables: true

jdbcds.schemas_in_tables_for_updates: true

jdbcds.subqueries_in_from: true

jdbcds.tables_in_columns: true

jdbcds.trailing_spaces_included_in_comparison: true

ds.conn_checkout_timeout: 30

jdbcds.with_clause: true

jdbcds.support_ddl: true

jdbcds.support_ddl.create_table: true

jdbcds.support_ddl.drop_table: true

jdbcds.support_ddl.alter_table: true

jdbcds.support_ddl.create_index: true

jdbcds.support_ddl.drop_index: true

jdbcds.ddl_cascade_restrict_support: true

jdbcds.ddl_drop_table_does_not_exist_error_string: ORA-00942

jdbcds.support_quantified_comparisons: true



jdbcds.no_support_xml_types_with_distinct: true

jdbcds.set_statement_escape_processing_false: true

jdbcds.no_shared_connection_in_procedure_call: fals e

3.2.4 Native-to-TDV and TDV-to-Native Data Type Map ping This section outlines how JDBC data types are mapped to TDV data types and vice-versa.

jdbcds.datatype.bfile: blob

jdbcds.datatype.blob: blob

jdbcds.datatype.char: char

jdbcds.datatype.clob: clob

jdbcds.datatype.date: timestamp

jdbcds.datatype.float: float

jdbcds.datatype.long: clob

jdbcds.datatype.nchar: char

jdbcds.datatype.number: decimal

jdbcds.datatype.varchar2: varchar

jdbcds.datatype.max_binary_length: 255

jdbcds.datatype.max_varbinary_length: 255

jdbcds.datatype.max_char_length: 2000

jdbcds.datatype.max_varchar_length: 4000

jdbcds.datatype.max_precision: 38

jdbcds.datatype.min_binary_length: 1

jdbcds.cast_function_max_precision: 38

jdbcds.cast_function_max_scale: 38

composite.datatype.bigint: number(&p, 0)

composite.datatype.binary: raw(&l)

composite.datatype.binary_promote: blob

composite.datatype.bit: number(&p, 0)

composite.datatype.blob: blob

composite.datatype.boolean: number(1,0)

composite.datatype.char: char(&l)

composite.datatype.char_promote: clob

composite.datatype.date: varchar(10)

composite.datatype.decimal: number(&p, &s)

composite.datatype.double: varchar(24)

composite.datatype.float: float

composite.datatype.integer: number(&p, 0)

composite.datatype.numeric: number(&p, &s)

composite.datatype.varchar: varchar(&l)

composite.datatype.varchar_promote: clob

composite.datatype.xml: clob

composite.datatype.interval_day: varchar(30)

composite.datatype.interval_day_to_hour: varchar(30 )

3.2.5 Expression Capabilities All expression patterns supported by the data source and how they are used in TDV are listed in this section.

expr.and_pattern: $1 AND $2



expr.or_pattern: $1 OR $2

expr.not_pattern: NOT ($1)

expr.like_pattern: $1 LIKE $2

expr.in_pattern: $1 IN ($2)

expr.exists_pattern: EXISTS ($1)

expr.like_pattern.wildcard: %

#expr.like_pattern.quote_char: '$1'

expr.like_pattern.single_char_wildcard: _

expr.ne_pattern: $1 <> $2

expr.eq_pattern: $1 = $2

expr.lt_pattern: $1 < $2

expr.gt_pattern: $1 > $2

expr.le_pattern: $1 <= $2

expr.ge_pattern: $1 >= $2

expr.is_null_pattern: $1 IS NULL

expr.is_not_null_pattern: $1 IS NOT NULL

expr.escape_character:

3.2.6 Comparison Capabilities All comparison patterns supported by the data source and how they are used in TDV are listed in this section.

comp.like_pattern:

(~string,~string)||(raw,~string)||(~number,~string) ||(@timestamp,~string)||(raw,~string)

comp.is_null_pattern: (~any)

comp.is_not_null_pattern: (~any)

comp.ne_pattern:

(~number,~number)||(@char,@char)||(@char,~number)|| (@char,raw)||(@char,@varchar)||(@char,@timestamp

)||(@varchar,@varchar)||(varchar,~number)||(varchar 2,~number)||(nvarchar,~number)||(nvarchar2,~numb

er)||(varchar,raw)||(varchar2,raw)||(nvarchar,raw)| |(nvarchar2,raw)||(varchar,@timestamp)||(varchar

2,@timestamp)||(nvarchar,@timestamp)||(nvarchar2,@t imestamp)||(@timestamp,@timestamp)||(raw,raw)||(

mlslabel,mlslabel)||(@date,@timestamp)||(@timestamp ,@date)

comp.eq_pattern:






comp.lt_pattern:






3.2.7 Supported Function Patterns This section contains list of functions that are supported by the data source. Along with the function signature, its usage in TDV is also shown. All functions may have variations on usage and thus multiple occurrences of a same function with different data type may be seen.

AVG(@char).distinct: AVG(DISTINCT $1)

CONCAT(~number,~number): (CASE WHEN ($1 IS NULL) OR ($2 IS NULL) THEN NULL ELSE

CONCAT($1, $2) END)



LOWER(@null): LOWER($1)

REPLACE(@char,@char,@char): REPLACE($1, $2, $3)

POSITION(~numberIN~number): INSTR($2, $1)

CAST(@timestampAS@timestamp): TO_DATE($1)

CAST(@timestampAS@null): NULL

TO_CHAR(~number): TO_CHAR($1)

TO_DATE(@varchar): TO_DATE(SUBSTR($1, 1, 10), 'YYYY -MM-DD')

CURRENT_TIMESTAMP(): SYSDATE

DAY(@date): TO_NUMBER(TO_CHAR($1, 'DD'))

ABS(@char): ABS($1)

CEILING(~number): CEIL($1)

CEILING(@null): CEIL($1)

EXP(~number): EXP($1)

EXP(@null): EXP($1)

FLOOR(@char): FLOOR($1)

3.2.8 Operator Capabilities This section lists all operators supported by the data source.

@char||@char: (CASE WHEN ($1 IS NULL) OR ($2 IS NUL L) THEN NULL ELSE ($1 || $2) END)

@char||@varchar: (CASE WHEN ($1 IS NULL) OR ($2 IS NULL) THEN NULL ELSE ($1 || $2) END)

@timestamp||@varchar: (CASE WHEN ($1 IS NULL) OR ($ 2 IS NULL) THEN NULL ELSE ($1 || $2) END)

@timestamp||@null: NULL

ABS(@char): ABS($1)

ABS(@varchar): ABS($1)@null/@char: ($1 / $2)

@varchar/@varchar: ($1 / $2)

@varchar/~number: ($1 / $2)

@varchar/@null: ($1 / $2)

3.3 How Capabilities Files Are Used At server start up, the TDV server loads all capabilities files into memory. As noted above, there might be multiple versions of capabilities files. The order in which these files are used is very important to the behavior of the TDV query engine. The general rule governing the usage Capabilities file by the TDV query engine is as follows (For example purposes, Oracle 10g data source capabilities files shown in figure above will be used).

• TDV will read the base capabilities file which follows the pattern <data source name>.capabilities: oracle.capabilities

• Next, TDV will read the driver specific capabilities file and overlay the contents of driver specific file over the base data source file loaded in step 1. This file follows a pattern <data source name>_<data source version>_<driver type/version>.<data source type>.capabilities: oracle_10g_thin_driver.oracle.capabilities

• Finally, TDV will read the capabilities file with the highest patch level for the specific data source and overwrite the contents of created by step 2. This file usually follow a pattern <data source name>_<data source

version>_<driver type/version>.<data source type>-<patch number>.capabilities: oracle_10g_thin_driver.oracle-0600.capabilities

• Starting way back in TDV version 4.5, the concept of adapters was introduced that allows capabilities to be updated via an XML file for each data source adapter rather than through these capabilities files. Therefore, as a last step in this process the XML files will be loaded. Adapters are discussed in following sections.



3.4 Modifying Default Capabilities of a Data Source The above example of a capabilities files demonstrates that all of the entries are name value pairs; which describe a

property and its associated value. Once loaded by the TDV Server, the query engine can then determine how to rewrite a query using the data source capabilities.

For example, when working with Oracle, the query engine knows from the capabilities file, that the data source supports ORDER BY thereby allowing it to use SORT MERGE algorithm when performing a federated join. Similarly, TDV query engine can also identify through the capabilities file, that Oracle supports functions such as RTRIM, TO_DATE, DECODE etc and can push them down.

On the other hand, POSITION function in TDV is translated to INSTRING function of Oracle and TDV query engine can rewrite the POSITION function to INSTRING based on the pattern in capabilities file.

By adding or modifying the Capabilities file, new functionality can be added to a data source or existing behavior can be modified. For example, if a new data type VARCHAR2 needs to be added in Oracle capabilities file, the following line can be added (in Native-to-TDV data type mapping section)

jdbcds.datatype.varchar2: varchar

The above line tells the TDV server that Oracle VARCHAR2 data type is equivalent to VARCHAR in TDV.

Occasionally there is a need to expose a new function capability in a data source. This may be required to push the SQL with a function to the data source. Adding a new data source functionality capability requires multiple steps as

follows:

• A function with the same name and signature as data source function is created in TDV using TDV Studio.

This function may be a placeholder with a valid signature or it could be a fully implemented SQL script procedure.

If the function is a placeholder, only the data source that has the functional capability may use the new functionality and all other data sources will return an error. On the other if the same function is implemented in TDV, when invoked within a SQL, Query Engine will decide if the function can be pushed to the data source. DECODE function is an example of such function where it has been implemented in TDV but is also a built in function in Oracle.

• The created function must be enabled as a global function. This may be done from Administration -> Custom Functions menu.

NOTE: User must be part of an ADMIN group to create a global function.

• The capabilities file must be changed to show the usage. Below is an example of DECODE function in Oracle

DECODE\\$\(\\\~string|string|@char|@varchar|char\\ \(\\\&l\\$|nchar\\$\\\&l\\$|varchar2\\$\\\&l

\\$|nvarchar2\\$\\\&l\\$|\\\~number|\\\~whole_nu mber|@bit|@integer|@smallint|@tinyint|@bigint|\\

\~floating_point|@float|@real|@double|@decimal|@num eric|\\\~date|@date|@time|@timestamp|\\\~interva

l_year|\\\~interval_day|@xml|@null|char|varchar|var char2|bit|integer|number|smallint|tinyint|bigint

|float|real|double|decimal|numeric|date|time|timest amp|xml|null\)$,\(\\\~string|string|@char|@varc

har|char\\\(\\\&l\\$|nchar\\$\\\&l\\$|varchar2\\ $\\\&l\\$|nvarchar2\\$\\\&l\\$|\\\~number|\\

\~whole_number|@bit|@integer|@smallint|@tinyint|@bi gint|\\\~floating_point|@float|@real|@double|@de

cimal|@numeric|\\\~date|@date|@time|@timestamp|\\\~ interval_year|\\\~interval_day|@xml|@null|char|v

archar|varchar2|bit|integer|number|smallint|tinyint |bigint|float|real|double|decimal|numeric|date|t

ime|timestamp|xml|null\)\)*\\\).variable : DECODE$ $1 $*1*,*$



NOTE: A TDV Server restart is required for new/modified capabilities to take effect

3.5 Effect of Patches on Capabilities File In the previous two sections, it was discussed that how we can modify the capabilities and how the TDV server uses the capabilities file. Occasionally, patches are installed on the TDV server to provide new functionality or for bug fixes. These patches may sometimes include new capabilities files for one or more data sources.

As the TDV server loads the latest patched capabilities file, any custom modifications done to the existing capabilities files may be over written once a patch installation creates as new capabilities file version. This becomes a maintenance nightmare if TDV administrators have to manually manage the changes in the capabilities files.

3.6 Data Source Adapters – System and Custom Adapte rs Starting way back in TDV version 4.5, TIBCO introduced a concept of Data Source Adapters. Data source adapters provide an easy interface to manage data source customizations. Using the adapters, default capabilities of a relational data sources can be modified without the need to modify the capabilities file as discussed previously. Custom adapters allow similar modifications when required changes to capabilities are limited.

Details on creating and managing adapters are discussed in the “Working with Data Sources” section of the TDV User’s Guide. For purpose of this document, both System and Custom adapters are created under <TDV_HOME>/conf/adapters folder. The system subfolder contains folders for each supported data source version in

TDV with a custom XML values file outlining the capabilities of the data source driver. The example below shows the

structure for Oracle 12c Thin JDBC drivers.

The format of the XML file in this folder is as follows. By default usually the whole file is commented out using XML

Syntax which indicates no overrides are enabled in the XML file and TDV should use the default values from the capabilities files instead. However, here you can see that some overrides have been made into the Oracle 12c



values.xml file to enable some mappings of TDV-defined custom functions to Oracle functions to enable push-down of certain operations:

<?xml version="1.1" encoding="UTF-8"?>

<common:attributes xmlns:common="http://www.composi tesw.com/services/system/util/common">

<common:attribute xmlns:common="http://www.composit esw.com/services/system/util/common">

<common:name>/custom/BLOB_TO_XML(~any,~any)</comm on:name>

<common:type>STRING</common:type>

<common:value>XMLType($1, $2).getClobVal()</commo n:value>

<common:configID>BLOB_TO_XML(~any,~any)</common:c onfigID>

</common:attribute>

<ns1:attribute xmlns:ns1="http://www.compositesw.co m/services/system/util/common">

<ns1:name>/custom/DECOMPRESS_BLOB(~any)</ns1:name >

<ns1:type>STRING</ns1:type>

<ns1:value>utl_raw.cast_to_varchar2(UTL_COMPRESS. LZ_UNCOMPRESS($1))</ns1:value>

<ns1:configID>DECOMPRESS_BLOB(~any)</ns1:configID >

</ns1:attribute>

<ns2:attribute xmlns:ns2="http://www.compositesw.co m/services/system/util/common">

<ns2:name>/custom/jdbcds.can_cancel_statement_on_ cursor_close</ns2:name>

<ns2:type>BOOLEAN</ns2:type>

<ns2:value>true</ns2:value>

<ns2:configID>jdbcds.can_cancel_statement_on_curs or_close</ns2:configID>

</ns2:attribute></common:attributes>

At TDV server startup, this data source XML file is loaded after the base data source capabilities and patched version capabilities files and it overwrites the contents of the capabilities files.

If any further changes are required to the capabilities, either the required sections may be un-commented or new changes can be added in a valid XML syntax. The example below of modified files shows how a capability showing

support for IN clause can be implemented:

<?xml version="1.1" encoding="UTF-8"?>

<common:attributes mlns:common= "http://www.composi tesw.com/services/system/util/common">

<common:attribute xmlns:common="http://www.composit esw.com/services/system/util/common">

<common:name>/custom/expr.in_pattern</common:name >

<common:type>STRING</common:type>

<common:value>$1 IN ($2)</common:value>

<common:configID>expr.in_pattern</common:configID>

</common:attribute>

The default XML file does not contain all capabilities of a specified data source and thus to make required changes, it may be necessary to get the correct XML code from TIBCO Support. A field-implemented utility is also available, which converts the text capabilities file to its equivalent XML format.

NOTE: TDV server should be restarted for the capabilities changes to take effect.

3.7 Best Practices Regarding Modification of Data S ource Capabilities Allowing modification of data source capabilities in the field provides flexibility and adds tremendous advantage to TDV, but it also adds maintenance complexity. Some issues that come up are

• As changes to capabilities file effects the data source and all connections to that data sources, thus it can

have detrimental effect on other application if changes are not properly tested.



• If the changes to capabilities files are not properly documented and communicated, troubleshooting may become difficult.

• If changes to capabilities file enable new functionality, the field team or customer may not communicate it back to TIBCO, thereby making the install out of sync with the core product.

• Any patches applied to a TDV instance may create a new capabilities file thereby overwriting the changes

Therefore, TIBCO Professional Services best practices strongly discourage making changes to the capabilities file

unless absolutely necessary. If the changes are required, they should be communicated to TIBCO Support for logging and general testing purpose.

On the other hand, if changes are required to the capabilities, XML Capabilities file under System or Custom adapters.

Depending on type of modification, either System or Custom adapters should be used. For adding new functionality

that will enhance the data source, like DECODE in Oracle, TIBCO Professional Services best practice recommends modifying the System adapters. On the other hand, if data source behavior needs to be changed, as with case

sensitivity or trailing spaces settings, then creation of custom adapters are recommended.

Finally, some other advantages of using the adapter capabilities files are:

• Subsequent patches applied to the TDV instances do not overwrite the changes made to the adapter XML

files.

• Changes to the adapters capabilities files ARE carried over during Export or Import of a data source to a new instance. Built-in scripts pkg_export/pkg_import commands include the adapter changes as part of export or import.