ROP Mainframe Data Services

ITON Approach and Capabilities for ROP Mainframe Integration and Migration Services

A Technical Report

June 2011

ROP Mainframe Data Services

ITON Software LLC. #02041, Knowledge Oasis Muscat, Oman.

© Copyright ITON 2011 All Rights Reserved

ROP Mainframe Data Services Ref No: ITON/WP-41/2011No. of Pages: 30

Author:ITON Software GroupBusiness Contact:Siva Prasad+968 [email protected] Corporation PO Box 54667, DubaiDubai Airport Free ZoneUnited Arab EmiratesPhone (+971) 4 204 5518Fax (+968) 4 204 5519URL www.itoncorp.comVersion 1.0 (04 June 2011)No part of this document may be reproduced in any form or by any means without written permission of:ITON Corporation

NOTICE:This document contains proprietary details of ITON and ROP and is derived in whole or in part from materials developed by ITON and ROP. The information in this document is subject to change without notice. ITON assumes no responsibilities for any errors that may appear in this document.ITON and the ITON logo are trademarks or registered trademarks of ITON CorporationOther company, product and service names may be trademarks or service marks of others.

Annotations[ITON] ITON Corporation[ROP] Royal Oman Police

Document HistoryVersion 1.0: Related Documents

ROP Mainframe Data Services 1.0

In the event of any ambiguity or conflict this response takes precedence over the above documents and any other communication either written or verbal.

ITON Mainframe Data Services Confidential 2

TABLE OF CONTENTS:

1.0 INTRODUCTION.........................................................................................................4

2.0 ITON EXPERIENCE....................................................................................................5

3.0 ISSUES IN MIGRATING NATURAL/ADABAS...........................................................9

4.0 ITON APPROACH & METHODOLOGY:..................................................................11TRELATIONAL AND DPS FUNCTIONALITY:........................................................................16

TRELATIONAL DATA ANALYSIS AND MODELING................................................................17

TRELATIONAL FILE IMPLEMENTATION:.............................................................................18

TRELATIONAL ADABAS FILE ANALYSIS..........................................................................19

RDBMS SCHEMA AUTO-GENERATION............................................................................21

TRELATIONALPC AUTO-GENERATION.............................................................................21

DPS ARCHITECTURE......................................................................................................23

DPS MATERIALIZATION (ETL)........................................................................................23

DPS MATERIALIZATION DATA.........................................................................................24

DPS PROPAGATION (CDC)............................................................................................25

DPS PROPAGATION.......................................................................................................26

5.0 APPENDIX.................................................................................................................29


1.0 INTRODUCTIONROP Mainframe Environment:

ROP Business Applications are developed using Natural Development Tools. The

ADABAS database is the repository of defined pieces of data that make up records.

These records are stored using ADABAS management System. ADABAS is a DBMS

that uses inverted lists with relational capabilities developed by Software AG for

Mainframe. It comprises of ADABAS data files, which are collections of logically related

data, organized by data fields and logical records where Indexes are not embedded into

the Data.

ROP uses EntireX V7 as middleware for Application, Data Connectivity and Integration,

which ITON team has worked with ROP for various other Projects that are delivered with

in ROP like ROP Immigration & ROP Prison Management. EntireX V7 provides data and

application connectivity. The EntireX suite is comprised of an EntireX Kernel plus

EntireX Broker and Security Components that support RPC and Messaging

Communication among Client-Server applications. EntireX is a Message orinented

Middleware (MOM) that offers guarantee delivery of messages along with Push-pull

services.

Data Volume: 17 Million Travel Documents to be considered to be moved out of

Mainframe.

MOFA Lotus DOMINO Architecture:

Lotus Domino Application architecture at MOFA is designed by ITON to serve all MOFA

missions using Lotus Databases with Replication facilities running on Domino Clusters.

ITON has built and designed the Entire MOFA Domino application Infrastructure and is

well versed the architecture and data models that are applied with in the whole solution.

ITON has custom tools and processes ready for MOFA migration and will be mapped to

the new eVISA System and Data Architecture.


2.0 ITON ExperienceITON has been working with Royal Oman Police for over 10 years and has successfully

integrated with Mainframe for various ROP Business Applications. The key challenge is

always integration with ROP Mainframe for various business applications like ROP

Prison System, ROP Immigration DMS and ROP FIU. ITON has successfully handled

the Integration and Migration Services with ROP using EntireX as a middleware platform

provided by ROP. ITON can utilize Software AG tools like Natural Engineer and SQL

Gateway, if available with ROP which is not mandatory.

ITON has partnered with various international experts from organizations like TreeHouse

Inc, USA who are specialized in dealing with Mainframe Migrations especially ADABAS

to handle ROP Mainframe Migration and Integration Requirements. We at ITON

understand how Important and Complicated is the Mainframe Data Migration for eVisa

Project. ITON will be putting in all the best resources for this entire activitiy. As part of

this Program, ITON will have Mr. Moosa Al Lawati, who is ex-ROP and has worked in

DGIT for over 27 years in various projects, planning, development and managing

Mainframe based Applications. Mr. Moosa, having years of hands on Expereince, he

clearly understands the core mainframe design and the migration requirements. Many

Expert teams and Consultants are made available for this Data Migration Project.

ITON has worked extensively with EntireX .Net for integration services. ITON has

experience working with .Net Wrapper, .Net Client or .Net Server DLL. The EntireX .Net

wrapper provides acces to the EntireX RPC servers for .Net client application and .Net

servers for any EntireX client. The .Net Wrapper takes as input an EntireX IDL file, that

describes the interface for the RPC, and generates C# classes that implement the

methods and data types of the interface.


The EntireX .NET Wrapper works as follows:

• C# code is generated from the EntireX IDL file. Using C# is a natural

choice when full fledged .NET programming is required, since C# was

designed for the .NET platform.

• The .NET Wrapper runtime implements functionality that is not specific to

a given IDL file (e.g., marshalling and unmarshalling of data). The C#

code makes use of the .NET wrapper runtime functionality. The customer

interface and the .NET Wrapper runtime is "managed" .NET code (C#)

and makes use of advanced .NET features such as Attributes,

VersionInfo, etc.

• The .NET Wrapper runtime makes use of the functionality of the

"unmanaged" EntireX runtime ("dllimport" in C#). "Managed" .NET code

and "unmanaged" DLL code can be combined safely.

The EntireX IDL compiler and an appropriate template are used for the C# code

generation.

The .Net Client Application for a given IL file, the For a given IDL file, the EntireX IDL

compiler (erxidl.exe) and a C# code generation template csharp_client.tpl are used to

generate a client stub. The source code generated by the .NET Wrapper can be

compiled into a .NET assembly with the Visual Studio .NET C# compiler. Application

developers can use the generated client stub assembly to write .NET applications that

access EntireX RPC servers. They are not limited to C# as programming language.

Any .NET programming language based on the Common Language Runtime (CLR) can

make use of the client stub assembly. Choices are C#, VisualBasic.NET or managed C+

+.


The .Net Server DLL model

The EntireX IDL compiler and a C# code generation template csharp-server.tpl are used

to generate a C# code frame for a specific IDL. Application developers can use the

generated frame to write their own server code for each program in the IDL. The source

code can be compiled into a .NET assembly (DLL) with the Visual Studio.NET C#

compiler.


Following are some of ITON Implementations implemented using SoftwareAG EntireX

Components and Services:

ROP ePrison System:

ITON has worked closely with ROP team to build EntireX components to provide

mainframe data services for all Prisoner related information which was used to connect

from Lotus Domino to extract all Prisoner details from various files in mainframe and

store in the ePrison Domino system. The data would be updated on a periodic basis into

Domino System by communicating with EntireX Gateway.

ROP Immigration EDMS:

ITON again has worked closely in integrating Passport EDMS system ROP Mainframe

again using EntireX components for searching Omani Citizen information for indexing

purposes at the Passport Imaging and Indexing facility in ROP and store into RDBMS

System.

ROP FIU System:

ITON has successfully worked with ROP DGIT and ROP DGCS to connect to NRS Web

Services to extract various personal data from the system and save into the Local

RDBMS System and maintain the updated information.


3.0 Issues in Migrating Natural/ADABAS

There are a variety of problems that must be handled in migrating Natural and

ADABAS-based COBOL code:

Database Conversion

o Conversion of the ADABAS data model to a Relational model. The

ADABAS data model is not relational and definitely not normalized.

The data model must be converted into an efficient, maintainable

Relational model, using data types supported by the Relational

model as opposed to the legacy ADABAS data types. A key

consequence is that the placement and representation of the data in

the database will not be the same and therefore data accesses and

updates must change.

o Conversion of ADABAS data base content into Relational database

content. It is called "data processing" because ultimately these

applications are about the data! The existing ADABAS database

content must be extracted and moved into the new Relational form

without losing any information. If the new target platform is not the

mainframe, the data must moved off the mainframe.

COBOL code modification to change ADABAS calls to relational calls. If

COBOL programs will be retained on the mainframe, then ADABAS calls in the

COBOL programs must be modified to access the data from the new

Relational model.

Code translation of Natural or COBOL code with ADABAS calls to modern OO

languages. If the goal is to move from mainframe then

o The source code of the application must be converted to Java or C#

o Explicit ADABAS calls in COBOL code and implicit calls in Natural

must be replaced by database calls that understand the relational

model and the representational changes. Reads and writes of flat

files must be replaced by modern file reads and writes.

o External interfaces to other systems must be replaced by some

equivalent


o EntireX interfaces must be converted in a way that enables short-

term continued access through EntireX with eventual ability to

replace by equivalent remote access facilities such as Web services.

o Job Control and Scheduling must be replaced

o Security must revised to support the broader accessibility of the

application

Application Testing. This provides assurance to all parties that the result

preserves the business logic and functionality.

Maintenance change Integration.

Migrations take time.

o The legacy application will continue to serve the organization during

the migration, and business needs will force changes to the legacy

application to even as the migration occurs. These changes must be

moved into the migrated application.


4.0 ITON Approach & Methodology:ITON works with various Adabas migration tools and services from Tree House Inc,

USA, which is a very specialized, company dealing primarily with Adabas Migration

Services. Tree House, primarily looks in to the 10,000 IBM mainframe sites worldwide,

with particular focus on the 2,500 organizations using ADABAS and NATURAL, 500 of

these being in North America.

Most mainframe users face critical data management challenges due to the complexity

and proprietary nature of ADABAS and other databases. TSI meets the needs of these

customers through a comprehensive and robust set of products—with supporting

services—to enable standards-based access to, and replication and integration of,

mainframe data sources throughout the technology architecture. The company’s

visionary leadership in this market has included pioneering ADABAS-to-RDBMS ETL

(extract-transform-load) and CDC (change data capture) with tRelational/DPS in the mid-

1990s, and launching of the era of real-time ADABAS-to-RDBMS replication with

DPSync in 2004. Today, TSI stands alone in its product maturity, capability and breadth

of product portfolio. Despite the rapid pace of change in the IT landscape, TSI’s

customer base can be assured that there remains a strong commitment to providing

continued support and upgrades for the product suite.

The increasing demand for mainframe application modernization and migration—

particularly in the Software AG customer arena—has enabled TSI to position itself as an

important contributor to such initiatives. TSI’s long experience, deep knowledge and

wide-ranging capabilities in ADABAS/NATURAL and other mainframe technologies

make the company a valued partner for third-party solution providers and a trusted

advisor to customers.


Finally, for customers and service providers struggling with a shortage of mainframe-

skilled resources and rapidly changing IT environments, TSI offers staff augmentation

and outsourcing services for technical roles. TSI works with over 700 large Corporations

in the world. Some of the references:



Banking & Finance Asian Development Bank, PhilippinesB-Source SA, SwitzerlandBanco Real, Brazil Bank Vontobel AG, SwitzerlandDexia Banque Internationale à Luxembourg, Luxembourg KAS BANK N.V., The NetherlandsVolkswagen Bank, Germany

Communication Verizon, ColoradoTelemar, BrazilU.S. Sprint, California, Kansas, Texas

Education Penn State University, PennsylvaniaUniversity of Texas, Austin (several campuses)University of Toronto, Canada

Entertainment Club Med, FranceUniversal Music, International

Food Dansk Supermarked A/S, DenmarkDole Fresh Vegetables, CaliforniaEDEKA ZENTRALE AG & Co. KG, GermanyFood Lion, North Carolina

Government (Federal) NASA, U.S.Royal Oman Police, Oman Department of Labor, United States

Government (Local) County of Sacramento, CaliforniaCounty of Los Angeles, California

Government (State) Informatikzentrum Landesverwaltung Baden-Wuerttemberg, GermanyState of California (Several Sites)State of Minnesota State of Utah

Services First American Real Estate, California, Florida, Texas

Insurance GMAC Insurance, Missouri Berufsgenossenschaft Druck und Papierverarbeitung, GermanyBerufsgenossenschaft Handel und Warenwirtschaft, GermanyBerufsgenossenschaft Metall Nord Süd, GermanyRegence Blue Cross & Blue Shield of OregonSIGNAL IDUNA, Germany

Manufacturing Bradford Exchange, Illinois EATON Corporation, PennsylvaniaDow Chemical Company, MichiganNewell Companies, IllinoisNissan Motor Manufacturing Ltd., United Kingdom, United StatesREHAU AG + Co, GermanyYves Rocher, Germany

Outsourcing CSC, United States EDS, United Kingdom, CanadaIBM Global Services, United States, Canada

Publishing Hachette Book Group, MassachusettsCurtis Circulation Co., New Jersey

Transportation Federal Express, TennesseeNMBS Holding, BelgiumSouthwest Airlines, Texas

Utilities American Electric Power Service, Ohio Tampa Electric Company (TECO), Florida

High Level Migration Process for Open Technical Platform:


ITON uses various TSI Products and Tools to perform Adabas Data Migration in

cooperation with TSI. Various product components that are used for ROP eVisa data

migration services include:

tRelational, An ADABAS-to-RDBMS modeling, mapping and data analysis tool

Data Propagation System (DPS), an ADABAS-to-RDBMS data migration and

propagation system for data distribution and warehousing

tRelationalPC, a windows-based client/server GUI data modeling and mapping

environment (included with tRelational)

Treehouse Remote Access (TRA), middleware that allows tRelationalPC to

communicate with tRelational on the Mainframe (included with tRelational)

tRelational Features:

Modeling and Mapping

- Native ADABAS/NATURAL application

- Predict metadata “discovery” and ADABAS file analysis

- Automated generation of normalized RDBMS schemata with explicit ADABAS

field to RDBMS column mapping

- Robust modeling and mapping – normalize, denormalize, substring,

concatenate

- Single rule base and metadata repository

“Code” Generation

- RDBMS Data Definition Language (DDL) – Create tables, columns and

contraints

- DPS Parameters – Extract and Transformation parameters


tRelational and DPS Functionality:

- Captures Logical (PREDICT) and Physical (ADABAS FDT) file definitions and

resolves any discrepancies.

- The implemented file provides the basis for modeling and mapping to the

RDBMS table(s).

- Captures statistical analysis to provide or confirm the understanding of the

source data. The analysis provides for improved modeling and early

identification of “problem” data.

- Provides physical modeling and explicit ADABAS to RDBMS mapping. Auto

Generation provides intelligent and “automatic” modeling and mapping from

an Implemented File.

- tRelational generates all input parameters needed to begin Materialization

and Propagation.


tRelational Data Analysis and Modeling

- tRelational generates output for the creation of tables, columns, and

constraints for your target RDBMS.

- The Materialization process requires No Direct ADABAS Access

- Extracts from an ADABAS utility backup

- Extracted data is transformed into a relational form

- RDBMS tables are then populated by the native RDBMS loader utility

- The Propagation process requires No Direct ADABAS Access

- ADABAS transaction data is extracted from the ADABAS Protection Log Files

- The extracted data is transformed into SQL “UPDATE”, “INSERT”, and

“DELETE” statements


tRelational File Implementation:

Capture Logical (Predict) file, Userviews, and Physical (FDT) definitions.

Fields that are defined logically and physically are highlighted.


tRelational ADABAS File Analysis

One time capture of statistical analysis of repeating data (MUs and PEs), Candidate variable text data, and descriptors for improved modeling.

Statistics of MUs and PEs for sizing of child tables and potential de-normalization of tables to individual columns.


Statistics of alphanumeric fields for candidate variable character text columns

This Screen shows descriptor/superdescriptor usage statistics to determine candidate Primary Keys and Indexed Columns.

RDBMS Schema Auto-Generation


Generates table(s), Columns, constraints, and mappings for a selected implemented file.

tRelationalPC Auto-Generation

Auto Generation Example: 4 tables with Primary Key and Foreign Key Constrains, and the added DPS PE Sequencer (PE Occurrence).Output Generated from Metadata


- RDBMS Data Definition Language (DDL)- DPS specifications (parameters) for ETL and CDC Processing- Metadata Reports (Summary and Detail)- tRelational API enables Metadata export to other tools and repositories

Generated RDBMS DDL from Sample:


DPS Architecture

- Written in Assembler for efficiency- Executed as batch job- No Calls to active ADBAS required- No Impact on Production Environment- External Transformation Routines (ETRs)- A Call to a Linked Object- Dozens of Built-ins- Custom Transformation and Data Cleansing

DPS Materialization (ETL)

- Provides initial load of the RDBMS- Extracts from ADASAV- Intelligent Transformation based on Model/Mappings- Generates rows for target tables and SQL Utility Load Control Statements- Provide refresh of the RDBMS when required or desired


DPS Materialization Data

Contains all row images to be loaded into the RDBMS REPOSITORY. Each row is prefixed with a Table ID and is formatted and delimited natively for the RDBMS loader.

Native loader control statements are automatically generated with each DPS Materialization run.


DPS Propagation (CDC)

DPS Propagation

- provides periodic synchronization of the RDBMS target with the source ADABAS database

- Extracts from PLOG archives- Intelligent transformation based on Update and model/mappings- Generates SQL for Inserts, Updates, Deletes and Commits


DPS Propagation

Sample SQL resulting from an update to Personnel ID, mapped to a Primary Key showing the Deletes and Inserts generated to maintain referential integrity.


Overall Data Migration and Integration Challenges:

Solution Architecture – Data Migration:

ITON Mainframe Data Services Confidential

ROPMainframe

iData Proces

sor

Storage

Manager

Runtime

Manager

Replication

Manager

MF Data

Natural Scripts for Data Profiling, Cleansing and Export MOFA

Lotus Domino

MOFA Data

Domino Scripts for Data Profiling, Cleansing and Export

Business

Logic Processor

Apartar

ETL Servic

es

eVISA Apps

Oracle DB

Data Available for eVisa Applications

Data Acces

s Processor

Oracle DB

Oracle DB

Oracle DB

VIS Apps

Oracle DB

27

ITON will be making expert Adbas/Natural Programmers to run the entire process under the Guidance of Mr. Moosa Al Lawati. These expert Programmers will be analysing the data models and use the right tools to deliver the data out of the mainframe. In required the Programming team will be generating custom scripts in Mainframe to handle the complex data migration scenarios.

ITON approach is mainly utilizing powerful TSI tools for ADABAS Migration and Software AG EntireX, SOA Gateway for Mainframe Integration Services.

Availability of Specialized Adabas / Natural Developers in the team will give more flexibility to generate required scripts for smooth Migration of Data.


5.0 APPENDIX

(Additional reference material and profiles)
