Atrium Webinar Series Understanding Atrium … Webinar Series – Understanding Atrium Integrator ... Troubleshooting ... all Pentaho repository tables are converted to AR regular

© Copyright 6/20/2013 BMC Software, Inc 1

Atrium Webinar Series – Understanding Atrium Integrator

Presentation Materials will be posted to the following location: https://communities.bmc.com/communities/docs/DOC-22363

Int'l Toll and US Cell Phone: 913-312-0414 US/CAN Toll free: 877-702-9054

LOCAL: Australia, Brisbane: +61 (0) 7 3123 0047 Australia, Canberra: +61 (0) 2 6111 2007 Australia, Melbourne: +61 (0) 3 9034 3217 Australia, Sydney: +61 (0) 2 9126 3023 Austria, Vienna: +43 (0) 1 9945 6051 Bahrain, Manama: +973 1619 9040 Belgium, Brussels: +32 (0) 2 303 2273 Brazil, Sao Paolo: +55 11 3163 0461 Bulgaria, Sofia: +359 (0) 2 491 6411 Canada, Montreal: +1 514 669 6112 Canada, Toronto: +1 647 426 9209 China: +86 400 120 2693 China: +86 400 120 26 93 Czech Republic, Prague: +420 234 147 003 Denmark, Copenhagen: +45 78 78 79 63 Estonia, Tallinn: +372 622 5742 Finland, Helsinki: +358 (0) 9 7479 0101 France, Lille: +33 (0) 359 69 03 42 France, Lyon: +33 (0) 426 10 30 24 France, Paris: +33 (0) 1 70 71 29 53 Germany, Berlin: +49 (0) 30 2555 5430 Germany, Frankfurt: +49 (0) 69 1200 9864 Germany, Munich: +49 (0) 89 1436 7911 Hong Kong, Hong Kong: +852 3008 0383 Hungary, Budapest: +36 1 577 9957 India, Bangalore: +91 (0) 80 6127 5134 India, Mumbai: +91 (0) 22 6150 2334

Ireland, Dublin: +353 (0) 1 437 0560 Israel, Tel Aviv: +972 (0) 3 721 9373 Italy, Milan: +39 02 9978 1800 Italy, Rome: +39 06 8743 4377 Japan, Tokyo: +81 (0) 3 4455 1996 Latvia, Riga: +371 6601 3678 Lithuania, Vilnius: +370 5205 5590 Luxembourg, Luxembourg: +352 2786 0224 Malaysia, Kuala Lumpur: +60 (0) 3 7724 0847 Mexico, Mexico City: +52 55 4777 2663 Netherlands, Amsterdam: +31 (0) 20 262 0137 New Zealand, Auckland: +64 (0) 9 929 1884 Norway, Oslo: +47 21 95 32 33 Poland, Warsaw: +48 (0) 22 295 36 31 Portugal, Lisbon: +351 21 120 9698 Romania, Bucharest: +40 (0) 21 529 1340 Russian Federation, Moscow: +7 495 620 9818 Singapore, Singapore: +65 6416 9957 Slovakia (Slovak Republic), Bratislava: +421 (0) 2 3278 6632 Slovenia, Ljublijana: +386 (0) 1 888 8397 South Africa, Johannesberg: +27 11 589 8382 Spain, Madrid: +34 91 080 0153 Sweden, Stockholm: +46 (0) 8 4030 4953 Switzerland, Geneva: +41 (0) 22 555 0258 Switzerland, Zurich: +41 (0) 44 556 8481 Taiwan, Taipei: +886 (0) 2 2650 7292 United Kingdom, London: +44 (0) 20 8150 0796

TOLL FREE: Argentina: 0800 666 2571 Australia: 800 635 764 Austria: 0800 295 994 Bahamas: 800 205 6295 Belgium: 0 800 72 785 Brazil: 0800 891 8445 Bulgaria: 00 800 115 1110 Chile: 123 0020 9601 China, Southern Region: 10 800 140 1378 Colombia: 01 800 518 0504 Czech Republic: 800 142 277 Denmark: 8088-6980 Dominican Republic: 1 888 751 4437 France: 0 800 914 669 Germany: 0 800 182 4414 Greece: 00 800 161 2205 1068 Hong Kong: 800 908 710 Hungary: 06 800 184 72 India: 000 800 100 7363 Indonesia: 001 803 011 2660 Ireland: 1 800 760 204 Israel: 1 80 945 2068 Italy: 800 873 593 Japan: 00531 12 0058 Korea, Republic Of: 00798 14 800 4574 Latvia: 800 03 656 Lithuania: 88 003 0200 Luxembourg: 800 25 164 Malaysia: 1 800 812 644 Mexico: 001 800 514 1058

Monaco: 800 93 304 Netherlands: 0 800 024 9645 New Zealand: 0 800 440 611 Norway: 800 191 83 Panama: 00 800 226 7179 Philippines: 1 800 111 00400 Poland: 00 800 112 41 37 Portugal: 800 819 382 Russian Federation: 810 800 2544 1012 Singapore, Singapore: 800 101 1738 Slovenia: 0 800 80815 South Africa: 0 800 999 562 South Korea, Korea, Republic Of: 003 0813 1654 Spain: 900 941 991 Sweden: 02 079 3671 Switzerland: 0 800 896 853 Taiwan: 00 801 126 826 Thailand: 001 800 12 066 3284 Trinidad and Tobago: 800 205 7179 United Arab Emirates: 800 017 7175 United Kingdom: 0 800 051 7166 Uruguay: 000413 598 2551 Venezuela: 0 800 100 2525

Participant passcode: 378410.

Participant passcode: 378410.

Understanding Atrium Integrator

Vaibhav Kulkarni


What will be covered?

AI Overview The Basics 8.X Architecture Overview Deployment / Installation AI Jobs & Job Scheduling Server Group Changes & Enhancements in 8.X Demo - Job Creation using the

Wizard

Best Practices Upgrade from 7604 to 8.X Demo – How to Edit jobs in Spoon Modes in CMDBOutput Step Carte Server details

AIE to AI Migration How AI is different from AIE AIE to AI migration tool Best Practices

Troubleshooting Log file locations

Troubleshooting tips

References

Q&A

Atrium Integrator Overview


Atrium Integrator Overview The Basics

What is it?

Why is it useful?

When do you use it?


Situation:

The Atrium CMDB Administrator needs to load data into CMDB from database tables of a discovery source. The discovery source provides both CI and Relationship data.

Steps:

• Launch the AI Console and create a new

connection in the AI Data Store

• Start the job creation wizard and identify

classes/relationships to be populated

• For each class/relationship, map fields in the

source database tables to CMDB attributes

• Add a schedule for the job



ETL Capabilities - Read input from a source (Extract) - Apply transformations on data (Transform) - Write to the target (Load)

Pentaho Data Integration Story - Industry leading ETL capabilities - Custom jobs using spoon - More flexibility to transform

Runtime job monitoring



Integration Job Builder

Wizard

Job Scheduler

Job Monitoring

Manage Data Store

Create job Schedule job Monitor job Create data store

connections

AR Systems Database

BMC Atrium Integrator

Carte Server

CMDB Plug in - CMDB Input - CMDB Output - CMDB Lookup

AR System Plug in - AR Input - AR Output - ARX Input

(Desktop Designer Tool)

Atrium Integrator Console (Accessible via Mid tier/Browser)

Remedy AR System Server

CMDB

Atrium Integrator

Spoon

Core Data Integration Engine

Atrium Integrator Overview Architecture Overview (v8.x)


Step1: AR Server Installer installs BMC Atrium Integrator - Spoon - Carte Server - AR Steps, e.g. ARInput / AROutput

Step2: Atrium Integrator Server 8.x - Deploys the UI - Flex-based guided workflow - CMDB Steps Input, Output, and Lookup in the already Installed Spoon

directory on the AR server

Step3 (Optional): Client Installer - Allows remote connection to AR System DB to view / design Pentaho JOBS - Requires installing AR Client components during AR Server Installation - Install the Atrium Integrator Client after the AR client installation

Atrium Integrator Overview Deployment/Installation


Job Sub-Job

Transformation(Steps)

Job

Sub Job

Sub Job

Transformation

Transformation

Transformation

Transformation

Atrium Integrator Overview Jobs, Sub job & Transformation


Entry created in UDM:Execution

Form

Carte server reads the entry

Job executes in carte server

Result/Logs are saved

Start/Schedule Job

Atrium Integrator Overview Job Execution Lifecycle


Login as AIUser/AIAdmin, select a JOB from the JOB LIST, and click on the Schedule icon to create a Schedule

Every JOB has its own independent Schedule

Multiple Schedules per JOB is allowed

Atrium Integrator Overview Job Scheduling


In a Server Group installation, multiple servers will appear in the Carte Server drop down list

Select the specific Carte server so that a JOB will always run on the configured Carte server

Note: No failover will occur in a Server Group setup

Atrium Integrator Overview AI in Server Group


Design - In 8.x, all Pentaho repository tables are converted to AR regular forms

- Carte server is now monitored by ARServer

- Start/Stop/Pause/Resume of job is now controlled through an AR Plug-in

Implementation - AR Forms with the prefix “UDM:Repository” were tables in AINTRepository

in 7.6.04.

- The UDM:Execution form controls Carte server job operations like Start/Stop/Resume

Atrium Integrator Overview Architecture Changes in v8.0


Pentaho upgrade from 3.2.4 to 4.1

New AISAMPLE_MachineOS _DELTA sample job

CSV and XML data store connection refresh

X-Path is displayed for XML jobs

Delete option in the Integration Job Builder wizard

Job scheduling enhancements

New access point to launch Atrium Integrator

Atrium Integrator Overview New Features in 8.0/8.1


Job Creation using the Wizard

Atrium Integrator Overview Demo


Few points to remember about AI Wizard

Wizard is available for transferring data to key CMDB classes. - It doesn’t include feature for designing jobs that involves data transfer to

AR forms

You need to use AI spoon on Windows for modifying any job that you created using wizard

Use AI wizard as starting point for designing your integration job - It provides you with a structure that you can refine using AI Spoon later

You can define job schedule as part of the wizard. - Schedule can be added/modified from AI console any time

You should select ‘Delta’ and ‘Response’ options during job design in wizard. - Else, modifying jobs or transformations to handle ‘Delta’ or ‘Response’ in

Spoon at later stage will be significant manual effort

Best Practices


AI Best Practices 7.6.04 to 8.X Upgrade


Edit Job using the Spoon

AI Best Practices Editing Job in Spoon


Only Insert New CIs

Identical to the AIE option ‘Do not update’ and ‘Create only new CIs’

Always Insert CIs

Insert CIs into CMDB without checking for duplicates

AI Best Practices Options in ‘CMDB Output’ Step


Use Cache

Should always be ON to reduce CMDB lookups and improve performance

Cache files are created for every class mapping

Use CheckSum

Equivalent to Checksum option in AIE.

Very useful when the delta option is not used in AI

AI Best Practices Options in ‘CMDB Output’ Step


Carte Server Behavior

Atrium Integrator Carte server is a web server used to execute

transformations and jobs remotely

- All the jobs scheduled via Atrium UI run on the Carte server.

Accessing the Carte server

- http://<ARserver_host>:20000

20000 is the default port

- Use the Remedy Application Service userid / password

Changing the Carte server port

- Carte server port is specified in UDM:Config form and armonitor.cfg file

- After changing the port, restart the BMC Remedy AR System server

- If it is a Server Group enviornment, restart all other AR servers in the

server group

AIE to AI Migration Tool


AIE

• Limited adapters available out-of-the-box

• Limited set of transforming capability

• No support for hierarchical XML

AI

• Exhaustive infrastructure connectors (JDBC, complex XML, web services, JMS etc)

• Support for JavaScript with many transformation functions

• Out-of-the-box templates to simplify most new CMDB integrations

• Wizard-based UI for creating integrations

AIE to AI Migration Tool Overview


BMC Atrium Integration Engine term - Data exchange - Data mapping

Corresponding Atrium Integrator term - Job - Transformation

Data Exchange

Data Mapping Relationship Exchange

Relationship Mapping

Job

Sub Job

Sub Job

Transformation

Transformation

Transformation

Transformation

AIE to AI Migration Tool Terminology


Uses the AR API to connect to the AIE source

Uses the same API library used for generating JOBS from the AI UI

Internal flow for migrating the AIE exchanges to AI - Read all exchanges from the AIE source - ALL Schedule exchanges are migrated as Parent JOBS on the target server

AI Migration Tool

Target AR Server with

AI Server Installed

(Carte server)

Source AR Server with

AIE Exchanges

AR API

Block Diagram

AIE to AI Migration Tool Design


AIE Server Details Server where the AIE Exchanges are

present. All the exchanges from the server are displayed in the list.

Important: You have to know the Parent Exchange to Start the Migration.

Target Server Details Server where the AI JOBS / Transformations converted from the AIE exchanges will reside.

Important: You have to know the Application Service password of the target AR server where the JOBS are getting migrated. File Name/Table Name column is editable for all file type

exchanges. In case if the paths for these files is going to be different on the target then one can type in the correct path from target server.

AIE to AI Migration Tool UI Overview


In this example, we will use the Sample ITSM exchanges which are shipped with AIE - The source for all these exchanges is a CSV file

Select the Parent exchange in the Migration tool

Because the underlying technology is different, after the exchanges have been migrated to AI they will be slightly different than they were in AIE

AIE to AI Migration Tool View Migrated Jobs in Spoon


Configuration - Before migrating Oracle/DB2 exchanges, you have to insert the SID and

Host Name of the AIE server into the DB.properties file on the AI server

Error & Logging messages - The Migration tool displays high level messages on the message console - You can also look in the log4j file in the log folder for detailed messaging

Best Practices - See section Migrating existing data exchanges in the Atrium Integrator

User Guide

AIE to AI Migration Tool Additional Information


Exchanges with multiple mappings will be migrated only with the main mapping. To create the transformations for child mappings refer to section Creating multiple mappings in the AI User Guide

No Support for .dat type file exchanges, but richer support in AI for text files

Independent Exchanges with Type = Relationship will not be migrated

AIE to AI Migration Tool Known Limitations

Troubleshooting


Logging: Importance of various Log files

Log File Name Log File Location Purpose of the Log file

arfileout .txt <AR Server Installed

Directory>\diserver\data-

integration\ngie\bin\log

This log file will contain info about file

metadata and execute Run Process call.

arcarte-stderr-2012-09-

04.log/arcarte-stdout-2012-

09-04.log/arcarte.log

<AR Server Installed

Directory>\Arserver\Db

This log file will contain info about

Start/ Stop and Job/Transformation

Execution for Carte Server

arjavaplugin.log <AR Server Installed

Directory>\Arserver\Db

Related to AR ARDBC Pentaho Plug-in

“ARSYS.ARDBC.PENTAHO”

AI cache file is generated

with the below format

[ServernamePort-

NameSpace-ClassName-

DatasetId-Keys

]

It will be located at any of

the following locations based

on environment variable set

•ATRIUMINTEGRATOR_HO

ME

•ATRIUMCORE_HOME

•BMC_AR_SYSTEM_HOME

The cache files are always generated

and filled by CMDBOuput Plug-in.

Mid-tier Log/Web Server Log(Tomcat)

<Mid-tier Installed Location>\Logs\

AI activities from AUI console i.e.

Start/Stop/Create/Delete can be

monitored here.


Common Troubleshooting Steps

- Class /Relationship list not appearing: Check Midtier stdout logs to find out the API error while fetching data from DatamodelRefernceCIMapping forms. CSV/XML Connection fails: Capture AR server logs to check for ARExecute Process failure which in turns give a call to script file to populate form NGIE:FileMetadata. Data Store Connection fails: Check for Midtier stdout logs and verify if the DB server is not behind the fire wall. Create job fails: Check Midtier stdout logs


Common Troubleshooting Steps

Cannot Start a job: Check Midtier stdout + AR Plug-in logs Job Schedule: Check in AR System Job/System Pending Job Queue form for the next schedule Plugin server log Cannot Delete AICache files: Stop AR Server and Close Spoon. Run a JOB on specific Carte server: Check if the Carte server is up and running and job is associated to Carte Server in UDM:Permissioninfo form.


References

- Understanding Atrium Integrator

- Data store connections in AI

- AI Job Monitoring

https://docs.bmc.com/docs/display/public/ac81/Understanding+Atrium+Integrator

https://docs.bmc.com/docs/display/public/ac81/Working+with+Atrium+Integrator+data+store+connections

https://docs.bmc.com/docs/display/public/ac81/Monitoring+jobs+in+Atrium+Integrator


Q & A


Feedback

Any follow up questions – post on BMC DN with prefix “Webinar –” in subject line 1. http://communities.bmc.com/communities/community/bmcdn/bmc_atrium_and_foun

dation_technologies/bmc_atrium_cmdb/bmc 2. https://communities.bmc.com/communities/community/bmcdn/bmc_atrium_and_fou

ndation_technologies/bmc_atrium_cmdb

http://communities.bmc.com/communities/community/bmcdn/bmc_atrium_and_foundation_technologies/bmc_atrium_cmdb/bmc

http://communities.bmc.com/communities/community/bmcdn/bmc_atrium_and_foundation_technologies/bmc_atrium_cmdb/bmc

https://communities.bmc.com/communities/community/bmcdn/bmc_atrium_and_foundation_technologies/bmc_atrium_cmdb



Thank you!

Documents

Atrium Webinar Series Understanding Atrium … Webinar Series – Understanding Atrium Integrator ... Troubleshooting ... all Pentaho repository tables are converted to AR regular