ICC Utility - User Guide - Informatica Marketplace Utility... · ICC Utility User Guide ... ready to be migrated in Test ... Template driven approach makes the code review process

INFORMATICA CORPORATION

ICC Utility

User Guide

Ajit Singh Kalura

2/10/2011

ICC Utility 1.0

Page 2

Contents

About ...................................................................................................................................................... 3

System Requirement ........................................................................................................................... 4

Directory Structure .............................................................................................................................. 4

ICC Utility Syntax ................................................................................................................................. 4

ICCUtility.bat file content ..................................................................................................................... 5

Configuration Files ............................................................................................................................... 5

CodeReviewConfiguration.xml ......................................................................................................... 5

log4j.properties ............................................................................................................................... 7

InfaProcesses.properties ................................................................................................................ 10

Execution Steps ................................................................................................................................. 10

Step1 ............................................................................................................................................. 10

Step2 ............................................................................................................................................. 11

Step3 ............................................................................................................................................. 12

Step4 ............................................................................................................................................. 20

Step5 ............................................................................................................................................. 21

Step6 ............................................................................................................................................. 22

Known Limitations ............................................................................................................................. 22

ICC Utility 1.0

About The main goal of ICC Utility is to streamline some of the labor intensive task in a typical ETL

environment, this utility consist of various independent components that helps following folks

manner listed below:

1) Helps ETL Developer document their Po

mapping.

2) Helps ETL Manager in doing Capacity Planning by providing Memory usage for code

ready to be migrated in Test / Production environment

3) Automated Code review option reduces the time an ETL Lead has to spend on doing

code reviews. Template driven approach makes the code review process very

flexible.

4) Helps Informatica Administrator receive code migration request in a consist

manner. Code migration component uses Informatica Velocity template and

extract all the information that a

PowerCenter code between two environments.

5) Normalized PowerCenter Code helps a

attribute values for any transformation between two different version of mapping

or workflow.

ICC Utility consist of a bat file, this bat file processes

The main goal of ICC Utility is to streamline some of the labor intensive task in a typical ETL

environment, this utility consist of various independent components that helps following folks

Helps ETL Developer document their PowerCenter code both workflow and

Helps ETL Manager in doing Capacity Planning by providing Memory usage for code

ready to be migrated in Test / Production environment.

Automated Code review option reduces the time an ETL Lead has to spend on doing


Helps Informatica Administrator receive code migration request in a consist

manner. Code migration component uses Informatica Velocity template and

the information that an administrator needs when migrating

PowerCenter code between two environments.

Normalized PowerCenter Code helps an ETL developer compare attribute by


or workflow.

this bat file processes the instructions as shown in the diagram below:

Page 3

The main goal of ICC Utility is to streamline some of the labor intensive task in a typical ETL

environment, this utility consist of various independent components that helps following folks in the

werCenter code both workflow and

Helps ETL Manager in doing Capacity Planning by providing Memory usage for code

Automated Code review option reduces the time an ETL Lead has to spend on doing


Helps Informatica Administrator receive code migration request in a consistent

manner. Code migration component uses Informatica Velocity template and will

administrator needs when migrating

ETL developer compare attribute by


the instructions as shown in the diagram below:

ICC Utility 1.0

System Requirement

• Microsoft Windows 2000/XP/2003/Vista/7

• JRE 1.5 or higher

• 512 MB RAM

• 10 MB free hard disk space

• PowerCenter 8.6 and above

Directory Structure

ICC Utility Syntax

Syntax : ICCUtility.bat <

Example: C:\ICCUtility.bat DataQuality wkf_Step_Load

Where:

Microsoft Windows 2000/XP/2003/Vista/7

10 MB free hard disk space

and above

ICCUtility.bat <PowerCenter Folder> <PowerCenter Workflow

ICCUtility.bat DataQuality wkf_Step_Load

[DataQuality] is PowerCenter folder name

[Wkf_Step_Load] is a workflow within DataQuality Folder

Page 4

PowerCenter Workflow Name>

is a workflow within DataQuality Folder

ICC Utility 1.0

Page 5

ICCUtility.bat file content

REM Syntax: ICCUtility.bat <PowerCenter Folder> <PowerCenter Workflow Name>

echo *** Extract Code from PowerCenter Repository ***

pmrep connect -r Repository_Service -h localhost -o 6005 -n Administrator -x ****

pmrep objectexport -n %2 -o Workflow -f %1 -m -s -b -r -u input/%2.xml

echo *** Converts PowerCenter Code into a normalized XML Format ***

java -Xms128m -Xmx512m -jar InfaParseInformaticaXML.jar input/%2.xml output/%2.parsed

echo *** Document PC Mapping and Workflow ***

java -Xms128m -Xmx512m -jar InfaDocumentor.jar output/%2.parsed output/%2_mapping_doc.html output/%2_workflow_doc.html

echo *** Perform Automated Code Review using pre-defined template ***

java -Xms128m -Xmx512m -jar InfaCodeReview.jar output/%2.parsed output/%2_rev.html

echo *** Document all sessions Memory properties in a workflow ***

java -Xms128m -Xmx512m -jar InfaMemoryRequirement.jar output/%2.parsed output/%2_mem.html

echo *** Generate code Migration checklist ***

java -Xms128m -Xmx512m -jar InfaCodeMigrationDocumentor.jar output/%2.parsed output/%2_mig.html

Configuration Files

CodeReviewConfiguration.xml

This configuration file is used to perform automated code review and also highlight modified values

when documenting PowerCenter workflow and mapping. This file contains all the transformation that

you typically see in PowerCenter Designer (Including IDQ, B2B DT, B2B DX transformation). Figure below

shows all the attributes associated with a Worfklow.

Depending on your organization you skip code review for Workflow by chaning SKIP_CODE_REVIEW

attribute from ‘N’ to ‘Y’. Similarly depending on your naming standard you can modify the workflow

prefix by modifying the attribute OBJECT_PREFIX from ‘wkf’ to ‘wf’, assuming you would want all your

workflows to start with wf instead of wkf.

ICC Utility 1.0

Page 6

This configuration file will contain naming standard for all the PowerCenter transformation as shown

below:

Apart from modifying two attributes you might not have to modify any other value in the configuration

file, the schema definition for xml file is as shown below:

This configuration file is located under <conf> folder.

ICC Utility 1.0

Page 7

log4j.properties

This file controls the logging portion of ICC Utility, if you need see detail progress for each component then you can change that behavior by modifying message level. For example if you need to detail progress for Step3 then modify log4j.logger.com.kalura.InfaDocumentor=WARN line to log4j.logger.com.kalura.InfaDocumentor=INFO Content of log4j.properties file:

# Set root logger level to DEBUG and its only appender to A1. log4j.rootLogger=DEBUG, A1 # A1 is set to be a ConsoleAppender. log4j.appender.DEBUG=org.apache.log4j.ConsoleAppender log4j.appender.DEBUG.layout=org.apache.log4j.PatternLayout log4j.appender.A1=org.apache.log4j.ConsoleAppender log4j.appender.A1.layout=org.apache.log4j.PatternLayout # Print the date in ISO 8601 format log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c - %m%n log4j.appender.R=org.apache.log4j.RollingFileAppender log4j.appender.R.File=ICCUtility.log log4j.appender.R.MaxFileSize=100KB # Keep one backup file log4j.appender.R.MaxBackupIndex=1 # Print only messages of level DEBUG or above in the package com.kalura. log4j.logger.com.kalura.common.XmlUtils=WARN log4j.logger.com.kalura.InfaParseInformaticaXML=WARN log4j.logger.com.kalura.InfaDocumentor=WARN log4j.logger.com.kalura.InfaCodeReview=WARN log4j.logger.com.kalura.InfaMemoryRequirement=WARN log4j.logger.com.kalura.InfaCodeMigrationDocumentor=WARN

Typical output console output generated by ICC Utility just using the WARN option in log4j.properties file

***************************************************************** *** Parameters passed to ICC Utility **************************** ***************************************************************** * * PowerCenter Folder = DemoFolder * PowerCenter Workflow Name = wkf_Demo * ***************************************************************** *** Extract Code from PowerCenter Repository ******************** ***************************************************************** * * Connecting to informatica repository .... Informatica(r) PMREP, version [9.0.1 HotFix2], build [242.1111], Windows 32-bit Copyright (c) Informatica Corporation 1994 - 2010 All Rights Reserved. This Software is protected by U.S. Patent Numbers 5,794,246; 6,014,670; 6,016,50 1; 6,029,178; 6,032,158; 6,035,307; 6,044,374; 6,092,086; 6,208,990; 6,339,775; 6,640,226; 6,789,096; 6,820,077; 6,823,373; 6,850,947; 6,895,471; 7,117,215; 7,1 62,643; 7,254,590; 7,281,001; 7,421,458; and 7,584,422, international Patents an d other Patents Pending. Invoked at Tue Feb 08 18:10:01 2011 Connected to repository Repository_Service in Domain_vm as user Administrator connect completed successfully. Completed at Tue Feb 08 18:10:01 2011 * Exporting wkf_Demo workflow to input/wkf_Demo.xml .... Informatica(r) PMREP, version [9.0.1 HotFix2], build [242.1111], Windows 32-bit Copyright (c) Informatica Corporation 1994 - 2010 All Rights Reserved. This Software is protected by U.S. Patent Numbers 5,794,246; 6,014,670; 6,016,50 1; 6,029,178; 6,032,158; 6,035,307; 6,044,374; 6,092,086; 6,208,990; 6,339,775; 6,640,226; 6,789,096; 6,820,077; 6,823,373; 6,850,947; 6,895,471; 7,117,215; 7,1 62,643; 7,254,590; 7,281,001; 7,421,458; and 7,584,422, international Patents an d other Patents Pending.

ICC Utility 1.0

Page 8

Invoked at Tue Feb 08 18:10:02 2011 02/08/2011 18:10:02 Analyzing Object Dependencies... Exporting selected objects in repository Repository_Service ... Exporting selected objects in folder DemoFolder ... Fetching Source Definition [ing_direct_source] ... Exporting Source Definition [ing_direct_source] ... Exporting Target Definition [to_cep_flow_1] ... Fetching Mapping [m_Demo] ... Exporting Mapping [m_Demo] ... Fetching SessionConfig [default_session_config] ... Exporting SessionConfig [default_session_config] ... Fetching Workflow [wkf_Demo] ... Exporting Workflow [wkf_Demo] ... Export is completed. Exported 5 object(s) - 0 Error(s), - 0 Warning(s) objectexport completed successfully. Completed at Tue Feb 08 18:10:03 2011 * ***************************************************************** *** Converts PowerCenter Code into a normalized XML Format ****** ***************************************************************** * * Parsing input/wkf_Demo.xml and generating output/wkf_Demo.parsed file .... Done. * ***************************************************************** *** Document PC Mapping and Workflow **************************** ***************************************************************** * * Generating output/wkf_Demo_mapping_doc.html and output/wkf_Demo_workflow_doc.html files .... Done. * ***************************************************************** *** Perform Automated Code Review using pre-defined template **** ***************************************************************** * * Generating output/wkf_Demo_rev.html file .... Done. * ***************************************************************** *** Document all sessions Memory properties in a workflow ******* ***************************************************************** * * Generating output/wkf_Demo_mem.html file .... Done. * ***************************************************************** *** Generate code Migration checklist *************************** ***************************************************************** * * Generating output/wkf_Demo_mig.html file .... Done. * ***************************************************************** Volume in drive C has no label. Volume Serial Number is B01D-872D Directory of C:\Kalura\ICCUtil\output 02/08/2011 05:54 PM 1,277,412 wkf_Demo.parsed 02/08/2011 05:55 PM 1,174,252 wkf_Demo_mapping_doc.html 02/08/2011 05:56 PM 53,571 wkf_Demo_mem.html 02/08/2011 05:56 PM 23,364 wkf_Demo_mig.html 02/08/2011 05:55 PM 140,642 wkf_Demo_rev.html 02/08/2011 05:55 PM 216,628 wkf_Demo_workflow_doc.html 6 File(s) 2,885,869 bytes 0 Dir(s) 60,012,584,960 bytes free *****************************************************************

ICC Utility 1.0

Page 9

Changing log4j.logger.com.kalura.InfaParseInformaticaXML=WARN entry in log4.properties file to

log4j.logger.com.kalura.InfaParseInformaticaXML=INFO will now produce detail output as shown below:

*

*****************************************************************

*** Converts PowerCenter Code into a normalized XML Format ******

*****************************************************************

*

* Parsing input/wf_s_m_flow_1.xml and generating output/wf_s_m_flow_1.parsed fil

e ....

2011-02-08 18:18:52,452 [main] INFO com.kalura.InfaParseInformaticaXML - Loggin

g initialized.

2011-02-08 18:18:52,468 [main] INFO com.kalura.InfaParseInformaticaXML - Initia

lizing Local Properties.

2011-02-08 18:18:52,562 [main] INFO com.kalura.InfaParseInformaticaXML - REPOSI

TORY:Repository_Service

2011-02-08 18:18:52,577 [main] INFO com.kalura.InfaParseInformaticaXML - FOLDER

:CEP

2011-02-08 18:18:52,656 [main] INFO com.kalura.InfaParseInformaticaXML - SOURCE

:ing_direct_source

2011-02-08 18:18:52,671 [main] INFO com.kalura.InfaParseInformaticaXML - TARGET

:to_cep_flow_1

2011-02-08 18:18:52,718 [main] INFO com.kalura.InfaParseInformaticaXML - MAPPIN

G:m_flow_1


:ing_direct_source


:to_cep_flow_1


_TRANSFORMATION_SUMMARY:to_cep_flow_1


_TRANSFORMATION_SUMMARY:ing_sourcefile

2011-02-08 18:18:52,843 [main] INFO com.kalura.InfaParseInformaticaXML - TRANSF

ORMATION:SQ_ing_sourcefile

2011-02-08 18:18:52,859 [main] INFO com.kalura.InfaParseInformaticaXML - TRANSF

ORMATION:SEQTRANS

Please visit http://logging.apache.org/ for more logging options that can be set in log4.properties file.

ICC Utility 1.0

Page 10

InfaProcesses.properties

This file is currently not being used for this release of ICC Utility.

Execution Steps

Step1

[Extract Code from PowerCenter Repository]

In this step ICC Utility will first connect to Informatica Repository using pmrep command.

pmrep connect -r Repository_Service -h localhost -o 6005 -n username -x password

To keep things simple hostname along with few other parameters are currently hardcoded in

ICCUtility.bat . Please modify these parameters depending on your environment; you should be able to

find details instruction on pmrep in “Informatica command line reference.pdf” file. In the help manual

you will also find other ways to convert these parameter as environment variables.

*****************************************************************

*** Extract Code from PowerCenter Repository ********************

*****************************************************************

*

* Connecting to informatica repository ....

Informatica(r) PMREP, version [9.0.1 HotFix2], build [242.1111], Windows 32-bit

Copyright (c) Informatica Corporation 1994 - 2010

All Rights Reserved.

This Software is protected by U.S. Patent Numbers 5,794,246; 6,014,670; 6,016,50

1; 6,029,178; 6,032,158; 6,035,307; 6,044,374; 6,092,086; 6,208,990; 6,339,775;

6,640,226; 6,789,096; 6,820,077; 6,823,373; 6,850,947; 6,895,471; 7,117,215; 7,1

62,643; 7,254,590; 7,281,001; 7,421,458; and 7,584,422, international Patents an

d other Patents Pending.

Invoked at Tue Feb 08 18:18:50 2011

Connected to repository Repository_Service in Domain_vm as user Administrator

connect completed successfully.

Next command performed in this step is to export the workflow into a Powercenter Xml file.

pmrep objectexport -n %2 -o Workflow -f %1 -m -s -b -r -u input/%2.xml

ICC Utility will name the exported file as [Workflow Name].xml and store the file in input folder. On

successful completion you’ll see the detail message on your console

* Exporting wf_s_m_flow_1 workflow to input/wf_s_m_flow_1.xml ....

Informatica(r) PMREP, version [9.0.1 HotFix2], build [242.1111], Windows 32-bit

Copyright (c) Informatica Corporation 1994 - 2010

All Rights Reserved.

This Software is protected by U.S. Patent Numbers 5,794,246; 6,014,670; 6,016,50

1; 6,029,178; 6,032,158; 6,035,307; 6,044,374; 6,092,086; 6,208,990; 6,339,775;

6,640,226; 6,789,096; 6,820,077; 6,823,373; 6,850,947; 6,895,471; 7,117,215; 7,1

62,643; 7,254,590; 7,281,001; 7,421,458; and 7,584,422, international Patents an

d other Patents Pending.

Invoked at Tue Feb 08 18:18:51 2011

02/08/2011 18:18:51

Analyzing Object Dependencies...

ICC Utility 1.0

Exporting selected objects in repositor

Exporting selected objects in folder CEP ...

Fetching Source Definition [ing_direct_source] ...

Exporting Source Definition [ing_direct_source] ...

Exporting Target Definition [to_cep_flow_1] ...

Fetching Mapping [m_flow_1] ...

Exporting Mapping [m_flow_1] ...

Fetching SessionConfig [default_session_config] ...

Exporting SessionConfig [default_session_config] ...

Fetching Workflow [wf_s_m_flow_1] ...

Exporting Workflow [wf_s_m_flow_1] ...

Export is completed.

Exported 5 object(s) - 0 Error(s), - 0 Warning(s)

objectexport completed successfully.

Completed at Tue Feb 08 18:18:52 2011

Step2

[Converts PowerCenter Code into a normalized XML Format

This entry in ICC Utility will convert exported workflow from Step1 and normalize it

The process takes all the attributes from exported

at one location along with the tr

down the line.

java -Xms128m -Xmx512m

Shown below is an sample output of parsed

Usually you would not need the parsed file, but as a developer its very handy especially when you

compare attribute by attribute values between two different version of the code.

Exporting selected objects in repository Repository_Service ...

Exporting selected objects in folder CEP ...

Fetching Source Definition [ing_direct_source] ...

Exporting Source Definition [ing_direct_source] ...

Exporting Target Definition [to_cep_flow_1] ...

Exporting Mapping [m_flow_1] ...

Fetching SessionConfig [default_session_config] ...

Exporting SessionConfig [default_session_config] ...

Fetching Workflow [wf_s_m_flow_1] ...

orkflow [wf_s_m_flow_1] ...

0 Warning(s)

Completed at Tue Feb 08 18:18:52 2011

PowerCenter Code into a normalized XML Format]

This entry in ICC Utility will convert exported workflow from Step1 and normalize it

process takes all the attributes from exported XML and combines them together so that they save

ransformation. This parsed xml makes it things easier for other steps

Xmx512m -jar InfaParseInformaticaXML.jar input/%2.xml output/%2.parsed

Shown below is an sample output of parsed PowerCenter xml file

Usually you would not need the parsed file, but as a developer its very handy especially when you


Page 11

This entry in ICC Utility will convert exported workflow from Step1 and normalize it for later processing.

XML and combines them together so that they save

ansformation. This parsed xml makes it things easier for other steps

jar InfaParseInformaticaXML.jar input/%2.xml output/%2.parsed

Usually you would not need the parsed file, but as a developer its very handy especially when you’ve to


ICC Utility 1.0

Page 12

Step3

[Document PC Mapping and PC Workflow]

This step onward parsed xml file will be used to document PowerCenter mapping and workflow

java -Xms128m -Xmx512m -jar InfaDocumentor.jar output/%2.parsed

output/%2_mapping_doc.html output/%2_workflow_doc.html

The above command will go thru the parsed xml file and produce two html files.

1. Mapping Htmlfile

2. Workflow Htmlfile

Documenting Mapping & Workflow in Html file has lot of advantages, to list few of them:

• Html file allows you to perform text based search on any transformation and any logic

within a mapping.

• You can read your code without having to open Informatica Designer or Workflow

Manager

• Html file will highlight all the modified attributes for any transformation in a mapping or

workflow

ICC Utility 1.0

Mapping Htmlfile

This file documents all the mapping referenced in a workflow.

mapping is as following:

Sources

All the sources in a folder are documented under

below the field name. If it’s a XML source

value linked with that field.

This file documents all the mapping referenced in a workflow. The structure used to document

documented under this section. You’ll also notice field description listed

XML source (Webservice source etc) then an expression

Page 13

The structure used to document

lso notice field description listed

xpression will display xpath

ICC Utility 1.0

Page 14

Targets

All the targets in a folder are documented under this section. You’ll also notice field description listed

below the field name. If it’s a XML target (Webservice target etc) then an expression will display xpath

value linked with that field.

Mapplets

All the mapplets in a folder are documented under this section. ICC Utility will also document all the

transformation associated with the mapplet.

Mapplet Parameter

ICC Utility 1.0

Page 15

Mappings

All the mappings in a folder are documented under this section.

Transformations

All the transformations in a mapping are documented below mapping section. Depending on the

transformation you might see a slightly different layout, few of them a discussed and screenshot are

attached.

Expression Transformation

Output Port, Local Variable, Input/Output Port

will be displayed in different colors

ICC Utility 1.0

Page 16

XML Generator

Original XML Schema can be viewed under XMLInfo attribute

SQL Transformation

Original SQL Query can be viewed under SQL_Query attribute

ICC Utility 1.0

Page 17

XML Parser

Different Groups can be seen along the group name

There are other transformations that will display valuable information.

User Defined Functions

All the UDF’s in a folder are documented under this section.

ICC Utility 1.0

Workflow Htmlfile

This file documents all the session

workflow html file is as following:

Workflow

Workflow attributes will be displayed under this

different color.

Session

All the session in a folder are documented under

session referenced in a workflow. The structure used to document

is as following:

Workflow attributes will be displayed under this session, any modified attribute will be shown with a

documented under this section.

Page 18

The structure used to document session in

attribute will be shown with a

ICC Utility 1.0

Page 19

Connections

All the connections in a session are documented below session section.

Other Transformation Level Properties

All other transformations in a session are documented below session section.

ICC Utility 1.0

Page 20

Step4

[Perform Automated Code Review using pre-defined template]

Using the parsed xml file this step will perform automated code review.

java -Xms128m -Xmx512m -jar InfaCodeReview.jar output/%2.parsed output/%2_rev.html

The above command will perform following check:

• Transformation

o Prefix does not match naming convention

o Name is not camel case

o Instance description is missing

• Variables

o Local variable should start with v_

o Description should be present for local variables

o Output variable should start with o_

• Tracing Level set to Normal for all transformation

• Highlight missing description

o In Workflow

o In Mapping

Result is detailed html report produced in the output directory as shown below:

ICC Utility 1.0

Page 21

Step5

[Document all sessions Memory properties in a workflow]

Using the parsed xml file this step will extract memory details for all the session within a workflow.

java -Xms128m -Xmx512m -jar InfaMemoryRequirement.jar output/%2.parsed

output/%2_mem.html

The result of command shown above is html report with memory properties for all the session as shown

in the report below:

ICC Utility 1.0

Page 22

Step6

[Generate code Migration checklist]

Using the parsed xml file this step will generate migration checklist.

java -Xms128m -Xmx512m -jar InfaCodeMigrationDocumentor.jar output/%2.parsed

output/%2_mig.html

The result of command shown above is html migration checklist as shown below:

Known Limitations

• None

Documents

ICC Utility - User Guide - Informatica Marketplace Utility... · ICC Utility User Guide ... ready to be migrated in Test ... Template driven approach makes the code review process