SAP Information Steward

SAP Information StewardDocument Version: 4.2 Support Package 1 (14.2.1.0) - 2013-12-05

User Guide

Table of Contents1 Access Information Steward. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.1 Accessing Information Steward from BI launch pad. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2 Data Insight. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.1 Data Insight overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.1.1 Enterprise project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.2 Add tables and files to projects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.2.1 Adding tables by name. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.2.2 Adding tables by browsing metadata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.2.3 Adding files by browsing metadata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.2.4 Searching for tables or files within a connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.3 Manage imported metadata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192.3.1 Reconciling imported metadata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202.3.2 Removing a table or file from a project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .202.3.3 Object dependents and dependencies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.3.4 Deleting imported metadata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2.4 Manage file formats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232.4.1 File formats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .232.4.2 Creating a file format for a flat file. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242.4.3 Creating a file format for an excel workbook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .282.4.4 Organize fields in a file format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.4.5 Adding files by browsing metadata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.4.6 Creating a file group for related files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .312.4.7 Editing a file format. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322.4.8 Copying a file format. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332.4.9 Assigning a file format to a file. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332.4.10 Viewing usage of a file format. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342.4.11 Deleting a file format. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

2.5 About Data Insight views. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352.5.1 Tabs in View Editor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362.5.2 Creating a view. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372.5.3 Create a new output column. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382.5.4 Editing a view. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392.5.5 Adding sources to a view. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402.5.6 Deleting sources from a view. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402.5.7 Editing output schema to customize views. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412.5.8 Showing usage of a view. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412.5.9 Copying a view. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422.5.10 Working with SAP tables in views. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

2 2013 SAP AG or an SAP affiliate company. All rights reserved.User Guide

Table of Contents

2.5.11 Advanced expression editors for views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442.5.12 Join conditions for views. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462.5.13 Creating a join condition expression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462.5.14 Cache options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482.5.15 Creating a join ranking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492.5.16 Filter conditions for views. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .492.5.17 Aggregate condition expressions and Group By. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512.5.18 View Editor window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

2.6 View source data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542.6.1 Viewing source data from Browse Metadata window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .552.6.2 Viewing source data from the Source Tables list. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552.6.3 Viewing a subset of columns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .562.6.4 Filtering viewed data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 562.6.5 Sorting viewed data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 562.6.6 Exporting viewed data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

2.7 Profile tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582.7.1 Creating a columns profiling task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592.7.2 Creating an Addresses profiling task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602.7.3 Creating a Dependency profiling task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 622.7.4 Creating a Redundancy profiling task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 632.7.5 Creating a Uniqueness profiling task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 642.7.6 Define column profile task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 662.7.7 Creating a content type profiling task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 662.7.8 View profile results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

2.8 Managing profile tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 742.8.1 Editing a profiling task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 742.8.2 Running a profiling task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 752.8.3 Viewing a profile task in the Enterprise Project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 762.8.4 Removing a profiling task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 762.8.5 Creating a profiling tasks filter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .772.8.6 Viewing and changing profile display settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 772.8.7 Profile display settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 772.8.8 Associating content types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

2.9 Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 792.9.1 About validation rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 792.9.2 Calculating scores. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 802.9.3 Rules sample failed data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 812.9.4 Creating rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 822.9.5 Advanced expressions for rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 872.9.6 Custom attributes for rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 912.9.7 Processing a rule. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

User GuideTable of Contents 2013 SAP AG or an SAP affiliate company. All rights reserved. 3

2.9.8 Viewing rule results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 972.9.9 Managing rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

2.10 Data Cleansing Advisor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1202.10.1 Setting up a new data cleansing solution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1212.10.2 Managing your data cleansing solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1302.10.3 Cleanse and match results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1332.10.4 Cleanse results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1352.10.5 Match results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1432.10.6 Data columns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1682.10.7 Data cleansing solution summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1802.10.8 Publishing your data cleansing solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1812.10.9 Closing a data cleansing solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

2.11 Scorecards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1812.11.1 Adding a key data domain. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1822.11.2 Removing objects from the key data domain. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1832.11.3 Viewing scorecards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1842.11.4 Viewing failed data samples from a scorecard. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1862.11.5 Viewing impact and lineage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1872.11.6 Scorecard links. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1882.11.7 How scorecards are calculated. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188

2.12 Business value analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1902.12.1 Selecting the currency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1902.12.2 Configuring costs per failed record. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1912.12.3 Business value analysis views. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

2.13 Scripting language. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1942.13.1 Categories for building expressions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1942.13.2 Validation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1972.13.3 Language syntax. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

2.14 Data types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3022.14.1 Descriptions of data types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3022.14.2 Data type processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309

2.15 Content types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3232.15.1 Creating custom content types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3242.15.2 Viewing content type dependencies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3252.15.3 Managing content types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3252.15.4 Content type best practices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328

2.16 Export and import projects and objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3292.16.1 Exporting a project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3302.16.2 Exporting an object. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3302.16.3 Exporting content types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3322.16.4 Exporting rule bindings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .333


Table of Contents

2.16.5 Exporting rules to Data Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3342.16.6 Importing a project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3352.16.7 Importing an object. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3372.16.8 Importing content types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3382.16.9 Importing rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3392.16.10 Importing rule bindings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340

2.17 Exporting profile results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342

3 Metadata Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3433.1 Metadata Management overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343

3.1.1 Metadata Management Directory page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3443.1.2 Search for objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3463.1.3 Metadata relationships overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3523.1.4 User-defined relationships between objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3643.1.5 Adding annotations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3663.1.6 Custom attributes for metadata objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3663.1.7 Change preferences for relationships. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3673.1.8 Export relational objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3703.1.9 Viewing custom metadata reports. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3753.1.10 View data from Data Insight connections in Metadata Management. . . . . . . . . . . . . . . . . . . 377

3.2 SAP BusinessObjects Enterprise Metadata integrator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3793.2.1 Objects in an SAP BusinessObjects Enterprise integrator source. . . . . . . . . . . . . . . . . . . . . 3793.2.2 Metadata displayed for SAP BusinessObjects Enterprise objects. . . . . . . . . . . . . . . . . . . . . 3853.2.3 Impact analysis of Crystal Report fields. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3883.2.4 View personal folder collection results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3893.2.5 Relationships for universe objects and reports. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3903.2.6 Viewing lineage diagrams from BI launch pad. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3963.2.7 Viewing relationships with SAP NetWeaver BW sources. . . . . . . . . . . . . . . . . . . . . . . . . . . 3973.2.8 Relationships for users and groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400

3.3 SAP NetWeaver Business Warehouse Metadata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4033.3.1 SAP NetWeaver Business Warehouse metadata objects. . . . . . . . . . . . . . . . . . . . . . . . . . . 4033.3.2 Metadata displayed for SAP NetWeaver Business Warehouse objects. . . . . . . . . . . . . . . . . 4083.3.3 Viewing lineage diagrams from BI launch pad. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4093.3.4 Viewing relationships of SAP NetWeaver BW objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410

3.4 SAP Data Services Metadata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4163.4.1 Metadata displayed for Data Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4163.4.2 SAP Data Services objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4193.4.3 Defining data flow metadata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4223.4.4 Displaying an instance of a data flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4223.4.5 Defining datastore metadata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4223.4.6 Displaying metadata for a table or column. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4233.4.7 Exploring a column instance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424


3.4.8 Data Services object relationships. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4243.4.9 Viewing Data Services Auto Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429

3.5 SAP HANA integrator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4303.5.1 SAP HANA database objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4303.5.2 Limiting SAP HANA package collection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4313.5.3 Support for multiple versions of SAP HANA Studio. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4333.5.4 SAP HANA Information Model and Column views. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4333.5.5 Viewing relationships in an SAP HANA database source. . . . . . . . . . . . . . . . . . . . . . . . . . . 434

3.6 SAP Sybase PowerDesigner metadata integrator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4343.6.1 PowerDesigner metadata objects for Metadata Management. . . . . . . . . . . . . . . . . . . . . . . 4353.6.2 PowerDesigner metadata integrator relationships. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436

3.7 Data Modeling Metadata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4363.7.1 Metadata displayed for data modeling objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4363.7.2 Viewing relationships of CWM objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437

3.8 Data Federator Metadata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4383.8.1 Metadata displayed for Data Federator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4383.8.2 Data Federator object relationships. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4403.8.3 Navigating Data Federator Metadata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4413.8.4 Viewing Data Federator object relationships. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .442

3.9 Relational Database Metadata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4463.9.1 Relational Database objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4463.9.2 Exploring an instance of a view. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4463.9.3 Viewing database source relationships. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447

4 Metapedia. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4494.1 Metapedia overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .449

4.1.1 Understanding terms and categories. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4504.1.2 Metapedia users and groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .450

4.2 Working with Metapedia terms and categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4514.2.1 Approval process overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4514.2.2 Creating terms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4534.2.3 Approving or rejecting terms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4544.2.4 Associating terms with objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4554.2.5 Viewing Metapedia terms associated with objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4574.2.6 Editing properties of terms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4584.2.7 Relating terms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4584.2.8 Deleting terms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4594.2.9 Filtering and editing terms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4594.2.10 Reverting a term. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4604.2.11 Canceling a term task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4614.2.12 Creating categories. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4614.2.13 Editing properties of categories. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462


Table of Contents

4.2.14 Deleting a category. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4624.2.15 Associating a term with categories. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4634.2.16 Email notifications for terms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463

4.3 Metapedia search. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4644.3.1 Metapedia search results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4654.3.2 Performing an advanced search in Metapedia. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466

4.4 Viewing Metapedia terms from BI launch pad. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4674.5 Custom attributes for Metapedia terms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4684.6 Importing and exporting terms and categories with Excel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469

4.6.1 Standard format file import. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4694.6.2 Custom format file import. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4714.6.3 Exporting to Excel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472

4.7 Glossary information from PowerDesigner. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4734.7.1 Security for glossary terms from PowerDesigner metadata integrator. . . . . . . . . . . . . . . . . 475

5 Cleansing Package Builder. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4765.1 Getting started with Cleansing Package Builder. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476

5.1.1 About cleansing data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4765.1.2 About cleansing packages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4775.1.3 Cleansing package types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4785.1.4 Cleansing package links. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4795.1.5 Terminology used in Cleansing Package Builder. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4805.1.6 Comparing custom and person and firm cleansing packages. . . . . . . . . . . . . . . . . . . . . . . . 4825.1.7 Planning and creating a custom cleansing package for operational data. . . . . . . . . . . . . . . . 4845.1.8 Refining a person and firm cleansing package. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4885.1.9 Cleansing package lifecycle: develop, deploy and maintain . . . . . . . . . . . . . . . . . . . . . . . . . 4925.1.10 Starting the Cleansing Package Builder. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4945.1.11 Closing cleansing packages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494

5.2 Create cleansing packages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4945.2.1 Creating new cleansing packages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4945.2.2 About parsing strategy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4955.2.3 Creating a custom cleansing package. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4965.2.4 Creating a person and firm cleansing package. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5005.2.5 Import Data Services 3.2 cleansing package files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5005.2.6 Creating a cleansing package from a Data Services 3.2 dictionary. . . . . . . . . . . . . . . . . . . . 508

5.3 Refine cleansing packages in Design mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5095.3.1 Introducing Design mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5095.3.2 IGNORED and EXTRA attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5125.3.3 Assign standard forms and variations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5125.3.4 About phrases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5185.3.5 Context definition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5195.3.6 Comparing phrases and context definition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536


5.3.7 Defining category format. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5365.3.8 Adding additional sample data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5375.3.9 Parsed values assigned to more than one attribute (conflict resolution). . . . . . . . . . . . . . . . 5385.3.10 Resolving conflicts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5405.3.11 Searching in Design mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5425.3.12 Drilling down into record groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542

5.4 Refine cleansing packages in Advanced mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5435.4.1 About Advanced mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5435.4.2 About domains. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5435.4.3 Searching in Advanced mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5475.4.4 Manage Classifications and Variations tab. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5495.4.5 Edit rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565

5.5 Publish cleansing packages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5885.5.1 Publishing a cleansing package. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5885.5.2 Exporting an ATL file for Data Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5885.5.3 Updating a published cleansing package. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5895.5.4 Viewing published cleansing packages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 590

5.6 Person and firm cleansing packages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5905.6.1 Standardization in person and firm cleansing packages. . . . . . . . . . . . . . . . . . . . . . . . . . . 5905.6.2 Person and firm attributes in discrete data fields. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5915.6.3 About parsing data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5925.6.4 Region-specific data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593

5.7 Parse common types of data (reference data). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6145.7.1 Parsing common types of data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6145.7.2 About phone numbers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6165.7.3 Adding international phone number patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6185.7.4 Editing international phone number patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6185.7.5 Adding email domain names . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6195.7.6 Editing email domain names. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6205.7.7 About Social Security number parsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6205.7.8 Social Security data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6205.7.9 About user-defined pattern matching rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6215.7.10 Simple user-defined rule example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6225.7.11 Complex user-defined rule example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6225.7.12 Adding user-defined patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6235.7.13 Adding user-defined rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6245.7.14 Editing user-defined patterns or rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6245.7.15 Deleting a user-defined pattern or rule. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6255.7.16 User-defined pattern rule elements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625

5.8 How Data Cleanse processes data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6265.8.1 Data parsing overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 626


Table of Contents

5.8.2 Parse and classify input for person and firm cleansing packages. . . . . . . . . . . . . . . . . . . . . 6265.8.3 Parse and classify input for custom cleansing packages. . . . . . . . . . . . . . . . . . . . . . . . . . . 6305.8.4 Rule-based parsing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6325.8.5 Output and standardization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633

5.9 Regular expressions and patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6355.9.1 Regular expressions within Cleansing Package Builder. . . . . . . . . . . . . . . . . . . . . . . . . . . . 6355.9.2 Consider the parsing strategy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6365.9.3 How regular expressions work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6375.9.4 Operators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6375.9.5 Special characters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6385.9.6 Bracketed list matches a single input character . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6395.9.7 Parentheses group characters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6395.9.8 Define a regular expression pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6405.9.9 Examples of regular expressions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643

5.10 Data Cleanse transform options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6445.10.1 Cleansing Package options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6455.10.2 Date options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6475.10.3 Other standardization options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6495.10.4 Person standardization options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6525.10.5 Gender standardization options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6545.10.6 Firm standardization options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6555.10.7 Report and analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656

5.11 Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6565.11.1 Troubleshooting FAQs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6565.11.2 Unlocking a cleansing package. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658

5.12 Keyboard shortcuts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6605.13 Reserved words. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6605.14 Maximum field length. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6645.15 Cleansing package states and statuses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664

6 Match Review. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6676.1 Match Review. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6676.2 Match and consolidation using Data Services and Information Steward. . . . . . . . . . . . . . . . . . . . . . . 6686.3 Managing Match Review Configurations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 669

6.3.1 Creating or editing a match review configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6696.3.2 Best record strategy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6746.3.3 Defining a best record strategy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6746.3.4 Best record strategy descriptions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6756.3.5 Running a match review configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6786.3.6 Deleting a Match Review Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 678

6.4 Managing Match Review Tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6796.4.1 Editing a match review task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 679


6.4.2 Force completing a match review task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6806.4.3 Canceling a match review task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6816.4.4 Email notifications for match review tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 681

6.5 Match review summary for task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6826.5.1 Match Groups by Status. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6836.5.2 Match Groups by Size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6836.5.3 Match Groups by Reviewer/Approver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6836.5.4 Match Review Progress. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684

6.6 Reviewing match groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6856.7 Approving match groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6866.8 Creating or changing a best record. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6886.9 Monitoring match review. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6906.10 Moving records. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 691

6.10.1 Moving records to an existing group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6916.10.2 Moving records into the group being reviewed. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6926.10.3 Moving records to a new group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6936.10.4 Master record assignment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 694

6.11 Modifying a match group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6946.12 Best record lineage table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6956.13 Match review activity table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6976.14 Data Review configuration file. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .698

7 Custom Attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7007.1 Custom Attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7007.2 Creating and associating custom attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7007.3 Modifying a custom attribute. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7027.4 Reordering the display of custom attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7037.5 Importing and exporting custom attribute definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7047.6 Deleting a custom attribute. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 705

8 SAP Information Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 706

9 Glossary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 707


Table of Contents

1 Access Information StewardPrior to accessing Information Steward, you must have your user rights set up and a project created in the Central Management Console (CMC).Access Information Steward by typing the name of the computer you are accessing into your browser. For example:http://webserver:8080/BOE/InfoSteward

You can also access Information Steward through SAP BusinessObjects Business Intelligence Launchpad (Metapedia and Metadata Management modules only).To be able to perform certain tasks in Information Steward, you must obtain the correct permissions, or be assigned a particular user or group role by an administrator. Based on the permissions, you may be able to perform some tasks on development or test systems, but not necessarily on SAP application production systems. Use one of the following options as a workaround: Contact the system administrator to receive development permissions to the SAP application. Perform the task on a development or test system. After doing so, export the project and import it into the

production system.

Related Information

Administrator Guide: Accessing Information Steward for administrative tasks

1.1 Accessing Information Steward from BI launch padAccess SAP Information Steward from within BI launch pad to view lineage diagrams for Crystal reports or Web Intelligence documents, or to view Metapedia terms and definitions associated with a Crystal report or Web Intelligence document.To access Information Steward from BI launch pad:1. Log into BI launch pad and open the Home tab.2. Click the Information Steward icon in the My Applications list on the right of the Home tab.

Information Steward opens.

Related Information

Viewing lineage diagrams from BI launch pad [page 396]Access SAP Information Steward from within BI launch pad to view the data lineage of a Crystal report or Web Intelligence document.

User GuideAccess Information Steward 2013 SAP AG or an SAP affiliate company. All rights reserved. 11

Viewing Metapedia terms from BI launch pad [page 467]


Access Information Steward

2 Data Insight2.1 Data Insight overviewUse Data Insight to find data quality issues, validate how relationships comply with standards, and to monitor, measure and improve the quality of the data. When considering the purchase of technology, use Data Insight to show the return on investment and the impact of poor data on the business. Gain a high-level view into the quality of data and drill down to see scores for tables and columns as well as to view a sample of the data that does not pass the rules. Then monitor the trends in data quality over time.Each deployment of SAP Information Steward supports multiple users in one or more Data Insight projects to assess and monitor the quality of data from various sources. A Data Insight project is a collaborative workspace for data stewards and data analysts to assess and monitor

the data quality of a specific domain and for a specific purpose (such as customer quality assessment, sales system migration, and master data quality monitoring).

A Data Insight project and the users that belong to the project are defined by the Data Insight Administrator on the Central Management Console (CMC) of SAP Business Intelligence Platform. For information about the CMC, see the Business Intelligence Platform Administrator Guide.

A connection defines the access to a data source. A data source can be a relational database, application, or file. The connections to data sources and user rights to each data connection are defined by the Data Insight Administrator on the CMC.

When you open a Data Insight project, Information Steward provides two views: Workspace

A workspace is a collaboration space for data stewards to collaborate with data analysts to perform tasks such as profile and analyze data, define rules, define file formats, and set up data quality scorecards.

ScorecardA scorecard is a visual way for data stewards to easily understand and analyze data quality from a data domain perspective.

Data Quality Advisor

Data Quality Advisor encompasses several new product features that guides data stewards to rapidly develop a solution to measure and improve the quality of their information assets. Data stewards can use Data Quality Advisor features to assess their data and get a recommendation on validation and cleansing rules with additional capabilities to review and tune the cleansing rules to better meet their business needs.1. Assess the data using content type profiling.

Content type profiling provides suggested meaning based on the data and the metadata attributes in the column. Automatically identifying the content types helps to suggest appropriate validation rules after running a content type profiling task.

2. Receive validation and cleansing rule recommendations.To better define best practices regarding the implementation of validation, cleansing and matching rules, use the Data Validation Advisor and Data Cleansing Advisor.After content type profiling, users can view new generated validation rule recommendations by launching Data Validation Advisor. These rules are automatically suggested based on the content type profiling results

User GuideData Insight 2013 SAP AG or an SAP affiliate company. All rights reserved. 13

for person, firm, and address-related data as well as statistical analysis of the data. Then data stewards determine if they want to use the validation rule on the profiled data. Likewise, if an existing rule is available for a defined content type, Information Steward suggests using the rule on other profiled data that has the same content type, when applicable.Data Cleansing Advisor is a new feature within Data Insight that simplifies the data cleansing process by intelligently recommending a data cleansing solution for your data. The recommended data cleansing solution is based on your data's content-type identification results and SAP best practices for your specific data.A data cleansing solution is a collection of cleansing and matching rules for party data that you can publish for use in Data Services Workbench. Party data is data about individuals or organizations, such as name, address, and phone data. Examples of party data include customer data and vendor data.

3. Tune the cleansing and matching rules using Data Cleansing Advisor.In Data Cleansing Advisor, you can see the data cleansing solution's effects on your data by viewing cleanse and match results, including statistics, graphs, and a data preview. If the data preview reveals unsatisfactory results, you can change option settings and see a new preview of the results on your data.The data changes that you preview in Data Cleansing Advisor are temporary and for planning purposes only. Data Cleansing Advisor creates staged data based on your data and works on the staged data. Nothing is changed in your actual data until the job is processed in Data Services Workbench.

For details about content type profiling, see the Information Steward User Guide sections Creating a content type profiling task and Editing content types.For details about automatic validation rule suggestions, see the Information Steward User Guide sections Managing profile tasks and Rules overview.For details about Data Cleansing Advisor, see the Information Steward User Guide section Data Cleansing Advisor.

Related Information

Add tables and files to projects [page 15]Creating a file format for a flat file [page 24]Creating a view [page 37]Profile tasks [page 58]Running a profiling task [page 75]Creating a rule [page 83]Create rules to ensure the data complies with business requirements.Creating and scheduling a rule task [page 96]Adding a key data domain [page 182]How scorecards are calculated [page 188]

2.1.1 Enterprise projectSAP Information Steward creates a special project, named Enterprise, which cannot be deleted. Only the profile and rule results in the Enterprise project are available for the Data Insight Connection category on the Metadata Management tab. Therefore, you can view profile results and data quality scores at the table and column level for tables under the Data Insight Connection category in Metadata Management.


Data Insight

Related Information

Viewing a profile task in the Enterprise Project [page 76]Running a rule task in the Enterprise project [page 97]Viewing profile and scorecard results in Metadata Management [page 72]

2.2 Add tables and files to projectsAdd tables and files (flat files or Excel workbooks) to an SAP Information Steward Data Insight project so that you can profile the data, run validation rules to monitor the data quality, and create data quality scorecards to measure quality against data quality metrics (or expected data quality targets). The tables, flat files and Excel workbooks that you add to your project are displayed on the Workspace Home tab in your project.The software provides options for adding tables and files through the Add menu in the Workspace Home window.You can add tables and files when you know the exact name, or, you can browse Data Insight connections to find the table or file.

Related Information

Adding tables by name [page 15]Adding tables by browsing metadata [page 16]Adding files by browsing metadata [page 18]

2.2.1 Adding tables by nameOne efficient way to add tables to your project is by knowing the names of your tables.

NoteWhen using SAP application connections, some users may not be able to perform this task on the production system because they may not have the necessary permissions to read metadata from SAP systems. Use one of the following options as a workaround: Contact the system administrator to receive development permissions to the SAP application. Perform the task on a development or test system. After doing so, export the project and import it into the

production system.

Follow these steps to add a table to your Data Insight project when you know the table name:1. In the Workspace Home tab of your project in the Data Insight tab, click Add Table By Name in the top

menu bar.The Add Table to Project window opens.


2. Select the connection name from the Connection drop-down list.3. Type the table name.4. If needed, change the value in Connection.5. If needed, select a different value in Object Type.6. Click Import.After you add a table to your Data Insight project, the table name appears in the Workspace Home tab of your project.

Related Information

Searching for tables or files within a connection [page 18]Removing a table or file from a project [page 20]Export and import projects and objects [page 329]

2.2.2 Adding tables by browsing metadataWhen you do not know the exact name of the table that contains the data that you want to profile, you can browse the metadata in each SAP Data Insight connection to find the table.

NoteWhen using SAP application connections, some users may not be able to perform this task on the production system because they may not have the necessary permissions to read metadata from SAP systems. Use one of the following options as a workaround: Contact the system administrator to receive development permissions to the SAP application. Perform the task on a development or test system. After doing so, export the project and import it into the

production system.

To add a table by browsing metadata, follow these steps:1. Open the applicable project in the Projects page of the Data Insight tab.

The Workspace Home window opens.2. Click Add Tables from the top menu bar.

The Browse Metadata window opens. The left pane of the window lists the names of connections that you are authorized to use.

3. Click the name of the connection in the list.The applicable objects appear in the right pane. The objects that appear depend on the type of connection.


Data Insight

Table 1: Connection types and related objectsConnection type Objects that appear in

Metadata Browsing window

Viewing the definition of tables

Database Tables and their columns

For a database or application connection, view the definition of the tables as they exist in the external source by following these steps:

1. Select View External Metadata at the top of the Browse Metadata window.

2. Expand the schema name to view the table names.3. If you know the table name, type it in the search box

(located above the list of table names). Or, filter the list of tables that are displayed by typing the first few letters of the name in the search box.

4. Expand the table name node to view the table columns and attributes.

SAP Application Tables and Extractors For a file connection, display the fields and attributes of the file by selecting the format definition for each file from the drop-down list in the Format Name column.

4. Select the check box in front of the name of each object you want to add to your project and click Add to Project in the top menu bar.

After you add a table to your Data Insight project the software does the following: Imports the metadata from the external source into the Information Steward repository for all projects (if it

has not already been imported by another project). Displays a check mark in the Imported column for the table (in External Metadata view). Displays the table name and definition in the Imported Metadata view. Adds the table name to the list of tables in the Workspace Home window of your project. Adds the connection and table definition to the Metadata Management tab in the Data Insight Connections

area.

NoteIf you remove a table, view or file from your project, the metadata still exists in the repository and other projects can use it.

Related Information



2.2.3 Adding files by browsing metadataYou must define file formats before you can add a file to your project.To add files to a project:1. Open a project from the Data Insight Projects window2. Open the Workspace Home page and select Add Files from the top menu bar.

The Browse Metadata window opens. The left pane of the window lists the names of connections to which you are authorized.

3. (Optional) Type the first few letters of the connection name in the Filter Connections text box located above the list of connection names.The list of connections is reduced to a subset.

4. Click the connection name from the subset to list the files in the connection.The file names appear in the right panel.

5. Select the drop list in the Format Name column and select the format definition for the file.6. Select the check box in front of the name of each file name you want to add to your project and click Add to

Project from the top menu bar.After you add a file to your Data Insight project, the file name appears in the Workspace Home tab of your project.

Related Information


2.2.4 Searching for tables or files within a connectionThe list of files and tables for a connection in the Browse Metadata window can be very lengthy. To make the search for a table or file within a connection easier, filter the list of objects in the connection by using the Search text box.

NoteYou cannot use this method to search for a hierarchy name.

Access the Browse Metadata window by selecting Tables or Files in the Add menu located at the top of the Workspace Home window.

NoteWhen using SAP application connections, some users may not be able to perform this task on the production system because they may not have permissions to read or search the metadata.


Data Insight

Follow the steps below to reduce the number of tables or files listed in the left panel of the Browse Metadata window:1. Select a connection from the list in the left pane of the Browse Metadata dialog to show a list of table schema

names.2. Expand a table schema name to show its objects (tables or files) on the right pane of the window.3. Type the exact name or a substring of the name that you are searching for in the Search text box located

above the list of objects in the right side of the window and click the Search icon.The list of objects includes only the names that contain the string that you typed. You can type any substring that appears in the table or file name and the Browse Metadata window displays all names that contain that string. For example, if you type the word order, the following tables could appear: Production.WorkOrder, Production.WorkOrderRouting, Purchasing.PurchaseOrderDetail, Purchasing.PurchaseOrderHeader.

4. If your connection type is an SAP application, you can also search for a table by its description.a) Type any substring that appears in the description of the object into the Search text box above the list of

object names.b) In the drop-down list next to the search icon, click Description.c) Click the Search icon.

5. Clear the search filter by clicking the Delete icon to the right of the Search text box above the list of object names.

Related Information

Adding tables by browsing metadata [page 16]View source data [page 54]Export and import projects and objects [page 329]

2.3 Manage imported metadataWhen you add a table to your Data Insight project, the metadata is imported into the SAP Information Steward repository. If two or more projects add the same table, Information Steward stores only one copy of the database table metadata in the Information Steward repository. Therefore, all projects that have the same table share the same copy of the imported table metadata, which results in the following conditions: If the schema has changed at the external source and a user reimports the metadata into one project, the

reimported metadata will be visible to all projects. A user cannot delete the table metadata (from the Browse Metadata dialog) if other projects are still using the

table.

Related Information

Reconciling imported metadata [page 20]


Removing a table or file from a project [page 20]Object dependents and dependencies [page 21]

2.3.1 Reconciling imported metadataWhen you add a table to your Data Insight project, the metadata from the external source is imported into the Information Steward repository. It is possible that the definition of the external table will change after you add it to your project, which could cause some of your profile tasks to fail.

NoteWhen using SAP application connections, some users may not be able to perform this task on the production system because they may not have the necessary permissions to read metadata from SAP systems. Use one of the following options as a workaround: Contact the system administrator to receive development permissions to the SAP application. Perform the task on the development or test system. After doing so, export the project and import it into

the production system.

To determine if the schema definition in the external source has changed since it was imported into Information Steward:1. Go to the Browse Metadata window.2. Select the name of your connection from the list on the left panel, and expand the schema name to display the

list of tables.3. Select the check box in front of the name of each table you want in the list on the right panel.4. Click Reconcile in the top menu bar.

If one or more object definitions changed in the source system since they were imported into Data Insight , a check mark appears in the Changed column.

5. To import the most recent metadata from the external system, select the objects you want on the right panel and click Re-import in the top menu bar.

6. If you receive a warning message that lists projects that are currently using the object that you are trying to reimport, notify the users of those projects to check if they need to redefine their profile tasks because of the revised schema.

Related Information

Adding tables by browsing metadata [page 16]

2.3.2 Removing a table or file from a projectIf the table or file is no longer required for a project, you can remove it.


Data Insight

NoteWhen you remove a table or file from a project, you delete all the profile results, sample data, rule results, and sample failed data from the Information Steward repository. For more information about dependent objects that are deleted, see Object dependents and dependencies [page 21].

1. Log on to Information Steward with a user name that belongs to the Data Insight Analyst group or higher. Users in the Data Insight Basic User group cannot remove a table or file from a project. For more information, see Type-specific rights for Data Insight objects in the Administrator Guide.

2. In the Workspace Home tab of your project, expand the name of the connection, then expand the schema name that contains the table or file that you want to remove.

3. Select the name of the table or file that you want to remove.4. Determine if the table is being used by a dependent object.

a) Select the Show usage ( ) icon located in the top menu bar.b) On the Usage for the selected tables dialog, expand the node for the table to display the dependent

objects.c) To delete the dependent object, select it, click Delete, and click Yes to the warning message that asks you

if you want to remove the selected objects.d) To close the Usage for the selected tables dialog, click the X icon in the upper right corner of the dialog.The table is removed from the project and its associated profile results, sample data, rule results and sample data that failed the rules are logically deleted from the Information Steward repository. This logically deleted information will be physically deleted by the Purge utility the next time it runs.

5. If there are no more dependent objects, you can select the table name and click Delete.6. On the warning message that asks you if you want to remove the selected objects, click OK.

2.3.3 Object dependents and dependenciesIf you delete an object from the Workspace of your Data Insight project, its dependent objects are also deleted. However, you cannot delete an object if another project is using one or more of its dependents. The following table shows the dependencies and dependents of objects that you delete from the Workspace in Data Insight.

Table 2: Delete object dependents and dependenciesObject to delete from Workspace of a project on Data insight

Object dependencies that prevent deletion

Dependent objects that will also be deleted

Profile Task none Profile Task InstanceRule Binding To delete a rule binding, you must

delete the following object that uses the rule binding from the current project: Key Data Domain

Rule results Sample data that failed rule


Object to delete from Workspace of a project on Data insight



Rule To delete a rule, you must delete the following objects that use the rule from the current project: Rule Binding Key Data Domain

Sample data that failed rule Threshold settings

Key Data Domain none Following objects that are related to the Key Data Domain: Quality Dimension Rule Rule Binding scores and

weights

Remove Table or File To remove a table or file from a project, you must delete the following objects that use the table or file in this project: Rule Binding Profile or Rule Task View Rule with Lookup function

referring to the table

Profile results Sample data Rule results Sample data that failed rule

View To delete a view, you must remove the following objects that use the view from all projects. Rule Binding Profile or Rule Task

Profile results Sample data Rule results Sample data that failed rule

Table metadata deleted from Browse Metadata dialog

To delete the table metadata, you must remove the following objects that use the metadata from all projects: Table

File Format To delete a file format, you must remove the following object that uses the file format from all projects. File

Column, when deleted as a result of reimporting a table on Browse Metadata dialog

To delete a column, you must remove the following objects that use the deleted column from all projects:


Data Insight

Object to delete from Workspace of a project on Data insight



Rule Lookup function View Rule binding Profile or Rule Task

2.3.4 Deleting imported metadataIf the table metadata is no longer required for any projects, you can delete it.1. Log on to Information Steward with a user name that belongs to the Data Insight Administrator group or

Administrator group.2. On the Data Insight tab, select the name of the project that contains the table whose metadata you want to

delete.3. In the Workspace Home tab of your project, click Add Tables in the top menu bar to display the Browse

Metadata window.4. From the list in the left pane, select the name of the connection and expand the schema name that contains

the table.5. Select the Imported Metadata option to display only the imported tables and enable the options in the top

menu bar.6. From the list in the right pane, select the check box in front of the name of the table whose metadata you want

to delete.7. To determine if the table is being used by a dependent object:

a) Select the Show usage icon.b) On the Projects and functions using the selected tables screen, expand the node for the table to display

the names of the projects using this table.c) If projects are currently using this table, you must remove the table from each project before you can

delete the table metadata. For details, see Removing a table or file from a project [page 20].8. If no projects are using the table, select the check box in front of the name of the table whose metadata you

want to delete and click Delete.

2.4 Manage file formats

2.4.1 File formatsTo view file formats, open the Data Insight tab and then click the File Formats icon from the tabs listed at the left side of the window. Selecting a file format displays read-only properties.A flat file is a database that stores data in a plain text file. To process flat files, the format of the flat file must be defined. In a delimited file, each field is separated by delimiters, such as commas or tabs. In a fixed-width file, each


field is set to a certain length of characters. For example, the Last Name fields may be set to a width of 25 characters.An Excel workbook is a spreadsheet file that contains several pages of worksheets. To process Excel workbooks in Information Steward, you must define the format of the Excel workbook. The format definition specifies the data to access by worksheet name, worksheet index number, or a range of cells. For example, a workbook might contain separate worksheets for customer data and orders data, and you define a file format for each worksheet.Once you associate a file with a format and add it to a project, the file is treated like a table object. You can perform the same tasks on a file format that you can on a table. For example, you can profile a file, run validation rules against it without adding it to a view, and create a view on top of the file.

2.4.2 Creating a file format for a flat fileBefore you can create a file format, you need to create a connection to the location of the files using the Central Management Console (CMC). For more information, see the SAP Information Steward Administrator Guide. To process a file, SAP Information Steward needs a file format.To create a file format, follow these steps:1. In the File Formats tab of your project in the Data Insight tab, click New Flat File Format in the top menu

bar.The File Format Editor opens.

2. Enter a name for the format in the File format name text box.3. (Optional) Enter a description of the file format in the Description text box.

After you save the file format, the file name and description displays as read-only properties on the upper right side of the File Formats tab.

4. Click Browse to select a sample file from the file connection. A sample file allows you to view sample data from a file so that you can validate the definition of the format.

5. Define the settings in your file format. If your file format is separated by a delimiter, select the Delimited by option and then select the character

type that marks the separation of the fields. If the columns of your file are defined by character units, select the Fixed Width option. Select First row contain field names to specify that the first row in the record contains field names. To set more advanced options for formatting, headers, and so on, click More Options.

6. Organize the fields if necessary by using Move Up and Move Down.7. To validate the file format definition, click View Sample Data.8. To pre-populate the field definitions with names and data types, click Propose Schema.

The application uses sample data from the file and user-defined options in the sample file and proposes the name of fields and data types. The fields are populated in the fields panel. You can rename the fields, change the type, and so on in the fields panel. View the sample data at the bottom of the window.

NoteField lengths and types are based on a few sample rows. You may need to make changes for the file format to apply to the whole file. Verify the length and types carefully.


Data Insight

9. Click Save.

Related Information

Editing a file format [page 32]Assigning a file format to a file [page 33]Organize fields in a file format [page 30]Creating a file group for related files [page 31]Copying a file format [page 33]Viewing usage of a file format [page 34]Deleting a file format [page 34]Also see: Administrator Guide: Defining a Data Insight connection to a file

2.4.2.1 File format options for flat filesThe following options are available in the File Format - more options window for fully defining file formats for flat files.

From the File Formats tab, click New Flat File Format and then More Options.

Table 3: File Format - More Options windowName Value DescriptionGeneralType Delimited or Fixed Width Delimited files have an indicator such as a

comma or a tab between each field and each record.

Fixed width files have a specific record length and field length.

Adaptable schema For delimited files: No or Yes Indicates whether the schema of a delimited file format is adaptable or fixed. Yes indicates that the schema is

adaptable. The actual file can contain fewer or more columns than indicated by the file format. If a row contains fewer columns than expected, the application loads null values into the columns with missing data. If a row contains more columns than expected, the application ignores the additional data.


Name Value Description No indicates that the schema is fixed. The

application requires the number of columns in each row to match the number of columns specified in the file format. If the data contains fewer or more columns than expected, an error is generated. Refer to the advanced options for Error Handling.

Data Alignment For fixed width files: Byte or Character

Character indicates that fields in your data are measured as character. All processing is in character semantics.

Byte indicates that fields in your data are measured as bytes. All processing is in byte semantics.

For example, in character semantics, if you define a column as varchar(30), it means that the column has 30 characters regardless of the number of bytes for each character. In byte semantics, varchar(30) means 30 bytes. Byte semantics recognizes bytes only, it does not recognize characters.

NoteThe file reader reads as many bytes in a column based on the length of the column. For multibyte code pages, a character can be more than one byte. There could be times when trailing bytes of the last character of a column exceeds the length of the column. When this happens, the reader continues until it reaches the total length of the character.

Delimiters Column For delimited files:

Tab, semicolon, comma, space, or any character sequence

The character sequence that indicates the end of one column and the beginning of the next.

Row {new line}, {Windows newline}, {Unix new line}, or any character sequence

A character sequence that indicates where one row of data ends and the next begins.


Data Insight

Name Value DescriptionText Single quotation marks,

double quotation marks, or {none}

Denotes the start and end of a text string. All characters (including those specified as column delimiters) between the first and second occurrence of this character is a single text string.

FormatEscape character Any character sequence, or

{none}A special character sequence that causes the application to ignore the normal column delimiter. Characters following the escape character sequence are never used as column delimiters.For example, suppose you specify a forward slash as the escape character and a comma as the column delimiter. Then you must have a forward slash to have a comma appear inside a field.

NULL indicator Any character sequence, or {none}

Special character sequence that the application interprets as NULL data.

Ignore row marker(s) Any character sequence, or {none}

A character sequence, which when found at the beginning of rows, causes the application to ignore the row when reading the file or automatically creating metadata. To enter multiple character sequences, separate each with a semicolon. To include a semicolon or backslash as a marking character, precede with a backslash.

Date yyyy.mm.dd or {none} The date format for reading date values to and from the file.

Time hh24:mi:ss or {none} The time format for reading time values to and from the file.

Date Time yyyy.mm.dd hh24:mi:ss or {none}

The date time format for reading date time values to and from the file.

Blank padding For Fixed width files: Leading or Trailing

The method for trimming extra blanks before or after the fields. Setting this option informs the application where the padding is located. Leading: Blanks added to the left of

(before) the data. Trailing: Blanks added to the right of

(after) the data.

HeadersSkip rows Integer When the file has heading information,

comments or other non-data information,


Name Value Descriptionenter the number of rows the application should skip when reading the file.

Skip row header No or Yes When the file has heading information, this indicates whether the first row of data contains column names and should be skipped. Selecting Yes causes the application to use the data in the first row as column names.

LocaleLanguage The three-letter language

abbreviation specified in the ISO 639-2T standard

Specifies the human language (for example, Korean, Japanese, or English) in which the data is stored and processed.

Code page Supported code pages Specifies the sequence of bits that define a character. For example, the Japanese code page contains ASCII, Greek, Cyrillic, and Japanese characters, thereby supporting the English, Greek, Russian, and Japanese languages.

Error HandlingLog data conversion warnings No or Yes Determines whether to include data type

conversion warnings in the error log.Maximum warnings to log Integer greater than 0 or No

limitIf Log data conversion warnings is enabled, you can limit how many warnings the application logs.

Capture data conversion errors

No or Yes Determines whether to capture data type conversion errors when processing.

Capture row format errors No or Yes Determines whether to capture row format errors when processing.

Max errors to stop processing Integer greater than 0 or No limit

If Capture data conversion errors or Capture row format errors is enabled, you can limit how many invalid rows the application processes before stopping the job.

Write errors to error file No or Yes If Capture row format errors is enabled, you can write invalid rows to an error file.

Error File File name, file name including full path name, or blank

If Write error rows to file is enabled, type the name of the file in which to record the invalid rows.

2.4.3 Creating a file format for an excel workbookThis procedure creates a file format for an excel workbook.


Data Insight

Before you can create a file format, you need to create a connection to the location of the files using the Central Management Console (CMC). For more information, see the SAP Information Steward Administrator Guide. To process a file, SAP Information Steward needs a file format.To create a file format, follow these steps:1. In the File Formats tab of your project in the Data Insight tab, click New Excel File Format in the top

menu bar.The File Format Editor opens.

2. Enter a name for the format in the File format name text box.3. (Optional) Enter a description of the file format in the Description text box.

After you save the file format, the file name and description display as read-only properties on the upper right side of the File Formats tab.

4. Click Browse to select a sample file from the file connection. A sample file allows you to view sample data from a file so that you can validate the definition of the format.

5. Define Data access settings to specify how Information Steward will access the data: Worksheet name: Information Steward will access the worksheet identified by the worksheet name. If you

have clicked Browse to select an Excel file, the Worksheet name combo box will be populated with the names of worksheets in that file.

Worksheet Index: The index (position) of the worksheet. The leftmost worksheet is index 1, the next one is 2, and so on.

Named Range: A name that specifies a range of cells on a particular worksheet. Information Steward will look only at the data in the cells identified in the named range. The source spreadsheet must also have such a named range, which can be in a different location than what was used to create the format. The named range must be a finite selection of cells.

More Options: To set more advanced options for formatting, headers, and so on.Table 4: File Format - More Options windowName Value DescriptionExtend Range No or Yes This option is visible only if the value of the

Range option is not {All cells}. When set to No, Information Steward should look only at the data rows in the given range.For example, the data only in rows 1 through 5 (minus the row header) are visible to Information Steward. When set to Yes, Information Steward should look at all rows, not just the ones in the given range.

Skip row header No or Yes When the file has heading information, this indicates whether the first row of data contains column names and should be skipped. Selecting Yes causes the application to use the data in the first row as column names.

Code page Supported code pages Specifies the sequence of bits that define a character. For example, the Japanese code

User GuideData Insight 20

Documents

SAP Information Steward