Upload
phungphuc
View
272
Download
0
Embed Size (px)
Citation preview
SAP SE Dietmar-Hopp-Allee 16 69190 Walldorf Germany +49 1805 34 34 34 fax +49 1805 34 34 20
SAP NetWeaver
Master Data Management (MDM)
MDM Import Manager
Reference Guide
Release: MDM 7.1 SP17
December 2016
ii MDM Import Manager Reference Guide
© 2016 SAP SE or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. Adobe, the Adobe logo, Acrobat, PostScript, and Reader are trademarks or registered trademarks of Adobe Systems Incorporated in the United States and other countries. Apple, App Store, FaceTime, iBooks, iPad, iPhone, iPhoto, iPod, iTunes, Multi-Touch, Objective-C, Retina, Safari, Siri, and Xcode are trademarks or registered trademarks of Apple Inc. Bluetooth is a registered trademark of Bluetooth SIG Inc. Citrix, ICA, Program Neighborhood, MetaFrame now XenApp, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems Inc. Computop is a registered trademark of Computop Wirtschaftsinformatik GmbH. Edgar Online is a registered trademark of EDGAR Online Inc., an R.R. Donnelley & Sons Company. Facebook, the Facebook and F logo, FB, Face, Poke, Wall, and 32665 are trademarks of Facebook. Google App Engine, Google Apps, Google Checkout, Google Data API, Google Maps, Google Mobile Ads, Google Mobile Updater, Google Mobile, Google Store, Google Sync, Google Updater, Google Voice, Google Mail, Gmail, YouTube, Dalvik, and Android are trademarks or registered trademarks of Google Inc. HP is a registered trademark of the Hewlett-Packard Development Company L.P. HTML, XML, XHTML, and W3C are trademarks, registered trademarks, or claimed as generic terms by the Massachusetts Institute of Technology (MIT), European Research Consortium for Informatics and Mathematics (ERCIM), or Keio University. IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, z10, z/VM, z/OS, OS/390, zEnterprise, PowerVM, Power Architecture, Power Systems, POWER7, POWER6+, POWER6, POWER, PowerHA, pureScale, PowerPC, BladeCenter, System Storage, Storwize, XIV, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, AIX, Intelligent Miner, WebSphere, Tivoli, Informix, and Smarter Planet are trademarks or registered trademarks of IBM Corporation. Microsoft, Windows, Excel, Outlook, PowerPoint, Silverlight, and Visual Studio are registered trademarks of Microsoft Corporation. INTERMEC is a registered trademark of Intermec Technologies Corporation. IOS is a registered trademark of Cisco Systems Inc. The Klout name and logos are trademarks of Klout Inc. Linux is the registered trademark of Linus Torvalds in the United States and other countries. Motorola is a registered trademark of Motorola Trademark Holdings LLC. Mozilla and Firefox and their logos are registered trademarks of the Mozilla Foundation. Novell and SUSE Linux Enterprise Server are registered trademarks of Novell Inc. OpenText is a registered trademark of OpenText Corporation. Oracle and Java are registered trademarks of Oracle and its affiliates. QR Code is a registered trademark of Denso Wave Incorporated. RIM, BlackBerry, BBM, BlackBerry Curve, BlackBerry Bold, BlackBerry Pearl, BlackBerry Torch, BlackBerry Storm, BlackBerry Storm2, BlackBerry PlayBook, and BlackBerry AppWorld are trademarks or registered trademarks of Research in Motion Limited. SAVO is a registered trademark of The Savo Group Ltd. The Skype name is a trademark of Skype or related entities. Twitter and Tweet are trademarks or registered trademarks of Twitter. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Wi-Fi is a registered trademark of Wi-Fi Alliance. SAP, R/3, ABAP, BAPI, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, SAP HANA, the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, Sybase, Adaptive Server, Adaptive Server Enterprise, iAnywhere, Sybase 365, SQL Anywhere, Crossgate, B2B 360° and B2B 360° Services, m@gic EDDY, Ariba, the Ariba logo, Quadrem, b-process, Ariba Discovery, SuccessFactors, Execution is the Difference, BizX Mobile Touchbase, It's time to love work again, SuccessFactors Jam and BadAss SaaS, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE in Germany or an SAP affiliate company. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.
SAP SE Neurottstrasse 16 69190 Walldorf Germany +49 6227 74 74 74 fax +49 6227 75 75 75
MDM Import Manager Reference Guide iii
Contents
Introduction .............................................................................. 13
Welcome .......................................................................................... 15 MDM Import Manager Overview ...................................................... 16
Data Import Fantasy..................................................................... 17 Data Import Reality ...................................................................... 18
Challenge: Combine Information from Multiple Tables ......................... 19 Challenge: Create a Pivot Table .......................................................... 19 Challenge: Create a Reverse Pivot Table ............................................ 20 Challenge: Map Fields ......................................................................... 20 Challenge: Map Values ........................................................................ 20 Challenge: Convert Data Types ........................................................... 21 Challenge: Create Measurement Values.............................................. 22 Challenge: Convert Data Values .......................................................... 22 Challenge: Split a Single Field to Expand Hierarchy ............................ 23 Challenge: Split a Single Field into Multiple Fields ............................... 24 Challenge: Split a Single Field into Multiple Data Values ..................... 24 Challenge: Combine Two Fields to Build a Hierarchy ........................... 25 Challenge: Combine Fields to Create Value Combinations .................. 25 Challenge: Combine Fields to Reconstitute Hierarchy ......................... 26 Challenge: Combine Fields to Merge Values ....................................... 27 Challenge: Combine Fields to Create a Multi-Valued Field .................. 27 Challenge: Import Attribute Data as Name/Value Pairs ........................ 28 Challenge: Match Records ................................................................... 28
Key Advantages ........................................................................... 29 About This Guide ............................................................................. 31
What You Will Find in This Guide ................................................ 31 What You Will Not Find in This Guide .......................................... 31 Assumptions ................................................................................ 31 Screen Captures .......................................................................... 32
Part 1: Basic Concepts ............................................................ 33
What is Parametric Import?.............................................................. 35 The Current Tables ...................................................................... 36
Non-Main Table Imports ...................................................................... 37 Relationship Imports ............................................................................ 37
Table Joins and Lookups ............................................................. 38 Pivoting and Reverse Pivoting ..................................................... 38 Partitioning ................................................................................... 38 Value Mapping ............................................................................. 39 Value Conversion ......................................................................... 39 Record Matching .......................................................................... 39 Import Maps ................................................................................. 39 Record Filtering ............................................................................ 40
iv MDM Import Manager Reference Guide
Part 2: Guided Tour ................................................................. 41
Starting Import Manager ................................................................... 43 Connecting Import Manager to an MDM Repository ..................... 43
Setting Up Unencrypted Repository Connections ................................. 43 Setting Up Secure Repository Connections .......................................... 44
Connecting Import Manager to Source Data ................................ 45 Source Properties ................................................................................ 45
Getting the Next Source Data File ................................................ 46 Connecting to a New Data Source ............................................... 46 Data Staging ................................................................................. 47 Excel-Related Import Challenges ................................................. 47
MDM Import Manager Main Window ................................................ 49 Switching Layout .......................................................................... 49 Selecting the Current Tables ........................................................ 50
Part 3: Import Workflow .......................................................... 51
Working with the MDM Import Manager ........................................... 53 MDM Import Manager at a Glance ............................................... 54
Source Hierarchy Pane ........................................................................ 55 Destination Hierarchy Pane .................................................................. 55 Source Preview Tab ............................................................................. 56 Partition Field/Value Tab ...................................................................... 57 Map Fields/Values Tab......................................................................... 57 Match Records Tab .............................................................................. 58 Import Status Tab................................................................................. 59 Status Bar ............................................................................................ 60
Configuration Options ....................................................................... 62 Import Maps...................................................................................... 69 Import Map Operations ..................................................................... 70
Opening and Closing Maps .......................................................... 70 Saving Maps ................................................................................. 72 Importing and Exporting Map Files ............................................... 73 Updating Maps for MDM 7.1 ......................................................... 74
Part 4: Browsing Source Data ................................................ 77
Overview .......................................................................................... 79 The Source Preview Tab .................................................................. 80 The Source Hierarchy Tree .............................................................. 81
Tables, Fields, and Data Values ................................................... 82 Table Joins and Lookups .............................................................. 82 Field Types in the Source Hierarchy Tree .................................... 83 XML Source Files and the Source Hierarchy Tree ....................... 84
Source Hierarchy Operations ........................................................... 85 Displaying Distinct Values ............................................................ 86 Removing Distinct Values ............................................................. 88 Setting The Case Sensitivity Option ............................................. 88
MDM Import Manager Reference Guide v
Adding an Empty Field ................................................................. 89 Adding a Clone Field .................................................................... 90 Renaming a Field ......................................................................... 91 Deleting a Field ............................................................................ 91 Joining Tables .............................................................................. 91 Looking Up a Field ....................................................................... 93 Splitting a Field ............................................................................ 94 Building a Hierarchy ..................................................................... 95 Creating a Compound Source Field ............................................. 96 Creating a Pivot Table.................................................................. 96 Creating a Reverse Pivot Table ................................................... 96
Part 5: Table Joins and Lookups ........................................... 97
Overview .......................................................................................... 99 Multiple Tables of Source Data .................................................. 100 Table Joins and Lookups ........................................................... 101 Joining Tables ............................................................................ 103 Looking Up a Field ..................................................................... 104
Part 6: Pivoting/Reverse Pivoting ........................................ 107
Overview ........................................................................................ 109 Pivoting ...................................................................................... 110
Challenge: Single Pair of Metadata/Data Fields ................................. 111 Challenge: Metadata Field Combination ............................................ 112 Challenge: Data Field Combination.................................................... 113 Challenge: Single Name/Value Field Pair .......................................... 114 Challenge: Multiple Name/Value Field Pairs ...................................... 115 Challenge: Multi-Valued Information .................................................. 116 Challenge: No Key Fields .................................................................. 117
Reverse Pivoting ........................................................................ 118 Challenge: Single Set of Data Fields .................................................. 119 Challenge: Metadata Field Combinations ........................................... 120 Challenge: Multiple Sets of Data Fields ............................................. 121
Pivoting and Reverse Pivoting Operations ..................................... 122 Creating a Pivot Table................................................................ 123
Example: Single Pair of Metadata/Data Fields ................................... 127 Example: Metadata Field Combination .............................................. 128 Example: Data Field Combination ...................................................... 129 Example: Single Name/Value Field Pair............................................. 130 Example: Multiple Name/Value Field Pairs ........................................ 131 Example: [Natural Order] Pivots ......................................................... 132
Creating a Reverse Pivot Table ................................................. 133 Example: Single Set of Data Fields .................................................... 135 Example: Metadata Field Combinations ............................................. 136 Example: Multiple Sets of Data Fields ................................................ 137
Imagining Horizontal Pivots ........................................................ 138 Pivoting vs. Adding Horizontal Pivots ................................................. 138 Reverse Pivoting vs. Removing Horizontal Pivots .............................. 140
vi MDM Import Manager Reference Guide
Part 7: Splitting Fields ........................................................... 143
Overview ........................................................................................ 145 Splitting Fields ................................................................................ 146
Splitting into Hierarchy ................................................................ 146 Splitting into Multiple Fields ........................................................ 148 Setting the Split Delimiters.......................................................... 150
Part 8: Classification Data Import ........................................ 153
Overview ........................................................................................ 155 Building a Hierarchy ....................................................................... 156
Parent/Child Value Pairs w/ Node Name .................................... 156 Parent/Child Value Pairs w/o Node Name .................................. 157 Hierarchy Codes w/ Node Name ................................................ 157 The Create Hierarchy Field Command ....................................... 158
Taxonomy Import ............................................................................ 161 Importing the Hierarchy .............................................................. 162 Importing the Attributes............................................................... 163 Importing the Attribute Text Values ............................................ 165 Importing Attribute Links ............................................................. 166
Attribute Data Import as Name/Value Pairs .................................... 167 The Attribute Display Configuration Option................................. 168 The [Attribute/Value] Hierarchy Field .......................................... 168 The Attribute/Value Hierarchy ..................................................... 169
Part 9: Partitioning Fields/Values ........................................ 171
Overview ........................................................................................ 173 Partitioning and Field Mapping ................................................... 173
Source Field Partitioning .................................................................... 174 Destination Field Partitioning .............................................................. 176 Source and Destination Field Partitioning ........................................... 178 Field Value Partitioning ...................................................................... 179
Distinct Value Lists and Value Combinations ............................. 180 Record Subsets .......................................................................... 182 Partitions in the Source and Destination Hierarchies .................. 183 The Partition Field/Value Tab ..................................................... 184 Field and Value Partitioning ........................................................ 185 Inheritance .................................................................................. 185 Multiple Partitions ....................................................................... 186 Nested and Combined Partitions ................................................ 187
Nested vs. Combined ......................................................................... 189 To Combine or Not to Combine .......................................................... 191 Merged Values ................................................................................... 192
Mathematical Analysis ................................................................ 193 Infinite Restructuring Flexibility ................................................... 194 Splitting vs. Partitioning: A Comparison ...................................... 194
MDM Import Manager Reference Guide vii
Partitioning Operations .................................................................. 196 Partitioning by a Single Field or Attribute ................................... 197 Partitioning by Multiple Items ..................................................... 198 Combining Partitions .................................................................. 199 Setting the Partition Combine Delimiter ..................................... 200 Splitting a Partition ..................................................................... 201 Removing Partitions ................................................................... 202 Reordering Partitions ................................................................. 202 Restoring Inheritance ................................................................. 203
Part 10: Mapping Fields/Values ............................................ 205
Overview ........................................................................................ 207 The Map Fields/Values Tab ....................................................... 208
The Field Mapping Grids.................................................................... 209 The Value Conversion and Mapping Grids ......................................... 210
Value Mapping ........................................................................... 212 Eliminating Value Differences ............................................................ 213 Collapsing Source Value Variations ................................................... 214 Mapping to Multi-Valued Destination Fields ....................................... 215 Mapping to Hierarchy Destination Fields ............................................ 216 Mapping to Object Lookup Destination Fields .................................... 217 Mapping to Main Table Lookup Destination Fields ............................. 218 When Value Mapping Is Required ..................................................... 220
Value Conversion ....................................................................... 221 Automatic Trimming of Leading and Trailing Spaces ......................... 221 Automatic Data Type Conversion ...................................................... 222 Measurement Value Creation ............................................................ 222
Source Field-Level Settings ....................................................... 224 Destination Field-Level Settings ................................................. 225 Hierarchy Values ........................................................................ 226
Hierarchy Representation in Source Data .......................................... 227 Importing Into a Hierarchy Lookup Field ............................................ 228 Importing Into the Hierarchy Lookup Table ........................................ 228
NULL Values .............................................................................. 230 Mapping To and From NULL Values .................................................. 230 Converting To and From NULL Values .............................................. 231 Missing Source Data Fields ............................................................... 231 The NULL Handling Options .............................................................. 231 The Flat NULL Handling Option ......................................................... 232 The Hierarchy NULL Handling Option ................................................ 233 The NULL Interpretation Option ......................................................... 236
The Case Sensitivity Option ....................................................... 237 Field Mapping Delimiters ............................................................ 238 Value Conversion Filters ............................................................ 238 Lookups Consisting of Multiple Display Fields ........................... 241
Compound Lookup Fields .................................................................. 241 Mapping From a Single Corresponding Source Field ......................... 242 Mapping From Multiple Corresponding Source Fields ........................ 242 Promoted Lookup Table Display Fields .............................................. 243
viii MDM Import Manager Reference Guide
Map-Within-a-Map (Create Compound Field Command) ..................... 243 Qualified Lookup Fields and Lookups-Within-Lookups ....................... 244 Value Mapping and Lookup Table Display Fields ............................... 247
Multi-Valued Destination Fields .................................................. 249 Field Mapping to Multi-Valued Fields .................................................. 249 Field Mapping to Coupled Numeric Attributes ..................................... 250 The Multi-Valued Update Option ........................................................ 251 The Qualified Update Option .............................................................. 252 Product Applications and Automotive Parts Data................................ 257 The Tuple Update Option ................................................................... 259
Multilingual Destination Fields .................................................... 265 Object Lookup Destination Fields ............................................... 265
The Data Group Display Option ......................................................... 265 Writeable Once Destination Fields ............................................. 266 GM Time Fields .......................................................................... 267 The MDIS Unmapped Value Handling Option ............................ 268
Field Mapping Operations .............................................................. 269 Source Field Mapping ................................................................. 270 Mapping Fields ........................................................................... 271
Field Mapping Grid Icons ................................................................... 271 Source Fields ..................................................................................... 271 Destination Fields............................................................................... 271
Unmapping Fields ....................................................................... 273 Automapping Source Fields ........................................................ 274 Finding Matching Destination Fields ........................................... 274 The Find Matching Field Options ................................................ 275 Creating a Compound Source Field ........................................... 275 Setting the Field Mapping Delimiters .......................................... 276 Setting The Case Sensitivity Option ........................................... 278 Setting the Flat NULL Handling Option ....................................... 279 Setting the Hierarchy NULL Handling Option ............................. 280 Applying Field-Level Value Conversion Filters ........................... 281 Filtering Unlinked Attributes ........................................................ 283 Adding a New Destination Attribute ............................................ 284 Setting the NULL Interpretation Option ....................................... 285 Setting the Multi-Valued Update Option ..................................... 286 Setting the Qualified Update Option ........................................... 287 Setting the Tuple Update Option ................................................ 289 Setting the Data Group Display Option ....................................... 290 Setting the MDIS Unmapped Value Handling Option ................. 291
Value Conversion and Mapping Operations ................................... 294 Source Value Conversion and Mapping ..................................... 295
Source Values Grid ............................................................................ 296 Destination Values Grid ...................................................................... 297
Editing Source Values ................................................................ 297 Single-Valued Destination Field ......................................................... 297 Multi-Valued Destination Field ............................................................ 298
Mapping Source Values.............................................................. 299 Mapping a Source Value .................................................................... 303
MDM Import Manager Reference Guide ix
Unmapping Values ............................................................................ 304 Adding New Destination Values ......................................................... 304 Adding New Destination Hierarchy..................................................... 306 Removing New Values ...................................................................... 309 Automapping Source Values ............................................................. 309 Mapping to a Compound Lookup Field .............................................. 310 Mapping to a Coupled Numeric Attribute............................................ 312
Mapping to an Object Lookup Destination Field ......................... 312 Using the Data Groups Hierarchy ...................................................... 313 Adding a New Data Group ................................................................. 313
Applying Value-Level Value Conversion Filters ......................... 314 Restoring Converted Values .............................................................. 314 Setting to Original Values .................................................................. 314 Setting Values to NULL ..................................................................... 315 Clearing Values ................................................................................. 315 Applying Operators ............................................................................ 316 Adjusting Measurement Units ............................................................ 317 Normalizing Values ............................................................................ 318 Changing the Case of Values ............................................................ 318 Replacing Values ............................................................................... 318 Accepting Truncated Values .............................................................. 320 Expanding Year Ranges .................................................................... 321 Expanding Month Qualifications ......................................................... 322
Finding Unmapped Source Values ............................................ 322 Reordering Mapped Destination Values..................................... 323 Finding Mapped Destination Values .......................................... 324 Selecting Source Values in a Hierarchy ..................................... 325 Handling Missing Source Data ................................................... 326
Mismatches Between Number of Header and Record Fields ............. 327
Part 11: Matching Records ................................................... 329
Overview ........................................................................................ 331 The Match Records Tab............................................................. 332
The Value Matching Lists................................................................... 333 The Default Import Actions Grid ......................................................... 334 The Record Matching Grids ............................................................... 336
Record Matching Concepts ........................................................ 338 Value Matches ................................................................................... 338 Record Matches ................................................................................ 339 Match Level ....................................................................................... 340 Match Type ........................................................................................ 340 Example: Single Matching Field ......................................................... 342 Example: Single Matching Field Combination .................................... 342 Example: Multiple Matching Fields ..................................................... 343 Example: Multiple Matching Fields with Field Combination ................ 344
Value Mapping and Record Matching: A Comparison ............... 345 Record Matching and Lookup Table Display Fields............................ 345
Import Actions ............................................................................ 346 Import Action Inheritance ................................................................... 347 Source and Destination Record Import Actions .................................. 348 Active Records and Inheritance ......................................................... 349
x MDM Import Manager Reference Guide
Source Record Merging .............................................................. 350 The Key Field ..................................................................................... 350 The Merge Source Records Option .................................................... 351 The Merge Qualified Links Option ...................................................... 352 To Merge or Not to Merge .................................................................. 354 Merging Qualified Links and the Qualified Update Option ................... 355
Record Filtering .......................................................................... 356 Full Refresh vs. Net Change Synchronizations ................................... 356 Filtered Match Level and Match Types ............................................... 357 Record Filtering Behavior ................................................................... 358
The Virtual [Match Level/Type] Field .......................................... 359 Updating the Record Matching Field .......................................... 359 Normalized Record Matching Field ............................................. 360 Mask Updates and Record Replacement ................................... 361 Key Mappings and Record Replacement ................................... 362 Checkout and Record Replacement ........................................... 362 Checkout and Record Matching ................................................. 363 Record Matching on Non-Sortable Fields ................................... 363
Record Matching Operations .......................................................... 364 Adding a Single Matching Field .................................................. 365 Adding Multiple Matching Fields ................................................. 365 Combining Matching Fields ........................................................ 366 Splitting a Matching Field Combination ...................................... 366 Removing Matching Fields ......................................................... 366 Setting the Key Field .................................................................. 367 Setting the Active Records ......................................................... 367 Setting the Record Filtering Criteria ............................................ 368 Setting a Default Import Action ................................................... 370 Setting a Source Record Import Action ...................................... 371 Setting a Destination Record Import Action ................................ 372
Part 12: Importing Data ......................................................... 373
Import Execution ............................................................................. 375 Monitoring Import Status............................................................. 375 Performing the Import ................................................................. 377 Refreshing the Record Matching ................................................ 377
Import, Checkout, and Workflow ..................................................... 378 The Auto-Join Checked Out Records Option.............................. 378 The Checkout/Workflow Option .................................................. 379 Checkout and Workflow Failure Conditions ................................ 380 A Note about Import and Validations .......................................... 381
Import Error Logs ............................................................................ 382
Part 13: Importing Relationships ......................................... 383
Importing Parent/Child Relationship Links ...................................... 385 Relationship Configuration Options ............................................ 386
MDM Import Manager Reference Guide xi
Aggregation Option ............................................................................ 387 Update Options .................................................................................. 387
Relationship File Format ............................................................ 388
Part 14: Multilingual Support ................................................ 391
Multilingual Basics ......................................................................... 393 Language Layers ....................................................................... 393 Language Inheritance ................................................................ 394
Multilingual Operations .................................................................. 396 Connecting to the MDM Import Manager ................................... 396 Field Mapping to Multilingual Destination Fields ........................ 397
The Expand Multilingual Fields Option ................................................ 397 As Multiple Fields Setting ................................................................... 397 As Field and Language Setting ........................................................... 398
Value Mapping to Multilingual Destination Values ..................... 399 Importing Multilingual Data into Lookup Tables ......................... 400
Import into Flat Tables ....................................................................... 402 Import into Hierarchy Tables .............................................................. 402
Record Matching and Multilingual Destination Fields ................. 403 Source Record Merging and Multilingual Data ........................... 403
Part 15: Remote Systems and MDM .................................... 405
Introduction .................................................................................... 407 What is a Remote System? ........................................................ 407 Key Mapping .............................................................................. 407 [Remote System] and [Remote Key] Fields ............................... 408 Key Mapping Behavior ............................................................... 409 Key Mapping Reassignment ...................................................... 410 What is a Port? .......................................................................... 411
Remote System Operations ........................................................... 412 Connecting to a Remote System ............................................... 412 Connecting to a Port .................................................................. 412 Field Mapping ............................................................................ 413 Value Mapping ........................................................................... 413 Attribute Mapping ...................................................................... 414 Record Matching ........................................................................ 414
Lookup Table Imports Using the [Remote Key] Field ......................... 414 [Remote Key] Values and Lookup Field Automapping ............... 415 [Remote Key] Values and Record Replacement ........................ 416
Part 16: Reference ................................................................. 417
Command Reference ..................................................................... 418 File Menu ................................................................................... 418 View Menu ................................................................................. 418 Source Menu .............................................................................. 419
xii MDM Import Manager Reference Guide
Fields Menu ................................................................................ 419 Values Menu ............................................................................... 421 Records Menu ............................................................................ 421 Configuration Menu .................................................................... 422 Help Menu .................................................................................. 422 Partition Field/Value Tab Buttons ............................................... 422 Map Fields/Values Tab Buttons .................................................. 423 Match Records Tab Buttons ....................................................... 424 Miscellaneous Context Menus .................................................... 424
Display and Format Conventions .................................................... 425 Glossary ......................................................................................... 428
13
INTRODUCTION
This part of the reference guide provides an overview of the MDM Import Manager, including its distinctive features, terminology, and overall design philosophy
For information about installing MDM and using MDM repositories, see the MDM documentation on the SAP Help Portal at http://sap.help.com/nwmdm71.
.
MDM Import Manager Reference Guide 15
Welcome
Congratulations on choosing SAP NetWeaver MDM. The MDM Import Manager is one of the powerful suite of MDM tools for next-generation master data management and single-source, cross-media catalog publishing.
Most MDM customers have worked to painstakingly create an MDM repository that contains from tens of thousands to millions of products. The major impediment to creating these large repositories has been the complexity of aggregating – and normalizing – rich electronic master data.
Because of the shortcomings of existing import tools, such master data aggregation has required that source data be meticulously pre-processed into a format that can be imported, a labor-intensive and manual process that is very slow and very, very expensive.
By contrast, the MDM Import Manager is a GUI tool that eliminates most of this pre-processing and provides a WYSIWYG interface for cleansing, normalizing, and restructuring master data as part of the import process itself. Moreover, it supports the import of electronic master data from any tabular or XML format, with a consistent user interface and identical functional capabilities regardless of the source application file format.
Just as MDM’s patent-pending indexing and parametric search technology broke through the SQL performance bottleneck to deliver performance that is 100–1000 times faster than SQL alone, the MDM Import Manager radically simplifies and streamlines the import process to break through the “master data bottleneck,” dramatically reducing the time and expense associated with master data aggregation while simultaneously increasing accuracy and throughput by up to two or more orders of magnitude.
16 MDM Import Manager Reference Guide
MDM Import Manager Overview
Rich master data is the essential lifeblood of a truly usable MDM repository, which must contain not only basic transactional data about each product (e.g. a part number, price, and short description), but also product attribute information, long descriptions, images, PDFs, and even more-detailed descriptive information, such as marketing text, features and bullets, and so on.
Moreover, in a master data management system such as MDM that supports single-source cross-media publishing, the master data must be properly structured to be fully searchable in a published electronic catalog, and also sufficiently rich and of sufficiently high-quality to satisfy the requirement for product families and structured table layouts in a published paper catalog. Finally, for the repository to be of any use, the master data must be comprehensive and up-to-date.
Unfortunately, most of what passes for master data is really just raw transactional data; truly rich master data usually does not exist, and often must be painstakingly created from scratch – and at great expense – for each MDM repository. And even if the raw master data exists in electronic format, it often resides in hundreds if not thousands of data sources in a variety of incompatible formats and structures that are completely different from that of the MDM repository.
Aggregating raw product data into something that deserves to be called rich master data has historically been an expensive and almost insurmountable challenge, one that requires a tremendous amount of manual preprocessing (which must be repeated for each source data file), as well as direct user intervention for exception handling on a record-by-record basis when the records are actually imported.
All this is history with the MDM Import Manager, which solves the import challenge once and for all. It is a powerful GUI tool that allows you to import data from virtually any flat or relational electronic source file (e.g. Access, Excel, delimited text, SQL, XML, and any ODBC-compliant source), and to completely restructure, cleanse, normalize, and rationalize raw data into rich master data as part of the import process itself.
The MDM Import Manager features an intuitive and highly efficient user interface coupled with innovative, patent-pending functionality that reduce the time and effort required to import raw electronic data by a factor of 100–1000 times. Instead of performing the import a record at a time, as with other import tools, it allows you to preprocess the source data a field at a time, dramatically reducing from hours to literally minutes the amount of time spent processing each source data file.
Perhaps most importantly, the MDM Import Manager produces cleansed, normalized, and rationalized master data that is 100% accurate.
MDM Import Manager Reference Guide 17
DATA IMPORT FANTASY
Historically, most import tools that have been used to aggregate master data have operated on one single record at a time. Each field in the source (import) table is first mapped to a corresponding field in a destination (repository) table, and then each record is imported.
The problem is that for a successful data import using such tools, the source data must have the same overall table and field structure as the destination data, and the following additional conditions must all be met:
Single import table. All the source data fields to be imported must
be from a single table.
Values isolated in distinct fields. All the source data values
represented by the source data must be stored in distinct fields rather than embedded within one or more text description fields. In addition, each field must contain a single data value per record rather than multiple delimited values that represent either a hierarchy or multiple fields of information, or multiple values for the same field.
Fields and field names match. There must be a direct one-to-one
relationship between individual source fields and individual destination fields, and moreover, corresponding fields must have the same names.
NOTE ►► The relationship between individual source and destination fields identifies a field-oriented one-to-one relationship rather than the record-oriented one-to-one relationship typical of a relational DBMS. This is a new way of looking at data values field-by-field from a vertical point of view rather than row-by row from a horizontal point of view.
Values match. When mapping to destination fields whose values are
from a fixed domain of valid values (lookup fields and text attributes), there must be a direct mapping between the values within the individual source and destination fields that were previously mapped. Not only that, the source data values themselves must precisely match the corresponding destination data values (which must already exist) to avoid polluting the repository with illegal and inconsistent values from outside the valid value domain.
Data types match. When mapping to destination fields without a
value domain, the data types of the source and destination fields must be the same, and when mapping to numeric measurements, all source data values must have the same physical unit of measure.
Records match. Source records must be easily matched against
existing destination records, to avoid duplication of data. Based on the value of a single key field, new records must all be created and existing records must all be either all updated or all replaced.
18 MDM Import Manager Reference Guide
In the unlikely event that all of the above conditions were met – a data import fantasy – the import could proceed automatically without user intervention. The field mappings and the value mappings could both be done automatically because of the obvious mapping relationship between corresponding source and destination fields and between matching source and destination values, no data values would need to be converted, no exceptions would need to be handled, and the import could proceed uninterrupted against existing records because new records would all be created and existing records either all updated or all replaced.
DATA IMPORT REALITY
Unfortunately, all of the requirements for a clean import are rarely if ever met, and so the master data aggregator must perform a series of labor-intensive pre-processing steps that can take hours or days, to sufficiently manipulate, massage, and transform the source data into a format that can be properly handled by the import tool. Moreover, during record-at-a-time imports, the import tool must flag each discrepancy or unexpected value for exception handling, which means that the import can never proceed smoothly without interruption.
Specific, real-world import challenges include the following:
Combine information from multiple tables
Create a pivot table
Create a reverse pivot table
Map fields
Map values
Convert data types
Create measurement values
Convert data values
Split a single field to expand hierarchy
Split a single field into multiple fields
Split a single field into multiple data values
Combine two fields to build a hierarchy
Combine fields to create value combinations
Combine fields to reconstitute hierarchy
Combine fields to merge values
Combine fields to create a multi-valued field
Import attribute data as name/value pairs
Match records
Each of these challenges – and the MDM Import Manager Innovations designed to address them – are described in the following sections.
MDM Import Manager Reference Guide 19
Challenge: Combine Information from Multiple Tables
Import fantasy: Source data values all from a single import table
Import reality: Values in multiple import tables
MDM feature: Table joins and lookups
The source data fields to be imported may not all belong to a single source table. In this case, there must be a way to first join tables together and to then look up fields from each lookup source table into the primary source table, to create virtual extended source records (containing both primary table fields and lookup fields) for import. Alternatively, source tables must be “flattened” into a single table as a pre-processing step in an external application.
INNOVATION ►► Source data can be combined from multiple tables using the table join and lookup features of the MDM Import Manager, eliminating the need to pre-flatten source data in an external application.
Challenge: Create a Pivot Table
Import fantasy: Source data is structured as distinct fields in import table
Import reality: Data is structured as name/value pairs
MDM feature: Pivot table creation
When field and attribute data is stacked “vertically” in the source table as name/value pairs and needs to be rearranged into a “horizontal” table structure from which it can be imported into the fields and attributes of a regular table, the import tool must provide a mechanism to “rotate” the source data.
INNOVATION ►► The records and fields of a source table can be completely restructured using the pivot mechanism of the MDM Import Manager.
NOTE ►► Attribute data that is stacked vertically as name/value pairs can also be imported directly without pivots by first partitioning the name and value source fields, and then mapping the resulting name/value hierarchy to a special destination field containing a name/value hierarchy of attributes and attribute data values.
20 MDM Import Manager Reference Guide
Challenge: Create a Reverse Pivot Table
Import fantasy: Source data is stacked for import into qualified table
Import reality: Data is structured as rectangular table of records and fields
MDM feature: Reverse pivot table creation
When qualified data is arranged “horizontally” in the source table and needs to be rearranged into a “vertical” table structure from which it can be imported into the fields and qualifiers of a qualified table, the import tool must provide a mechanism to “rotate” the source data.
INNOVATION ►► The records and fields of a source table can be completely restructured using the reverse pivot mechanism of the MDM Import Manager.
Challenge: Map Fields
Import fantasy: Source and destination field names match
Import reality: Field names do not match
MDM feature: Field-level mapping
Corresponding source and destination fields may not have the same name. In this case, field mapping must be done manually before proceeding with the import.
INNOVATION ►► Fields can be mapped using the MDM Import Manager, to indicate not only the correspondence between source and destination fields, but also which source fields should be imported.
Challenge: Map Values
Import fantasy: Source and destination field values match
Import reality: Values do not match
MDM feature: Value-level mapping
When mapping to destination fields whose values are from a fixed domain of valid values (lookup fields and text attributes), corresponding source and destination values may not precisely match, and some source values may not already exist as destination values (e.g. source values USA, UK, and FRA correspond to destination values United States, England, and France, respectively, while source value MEX does not yet have a corresponding destination value). In this case, a record-at-a-time import tool must flag each discrepancy for exception handling during the import process itself, and the user must manually and tediously identify the corresponding destination value or indicate that a new value should be created to correspond to the source value. Alternatively, the source data must be meticulously cleansed and scrubbed as a pre-processing step in an external application.
MDM Import Manager Reference Guide 21
INNOVATION ►► Source fields that contain text values and are mapped to lookup fields or text attributes can be mapped not only at the field level, but also at the value level against the domain of legal destination values, eliminating the need to pre-cleanse source data in an external application.
INNOVATION ►► For each field, the MDM Import Manager collapses the set of all values across the entire set of source records down to the set of distinct values, whose number will usually be dramatically smaller than the total number of records, often by several orders of magnitude.
DATA INTEGRITY ►► Value-level mapping of source values forces a user to normalize source data, implicitly defining a precise set of transformation rules that: (1) eliminate differences between source and destination values; (2) collapse multiple source value variations into a single destination value; (3) properly map multiple delimited source values into a multi-valued MDM field; and (4) explicitly add new values to the set of destination values. The import cannot occur until each distinct source data value is mapped and all discrepancies in the source data resolved.
Challenge: Convert Data Types
Import fantasy: Source and destination field data types match
Import reality: Data types do not match
MDM feature: Automatic data type conversion
When mapping to destination fields without a value domain, the data types of the corresponding source and destination fields may be different, such as when numeric, currency, date, time, or Boolean source data are each stored as text values within text fields. In this case, the source data values must be converted to the proper data type as part of the import process.
INNOVATION ►► The data types of source data values are automatically converted by the MDM Import Manager based on the data type of the destination field to which the source field is mapped, even when stored in the source data as text values. If necessary, individual values can be manually edited to override errors or omissions in the source data.
DATA INTEGRITY ►► The import cannot occur until the data type of each distinct source data value is properly converted.
22 MDM Import Manager Reference Guide
Challenge: Create Measurement Values
Import fantasy: Source values all have the appropriate unit of measure
Import reality: Units of measure missing and inconsistent
MDM feature: Automatic measurement value creation
When mapping to destination fields that represent physical measurements (measurement fields and numeric attributes), the units of measure in the source data may be: (1) missing entirely, such as when all the values in a source field have the same implicit unit; (2) inconsistent, such as when different text strings are used to represent the same physical unit (e.g. inches, inch, in, or ‘"’); or (3) different for each source data value. In each case, the unit of measure strings must be normalized and appended to each numeric source data value to create measurement values as part of the import process.
NOTE ►► The syntactical challenge of creating measurements is further compounded because the source data values may be stored either: (1) in a single numeric field; (2) in a single text field; or (3) in a pair of fields (one containing the numeric value and the other containing the corresponding unit of measure string); additionally, each field may contain: (4) a single value; or (5) multiple delimited values. Each of these distinct cases must be understood and properly handled by the import tool.
INNOVATION ►► Unit of measure strings are automatically normalized and appended to each numeric source data value based on the physical dimension of the destination measurement field or numeric attribute to which the source field is mapped. If necessary, each numeric measurement value can be converted to a target unit, or manually edited to override errors or omissions in the source data.
DATA INTEGRITY ►► MDM’s measurement data type guarantees unit of measure consistency and permits each individual measurement value to have its own distinct unit. Moreover, unit of measure strings are automatically normalized against MDM’s built-in dictionary of over 2000 synonyms for over 750 units in over 70 physical dimensions. The import cannot occur until each numeric source data value has an associated unit of measure.
Challenge: Convert Data Values
Import fantasy: Source values properly formatted for import
Import reality: Values need to be adjusted, corrected, or reformatted
MDM feature: Value conversion grids and filters
MDM Import Manager Reference Guide 23
With or without value mapping, data type conversion, or measurement value creation, individual source data values may need to be manually adjusted, corrected, or reformatted.
INNOVATION ►► Not only can each individual source value be manually edited, built-in and custom-designed value conversion filters can automatically convert or reformat a single value or the entire set of distinct values with a single command. Built-in filters include: (1) conversion from one unit of measure to another, such as from inches to millimeters; (2) case change, such as from UPPER CASE to Mixed Case; or (3) month and year range expansion for efficient import of automotive data.
DATA INTEGRITY ►► Value conversion filters automate repetitive and error-prone transformations, eliminating manual typing and the possibility of user error. They also allow powerful reformatting algorithms to be applied to an entire group of values.
Challenge: Split a Single Field to Expand Hierarchy
Import fantasy: Source data values isolated in distinct fields
Import reality: Delimited values represent a hierarchy
MDM feature: Split into hierarchy
A single field may contain a set of delimited values that implicitly represent the full path of a leaf-node value in a hierarchy (e.g. “Tools > Power Tools > Drills > Accessories” represents the Accessories category for power drills in a four-level hierarchy). The delimited values must be expanded into a hierarchy as part of the import.
INNOVATION ►► The values in a source field can be expanded into a hierarchy based on the specified delimiter using the split-into-hierarchy feature of the MDM Import Manager, so that you can precisely map to existing destination hierarchy – and even automatically create new destination hierarchy – as part of the value-mapping process.
DATA INTEGRITY ►► When expanding into a hierarchy, the MDM Import Manager automatically synthesizes the delimited values across the entire set of source records into a single integrated hierarchy.
DATA INTEGRITY ►► You can specify how to handle trailing NULLs in delimited values based on whether you are importing into a hierarchy lookup field or into the hierarchy table itself. However, regardless of the hierarchy NULL-handling setting, internal NULLs are always preserved as nodes because ignoring them would collapse hierarchy structure.
24 MDM Import Manager Reference Guide
Challenge: Split a Single Field into Multiple Fields
Import fantasy: Source data values isolated in distinct fields
Import reality: Delimited source field value represents multiple fields of data
MDM feature: Split into multiple fields
A single field may contain a set of delimited values that actually belong in distinct fields (e.g. “Printers; Laser; 1200” represents a product with Category=Printers; Printer Type=Laser; and DPI=1200 dpi). The delimited values must be split into multiple fields as part of the import.
NOTE ►► Sometimes multiple delimited values do not belong in distinct fields. Rather, they represent multiple values for a single field (e.g. “110V; 115V; 220V; 230V”), in which case no splitting is necessary and the delimited field should instead be mapped to an MDM multi-valued field or attribute.
INNOVATION ►► The values in a source field can be split into multiple fields based on the specified delimiter using the split-into-multiple-fields feature of the MDM Import Manager, so that you can map, convert, and create measurements from the individual values in each split field.
Challenge: Split a Single Field into Multiple Data Values
Import fantasy: Source data values isolated in distinct fields
Import reality: Delimited source field value represents multiple data values
MDM feature: Multi-valued delimiter
Sometimes multiple delimited values do not belong in distinct fields.
Rather, they represent multiple values for a single field (e.g. “110V; 115V; 220V; 230V” represents a product that is available for all four voltages). In this case, no splitting is necessary and the delimited field value must be split into multiple values as part of the import, and the delimited field mapped to an MDM multi-valued field or attribute.
INNOVATION ►► The value in a source field can be split into multiple values based on the specified multi-valued delimiter, so that you can map, convert, and create measurements from each of the individual values.
MDM Import Manager Reference Guide 25
Challenge: Combine Two Fields to Build a Hierarchy
Import fantasy: Direct mapping between individual source and destination fields
Import reality: Two fields represent a single hierarchy of values
MDM feature: Create hierarchy field
Hierarchy in the source data may have been flattened into a pair of fields in the source table, with one field for the node and one for its parent node (e.g. “Printers”; “Printers; Laser”; and “Printers; Dot Matrix” represent the Printers node and its two children). The parent/child value pairs must be assembled to build this hierarchy as part of the import.
INNOVATION ►► MDM allows you to build a hierarchy from two fields whose values identify a set of parent/child hierarchy relationships.
Challenge: Combine Fields to Create Value Combinations
Import fantasy: Direct mapping between individual source and destination fields
Import reality: Mapping between field combinations rather than individual fields
MDM feature: Partitions
It often may not be possible to directly map individual source fields to individual destination fields – or to map, convert, or create measurements from the underlying values within those individual source and destination fields – because there may not be a direct one-to-one relationship between individual source and destination fields.
Instead, the appropriate field mapping relationship may exist between the combination of two or more source fields and/or the combination of two or more destination fields and/or attributes (such as when the CATEGORY and SUBCATEGORY source fields together correspond to the Category destination field, or alternatively, the CATEGORY source field corresponds to the Category destination field together with the Printer Type and DPI attributes). In these cases, fields must be combined to allow the mapping of field combinations rather than individual fields; meanwhile, combining fields into field combinations simultaneously combines the underlying values into value combinations, to allow the value-level mapping, conversion, and measurement creation of value combinations rather than individual values.
NOTE ►► Field and value combinations represent one-to-many, many-to-one, and many-to-many relationships between the fields rather than the rows of two tables.
26 MDM Import Manager Reference Guide
INNOVATION ►► MDM’s generalized partitioning mechanism allows you to combine two or more fields into field combinations (and underlying values into a corresponding set of value combinations), so that you can map, convert, and create measurements from value combinations rather than individual values. Multiple partitions can be nested (creating hierarchy among value combinations) or combined (merging value combinations).
INNOVATION ►► For a field combination created by partitioning, the set of all possible value combinations is collapsed down to the set of those that actually exist in the underlying records.
DATA INTEGRITY ►► Although the set of distinct value combinations will be larger than each individual set of distinct values, each particular value combination will then correspond to a smaller, more specific and more precisely defined subset of underlying records, and thereby permit an accurate value-level mapping that would not otherwise be possible.
Challenge: Combine Fields to Reconstitute Hierarchy
Import fantasy: Direct mapping between individual source and destination fields
Import reality: Multiple fields represent a single hierarchy of values
MDM feature: Nested partitions
Hierarchy in the source data may have been flattened into multiple fields in the source table, such as when a category hierarchy is stored as three distinct source fields, with the first two representing internal category nodes and the third representing the leaf node category. Again, fields must be combined to reconstitute this hierarchy as part of the import.
INNOVATION ►► Nested partitions allow you to create field combinations from multiple fields in the source data to create or reconstitute hierarchy represented by the underlying value combinations, so that you can precisely map to existing destination hierarchy – and even automatically create new destination hierarchy – as part of the value-mapping process.
DATA INTEGRITY ►► You can specify how to handle trailing NULLs in delimited values based on whether you are importing into a hierarchy lookup field or into the hierarchy table itself. However, regardless of the hierarchy NULL-handling setting, internal NULLs are always preserved as nodes because ignoring them would collapse hierarchy structure.
MDM Import Manager Reference Guide 27
Challenge: Combine Fields to Merge Values
Import fantasy: Direct mapping between individual source and destination fields
Import reality: Multiple fields represent a single merged value
MDM feature: Combined partitions
Values from two or more fields in each source record may need to be merged into a single value for import, such as when a first name and last name are stored in two distinct source fields and must be combined into a single name field within the repository, or a numeric value and an associated unit of measure are stored in two distinct source fields but are properly stored in a single measurement field in the repository. Again, fields must be combined to merge values as part of the import.
INNOVATION ►► Combined partitions allow you to create field combinations from multiple fields in the source data to concatenate the underlying value combinations into a single merged value for each record, so that you can map, convert, and create measurements from the merged values.
DATA INTEGRITY ►► You can preserve or collapse empty positions in the single merged value (which correspond to NULLs in the source data) based on whether position matters in the merged value.
Challenge: Combine Fields to Create a Multi-Valued Field
Import fantasy: Direct mapping between individual source and destination fields
Import reality: Multiple fields represent the multiple values of a multi-valued field
MDM feature: Combined partitions
Instead of being merged, values from two or more fields in each source record may need to be concatenated into a set of delimited values that belong in a single multi-valued field, such as when two size values are stored in two distinct source fields and must be stored in a multi-valued measurement field within the repository. Again, fields must be combined to concatenate values as part of the import.
INNOVATION ►► Combined partitions allow you to create field combinations from multiple fields in the source data to concatenate the underlying value combinations into a set of delimited values for each record, so that you can map the field combination and the underlying set of delimited values to a multi-valued field.
DATA INTEGRITY ►► You can preserve or collapse empty positions in the set of delimited values (which correspond to NULLs in the source data) based on whether position matters.
28 MDM Import Manager Reference Guide
Challenge: Import Attribute Data as Name/Value Pairs
Import fantasy: Source attribute data is structured as distinct fields in import table
Import reality: Attribute data is structured as name/value pairs
MDM feature: Partitions; special destination field containing attribute name/values
When attribute data is stacked “vertically” in the source table as name/value pairs and the number of attributes represented by the source data is too large to use the pivot and map mechanism, the attribute data needs to be imported into the attributes directly.
INNOVATION ►► Attribute data that is stacked vertically as name/value pairs can be imported directly by first partitioning the name and value source fields, and then mapping the resulting name/value hierarchy to a special destination field containing a name/value hierarchy of attributes and attribute data values.
Challenge: Match Records
Import fantasy: Source records easily matched against destination records
Import reality: Duplicate matching records and matching conflicts
MDM feature: Matching against multiple key fields and field combinations
Matching source records against existing destination records is extremely difficult. Source records may correspond to a single existing destination record or to multiple existing destination records. The match may be based on the value of a unique key field, multiple key fields (only some of which may match), or key field combinations. Finally, for each source record, the user must be able to indicate that the source record should be: (1) skipped; (2) created; (3) used to update existing destination records; or (4) used to replace existing destination records.
INNOVATION ►► Record matching can be done simultaneously against multiple key fields and/or key field combinations to locate existing records that correspond to the source records. A default import action (skip, add, update, replace) can then be applied to each matching group and then overridden for each source record.
DATA INTEGRITY ►► Using multiple key fields for record matching does an extremely thorough job of locating existing repository records that match each source record, which highlights: (1) potential data duplication among existing records, when there are multiple matching records; and (2) potential errors in the incoming import data, when there are conflicts between the various key fields.
MDM Import Manager Reference Guide 29
KEY ADVANTAGES
Advantages of using the MDM Import Manager include the following:
Infinite flexibility. The MDM Import Manager can handle any tabular
electronic source data, enabling you to arbitrarily restructure and rationalize it against destination data that has an entirely different structure. Moreover, the underlying engine can not only import but also transform data for destinations outside the repository.
No programming. Despite its flexibility, the MDM Import Manager
requires no programming whatsoever, with a WYSIWYG interface that allows you to implicitly define a set of intricate transformation rules that can be saved and then reused on a recurring basis, either through the GUI or programmatically through an API.
End-to-end application. The MDM Import Manager includes
comprehensive functionality to interactively browse, sort, restructure, and import source data that completely eliminates the need to use an external application for any step in the import process.
No pre- or post-processing. Not only is the pre-processing typically
required by other import tools completely eliminated, no post-processing is required after the source data is within the repository, since it is completely transformed as part of the import process.
Consistent view of source data. Regardless of the application file
format of the source data (Access, Excel, delimited text, SQL, XML), the MDM Import Manager offers precisely the same user interface, functionality, and overall user experience for performing the import.
Full relational support. The MDM Import Manager can combine
related information from multiple tables, so that relational source data (even multiple sheets within an Excel file) do not need to be pre-flattened into a single table in an external application prior to import.
Single interface. An “at-a-glance” interface simultaneously provides
both a high-level “birds-eye” view of the schema structure of the source data (including tables, fields, and table relationships) and a low-level detailed view of the data itself within each of the tables.
Field-oriented processing. User processing within the MDM Import Manager is performed field-by-field prior to import rather than record-by-record during import, reducing the magnitude of the processing
effort from the number of records to the number of fields.
Distinct value lists. For each field, the MDM Import Manager collapses the set of all values across the entire set of source records down to the set of distinct values, whose number will usually be dramatically smaller than the total number of records, often by several orders of magnitude.
30 MDM Import Manager Reference Guide
Value conversion and mapping. Data types are automatically
converted and unit of measure strings automatically normalized and appended to numeric source data values based on the data type and physical dimension of the destination field, even when stored in the source data as text. In addition, you can apply value conversion filters both at the field level and at the individual value level, and map source data values to the domain of legal destination values.
100% accuracy. Unlike other import applications that claim up to
96% accuracy for importing source data, the approach used by the MDM Import Manager supports and even guarantees 100% accuracy of the imported data, because the precise transformations must all be defined before you perform the import.
DATA INTEGRITY ►► Each source data field is handled in sequence, with a beginning, a middle, and an end to the process of validating, correcting, and/or converting source data values. All exceptions are clearly visible in the set of distinct values for each field, and each can be handled accordingly. With or without value mapping, data type conversion, or measurement value creation, each distinct source value can be manually adjusted or corrected to completely eliminate source data inconsistencies.
MDM Import Manager Reference Guide 31
About This Guide
This reference guide focuses on providing the most useful information in the fewest pages. However, making the most of the MDM Import Manager requires an understanding of the concepts that make it so powerful; therefore we provide ample conceptual background as well. Our ultimate goal is to help you learn to use the MDM Import Manager as it was designed to be used, for maximum ease of use and power in aggregating your master data.
WHAT YOU WILL FIND IN THIS GUIDE
Concepts required to make best use of the MDM Import Manager
Step-by-step procedures for preparing source data for import
Step-by-step procedures for importing data and tracking status
Operational tips and techniques
WHAT YOU WILL NOT FIND IN THIS GUIDE
How to create an MDM repository, and other DBMS administration topics (described in the MDM Console Reference Guide).
A description of the distinctive features of the MDM user interface (described in the MDM Data Manager Reference Guide)
How to manage and edit records in the MDM repository (also described in the MDM Data Manager Reference Guide).
MDIS documentation (described in the MDM Console Reference Guide).
ASSUMPTIONS
We have made certain assumptions in this reference guide, to better focus on the information you will find most useful.
The data to be imported resides in Access, Excel, SQL Server, Oracle, or other databases supported by MDM that have been set up as ODBC data sources.
You are familiar with the MDM Client software, or at least the MDM Data Manager Reference Guide. In particular, you are familiar with the different types of controls in the MDM user interface.
NOTE ►► Since each MDM repository is unique, we cannot choose examples of table names, field names, attributes, and so on, that would apply to all users. For the sake of example, we call the main table that contains product records “Products” and the taxonomy table that contains product categories “Categories.”
32 MDM Import Manager Reference Guide
SCREEN CAPTURES
All MDM 7.1 applications now follow the standard SAP color scheme, as illustrated in Figure 1. However, screen captures included in this reference guide may continue to show the old color scheme if the functionality illustrated by the screen capture has not changed. In cases where functionality has changed or new features have been added, the screen captures have been updated to the new color scheme.
Figure 1. MDM 7.1 color scheme
33
PART 1: BASIC CONCEPTS
This part of the reference guide explains essential concepts that will enable you to make the best use of the MDM Import Manager. It explains in conceptual terms the processes for which the remainder of this reference guide provide the step-by-step procedures.
MDM Import Manager Reference Guide 35
What is Parametric Import?
The MDM Import Manager is a GUI “cockpit” that defines a very specific sequence of steps to prepare source data for import. But rather than perform the import a record at a time as with other import tools, the MDM Import Manager offers a comprehensive set of data transformation capabilities that allow you to pre-process the source data a field at a time prior to import.
Moreover, for each field, the MDM Import Manager collapses the set of all values across the entire set of source records down to the set of distinct values, whose number will usually be dramatically smaller than
the total number of records, often by several orders of magnitude.
Field-at-a-time rather than record-at-a-time processing and the distinct value lists of the MDM Import Manager are together the lynchpin of its patent-pending Parametric Import™ technology, which supports the import of electronic source data from any tabular or XML format.
Parametric Import is a new and radically more efficient approach to importing and transforming data that is conceptually similar to parametric search, as follows:
Parametric search lists the complete set of distinct values for each field or attribute in a repository. Selecting a value during a drill-down search selects all of the underlying records that contain that value.
Parametric import lists the complete set of distinct values for each field in the source data. Changing a value as part of the import (by mapping or converting it) changes it in all of the underlying records that contain that value.
INNOVATION ►► In conjunction with field-at-a-time rather than record-at-a-time processing, distinct value lists make import exception handling much more efficient because:
(1) the number of distinct values for each field is dramatically smaller than the total number of values;
(2) each distinct value represents and is a proxy for not only all instances of the value across the entire set of underlying records, but also the subset of records that contain the value;
(3) examining and then transforming each source data value must be done only once for each distinct value rather than once for each instance of the value; and as a result,
(4) the maximum number of exceptions is the sum rather than the product of the number of distinct values in each mapped field.
INNOVATION ►► Distinct value lists also support partitioning, another patent-pending innovation of the MDM Import Manager that allows you combine fields to create field and value combinations, for additional transformation flexibility.
36 MDM Import Manager Reference Guide
For example, suppose there are 5000 source records, each with a value for a Country field that represents the country of origin, and moreover, there are twelve distinct values for Country across the 5000 records. If
all of the source values for Country were in upper case and must be converted to mixed case for import, using the set of distinct values would require you to convert each of twelve values once rather than individually convert its value in each of 5000 source records. Similarly, if all of the source values for Country were three-letter abbreviations that must be mapped to the standard country names within the repository, using the set of distinct values would require you to map each of twelve values once rather than individually replace its value in each of 5000 source records. The benefit for the Country field is clear: twelve transformations rather than 5000.
INNOVATION ►► Parametric import replaces the traditional concept of exception handling with a very specific and predictable sequence of steps to prepare source data for a clean and exception-free import.
THE CURRENT TABLES
The MDM Import Manager imports records from the current source table into the current destination table. The current source and destination tables are displayed in the pair of drop-down list controls at the far left of the toolbar (Figure 2).
Figure 2. Current source and destination tables shown at left of toolbar
NOTE ►► The MDM Import Manager imports records from a single table into a single table.
When you open the drop-down list of source tables (Figure 3), the table
list includes the None choice separated by a gray line from all the tables in the source file, in alphabetical order.
Figure 3. Drop-down table list open for source table selection
NOTE ►► For XML source files, the source table list displays the nested elements defined in the XML schema.
MDM Import Manager Reference Guide 37
When you open the drop-down list of destination tables (Figure 4), the
table list includes None, the main table of the MDM repository, the normal subtables in alphabetical order, the object tables, and the parent/child relationships, with each set separated by a gray line.
Figure 4. Drop-down table list open for destination table selection
NOTE ►► When you first start the MDM Import Manager, it sets both
the current source table and the current destination table to None. You must select a current source table and a current destination table before you can perform most of the MDM Import Manager operations.
Non-Main Table Imports
The MDM Import Manager can be used to import not only to the main table but also to the subtables of an MDM repository. However, there are several differences when importing to a subtable with respect to which fields require or permit optional value mapping and which fields must be used for record matching.
NOTE ►► See “When Value Mapping Is Required” on page 218, “Value Mapping and Lookup Table Display Fields” on page 247, and “Record Matching and Lookup Table Display Fields” on page 345.
NOTE ►► You cannot import to the Masks or Named Searches tables.
Relationship Imports
You can also use the MDM Import Manager to import into parent/child product relationships; as above, there are differences with respect to field mapping and record matching when importing the links that define the related records of these relationships (see “Importing Parent/Child Relationship Links” on page 385).
38 MDM Import Manager Reference Guide
TABLE JOINS AND LOOKUPS
The MDM Import Manager eliminates the need to flatten relational source data in an external application prior to import. It automatically detects join relationships between tables defined in the data source, and also lets you manually create additional relationships. You can then look up fields from lookup tables into the primary source table to create a virtual “lookup-extended” source table for import whose virtual records consist of both the fields of the primary source table and the fields looked up from each of the lookup tables.
Table joins and lookups are described in more detail in “Table Joins and Lookups” on page 101.
PIVOTING AND REVERSE PIVOTING
The pivoting and reverse pivoting mechanisms of the MDM Import Manager allow you to “rotate” source data, creating an entirely new table from the source table, to completely transform the data in ways that the other restructuring mechanisms cannot address.
In particular, pivoting is often required when field and attribute data is stacked “vertically” in the source table and needs to be rearranged into a “horizontal” table structure from which it can be imported into the fields and attributes of a regular table. Conversely, reverse pivoting is often required when qualified data is arranged “horizontally” in the source table and needs to be rearranged into a “vertical” table structure from which it can be imported into the fields and qualifiers of a qualified table.
Pivoting is described in more detail in “Pivoting” on page 110; reverse pivoting is described in more detail in “Reverse Pivoting on page 117.
PARTITIONING
The MDM Import Manager extends the MDM Client partitioning mechanism to combine fields into field combinations and thereby create value combinations, allowing you to deal with the value combinations as a proxy for the corresponding subsets of underlying records.
Partitioning can be used to achieve several very different objectives: (1) establish a one-to-one relationship between source and destination fields and field combinations that might not otherwise exist between individual source and destination fields; (2) create or reconstitute hierarchy represented by the source data; and (3) concatenate values stored in two or more distinct source fields into a single merged value for each record.
Partitioning is described in more detail in “Partitioning and Field Mapping” on page 173.
MDM Import Manager Reference Guide 39
VALUE MAPPING
The value mapping mechanism provided by the MDM Import Manager offers an entirely new approach to handling exceptions in source data, which can be pre-processed a field-at-a-time prior to import rather than processed a record-at-a-time during import. All exception handling can be handled up-front and the actual import can then proceed without interruption regardless of the size of the record set.
Value mapping also provides a powerful way to map from source to destination hierarchy, and to automatically create destination hierarchy (internal and leaf nodes) as part of the import.
Value mapping is described in more detail in “Value Mapping” on page 212.
VALUE CONVERSION
Even source data values that do not need to be mapped may need to be converted, either because the values are inconsistent with repository values, because their data types do not correspond, or because numeric values are missing a physical unit of measure. Such data conversion is handled automatically by the MDM Import Manager, and can be overridden on a value-by-value basis.
Value conversion is described in more detail in “Value Conversion” on page 221.
RECORD MATCHING
The preceding operations transform records for import, which can then be matched against existing records to determine whether a new record should be created, or an existing record should be updated or replaced.
Record matching can be done against a single key field, multiple key fields, key field combinations, and even against non-key fields to detect differences between source and destination field values.
Record matching is described in more detail in “Record Matching Concepts” on page 338.
IMPORT MAPS
All of the partitions, mappings, and other conversions can be saved in an import map within the MDM repository. A map is useful if you want to save your work before you have performed the import, if you want to subsequently apply the same transformations for future imports of additional records or updates from the same data source, or if you want to later use the map in conjunction with MDIS processing.
Import maps are described in more detail in “Import Map” on page 69.
40 MDM Import Manager Reference Guide
RECORD FILTERING
Record filtering expands the concept of import by complementing the source-centric set of import records with a destination-centric set of records that includes not only matched records but also unmatched records in the destination set.
Record filtering is described in more detail in “Record Filtering” on page 356.
41
PART 2: GUIDED TOUR
This part of the reference guide explains how to start and stop the MDM Import Manager. It assumes that you are familiar with the MDM Client software, or at least have familiarized yourself with the different types of controls in the MDM user interface using the MDM Data Manager Reference Guide.
MDM Import Manager Reference Guide 43
Starting Import Manager
When you start MDM Import Manager, a connection wizard connects Import Manager to the MDM repository you want to import to, and then connects Import Manager to the source data you want to import.
A repository must be running in order for Import Manager to connect to it.
In order to connect to the source data, the source file or database must be available through the Windows file system, an MDM repository port, or the DBMS.
CONNECTING IMPORT MANAGER TO AN MDM REPOSITORY
The first step in starting Import Manager is connecting to a repository.
To connect Import Manager to a repository:
1. Double-click the MDM Import Manager icon.
2. In the Connect to MDM Repository screen, select the desired MDM repository, the language layer to work in, and enter your user name and password.
3. Click Next to connect to the source data you want to import (see
“Connecting Import Manager to Source Data” on page 45).
NOTE ►► If an MDM repository does not appear in the list, you must
add it by clicking “…” (browse) to set up the connection (see the following sections for more information).
NOTE ►► If a repository’s TCP/IP port number changes, you must re-add the repository as the old entry will load whatever repository is loaded on the old TCP/IP port.
MULTILINGUAL ►► More information about language layers and multilingual repositories is provided in “Part 14: Multilingual Support.”
Setting Up Unencrypted Repository Connections
To set up an unencrypted connection to an MDM repository:
1. In the Connect to MDM Repository screen, click “…” (browse).
2. In the Choose Repository dialog, select the Master Data Server on which the repository is running.
If the Master Data Server has not been previously connected to by
Import Manager, type the name or IP address of its host, or click “…” (browse) to select it from a list.
44 MDM Import Manager Reference Guide
NOTE ►► If the Master Data Server is configured to listen on non-"MDM default" ports, you must type in the port number after the Master
Data Server name, using the format ServerName:PortNumber (for
example, ServerXYZ:54321). Otherwise, Import Manager will be
unable to connect to the Master Data Server.
3. Select the repository (you may have to click the “refresh” button next to the Repository field to load the repository names).
4. Click OK to add the repository to the Connect to MDM Repository list.
Setting Up Secure Repository Connections
Secure connections are only possible for repositories on SSL-enabled Master Data Servers (see “Network and Communication Security” in the MDM 7.1 Security Guide for more information).
To set up a secure connection to an MDM repository:
1. In the Connect to MDM Repository screen, click “…” (browse).
2. In the Choose Repository dialog:
a. Select the SSL-enabled Master Data Server
b. Click Secure Connection
c. Enter the paths to the client Key File and SSL Library
3. Click the “refresh” button next to the Repository field and choose the repository you want to connect securely to.
4. Click OK to add the repository to the Connect to MDM Repository list.
5. In the Connect to Repository dialog, a lock icon similar to the one shown below indicates that communications with the selected repository will be established on a secure connection.
Figure 5. The lock icon on a Connect to MDM Repository dialog
MDM Import Manager Reference Guide 45
CONNECTING IMPORT MANAGER TO SOURCE DATA
Once Import Manager connects to a repository, the next step is to connect it to the source data you want to import.
To connect Import Manager to source data:
1. In the Connect to Source screen, choose the source Type and the Remote System the data originated from.
2. Select the source properties that correspond to the selected Type (see below for more information about each property).
3. Click OK to open Import Manager.
Source Properties
Each source Type requires a different set of source properties, as indicated by the bullets in Table 1.
Table 1. Source Properties
Property Ty
pe
Acc
ess
Delim
ited T
ext
Exce
l
Fix
ed T
ext
OD
BC
Ora
cle
Port
SQ
L S
erv
er
XM
L
XM
L S
chem
a
Description
Remote System ● ● ● ● ● ● ● ● ● Name of the remote system
Port ● Name of the port
XML Schema ● Name of the XML schema
File Name ● ● ● ● ● ● Name of the source data file
DBMS Server ● ● Name of the DBMS server
Database ● ● Name of the source database
User ● ● Username of the connection
Password ● ● Password of the connection
Columns ● Columns of the fixed text
Delimiter ● Delimiter of the delimited text
Read-Only ● ● Open source as read-only?
Root ● Root element of XML schema
NOTE ►► A maximum field name length of 64 characters is imposed on Access and Excel import files; all other types have no length limits.
NOTE ►► For Type=Delimited Text, use \t to specify a tab delimiter.
NOTE ►► For Type=Fixed Text, the Columns values are entered as a semi-colon delimited list (e.g. 4;7;9:) with a colon used to separate column header widths and column data widths (e.g. 25;50;10:4;7;9)
46 MDM Import Manager Reference Guide
NOTE ►► For Type=Oracle, a 32-bit Oracle client is required (64-bit clients are not supported).
NOTE ►► For Type=Port, selecting a Remote System limits the set of ports but is not strictly required, since port names should be unique.
PORTS ►► The Port list includes an entry for each inbound port defined in the MDM Console for the selected Remote System; those that do not have files waiting or are locked by another user are disabled in the drop-down list. When you select a port, the MDM Import Manager automatically selects the first source data file that is waiting
to be processed; you can then use the Get Next Source File command to select each succeeding file.
XML SCHEMAS ►► The XML Schema list includes an entry for each XML schema defined in the MDM Console.
TIP ►► For Type=XML and Type=XML Schema, you can select multiple XML source files, each having the same structure.
TIP ►► To reference a SQL Server named instance in the DBMS Server property, append a backslash (\) and the instance name after the machine name.
GETTING THE NEXT SOURCE DATA FILE
For source Type=Port, MDM Import Manager automatically selects the
first source data file waiting to be processed. You can use the Get Next Source File command to get the next source data file waiting at the current port without closing your MDM Import Manager session.
To get the next source file from the selected port:
1. From within the MDM Import Manager, choose File > Get Next Source
File from the main menu.
2. The MDM Import Manager selects the next source data file waiting at the current port.
3. The existing MDM Import Manager session starts fresh with new source data as if you exited and restarted the MDM Import Manager.
CONNECTING TO A NEW DATA SOURCE
You can use the Connect to Source command to connect to an entirely new data source without closing your MDM Import Manager session, as described in this section.
MDM Import Manager Reference Guide 47
To connect to a new data source without closing your existing MDM Import Manager session:
1. From within the MDM Import Manager, choose File > Connect to
Source from the main menu.
2. The MDM Import Manager opens the Connect to Source dialog.
3. Choose a new Type and Remote System and other source properties.
4. The existing MDM Import Manager session starts fresh with new source data as if you exited and restarted the MDM Import Manager.
DATA STAGING
Prior to MDM 7.1, Import Manager used MS Access to stage data from MS Access, MS Excel, text, and XML source files. As part of the staging process, Import Manager converted source data from its original structure into MS Access tables. These MS Access tables, and not the original source files, were then displayed in the Import Manager.
For XML and XML Schema source files, the conversion flattened the nested hierarchical data structures contained in the original XML file into a set of Access tables. These tables required joins in order to reconstruct the data relationships in the original file, and the end result was that the “source” tables displayed in Import Manager bore little resemblance to the nested structure of the original XML source file.
In MDM 7.1, Import Manager no longer uses MS Access to stage data from text and XML source files. As a result, the previous limitations on field name lengths (64 characters) is removed for text and XML files. In addition, Import Manager now retains and displays the nested structure of source XML files, which should result in a more intuitive Import Manager experience (see “XML Source Files and the Source Hierarchy Tree” on page 84 for more information).
NOTE ►► Import Manager still uses MS Access to stage data from MS Access and MS Excel source files.
EXCEL-RELATED IMPORT CHALLENGES
Because an Excel spreadsheet does not have an associated schema that defines the data type of each column, the Jet driver used to connect
48 MDM Import Manager Reference Guide
to Excel has several limitations that must be addressed by the MDM Import Manager:
Empty columns. The Jet driver may incorrectly conclude that a
spreadsheet column is empty.
Data types. If the column is not empty, Jet may incorrectly determine
the data type of the values in the column.
Mixed data types. If the column contains both numeric and text
values, Jet will return NULL for values of the minority data type.
The mixed data type problem is overcome internally by the MDM Import Manager in the way it connects to the Jet driver, and does not require any changes to your system.
However, MDM Import Manager can overcome the other problems only by changing the value of a Windows registry setting, which it offers to do for you automatically the first time you connect to an Excel data source.
NOTE ►► If you do not allow the MDM Import Manager to change the registry setting, some Excel source data values may be ignored or truncated, or their data type improperly converted.
MDM Import Manager Reference Guide 49
MDM Import Manager Main Window
The MDM Import Manager main window includes three primary panes, with several tabs in the bottom pane. Panes may be resized by dragging the splitters that separate them (Figure 6).
Figure 6. MDM Import Manager main window (default layout)
TIP ►► To cycle among the panes in the main window, press F6.
TIP ►► To resize panes, place the mouse pointer over a splitter until the pointer changes to a vertical or horizontal pipe with arrows, then click and hold down the mouse button while you drag the splitter in the direction of the arrows.
SWITCHING LAYOUT
You can rearrange the panes in the main window by using the Switch
Layout command..
To switch the layout of the main window:
Choose View > Switch Layout from the main menu.
50 MDM Import Manager Reference Guide
SELECTING THE CURRENT TABLES
When you first start the MDM Import Manager, it sets both the current
source table and the current destination table to None. You must select a current source table and a current destination table before you can perform most of the MDM Import Manager operations.
To specify the current source table:
Click on the drop-down table list (Figure 7), and select the source table whose records you want to import.
Alternatively, choose View > Source Table from the main menu and choose from the cascading menu of tables.
Figure 7. Drop-down table list open for source table selection
To specify the current destination table:
Click on the drop-down table list (Figure 8), and select the destination MDM repository table into which you want to import.
Alternatively, choose View > Destination Table from the main menu and choose from the cascading menu of tables.
Figure 8. Drop-down table list open for destination table selection
51
PART 3: IMPORT WORKFLOW
This part of the reference guide explains how the import workflow and the panes and tabs of the MDM Import Manager main window.
MDM Import Manager Reference Guide 53
Working with the MDM Import Manager
The MDM Import Manager allows you to import data into an MDM repository from any tabular or XML format, with the same user interface and identical functional capabilities regardless of the source file format.
Importing source data is akin to stir frying – almost all of the work is in the preparation. In the case of import, preparation of a source data file includes some or all of the following steps:
1. Connect to the source data file or database and to the destination MDM repository, to establish programmatic access from the Import Manager to the source and destination schemas and data.
2. Browse source tables, fields, and records, examine the set of distinct data values for each field, to become familiar with the source data and its structure.
3. Join tables and then look up fields from secondary tables into the primary source table, to combine information from multiple tables and create virtual extended source records for import.
4. Split fields, to break delimited field values into a hierarchy or into multiple fields, for subsequent field-level processing on individual field values in each new field.
5. Partition fields, to restructure source data by combining fields into field combinations, for subsequent field-level processing on value combinations rather than individual field values.
6. Map fields, the most traditional step in the import process, to identify the destination field for each source field, and also the type of mapping and conversion required for the underlying data values.
7. Map and convert values, to transform and normalize source data by eliminating differences between source and destination values, converting data types, and creating measurement values.
8. Match records, to specify key fields for identifying existing records that match the source records, and whether new records should be created and existing records should be updated or replaced.
Each step is a part of the process of implicitly defining a precise set of transformation rules prior to import that allows you to completely restructure, cleanse, normalize, and rationalize raw product data into rich master data as part of the import process itself.
54 MDM Import Manager Reference Guide
MDM IMPORT MANAGER AT A GLANCE
Figure 9. MDM Import Manager main window
The main window of the MDM Import Manager consists of the panes and tabs shown in the numbered callouts of Figure 9, listed below, and described in the following sections:
1. Source Hierarchy pane
2. Destination Hierarchy pane
3. Source Preview tab
4. Partition Field/Value tab
5. Map Fields/Values tab
6. Match Records tab
7. Import Status tab
8. Status bar
The tabs in the bottom pane of the main window are arranged in “workflow” order, from left to right. Various functions are performed in the Source Hierarchy pane, but most of your work in preparing source data for import is done in these tabs. Once you have completed partitioning, field mapping, value conversion and mapping, and record matching in these tabs, all that’s left to do to import the data is…press
the Execute Import toolbar button.
1 2
3 4 5
8
6 7
MDM Import Manager Reference Guide 55
Source Hierarchy Pane
The Source Hierarchy pane (top-left pane) contains a tree representing the hierarchy of tables, fields, and data values of the source file whose records you intend to import (Figure 10). Use the tree in the Source Hierarchy pane to view the source table and field structure, to view distinct data values for each field, to define and view join and lookup relationships between tables, and to perform various source operations.
Figure 10. Source Hierarchy pane
Destination Hierarchy Pane
The Destination Hierarchy pane (top-right pane) contains a tree representing the hierarchy of tables, fields, attributes, qualifiers, and data values of the destination MDM repository into which you intend to import the source records (Figure 11). Use the tree in the Destination Hierarchy pane to view the destination repository table and field structure and to view distinct data values for each field, attribute, and qualifier.
Figure 11. Destination Hierarchy pane
56 MDM Import Manager Reference Guide
Source Preview Tab
The Source Preview tab (tab in bottom pane) contains two subpanes (Figure 12). The left Tables subpane lists the source tables; the right Records subpane contains a list view of the selected source table’s records in a row/column grid, with a row for each record and a column for each of the fields. Use the Source Preview tab to interactively browse the records in each of the source tables and sort by any of the columns in ascending or descending order.
Figure 12. Source Preview tab
MDM Import Manager Reference Guide 57
Partition Field/Value Tab
The Partition Field/Value tab (tab in bottom pane) lists, on the left, the source fields that are available to partition the selected source field or value in the Source Hierarchy tree, and on the right, the destination fields and attributes that are available to partition the selected destination field or attribute in the destination hierarchy (Figure 13). Use the Partition Field/Value tab to create the partitions that combine fields into field combinations, for subsequent processing of value combinations rather than individual field values.
Figure 13. Partition Field/Value tab
NOTE ►► You cannot partition a source field or value until you have selected the current source table. You cannot partition a destination field or value until you have selected the current destination table.
Map Fields/Values Tab
The Map Fields/Values tab (tab in bottom pane) contains two pairs of grids (Figure 14). The top pair of Field Mapping grids contain the source fields on the left and the destination fields, attributes, and qualifiers on the right; the bottom pair of Value Conversion and Mapping grids contain the set of distinct data values corresponding to the current source field on the left and corresponding to the current destination field on the right. Use the Field Mapping grids to map source fields to destination fields, attributes, and qualifiers. Then, for each pair of mapped source and destination fields, use the Value Conversion and Mapping grids to: (1) map values (when mapping to lookup fields and text attributes); (2) create measurement values (when mapping to measurement fields or numeric attributes); and (3) manually adjust, correct, and reformat individual data values.
58 MDM Import Manager Reference Guide
Figure 14. Map Fields/Values tab
NOTE ►► You cannot map fields until you have selected the current source and destination tables. You cannot map and convert values until you have mapped the corresponding field.
Match Records Tab
The Match Records tab (tab in bottom pane) contains several controls (Figure 15). Use the dual-list Value Matching control to specify the destination fields to use as matching fields. Use the Default Import Actions grid to specify the default import action for each group of source records. And use the Source Records grid and Matching Destination Records grid to override the default import action for each individual source record, and for each source record, each of its matching destination records.
MDM Import Manager Reference Guide 59
Figure 15. Match Records tab
NOTE ►► You cannot perform record matching until you have selected the current source and destination tables and mapped the fields that you intend to use as matching fields.
Import Status Tab
The Import Status tab (tab in bottom pane) summarizes the status of the current import (Figure 16). Use the Import Status tab to view the status of: (1) table mapping; (2) table joins; (3) field lookups; (4) field mapping; (5) field mapping; (6) field and value mapping; (7) matching fields; (8) record matching; and (9) import actions. The Import Status tab also identifies the next required action items in the import workflow.
60 MDM Import Manager Reference Guide
Figure 16. Import Status tab
Status Bar
The Status bar displays different information for each tab, as summarized in Table 2.
Table 2. Status Bar Information
Tab Position Status Bar Items
Source Preview
1 “p records”
2 “q fields”
3 “r tables”
Partition Field/Value n/a none
Map Fields/Values
1*
“r selected”
“r leaf nodes selected”
“s overmapped”
2*
“m values”
“m leaf node values”
“m of n values converted”
“m of n values mapped”
“m of n leaf node values converted”
“m of n leaf node values mapped”
“m of n values mapped (p added)”
MDM Import Manager Reference Guide 61
Tab Position Status Bar Items
“m of n leaf node values mapped (p added)”
3 “x of y fields mapped”
Match Records 1
“r source selected”
“r destination selected”
2 “m of n records active”
Import Status n/a none
* Only leaf nodes considered when the mapped destination field is a hierarchy lookup field.
NOTE ►► The MDM Import Manager displays certain information listed above only if the value is: (1) greater than one (e.g. “r selected”); or (2) nonzero (e.g. “s overmapped”). Moreover, it sometimes uses the same embossed position on the status bar to display different information, depending upon the circumstances (e.g. “m values”, “m of n values converted” and “m of n values mapped”).
62 MDM Import Manager Reference Guide
Configuration Options
Configuration options affect how the MDM Import Manager interprets and processes source data.
You can use the Options command to open the Configuration Options dialog and change the option settings for the MDM Import Manager. The dialog contains a grid that enables you to specify values for each MDM Import Manager option.
Options for the MDM Import Manager are listed in Table 3.
Table 3. Configuration Options
Option Description
Default flat NULL handling
The default setting for how implicit and explicit NULLs are handled in source data when you map a field that contains multiple delimited values:
Preserve NULLs Ignore NULLs
Default hierarchy NULL handling
The default setting for how implicit and explicit NULLs are handled in source data when you map a field that contains hierarchy values (see “The Hierarchy NULL Handling Option” on page 233):
Preserve NULLs
Ignore Non-Leaf Trailing NULLs > Preserve Leaf NULLs as NULLs
> Preserve Leaf NULLs as Internal Leaf Nodes Ignore All Trailing NULLs
Ignore Non-Leaf Trailing NULLs has been split into two subsettings for flexible handling of the leaf trailing NULLs.
Default case sensitivity
The default setting for how case differences are handled when determining the set of distinct values for a source field:
Case insensitive
Case Sensitive
For SQL Server or Oracle data sources, case sensitivity of the source field is determined by database collation and cannot be changed by the MDM Import Manager.
Default value delimiters The value delimiters are the strings used by various operations both to identify delimited source data values and to separate merged data values.
Split Hierarchy [SH] The default delimiter string used by the Split into Hierarchy command to identify individual delimited values within a single field (“\”).
Split Multiple Fields [SM] The default delimiter string used by the Split into Multiple Fields command to identify individual delimited values within a single field (“;”).
Partition Combine [PC] The default delimiter string used by the Combine Partitions command to separate each pair of individual merged values (“;”).
MDM Import Manager Reference Guide 63
Option Description
Multi-Valued [MV]
The default delimiter string used to identify individual delimited values within a single field when mapping to a multi-valued field (“;”).
Accepts “\t” for Tab and “\n” for Return.
Coupled Numeric [CN] The default delimiter string used to identify coupled numeric value pairs when mapping to a coupled numeric attribute (“@”).
Default NULL interpretation
The default setting for whether NULLs in source data are ignored or used to overwrite existing values:
Ignore NULLs
Overwrite existing values with NULLs
Default multi-valued update
The default setting for whether incoming source data values either are appended to or completely replace the set of existing destination data values of a multi-valued field when updating existing records:
Append Replace
Default qualified update
The default setting for whether incoming source subrecords either are appended to, completely replace, or update the set of existing qualified links of a qualified lookup field when updating existing records:
Append Replace
Update
See also the Default Matching Qualifiers option below.
Note that the Append and Replace options use matching qualifiers for qualified link aggregation only, while the
Update option uses them both for qualified link aggregation and for matching.
Default matching qualifiers
The default setting for which qualifiers to use as matching qualifiers when matching existing qualified links when updating existing records:
None All
You can set specific subsets of matching qualifiers only at the field level, since the set of qualifiers is different for each qualified lookup field.
Default tuple update Tuple updating is performed based on these settings.
Matching tuple fields
The default setting for which tuple member fields to use as matching fields for the tuple’s records:
None All
Because each tuple field can have a different set of member fields, you can set specific subsets of matching tuple member fields only at the field level.
New records option
The default setting for handling incoming tuple records which do not already exist for the matching record:
Skip Create
64 MDM Import Manager Reference Guide
Option Description
Existing records option
The default setting for handling incoming tuple records which already exist for the matching record:
Skip
Update (NULL Fields Only) Update (All Mapped Fields) Replace
Delete
Default data group display
The default setting for whether the data groups hierarchy is hidden or displayed in the Destination Values grid for object lookup destination fields:
Hide data group Display data group
Find matching field
The Automap and Find next matching destination field
commands match source and destination field names based on these settings (see “The Find Matching Field Options” on page 275).
Operator
The type of comparison to perform of the source field name against the destination field name:
Contains
Exact
Type
Whether the comparison should be against normalized values or alternatively, whether or not to match case:
Normalized
Case insensitive Case sensitive
Automap fields when loading map
Whether to automatically automap fields when loading an import map into Import Manager. Does not apply to
MDIS. (No/Yes).
Value automap Value automap is performed based on these settings.
Hierarchy
How to match hierarchy values when using Automap in the Value Conversion and Mapping grids:
Full path Leaf nodes only
The Full path setting ignores the root node on the source and/or the destination hierarchies.
Normalized Whether to normalize source and destination values
when attempting to automap (No/Yes).
Automatically map NULL value
Whether to automatically map the NULL source value to
the NULL destination value (No/Yes).
Automatically add missing units
How to handle missing units when importing a taxonomy classification (see “Taxonomy Import" on page 161):
None
When all missing When any missing
Record Matching Record matching is performed based on these settings.
Perform record matching Whether to perform record matching when you select
each matching field (No/Yes).
MDM Import Manager Reference Guide 65
Option Description
Normalized record matching field
How to match against the value of a normalized record matching field (see “Normalized Record Matching Field” on page 360):
Normalized match against normalized value
Exact match against non-normalized value
Import The actual import is performed based on these settings.
Writeable once field update
How to handle import into Writeable Once fields (see “Writeable Once Destination Fields” on page 266):
Ignore Conflicting Value
Reject Record on Conflicting Value
Merge source records using key field
Whether to collapse and merge duplicate source records for import as a single virtual source record (see “The Merge Source Records Option” on page 351) (No/Yes).
Merge qualified links using matching qualifiers
Whether to collapse and merge duplicate subrecords as a single qualified link (see “The Merge Qualified Links Option” on page 352) (No/Yes).
Synchronize Family Hierarchy after import
Whether to synchronize the entire Family Hierarchy
after import (No/Yes).
Do not update record matching field value
Whether to update the value of a record matching field (see “Updating the Record Matching Field” on page
356) (No/Yes).
Auto-Join checked out records
Whether to auto-join checked out records during import
(see “The Auto-Join Checked Out Records Option” on
page 378) (No/Yes).
Checkout/Workflow
Whether or not to check out the imported records or
launch a workflow job containing the imported records
(see “The Checkout/Workflow Option” on page 379):
None
Checkout workflowname : workflowname
Reassign existing key mappings
Whether or not – and how – to reassign existing key mappings during the import (see “Key Mapping Reassignment” on page 410):
No Into original records only
Into original and checked out records
Eliminate duplicate qualified links
Whether or not to delete duplicate qualified links:
Do not eliminate duplicates Eliminate duplicates (cached qualifiers only)
Autolink attributes The MDM Import Manager can automatically link unlinked attributes to a category as part of the import based on these settings.
When mapping to an unlinked attribute
Whether to automatically link unlinked attributes to the category value of a source record when source record fields containing non-NULL values are mapped to those
attributes (No/Yes).
66 MDM Import Manager Reference Guide
Option Description
When reassigning an existing record to
another category
Whether to automatically link unlinked attributes to a category when the category value is changed for an existing record and data values would otherwise be
discarded (No/Yes).
Default MDIS handling
The MDM Import Manager can automatically map unmapped values (see “The MDIS Unmapped Value Handling Option” on page 268) and set the Default Import Action for MDIS based on these settings.
Automap unmapped values
Whether to first attempt to automap to existing destination values source data values that do not exist
in the map file (No/Yes)
Unmapped value handling
How to handle still unmapped source data values (either because they do not exist or because automapping was turned off):
Ignore Add
Fail
You can set the Map option only at the field level, since the set of destination values is different for each field.
Numeric format The numeric format options allow you to specify the format of incoming numeric data values.
Decimal symbol The decimal symbol for source data values (“.”).
1000s separator The thousands separator for source data values (“,”).
Dates The date options allow you to specify the format of incoming date values.
Date format
The format of incoming dates:
YYYYMMDD MM/DD/YYYY
DD.MM.YYYY
NULL value string The string to interpret as a NULL date value (000000).
Relationships The Relationships options allow you to specify how the MDM Import Manager handles the import of relationship links.
Aggregate by
How to group the source records when replacing existing relationship links:
Parent
Child
Links update
How to handle the set of existing relationship links:
Append
Replace Update
New links (update only)
How to handle new links when updating existing links:
Skip Create
MDM Import Manager Reference Guide 67
Option Description
Existing Links (update only)
How to handle existing links when updating existing links:
Skip
Update (Mapped Qualifiers) Update (All Mapped Fields) Replace
Delete
Expand Multilingual Text Fields
Whether or not – and how – to expand multilingual text fields into multiple fields in the Destination Fields grid of the Map Fields/Values tab (see “Field Mapping to Multilingual Destination Fields” on page 397):
No As Multiple Fields
As Field and Language
Attribute display
How to display attributes and attribute values in the Destination Fields and Destination Values grids of the Map Fields/Values tab (see “Attribute Data Import as Name/Value Pairs” on page 167):
As multiple attributes
As attribute/value hierarchy
Attribute/value hierarchy automap operator
The type of comparison to perform of the source field value against the destination field value when automapping to the Attribute/Value hierarchy:
Contains Exact
Display priority field for Attribute taxonomy links table
using
Whether to use predefined priority values (Highest, High, Normal, Low, Lowest) or allow import of custom integer values:
Predefined values Custom values
Disable automatic hierarchy value mapping according to
remote key
Whether or not to disable the automatic mapping of values by remote key for source fields mapped to keymappable hierarchy lookup fields :
Yes No
68 MDM Import Manager Reference Guide
To open the Configuration Options dialog that allows you to view and edit options settings:
1. Choose Configuration > Options from the main menu.
2. MDM opens the Configuration Options dialog shown in Figure 17.
Figure 17. Configuration Options dialog
TIP ►► Click the “+” icon to expand any node in the tree.
3. Click in the Value column for the option setting you want to change.
4. If the Value cell is a drop-down list, select the desired option. If the Value cell is an edit field, double-click inside the field and replace the existing value with a new value.
5. Click OK to save any new values and close the dialog.
MULTILINGUAL ►► More information about languages and multilingual repositories is provided in “Part 14: Multilingual Support.”
REMOTE SYSTEMS AND MDM ►► More information about remote systems and MDM is provided in “Part 15: Remote Systems and MDM.”
MDM Import Manager Reference Guide 69
Import Maps
All of the partitions, mappings, and other conversions can be saved in an import map within the MDM repository. A map is useful if you want to save your work before you have performed the import, if you want to subsequently apply the same transformations for future imports of additional records or updates from the same data source, or if you want to later use the map in conjunction with MDIS processing.
Specifically, saving a map takes a snapshot of the state of the current Import Manager session so that it can later be restored when you load the map during a future session, as summarized in Table 4.
Table 4. Import Map Components
Setting Description
Configuration Options
All of the global default configuration option settings are saved in the map, along with overridden field-level settings. When you load the map, all the settings are restored.
Joins and Lookups
All of the joins and lookup fields are saved in the map, including whether the
lookup fields were looked up using [All]. When you load the map, saved joins and lookups that do not exist in the source file are ignored.
Partitions All of the field and value level partitions are saved in the map. When you load the map, saved partitions of fields or values that do not exist in the source file are ignored.
Field Mappings
For each field, all of the field mappings are saved as source/destination field name pairs. When you load the map, saved mappings for which either field name does not exist in the source file are ignored.
Value Conversions
For each field, value conversions are saved as follows: (1) at the field level, the filters are saved; and (2) at the value level, all of the manually overridden value conversions are saved as source/destination value pairs. When you load the map, saved values that do not exist in the source file are ignored.
Value Mappings
For each field, all of the value mappings are saved as source/destination value pairs. When you load the map, saved mappings for which either value does not exist in the source file are ignored.
Record Matching
The record matching fields and the matching options are saved in the map.
Import Actions
The default import action for each class and type of match are saved in the map.
CAUTION ►► When you open an existing map against a new source file and resave it, the MDM Import Manager discards old map data that does not exist in the new source file and replaces it with current data. This means, for example, you will lose the mapping and/or cloning of a field that does not exist, and/or the mapping of values for fields and/or values that do not exist. To open and resave a map, be sure to open it against the original source file against which the map was created.
70 MDM Import Manager Reference Guide
Import Map Operations
The following sections describe the various import map operations, and are summarized in Table 5.
Table 5. Import Map Operations
Operation Description
Open Map Opens the selected map.
Close Map Closes the current map.
Save Map Saves the current map.
Save Map As Saves the current map to a new name.
Save Map Update Saves an update of the current map.
Import Map Imports a map file into the repository.
Export Map Exports a map file from the repository.
OPENING AND CLOSING MAPS
When you open a map, MDM performs a series of tests to ensure that the import map is compatible with the source file and destination repository currently connected to the Import Manager.
These tests and their error conditions are described in Table 6.
Table 6. Tests Performed When Opening an Import Map
Test Error Condition Outcome
Source tables in map are present in connected source.
Source table name in map is not found in connected source.
Import Manager cannot open map.
Destination tables in map are
present in connected MDM repository.
Destination table in map is not found in connected repository.
Import Manager cannot open map.
Mapped source fields are present in connected source.
Source field name in map is not found in connected source.
Import Manager opens map without the missing field.
Mapped destination fields are present in connected MDM repository.
Destination field in map is not found in connected repository
Import Manager opens map without the missing field.
MDM Import Manager Reference Guide 71
When you open an import map, the MDM Import Manager loads the configuration option settings that were stored in the map, overriding the settings currently in effect. When you later close the map or exit the MDM Import Manager, MDM asks whether or not you want to save the map’s settings as the new default settings, as shown in Figure 18.
Figure 18. Configuration options save confirmation dialog
Only one import map can be open at a time in the Import Manager. If you try to open a new map without closing the current one, the Import Manager prompts you to save changes to the current map before it opens the new map. If there are no changes to save, the Import Manager automatically closes the old map and opens the new one.
To open an import map:
1. Choose File > Open from the main menu to open the Open Map dialog, shown in Figure 19 below.
Figure 19. Open dialog
2. Select a map from the Available maps list and click OK.
3. When opening the map, Import Manager verifies that the map is compatible with the current repository and alerts you if it finds any problems.
To close the currently open map:
Choose File > Close from the main menu.
72 MDM Import Manager Reference Guide
SAVING MAPS
You can save import maps as described in this section.
To save the current MDM Import Manager state to the current map:
1. Choose File > Save from the main menu.
2. MDM saves the current MDM Import Manager state to the current map.
To save the current MDM Import Manager state to a new map:
1. Choose File > Save As from the main menu.
2. MDM opens the Save Map As dialog shown in Figure 20.
Figure 20. Save As dialog
3. Select or type a name for the new map and click OK to close the dialog
4. MDM saves the current state in a map within the MDM repository.
TIP ►► The File > Save As command saves to a new map. You can
also use the File > Save command to save to the current map if one has already been opened or saved.
To save updated header info to a map:
1. Open the map.
2. Choose File > Save Update from the main menu.
3. MDM saves updated header info to the current map.
MDM Import Manager Reference Guide 73
IMPORTING AND EXPORTING MAP FILES
Sometimes you may want to transfer maps among MDM repositories. You can export a map from a repository by saving it as a map file on your file system. You can then import the map file into the target repository, as described in this section.
To export a map from the repository into a map file:
1. Choose File > Export from the main menu.
2. MDM opens the Select Map dialog. shown in Figure 21.
Figure 21. Select Map dialog
3. Select a map from the list of maps and click OK to close the dialog.
4. MDM opens the Windows file Open dialog for you to name the map file. Navigate to the appropriate folder, select or type a file name, and
click Save.
5. MDM exports the map into a map file.
To import a map file into a map within the repository:
1. Choose File > Import from the main menu.
2. MDM opens the Windows file Open dialog.
3. Browse to the map file and click Open.
4. MDM opens the Save As dialog shown in Figure 20 above.
5. Select or type a name for the new map and click OK to close the dialog
6. MDM saves the map file into a map within the MDM repository.
NOTE ►► After importing a map file from another repository, it is a good idea to open the map in Import Manager and verify that (a) the map is compatible with the new repository; and (b) the field mappings are correct (see “Opening and Closing Maps” on page 70 for more information).
74 MDM Import Manager Reference Guide
UPDATING MAPS FOR MDM 7.1
All import maps created in MDM 5.5 must be updated by MDM before they can be used with the MDM 7.1 versions of Import Manager and MDIS. Until MDM updates them, you cannot open outdated maps in the Import Manager or use them with MDIS.
Whenever you import an old map or connect MDM 7.1 Import Manager to an MDM repository containing outdated maps, MDM offers to automatically update the outdated import maps. If MDM cannot completely update a map, you must update it manually in Import Manager before it can be used.
NOTE ►► Outdated maps are grayed-out and labeled “[Outdated]” in the Import Manager’s Open Map dialog.
NOTE ►► You cannot manually update a map until MDM has attempted to update it automatically
CAUTION ►► Updating maps is irreversible. You may want to archive a repository before updating its maps.
You can update outdate maps as described below.
To automatically update all outdated maps in an MDM repository:
1. Connect Import Manager to the MDM repository.
2. Import Manager opens the dialog shown in Figure 22.
Figure 22. Outdated Maps dialog
3. Click Yes on the dialog to update the maps.
NOTE ►► If you click No, you will be unable to open the outdated maps in Import Manager or use them with MDIS, and Import Manager will reopen the dialog the next time you connect to the MDM repository.
4. MDM updates the old maps and writes the results to the
UpdateMapLog xml file in the Import Manager’s installation directory.
MDM Import Manager Reference Guide 75
To manually update an outdated map in an MDM repository:
1. In Import Manager, connect to a source file that corresponds to the outdated map.
2. Choose File > Open from the main menu to open the Open Map dialog.
3. Select the map from the Open Map dialog and click OK.
NOTE ►► Maps in need of manual repair have [Requires Manual
Update] appended to their names.
4. Correct the problems noted for the map in the UpdateMapLog file.
5. Save the map.
77
PART 4: BROWSING SOURCE DATA
This part of the reference guide explains how to familiarize yourself with the source data to determine what processing will be necessary to perform the import.
MDM Import Manager Reference Guide 79
Overview
Data that needs to be imported usually comes from a foreign source and you’ve probably never seen it before.
Consequently, the first step in preparing source data for import is simply to familiarize yourself with its table and field structure, as well as the underlying source records and data values. Becoming familiar with source data allows you to determine what MDM Import Manager processing will be necessary before you can perform the import.
The MDM Import Manager allows you to browse and become familiar with source data in several different ways:
Source Preview tab. For browsing all of the records and fields of
each table in a traditional row/column grid.
Source Hierarchy pane. For browsing the overall table and field
structure of the source file – and the underlying set of distinct data values for each field – in a multi-level tree.
These two complementary methods of browsing source data are described in the following sections.
80 MDM Import Manager Reference Guide
The Source Preview Tab
The Source Preview tab displays a grid containing the records of each source table or nested XML element, with a row for each record and a column for each of the fields, as shown in Figure 23.
Figure 23. The Source Preview tab
NOTE ►► The Tables subpane displays nested XML elements in their natural hierarchy, rather than in a list.
To browse the records of any of the source tables:
1. If necessary, click on the Source Preview tab to make it the active tab.
2. In the Tables subpane, select the source table or nested XML element whose records you want to browse.
3. In the Records subpane, interactively browse the records of the selected table and sort by any of the columns in ascending or descending order.
NOTE ►► When you look up fields based on join relationships
between tables, the MDM Import Manager appends [Lookup] to the name of the table in the Tables subpane, and adds each lookup field to the grid in the Records subpane as “field <table>” (where ‘field’ is the name of the lookup field and ‘table’ is the name of the lookup table). (See “Table Joins and Lookups” on page 101 for more information.)
MDM Import Manager Reference Guide 81
The Source Hierarchy Tree
The Source Hierarchy tree provides a high-level view of the table and field structure of the source file, the join relationships between tables,
the lookup fields, and – if you have used the Display Distinct Values command to display the distinct data values for one or more fields – the set of values for each field, as shown in Figure 24.
The Source Hierarchy tree also provides a place from which to perform other source operations, such as creating join and lookup relationships, adding, cloning, and splitting fields, and partitioning fields and values.
Figure 24. Nodes of the Source Hierarchy tree
The types of nodes of the Source Hierarchy tree are described in the following sections, and summarized in Table 7.
Tables
Lookup field
Fields
Data values
Source file
Lookup table
Matching fields
82 MDM Import Manager Reference Guide
TABLES, FIELDS, AND DATA VALUES
At a minimum, the Source Hierarchy tree contains two levels that include the tables and fields of the source file whose records you intend to import.
If you display the list of distinct data values for any field in the current source table, the Source Hierarchy tree contains an additional level that includes the data values.
TIP ►► You can use the Display Distinct Values command to display the list of distinct data values for any field in the current source table (see “Displaying Distinct Values” on page 86 for more information).
TABLE JOINS AND LOOKUPS
If the source file contains join relationships between source tables, or if you create your own join relationships, the Source Hierarchy tree identifies the joined matching fields, and also includes the joined lookup tables and corresponding matching fields as children of the corresponding table and matching field nodes, respectively.
TIP ►► You can use the Join command to create join relationships that don’t already exist in the source data file (see “Joining Tables” on page 103 for more information).
NOTE ►► The MDM Import Manager appends [Join] to the name of both matching fields that participates in a join relationship.
Finally, if you have looked up fields from joined lookup tables to combine all of the fields for import into a single source table, the Source Hierarchy tree also includes the lookup fields as children of the joined lookup table nodes.
TIP ►► You can use the Lookup command to look up fields from joined lookup tables into the current source table (see “Looking Up a Field” on page 104 for more information).
NOTE ►► Joins are transitive: if Table A is joined to Table B and Table B is joined to Table C, Table C is effectively joined to Table A. The MDM Import Manager automatically detects this condition and allows “transitive lookups” from Table C into Table A.
NOTE ►► See “Table Joins and Lookups” on page 101 for more information about how tables and fields associated with joins and lookups are reflected in the Source Hierarchy tree.
MDM Import Manager Reference Guide 83
Table 7. Source Hierarchy Tree Nodes
Level Node Type Description
Root File/Database The name of the source file or database.
1 Tables The list of tables, in alphabetical order.
2 Fields
The list of fields of each table, in alphabetical order (as
children of each table node). (For matching fields, [Join] is appended to the name of the field.)
3 Data Values The distinct data values of each field (as children of each field node, if displayed).
2 Lookup Tables Each table node has as its last children (after the field nodes) any lookup tables to which it is joined.
3 Matching Fields Each field node has as its first children (before the value nodes) any matching fields to which it is joined.
3 Lookup Fields Each lookup table node has as its first children any lookup fields that have been looked up from it.
FIELD TYPES IN THE SOURCE HIERARCHY TREE
The Source Hierarchy tree includes fields from the original data source, as well as additional fields that you create during the import process.
The different types of fields as well as the operations you can perform on them are summarized in Table 8.
Table 8. Field Types and Permitted Operations
Field Type Clone Delete Rename Split Partition
Original Source Field
Lookup Field
Added Field
Cloned Field
Partitioned Field (1)
Split Hierarchy Field (1) (2)
Split Multiple Field
Created Hierarchy Field [delimited]
Created Hierarchy Field [hierarchy] (1) (2)
Compound Field
‘’ = permitted; 1 Clone before partitioning or splitting, then apply the operation to the original and the clone. 2 Split original into multiple rather than into hierarchy, and then partition the split fields.
84 MDM Import Manager Reference Guide
XML SOURCE FILES AND THE SOURCE HIERARCHY TREE
Prior to MDM 7.1, Import Manager flattened all record data from XML source files before importing the data into MDM (see “Data Staging” on page 47). Specifically, the hierarchical nesting that occurs naturally in XML files was replaced by “artificial” tables which then had to be “joined” in the Import Manager to restore the original data relationships.
Starting in MDM 7.1, Import Manager no longer flattens XML record data. Instead, Import Manager now preserves the original hierarchical structure of XML source files and reflects this structure in the Source Hierarchy tree, as shown in Figure 25.
Figure 25. XML Structure in the Source Hierarchy Tree
Ways in which the Source Hierarchy tree now reflects an XML file’s structure include:
Top-level node is the source XML file
Tables in the tree represent nested XML structures
All tables are nested under the root element
Fields in the tree represent data-storing XML elements
Joins and _ID fields are no longer added or required
With these changes, users no longer have to manually recreate the relationships implicit in the XML schema, as they are preserved by MDM and accurately reflected in the source hierarchy tree.
NOTE ►► To ensure that Import Manager correctly interprets the structure of an XML file, specify its corresponding XML schema file in the Connect to Source dialog (see “Starting Import Manager” on page 43 for more information).
MDM Import Manager Reference Guide 85
Source Hierarchy Operations
The following sections describe the various operations you can perform on nodes of the Source Hierarchy tree, as summarized in Table 9.
Table 9. Source Hierarchy Operations
Operation Description
Display Distinct Values
Displays the distinct values for the selected field.
Add Field Adds an empty field to the hierarchy.
Clone Field Clones the selected field and its values.
Rename Field Renames the selected field.
Delete Field Removes the selected field from the hierarchy.
Join Joins the selected field to a field in another table.
Unjoin Unjoins the selected field.
Lookup Looks up a field from another joined table.
Unlookup Removes the selected lookup field from the hierarchy.
Split into Hierarchy
Splits the data values within the selected field into a hierarchy value.
Split into Multiple Fields
Splits the data values within the selected field into multiple fields.
Create Hierarchy Field Creates a new field based on the selected fields and then splits it into a hierarchy.
Create Compound Field
Combines fields that have been mapped to the multiple display fields of a destination lookup field.
Create Pivot Table
Creates a new table by pivoting on the selected fields.
Create Reverse Pivot Table
Creates a new table by reverse pivoting on the selected fields.
NOTE ►► As of MDM 7.1, operations that previously required an updateable data source are now performed virtually. As a result, all source hierarchy operations are now available to all data types.
86 MDM Import Manager Reference Guide
Table 10. Default Names for Fields Added to the Source Hierarchy Tree
Command Field Name
Add Field1,2 New Field (n)
Create Compound Field field <destination lookup table name>
Clone Field1,2 field <Clone> (n)
Split into Hierarchy1,2 field <Split Hierarchy> (n)
Split into Multiple Fields1,3 field <Split Multiple> <m> (n)
1 “field” is the name of the original field 2 “(n)” does not appear unless n >= 2 3 ‘m’ ranges from 1 to the number of new fields
NOTE ►► The maximum field name length is 64 characters for import maps which use Access or Excel source types.
DISPLAYING DISTINCT VALUES
In conjunction with field-at-a-time rather than record-at-a-time processing, distinct value lists make import exception handling much more efficient because:
The number of distinct values for each field is dramatically smaller than the total number of values.
Each distinct value represents and is a proxy for not only all instances of the value across the entire set of underlying records, but also the subset of records that contain the value.
Examining and then transforming each source data value must be done only once for each distinct value rather than once for each instance of the value; and as a result.
The maximum number of exceptions is the sum rather than the product of the number of distinct values in each mapped field.
When you use the Display Distinct Values command to display the list of distinct values for a field in the Source Hierarchy tree, the MDM Import Manager adds the values to the tree as children of the field node, as shown in Figure 26.
MDM Import Manager Reference Guide 87
Figure 26. Display Distinct Values command
NOTE ►► The distinct value list is like the lookup values of the corresponding lookup table if such a table actually existed.
NOTE ►► You can only display data values in the Source Hierarchy tree for fields of the current source table.
Displaying values as nodes in the tree is an optional convenience that allows you to efficiently familiarize yourself with the set of distinct values for a field (as opposed to browsing multiple occurrences of each value in the Source Preview grid).
NOTE ►► The set of distinct values is meaningful primarily for fields that have a relatively small set of distinct values relative to the total number of records, not for an index or description field, where the values may be unique for each record. The MDM Import Manager warns you before displaying the values if the number of distinct values is greater than 100, as shown in Figure 27.
Figure 27. Confirm Distinct Value Display dialog
Value nodes also provide a foundation for partitioning fields and values. (See “Partitioning and Field Mapping” on page 173 for more information about partitioning.)
NOTE ►► If you haven’t already used the Display Distinct Values command to manually add the values as tree nodes, the values and value combinations are automatically added when you partition a field.
By contrast, you must use the Display Distinct Values command to manually add the value nodes prior to partitioning an individual field value.
Distinct values
88 MDM Import Manager Reference Guide
NOTE ►► The set of distinct values that are added as nodes to the tree correspond to the set of distinct values that appear in the Value Conversion and Mapping grids of the Map Fields/ Values tab (see “The Value Conversion and Mapping Grids” on page 210).
TIP ►► You can also use the Display Distinct Values command in the Destination Hierarchy tree to display the set of values for destination fields, attributes, and qualifiers of the current destination table. If the field is a hierarchy field, the entire tree structure is added to the Destination Hierarchy tree.
To display the distinct data values for a field:
1. In the Source Hierarchy tree, select the field node whose distinct data values you want to display.
2. Right-click on the node and choose Display Distinct Values from the
context menu, or choose Source > Display Distinct Values from the main menu.
3. MDM adds the set of distinct data values as children of the selected field node (Figure 26 above).
REMOVING DISTINCT VALUES
You can use the Remove Distinct Values command to remove the set of distinct values from the hierarchy tree, as described in this section.
NOTE ►► You cannot use the Remove Distinct Values command if the field or any of its values have been partitioned.
To remove the distinct data values for a field:
1. In the Source Hierarchy tree, select the field node whose distinct data values you want to remove.
2. Right-click on the node and choose Remove Distinct Values from the
context menu, or choose Source > Remove Distinct Values from the main menu.
3. MDM remove the set of distinct data values as children of the selected field node.
SETTING THE CASE SENSITIVITY OPTION
You can specify whether to ignore or recognize case differences in the source data when the MDM Import Manager determines the set of distinct values for a field.
NOTE ►► The procedure for using the Set Case Sensitivity command is included here for reference. (See “The Case Sensitivity Option” on page 237 for complete information about case sensitivity.)
MDM Import Manager Reference Guide 89
To set the Case Sensitivity option for one or more fields:
1. In the Source Hierarchy tree, select the field nodes for which you want to set the Case Sensitivity option.
2. Right-click on one of the nodes and choose Set Case Sensitivity from
the context menu, or choose Fields > Set Case Sensitivity from the main menu.
TIP ►► Alternatively, you can right-click on the field in the Source
Fields grid and choose Set Case Sensitivity from the context menu.
3. MDM displays a cascading menu of three choices:
Case Insensitive Case Sensitive Inherit Default
4. To set the Case Sensitivity option and override the global default
setting, choose either Case Insensitive or Case Sensitive, or
alternatively, to restore inheritance, choose Inherit Default.
NOTE ►► MDM places a check next to the cascading menu choice that corresponds to the current setting unless you have selected multiple fields for which the settings are not all the same.
ADDING AN EMPTY FIELD
You can add to the source data an empty field with the single NULL value in its list of distinct values. For example, if the source data is missing a field for which all of the values would be the same (such as when data from a particular manufacturer is missing the manufacturer field), you can add a field, and then use the Map Fields/Values tab to map the single NULL value to the corresponding destination value (see “Missing Source Data Fields” on page 231 for more information).
When you use the Add Field command, the MDM Import Manager creates an empty field and adds it to the Source Hierarchy tree. Added fields are highlighted in italics, as shown in Figure 28.
Figure 28. Adding an empty field
90 MDM Import Manager Reference Guide
TIP ►► You can delete an added field using the Delete Field command.
To add an empty field to the Source Hierarchy tree:
1. In the Source Hierarchy tree, select the field node after which you want to add an empty field.
2. Right-click on the node and choose Add Field from the context menu,
or choose Source > Add Field from the main menu.
3. MDM adds a new field node named “New Field ” as a sibling of and just after the original node (Figure 28 above).
4. Press F2 to edit the name of the new field.
5. MDM highlights the name of the new field for editing.
6. Type the name you want for the new field and press Enter.
ADDING A CLONE FIELD
If you need to perform two different operations on a single source field, such as: (1) mapping it to two different destination fields; or (2) mapping it to one destination field by itself and also partitioning it by another source field, you can clone the field to create a copy of the original field.
When you use the Clone Field command to create a clone of a source field, MDM creates a duplicate of the original field that contains the same data value for each source record. You can then perform subsequent processing on the clone field just as if it were an original field in the source data. Clones in the Source Hierarchy tree are highlighted in italics, as shown in Figure 29.
Figure 29. Adding a clone field
TIP ►► You can delete a clone field using the Delete Field command.
To create a clone of a field:
1. In the Source Hierarchy tree, select the field node you want to clone.
2. Right-click on the node and choose Clone Field from the context menu,
or choose Source > Clone Field from the main menu.
MDM Import Manager Reference Guide 91
3. MDM adds a new field node named “field <Clone>” as a sibling of and just after the original node (Figure 29 above).
4. Press F2 to edit the name of the clone.
5. MDM highlights the name of the clone for editing.
6. Type the name you want for the clone and press Enter.
RENAMING A FIELD
Fields that you add to the Source Hierarchy tree are automatically named based on the original field name and the command that you use to create the field. You can then rename the field as described in this section.
NOTE ►► You cannot rename a compound field.
The commands that add fields to the Source Hierarchy tree and the corresponding default field names are summarized in Table 10.
To rename a field in the Source Hierarchy tree:
1. In the Source Hierarchy tree, select the field node you want to rename.
2. Right-click on the node and choose Rename Field from the context menu,
or press F2, or choose Source > Rename Field from the main menu.
3. MDM highlights the field name for editing. Type the new name you
want for the field and press Enter.
DELETING A FIELD
You can delete a field that you previously added to the Source Hierarchy tree as described in this section.
To permanently delete a field from the Source Hierarchy tree:
1. In the Source Hierarchy tree, select the field node you want to delete.
2. Right-click on the node and choose Delete Field from the context menu,
or press Del, or choose Source > Delete Field from the main menu.
3. MDM prompts you to confirm that you really want to delete the field.
Click OK to remove the field from the Source Hierarchy tree.
JOINING TABLES
For data sources that do not store relationships between tables or do not make them available through a programmatic interface, or alternatively, to create additional relationships that are not explicitly defined in the data source, the MDM Import Manager allows you to manually define join relationships from a primary table to a lookup table.
92 MDM Import Manager Reference Guide
NOTE ►► The procedures for using the Join and Unjoin commands are included here for reference. (See “Table Joins and Lookups” on page 101 and “Joining Tables” on page 103 for complete information about joining tables.)
To manually join a primary table to a lookup table:
1. In the Source Hierarchy tree, select the primary matching field node in the primary table from which you want to establish the join relationship.
NOTE ►► The primary table does not need to be the current source table.
2. Right-click on the node and choose Join from the context menu, or
choose Source > Join from the main menu, and select first the lookup table and then the lookup matching field from the two-level cascading menu of source tables and fields for each table.
NOTE ►► You can only join fields with the same data type; other fields are disabled in the cascading menu of fields.
NOTE ►► You can also drag-and-drop the primary matching field from the primary table onto the lookup matching field in the lookup table to create the join relationship.
3. MDM creates the new join relationship and reflects it in the Source Hierarchy tree.
NOTE ►► See “The Source Hierarchy Tree” on page 81 for more information about the nodes of the Source Hierarchy tree.
To remove a join relationship between two tables:
1. In the Source Hierarchy tree, select either the primary matching field
node in the lookup table (i.e. the [To: …] node) or the lookup matching
field in the primary table (i.e. the [From: …] node) that corresponds to the join relationship you want to remove.
2. Right-click on the node and choose Unjoin from the context menu, or
choose Source > Unjoin from the main menu.
3. MDM eliminates the join relationship and removes the matching field nodes, the lookup table node, and any lookup field nodes from the Source Hierarchy tree.
MDM Import Manager Reference Guide 93
LOOKING UP A FIELD
Once the join relationships have been established, you can look up fields from a lookup table into a primary table, adding the field into the list of primary table fields, and in effect, building an extended virtual record consisting of fields from the primary table and lookup fields from one or more lookup tables.
NOTE ►► The procedures for using the Lookup and Unlookup commands are included here for reference. (See “Table Joins and Lookups” on page 101 and “Looking Up a Field” on page 104 for complete information about looking up fields.)
To look up a field from a lookup table into a primary table:
1. In the Source Hierarchy tree, select the lookup table node in the primary table into which you want to look up a field.
NOTE ►► The primary table does not need to be the current source table.
2. Right-click on the node and choose Lookup from the context menu, or
choose Source > Lookup from the main menu, and select the lookup field from the cascading menu of fields in the lookup table.
TIP ►► You can also drag-and-drop the lookup field from the lookup table onto the lookup table in the primary table to create the lookup.
TIP ►► You can select [All] from the cascading menu to lookup all of the fields of the lookup table.
3. MDM creates the new lookup field and adds it to the Source Hierarchy tree.
NOTE ►► See “The Source Hierarchy Tree” on page 81 for more information about the nodes of the Source Hierarchy tree.
To remove a lookup field from a primary table:
1. In the Source Hierarchy tree, select the lookup field node you want to remove in the primary table.
2. Right-click on the node and choose Unlookup from the context menu,
or choose Source > Unlookup from the main menu.
3. MDM eliminates the lookup field and removes its node from the Source Hierarchy tree.
NOTE ►► When you remove a join relationship, the MDM Import Manager automatically removes the lookup fields as well.
94 MDM Import Manager Reference Guide
SPLITTING A FIELD
You can split a field containing multiple delimited values into: (1) a single field containing the values expanded into a hierarchy; or (2) multiple fields, each containing a single value.
NOTE ►► The procedures for using the Split into Hierarchy and Split
into Multiple Fields commands are included here for reference. (See “Splitting Fields” on page 146 for complete information about splitting fields.)
NOTE ►► The Split into Hierarchy and Split into Multiple Fields commands described in this section identify the individual delimited values within a single field using the current Split Hierarchy or Split Multiple Fields delimiter strings, respectively, which must be properly set prior to performing the commands (see “Setting the Split Delimiters” on page 150 for more information).
To split a field with multiple delimited values into a hierarchy:
1. In the Source Hierarchy tree, select the field node whose multiple delimited values you want to expand into a hierarchy.
2. Right-click on the node and choose Split into Hierarchy from the
context menu, or choose Source > Split into Hierarchy from the main menu.
3. MDM adds a new field node named “field <Split Hierarchy>” as a sibling of and just after the original node.
4. Press F2 to edit the name of the split field.
5. MDM highlights the name of the split field for editing.
6. Type the name you want for the split field and press Enter.
To split a field with multiple delimited values into multiple fields:
1. In the Source Hierarchy tree, select the field node whose multiple delimited values you want to split into multiple fields.
2. Right-click on the node and choose Split into Multiple Fields from the
context menu, or choose Source > Split into Multiple Fields from the main menu.
3. MDM adds one or more new field nodes named “field <Split Multiple> <m>” (where ‘m’ ranges from 1 to the number of new fields) as a sibling of and just after the original node.
4. Press F2 to edit the name of each split field.
5. MDM highlights the name of the split field for editing.
6. Type the name you want for the split field and press Enter.
MDM Import Manager Reference Guide 95
BUILDING A HIERARCHY
You can build a hierarchy from a set of fields identifying parent/child relationships, as described in this section.
NOTE ►► The procedure for using the Create Hierarchy Field command is included here for reference. (See “Building a Hierarchy” on page 156 for complete information about building a hierarchy.)
To combine two or three fields containing hierarchy data into a hierarchy:
1. In the Source Hierarchy tree, select the two or three field nodes you want to use build into a hierarchy:
Parent/child value pairs with node name – three
Parent/child value pairs w/o node name – two Hierarchy codes with node name – two
2. Right-click on one of the nodes and choose Create Hierarchy Field from
the context menu, or choose Source > Create Hierarchy Field.
3. MDM opens the Create Hierarchy Field dialog shown in Figure 30.
Figure 30. Create Hierarchy Field dialog
4. In the Hierarchy Field Name edit control, type the name for the new field.
5. In the Parent Field, Child Field, and Node Name Field drop-down controls, identify the field containing the parent value, the child value, and the node name value respectively.
TIP ►► For two-node selection of parent/child value pairs without a
node name, set the Node Name Field to None. MDM uses the values of
Child Field as the name of each node.
TIP ►► For two-node selection of hierarchy codes with a node name,
set the Parent Field to None. MDM automatically decodes the values of Child Field to determine the parent/child hierarchy relationships.
6. Click OK to close the Create Hierarchy Field dialog.
7. The MDM Import Manager creates a hierarchy field named “field <Split Hierarchy>” as a sibling of and just after the first new node (Figure 104 above).
96 MDM Import Manager Reference Guide
8. Press F2 to edit the name of the hierarchy field.
9. MDM highlights the name of the hierarchy field for editing.
10. Type the name you want for the hierarchy field and press Enter.
NOTE ►► The Create Hierarchy Field command uses the current Split Hierarchy delimiter string to delimit the nodes in the new hierarchy field and then to split the individual delimited values within the new field (see “Setting the Split Delimiters” on page 150 for more information).
CREATING A COMPOUND SOURCE FIELD
Please see "Creating a Compound Source Field" on page 275 for complete information about compound source fields and how to create them
CREATING A PIVOT TABLE
Please see “Pivoting” on page 110 and “Creating a Pivot Table” on page 123 for complete information about pivot tables and how to create them.
CREATING A REVERSE PIVOT TABLE
Please see “Reverse Pivoting” on page 117 and “Creating a Reverse Pivot Table” on page 133 for complete information about reverse pivot tables and how to create them.
97
PART 5: TABLE JOINS AND LOOKUPS
This part of the reference guide explains how to combine information from multiple source tables into a single table for import by creating join relationships between the tables, and then looking up fields from one table into another, so that all of the fields exist in the records of a single table.
MDM Import Manager Reference Guide 99
Overview
The MDM Import Manager imports records from a single table at a time into a single table at a time.
However, when the source data is relational, and the data to be imported exists as fields in multiple source tables, you must first create join relationships between the tables, and then look up fields from one table into another, so that all of the fields exist in the records of a single table. In effect, table joins and lookups allow you to combine information from multiple source tables into a single virtual table for import.
NOTE ►► When you have a single-table data source or an XML data source, no table joins or lookups are necessary, and this step can be skipped entirely. Table joins and lookups are necessary only with multi-table data sources and only when the data to be imported exists as fields in multiple tables.
The benefit of using table joins and lookups to create a virtual “lookup-extended” source table consisting of fields from more than one table is that the source data does not need to be “pre-flattened” (i.e. normalized) into a single table within an external application prior to import, nor do additional tables need to be created in the original data source. Instead, the flattening can be done “on-the-fly” within the virtual workspace created by the MDM Import Manager, as part of the import process itself.
NOTE ►► The virtual lookup-extended source table contains all of the records and all of the fields of the primary table, plus each of the lookup fields from each of the lookup tables.
Each of the operations that support table joins and lookups is described in the following sections, and summarized in Table 11.
Table 11. Table Join and Lookup Operations
Operation Description
Join Joins the selected field to a field in another table.
Unjoin Unjoins the selected field.
Lookup Looks up a field from another joined table.
Unlookup Removes the selected lookup field from the hierarchy.
100 MDM Import Manager Reference Guide
MULTIPLE TABLES OF SOURCE DATA
By way of example, consider the Products, Manufacturers, and Countries tables shown in Figure 31, three tables that contain related information on products.
Each product record in the Products table contains a MFG code but no manufacturer name, which is contained in the Manufacturers table for each manufacturer record.
Similarly, each manufacturer record in the Manufacturers table contains a CNTRY code but no country name, which is contained in the Countries table for each country record.
Finally, the Products table contains no information whatsoever on the country of origin.
Products:
NAME … MFG Countries:
Belt … 2011 Manufacturers: CODE NAME
End Brush … 1212 CODE NAME CNTRY E789 ENG
Flap Disc … 2011 1212 Weiler U123 F456 FRA
Grinder … 2011 2011 Radnor E789 U123 USA
Grinder … 2011 3478 CGW U123
Nut-Driver … 5124 5124 Morgan F456
Ratchet … 3478
Figure 31. Multiple tables containing related information
If you want to import records from the Products table along with the manufacturer name and country name for each product record, then you must combine the information from the three tables using table joins and lookups, as described in the following sections and as illustrated in Figure 32.
Products: Manufacturers: Countries:
NAME … MFG NAME NAME
Belt … 2011 Radnor ENG
End Brush … 1212 Weiler USA
Flap Disc … 2011 Radnor ENG
Grinder … 2011 Radnor ENG
Grinder … 2011 Radnor ENG
Nut-Driver … 5124 Morgan FRA
Ratchet … 3478 CGW USA
Figure 32. Information combined from multiple tables
MDM Import Manager Reference Guide 101
TABLE JOINS AND LOOKUPS
In simple terms, a join combines information from the records in two tables by using the values in a pair of matching fields to match records
in the two tables.
A lookup is a type of join that uses the value of the primary matching field for each record in a primary table to match the corresponding record with the same value for the lookup matching field in a lookup table, and then “looks up” additional information from one or more lookup fields in the lookup table record into the primary table record.
These table join and lookup terms – and how each type of table and field associated with joins and lookups is reflected in the Source Hierarchy tree – are shown in Figure 33 and summarized in Table 12.
Figure 33. Table joins and lookups in the Source Hierarchy tree
Lookup matching fields
Lookup table
Transitive lookup table
Lookup fields
Primary matching fields
102 MDM Import Manager Reference Guide
Table 12. Table Join and Lookup Terms and the Source Hierarchy Tree
Term Description / Display in Source Hierarchy Tree
Join Combines information from records in multiple tables.
Matching Fields The pair of fields whose values are used to match records.
Lookup Uses a join to look up information from one table into another.
Primary Table The table into which fields are looked up.
Lookup Table The table from which fields are looked up.
Added as the last child of the primary table
Primary Matching Field
The matching field in the primary table.
[Join] appended to the name of the primary matching field.
Also added as the first child of the lookup matching field, as
[From: primary table.primary matching field]
Lookup Matching Field
The matching field in the lookup table.
[Join] appended to the name of the lookup matching field.
Also added as the first child of the primary matching field, as
[To: lookup table.lookup matching field]
Lookup Fields Fields looked up from the lookup table into the primary table.
Added as children of the lookup table node
NOTE ►► Joins are transitive: if Table A is joined to Table B and Table B is joined to Table C, Table C is effectively joined to Table A. The MDM Import Manager automatically detects this condition, displays “transitive join” Table C as the child not only of Table B but also of the joined lookup Table B node that is a child of Table A, and allows “transitive lookups” from Table C into Table A.
TECHNICAL NOTE ►► In a typical relational DBMS, joins are usually supported by explicit primary key and foreign key matching fields within each of the two tables, and lookups occur as a result of explicit queries or views. By contrast, MDM automatically creates implicit primary key and foreign key matching fields when you create a field in the repository of type Lookup, and the value of the lookup field is the value(s) of the display field(s) of the lookup table record.
TECHNICAL NOTE ►► In a typical DBMS application, most joins are equi-joins. An equi-join is symmetrical, in that both tables are equal with no “from” or “to” table; combining information from the two tables creates a “result” table that includes the specified fields from either of the tables, but might not include all of the records of either table. By contrast, the MDM Import Manager uses left-outer joins to creates the lookup-extended table. A left-outer join treats the “from” table as the primary table and the “to” table as the lookup table, and the result table includes all of the primary table fields and each of the lookup fields, but most importantly, includes all of the records of the primary table.
MDM Import Manager Reference Guide 103
In the MDM Import Manager, you can use the Join command (or drag-and-drop) to join a primary source table to one or more lookup source
tables, and then use the Lookup command (or drag-and-drop) to look up fields from the lookup tables into the primary table, as described in the following sections.
JOINING TABLES
For data sources that do not store relationships between tables or do not make them available through a programmatic interface, or alternatively, to create additional relationships that are not explicitly defined in the data source, the MDM Import Manager allows you to manually define join relationships from a primary table to a lookup table.
NOTE ►► For data sources that store “primary key / foreign key” and/or “main table / related table” relationships and that make this information available through a programmatic interface, the MDM Import Manager automatically detects these join relationships when you connect to the data source and reflects them in the Source Hierarchy tree. For example, Microsoft Access stores them and makes them available through the ADO interface used by the MDM Import Manager to connect to Access.
When you use the Join command to manually join a primary table to a lookup table, as shown in Figure 34, MDM does the following:
Creates a link between the primary matching field in the primary table and the lookup matching field in the lookup table
Appends [Join] to the name of each matching field in the tree.
Adds the corresponding matching field of the other table as the first child of each matching field.
Adds the lookup table as the last child of the primary table.
Figure 34. Source Hierarchy tree before and after Join
NOTE ►► The Join command joins two tables in the Source Hierarchy tree and in the MDM Import Manager’s virtual workspace, but does not change the structure of the underlying data source.
104 MDM Import Manager Reference Guide
To manually join a primary table to a lookup table:
1. In the Source Hierarchy tree, select the primary matching field node in the primary table from which you want to establish the join relationship.
NOTE ►► The primary table does not need to be the current source table.
2. Right-click on the node and choose Join from the context menu, or
choose Source > Join from the main menu, and select first the lookup table and then the lookup matching field from the two-level cascading menu of source tables and fields for each table.
NOTE ►► You can only join fields with the same data type; other fields are disabled in the cascading menu of fields.
NOTE ►► You can also drag-and-drop the primary matching field from the primary table onto the lookup matching field in the lookup table to create the join relationship.
3. MDM creates the new join relationship and reflects it in the Source Hierarchy tree.
NOTE ►► See “The Source Hierarchy Tree” on page 81 for more information about the nodes of the Source Hierarchy tree.
To remove a join relationship between two tables:
1. In the Source Hierarchy tree, select either the primary matching field
node in the lookup table (i.e. the [To: …] node) or the lookup matching
field in the primary table (i.e. the [From: …] node) that corresponds to the join relationship you want to remove.
2. Right-click on the node and choose Unjoin from the context menu, or
choose Source > Unjoin from the main menu.
3. MDM eliminates the join relationship and removes the matching field nodes, the lookup table node, and any lookup field nodes from the Source Hierarchy tree.
LOOKING UP A FIELD
Once the join relationships have been established, you can look up fields from a lookup table into a primary table, adding the field into the list of primary table fields, and in effect, building an extended virtual record consisting of fields from the primary table and lookup fields from one or more lookup tables.
MDM Import Manager Reference Guide 105
When you use the Lookup command to look up a field from a lookup table into the primary table, as shown in Figure 35, the MDM Import Manager adds the lookup field as a child of the lookup table. You can then perform subsequent processing on the lookup field just as if it were an original field in the source data.
Figure 35. Source Hierarchy tree before and after Lookup
NOTE ►► The Lookup command adds a lookup field to the Source Hierarchy tree and to the MDM Import Manager’s virtual workspace, but does not change the structure of the underlying data source.
NOTE ►► The lookup table can be nested more than one level below the primary table through the effect of “transitive” joins, as described in the previous section.
To look up a field from a lookup table into a primary table:
1. In the Source Hierarchy tree, select the lookup table node in the primary table into which you want to look up a field.
NOTE ►► The primary table does not need to be the current source table.
2. Right-click on the node and choose Lookup from the context menu, or
choose Source > Lookup from the main menu, and select the lookup field from the cascading menu of fields in the lookup table.
TIP ►► You can also drag-and-drop the lookup field from the lookup table onto the lookup table in the primary table to create the lookup.
TIP ►► You can select [All] from the cascading menu to lookup all of the fields of the lookup table.
3. MDM creates the new lookup field and adds it to the Source Hierarchy tree.
NOTE ►► See “The Source Hierarchy Tree” on page 81 for more information about the nodes of the Source Hierarchy tree.
106 MDM Import Manager Reference Guide
To remove a lookup field from a primary table:
1. In the Source Hierarchy tree, select the lookup field node you want to remove in the primary table.
2. Right-click on the node and choose Unlookup from the context menu,
or choose Source > Unlookup from the main menu.
3. MDM eliminates the lookup field and removes its node from the Source Hierarchy tree.
NOTE ►► When you remove a join relationship, the MDM Import Manager automatically removes the lookup fields as well.
107
PART 6: PIVOTING/REVERSE PIVOTING
This part of the reference guide explains the pivoting operations of the MDM Import Manager, and how to use them to “rotate” source data prior to subsequent restructuring operations.
MDM Import Manager Reference Guide 109
Overview
Pivoting and reverse pivoting are perhaps the most powerful restructuring mechanisms available within the MDM Import Manager.
They “rotate” source data, creating an entirely new table from the source table, to completely transform the data in ways that the other restructuring mechanisms cannot address.
In particular, the pivot and reverse pivot operations behave as follows:
Pivot. Translates the distinct data values in one or more fields into
field names, increasing the number of fields and reducing the number of rows in the pivot table.
Reverse pivot. Translates one or more field names into the data
values within a single field, reducing the number of fields and increasing the number of rows in the reverse pivot table.
Figure 36 illustrates the symmetry of the pivoting and reverse pivoting operations, each of which reverses the action of the other.
Field Field Field Field X Y Z
1 X a 1 a b c
1 Y b 2 d e f
1 Z c Pivot ►
◄ Reverse Pivot 2 X d
2 Y e
2 Z f
Figure 36. Pivot vs. reverse pivot
Pivoting and reverse pivoting are fundamentally different from – and if they are necessary, must precede – the splittingand partitioning operations described in subsequent parts of this guide, since they create an entirely new table from the source table on which all of those other restructuring operations should be performed.
NOTE ►► Splitting expands a single field of values into multiple fields, each containing a single value extracted from the original field. Conversely, partitioning combines multiple fields and the values within them into field and value combinations. In both cases, the operations are within and between fields of the original source table, leaving the set of original source records untouched.
TIP ►► When pivoting or reverse pivoting is necessary, the basic sequence of steps is to: (1) select the source table you want to restructure; (2) perform the pivot or reverse pivot; and (3) change the current source table to the newly created pivot or reverse pivot table.
110 MDM Import Manager Reference Guide
PIVOTING
Pivoting creates a new table by translating the distinct data values in one or more source fields into field names, increasing the number of fields and reducing the number of rows in the pivot table.
Specifically, pivoting: (1) treats as metadata the values within one or more source fields and uses them as the name of new fields in the pivot table; (2) rearranges the corresponding data values from one or more other source fields within the new fields, and (3) collapses duplicates of each product record in the source table into a single record.
Figure 37 illustrates a simple pivot operation on a single pair of fields.
Original table: Pivot table:
Key Metadata Data Key X Y Z
1 X a 1 a b c
1 Y b 2 d e f
1 Z c
Pivot ►
2 X d
2 Y e
2 Z f
Figure 37. Pivot translates metadata values into field names
NOTE ►► A pivot in the MDM Import Manager is like adding one or more horizontal pivots in MDM Publisher, which takes the distinct data values from the pivot columns and uses them to label the base tables which have been arranged side-by-side.
Pivoting is very flexible. You can pivot by a single pair of source fields, one containing the metadata values and the other the data values. You can pivot by multiple metadata field combinations and/or multiple data field combinations. Finally, you can pivot simultaneously by multiple pairs of corresponding fields of metadata and data.
In each case, you must identify the source fields that participate in the pivot, which ones contain metadata and which ones contain data, which ones must be combined, the one-to-one correspondence between fields and/or field combinations, and the key field that the MDM Import Manager should use to identify duplicate records.
Various pivoting challenges are illustrated in the following sections.
NOTE ►► Pivoting is often required when field and attribute data is stacked “vertically” in the source table as name/value pairs and needs to be rearranged into a “horizontal” table structure from which it can be imported into the fields and attributes of a regular table, as described in subsequent parts of this guide.
MDM Import Manager Reference Guide 111
Challenge: Single Pair of Metadata/Data Fields
Most frequently, the fields by which to pivot consist of an individual field of metadata and an individual field of data, which together comprise a single pair of corresponding source fields.
Consider the table shown in Figure 38, which contains two fields, QTY and PRICE, that are used to store quantity pricing information.
PART NO QTY PRICE
101 50 $70
101 250 $60
101 500 $50
102 50 $80
102 250 $70
102 500 $60
103 50 $90
103 250 $75
103 500 $60
Figure 38. Single pair of corresponding metadata and data fields
NOTE ►► PART NO is used as the key field to identify the product records that have been duplicated for each quantity.
In the figure above, the set of distinct values of QTY are the metadata values that represent the field names for each corresponding PRICE data value. Figure 39 illustrates the table that results from pivoting by QTY against PRICE.
PART NO 50 250 500
101 $70 $60 $50
102 $80 $70 $60
103 $90 $75 $60
Figure 39. Pivot table
112 MDM Import Manager Reference Guide
Challenge: Metadata Field Combination
Sometimes, the fields by which to pivot consist of multiple metadata fields that correspond to a single data field in the source table.
Consider the table shown in Figure 40, which contains three fields, DIST, QTY, and PRICE, that are used to store distributor-specific quantity pricing information.
PART NO DIST QTY PRICE
101 McMaster 50 $70
101 McMaster 250 $60
101 McMaster 500 $50
101 Grainger 50 $68
101 Grainger 250 $58
101 Grainger 500 $48
102 McMaster 50 $80
102 McMaster 250 $70
102 McMaster 500 $60
102 Grainger 50 $78
102 Grainger 250 $68
102 Grainger 500 $58
103 McMaster 50 $90
103 McMaster 250 $75
103 McMaster 500 $60
103 Grainger 50 $88
103 Grainger 250 $73
103 Grainger 500 $58
Figure 40. Metadata field combination
In the figure above, the set of distinct value combinations of DIST and
QTY are the metadata values that represent the field names for each corresponding PRICE data value. Figure 41 illustrates the table that results from pivoting by DIST and QTY against PRICE.
PART NO M/50 M/250 M/500 G/50 G/250 G/500
101 $70 $60 $50 $68 $58 $48
102 $80 $70 $60 $78 $68 $58
103 $90 $75 $60 $88 $73 $58
Figure 41. Pivot table
NOTE ►► Each field name in the pivot table is a metadata value combination (abbreviated in the table above for space considerations).
MDM Import Manager Reference Guide 113
Challenge: Data Field Combination
Alternatively, the fields by which to pivot consist of a single metadata field in the source table that corresponds to multiple data fields, each of which contains a different type of data.
Consider the table shown in Figure 42, which contains three fields, DIST, PART NO, and PRICE, that are used to store distributor-specific part numbers and distributor-specific pricing information (two pieces of data for each distributor).
SKU DIST PART NO PRICE
101 McMaster M101 $70
101 Grainger G101 $68
101 Applied A101 $50
102 McMaster M102 $80
102 Grainger G102 $78
102 Applied A102 $80
103 McMaster M103 $90
103 Grainger G103 $88
103 Applied A103 $73
Figure 42. Data field combination
NOTE ►► In this case, SKU is used as the key field to identify the product records that have been duplicated for each distributor.
In the figure above, the set of distinct values of DIST are the metadata values that represent the field names for each corresponding PART NO and PRICE data value. Figure 43 illustrates the table that results from pivoting by DIST against both PART NO and PRICE.
SKU P.NO: M P.NO: G P.NO: A PRICE: M PRICE: G PRICE: A
101 M101 G101 A101 $70 $68 $50
102 M102 G102 A102 $80 $78 $80
103 M103 G103 A103 $90 $88 $73
Figure 43. Pivot table
NOTE ►► When multiple data fields correspond to the metadata field (so that multiple data values correspond to each metadata value), the pivot table contains a complete set of columns for each data field.
114 MDM Import Manager Reference Guide
Challenge: Single Name/Value Field Pair
A very common real-world challenge is when the pair of fields by which to pivot represent both field and attribute name/value pairs, where each attribute applies to just some of the product records, and each product record is duplicated for each name/value pair.
NOTE ►► This name/value structure for attribute information is a common alternative to very wide (and sparse) source records with a large number of fields, most of which would contain NULL values.
Consider the table shown in Figure 44, which contains two fields, NAME and VALUE, that are used to store field and attribute information for each source record.
PART NO NAME VALUE
101 Width 4 mm
101 Diameter 2 mm
101 Weight 8 oz
101 Material Steel
102 HP 4 hp
102 RPM 600 rpm
102 Weight 5 lb
103 Width 8 mm
103 Diameter 4 mm
103 Weight 1 lb
103 Material Copper
Figure 44. Single pair of name/value fields
In the figure above, the set of distinct values of NAME represents the field and attribute names for each corresponding data value. Figure 45 illustrates the table that results from pivoting by NAME against VALUE.
PART NO Width Diameter Weight Material HP RPM
101 4 mm 2 mm 8 oz Steel
102 5 lb 4 hp 600 rpm
103 8 mm 4 mm 1 lb Copper
Figure 45. Pivot table
NOTE ►► The new table is now structured so that each field and attribute can be individually split, partitioned, and/or mapped as described in subsequent parts of this guide.
MDM Import Manager Reference Guide 115
NOTE ►► Attribute data that is stacked vertically as name/value pairs can also be imported directly without pivots by first partitioning the name and value source fields, and then mapping the resulting name/value hierarchy to a special destination field containing a name/value hierarchy of attributes and attribute data values (see “Attribute Data Import as Name/Value Pairs” on page 167 for more information).
Challenge: Multiple Name/Value Field Pairs
Another real-world challenge is when each source record contains multiple name/value pairs of field and attribute information.
NOTE ►► Multiple name/value field pairs for each source record is a common alternative to duplicating the product record for each name/value pair.
Consider the table shown in Figure 46, which contains five pairs of name/value fields and just a single source record for each product record.
PART NO NAME1 VAL1 NAME 2 VAL2 NAME 3 VAL3 NAME 4 VAL4 NAME 5 VAL5
101 Width 4 mm Height 6 mm Diameter 2 mm Weight 8 oz Material Steel
102 HP 4 hp RPM 600 rpm Weight 5 lb
103 Height 12 mm Width 8 mm Weight 1 lb Material Copper Diameter 4 mm
Figure 46. Multiple pairs of name/value fields
Figure 47 illustrates the table that results from pivoting by each of the five NAME fields against each of the five VAL fields.
PART NO Width HP Height RPM Diameter Weight Material
101 4 mm 6 mm 2 mm 8 oz Steel
102 4 hp 600 rpm 5 lb
103 8 mm 12 mm 4 mm 1 lb Copper
Figure 47. Pivot table
NOTE ►► In this example, the pivot does not reduce the number of rows in the table because the product records had not been duplicated in the source table.
116 MDM Import Manager Reference Guide
Challenge: Multi-Valued Information
A variation on name/value pairs is when the source data represents multi-valued field and attribute information. Each product record is once again duplicated, not only for each name/value pair, but also for each instance of a field or attribute name, which can occur once for each value of the multi-valued field.
Consider the table shown in Figure 48, in which the Color attribute appears multiple times for each product record.
PART NO NAME VALUE
101 Width 4 mm
101 Diameter 2 mm
101 Weight 8 oz
101 Color Red
101 Color Green
101 Color Blue
102 HP 4 hp
102 RPM 600 rpm
102 Weight 5 lb
102 Color Black
102 Color White
103 Width 8 mm
103 Diameter 4 mm
103 Weight 1 lb
103 Color White
Figure 48. Name/value fields representing multi-valued information
Figure 49 illustrates the table that results from pivoting by NAME against VALUE.
PART NO Width Diameter Weight HP RPM Color
101 4 mm 2 mm 8 oz Red; Green; Blue
102 5 lb 4 hp 600 rpm Black; White
103 8 mm 4 mm 1 lb White
Figure 49. Pivot table with multi-valued field
NOTE ►► All of the multiple Color values for each product record are automatically combined into a single field in the pivot table: (1) in the order the values appear in the source table; and (2) using the Multi-Valued Delimiter to separate individual values.
MDM Import Manager Reference Guide 117
Challenge: No Key Fields
Sometimes, you may need to base a pivot on all of the fields in a table, such as when a table has only two fields. In these cases, there are no fields available to use as key fields.
Consider the table shown in Figure 50, which contains two fields, MAKE and TRANSMISSION.
MAKE TRANSMISSION
Ford Automatic
VW Automatic
VW Manual
Toyota Automatic
Toyota Manual
Porsche Manual
Figure 50. A two-field table
In the figure above, the set of distinct values of the MAKE field are the metadata values that represent the field names for each corresponding TRANSMISSION value. Prior to MDM 7.1, however, the lack of a key field meant you could not pivot on this table. With MDM 7.1, Import
Manager includes a [Natural Order] key field option, which uses the order in which data appears in the source file as the table’s implied key values. Import Manager then creates an Implicit Key field on the pivot table to serve as the table’s key field.
Figure 49 illustrates the table that results from pivoting by MAKE against TRANSMISSION, using the [Natural Order] option as the key field.
Implicit Key Ford VW Toyota Porsche
1 Automatic Automatic; Manual Automatic; Manual Manual
Figure 51. Pivot table with Implicit Key field added
NOTE ►► The [Natural Order] option is especially useful for creating pivot tables from XML source files, as in MDM 7.1, Import Manager no longer adds _ID join fields to XML source hierarchies.
118 MDM Import Manager Reference Guide
REVERSE PIVOTING
Reverse pivoting creates a new table by translating one or more source field names into data values within a single field, reducing the number of fields and increasing the number of rows in the reverse pivot table.
Specifically, reverse pivoting: (1) places the names of one or more source fields into a new field of metadata values in the pivot table; (2) eliminates those fields and rearranges the data values within them into another new field of corresponding data values; and (3) duplicates each product record in the source table for each field that is eliminated.
Figure 52 illustrates a simple reverse pivot operation.
Original table: Reverse pivot table:
Key X Y Z Key Metadata Data
1 a b c 1 X a
2 d e f 1 Y b
Pivot
Reverse Pivot ►
1 Z c
2 X d
2 Y e
2 Z f
Figure 52. Reverse pivot translates field names into metadata values
NOTE ►► A reverse pivot in the MDM Import Manager is like removing one more horizontal pivots in MDM Publisher.
Like pivoting, reverse pivoting is very flexible. You can reverse pivot by a single set of source fields, and you can reverse pivot by multiple sets of source fields, where each set contains a different type of data values. In each case, you must identify the source fields that participate in the reverse pivot and the source fields in each set.
Various reverse pivoting challenges are illustrated in the following sections.
NOTE ►► Reverse pivoting is often required when qualified data is arranged “horizontally” in the source table and needs to be rearranged into a “vertical” table structure from which it can be imported into the fields and qualifiers of a qualified table, as described in subsequent parts of this guide.
MDM Import Manager Reference Guide 119
Challenge: Single Set of Data Fields
Most frequently, the fields by which to reverse pivot consist of a single set of data fields, all of which contain the same type of data, and each field corresponds to a single metadata value.
Consider the table shown in Figure 53, which contains three fields that are used to store quantity pricing information, where each field corresponds to a single quantity value.
PART NO Q50 Q250 Q500
101 $70 $60 $50
102 $80 $70 $60
103 $90 $75 $60
Figure 53. Single set of data fields
Figure 54 illustrates the table that results from reverse pivoting by each of the three quantity pricing fields.
PART NO Names Values
101 Q50 $70
101 Q250 $60
101 Q500 $50
102 Q50 $80
102 Q250 $70
102 Q500 $60
103 Q50 $90
103 Q250 $75
103 Q500 $60
Figure 54. Reverse pivot table
TIP ►► During field and value mapping in a later step of the import process, the Names field would most likely be mapped to the Quantity display field of the qualified lookup table (and each Names value converted or mapped to a quantity value), and the Values field would most likely be mapped to the Prices qualifier.
120 MDM Import Manager Reference Guide
Challenge: Metadata Field Combinations
Sometimes, each field in the single set of data fields by which to reverse pivot represents a combination of metadata values, so that when you perform the reverse pivot, each field value corresponds to a metadata value combination.
Consider the table shown in Figure 55, which contains four fields that are used to store distributor-specific quantity pricing information, where each field corresponds to a single distributor/quantity value combination.
PART NO M100 M500 G100 G500
101 $70 $50 $68 $48
102 $80 $60 $78 $58
103 $90 $60 $88 $58
Figure 55. Each data field corresponds to a metadata value combination
Figure 56 illustrates the table that results from reverse pivoting by each of the four distributor/quantity pricing fields.
PART NO Names Values
101 M100 $70
101 M500 $50
101 G100 $68
101 G500 $48
102 M100 $80
102 M500 $60
102 G100 $78
102 G500 $58
103 M100 $90
103 M500 $60
103 G100 $88
103 G500 $58
Figure 56. Reverse pivot table
TIP ►► When the Names column represents destination value combinations, the display fields and/or qualifiers of the destination qualified lookup table would most likely need to be partitioned prior to field mapping.
MDM Import Manager Reference Guide 121
Challenge: Multiple Sets of Data Fields
The most complex challenge occurs when the fields by which to reverse pivot consist of multiple sets of data fields, where the fields within each set contain the same type of data, each set contains a different type of data, and each field corresponds to a single metadata value.
Consider the table shown in Figure 57, which contains six fields of distributor-specific information and two types of data. Three of the fields contain distributor-specific part numbers and three of them contain distributor-specific prices.
SKU M P.NO M PRICE G P.NO G PRICE A P.NO A PRICE
101 M101 $70 G101 $68 A101 $50
102 M102 $80 G102 $78 A102 $80
103 M103 $90 G103 $88 A103 $73
Figure 57. Multiple sets of data fields
Figure 58 illustrates the table that results from reverse pivoting by the three pairs of distributor-specific fields.
SKU Names Values (1) Values (2)
101 M P.NO; M PRICE M101 $70
101 G P.NO; G PRICE G101 $68
101 A P.NO; A PRICE A101 $50
102 M P.NO; M PRICE M102 $80
102 G P.NO; G PRICE G102 $78
102 A P.NO; A PRICE A102 $80
103 M P.NO; M PRICE M103 $90
103 G P.NO; G PRICE G103 $88
103 A P.NO; A PRICE A103 $73
Figure 58. Reverse pivot table
TIP ►► When there are multiple sets of source fields, the relationship between the fields of source data – and the proper reverse pivot – may not be immediately obvious. In these cases, it can help to imagine the source table as having been horizontally pivoted, and each subset of fields as representing a base table (see “Imagining Horizontal Pivots” on page 138 for more information).
122 MDM Import Manager Reference Guide
Pivoting and Reverse Pivoting Operations
The following sections describe the pivoting and reverse pivoting operations you can perform on a source table, including:
Pivoting by a single pair of source fields.
Pivoting by multiple metadata fields.
Pivoting by multiple data fields.
Pivoting by multiple pairs of source fields.
Reverse pivoting by a single set of source fields.
Reverse pivoting by multiple sets of source fields.
The operations are summarized in Table 13.
Table 13. Pivoting and Reverse Pivoting Operations
Operation Description
Create Pivot Table
Creates a new table by pivoting on the selected fields.
Create Reverse Pivot Table
Creates a new table by reverse pivoting on the selected fields.
NOTE ►► Pivoting and reverse pivoting are fundamentally different from – and if they are necessary, must precede – the splitting and partitioning operations described in subsequent parts of this guide, since they create an entirely new table from the source table on which all of those other restructuring operations should be performed.
TIP ►► When pivoting or reverse pivoting is necessary, the basic sequence of steps is to: (1) select the source table you want to restructure; (2) perform the pivot or reverse pivot; (3) change the current source table to the newly created pivot or reverse pivot table; and (4) proceed with the subsequent steps in the import process.
NOTE ►► If the proper pivot or reverse pivot is not immediately obvious, it can help to imagine the tables as having been horizontally pivoted (see “Imagining Horizontal Pivots” on page 138 for more information).
NOTE ►► In prior versions of Import Manager, pivoting and reverse pivoting operations were permitted only when the data source was updateable. This limitation is removed as of MDM 7.1.
MDM Import Manager Reference Guide 123
CREATING A PIVOT TABLE
To create a pivot table, you must identify the source fields that participate in the pivot, which ones contain metadata and which ones contain data, which ones must be combined, the one-to-one correspondence between metadata and data fields and/or field combinations, and the key field or fields.
NOTE ►► You use the same generic procedure to create a pivot table regardless of the structure of the source table and the restructuring that must be performed. Specific pivot examples are then illustrated in the subsections that follow.
To create a pivot table:
1. In the drop-down list of source tables, make sure the table you want to pivot is the current source table.
2. In the Source Hierarchy tree, select all the field nodes corresponding to both the metadata fields and the data fields by which you want to pivot.
3. Right-click on one of the nodes and choose Create Pivot Table from the
context menu, or choose Source > Create Pivot Table from the main menu, as shown in Figure 59.
Figure 59. Create Pivot Table context menu command
4. MDM opens the Create Pivot Table dialog shown in Figure 60.
124 MDM Import Manager Reference Guide
Figure 60. Create Pivot Table dialog
5. In the Key Fields dual-list drop-down control, move one or more fields from the Available Fields list to the Selected Fields list to identify the key fields on which to perform the pivot, as shown in Figure 61.
Figure 61. Key Fields dual-list drop-down control
NOTE ►► The Key Fields drop-down control contains the source fields that were not selected as fields by which to pivot.
NOTE ►► For XML, XML Schema, and text sources, the Key Fields
drop-down control includes a [Natural Order] option. This option cannot be combined with other fields.
6. In the dual-list control of fields, drag-and-drop the data fields from the Field Values Become Field Names list to the Field Values Become Field Values list, as shown in Figure 62.
MDM Import Manager Reference Guide 125
Figure 62. Drag-and-drop between field lists
NOTE ►► The Field Values Become Field Names list initially contains the entire set of source fields that were selected in the Source Hierarchy tree as fields by which to pivot.
7. If necessary, select two or more fields in either list that must be
combined into a field combination, and click on the Combine button, or
right-click on one of the fields and choose Combine from the context menu, as shown in Figure 63.
Figure 63. Combine context menu command
8. If necessary, drag-and-drop fields or field combinations within each list to create the one-to-one correspondence between metadata fields and data fields.
NOTE ►► The two lists must contain the same number of distinct items before you can proceed.
9. Click on the Preview button to display a preview of the first ten records of the pivot table, as shown in Figure 64.
126 MDM Import Manager Reference Guide
Figure 64. Pivot preview
NOTE ►► You must perform the Preview before you can proceed.
10. When you have verified that the pivot operation you have defined will
have the desired effect, click OK to close the Create Pivot Table dialog.
11. In the drop-down list of source tables, select the newly created pivot table on which to perform subsequent import processing.
NOTE ►► Pivoting creates additional fields out of field values. MDM automatically converts any characters in the field values that are invalid for field names to valid ones, as follows: period (.) underscore (_); exclamation (!) vertical bar (|); square brackets ([]) angular brackets (<>); accent grave (`) single quote ('); and double quote (") single quote ('). In addition, MDM removes preceding and trailing spaces.
NOTE ►► Various pivoting examples are illustrated in the following subsections.
MDM Import Manager Reference Guide 127
Example: Single Pair of Metadata/Data Fields
Figure 65 shows a source table with a single pair of fields by which to pivot.
Figure 65. Single pair of fields by which to pivot
Figure 66 shows the corresponding field settings and preview in the Create Pivot Table dialog.
Figure 66. Field settings in the Create Pivot Table dialog
128 MDM Import Manager Reference Guide
Example: Metadata Field Combination
Figure 67 shows a source table with a combination of metadata fields by which to pivot.
Figure 67. Metadata field combination
Figure 68 shows the corresponding field settings and preview in the Create Pivot Table dialog.
Figure 68. Field settings in the Create Pivot Table dialog
NOTE ►► DIST and QTY have been combined in the Field Values Become Field Names list.
MDM Import Manager Reference Guide 129
Example: Data Field Combination
Figure 69 shows a source table with a combination of data fields by which to pivot.
Figure 69. Data field combination
Figure 70 shows the corresponding field settings and preview in the Create Pivot Table dialog.
Figure 70. Field settings in the Create Pivot Table dialog
NOTE ►► PART NO and PRICE have been combined in the Field Values Become Field Values list.
130 MDM Import Manager Reference Guide
Example: Single Name/Value Field Pair
Figure 71 shows a source table with a single pair of name/value fields by which to pivot.
Figure 71. Single pair of name/value fields
Figure 72 shows the corresponding field settings and preview in the Create Pivot Table dialog.
Figure 72. Field settings in the Create Pivot Table dialog
NOTE ►► If multiple instances of the same Name appear for each product record (which can represent multi-valued information), all of the values are automatically combined into a single field in the pivot table: (1) in the order the values appear in the source table; and (2) using the Multi-Valued Delimiter to separate individual values.
MDM Import Manager Reference Guide 131
Example: Multiple Name/Value Field Pairs
Figure 73 shows a source table with multiple pairs of name/value fields by which to pivot.
Figure 73. Multiple pairs of name/value fields
Figure 74 shows the corresponding field settings and preview in the Create Pivot Table dialog.
Figure 74. Field settings in the Create Pivot Table dialog
NOTE ►► Each of the NAME fields have been included in the Field Values Become Field Names list and each of the VALUE fields have been included – and arranged in the same order – in the Field Values Become Field Values list.
132 MDM Import Manager Reference Guide
Example: [Natural Order] Pivots
Figure 73 shows a source table with only two fields.
Figure 75. Multiple pairs of name/value fields
Since both fields are needed for the pivot, [Natural Order] must be selected as the key value. This results in a new field, Implicit Key, being added to the table.
Figure 74 shows the corresponding field settings and preview in the Create Pivot Table dialog.
Figure 76. Field settings in the Create Pivot Table dialog
NOTE ►► The [Natural Order] field is new for MDM 7.1; in previous versions you could not pivot a two-field table.
MDM Import Manager Reference Guide 133
CREATING A REVERSE PIVOT TABLE
To create a reverse pivot table, you must identify the source fields that participate in the reverse pivot and the source fields in each set.
NOTE ►► You use the same generic procedure to create a reverse pivot table regardless of the structure of the source table and the restructuring that must be performed. Specific reverse pivot examples are then illustrated in the subsections that follow.
To create a reverse pivot table:
1. In the drop-down list of source tables, make sure the table you want to reverse pivot is the current source table.
2. In the Source Hierarchy tree, select all the field nodes corresponding to the data fields by which you want to reverse pivot.
3. Right-click on one of the nodes and choose Create Reverse Pivot Table
from the context menu, or choose Source > Create Reverse Pivot Table from the main menu, as shown in Figure 77.
Figure 77. Create Reverse Pivot Table context menu command
134 MDM Import Manager Reference Guide
4. MDM opens the Create Reverse Pivot Table dialog shown in Figure 78.
Figure 78. Create Reverse Pivot Table dialog
NOTE ►► The Values Fields list contains the entire set of source fields that were selected in the Source Hierarchy tree as fields by which to reverse pivot.
5. If the fields by which to reverse pivot include multiple sets of data fields, combine one field from each set, so that there is one field or field combination in the list for each metadata value, as shown in Figure 79.
Figure 79. Combine context menu command
6. Click on the Preview button to display a preview of the first ten records of the reverse pivot table, as shown in Figure 80.
MDM Import Manager Reference Guide 135
Figure 80. Reverse pivot preview
NOTE ►► You must perform the Preview before you can proceed.
7. When you have verified that the reverse pivot operation you have
defined will have the desired effect, click OK to close the Create Reverse Pivot Table dialog.
8. In the drop-down list of source tables, select the newly created reverse pivot table on which to perform subsequent import processing.
NOTE ►► Various reverse pivoting examples are illustrated in the following subsections.
Example: Single Set of Data Fields
Figure 81 shows a source table with a single set of data fields by which to reverse pivot.
Figure 81. Single set of fields by which to reverse pivot
136 MDM Import Manager Reference Guide
Figure 82 shows the corresponding field settings and preview in the Create Reverse Pivot Table dialog.
Figure 82. Field settings in the Create Reverse Pivot Table dialog
Example: Metadata Field Combinations
Figure 83 shows a source table where each field by which to reverse pivot represents a combination of metadata values.
Figure 83. Each field represents a combination of metadata values
Figure 84 shows the corresponding field settings and preview in the Create Reverse Pivot Table dialog.
MDM Import Manager Reference Guide 137
Figure 84. Field settings in the Create Reverse Pivot Table dialog
Example: Multiple Sets of Data Fields
Figure 85 shows a source table with multiple sets of data fields by which to reverse pivot.
Figure 85. Multiple sets of fields by which to reverse pivot
Figure 86 shows the corresponding field settings and preview in the Create Reverse Pivot Table dialog.
138 MDM Import Manager Reference Guide
Figure 86. Field settings in the Create Reverse Pivot Table dialog
NOTE ►► One field from each set has been combined in the Value Fields list so that there is one field or field combination for each metadata value.
IMAGINING HORIZONTAL PIVOTS
Recall that creating a pivot table in the MDM Import Manager is like adding horizontal pivots in MDM Publisher, and creating a reverse pivot table is like removing horizontal pivots, a parallel that can be very useful to help you determine how to properly arrange the fields on which to pivot or reverse pivot in the applicable dialog.
Pivoting vs. Adding Horizontal Pivots
Pivots tend to be straightforward, but imagining the horizontal pivot that corresponds to a pivot table is a good warm-up for imagining reverse pivots, which require that you imagine the horizontal pivots on the original source table rather than the pivot table.
Figure 87 repeats the pivot table shown in Figure 39, which was based on a single pair of metadata/data fields.
PART NO 50 250 500
101 $70 $60 $50
102 $80 $70 $60
103 $90 $75 $60
Figure 87. Pivot table with a single pair of metadata/data fields
MDM Import Manager Reference Guide 139
In Figure 88, the pivot table above has been redrawn as a table layout with a horizontal pivot by QTY and a vertical pivot by PART NO, leaving just PRICE beneath the horizontal pivot span.
QTY 50 250 500
PART NO PRICE PRICE PRICE
101 $70 $60 $50
102 $80 $70 $60
103 $90 $75 $60
Figure 88. Alternative table layout with a single horizontal pivot
TIP ►► Pivot table correlation: (1) the vertical pivot becomes the Key Field; (2) the horizontal pivot is placed into the Fields Become Field Names list; and (3) the field beneath the horizontal pivot span is placed into the Fields Become Field Values list.
NOTE ►► The table layout above could be made to look like the pivot table in Figure 87 by setting the column header format in MDM Publisher to swap horizontal pivot values with column headers and hide the column names.
Figure 89 repeats the pivot table shown in Figure 41, which was based on multiple metadata fields.
PART NO M/50 M/250 M/500 G/50 G/250 G/500
101 $70 $60 $50 $68 $58 $48
102 $80 $70 $60 $78 $68 $58
103 $90 $75 $60 $88 $73 $58
Figure 89. Pivot table with multiple metadata fields
In Figure 90, the pivot table above has been redrawn as a table layout with two horizontal pivots by DIST and QTY and a vertical pivot by PART NO, leaving just PRICE beneath the horizontal pivot span.
DIST McMaster Grainger
QTY 50 250 500 50 250 500
PART NO PRICE PRICE PRICE PRICE PRICE PRICE
101 $70 $60 $50 $68 $58 $48
102 $80 $70 $60 $78 $68 $58
103 $90 $75 $60 $88 $73 $58
Figure 90. Alternative table layout with multiple horizontal pivots
140 MDM Import Manager Reference Guide
TIP ►► Pivot table correlation: (1) the vertical pivot becomes the Key Field; (2) the horizontal pivots are placed into and combined in the Fields Become Field Names list; and (3) the field beneath the horizontal pivot span is placed into the Fields Become Field Values list.
Finally, Figure 91 repeats the pivot table shown in Figure 43, which was based on multiple data fields.
SKU P.NO: M P.NO: G P.NO: A PRICE: M PRICE: G PRICE: A
101 M101 G101 A101 $70 $68 $50
102 M102 G102 A102 $80 $78 $80
103 M103 G103 A103 $90 $88 $73
Figure 91. Pivot table with multiple data fields
In Figure 92, the pivot table above has been redrawn as a table layout with a horizontal pivot by DIST and a vertical pivot by SKU, leaving both PART NO and PRICE beneath the horizontal pivot span.
DIST McMaster Grainger Applied
SKU PART NO PRICE PART NO PRICE PART NO PRICE
101 M101 $70 G101 $68 A101 $50
102 M102 $80 G102 $78 A102 $80
103 M103 $90 G103 $88 A103 $73
Figure 92. Alternative table layout with a single horizontal pivot
TIP ►► Pivot table correlation: (1) the vertical pivot becomes the Key Field; (2) the horizontal pivot is placed into the Fields Become Field Names list; and (3) the fields beneath the horizontal pivot span are placed into and combined in the Fields Become Field Values list.
NOTE ►► A pivot table with multiple data fields corresponds to a table layout with multiple fields beneath the horizontal pivot span.
Reverse Pivoting vs. Removing Horizontal Pivots
Reverse pivots with a single set of data fields tend to be relatively straightforward. The challenge occurs when the fields by which to reverse pivot consist of multiple sets of data fields, as illustrated below.
NOTE ►► Imagining reverse pivots requires that you imagine the horizontal pivots on the original source table rather than the pivot table.
Figure 93 repeats the original source table shown in Figure 57.
MDM Import Manager Reference Guide 141
SKU M P.NO M PRICE G P.NO G PRICE A P.NO A PRICE
101 M101 $70 G101 $68 A101 $50
102 M102 $80 G102 $78 A102 $80
103 M103 $90 G103 $88 A103 $73
Figure 93. Original source table with multiple sets of data fields
NOTE ►► The original source table above is effectively the same table as the pivot table in Figure 91, except that the fields above are arranged in pairs by distributor rather than in triplets by type of data.
In Figure 94, the original source table above has been redrawn as a table layout with a horizontal pivot by DIST and a vertical pivot by SKU, leaving both PART NO and PRICE beneath the horizontal pivot span.
DIST McMaster Grainger Applied
SKU PART NO PRICE PART NO PRICE PART NO PRICE
101 M101 $70 G101 $68 A101 $50
102 M102 $80 G102 $78 A102 $80
103 M103 $90 G103 $88 A103 $73
Figure 94. Alternative table layout with a single horizontal pivot
TIP ►► Reverse pivot table correlation: each pair of PART NO and PRICE fields beneath the horizontal pivot span are combined in the Value Fields list for each distributor.
NOTE ►► An original source table with multiple sets of data fields corresponds to a table layout with multiple fields beneath the horizontal pivot span.
By pulling out the implicit metadata value for the distributor from each pair of PART NO and PRICE fields in the original source table, how to proceed with the reverse pivot has become more obvious.
143
PART 7: SPLITTING FIELDS
This part of the reference guide explains how to extract data values that are buried within a field so that you can perform subsequent field-level processing, such as when a field contains a set of delimited values or one or more data values within an unstructured text description.
MDM Import Manager Reference Guide 145
Overview
Sometimes, a single field may contain more than just a single, isolated field value within each record. In these cases, the value (or multiple values) may need to be extracted from the field before you can perform subsequent field-level processing.
Extraction problems – and the corresponding MDM solution – include:
Delimited values represent a hierarchy. If a single field contains a
set of delimited values that implicitly represent the full path of a leaf-
node value in a hierarchy, you can use the Split into Hierarchy command to expand the values into a hierarchy based on the specified delimiter. For example, “Tools > Power Tools > Drills > Accessories” represents the Accessories category for power drills in a four-level hierarchy.
Delimited values belong in multiple fields. If a single field contains
a set of delimited values that actually belong in distinct fields, you
can use the Split into Multiple Fields command to split the values into multiple fields based on the specified delimiter. For example, “Printers; Laser; 1200” represents a product with Category=Printers; Printer Type=Laser; and DPI=1200 dpi.
Each of the operations for extracting values from within a field is described in the following sections, and summarized in Table 14.
Table 14. Splitting Operations
Operation Description
Split into Hierarchy
Splits the data values within the selected field into a hierarchy value.
Split into Multiple Fields
Splits the data values within the selected field into multiple fields.
TIP ►► You can use the Set Split Delimiter commands to set the delimiter strings that the MDM Import Manager uses to split values.
146 MDM Import Manager Reference Guide
Splitting Fields
You can split a field containing multiple delimited values into: (1) a single field containing the values expanded into a hierarchy; or (2) multiple fields, each containing a single value, as described in the following sections.
NOTE ►► Several different mechanisms allow you to interpret the multiple delimited values stored within a single field, each applicable
depending on the level of consistency of the source data: (1) Clone,
when the data values in a single field are either/or; and (2) Split, when the data values are positional and delimited.
NOTE ►► A symmetry exists between splitting a single field and partitioning multiple fields (see “Splitting vs. Partitioning: A Comparison” on page 194 for more information).
SPLITTING INTO HIERARCHY
A source data field may contain delimited values that implicitly represent the full path of a leaf-node value in a hierarchy, as shown in Figure 95.
Source: Corresponding hierarchy:
CATEGORY ► CATEGORY [hierarchy]
Printers; Laser └ Printers
├ Laser
├ Inkjet
│ ├ B&W
│ └ Color
└ Dot Matrix
├ 300 DPI
├ 600 DPI
└ 1200 DPI
Printers; Inkjet; B&W
Printers; Inkjet; Color
Printers; Dot Matrix; 300 DPI
Printers; Dot Matrix; 600 DPI
Printers; Dot Matrix; 1200 DPI
Figure 95. Delimited values and corresponding hierarchy
When you use the Split into Hierarchy command to split a field with multiple delimited values, the MDM Import Manager creates a single new field that contains the delimited values expanded into a hierarchy. You can then perform subsequent processing on the split field just as if it were an original field in the source data. Split fields in the Source Hierarchy tree are highlighted in italics, as shown in Figure 96.
MDM Import Manager Reference Guide 147
Figure 96. Splitting a field into a hierarchy
NOTE ►► The Split into Hierarchy command adds a hierarchy field to the Source Hierarchy tree and to the MDM Import Manager’s virtual workspace, but does not change the structure of the underlying data source.
TIP ►► You can delete a split field using the Delete Field command.
DATA INTEGRITY ►► When expanding into a hierarchy, the MDM Import Manager automatically synthesizes the delimited values across the entire set of source records into a single integrated hierarchy.
DATA INTEGRITY ►► You can specify how to handle trailing NULLs in delimited values based on whether you are importing into a hierarchy lookup field or into the hierarchy table itself. However, regardless of the hierarchy NULL-handling setting, internal NULLs are always preserved as nodes because ignoring them would collapse hierarchy structure.
To split a field with multiple delimited values into a hierarchy:
1. In the Source Hierarchy tree, select the field node whose multiple delimited values you want to expand into a hierarchy.
2. Right-click on the node and choose Split into Hierarchy from the
context menu, or choose Source > Split into Hierarchy from the main menu.
3. MDM adds a new field node named “field <Split Hierarchy>” as a sibling of and just after the original node (Figure 96 above).
4. Press F2 to edit the name of the split field.
5. MDM highlights the name of the split field for editing.
6. Type the name you want for the split field and press Enter.
148 MDM Import Manager Reference Guide
NOTE ►► The Split into Hierarchy command identifies the individual delimited values within a single field using the current Split Hierarchy delimiter string, which must be properly set prior to performing the command (see “Setting the Split Delimiters” on page 150 for more information).
NOTE ►► Regardless of the special NULL handling that occurs later
in the Map Fields/Values tab, the Split into Hierarchy command displays all express and implied source NULLs as nodes in the Source Hierarchy tree. The preserve or ignore NULLs behavior must be properly set prior to mapping the split field, where it is reflected in the values that are displayed in the Source Values grid. (See “The Hierarchy NULL Handling Option” on page 233 for more information about NULL handling in hierarchies).
SPLITTING INTO MULTIPLE FIELDS
A source data field may contain delimited values that that actually belong in distinct fields, as shown in Figure 97.
Source: Corresponding multiple fields:
CATEGORY ► Split <1> Split <2> Split <3>
Printers; Laser Printers Laser
Printers; Inkjet; B&W Printers Inkjet B&W
Printers; Inkjet; Color Printers Inkjet Color
Printers; Dot Matrix; 300 DPI Printers Dot Matrix 300 DPI
Printers; Dot Matrix; 600 DPI Printers Dot Matrix 600 DPI
Printers; Dot Matrix; 1200 DPI Printers Dot Matrix 1200 DPI
Figure 97. Delimited values and corresponding multiple fields
When you use the Split into Multiple Fields command to split a field with multiple delimited values, the MDM Import Manager creates n new fields, (where ‘n’ is the maximum number of delimited values in a single field across the entire set of source records, and the nth field contains the nth delimited value in each record). You can then perform
subsequent processing on the split fields just as if they were original fields in the source data. Split fields in the Source Hierarchy tree are highlighted in italics, as shown in Figure 98.
MDM Import Manager Reference Guide 149
Figure 98. Splitting a field into multiple fields
NOTE ►► Sometimes multiple delimited values do not belong in distinct fields. Rather, they represent multiple values for a single field (e.g. “110V; 115V; 220V; 230V”), in which case no splitting is necessary and the delimited field should instead be mapped to an MDM multi-valued field or attribute (see “Mapping to Multi-Valued Destination Fields” on page 215 for more information).
TIP ►► You can delete split fields using the Delete Field command.
To split a field with multiple delimited values into multiple fields:
1. In the Source Hierarchy tree, select the field node whose multiple delimited values you want to split into multiple fields.
2. Right-click on the node and choose Split into Multiple Fields from the
context menu, or choose Source > Split into Multiple Fields from the main menu.
3. MDM adds one or more new field nodes named “field <Split Multiple> <m>” (where ‘m’ ranges from 1 to the number of new fields) as a sibling of and just after the original node (Figure 98 above).
4. Press F2 to edit the name of each split field.
5. MDM highlights the name of the split field for editing.
6. Type the name you want for the split field and press Enter.
NOTE ►► The Split into Multiple Fields command identifies the individual delimited values within a single field using the current Split Multiple Fields delimiter string, which must be properly set prior to performing the command (see “Setting the Split Delimiters” on page 150 for more information).
150 MDM Import Manager Reference Guide
SETTING THE SPLIT DELIMITERS
The current split delimiters determine how the Split into Hierarchy and
Split into Multiple Fields commands identify the individual delimited values within a single field, and must be properly set prior to performing the commands.
The two split delimiters include:
Split Hierarchy delimiter. The Split into Hierarchy command
identifies the individual delimited values using the current Split Hierarchy delimiter string.
Split Multiple Fields delimiter. The Split into Multiple Fields
command the individual delimited values using the current Split Multiple Fields delimiter string.
Specifically, the commands use as the current delimiter either: (1) the global default delimiter string, if the field is set to inherit; or (2) the field-specific delimiter string, if it has been manually set for the field.
You can set the global default delimiter string for all fields that inherit the
default using the Configuration Options command, or alternatively,
override the default and set it on a field-by-field basis using the Set Split
Delimiter commands, as described in this section.
NOTE ►► Inherited global default values appear in the Source Fields grid in gray.
TIP ►► Be careful not to include leading or trailing spaces in the delimiter string unless your intent is to distinguish between occurrences of the delimiter with and without surrounding spaces (e.g. “X;Y” represents a single value but “X ; Y” represents two values). MDM will automatically trim the leading and trailing spaces from individual delimited values even if you do not include the spaces.
NOTE ►► See “Configuration Options” on page 62 for more information about the global default delimiter strings.
To set the Split Hierarchy or Split Multiple Fields delimiter string for one or more fields:
1. In the Source Hierarchy tree, select the field nodes for which you want to set the Split Hierarchy or Split Multiple Fields delimiter string.
2. Right-click on one of the nodes and choose Set Split Delimiter from the
context menu, or choose Fields > Set Value Delimiter from the main menu, and then choose from the cascading menu:
Split Hierarchy Split Multiple Fields
MDM Import Manager Reference Guide 151
3. MDM opens a Set Delimiter dialog like the one shown in Figure 99.
Figure 99. Set Delimiter dialog
4. MDM displays the current value for the delimiter string and whether or not the global default is being inherited.
NOTE ►► No value is displayed in the edit control if you have selected multiple fields for which the current values are not all the
same. Similarly, a gray check () appears in the Inherit Default checkbox if some but not all of the selected fields are inheriting the global default.
5. To set the delimiter string and override the global default value, enter the new delimiter string into the edit control, or alternatively, to restore
inheritance, check the Inherit Default checkbox.
6. Click OK when you are done to close the dialog.
153
PART 8: CLASSIFICATION DATA IMPORT
This part of the reference guide explains how to import classifications and attribute data; specifically, how to build a hierarchy from a set of fields containing flattened hierarchy data, so that you can perform subsequent field-level processing on the hierarchy; how to import a complete taxonomy, including its hierarchy, attribute definitions, and attribute linkages; and how to import attribute data that appears in the source data as name/value pairs.
MDM Import Manager Reference Guide 155
Overview
Sometimes, source data may comprise a classification scheme consisting of just a classification hierarchy or a complete taxonomy, or may comprise the attribute data itself, as follows:
Classification hierarchy. A set of fields represents a hierarchy that
has been flattened in various ways, and which must be rebuilt into the hierarchy prior to import into MDM.
Complete taxonomy. A set of tables represents an entire taxonomy,
consisting of the hierarchy itself, the attribute definitions, and the node-specific attribute linkages.
Attribute data. The source table represents attribute data in a pair of
fields that contain attribute names and attribute values as name/value pairs.
The operation for creating a hierarchy from a set of fields is described in the following section, and summarized in Table 15.
Table 15. Build Hierarchy Operation
Operation Description
Create Hierarchy Field Creates a new field based on the selected fields and then splits it into a hierarchy.
NOTE ►► See “Taxonomy Import” on page 161 for information on the sequence of operations for importing a complete taxonomy.
NOTE ►► See “Attribute Data Import as Name/Value Pairs” on page 167 for information on importing attribute data as name/value pairs.
156 MDM Import Manager Reference Guide
Building a Hierarchy
Source data fields may represent a hierarchy that has been flattened in various ways, as summarized in Table 16 and further described in the following sections.
Table 16. Flattened Hierarchy Representations
Type # Fields Source Fields Description
Parent/Child Value Pairs w/ Node Name
3 Parent node value Child node value Child node name
Pair of fields contain parent/child value pairs; third field contains node name.
Parent/Child
Value Pairs w/o Node Name
2 Parent node value Child node value
Pair of fields contain
parent/child value pairs; child value contains node name.
Hierarchy Codes w/ Node Name
2 Hierarchy node code Node name
Single field contains the encoded hierarchy; second field contains node name.
NOTE ►► Each record in the data source identifies a distinct internal or leaf node in the hierarchy.
PARENT/CHILD VALUE PAIRS W/ NODE NAME
A pair of source data fields may contain parent/child value pairs, with a third field indicating the node name, as shown in Figure 100.
Source: Corresponding hierarchy:
PARENT CATEGORY NODE NAME ► CATEGORY [hierarchy]
001 Printers └ Printers
├ Laser
├ Inkjet
│ ├ B&W
│ └ Color
└ Dot Matrix
├ 300 DPI
├ 600 DPI
└ 1200 DPI
001 002 Laser
001 003 Inkjet
003 004 B&W
003 005 Color
001 006 Dot Matrix
006 007 300 DPI
006 008 600 DPI
006 009 1200 DPI
Figure 100. Parent/child value pairs with a third name field
NOTE ►► In the illustration above, the parent/child values are codes rather than node names.
MDM Import Manager Reference Guide 157
PARENT/CHILD VALUE PAIRS W/O NODE NAME
A pair of source data fields may contain parent/child value pairs, with the child value indicating the node name, as shown in Figure 101.
Source: Corresponding hierarchy:
PARENT CATEGORY ► CATEGORY [hierarchy]
Printers └ Printers
├ Laser
├ Inkjet
│ ├ B&W
│ └ Color
└ Dot Matrix
├ 300 DPI
├ 600 DPI
└ 1200 DPI
Printers Laser
Printers Inkjet
Inkjet B&W
Inkjet Color
Printers Dot Matrix
Dot Matrix 300 DPI
Dot Matrix 600 DPI
Dot Matrix 1200 DPI
Figure 101. Parent/child value pairs
HIERARCHY CODES W/ NODE NAME
A single field may contain codes where the parent/child hierarchy relationship is encoded within the class code itself, with a second field indicating the node name, as shown in Figure 102.
Source: Corresponding hierarchy:
CODE CATEGORY ► CATEGORY [hierarchy]
17000000 Printers └ Printers
├ Laser
├ Inkjet
│ ├ B&W
│ └ Color
└ Dot Matrix
├ 300 DPI
├ 600 DPI
└ 1200 DPI
17010000 Laser
17020000 Inkjet
17020100 B&W
17020200 Color
17030000 Dot Matrix
17030100 300 DPI
17030200 600 DPI
17030300 1200 DPI
Figure 102. Single coded value with a second name field
158 MDM Import Manager Reference Guide
THE CREATE HIERARCHY FIELD COMMAND
Regardless of how the hierarchy is represented in the source data, you
can use the Create Hierarchy Field command to recreate the hierarchy.
When you use the Create Hierarchy Field command to convert the parent/child value pairs into a hierarchy, the MDM Import Manager creates two new fields: (1) a field that contains the full path of the node indicated by the corresponding record, as a set of delimited values; and (2) a field with the delimited values of the first new field automatically
expanded by MDM into a hierarchy using the Split into Hierarchy command, as shown in Figure 103
Source: Two new fields:
PARENT CATEGORY ► HIERARCHY [delimited] HIERARCHY [hierarchy]
Printers Printers └ Printers
├ Laser
├ Inkjet
│ ├ B&W
│ └ Color
└ Dot Matrix
├ 300 DPI
├ 600 DPI
└ 1200 DPI
Printers Laser Printers\Laser
Printers Inkjet Printers\Inkjet
Inkjet B&W Printers\Inkjet\B&W
Inkjet Color Printers\Inkjet\Color
Printers Dot Matrix Printers\Dot Matrix
Dot Matrix 300 DPI Printers\Dot Matrix\300 DPI
Dot Matrix 600 DPI Printers\Dot Matrix\600 DPI
Dot Matrix 1200 DPI Printers\Dot Matrix\1200 DPI
Figure 103. Two new fields of the Create Hierarchy Field command
You can then perform subsequent processing on the split field just as if it were an original field in the source data. The two new fields are added to the Source Hierarchy tree, with the second one highlighted in italics (to indicate that it can be deleted) as shown in Figure 104.
Figure 104. Creating a hierarchy field from a pair of fields
NOTE ►► In prior versions of Import Manager, creating a hierarchy field was permitted only when the data source was updateable. This limitation is removed as of MDM 7.1.
MDM Import Manager Reference Guide 159
TIP ►► You can delete the split hierarchy field using the Delete Field command.
To combine two or three fields containing hierarchy data into a hierarchy:
1. In the Source Hierarchy tree, select the two or three field nodes you want to use build into a hierarchy:
Parent/child value pairs with node name – three
Parent/child value pairs w/o node name – two Hierarchy codes with node name – two
2. Right-click on one of the nodes and choose Create Hierarchy Field from
the context menu, or choose Source > Create Hierarchy Field from the main menu, as shown in Figure 105.
Figure 105. Create Hierarchy Field context menu command
3. MDM opens the Create Hierarchy Field dialog shown in Figure 106.
160 MDM Import Manager Reference Guide
Figure 106. Create Hierarchy Field dialog
4. In the Hierarchy Field Name edit control, type the name for the new field.
5. In the Parent Field, Child Field, and Node Name Field drop-down controls, identify the field containing the parent value, the child value, and the node name value respectively.
TIP ►► For two-node selection of parent/child value pairs without a
node name, set the Node Name Field to None. MDM uses the values of
Child Field as the name of each node.
TIP ►► For two-node selection of hierarchy codes with a node name,
set the Parent Field to None. MDM automatically decodes the values of Child Field to determine the parent/child hierarchy relationships.
6. Click OK to close the Create Hierarchy Field dialog.
7. The MDM Import Manager creates a hierarchy field named “field <Split Hierarchy>” as a sibling of and just after the first new node (Figure 104 above).
8. Press F2 to edit the name of the hierarchy field.
9. MDM highlights the name of the hierarchy field for editing.
10. Type the name you want for the hierarchy field and press Enter.
NOTE ►► The Create Hierarchy Field command uses the current Split Hierarchy delimiter string to delimit the nodes in the new hierarchy field and then to split the individual delimited values within the new field (see “Setting the Split Delimiters” on page 150 for more information).
MDM Import Manager Reference Guide 161
Taxonomy Import
Source data tables may represent an entire taxonomy, consisting of the hierarchy itself, attribute definitions, and node-specific attribute links.
Importing a complete taxonomy – independently of main table records – is actually a multi-step operation that imports from multiple source tables into multiple destination tables, one at a time as follows:
1. Import the hierarchy into the taxonomy table.
2. Import the attributes into the attributes table.
3. Import the attribute text values into the attribute text values table.
4. Import the node-specific attribute links into the attribute links table.
TIP ►► Building the hierarchy itself is described in the previous section (see “Building a Hierarchy on page 156 for more information on
how to build the hierarchy using the Create Hierarchy Field command).
When you import a taxonomy, you import the hierarchy and the hierarchy records into the taxonomy table, and then you import the other elements of the taxonomy into three “virtual” destination tables related to the taxonomy, as summarized in Table 17.
Table 17. Taxonomy-Related Destination Tables
Table Name Description
taxonomy table The taxonomy table itself.
taxonomy table [Attributes] The related table of attribute definitions.
taxonomy table [Text Values] The related table of attribute text values.
taxonomy table [Links] The related table of node-specific attribute links.
NOTE ►► For each taxonomy table, the drop-down list of destination tables includes the taxonomy table in the first group of items and the three taxonomy-related tables in the last group (see “The Current Tables” on page 36 and “Selecting the Current Tables” on page 50).
TIP ►► In the following sections, set the Expand Multilingual Fields
configuration option to the As Multiple Fields or As Field and Language settings to import multilingual data.
162 MDM Import Manager Reference Guide
IMPORTING THE HIERARCHY
Assuming you have already created the hierarchy using the Create Hierarchy Field command, the first step of importing the taxonomy is to import the hierarchy into the taxonomy table, as described in this section.
To import the hierarchy into the taxonomy table:
1. Set the Source Table to the table that contains the hierarchy records
(and the hierarchy field you created using the Create Hierarchy Field command).
2. Set the Destination Table to the taxonomy table.
3. Make the Map Fields/Values tab the active tab.
4. In the Source Fields and Destination Fields grids, map the hierarchy field to the display field of the taxonomy table by selecting the fields
and clicking on the Map button.
5. In the Source Values and Destination Values grids, add the hierarchy to the taxonomy table by selecting: (1) all of the nodes one level under the root of the source hierarchy; and (2) the applicable node of the
destination hierarchy, clicking on the Add button, and choosing Add
Branch as Child from the cascading menu.
NOTE ►► For (1) The Add button will be enabled only when the source hierarchy is opened to only one level under the root (no further levels are seen); then you select all the nodes. Import Manager will then automap existing nodes and add new nodes in their respective place of the destination hierarchy
NOTE ►► If you want to save the map to be used by MDIS, then you should map only the field name and not any value. By using the Set MDIS Unmapped Value Handling command you will tell what to do with unmapped values and where to locate the root of the hierarchy of the source.
6. Map the other fields and values as necessary.
7. Make the Match Records tab the active tab.
8. In the Value Matching grids, perform record matching on the display field of the taxonomy table.
9. Set the import actions.
10. Perform the import.
11. Select File > Close from the main menu to prepare for the next step.
MDM Import Manager Reference Guide 163
IMPORTING THE ATTRIBUTES
The second step of importing the taxonomy is to import the attribute
definitions into the [Attributes] table, as described in this section.
NOTE ►► The [Attributes] table fields are summarized in Table 18.
To import the attribute definitions into the [Attributes] table:
1. Set the Source Table to the table that contains the attribute definitions.
2. Set the Destination Table to the [Attributes] table.
3. Make the Map Fields/Values tab the active tab.
4. In the Source Fields and Destination Fields grids, either: (1) map the
attribute name to the Name field; or (2) map the remote key value to
the Alias field.
TIP ►► You can use the Alias field to store remote key values since MDM supports key mapping for attribute text values but not for the attributes themselves.
5. In the Source Fields and Destination Fields grids, either: (1) map the
attribute dimension to the Dimension field; or (2) map the attribute
default unit to the Default Unit field.
NOTE ►► If you map: (1) to the Default Unit field, the MDM Import Manager automatically determines the applicable dimension for each
attribute; (2) to the Dimension field, the MDM Import Manager does not set a default unit; or (3) to both, the MDM Import Manager ignores the
Dimension mapping.
NOTE ►► To facilitate organization of the units across all of the
dimensions, the Default Unit values are a hierarchy of dimensions and the units for each dimension.
TIP ►► You can partition the attribute dimension by attribute default unit in the source data to create a hierarchy of dimensions and units.
NOTE ►► You must map manually dimensions or units one-by-one.
6. Map the other fields and values as necessary.
7. Make the Match Records tab the active tab.
8. In the Value Matching grids, perform record matching on the Name
field or the Alias field, as applicable.
9. Set the import actions and perform the import.
10. MDM imports the attribute definitions.
11. Select File > Close from the main menu to prepare for the next step.
164 MDM Import Manager Reference Guide
Table 18. [Attributes] Table Fields
Field Type Description
Name Text (multilingual) The name.
Type Lookup [Flat] The type (Text / Numeric / Coupled Numeric).
Definition Text (multilingual) The definition.
Alias Text The alias.
Multi-Valued Boolean Whether multi-valued (No/Yes)?
Decimal Places Integer The decimal places (3).
Show Fractions Boolean Whether show fractions (No/Yes).
Dimension Lookup [Flat] The dimension (list of dimensions).
Default Unit Lookup [Hierarchy] The default unit (hierarchy of units).
Coupled Name Text (multilingual) The coupled name.
Coupled Decimal Places
Integer The coupled decimal places (3).
Coupled Show Fractions
Boolean Whether coupled show fractions (No/Yes).
Coupled Dimension Lookup [Flat] The coupled dimension (list of dimensions).
Coupled Default Unit Lookup [Hierarchy] The coupled default unit.
Coupled Delimiter Text The coupled delimiter (@).
Ratings Lookup [Flat] (m/v) The ratings (Nom/Min/Max/Avg/Typ) (Nom).
Key Mapping Boolean Whether key mapping (No/Yes)?
[Remote Key] Key Mapping
The remote key field for the attribute name.
Can be used as a record matching field. Its values will be imported as attribute remote key values.
* Where applicable, the default value appears in bold.
MDM Import Manager Reference Guide 165
IMPORTING THE ATTRIBUTE TEXT VALUES
The third step of importing the taxonomy is to import the attribute text
values into the [Text Values] table, as described in this section.
NOTE ►► The [Text Values] table fields are summarized in Table 19.
To import the attribute text values into the [Text Values] table:
1. Set the Source Table to the table that contains the attribute text values.
2. Set the Destination Table to the [Text Values] table.
3. Make the Map Fields/Values tab the active tab.
4. In the Source Fields and Destination Fields grids, either: (1) map the
attribute name to the Attribute Name field; or (2) map the remote key
value to the Attribute Alias field.
TIP ►► You can use the Attribute Alias field to store remote key values since MDM supports key mapping for attribute text values but not for the attributes themselves.
5. In the Source Fields and Destination Fields grids, either: (1) map the
text value to the Text Value field; or (2) map the text value remote key
value to the [Text Value Remote Key] field.
NOTE ►► MDM supports key mapping for attribute text values using
the [Text Value Remote Key] field.
6. Map the other fields and values as necessary.
7. Make the Match Records tab the active tab.
8. In the Value Matching grids, perform record matching on the Attribute
Name field or the Attribute Alias field, as applicable.
9. Set the import actions and perform the import.
NOTE ►► You cannot create new attributes during this step and so
the None/None row in the Import Actions grid is disabled.
10. MDM matches each attribute and updates its set of text values.
11. Select File > Close from the main menu to prepare for the next step.
Table 19. [Text Values] Table Fields
Field Type Description
Attribute Name Text The name of the attribute to which it applies.
Attribute Alias Text The alias of the attribute to which it applies.
Text Value Text (multilingual) The attribute text value.
[Text Value Remote Key] Key Mapping The remote key value of the text value.
[Attribute Remote Key] Key Mapping The remote key value of the attribute name. Can be used as a record matching field.
166 MDM Import Manager Reference Guide
IMPORTING ATTRIBUTE LINKS
The fourth and final step of importing the taxonomy is to import the
node-specific attribute links into the [Links] table.
Table 20. [Links] Table Fields
Field Type Description
Attribute Name Text The name of the attribute to which it applies.
Attribute Alias Text The alias of the attribute to which it applies.
Node Name Lookup [Hierarchy] The name of the linked node (taxonomy hierarchy).
Priority Lookup [Flat] The custom integer value or predefined priority value
(Highest / High / Normal / Low / Lowest).
Link Type Lookup [Flat] The type of link (Ordinary / Master / Variant).
[Attribute Remote Key]
Key Mapping The remote key value of the attribute name. Can be used as a record matching field.
TIP ►► You can configure Import Manager to accept integer values in
the Priority field rather than predefined values (Highest / High / Normal / Low / Lowest) by setting the configuration option "Display priority field
for Attribute taxonomy links table using" to "custom values."
To import the node-specific attribute links into the [Links] table:
1. Set the Source Table to the table that contains the attribute links.
2. Set the Destination Table to the [Links] table.
3. Make the Map Fields/Values tab the active tab.
4. In the Source Fields and Destination Fields grids, either: (1) map the
attribute name to the Attribute Name field; or (2) map the remote key
value to the Attribute Alias field.
TIP ►► You can use the Attribute Alias field to store remote key values since MDM supports key mapping for attribute text values but not for the attributes themselves.
5. Map the other fields and values as necessary.
6. Make the Match Records tab the active tab.
7. In the Value Matching grids, perform record matching on the Attribute
Name field, the Attribute Alias field, and the Node Name field.
TIP ►► You can also combine either the Attribute Name or Attribute
Alias field with the Node Name field.
8. Set the import actions and perform the import.
NOTE ►► You cannot create new attributes during this step and so
the None/None row in the Import Actions grid is disabled.
9. MDM matches each attribute and updates its set of node-specific attribute links.
MDM Import Manager Reference Guide 167
Attribute Data Import as Name/Value Pairs
Recall from the section on pivoting that field and attribute data may be stacked “vertically” in the source table, with a pair of source fields representing name/value pairs and each product record duplicated for each name/value pair, and specifically, that when source data is structured this way, you can use a pivot to “rotate” the source data so that each field or attribute becomes a field in the source table.
NOTE ►► See “Pivoting” on page 114 for more information on pivoting name/value pairs.
However, attribute data that is stacked vertically as name/value pairs can also be imported directly without pivots by first partitioning the name and value source fields, and then mapping the resulting name/value hierarchy to a special destination field containing a name/value hierarchy of attribute names and data values.
Consider the table shown in Figure 107, which contains three fields, PART NO, which stores the part number, and NAME and VALUE, which store attribute name and value information for each source record.
PART NO NAME VALUE
101 Cable Length 10 in
101 Color Blue
101 Color Red
101 Color Green
102 Cable Length 5 in
102 Color Green
103 Cable Length 5 in
103 Color Blue
Figure 107. Attribute data as name/value pairs
TIP ►► Remember to partition the name field by the value field, to create a two-level hierarchy of attribute names and values.
This alternative approach allows you to avoid the tedious manual task of mapping at the field level, by turning mapping into a value mapping process. More importantly, by taking advantage of the automatic value mapping and exception handling features of MDIS, this approach also supports full MDIS automation of attribute data import, which would not otherwise be possible with a variable set of attributes.
NOTE ►► You must use this alternative approach when the number of attributes represented by the name/value pairs is too large to pivot (because it would result in a very wide table with more than 255 fields).
168 MDM Import Manager Reference Guide
THE ATTRIBUTE DISPLAY CONFIGURATION OPTION
In order to import the two-level hierarchy of attribute names and values that results from partitioning the name and value source fields, the destination attributes must appear in the Destination Fields grid as a single combined hierarchy item rather than a list of individual items.
You can use the Attribute Display configuration option to specify how to display attributes and attribute values in the Destination Fields and Destination Values grids of the Map Fields/Values tab, respectively, as summarized in Table 21.
Table 21. Attribute Display Configuration Option
Setting Description
As Multiple Attributes Display each attribute as a separate item with its corresponding list of attribute data values.
As Attribute/Value Hierarchy
Display a single item for all attributes with a two-level hierarchy of attributes and data values.
THE [ATTRIBUTE/VALUE] HIERARCHY FIELD
The single attribute/value hierarchy item in the Destination Fields grid of
the Map Fields/Values tab is named [Attribute/Value] Hierarchy and is
of type Attributes, as shown in Figure 108.
Figure 108. [Attribute/Value] Hierarchy field in Destination Fields grid
NOTE ►► If there are multiple taxonomy lookup fields, the name of the taxonomy lookup field is appended in angular brackets (<>).
NOTE ►► You can add new text attribute text values but you cannot add new attributes using this approach; to add a new attribute use the
Add attribute button before mapping to the destination field.
NOTE ►► The Set Value Conversion Filter command is disabled at the field level since: (1) source values can be text; and (2) numeric source values can be of different measurement types.
MDM Import Manager Reference Guide 169
THE ATTRIBUTE/VALUE HIERARCHY
The attribute/value hierarchy of attribute names and values appears in the Destination Values grid of the Map Fields/Values tab, as shown in Figure 109, and subject to the constraints summarized in Table 22.
Figure 109. Attribute/value hierarchy in Destination Values grid
Table 22. Attribute/Value Hierarchy of Names and Values
Item Description
Name/Value Hierarchy
Attributes displayed as a two-level hierarchy of names and values
Attribute names are at the first level, values at the second level
Text values of a text attribute appear as its children
Numeric values of a numeric attribute do not appear in the tree.
Automap Button
Finds and maps the matching attribute name
Finds and maps the matching text values of a text attribute
Converts the numeric values of a numeric attribute
Map Button
Attribute name can be mapped only to attribute name in a one-to-one mapping
Text values can be mapped to text values only if the text attribute name is already mapped and only to values of the same attribute (one-to-one, many-to-one, or one-to-many if multi-valued)
Numeric values are automatically converted when the numeric attribute name is mapped
Add Button Text values can be added only if the text attribute name is already
mapped and only to the same attribute
Unmap Button
All text attribute values automatically unmapped when the text attribute name is unmapped
All numeric attribute values will be deconverted back to text when the numeric attribute name is unmapped
Value Conversion Filters
Disabled at the field level
Enabled or disabled at value level based on selected source row
171
PART 9: PARTITIONING FIELDS/VALUES
This part of the reference guide explains the partitioning mechanism of the MDM Import Manager, and how to use it to effortlessly import source data into a destination repository even if the source and destination structures would otherwise be incompatible.
MDM Import Manager Reference Guide 173
Overview
Traditional import tools typically offer a field mapping capability that allows you to map an individual source field to an individual destination field, and then to import the data values from the source field directly into the destination field.
However, it often may not be possible to directly map individual source fields to individual destination fields – or to map, convert, or create measurements from the underlying values within those individual source and destination fields – because there may not be a direct one-to-one relationship between individual source and destination fields.
Instead, the appropriate field mapping relationship may exist between the combination of two or more source fields and/or the combination of two or more destination fields and/or attributes (such as when the CATEGORY and SUBCATEGORY source fields together correspond to the Category destination field, or alternatively, the CATEGORY source field corresponds to the Category destination field together with the Printer Type and DPI attributes). In these cases, fields must be combined to allow the mapping of field combinations rather than individual fields; meanwhile, combining fields into field combinations simultaneously combines the underlying values into value combinations, to allow the value-level mapping, conversion, and measurement creation of value combinations rather than individual values.
PARTITIONING AND FIELD MAPPING
To deal with the challenge of different source and destination field structures, MDM’s generalized partitioning mechanism allows you to combine two or more fields into field combinations and underlying values into a corresponding set of value combinations.
In each case, you create field combinations by partitioning either an entire field or just an individual value by another field or attribute. And multiple partitions can be nested (creating hierarchy among the value combinations) or combined (merging the value combinations).
Partitioning is one of the most powerful features of the MDM Import Manager. It is the mechanism that allows you to arbitrarily restructure electronic source data and rationalize it against destination data that has an entirely different structure.
NOTE ►► Partitioning is part of the process of rationalizing and preparing source data for import prior to the next transformation step comprising field mapping, value mapping, value conversion, and measurement value creation, all of which are fully described in the next part of this guide.
174 MDM Import Manager Reference Guide
Source Field Partitioning
Sometimes, multiple source fields may correspond to a single destination field.
Consider Figure 110, in which the individual source field CATEGORY (which contains a single Printers category) does not contain sufficient information to be mapped by itself to the more detailed destination field Category (which contains three corresponding printer categories: Laser Printers, Inkjet Printers, and Dot Matrix Printers).
Source: Destination:
CATEGORY ► Category
Printers Laser Printers
Bearings Inkjet Printers
Motors Dot Matrix Printers
Mounted Bearings
Unmounted Bearings
One-Phase Motors
Two-Phase Motors
Three-Phase Motors
Figure 110. Individual source field missing destination information
Now consider Figure 111, which illustrates that the missing detail exists in the SUBCATEGORY source field, which together with CATEGORY corresponds to destination field Category. In this case, you must combine CATEGORY and SUBCATEGORY with partitioning prior to field mapping, so that they can be field mapped as a combination to Category, and their underlying values mapped as value combinations rather than as individual values.
Source: Destination:
CATEGORY SUBCATEGORY ► Category
Printers Laser Laser Printers
Printers Inkjet Inkjet Printers
Printers Dot Matrix Dot Matrix Printers
Bearings Mounted Mounted Bearings
Bearings Unmounted Unmounted Bearings
Motors One-Phase One-Phase Motors
Motors Two-Phase Two-Phase Motors
Motors Three-Phase Three-Phase Motors
Figure 111. Source fields combined for value mapping
MDM Import Manager Reference Guide 175
NOTE ►► Strictly speaking, if SUBCATEGORY contains only unique values as in the example above, you can ignore CATEGORY and simply field map SUBCATEGORY directly to Category. However, if SUBCATEGORY contains duplicate values, partitioning is essential.
Similarly, in Figure 112, the two source fields FIRST NAME and LAST NAME together correspond to the single destination field Name, and must be merged with partitioning prior to field mapping, so that they can be field mapped as a combination to Name, and their underlying values mapped as merged value combinations rather than as individual values.
Source: Destination:
FIRST NAME LAST NAME ► Name
Joe Williams Joseph Williams
Jim Williams James Williams
Dave Rodgers David Rodgers
Dave Smith David Smith
Figure 112. Source fields combined for value mapping
Even without value mapping, multiple source fields may need to be combined with partitioning prior to field mapping. In Figure 113, the individual numeric source field SOCKET SIZE cannot be mapped by itself to the measurement destination field Socket Size because it is missing the unit of measure for each numeric value.
Source: Destination:
SOCKET SIZE ► Socket Size [length]
3 1 in
5 1.5 in
20 2 in
5 10 mm
2
Figure 113. Individual source field missing unit of measure information
Now consider Figure 114, which illustrates that the missing unit of measure information exists in the SOCKET UNIT source field, which together with SOCKET SIZE comprises the measurement values that are stored in destination field Socket Size. In this case, you must combine SOCKET SIZE and SOCKET UNIT with partitioning prior to field mapping, so that they can be field mapped as a combination to Socket Size, and measurement values created from their underlying merged value combinations.
176 MDM Import Manager Reference Guide
Source: Destination:
SOCKET SIZE SOCKET UNIT ► Socket Size [length]
3 in 1 in
5 in 1.5 in
20 mm 2 in
5 mm 10 mm
2 in
Figure 114. Source fields combined for measurement value creation
Finally, source fields may need to be combined so that the individual values within each field can be treated as the set of multiple delimited values for mapping to a multi-valued field, as shown in Figure 115.
Source: Destination:
SOCKET SIZE1 SOCKET SIZE2 ► Socket Size [length]
1/2 in 3/4 in 1 in; 1 1/2 in
1/4 in 3/8 in 16 mm; 18 mm
20 mm 22 mm 2 in; 2 1/2 in
5 mm 6 mm 10 mm; 12 mm
2 in 2 1/2 in
Figure 115. Source fields combined for mapping to a multi-valued field
Destination Field Partitioning
Just as multiple source fields may correspond to a single destination field, a single source field may correspond to multiple destination fields and attributes.
Consider Figure 116, in which the more detailed source field CATEGORY (which contains seven printer categories) cannot be mapped to the individual destination field Category (which contains only a single Printers category) without losing the additional detail about printers.
MDM Import Manager Reference Guide 177
Source: Destination:
CATEGORY ► Category
300 DPI Laser Printers Printers
600 DPI Laser Printers
1200 DPI Laser Printers
300 DPI Inkjet Printers
600 DPI Inkjet Printers
300 DPI Dot Matrix Printers
600 DPI Dot Matrix Printers
Figure 116. Destination field missing source information
Now consider Figure 117, which illustrates that the additional printer detail is stored in the Printer Type and DPI destination attributes, which together with Category correspond to source field Category. In this case, you must combine Category, Printer Type, and DPI with partitioning prior to field mapping, so that CATEGORY can be field mapped not just to Category by itself but rather to the field and attributes as a combination, and CATEGORY values mapped to their underlying value combinations rather than to their individual values.
Source: Destination:
CATEGORY ► Category Printer Type DPI
300 DPI Laser Printers Printers Laser 300
600 DPI Laser Printers Printers Laser 600
1200 DPI Laser Printers Printers Laser 1200
300 DPI Inkjet Printers Printers Inkjet 300
600 DPI Inkjet Printers Printers Inkjet 600
300 DPI Dot Matrix Printers Printers Dot Matrix 300
600 DPI Dot Matrix Printers Printers Dot Matrix 600
Figure 117. Destination field and attributes combined for value mapping
NOTE ►► If the multiple source values represent inconsistencies in the source data or unnecessary detail that can be eliminated, then partitioning is not necessary and the multiple source values should be collapsed during the value-mapping step. (See “Collapsing Source Value Variations” on page 214 for more information about collapsing source value variations during value mapping.)
178 MDM Import Manager Reference Guide
Source and Destination Field Partitioning
Sometimes, multiple source fields may correspond to multiple destination fields and/or attributes, so that both source and destination fields must be combined with partitioning prior to field mapping.
Consider Figure 118, in which the source field CATEGORY (which contains two lighting categories) cannot be mapped directly to the destination field Category (which also contains two lighting categories) because the categories are broken down completely differently and the source and destination values simply don’t correspond at all.
Source: Destination:
CATEGORY ► Category
Commercial Lighting ≠ Incandescent Lighting
Residential Lighting ≠ Fluorescent Lighting
Commercial Plumbing Brass Plumbing
Residential Plumbing PVC Plumbing
Figure 118. Source and destination fields broken down differently
Now consider Figure 119, which illustrates that CATEGORY, together with the additional information stored in the SUBCATEGORY source field, corresponds to Category, together with the additional information stored in the Type destination attribute. In this case, you must combine CATEGORY and SUBCATEGORY and also combine Category and Type, so that the combination of source fields can be field mapped to the combination of destination fields, and underlying source value combinations mapped to destination value combinations.
Source: Destination:
CATEGORY SUBCATEGORY ► Category Type
Commercial Lighting Incandescent Incandescent Lighting Commercial
Commercial Lighting Fluorescent Fluorescent Lighting Commercial
Residential Lighting Incandescent Incandescent Lighting Residential
Residential Lighting Fluorescent Fluorescent Lighting Residential
Commercial Plumbing Brass Brass Plumbing Commercial
Commercial Plumbing PVC PVC Plumbing Commercial
Residential Plumbing Brass Brass Plumbing Residential
Residential Plumbing PVC PVC Plumbing Residential
Figure 119. Source and destination both combined for value mapping
NOTE ►► You can partition as many source fields and destination fields and attributes as necessary to establish a correspondence between source and destination field and value combinations.
MDM Import Manager Reference Guide 179
Field Value Partitioning
Sometimes, multiple source or destination fields may need to be combined, but unlike in the earlier examples, you must partition individual field values rather than an entire field.
Consider Figure 120, in which the missing CATEGORY info exists not in one but rather in three additional source fields, each of which corresponds to a different CATEGORY value, and which collectively correspond to destination field Category. Here, you must combine value Printers with PRINTER TYPE, Bearings with BEARING TYPE, and Motors with MOTOR TYPE to create the proper value combinations.
Source: Destination:
CATEGORY PRINTER TYPE BEARING TYPE MOTOR TYPE ► Category
Printers Laser ├ Printers
Printers Inkjet │ ├ Laser
Printers Dot Matrix │ ├ Inkjet
Bearings Mounted │ └ Dot Matrix
Bearings Unmounted ├ Bearings
Motors One-Phase │ ├ Mounted
Motors Two-Phase │ └ Unmounted
Motors Three-Phase └ Motors
├ One-Phase
├ Two-Phase
└ Three-Phase
Figure 120. Source values combined with partitioning
Alternatively, consider Figure 121, in which you must combine Category value Printers with Printer Type, Bearings with Bearing Type, and Motors with Motor Type to create the proper value combinations.
Source: Destination:
CATEGORY ► Category Printer Type Bearing Type Motor Type
Laser Printers Printers Laser
Inkjet Printers Printers Inkjet
Dot Matrix Printers Printers Dot Matrix
Daisywheel Printers Printers Daisywheel
Mounted Bearings Bearings Mounted
Unmounted Bearings Bearings Unmounted
One-Phase Motors Motors One-Phase
Two-Phase Motors Motors Two-Phase
Three-Phase Motors Motors Three-Phase
Figure 121. Destination values combined with partitioning
180 MDM Import Manager Reference Guide
DISTINCT VALUE LISTS AND VALUE COMBINATIONS
Recall that both the Source Hierarchy tree and the Destination Hierarchy tree allow you to display distinct value lists for each field. Moreover, each distinct value represents and is a proxy for not only all instances of the value across the entire set of underlying records, but also the subset of records that contain the value.
Partitioning relies upon the concept of distinct value lists and extends it to value combinations. Instead of working with the set of distinct values in a single field, you work with the set of distinct value combinations across the multiple fields that have been combined with partitioning.
Partitioning relies upon and extends distinct value lists as follows:
Field-level partitioning. Partitioning a field by another field partitions each of the distinct values of the partitioned field as if it were directly partitioned.
Value-level partitioning. Partitioning a field value by another field
(either directly or by virtue of partitioning the field) creates the set of value combinations of that value with each of the distinct values of
the partitioning field that actually exist in the underlying records.
NOTE ►► For a field combination created by partitioning, the set of all possible value combinations is collapsed down to the set of those that actually exist in the underlying records.
Subsets of underlying records. Just as each distinct value
represents and is a proxy for the subset of records that contain the value, each distinct value combination represents and is a proxy for the smaller subset of records that contain the value combination, to allow the value-level mapping, conversion, and measurement creation of value combinations rather than individual values.
NOTE ►► Although the set of distinct value combinations will be larger than each individual set of distinct values, each particular value combination will then correspond to a smaller, more specific and more precisely defined subset of underlying records, and thereby permit an accurate value-level mapping that would not otherwise be possible.
TECHNICAL NOTE ►► In mathematical terms, partitioning n fields: (1) generates the n-dimensional Cartesian product of ordered value combinations across the n individual value domains; and (2) removes from the complete set those value combinations that do not exist in the underlying records. Creating this collapsed set of actual value combinations is equivalent to partitioning – in the mathematical sense – the set of underlying records into non-overlapping subsets of records, where each value combination represents a single subset.
Consider the table of records and field values shown in Figure 122.
MDM Import Manager Reference Guide 181
Record Field1 Field2
1 a x
2 c x
3 d z
4 b y
5 c x
6 b x
7 c y
Figure 122. Sample table of records and field values
For the table above, Figure 123 below illustrates the following as it relates to partitioning Field1 by Field2:
Distinct values. The set of distinct values for Field1 and Field2
(Field1 and Field2, respectively).
All possible value combinations. The set of all possible value combinations created by the Cartesian product of Field1 and Field2 (Field1 x Field2).
Actual value combinations. The set of actual value combinations
created by partitioning Field1 by Field2 (Field1 Field2).
Field1 + Field2 ► Field1 x Field2 ► Field1 Field2
a x a x a x
b y a y b x
c z a z b y
d 3 distinct values
b x c x
4 distinct values
b y c y
b z d z
c x Partitioning:
7 actual value combinations
c y
c z
d x
d y
d z
Cartesian product:
12 possible value combinations
Figure 123. Distinct value lists and value combinations
182 MDM Import Manager Reference Guide
RECORD SUBSETS
The partitioning mechanism of the MDM Client has been enhanced and extended within the MDM Import Manager to address the specific requirements for importing source data.
Recall that in the MDM Client, the Partition command in Taxonomy mode splits a category into multiple child categories according to the values of an attribute, thereby partitioning the set of records in the category into multiple subsets corresponding to each new child category.
Similarly, partitioning a family in Family mode splits the family into multiple child families according to the values of a field or attribute, thereby partitioning the set of records in the family into multiple subsets corresponding to each new child family.
In both cases, each subset of records corresponds to a particular value combination represented by: (1) either the value of the original category (in Taxonomy mode) or the value or value combination represented by the original family (in Family mode); and (2) one of the values of the field or attribute by which you partition.
Just as in the MDM Client, partitioning in the MDM Import Manager splits a set of records into smaller subsets (hence the term partitioning), and it does so by creating nodes that correspond to a set of actual value combinations among the partitioned items.
But whereas the emphasis in the MDM Client is on the nodes as a proxy for the underlying subset of records, the emphasis in the MDM Import Manager is instead on the value combinations themselves, to allow the value-level mapping, conversion, and measurement creation of value combinations rather than individual values.
NOTE ►► By default, the set of distinct values for a field automatically partitions the entire set of records into multiple subsets, each corresponding to a distinct value. Partitioning a field by another field further partitions each subset into multiple subsets, each corresponding to an actual value combination.
TECHNICAL NOTE ►► Partitioning has the effect of: (1) horizontally partitioning a single set of records into multiple subsets; by (2) vertically combining multiple fields into a single field combination and the underlying values into value combinations.
MDM Import Manager Reference Guide 183
PARTITIONS IN THE SOURCE AND DESTINATION HIERARCHIES
A partition can be applied in either the Source or Destination Hierarchy tree to: (1) a field node, for inheritance by each value node beneath it in the hierarchy that is not directly partitioned; or (2) a value node (either internal or leaf), overriding the inheritance, if any, from a partitioned node above it in the hierarchy (see the next section for more information about inheritance).
NOTE ►► In the Source Hierarchy tree, you partition a source field or value by another field in the current source table by selecting from the Available Fields list on the left side of the Partition Field/Value tab.
NOTE ►► In the Destination Hierarchy tree, you partition a source lookup field or value by another lookup field or attribute in the current destination table by selecting from the Linked Attributes list on the right side of the Partition Field/Value tab.
Table 23 summarizes how you can partition fields and values.
Table 23. Partitions in the Source and Destination Hierarchies
Partition… By… Description
Source Hierarchy
Field Field Partition each of the values of a field by each of the values of a second field.
Field Value
Field Partition a single field value by each of the values of a second field.
Destination Hierarchy (NYI)
Lookup Field
Lookup Field
Partition each of the values of a lookup field by each of the values of a second lookup field.
Lookup Field Value
Lookup Field
Partition a single lookup field value by each of the values of a second lookup field.
Taxonomy Lookup Field
Attribute Partition each of the values of a taxonomy lookup field by each of the values of an attribute. The partition is applied only to the values of the taxonomy field to which the attribute is linked.
Taxonomy Lookup Field Value
Linked Attribute
Partition a single taxonomy lookup field value by each of the values of an attribute linked to that value.
Linked Attribute
Linked Attribute
Within the existing partition of a taxonomy lookup field value by an attribute, partition each of the values of the attribute by each of the values of a second attribute linked to that lookup field value.
Linked Attribute Value
Linked Attribute
Within the existing partition of a taxonomy lookup field value by an attribute, partition a single value of the attribute by each of the values of a second attribute linked to that lookup field value.
TIP ►► Source partitions are usually applied at the field level, to partition an entire field by another field. Destination partitions are usually applied at the value level, to partition an individual category value by a linked attribute.
184 MDM Import Manager Reference Guide
THE PARTITION FIELD/VALUE TAB
The Partition Field/Value tab displays two pairs of dual-list controls for multiple-item selection, one for partitioning nodes of the Source Hierarchy and one for partitioning nodes of the Destination Hierarchy tree, as shown in Figure 124.
Figure 124. The Partition Field/Value tab
You can add both source and destination partitions using this tab, as follows:
Source partitioning. In the Source Hierarchy tree, you can partition
any field or value by any other field in the current source table by selecting from the Available Fields list on the left side of the tab.
Destination partitioning. In the Destination Hierarchy tree, you can
partition: (1) any lookup field or value by any other lookup field; or (2) any taxonomy lookup field or value by any other lookup field or linked attribute in the current destination table by selecting from the Linked Attributes list on the right side of the tab.
NOTE ►► The lists on the left are disabled if you have not yet selected the current source table. The lists on the right are disabled if you have not yet selected the current destination table.
NOTE ►► When the source is an XML file, the Available Fields list is limited to nodes which are siblings (on the same level as), or children of (nested below) the currently selected Source Hierarchy tree node.
NOTE ►► The name of the selected node always appears as the first item in the Partition By list as “fieldname” (for a field node) or “fieldname [value]” (for a value node) to allow you to combine nodes by which you partition with the partitioned node.
MDM Import Manager Reference Guide 185
FIELD AND VALUE PARTITIONING
When you partition a field node (or an internal value node), the MDM Import Manager applies the partition to each leaf value node beneath it that inherits the partition as if it were directly partitioned.
When you partition a value node (either directly or by virtue of partitioning the field), the MDM Import Manager creates the set of actual value combinations of that value (or value combination if the value node is already the result of a partition) with each of the distinct values of the partitioning field, and adds a child node to the hierarchy corresponding to each value combination, as shown in Figure 125.
Figure 125. Node in Source Hierarchy tree before and after partitioning
NOTE ►► Partitioning a single leaf value node results in a single set of value combinations. Partitioning a field node (or an internal value node) results in a set of value combinations for each leaf value node beneath it that inherits the partition.
NOTE ►► Nodes that are the result of partitioning have a purple icon ( ) to the left of the node name. Nodes that have been directly partitioned (as opposed to having been partitioned through inheritance) have a small purple square ( ) to the left of the node icon.
INHERITANCE
By default, when you partition an internal node (i.e. a node that has at least one child node), each of the child nodes inherits the parent’s
partitions, each of its child nodes inherits the inherited partitions, and so on, until the partitions are inherited by and applied to a leaf node.
NOTE ►► When a node is inheriting partitioning: (1) the Partition By list in the Partition Field/Value tab contains the partition items that are
inherited; and (2) the Inherit checkbox is checked in the Partition Field/Value tab to indicate that the items shown in the Partition By list are inherited.
186 MDM Import Manager Reference Guide
Inheritance of partitions makes defining and refining the set of value combinations extremely efficient. When you partition a field node, the partition is inherited by each of the child value nodes, so that by partitioning the field node, you can partition each of the values with a single command.
When you directly partition any internal node or a leaf value node, the inheritance of partitions from nodes above it in the hierarchy is broken, and the value node is partitioned only by the fields and/or attributes specified directly for that node.
In effect, directly partitioning the node breaks the chain of inheritance from nodes above it in the hierarchy, and the partitions for the node override any inherited partitions.
In this way, inheritance can be overridden for any node by directly partitioning it in the hierarchy. Moreover, any child of the directly partitioned node then inherits the partitions of the node unless the partitioning information of the child node is itself directly partitioned.
NOTE ►► Nodes that have been directly partitioned (as opposed to having been partitioned through inheritance) have a small purple square ( ) to the left of the node icon in the hierarchy tree.
NOTE ►► In addition to the purple square, the Inherit checkbox is not checked in the Partition Field/Value tab for nodes that have been
directly partitioned. To restore inheritance, recheck the Inherit checkbox (see “Restoring Inheritance” on page 203 for more information).
NOTE ►► Directly partitioning an internal node, all of whose children are also directly partitioned, will not result in any nodes being added to the hierarchy, since partitioning is only applied to leaf nodes and the chain of inheritance is broken by each of its children.
MULTIPLE PARTITIONS
The result of a partition is that additional child value nodes are added to the hierarchy.
When you partition a node by more than one field or attribute, each partition is applied sequentially. In other words, the first partition is applied to the leaf value node or to each leaf value node beneath it that inherits the partition to create additional child value nodes, the second partition is then applied to the newly created leaf value nodes beneath it that inherit the partition to create additional child value nodes, and so on, as shown in Figure 126, Figure 127, and Figure 128.
MDM Import Manager Reference Guide 187
Figure 126. Multiple partitions before first partition
Figure 127. Multiple partitions after first partition
Figure 128. Multiple partitions after second partition
NESTED AND COMBINED PARTITIONS
By default, each partition that you add for a particular node in the hierarchy is nested (i.e. applied sequentially) as described in the previous section. Nested partitions add multiple levels of hierarchy, with a level of child nodes created for the set of values in each of the nested partitions.
Partitions can also be combined (i.e. applied as a group). Combined partitions add a single level of hierarchy, with child nodes created for each of the merged value combinations of the combined partitions.
A nested partition displays each value of each value combination as a distinct node in the hierarchy, with the field nodes represented as sibling
188 MDM Import Manager Reference Guide
nodes at the top level of the hierarchy, the values of each field node represented as sibling nodes at a second level in the hierarchy as children of the first set of sibling nodes, the values of the first partitioning field or attribute represented as sibling nodes at a third level in the hierarchy, and so on for each field or attribute participating in the partition, as shown in Figure 129.
Figure 129. Nested partitions (Brush Diameter Brush Diameter Unit)
NOTE ►► Reordering nested partitions changes the hierarchy structure created by the multiple partitions but results in the same set of leaf nodes.
By contrast, a combined partition displays each value combination consisting of the individual values of the field node and each of the fields or attributes participating in the combined partition as a single merged value in a single node in a hierarchy, with the entire set of value combinations represented as sibling nodes at a single level in the hierarchy, as shown in Figure 130.
Figure 130. Combined partitions (Brush Diameter Brush Diameter Unit)
NOTE ►► The name of the selected node always appears as the first item in the Partition By list as “fieldname” (for a field node) or “fieldname [value]” (for a value node) to allow you to combine nodes by which you partition with the partitioned node.
MDM Import Manager Reference Guide 189
Consider the following table of original records that contains values for the fields CITY, STATE, and ZIP:
Original records:
CITY STATE ZIP
Los Angeles CA 90046
Los Angeles CA 90025
Hollywood CA 90069
Hollywood CA 90036
Miami Beach FL 33139
Hollywood FL 33019
Then the nested partition of STATE by CITY would be:
Nested:
STATE CITY
├ CA
│ ├ Los Angeles
│ └ Hollywood
└ FL
├ Miami Beach
└ Hollywood
… and the combined partition of STATE by CITY would be:
Combined:
STATE CITY
CA; Los Angeles
CA; Hollywood
FL; Miami Beach
FL; Hollywood
Nested vs. Combined
Not coincidentally, the value combination represented by each bottom-level leaf node in a nested partition corresponds to the value combination represented by each top-level sibling node in a combined partition, and also to precisely the same subset of records, as follows:
190 MDM Import Manager Reference Guide
Original records: Nested: Combined:
CITY STATE ZIP ► STATE CITY ► STATE CITY
Los Angeles CA 90046
► CA
└ Los Angeles ► CA; Los Angeles
Los Angeles CA 90025
Hollywood CA 90069
► CA
└ Hollywood ► CA; Hollywood
Hollywood CA 90036
Miami Beach FL 33139 ► FL
└ Miami Beach ► FL; Miami Beach
Hollywood FL 33019 ► FL
└ Hollywood ► FL; Hollywood
NOTE ►► Combining partitions results in fewer internal nodes but exactly the same set of leaf nodes being added to the hierarchy, so that the hierarchy has less depth and fewer levels than if the partitions were not combined.
Nested and combined partitions would thus appear to be equivalent and interchangeable, and in some ways, they are. However, there are subtle differences between nested and combined partitions, as illustrated above and further described below:
Node values. Even though the bottom-level leaf nodes of a nested
partition correspond to the top-level siblings of a combined partition, and both correspond to the same value combinations and subset of records, the node values themselves are very different. In a nested partition, the value of the bottom-level leaf node is the value of the last partition field, whereas in a combined partition, the value of the top-level sibling node is the merged value of all the partition fields.
Hierarchy. A nested partition creates the entire hierarchy
represented by the value combinations of the partition, along with all associated internal nodes, while a combined partition flattens the hierarchy and does not create any internal nodes.
Number of nodes for each field value. A nested partition creates a
single node for each value of the first partitioned field, with a child for each value of the first field or attribute by which you partition, “grandchild” nodes for each value of the second partition, and so on for each partition, whereas a combined partition creates multiple nodes for each value of the first partitioned field, with a child for each value combination of each of the partitions, making it more difficult with a combined partition to further partition a value of the partition.
MDM Import Manager Reference Guide 191
To Combine or Not to Combine
Regardless of whether partitions are nested or combined, partitioning creates the same set of value combinations, each of which corresponds to the same subset of records. So the obvious question is when to combine partitions and when to allow them to remain nested.
The partial answer is that nested and combined partitions achieve the same effect when: (1) partitioning fields by fields; (2) field mapping to a destination field that requires value mapping; and (3) value mapping to existing destination values and/or hierarchy.
NOTE ►► When value mapping to existing destination values, it doesn’t matter whether the partition is nested or combined, since the value of the destination field already exists and only the value combination represented by the node is relevant, not the value of the node itself.
However, nested and combined partitions can be used to very different effect when: (1) partitioning individual field values rather than an entire field; (2) merging field values without value mapping; (3) creating new destination values; or (4) creating new destination hierarchy; as follows:
Partition individual field values. In the example above, if the
partition of STATE value CA were to be further partitioned by ZIP, but value Florida were to remain partitioned only by CITY, the nested partition of STATE by CITY would be more appropriate than a combined partition, which would create two nodes containing the STATE value CA rather than one. In this case, the nested partitions would be as follows:
Value partition:
STATE CITY [ ZIP]
├ CA [ ZIP]
│ ├ Los Angeles
│ │├ 90046
│ │└ 90025
│ └ Hollywood
│ ├ 90069
│ └ 90036
└ FL
├ Miami Beach
└ Hollywood
Merge field values without value mapping. If values need to be
merged for value conversion without value mapping, a combined partition is appropriate, since these merged values are used to create the destination field values.
192 MDM Import Manager Reference Guide
Create new destination values. If new destination values need to
be created as part of value mapping, either a nested partition or a combined partition may be appropriate depending upon the circumstances, since nested and combined partitions result in different leaf node values and the value of each node can be used to be automatically define the new destination field values.
Create new destination hierarchy. If new destination hierarchy
needs to be created as part of value mapping, a nested partition is appropriate, since its internal nodes can be used to automatically define the new destination hierarchy.
Guidelines on when to nest partitions and when to combine them are summarized in Table 24.
Table 24. Guidelines for Nesting and Combining Partitions
Nest or Combine When…
Doesn’t Matter Partition is mapped to a destination field that requires value
mapping to existing destination values.
Nest
Partition is mapped to a destination hierarchy field that requires the creation of new destination hierarchy as part of the value-mapping process.
Individual field values need to be partitioned.
Combine
Partition is mapped to a destination field that requires value conversion without value mapping and the values of the distinct fields participating in the partition must be concatenated into a single merged value for effective conversion.
Depends Partition is mapped to a destination field that requires value
mapping to new destination values.
Merged Values
When you combine two or more partitions, MDM creates the merged value by: (1) concatenating the individual values of the field node and each of the fields or attributes participating in the combined partition; and (2) separating each pair of individual values with the Partition Combine delimiter string specified for the field corresponding to the first value of the pair.
NOTE ►► The Combine Partitions command separates each pair of individual values that are merged using the current Partition Combine delimiter string of the field corresponding to the first value of the pair, which must be properly set prior to performing the command (see “Setting the Partition Combine Delimiter” on page 200 for more information).
MDM Import Manager Reference Guide 193
Consider Figure 131, which illustrates: (1) three source fields PREFIX, CODE, and SUFFIX that must be concatenated into a single merged value to form a Part Number with the format PREFIX-CODE/SUFFIX;
and (2) the values of the combined partition PREFIX CODE SUFFIX if the Partition Combine delimiters are ‘-’ for PREFIX and ‘/’ for CODE.
Original: Combined partition:
PREFIX CODE SUFFIX ► PREFIX CODE SUFFIX
113 1234 21 ► 113-1234/21
113 1450 53 ► 113-1450/53
126 3308 43 ► 126-3308/43
109 2049 14 ► 109-2049/14
Figure 131. Merged Values in a Combined Partition
NOTE ►► Setting a different Partition Combine delimiter string for each pair of values allows you to precisely format the concatenated string for value mapping or conversion in the next step.
MATHEMATICAL ANALYSIS
In mathematical terms, partitioning n fields: (1) generates the n-dimensional Cartesian product of ordered value combinations across the n individual value domains; and (2) removes from the complete set those value combinations that do not exist in the underlying records. Creating this collapsed set of actual value combinations is equivalent to partitioning – in the mathematical sense – the set of underlying records into non-overlapping subsets of records, where each value combination represents a single subset.
NOTE ►► You can then further partition either the entire domain of value combinations or just a subset of the value combinations to create additional Cartesian products based on the previous results.
Each value of a field’s value domain corresponds to an underlying subset of records that contain the value, so that there is a correlation between values and value combinations and corresponding subsets of underlying records. In particular, each value domain partitions the entire set of records into a set of non-overlapping subsets such that each record belongs to exactly one subset, and the entire set of subsets contains the entire set of records. Each partition of a field by another field further partitions each subset into smaller subsets. Thus the term partitioning is used to describe the mechanism of creating value combinations because of the precise partitioning effect that it has on the underlying subsets of records.
194 MDM Import Manager Reference Guide
The partitioning mechanism is necessary when the subsets of source records implicitly created by partitioning the entire set of source records by the values of just a single source field do not correspond to the subsets of destination records implicitly created by partitioning the entire set of destination records by the values of just the single corresponding destination field. In these cases, the mechanism is used to further partition, or refine, the subsets of source and/or destination records into smaller subsets that do correspond to one another. Thus, the effect of establishing a one-to-one correspondence between source and destination values and/or value combinations is mathematically equivalent to establishing a correspondence between the underlying subsets of source and destination records.
INFINITE RESTRUCTURING FLEXIBILITY
Each of the various flavors of partitioning described above has a slightly different effect but are still just variants of the same overall partitioning mechanism, and each can be combined with any other, to achieve any desired effect.
Using the partitioning mechanism of the MDM Import Manager, nested and combined partitions can be applied to source fields and/or values and destination fields and attributes and/or values in an unlimited number of ways, enabling you to arbitrarily restructure and rationalize electronic source data against destination data that has an entirely different structure. Moreover, the underlying engine can not only import but also transform data for destinations outside the repository.
SPLITTING VS. PARTITIONING: A COMPARISON
In the MDM Client, virtually all of the commands in Taxonomy mode that allow you to structure and restructure the taxonomy exist as pairs of
symmetrical commands (Partition and Consolidate Children, Merge and
Split, Promote and Demote), where either command in the pair completely reverse the action of the other.
In the MDM Import Manager, this same symmetry exists between the effects of: (1) splitting a single field; and (2) partitioning multiple fields, as illustrated in Figure 132:
Splitting. Splitting starts with [A] a single field containing multiple
delimited values, and finishes with either [B] a single field containing
a hierarchy (Split into Hierarchy) or [C] multiple fields (Split into
Multiple Fields), each containing a single value.
Partitioning. By contrast, partitioning starts with [C] multiple fields,
each containing a single value, and finishes with either [B] a single field containing a hierarchy (Nested Partition) or [A] a single field containing multiple delimited values (Combined Partition).
MDM Import Manager Reference Guide 195
[A]
Single Field (delimited values)
Field
X; Y; Z
Split into Multiple Fields
▼
Split into Hierarchy ▼
▲ Combined Partition
[B]
Single Field (hierarchy value)
Field
└ X
└ Y
└ Z
▲ Nested Partition
[C]
Multiple Fields (individual values)
Field1 Field2 Field3
X Y Z
Figure 132. Symmetry of splitting and partitioning
196 MDM Import Manager Reference Guide
Partitioning Operations
The following sections describe the various partitioning operations you can perform on nodes of the Source and Destination Hierarchy trees, including:
Partitioning a field or value node by a single field or attribute.
Partitioning a field or value node by multiple fields and/or attributes.
Combining separate partitions.
Splitting partitions that were previously combined.
Removing partitions.
Reordering partitions.
Restoring inheritance of partitioning information.
The partitioning operations are summarized in Table 25.
Table 25. Partitioning Operations
Operation Description
Add Partition Adds the selected items to the list of partitions for the selected tree node.
Remove Partition Removes the selected partition items from the list of partitions for the selected tree node.
Combine Partitions Combines the selected partition items for the selected tree node.
Split Partitions Splits the previously combined partition items for the selected tree node.
Reorder Partitions Reorders the selected partition items for the selected tree node.
Inherit Partitions Eliminates custom partitions for the selected tree node and restores inheritance of partitions.
TIP ►► You can use the Set Combine Delimiter command to set the delimiter string that the MDM Import Manager uses to separate the individual values that are merged when combining partitions.
MDM Import Manager Reference Guide 197
PARTITIONING BY A SINGLE FIELD OR ATTRIBUTE
You can partition a source or destination field or value node by a single field or attribute item as described in this section.
To partition a field or individual field value by a single field or attribute item:
1. In the appropriate hierarchy tree, select the field or value node you want to partition.
2. If necessary, click on the Partition Field/Value tab to make it the active tab. This tab lists all the items by which you can partition the selected node in each tree, as shown in Figure 133.
Figure 133. Source Hierarchy field node before partitioning
NOTE ►► MDM displays the selected node as the first item in the Partition By list.
NOTE ►► In the Linked Attributes list for the Destination Hierarchy, field names are displayed in normal type and attribute names are highlighted in italics.
3. In the appropriate Available items list, select the item by which you
want to partition the selected node, and click on the Add button.
4. MDM partitions the selected node by the item, as shown in Figure 134. Note that in the figure there are now new value nodes underneath the partitioned node.
198 MDM Import Manager Reference Guide
Figure 134. Source Hierarchy node after partitioning by a single field
NOTE ►► You can repeat this procedure to successively partition any field or value node by other items.
NOTE ►► When you directly partition a node, MDM: (1) displays a small purple square ( ) to the left of the node icon in the hierarchy
tree; and (2) unchecks the Inherit checkbox in the Partition Field/Value
tab. To restore inheritance, recheck the Inherit checkbox (see “Restoring Inheritance” on page 203).
TIP ►► Nodes are nested in the hierarchy tree in the same order as the items shown in the Partition By list on the Partition Field/Value tab. You can change the nesting of the nodes by simply dragging-and-dropping the list items to rearrange them into the desired order.
PARTITIONING BY MULTIPLE ITEMS
You can partition a field or value node by a combination of multiple field and/or attribute items as described in this section.
To partition a field or individual value by multiple field and/or attribute items:
1. In the appropriate hierarchy tree, select the field or value node you want to partition.
2. If necessary, click on the Partition Field/Value tab to make it the active tab. This tab lists all the items by which you can partition the selected node in each tree, as shown in Figure 133 above.
3. In the appropriate Available items list, select the items by which you want to partition the selected node.
4. Click on the Add button and choose from the cascading menu:
Add as Individual Add as Combination
MDM Import Manager Reference Guide 199
5. MDM partitions the selected node by the selected items, either individually or simultaneously, as shown in Figure 135. Note that in the figure, there are now new value nodes underneath the partitioned node for each combination of values of the partitioning fields.
Figure 135. Source Hierarchy node after partitioning by multiple items
NOTE ►► When you directly partition a node, MDM: (1) displays a small purple square ( ) to the left of the node icon in the hierarchy
tree; and (2) unchecks the Inherit checkbox in the Partition Field/Value
tab. To restore inheritance, recheck the Inherit checkbox (see “Restoring Inheritance” on page 203).
NOTE ►► When you partition a field or value node by multiple items at the same time, MDM automatically creates a combined rather than a nested partition, as described in the next section.
COMBINING PARTITIONS
You can combine two or more partitions that were previously added separately as described in this section.
To combine two or more existing partitions for a node:
1. In the appropriate hierarchy tree, select the node whose partitions you want to combine.
2. If necessary, click on the Partition Field/Value tab to make it the active tab.
3. In the appropriate Partition By list, select the two or more partition items you want to combine into a single partition.
4. Click on the Combine button, or right-click on one of the items and
choose Combine Partitions from the context menu.
5. MDM combines the selected partition items.
NOTE ►► See “Splitting a Partition” on page 201 for more information about how to split combined partitions.
200 MDM Import Manager Reference Guide
NOTE ►► The Combine Partitions command separates each pair of individual values that are merged using the current Partition Combine delimiter string of the field corresponding to the first value of the pair, which must be properly set prior to performing the command (see “Setting the Partition Combine Delimiter” on page 200 for more information).
NOTE ►► Combining partitions for a node that is currently inheriting partitions from an ancestor in the hierarchy directly partitions the selected node and breaks the inheritance for that node, so that it no longer inherits partition changes made to its ancestor.
NOTE ►► When you directly partition a node, MDM: (1) displays a small purple square ( ) to the left of the node icon in the hierarchy
tree; and (2) unchecks the Inherit checkbox in the Partition Field/Value
tab. To restore inheritance, recheck the Inherit checkbox (see “Restoring Inheritance” on page 203).
SETTING THE PARTITION COMBINE DELIMITER
The Combine Partitions command separates each pair of individual merged values with the current Partition Combine delimiter string of the field corresponding to the first value of the pair, which must be properly set prior to performing the command.
Specifically, the command use as the current delimiter either: (1) the global default delimiter string, if the field is set to inherit; or (2) the field-specific delimiter string, if it has been manually set for the field.
You can set the global default delimiter string for all fields that inherit the
default using the Configuration Options command, or alternatively,
override the default and set it on a field-by-field basis using the Set
Combine Delimiter command, as described in this section.
NOTE ►► Inherited global default values appear in the Source Fields grid in gray.
NOTE ►► See “Configuration Options” on page 62 for more information about the global default delimiter strings.
To set the Partition Combine delimiter string for one or more fields:
1. In the appropriate hierarchy tree, select the field nodes for which you want to set the Partition Combine delimiter string.
2. Right-click on one of the nodes and choose Partitions > Set Combine Delimiter from the context menu, or choose Fields > Set Value Delimiter > Partition Combine from the main menu, to open the Set Combine Delimiter dialog shown in Figure 136.
MDM Import Manager Reference Guide 201
Figure 136. Set Combine Delimiter dialog
TIP ►► Alternatively, you can right-click on the item in the Partition By
list whose combine delimiter string you want to set and choose Set
Combine Delimiter from the context menu.
3. MDM displays the current value for the delimiter string and whether or not the global default is being inherited.
NOTE ►► No value is displayed in the edit control if you have selected multiple fields for which the current values are not all the
same. Similarly, a gray check () appears in the Inherit Default checkbox if some but not all of the selected fields are inheriting the global default.
4. To set the delimiter string and override the global default value, enter the new delimiter string into the edit control, or alternatively, to restore
inheritance, check the Inherit Default checkbox.
TIP ►► You can restore inheritance for all of the value delimiter
strings for the selected fields by choosing Fields > Set Value Delimiter
> Inherit Defaults from the main menu.
5. Click OK when you are done to close the dialog.
SPLITTING A PARTITION
You can split a partition that was previously combined as described in this section.
To split a partition for a hierarchy node:
1. In the appropriate hierarchy tree, select the node whose combined partition you want to split.
2. If necessary, click on the Partition Field/Value tab to make it the active tab.
3. In the appropriate Partition By list, select the combined partition item you want to split into its individual component partitions.
4. Click on the Split button, or right-click on one of the items and choose
Split Partitions from the context menu.
5. MDM splits the selected partition items.
NOTE ►► See “Combining Partitions” on page 199 for more information about how to combine partitions.
202 MDM Import Manager Reference Guide
NOTE ►► Splitting a partition for a node that is currently inheriting partitions from an ancestor in the hierarchy directly partitions the selected node and breaks the inheritance for that node, so that it no longer inherits partition changes made to its ancestor.
NOTE ►► When you directly partition a node, MDM: (1) displays a small purple square ( ) to the left of the node icon in the hierarchy
tree; and (2) unchecks the Inherit checkbox in the Partition Field/Value
tab. To restore inheritance, recheck the Inherit checkbox (see “Restoring Inheritance” on page 203).
REMOVING PARTITIONS
Removing partitions in the Source or Destination Hierarchy is a matter of simply selecting the fields and/or attributes by which a node was partitioned, and removing them from the list of partitions for the node, as described in this section.
To remove one or more partitions from a node:
1. In the appropriate hierarchy tree, select the node from which you want to remove the partitions.
2. If necessary, click on the Partition Field/Value tab to make it the active tab.
3. In the appropriate Partition By list, select the one or more partition
items you want to remove, and click on the Remove button.
4. MDM removes the partition item(s) from the selected node, and restructures the hierarchy to eliminate any nodes that had been created as a result of the partition(s).
NOTE ►► Removing a partition for a node that is currently inheriting partitions from an ancestor in the hierarchy directly partitions the selected node and breaks the inheritance for that node, so that it no longer inherits partition changes made to its ancestor.
NOTE ►► When you directly partition a node, MDM: (1) displays a small purple square ( ) to the left of the node icon in the hierarchy
tree; and (2) unchecks the Inherit checkbox in the Partition Field/Value
tab. To restore inheritance, recheck the Inherit checkbox (see “Restoring Inheritance” on page 203).
REORDERING PARTITIONS
You can reorder the fields and attributes by which a node is partitioned by dragging and dropping the items in the Partition By list of the Partition Field/Value tab.
MDM Import Manager Reference Guide 203
NOTE ►► Reordering the partitions for a node that is currently inheriting partitions from an ancestor in the hierarchy directly partitions the selected node and breaks the inheritance for that node, so that it no longer inherits partition changes made to its ancestor.
NOTE ►► When you directly partition a node, MDM: (1) displays a small purple square ( ) to the left of the node icon in the hierarchy
tree; and (2) unchecks the Inherit checkbox in the Partition Field/Value
tab. To restore inheritance, recheck the Inherit checkbox (see “Restoring Inheritance” on page 203).
RESTORING INHERITANCE
Directly partitioning an internal node or a leaf node breaks the inheritance for that node. You can restore the inheritance for that node as described in this section.
To restore inheritance for a node:
1. In the appropriate hierarchy tree, select the node for which you want to restore inheritance.
2. If necessary, click on the Partition Field/Value tab to make it the active tab.
3. Check the Inherit checkbox.
4. MDM restores the inheritance to the selected node.
NOTE ►► Restoring the inheritance for a directly partitioned node replaces the partition items for the node with the inherited partitions from its ancestor in the hierarchy.
205
PART 10: MAPPING FIELDS/VALUES
This part of the reference guide explains the field mapping and value conversion and mapping mechanisms of the MDM Import Manager, which allow you to cleanse, normalize, and rationalize source data against destination data as one of the steps in the import process.
MDM Import Manager Reference Guide 207
Overview
Field mapping is perhaps the most traditional and fundamental step in the import process, in which most import tools allow you to map an individual source field to an individual destination field, and then import data values from the source field directly into the destination field.
The MDM Import Manager takes this basic field mapping capability and extends it with a powerful value mapping and conversion capability, for transforming, cleansing, and rationalizing source data as part of the import process itself. In particular, value-level operations include:
Value mapping. Source fields that contain text values and are
mapped to lookup fields or text attributes can be mapped at the value level against the domain of legal destination values, eliminating the need to pre-cleanse source data in an external application.
Data type conversion. The data types of source data values are
automatically converted by the MDM Import Manager based on the data type of the mapped destination field, even when stored in the source data as text values.
Measurement value creation. Unit of measure strings are
automatically normalized and appended to each numeric source data value based on the physical dimension of the mapped destination measurement field or numeric attribute.
Data value conversion. Not only can each individual source value
be manually edited, built-in and custom-designed value conversion filters can automatically convert or reformat a single value or the entire set of distinct values with a single command.
Moreover, as described in the previous part of this guide, the MDM partitioning mechanism allows you to combine fields into field combinations prior to field mapping, so that you can: (1) field map to and/or from partitioned field combinations rather than individual fields; and then (2) value map to and/or from the underlying value combinations rather than individual values.
NOTE ►► If the Partition Field/Value tab is the workhorse that allows you to restructure source data with partitioning, the Map Fields/Values tab is the workhorse that allows you to transform source data with field-level mapping, and value-level mapping, conversion, and measurement value creation.
INNOVATION ►► Field and value mapping are where field-at-a-time processing and distinct value lists become enormously efficient. For each mapped field or field combination, you must map, convert, or adjust each distinct value or value combination just once to transform it for all instances across the entire set of source records.
208 MDM Import Manager Reference Guide
THE MAP FIELDS/VALUES TAB
The Map Fields/Values tab contains two pairs of dual-grid controls, as shown in Figure 137. These grids allow you to perform both field-level mapping and value-level operations, as follows:
Field level. For each source field or field combination whose data
values you want to import, you can use the top pair of Field Mapping grids to map the field to the corresponding destination field.
Value level. For each mapped source field, you can use the bottom
pair of Value Conversion and Mapping grids to either map, convert, or create measurements from the underlying field values, dependent upon the mapped destination field and data type.
Figure 137. Map Fields/Values tab
The two pairs of grids in the Map Fields/Values tab are described in Table 26.
Table 26. Grids of the Map Fields/Values Tab
Grid Description
Field Mapping Grids
Source Fields Displays the fields of the current source table.
Destination Fields Displays the fields, tuples, attributes, and qualifiers of the current destination table.
Value Conversion and Mapping Grids
Source Values Displays the set of distinct source values for the current source field, either as a list or as a tree.
Destination Values Displays the set of distinct destination values for the current destination field, either as a list or as a tree.
MDM Import Manager Reference Guide 209
The Field Mapping Grids
Field mapping is used to establish the correspondence between source and destination fields. The sets of source and destination fields available for mapping are displayed in the Source Fields and Destination Fields grids, respectively.
Source Fields grid. Displays the fields of the current source table as
a tree (if XML) or a flat list.
Destination Fields grid. Displays the fields of the current
destination table as a tree.
Both grids display fields according to their “natural” order. For example, if a source file is in XML format, Import Manager uses a tree in the Source Fields grid to depict the nested structure of fields within the associated XML schema. Likewise, the Destination Fields grid displays tuples, qualified lookup fields, and other nested objects in their natural hierarchies, preserving the relationships within them that could otherwise be lost in a flat list.
The Source Fields and Destination Fields grids are shown in Figure 138.
Figure 138. Field Mapping grids
NOTE ►► The Source Fields and Destination Fields grids are each disabled if you have not yet selected the current source or destination table, respectively.
NOTE ►► The use of trees in the Source Fields and Destination Fields grid is new in MDM 7.1. In previous versions, these grids contained only flat lists.
For each source field or field combination whose data values you want to import, you can use the Field Mapping grids to map the field to its corresponding destination field.
The columns of the Field Mapping grids are described in Table 27 (see “Field Mapping Operations” on page 269 for more information).
210 MDM Import Manager Reference Guide
The Value Conversion and Mapping Grids
Value conversion and mapping layers upon and augments field-level mapping and is used to transform source data at the value level.
Specifically, for each mapped source field, the MDM Import Manager uses the bottom pair of Value Conversion and Mapping grids to display: (1) the set of distinct values of the field as a list or a tree in the Source Values grid; and (2) the set of distinct values of the mapped destination field as a list or a tree in the Destination Values grid, as shown in Figure 139.
Figure 139. Value Conversion and Mapping grids
Table 27. Columns of the Field Mapping Grids
Column Description
Source Fields Grid
Mapped Whether the source field has been mapped ( ).
Position (Pos.) The display order of the field within the source hierarchy (n).
Sorting by this column restores the field order of the source hierarchy.
Name The name of the source field.
Type The data type of the source field (e.g. Text, Numeric).
Flat NULLs*
The setting for the Flat NULL Handling option (gray if inheriting default):
Preserve NULLs Ignore NULLs
Hierarchy NULLs*
The setting for the Hierarchy NULL Handling option (gray if inheriting default):
Preserve NULLs
Ignore Non-Leaf Trailing NULLs > Preserve Leaf NULLs as NULLs > Preserve Leaf NULLs as Internal Leaf Nodes
Ignore All Trailing NULLs
Ignore Non-Leaf Trailing NULLs has been split into two subsettings for flexible handling of the leaf trailing NULLs.
Case Sensitivity*
The setting for the Case Sensitivity option (gray if inheriting default):
Case Insensitive Case Sensitive
For SQL Server or Oracle data sources, case sensitivity of the source field is determined by database collation and cannot be changed by the MDM Import Manager.
SH* The Split Hierarchy delimiter string (gray if inheriting default).
SM* The Split Multiple Fields delimiter string (gray if inheriting default).
MDM Import Manager Reference Guide 211
Column Description
PC* The Partition Combine delimiter string (gray if inheriting default).
MV* The Multi-Valued delimiter string (gray if inheriting default).
CN* The Coupled Numeric delimiter string (gray if inheriting default).
Filters* The field-level filters (None, or semi-colon delimited list).
Destination Field The name of the corresponding mapped destination field.
Destination Fields Grid
Mapped Whether a source field is mapped to the destination field ( )
Position (Pos.) The display order of the field within the destination hierarchy (n).
Sorting by this column restores the field order of the destination hierarchy.
F/D/Q/A
The field type:
F – standard destination field
D – display field of a lookup with multiple display fields
Q – qualifier A – attribute
Name The name of the destination field.
Language The language instance of an expanded multilingual field.
Type The data type of the destination field (e.g. Integer, Text, Lookup [Flat]) or attribute (e.g. Text, Numeric, Coupled), where a dimension appears as “Measurement [dimension]” (for fields) or “Numeric [dimension]” (for attributes).
NULLs*
The setting of the NULL Interpretation option:
Ignore Overwrite
MV/Qual*
The setting of the Multi-Valued Update option or the Qualified Update option, as applicable for the corresponding multi-valued or qualified lookup destination field:
Append
Replace Update
Data Group*
The setting of the Data Group Display option:
Hide Data Group Display Data Group
MDIS*
The setting of the MDIS Unmapped Value Handling option:
Ignore Add Map
Fail
You can set the Map option only at the field level, since the set of destination values is different for each destination field.
Source Field The name of the mapped source field.
* Hidden by default; unhide to display.
MULTILINGUAL ►► More information about languages and multilingual repositories is provided in “Part 14: Multilingual Support.”
For each mapped source field, you must use the Value Conversion and Mapping grids to either map, properly convert, or create measurement values from each of the distinct values for the field, where the required mapping or conversion depends upon the mapped destination field type.
212 MDM Import Manager Reference Guide
NOTE ►► Distinct value lists allow you to map, convert, or adjust each distinct value just once to transform it for all instances across the entire set of source records.
NOTE ►► The Source Values grid is disabled if the current source field has not yet been mapped. The Destination Values grid is disabled if the current mapped source field does not require value mapping.
The columns of the Value Conversion and Mapping grids are described in Table 28 (see “Value Conversion and Mapping Operations” on page 294 for more information).
Table 28. Columns of the Value Conversion and Mapping Grids
Column Description
Source Values Grid
Status/Conv./Mapped1 Whether the source value has been converted ( ) / mapped ().
Value The original source value.
Converted Value The converted source value.
Destination Value The corresponding mapped destination value(s).
Destination Values Grid
Pos. The ordered position of the mapped destination value.
Mapped Whether a source value is mapped to the destination value ( )
Value The destination value.
Source Value The corresponding mapped source value(s).
1 Name changes depending on the selected field and whether value mapping is required.
NOTE ►► If the cell selection in the Source Values grid is on the Converted Value column, typing enters the Converted Value edit control rather than triggering typeahead seek on the sorted column.
VALUE MAPPING
When mapping to destination fields whose values are from a fixed domain of valid values (lookup fields and text attributes), corresponding source and destination values may not precisely match, and some source values may not already exist as destination values.
In these cases, source fields that contain text values and are mapped to lookup fields and text attributes can be mapped not only at the field level, but also at the value level against the domain of legal destination values, eliminating the need to pre-cleanse source data in an external application.
MDM Import Manager Reference Guide 213
NOTE ►► For each field, the MDM Import Manager collapses the set of all values across the entire set of source records down to the set of distinct values, whose number will usually be dramatically smaller than the total number of records, often by several orders of magnitude.
Value-level mapping of source values forces a user to normalize source data, implicitly defining a precise set of transformation rules that: (1) eliminate differences between source and destination values; (2) collapse multiple source value variations into a single destination value; (3) properly map multiple delimited source values into a multi-valued MDM field; and (4) explicitly add new values to the set of destination values.
NOTE ►► The value in the Converted Value column is most useful for adding new destination values or hierarchy, where it is used as the new item or node value. When mapping to existing destination values,
the Converted Value is just a convenience for Automap, since each value could be manually mapped to any destination value regardless of its value.
NOTE ►► Value mapping to the value domain of a lookup field or text attribute is like record matching to an index field in that both help to avoid creating duplicates of records or values that already exist.
For each source value, you can use the Value Conversion and Mapping grids to manually map to the corresponding destination value or values, or you can automatically map values using the Automap button.
When value mapping is required, each distinct source data value must be mapped and all discrepancies in the source data resolved in order to perform the import, as indicated by a colored thunderbolt ( ) in the Mapped column of the Source Values grid.
NOTE ►► When you map a source field to a destination field that requires value mapping, the name of the first column changes from Status to Mapped.
Eliminating Value Differences
Value mapping can be used to eliminate differences between source and destination values, including the addition of new destination values. This is necessary to avoid polluting the repository with illegal values, which must be translated to the valid domain of destination values during the import process.
214 MDM Import Manager Reference Guide
Consider Figure 140, in which source field STATE has been mapped to destination field State. The source values of the STATE field are completely different from the destination values of the State field, which is itself a lookup into the States table that contains the set of consistent, normalized two-letter state codes for each of the fifty states. Moreover, the source value Penna has no corresponding destination value.
Source: Destination:
STATE ► State [lookup]
Calif ≠ CA
New Jersey ≠ NJ
Tenn ≠ TN
Flor ≠ FL
Penna ?
Figure 140. Source and destination value differences
In this case, each source value can be mapped to its corresponding destination value, and a new destination value PA (shown in blue) added to correspond to the source value Penna, as shown in Figure 141.
Source: Destination:
STATE ► State [lookup]
Calif ► CA
New Jersey ► NJ
Tenn ► TN
Flor ► FL
Penna ► PA
Figure 141. Value mapping eliminates value differences
NOTE ►► Source values that do not have corresponding destination values can be added during the value mapping process.
Collapsing Source Value Variations
Value mapping can also be used to collapse multiple source value variations into a single destination value. This is necessary when source data originates in a system that permits entry of inconsistent data values, which must be normalized during the import process.
Consider Figure 142, in which source field STATE has once again been mapped to destination field State. This time, four source values (Calif, Ca., California, and Cal) all correspond to a single destination value CA.
MDM Import Manager Reference Guide 215
Source: Destination:
STATE ► State [lookup]
Calif CA
Ca.
California
Cal
Figure 142. Multiple source value variations
In this case, each source value can be mapped to the same destination value CA, as shown in Figure 143.
Source: Destination:
STATE ► State [lookup]
Calif ►
CA Ca. ►
California ►
Cal ►
Figure 143. Value mapping collapses value variations
Mapping to Multi-Valued Destination Fields
When the mapped destination field is an MDM multi-valued field, you can use value mapping to map each set of delimited values in a single source field to multiple values of the destination field. This is necessary for most source data since other systems usually represent multi-valued information as a delimited string.
Consider Figure 144, in which source field UOM, which contains multiple delimited values in each record, has been mapped to destination multi-valued field Unit of Measure.
Source: Destination:
UOM ► Unit of Measure [l]
Each; Carton Box
Box; Cylinder; Each Carton
Carton Each
Each; Package Package
Box
Figure 144. Multiple delimited source values
216 MDM Import Manager Reference Guide
As part of the value-mapping process, the set of delimited values represented by each UOM source value can be mapped to the one or more corresponding Unit of Measure destination values (including the new value Cylinder, shown in blue), as shown as Figure 145.
Source: Destination:
UOM ► Unit of Measure [l]
Each; Carton Box
Box; Cylinder; Each Carton
Carton Each
Each; Package Package
Box Cylinder
Figure 145. Value mapping to multi-valued field
Mapping to Hierarchy Destination Fields
When the mapped destination field is an MDM hierarchy field, you can use value mapping to map the values in a source field to the values of the destination field, and also to create new destination hierarchy
NOTE ►► see “Hierarchy Values” on page 226 and “Adding New Destination Hierarchy“ on page 306 for more information.
Consider Figure 146, in which source field CATEGORY has been mapped to destination hierarchy lookup field Category.
Source: Destination hierarchy:
CATEGORY ► Category [lookup]
Laser Printers ├ Printers
Inkjet Printers │ ├ Laser
Dot Matrix Printers │ └ Dot Matrix
Mounted Bearings └ Bearings
Unmounted Bearings ├ Mounted
One-Phase Motors └ Unmounted
Two-Phase Motors
Three-Phase Motors
Figure 146. Destination hierarchy lookup field
As part of the value-mapping process, each source field value is mapped directly to the corresponding value in the destination hierarchy (including the new value for the Inkjet printers category, and an entirely new branch for Motors and the three Motors categories, shown in blue).
MDM Import Manager Reference Guide 217
Source: Destination hierarchy:
CATEGORY ► Category [lookup]
├ Printers
Laser Printers ► │ ├ Laser
Inkjet Printers ► │ ├ Inkjet
Dot Matrix Printers ► │ └ Dot Matrix
├ Bearings
Mounted Bearings ► │ ├ Mounted
Unmounted Bearings ► │ └ Unmounted
└ Motors
One-Phase Motors ► ├ One-Phase
Two-Phase Motors ► ├ Two-Phase
Three-Phase Motors ► └ Three-Phase
Figure 147. Value mapping to destination hierarchy lookup field
Mapping to Object Lookup Destination Fields
An MDM object lookup field (i.e. image, text block, PDF, or binary object) is like any MDM lookup field, in that the value of the field is a link to one or more values in the underlying lookup table.
When the mapped destination field is an MDM object lookup field, you can map the values in a source field to the objects of the underlying object lookup table, thereby linking the corresponding object to each source record as part of the import.
The subtle differences among object lookup fields are summarized in Table 29.
Table 29. Mapping to Object Fields
Object Value Cell in Destination Values Grid
Image Lists the name of each image in the Images table rather than the image itself; each source value must be mapped to the name of an existing destination image.
Text Block Lists the value of each text block in the Text Blocks table; you can map to existing text blocks or add and map to new text blocks added as part of the import, and also add new data groups prior to mapping.
PDF Lists the name of each PDF in the PDFs table rather than the PDF itself; each source value must be mapped to the name of an existing destination PDF.
Binary Object Lists the name of each Binary Object in the Binary Objects table rather than the object itself; each source value must be mapped to the name of an existing destination Binary Object.
218 MDM Import Manager Reference Guide
NOTE ►► To distinguish among objects that have the same name, you can display the Data Groups hierarchy in the Destination Values grid so that each object appears as a leaf node beneath its data group. (See “The Data Group Display Option” on page 265 for more information.)
NOTE ►► You must use Data Manager to import new image, PDF, or binaryobjects into the repository, since the MDM Import Manager imports links to existing objects, not objects themselves.
NOTE ►► When the destination field is an image, PDF, or binary
object lookup field, the Add button and the Add Value commands for the Destination Values grid are both disabled.
NOTE ►► When the destination field is a text block object lookup field, the Values column in the Destination Values grid displays only the first 255 characters of each text block, but allows you to edit new text blocks of arbitrary length.
Mapping to Main Table Lookup Destination Fields
Import Manager handles main table lookup fields (Lookup [Main]) differently than other MDM lookup fields. Specifically, Import Manager does not display the complete set of display field values of the records of the underlying lookup table. Instead, the values it displays for a main table lookup field are limited by both the key mappings for the lookup table and the values in the source file.
Also, Import Manager does not automatically display the values of a
Lookup [Main] destination field in the Destination Values grid when you select the field in the Destination Fields grid. Instead, for a main table lookup field value to appear in the Destination Values grid, all of the following conditions must be met:
The lookup table must have key mapping enabled
The lookup field must be mapped to a source field
The source field must contain key values for the lookup table
The destination value must have a key on the current remote system
The destination value’s key must match a source field value
NOTE ►► The current remote system is the remote system that was selected in Import Manager’s Connect to Source dialog (see “Connecting to a Remote System” on page 412 for more information).
These differences are summarized in Table 30.
MDM Import Manager Reference Guide 219
Table 30. Differences between Lookup [Main] and Other Lookup Fields
Destination Values Lookup [Main] Fields Other Lookup Fields
Must have key mappings? Yes No
Visible only after field is mapped? Yes No
Limited by remote system? Yes No
Limited by source values? Yes No
Consider Figure 148, which displays key mappings for the lookup main table, Customers, and which also shows customer-related fields in a source file. Note that the values of source field ID match some of the remote key values for the Customers main table.
Customers: Source:
Name [DF] [Key Mapping] ID Name
John Smith MDM; 301 301 John Smith
CRM; 001 303 Susan Gray
Ken Hong CRM; 002 305 Mary Hsu
Susan Gray MDM; 303
Ed Bauer MDM; 307
Figure 148. Main table key mappings and source field values
Now consider Figure 149, in which source field ID is mapped to the destination field Customer, which is a lookup into the Customers main table. Note that the only destination values displayed for the Customer field are those whose key mapping values for the current remote system, MDM, match values in the mapped Source ID field.
Source: Destination:
ID ► Customer [lookup]
301 John Smith
303 Susan Gray
305
Remote System: MDM
Figure 149. Source values and main table lookup destination values
Once you map the source field to the main table lookup destination field, Import Manager automaps each source field value to its corresponding destination value for your convenience. You can also add and map destination values as needed (shown in blue in Figure 150).
220 MDM Import Manager Reference Guide
Source: Destination:
ID ► Customer [lookup]
301 ► John Smith
303 ► Susan Gray
305 ► Mary Hsu
Figure 150. Source values and main table lookup destination values
During import, MDM creates a new record on the lookup table for every new destination value you added and mapped. As shown in Figure 151, this new record is blank except for the mapped source value, which is added as the value of the lookup table’s primary display field.
Customers:
Name [DF] Telephone
John Smith 800-555-1212
Susan Gray 212-999-9876
Mary Hsu
Figure 151. New record added to destination table
NOTE ►► Source values are added to text display fields only. If the lookup table has no text display fields, the new record is entirely blank.
When Value Mapping Is Required
The Map checkbox resides at the bottom of the Map Fields/Values tab, just between the Value Conversion and Mapping grids. Based upon the current table and the type of the mapped destination field, it indicates whether or not the current mapped source field requires value mapping, and if checked, that you must map source to destination values.
Figure 152. Map checkbox (value mapping required)
NOTE ►► The Map checkbox is: (1) disabled and automatically checked when value mapping is required, as shown in Figure 152; (2) disabled and automatically unchecked when value mapping is neither required nor permitted; and (3) enabled for you to check or uncheck manually when value mapping is optional.
Whether or not particular destination field, attribute, and qualifier types require or permit optional value mapping is summarized in Table 31.
MDM Import Manager Reference Guide 221
Table 31. Value Mapping Requirements
Required or Optional* to… Not Required and Not Permitted to…
Current table = Any table
Lookup [Flat] Lookup [Hierarchy] Lookup [Taxonomy] Lookup [Qualified] Lookup [Object] Lookup [Main] Booleans Text attributes Lookup qualifiers Promoted Text lookup display fields*
Current table = Hierarchy lookup table
Primary display field*
Current table = Any lookup table
Text display fields*
Current table = Any table
Non-lookup fields Numeric attributes Coupled numeric attributes Non-lookup qualifiers AAA
AAA AAA
Current table = Main / non-lookup subtable
Display fields
NOTE ►► Value mapping is optional to fixed-width Text fields that are: (1) promoted lookup table display fields when the current table is the main table and the lookup table has multiple display fields; (2) the primary display field when the current table is a hierarchy lookup table; and (3) actual display fields when the current table is a flat lookup table (see “Value Mapping and Lookup Table Display Fields” on page 247).
VALUE CONVERSION
Regardless of whether or not a field requires value mapping, individual source data values may require: (1) manual adjustment or correction; (2) data type conversion based on the respective source and destination field data types; and (3) the assignment of a unit of measure.
You can manually perform value conversion on each source value by editing the contents of each cell in the Converted Value column of the Source Values grid, or you can automatically convert one or more values by applying a value conversion filter at either the individual value level or at the field level (see “Value Conversion Filters” on page 238 for more information about value conversion filters).
When value mapping is not required, each value must be properly converted in order to perform the import, as indicated by a green bullet (
) in the Conv. column of the Source Values grid.
Automatic Trimming of Leading and Trailing Spaces
When mapping source fields containing values with leading or trailing spaces, it is not necessary to manually trim the leading or trailing spaces from these values, as Import Manager automatically trims leading and trailing spaces when it imports values into a repository.
222 MDM Import Manager Reference Guide
Automatic Data Type Conversion
When value mapping is not required, the data types of the corresponding source and destination fields may be different, such as when numeric, currency, date, time, or Boolean source data are each stored as text values within text fields.
In these cases, the data types of source data values are automatically converted within the Source Values grid based on the data type of the destination field to which the source field is mapped, even when stored in the source data as text values.
Original source values appear with their original data type in the Value column, and the converted values appear in the Converted Value column with the data type of the mapped destination field and the proper grid cell type for editing (e.g. text, numeric, measurement, currency, date, time, Boolean). If necessary, individual values can then be manually edited to override errors or omissions in the source data.
NOTE ►► The Import Manager can parse and convert only Arabic numbers when doing automatic value conversion to numeric fields.
Measurement Value Creation
When mapping to destination fields that represent physical measurements (measurement fields and numeric attributes), the units of measure in the source data may be: (1) missing entirely, such as when all the values in a source field have the same implicit unit; (2) inconsistent, such as when different text strings are used to represent the same physical unit (e.g. inches, inch, in, or ‘"’); or (3) different for each source data value.
The syntactical challenge of creating measurements is further compounded because the source data values may be stored either: (1) in a single numeric field; (2) in a single text field; or (3) in a pair of fields (one containing the numeric value and the other containing the corresponding unit of measure string); additionally, each field may contain: (4) a single value; or (5) multiple delimited values.
For example, in Figure 153, all the measurement values have the same implicit unit (which is missing entirely), and the numeric values are stored in a single numeric field.
Source: Destination:
WIDTH (INCHES) ► Width [length]
3 3 in
2 1 in
1 1.5 in
2 ½
Figure 153. Measurement values with implicit unit in numeric field
MDM Import Manager Reference Guide 223
In Figure 154, multiple delimited measurement values each have different units and inconsistent unit strings, and are stored in a single text field.
Source: Destination:
SOCKET SIZE ► Socket Size [length]
2 “; 2.5 INCH 1 in; 1.5 in
10 millimeters; 12 mm 15 mm; 20 mm
15 mm; 20 millimeters 2.5 in; 3 in
2 1/2 IN; 3 “
Figure 154. Delimited measurement values with inconsistent unit strings
Finally, in Figure 155, the measurement values are stored in a pair of fields, one containing the numeric value and the other containing the corresponding unit of measure string.
Source: Destination:
WIDTH WIDTH (UNIT) ► Width [length]
3 inches 3 in
2 “ 1 in
1 inch 1.5 in
2 1/2 IN
Figure 155. Numeric value and unit string stored in two source fields
In each of these cases, unit of measure strings are automatically normalized and appended to each numeric source data value based on the physical dimension of the destination measurement field or numeric attribute to which the source field is mapped. If necessary, each numeric measurement value can be converted to a target unit, or manually edited to override errors or omissions in the source data.
DATA INTEGRITY ►► MDM’s measurement data type guarantees unit of measure consistency and permits each individual measurement value to have its own distinct unit. Moreover, unit of measure strings are automatically normalized against MDM’s built-in dictionary of over 2000 synonyms for over 750 units in over 70 physical dimensions. The import cannot occur until each numeric source data value has an associated unit of measure.
224 MDM Import Manager Reference Guide
SOURCE FIELD-LEVEL SETTINGS
The steps leading up to field mapping (and then to value conversion and mapping) require that each of the various field-level settings for source fields be set at the proper times and in the proper sequence.
Field-level settings for each source field include:
Flat NULL handling. The Flat NULL Handling option determines
how NULLs are interpreted in multiple delimited values.
Hierarchy NULL handling. The Hierarchy NULL Handling option
determines how NULLs are interpreted in hierarchy values.
Case sensitivity. The Case Sensitivity option determines how case
differences are interpreted in source values.
Value delimiters. The value delimiter strings determine how multiple
delimited values within a single field are identified and generated.
Value conversion filters. One or more value conversion filters are
applied in sequence to each of the distinct values of a field.
The field-level settings for source fields (and when they must be set) are summarized in Table 32. Each field-level setting can be set on a field-by-field basis.
NOTE ►► NULL handling, case sensitivity, and delimiters must be set before you field map because they affect how original values are interpreted and how converted values are generated; the NULL Handling and Case Sensitivity options also affect the set of original values. By contrast, value conversion filters must be set after you map because they depend on the destination data type.
Table 32. Field-Level Settings for Source Fields
Field Setting Must Be Set… Description
Flat NULL Handling Option (see page 279)
Flat NULL Handling
Prior to Field Mapping Specifies whether to preserve or ignore NULLs in multiple delimited values.
Hierarchy NULL Handling Option (see page 280)
Hierarchy NULL Handling
Prior to Field Mapping Specifies how to preserve or ignore NULLs in hierarchy values.
Case Sensitivity Option (see page 88)
Case Sensitivity Prior to Display Distinct Values,
Split, Join, and Field Mapping
Specifies whether to ignore or recognize case differences in source values.
Split Delimiters (see page 150)
Split Hierarchy Delimiter
Prior to Split into Hierarchy Delimits values when splitting a field into a hierarchy.
Split Multiple Fields Delimiter
Prior to Split into Multiple Fields Delimits values when splitting a field into multiple fields.
Partition Combine Delimiter (see page 200)
MDM Import Manager Reference Guide 225
Field Setting Must Be Set… Description
Partition Combine Delimiter
Prior to Combine Partitions Delimits merged values when combining partitions.
Field Mapping Delimiters (see page 276)
Multi-Valued Delimiter
Prior to Field Mapping Delimits values when mapping to a multi-valued field.
Coupled Numeric Delimiter
Prior to Field Mapping Delimits value pairs when mapping to a coupled numeric attribute.
Value Conversion Filters (see page 281 and 314)
Value Conversion Filters
After Field Mapping (and before value-level edits/conversions)
Specifies the value conversion filters to be applied to each individual field value.
DESTINATION FIELD-LEVEL SETTINGS
The various field-level settings for each destination field can be set at any time. Field-level settings for each destination field include:
NULL interpretation. The NULL Interpretation option determines
how NULL values are interpreted for existing values.
Multi-valued updates. The Multi-Valued Update option determines
how updates are handled in multi-valued destination fields.
Qualified updates. The Qualified Update option determines how
updates are handled in qualified lookup destination fields.
Data group display. The Data Group Display option determines
whether the Data Groups hierarchy is displayed for object lookup destination fields.
MDIS handling. The MDIS Unmapped Value Handling option
determines how unmapped values are handled during MDIS processing.
The field-level settings for destination fields are summarized in Table 33. Each field-level setting can be set on a field-by-field basis.
226 MDM Import Manager Reference Guide
Table 33. Field-Level Settings for Destination Fields
Field Setting Must Be Set… Description
NULL Interpretation Option (see page 285)
NULL Interpretation Any time prior to import Specifies whether NULL source values are ignored or used to overwrite existing destination values.
Multi-Valued Update Option (see page 286)
Multi-Valued Update Any time prior to import Specifies whether source values are appended to or replace existing multi-values.
Qualified Update Option (see page 287)
Qualified Update Any time prior to import Specifies whether source subrecords are appended to, replace, or update existing qualified links.
Data Group Display Option (see page 289)
Data Group Display Prior to Field Mapping Specifies whether the Data Groups hierarchy is hidden or displayed in the Destination Values grid for object fields.
MDIS Unmapped Value Handling Option (see page 291)
MDIS Unmapped Value Handling
Any time prior to import Specifies how to handle unmapped
values when running MDIS with a .map file.
HIERARCHY VALUES
Recall that MDM supports hierarchical, parent/child relationships among records with a special hierarchy table type and with lookup fields into the hierarchy table. By contrast, a typical relational DBMS does not offer native support for representing these hierarchical relationships.
As a result, importing hierarchy values into MDM encounters a variety of challenges that depend upon the various ways the hierarchy values are represented in the source data.
In addition, regardless of how the hierarchy values are represented, how they are interpreted by MDM depends upon whether you are importing into a hierarchy lookup field or into the hierarchy table itself.
NOTE ►► Whether you are importing into a hierarchy lookup field or into the hierarchy table itself specifically affects how NULLs should be interpreted (see “The Hierarchy NULL Handling Option” on page 233 for more information).
These various challenges and considerations are described more fully in the following subsections.
MDM Import Manager Reference Guide 227
Hierarchy Representation in Source Data
Since a typical DBMS does not offer native support for representing hierarchical relationships, hierarchy values can appear in source data in at least three different ways:
Delimited within a single field. A single source data field may be
used to represent each hierarchy value as multiple delimited values
(with or without trailing delimiters). You can then use the Split into Hierarchy command to expand the delimited values into a hierarchy (see “Splitting into Hierarchy” on page 146 for more information).
Category or Category
Printers Printers \ \
Printers \ Dot Matrix Printers \ Dot Matrix \
Printers \ Dot Matrix \ 300 DPI Printers \ Dot Matrix \ 300 DPI
Printers \ Dot Matrix \ 600 DPI Printers \ Dot Matrix \ 600 DPI
Printers \ Dot Matrix \ 1200 DPI Printers \ Dot Matrix \ 1200 DPI
Expanded into multiple fields. Multiple source data fields may be
used to represent each hierarchy value, with each field representing a single level of the hierarchy. You can then use nested partitions to combine the values from each field into a hierarchy (see “Source Field Partitioning” on page 174 for more information).
Category 1 Category 2 Category 3
Printers
Printers Dot Matrix
Printers Dot Matrix 300 DPI
Printers Dot Matrix 600 DPI
Printers Dot Matrix 1200 DPI
Pair of parent/child fields. A pair of source data fields may be used
to represent each hierarchy node value, with one field for the node
and one for its parent node. You can then use the Create Hierarchy Field command to materialize the hierarchy (see “Building a Hierarchy” on page 156 for more information).
Parent Category
Printers
Printers Dot Matrix
Dot Matrix 300 DPI
Dot Matrix 600 DPI
Dot Matrix 1200 DPI
NOTE ►► The three source data examples above represent exactly the same set of internal- and leaf-node hierarchy values.
228 MDM Import Manager Reference Guide
Importing Into a Hierarchy Lookup Field
When you are importing into a hierarchy lookup field, the source data represents the assignment of lookup values to nodes of the hierarchy.
Since lookup values can be assigned only to leaf nodes in the hierarchy, the source data should contain only leaf-node references, as illustrated using multiple fields below:
… Category 1 Category 2 Category 3 …
Printers Dot Matrix 300 DPI
Printers Dot Matrix 600 DPI
Printers Dot Matrix 1200 DPI
NOTE ►► The fields representing the hierarchy node values are contained in source records that also include other fields for the records of the table containing the hierarchy lookup field.
NOTE ►► When importing into a hierarchy lookup field, you can: (1) map to existing leaf nodes in the hierarchy; and (2) add new internal and leaf nodes to the hierarchy.
NOTE ►► You can optionally map the source field representing the hierarchy to a destination hierarchy lookup field. If the field is mapped, then: (1) value mapping is required; (2) you can map only leaf nodes; and (3) all of the leaf nodes must be mapped to proceed with the import. This is consistent with the standard requirement that all values of a lookup field must be mapped.
NOTE ►► Internal-node references in the source data may or may not be acceptable when importing to the hierarchy lookup field, and must be properly interpreted using the Hierarchy NULL Handling option (see “The Hierarchy NULL Handling Option” on page 233 for more information).
Importing Into the Hierarchy Lookup Table
When you are importing into the hierarchy lookup table itself, the source data represents the actual internal- and leaf-node records of the hierarchy table, as illustrated using multiple fields below:
… Category 1 Category 2 Category 3 …
Printers
Printers Dot Matrix
Printers Dot Matrix 300 DPI
Printers Dot Matrix 600 DPI
Printers Dot Matrix 1200 DPI
MDM Import Manager Reference Guide 229
NOTE ►► The fields representing the hierarchy node values are contained in source records that also include other fields for the records of the hierarchy table itself.
NOTE ►► When importing into the hierarchy table itself, you can: (1) map to existing internal and leaf nodes; and (2) add new internal and leaf nodes to the hierarchy.
NOTE ►► You must map the source field representing the hierarchy to the primary display field of the hierarchy table. Once the field is mapped: (1) value mapping is optional; (2) if you do map values, you can map both internal and leaf nodes; and (3) all of the internal and leaf nodes must be mapped to proceed with the import.
NOTE ►► With respect to record matching: (1) if you do map values,
the [Remote Key] field can be used as a matching field only if it is combined with the primary display field; and (2) if you do not map
values, the [Remote Key] field can be used as the matching field by
itself (see “Lookup Table Imports Using the [Remote Key] Field” on page 414 for more information).
NOTE ►► Internal node value mapping allows you not only to import a complete hierarchy consisting of internal and leaf nodes, but also to populate for internal nodes: (1) the other fields of the hierarchy table records (i.e. fields other than the primary display field); (2) other
language layers of all of the fields; and (3) the [Remote Key] field.
NOTE ►► Internal-node references in the source data are acceptable when importing to the hierarchy table, and must be properly interpreted using the Hierarchy NULL Handling option (see “The Hierarchy NULL Handling Option” on page 233 for more information).
NOTE ►► The Replace import action is disabled for import into the hierarchy table itself; the Import Manager and MDIS will substitute
Update (All Mapped Fields) in old maps that have the Replace import action.
230 MDM Import Manager Reference Guide
NULL VALUES
In a typical DBMS, a field that is empty is said to have a NULL value, which corresponds to missing or inapplicable data.
In the MDM Import Manager, a NULL value is displayed in trees and grids as <NULL> to distinguish it from an actual field value, and appears in the Value Conversion and Mapping grids as follows:
Source Values grid. A <NULL> appears in the list of distinct source
values for a particular source field only if one or more source data records contain no value for the field.
Destination Values grid. For convenience, a <NULL> value always
appears in the list or tree of distinct destination values as a target for mapping, regardless of whether or not the corresponding destination field actually includes empty values in any of its records.
NULL values pose a number of challenges when mapping and converting source data values, as described in the following sections.
Mapping To and From NULL Values
You can map to and from NULL values as follows:
NULL to NULL. In the most straightforward case, you can simply
value map the <NULL> source value that represents inapplicable or missing data to the <NULL> destination value. The <NULL> source value is ignored during the import process.
NULL to non-NULL. By contrast, if the <NULL> represents a default
value, or you know what the missing data should be, you can value map the <NULL> to any existing non-NULL destination value, or even change the converted value for the <NULL> source value and
then Add a new destination data value based on the converted value.
Non-NULL to NULL. Finally, if one or more non-NULL source data
values need to be ignored as part of the import, you can value map them to the <NULL> destination value.
NOTE ►► The MDM Import Manager interprets a field value containing only whitespace (i.e. spaces or tabs) as a NULL value.
NOTE ►► When you map a source value (either NULL or non-NULL) to the <NULL> destination value, the NULL is either ignored or used to set existing destination values to NULL based on the setting of the NULL Interpretation option (see “The NULL Interpretation Option” on page 236 for more information).
MDM Import Manager Reference Guide 231
Converting To and From NULL Values
You can also convert source data values to and from NULL simply by editing the corresponding value in the Converted Value column within the Source Values grid.
By default, <NULL> is converted to <NULL>. You can also set the converted value to a non-NULL value if the NULL represents a data value; alternatively, you can set to NULL the converted value for any
other non-NULL value using the Set Value to NULL value conversion filter to ignore it as part of the import.
Missing Source Data Fields
Sometimes, a field itself will be missing entirely from the source data, so that in effect, all of its values are NULL. For example, the source data from a particular manufacturer may not include a manufacturer field.
In these cases: (1) use the Add Field command in the Source Hierarchy tree to add a new field whose list of values will contain the single <NULL> value; (2) map the field to appropriate destination field; and (3) map or convert the <NULL> value to the appropriate destination value.
The NULL Handling Options
In the MDM Import Manager, implicit and explicit NULL values also arise in additional situations, as follows:
Multiple delimited values. When a single field contains multiple
delimited values and you map the field to a multi-valued field, empty positions in the delimited field values represent implicit NULLs.
Splitting into hierarchy. When you split a field containing multiple
delimited values using the Split into Hierarchy command, empty positions in the delimited field values again represent implicit NULLs that become NULL-valued nodes in the resulting hierarchy.
Nested partitions. When you combine fields with a nested partition,
explicit NULL values in any of the fields again become NULL-valued nodes in the resulting hierarchy.
Combined partitions. When you combine fields with a combined
partition, explicit NULL values in any of the fields become empty positions in the resulting set of delimited values; when you map the field combination to a multi-valued field, these empty positions again represent implicit NULLs.
Depending upon the circumstances and how you would like the MDM Import Manager to interpret the source data, you can specify that these implicit and explicit NULLs be either preserved or ignored.
232 MDM Import Manager Reference Guide
How to handle NULLs is based on one of the two NULL handling options (one applicable to handling delimited flat values and one to handling hierarchy values), as described in the following sections.
NOTE ►► The Map Fields/Values tab populates the Source Values grid with values based on the current NULL-handling settings, which affects not only how original values are interpreted and how converted values are generated, but also the set of original values, and therefore must be properly set prior to mapping the field.
The Flat NULL Handling Option
The Flat NULL Handling option applies when you have a source field that contains multiple delimited values or when you combine fields with a combined partition. It has two settings, as described in Table 34.
Table 34. The Flat NULL Handling Option
Setting Description
Preserve NULLs Every NULL value is preserved in the list of distinct values for the field, including implicit NULLs that appear as empty positions in a set of delimited values.
Ignore NULLs Every NULL value is ignored in the list of distinct values for the field, with empty positions collapsed and trailing NULLs eliminated in a set of delimited values.
NOTE ►► Unlike the four settings of the Hierarchy NULL Handling option, which have various subtleties, the two settings of the Flat NULL Handling option are simple and either preserve or ignore all NULLs.
The default flat NULL-handling setting is to preserve NULLs, which most accurately reflects the original source data. By contrast, ignoring NULLs can be used to efficiently collapse inconsistencies in the source data when they do not reflect real value differences. Whether to preserve or ignore NULLs depends in large part on whether the position of data values matters within the multiple delimited values.
For each source field, you can set the Flat NULL Handling option (see “Setting the Flat NULL Handling Option” on page 279), which depends upon whether the NULLs represent missing or inapplicable values.
NOTE ►► See “Source Field-Level Settings” on page 224 for more information about all of the source field-level settings.
Table 35 illustrates the two alternative NULL-handling settings for the values in two fields combined with partitioning. The same NULL-handling behavior also applies to the values in a single field that already contains delimited values and is mapped to a multi-valued field.
MDM Import Manager Reference Guide 233
Table 35. Preserve and Ignore NULL Handling for Delimited Flat Values
Field1 Field2 Preserve NULLs Ignore NULLs
value1 value2 value1; value2 value1; value2
value1 <NULL> value1; value1
<NULL> value2 ; value2 value2
<NULL> <NULL> ; <NULL>
Delimited flat values occur when mapping to a multi-valued field and with combined partitions.
The interpretation of implicit NULLs in a string of delimited values depends upon the value of the Multi-Valued delimiter string.
The field containing delimited values may itself have been created with a combined partition, in which case the Multi-Valued delimiter string and Partition Combine delimiter string should be the same.
For Ignore NULLs, all implicit NULLs are collapsed and eliminated; when all positions are empty, the result is <NULL>.
Ignoring NULLs may shrink the set of distinct values because differences are eliminated and duplicate values then collapsed.
Do not ignore NULLs if position matters because different source values with the same sequence of values in different positions would look like the same value after empty positions were eliminated, and would then be collapsed into a single distinct value.
The Hierarchy NULL Handling Option
The Hierarchy NULL Handling option applies when you have a source field that is split into hierarchy or when you combine multiple fields with a nested partition. It has four settings, as described in Table 36.
Table 36. The Hierarchy NULL Handling Option
Setting Description
Preserve NULLs Every NULL value is preserved, including implicit NULLs that appear as NULL node values.
Ignore Non-Leaf Trailing NULLs
Preserves internal NULLs and those trailing NULLs necessary to interpret all values as leaf nodes.
Preserve Leaf NULLs as NULLs
Trailing NULLs preserved as nodes named NULL.
Preserve Leaf NULLs as Internal Leaf Nodes
Trailing NULLs preserved as internal leaf nodes.
Ignore All Trailing NULLs
Preserves internal NULLs and ignores all trailing NULLs, so that internal node values are preserved.
NOTE ►► The Ignore Non-Leaf Trailing NULLs setting has two subsettings whose particular use depends upon whether or not you want to preserve inner node assignments in your hierarchy.
Consider the hierarchy values represented by the three fields in the three records below:
234 MDM Import Manager Reference Guide
… Field 1 Field 2 Field 3 …
A
A B
A B C
In the simplest analysis, ignoring NULLs using the hierarchy NULL-handling setting boils down to whether the three hierarchy values above should be: (1) NULL-extended for interpretation as three leaf nodes; or (2) interpreted instead as two internal nodes and one leaf node.
The three “ignore trailing NULLs” settings are summarized in Table 37.
Table 37. The Three “Ignore Trailing NULLs” Settings
Item
Ignore Non-Leaf Trailing NULLs: Preserve Leaf NULLs… Ignore All
Trailing NULLs As NULLs As Internal Leaf Nodes
Used when importing to…
A hierarchy lookup field.
The hierarchy lookup table.
Ignores… Those trailing NULLs not required for interpretation as a leaf node (based on other leaf nodes at same level).
All trailing NULLs.
Views each source value as…
A leaf-node assignment in the lookup hierarchy.
A leaf- or inner node assignment in the lookup hierarchy.
An internal- or leaf- node record in the hierarchy table.
TIP ►► Recall that for proper organization of the records within an MDM repository, a hierarchy lookup field can normally be assigned only to the value of a leaf node in the hierarchy. If, however, the inbound records have inner node assignments in which the lookup field is assigned to the value of an internal (i.e. non-leaf) node and you wish to preserve these inner node assignments during inbound processing
of records, you can use the Ignore Non-Leaf Trailing NULLs (Preserve Leaf NULLs as Internal Leaf Nodes) setting to create internal leaf nodes instead of nodes named NULL.
The default hierarchy NULL-handling setting is to preserve all NULLs, which most accurately reflects the original source data. Alternatively, you can ignore some or all trailing NULLs using the three “ignore trailing NULLs” settings. In all cases, internal NULLs are always preserved.
For each source field, you can set the Hierarchy NULL Handling option (see “Setting the Hierarchy NULL Handling Option” on page 280), which depends upon whether the NULLs represent missing or inapplicable values, and whether you are importing into a hierarchy lookup field or into the hierarchy table itself.
NOTE ►► See “Source Field-Level Settings” on page 224 for more information about all of the source field-level settings.
MDM Import Manager Reference Guide 235
Table 38 illustrates the three alternative NULL-handling settings for the values of three fields in six records combined with nested partitioning. The same NULL-handling behavior also applies to the values in a single field that already contains delimited values and is split into a hierarchy.
NOTE ►► Unlike the Flat NULL Handling option, which looks at each individual record by itself, the Hierarchy NULL Handling option looks at the entire set of source records and synthesizes the delimited values across the entire set into a single integrated hierarchy.
DATA INTEGRITY ►► Regardless of the hierarchy NULL-handling setting, internal NULLs are always preserved as nodes because ignoring them would collapse hierarchy structure. Similarly, with the
Ignore Non-Leaf Trailing NULLs settings, leaf-node NULLs are always preserved and/or automatically added as nodes to the hierarchy when other nodes at the same level of the same branch have a non-NULL value either: (1) because without the trailing NULL, the full-path value would represent an internal rather than a leaf node; or (2) to preserve the inner node assignment using an internal leaf node.
Table 38. Preserve and Ignore NULL Handling for Hierarchy Values
Field1 Field2 Field3 Preserve NULLs
Ignore Non-Leaf Trailing NULLs: Preserve Leaf NULLs… Ignore All
Trailing NULLs as NULLs
as Internal Leaf Nodes
<NULL> <NULL> <NULL> Field 1
├ NULL
│ └ NULL
│ └ └ NULL
└ Printers
├ Dot Matrix
│ ├ 300 DPI
│ ├ 600 DPI
│ ├ 1200 DPI
│ └ NULL
└ NULL
└ NULL
Field 1
├ NULL
└ Printers
├ Dot Matrix
│ ├ 300 DPI
│ ├ 600 DPI
│ ├ 1200 DPI
│ └ NULL
└ NULL
Field 1
├ NULL
└ Printers
├ Dot Matrix
│ ├ 300 DPI
│ ├ 600 DPI
│ ├ 1200 DPI
│ └ [Dot Matrix]
└ [Printers]
Field 1
├ NULL
└ Printers
└ Dot Matrix
├ 300 DPI
├ 600 DPI
└ 1200 DPI
Printers <NULL> <NULL>
Printers Dot Matrix <NULL>
Printers Dot Matrix 300 DPI
Printers Dot Matrix 600 DPI
Printers Dot Matrix 1200 DPI
Hierarchy values occur: (1) when splitting into hierarchy; and (2) with nested partitions.
The root node of the hierarchy is the field name itself.
The individual hierarchy values across the set of records are synthesized into a single integrated hierarchy rather than into a list of distinct values.
The string “NULL” (as opposed to the <NULL> value) is used as the node value when the source data has no value for an internal node.
For the two Ignore Non-Leaf Trailing NULLs settings, inner node assignments are assigned to either: (1) a leaf node named NULL; or (2) an internal leaf node.
For all three of the Ignore Trailing NULLs settings, when the entire path of node values is <NULL>, a single trailing NULL node is preserved under the root.
236 MDM Import Manager Reference Guide
The NULL Interpretation Option
Just as the NULL Handling options determines how NULLs are reflected in source values, the NULL Interpretation option determines how NULLs are reflected in destination values.
Consider first that when creating new destination records or replacing existing destination records, importing NULL source data values into destination fields can be handled in a single way: the destination field value corresponding to the NULL source value remains empty.
By contrast, when updating existing destination records as part of the import – and in particular when updating non-NULL destination fields that already have one or more data values – importing NULL source values can be handled in one of two ways.
Specifically, the set of incoming NULL source values can either be: (1) ignored, so that corresponding destination field values in the existing destination records remain undisturbed; or (2) used to overwrite the corresponding destination field values in the existing destination records, wiping them out so that destination field values becomes empty, as described in Table 39.
Table 39. The NULL Interpretation Option
Setting Description
Ignore NULLs NULL source values are ignored, preserving existing data values in existing destination records.
Overwrite Existing Values with NULLs
NULL source values are used to overwrite existing destination field values, wiping them out.
NOTE ►► Among the pair of update import actions, the NULL
Interpretation option only affects the Update (All Mapped Fields) action,
because the Update (NULL Fields Only) action updates only fields that
do not yet have data values, making the Ignore and Overwrite settings equivalent. (See “Import Actions” on page 346 for more information about the import actions of the Record Matching tab.)
The default NULL-interpretation setting is to ignore NULLs. By contrast, using NULLs to overwrite existing destination field values can be used to wipe out values that are no longer valid.
For each destination field, you can set the NULL Interpretation option (see “Setting the NULL Interpretation Option” on page 285) based on the requirements of the situation.
NOTE ►► See “Destination Field-Level Settings” on page 225 for more information about all of the destination field-level settings.
MDM Import Manager Reference Guide 237
THE CASE SENSITIVITY OPTION
Depending upon the circumstances and how you would like the MDM Import Manager to determine the set of distinct values for a field, you can specify that case differences between different instances of the same value are either ignored or recognized, as described in Table 40.
Table 40. The Case Sensitivity Option
Setting Description
Case Insensitive Case is ignored. Instances of the same value with case differences are collapsed into a single value in the list of distinct values for the field.
Case Sensitive Case is recognized. Every distinct value – including instances of the same value with case differences – is preserved in the list of distinct values for the field.
The default Case-Sensitivity setting is to be case insensitive, which ignores case and conveniently collapses inconsistencies in the source data when case differences do not reflect real value differences (such as with the values GENERAL MOTORS and General Motors, which represent the same value). By contrast, case sensitivity recognizes case and allows you to distinguish between different source values when case differences represent value differences (such as with the values 6 mHz and 6 MHz, which truly represent different values).
For each field, you can set the Case Sensitivity option (see “Setting The Case Sensitivity Option” on page 278), which depends upon whether case differences represent legitimate value differences.
NOTE ►► See “Source Field-Level Settings” on page 224 for more information about all of the source field-level settings.
NOTE ►► The Map Fields/Values tab populates the Source Values grid with values based on the current case-sensitivity setting, which affects the set of original values, and must be properly set prior to: (1)
displaying values using the Display Distinct Values command; (2)
partitioning the field; (3) splitting the field using the Split into Hierarchy
or Split into Multiple Fields commands; or (4) mapping the field.
NOTE ►► For SQL Server or Oracle data sources, case sensitivity of the source field is determined by database collation and cannot be changed by the MDM Import Manager.
238 MDM Import Manager Reference Guide
DATA INTEGRITY ►► If the Case Sensitivity option is set to Case Insensitive and you map a source field to a destination measurement field or attribute whose corresponding dimension has case-sensitive units (e.g. Frequency, Power, and Pressure, which have units with both “milli-“ and “mega-“ prefixes), MDM will offer to change the setting of the option for you automatically.
FIELD MAPPING DELIMITERS
The current field mapping delimiters determine how the MDM Import Manager interprets source data values as part of the value conversion and mapping process, and must be properly set prior to mapping a source field to a destination field.
The two field mapping delimiters include:
Multi-Valued delimiter. MDM identifies the individual delimited
values within a single field when mapping to a multi-valued field using the Multi-Valued delimiter string.
Coupled Numeric delimiter. MDM identifies the coupled numeric
value pairs when mapping to a coupled numeric attribute using the Coupled Numeric delimiter string.
NOTE ►► The delimiter for a compound lookup field is ", " (comma and a space) and cannot be changed.
For each field, you can specify the field mapping delimiters (see “Setting the Field Mapping Delimiters” on page 276).
NOTE ►► See “Source Field-Level Settings” on page 224 for more information about all of the delimiters and source field-level settings.
VALUE CONVERSION FILTERS
Not only can each source value be manually edited, built-in and custom-designed value conversion filters can automatically convert or reformat a single value or the entire set of distinct values with a single command.
For each field, you can apply one or more value conversion filters at the field level, and for each value, you can apply one or more value conversion filters at the individual value level (regardless of whether or not value mapping is required), as follows:
Field level. When you apply value conversion filters at the field level,
the conversions are applied sequentially to each converted value that has not been individually edited or converted at the value level, starting with the original value.
MDM Import Manager Reference Guide 239
Value level. When you apply a value conversion filter at the value level, the conversion is applied to the converted value resulting from
the prior application of field-level filters, value-level filters, and manual edits.
NOTE ►► When you first map a field, each value in the Converted Value column appears in gray to indicate that it is inheriting value conversion filters from the field level. When you apply a manual edit or a value-conversion filter to an individual value, the Converted Value then appears in black to indicate that you have overridden inheritance
from the field level. You can use the Restore Converted Value command to restore inheritance for a value. (See “Source Values Grid” on page 296 for more information about the Source Values grid.)
NOTE ►► Value conversions are saved in a map file and applied during future imports as follows: (1) at the field level, the filters themselves are saved and applied; and (2) at the value level, the final translation of each original value to a converted value (rather than the sequence of edits and conversions) is saved and applied.
NOTE ►► No conversion filter is necessary to trim leading and trailing spaces from source values, as Import Manager does this automatically when importing values into an MDM repository.
NOTE ►► See “Source Field-Level Settings” on page 224 for more information about all of the source field-level settings.
Value conversion filters automate repetitive and error-prone transformations, eliminating manual typing and the possibility of user error. They also allow powerful reformatting algorithms to be applied to an entire group of values.
The value conversion filters affect the Converted Values and are summarized in Table 41.
240 MDM Import Manager Reference Guide
Table 41. Value Conversion Filters
Filter Description
Restore Converted Value*
Restores inheritance of the field-level value conversion filters.
Set to Original Value* Sets the selected values to their original unconverted values.
Set Value to NULL
Sets the selected values to <NULL>.
When value mapping is not required, a <NULL> value is deemed to be converted
Clear Value
Clears the selected values.
When value mapping is not required, an empty value is deemed to be not yet converted
Apply Operator…
Applies the numeric or string operator to the selected values:
+ – add the specified value
- – subtract the specified value
* – multiply by the specified value
/ – divide by the specified value
Equals – set to the specified value
NULL Equals – set to the specified value only if NULL
Round – round to the specified number of decimal places
Ceiling – ceiling to the specified number of decimal places
Truncate – truncate to the specified number of decimal places
Prepend – prepend the specified string
Append – append the specified string
Truncate – truncate the string to the specified width
Set Measurement Unit Sets the unit for all of the selected measurement values, overriding the existing unit, if any. Unit is chosen from the cascading menu of applicable units based on the dimension of the mapped destination.
Add Missing Unit Adds the unit to just those selected measurement values that do not already have a unit. Unit is chosen from the cascading menu of applicable units based on the dimension of the mapped destination.
Convert to Unit Converts the selected measurement values (which must already have a unit) to the target unit. Unit is chosen from the cascading menu of applicable units based on the dimension of the mapped destination.
Normalize Normalizes the selected text values. Normalization returns the capitalized alphanumeric string from which all punctuation other than the underscore (_) and other “unwanted” characters have been removed.
Change Case
Changes the case of the selected text values:
UPPER CASE lower case
Sentence case Title Case tOGGLE cASE
Replace… Finds and replaces strings in the selected text values.
Accept Truncated Value
Accepts the selected text values that were truncated because they were too long for the width of the mapped destination text field.
Expand Year Range Expands the year range for the selected text values into multiple year values. For example, the single value 1997-94 expands to the four distinct values 1994, 1995, 1996, and 1997.
MDM Import Manager Reference Guide 241
Filter Description
Expand Month* Expands the month qualifier for the selected text values into multiple month values. For example, the value 1999,6-2001,8 expands to June 1999, NULL, and August 2001.
* Value-level filter only.
LOOKUPS CONSISTING OF MULTIPLE DISPLAY FIELDS
Recall that you must value map from a source field to the values of a destination lookup field, as shown in Figure 156.
Source: Destination:
NAME ► Name [lookup]
Joe Williams ► Joseph Williams
Jim Williams ► James Williams
Dave Rodgers ► David Rodgers
Dave Smith ► David Smith
Figure 156. Simple value mapping to destination lookup field
Compound Lookup Fields
When the underlying lookup table has just a single display field, each lookup field value is simply the value of that display field.
However, when the lookup table has multiple display fields, the lookup field is in effect a compound field, and each lookup field value is the combination of the values of the display fields, as shown in Figure 157.
Destination: Lookup display fields:
Name [lookup] First Name Last Name
Joseph Williams ◄ Joseph Williams
James Williams ◄ James Williams
David Rodgers ◄ David Rodgers
David Smith ◄ David Smith
Figure 157. Corresponding lookup table with multiple display fields
NOTE ►► Normally, when a single source field corresponds to multiple destination fields, you must combine the destination fields with partitioning prior to field and value mapping. However, partitioning is not necessary with a compound lookup field, which automatically combines its underlying lookup table’s multiple display fields.
242 MDM Import Manager Reference Guide
NOTE ►► The Add Value commands for the Destination Values grid are always disabled when the destination is a compound lookup field.
Mapping From a Single Corresponding Source Field
When just a single source field corresponds to a compound lookup field, you have no choice but to field and value map the source field directly to the compound lookup field. For value mapping purposes, the compound lookup field then behaves somewhat like an MDM multi-valued field, in that both consist of multiple values for each record.
Consequently, when you map a single source field to a compound destination lookup field, MDM uses the delimiter ", " (comma and space) to separate each source data value into the multiple delimited values corresponding to the value combinations of the compound lookup field. This delimiter cannot be changed.
NOTE ►► The edit controls in the Converted Value column of the Source Values grid are disabled and cannot be opened for editing any of the individual delimited values.
NOTE ►► The Add button between the Value Mapping grids and the
Add Value commands for the Destination Values grid are also both disabled.
TIP ►► If the source value combinations represented by the single source field do not all correspond to existing values of the destination
lookup field, you must use the Split into Multiple Fields command to split the delimited source values into multiple source fields, and then map them to the compound lookup field, described in the next section.
NOTE ►► Whereas a multi-valued field stores multiple values in a single field, a compound lookup field corresponds to a multi-field value combination across each of the multiple lookup table display fields.
Mapping From Multiple Corresponding Source Fields
When multiple source fields correspond to a compound lookup field –and each of the source fields corresponds to one of the individual display fields of the underlying lookup table – the MDM Import Manager offers a nested, two-step, “map-within-a-map” mechanism that allows you to more efficiently map source values to the value combinations represented by the compound lookup field.
Consider Figure 158, in which multiple source fields correspond to a compound lookup field. Specifically: (1) multiple source fields (FIRST
MDM Import Manager Reference Guide 243
NAME + LAST NAME) together correspond to a destination lookup field (Name); (2) the compound lookup field represents the combination of the underlying lookup table’s multiple display fields (First Name + Last Name); and (3) each source field corresponds to one of the display fields (FIRST NAME = First Name and LAST NAME = Last Name).
Source: Destination: Lookup display fields:
FIRST NAME LAST NAME ► Name [lookup] First Name Last Name
Joe Williams Joseph Williams ◄ Joseph Williams
Jim Williams James Williams ◄ James Williams
Dave Rodgers David Rodgers ◄ David Rodgers
Dave Smith David Smith ◄ David Smith
Figure 158. Multiple source fields and multiple display fields
Promoted Lookup Table Display Fields
To facilitate map-within-a-map to a compound lookup field, each of the individual display fields of its underlying lookup table is automatically promoted and appears in the Destination Fields grid, just as if it were a
field of the destination table, and its individual values appear in the Destination Values grid (see “Value Mapping and Lookup Table Display Fields” on page 247).
NOTE ►► Promoted lookup table display fields appear in the Destination Fields grid as “display field n [lookup field]” and compound lookup fields appear as “lookup field [display field 1; display field n]” (where “display field n” is the name of the nth lookup table display field and “lookup field” is the name of the compound lookup field).
Map-Within-a-Map (Create Compound Field Command)
Assuming the one-to-one correspondence between source fields and promoted display fields, you can then perform the map-within-a-map as follows: (1) map each source field to one of the display fields; (2) for each source field, convert its values, or optionally map to the values of a
Text display field; (3) use the Create Compound Field command to create a compound source field for mapping to the compound destination lookup field; (4) map the compound source field to the destination lookup field; and (5) automap the compound source field’s source value combinations to the lookup field values.
244 MDM Import Manager Reference Guide
NOTE ►► Normally, when multiple source fields correspond to a single destination field, you must combine the source fields with partitioning prior to field and value mapping. A combined partition merges original source data values, which makes value mapping of the
value combinations more difficult. By contrast, the Create Compound Field command merges converted or mapped values, so that validation
can be performed at the individual field level; moreover, Automap of the compound source field can then have a 100% success rate (for value combinations that already exist in the destination field) since the components of the value combination have already been validated.
Qualified Lookup Fields and Lookups-Within-Lookups
The map-within-a-map mechanism is especially useful – and its benefits best illustrated – when the mapped destination field is a qualified lookup field where: (1) the number of value combinations represented by the records of the underlying qualified lookup table is very large; and (2) the display fields are themselves nested lookups-within-lookups.
NOTE ►► With nested lookups, map-within-a-map corresponds logically to the multi-level search-within-a-search offered by the MDM Client, both of which use the nested lookups to restrict the larger set of lookup table records.
Consider Figure 159, which lists some of the vehicles in the Vehicles qualified table, which contains forty thousand vehicle specifications.
Vehicles
1996; Toyota; Celica
1997; Toyota; Celica
1998; Toyota; Celica
1999; Toyota; Celica
1996; Ford; Mustang
1997; Ford; Mustang
1997; Ford; Pinto
1998; Ford; Pinto
1998; Chevrolet; Impala
1999; Chevrolet; Impala
2000; Chevrolet; Impala
1997; Chevrolet; Caprice
1998; Chevrolet; Caprice
1999; Chevrolet; Caprice
2000; Chevrolet; Caprice
Figure 159. Records of a qualified table
MDM Import Manager Reference Guide 245
The records above correspond to the sets of values for each of the Vehicles table display / nested lookup fields shown in Figure 160.
Year [l] Model [l]
1996 Make [l] Celica
1997 Toyota Mustang
1998 Ford Pinto
1999 Chevrolet Impala
2000 Caprice
Figure 160. Nested lookup values of each qualified table display field
Now, suppose the three source data fields YEAR, MAKE, and MODEL have the sets of unnormalized values shown in Figure 161.
MAKE
YEAR Toyota
2000-1998 Toyota MODEL
97-98 Ford Celica
2000-1997 Ford Motor Mustang
97-96 Chevy Pinto
1996-1998 Chevrolet Impala
1999 Caprice
Figure 161. Source data fields and values
Using a combined partition to merge the values of YEAR, MAKE, and MODEL as a prelude to value mapping to the lookup values of the Vehicles table would combine their unmapped values, and in the process create multiple instances of each unnormalized value across the set of value combinations, as shown in Figure 162. As a result, automapping the value combinations would be impossible, and manually mapping them against the lookup values corresponding to the forty thousand records of the Vehicles table exceptionally tedious.
YEAR; MAKE; MODEL
2000-1998; Chevy; Impala
97-98; Ford Motor; Pinto
2000-1997; Chevrolet; Caprice
97-96; Ford; Mustang
1996-1998; Toyota; Celica
1999; Toyota; Celica
Figure 162. Combined partition of source data values
246 MDM Import Manager Reference Guide
This is where map-within-a-map comes into play. Rather than partition, you can first field and value map each of the source data fields to the corresponding qualified table lookup fields, each of which contains a much smaller number of distinct values than the qualified table itself, so that value mapping is much simpler. For example, YEAR maps to Year, and the source value 2000-1998 maps to the multiple values 1998, 1999, and 2000, while MAKE maps to Make, and the two source values Chevy and Chevrolet map to Chevrolet.
NOTE ►► Each promoted lookup table display field that is itself a nested lookup field allows multi-valued mapping, regardless of whether it is itself multi-valued.
After the source data values have been mapped, you can then use the
Create Compound Field command to combine the normalized mapped values (rather than the unnormalized original values) into a single combination value that corresponds to the multiple display fields of each record of the qualified table.
NOTE ►► Map-within-a-map allows you to normalize the source data values against the nested lookup values before the compound field creates multiple instances of each value, and then combine not the original source values but rather the mapped values, so that the strings of the combined values precisely correspond to the display fields of the qualified table, for an Automap that will have a 100% success rate.
In the automotive example above, the benefit of map-within-a-map over partitioning is more than just the convenience of being able to Automap rather than manually map source value combinations against a large set of destination values. Rather, map-within-a-map is essential when some of the individual source values themselves must be mapped to multiple destination values, so that map-within-a-map actually expands the number of source value combinations, sometimes dramatically.
Finally, consider Figure 163, another set of automotive data, in which not only the YEAR but also the MODEL must be expanded into multiple values.
YEAR MAKE MODEL
1997-94 Chevy ALL
1996-93 Olds Cutlass
1998-97 Chevy Camaro
2000-1997 Caddy Eldorado
1994 Buick ALL
Figure 163. Map-within-a-map with value expansion
In the first source record, the YEAR would be value mapped to 1994, 1995, 1996, and 1997, and the MODEL mapped to all Chevrolet
MDM Import Manager Reference Guide 247
models, and the value combination represented by the single record automatically expanded to perhaps tens or even hundreds of value combinations for automapping.
Value Mapping and Lookup Table Display Fields
Recall that a lookup table is a valid table, its display field or display field combination defines the legal values of the corresponding lookup field into the table, and value mapping to destination lookup fields is therefore required to enforce source data value validation against the set of legal destination values represented by the lookup table records.
In addition, value mapping is optional to fixed-width Text fields that are: (1) promoted lookup table display fields when the current table is the main table and the lookup table has multiple display fields; (2) the primary display field when the current table is a hierarchy lookup table; and (3) actual display fields when the current table is the lookup table.
While not absolutely necessary for source data value validation, value mapping to Text display field values affords less opportunity for data entry and editing error and supports more consistent normalization than simple value conversion. In particular, value mapping: (1) saves typing when normalizing against existing destination values; and (2) requires explicit action and precise definition when adding new values.
NOTE ►► When the subtable is not a lookup table (such as a subtable of manufacturer part numbers and part information), the table is not a valid table, no validity checking against the display field or display field combination is necessary, and hence value mapping to the Text display fields is neither required nor permitted.
When the lookup field and corresponding lookup table consist of multiple display fields, an additional objective is to ensure the validity of source data values not only against the individual values of each of the display fields, but also against the value combinations represented by the display fields taken together, as follows:
Main table. When the current table is the main table, validate source
values against: (1) the individual display field values by permitting optional value mapping to each of the promoted Text display fields; and (2) display field value combinations by requiring value mapping
of the compound source field created with Create Compound Field.
Lookup table. When the current table is a lookup table, validate
source values against: (1) the individual display field values by permitting optional value mapping to each of the Text display fields; and (2) display field value combinations by requiring record matching against the display fields as a matching field combination.
248 MDM Import Manager Reference Guide
NOTE ►► When the lookup table is a hierarchy table, value mapping is optional to its primary display field (which appears as a hierarchy value) and also to its other display fields (which appear as flat values); when the hierarchy lookup table is the current table, record matching is
required only to its primary display field and/or the [Remote Key] field
(see “Lookup Table Imports Using the [Remote Key] Field” on page 414 for more information).
For example, consider a main table with a Manufacturer lookup field into the Manufacturers lookup table, which has two display fields: (1) Name (a fixed-width Text field); and (2) Code (an Integer field).
In this case, you can optionally value map to the Name field; when the current table is the main table, value mapping is required from the compound source field to Manufacturer, and when the current table is the Manufacturers table, record matching is required against Name and Code as a matching field combination.
If the Manufacturers lookup table is a hierarchy table, then value mapping would still be optional to the Name field as its primary display field, and record matching would be required to the Name field alone
rather than with Code and/or to the [Remote Key] field (see “Lookup
Table Imports Using the [Remote Key] Field” on page 414 for more information).
The value mapping rules are summarized in Table 42.
Table 42. Value Mapping Rules Summary
Current Table Single Lookup Display Field Multiple Lookup Display Fields
Main Table
Value map required from individual source field to lookup field (which is the lookup table display field).
Value map: (1) optional from individual source fields to promoted Text lookup table display fields; (2) optional to promoted hierarchy table primary display field; and (3) required from compound source field to compound lookup field (which is the lookup table display field combination).
Hierarchy Lookup Table
Value map optional from individual source field to hierarchy table display field (record match required against hierarchy table display field).
Value map optional from individual source field to hierarchy table primary display field (record match required against hierarchy table primary display field and/or the
[Remote Key] field).
Any Lookup Table
Value map optional from individual source field to Text lookup table display field (record match required against lookup table display field).
Value map optional from individual source fields to Text lookup table display fields (record match required against lookup table display fields as matching field combination).
MDM Import Manager Reference Guide 249
NOTE ►► Current table display fields appear in the Destination Fields grid as “display field [DF]”, promoted lookup table display fields appear as “display field n [lookup field]”, and compound lookup fields appear as “lookup field [display field 1; display field n]”; (where “display field” is the name of a current table display field, “display field n” is the name of the nth lookup table display field, and “lookup field” is the name of the compound lookup field).
NOTE ►► See “Record Matching and Lookup Table Display Fields” on page 345 for more information about record matching.
NOTE ►► Field mapping to promoted lookup table display fields and creating a compound source field for mapping to the compound lookup field is possible only if the source data has individual fields that correspond to the individual display fields of the lookup table. Otherwise, you must simply field and value map from the applicable source field to the compound lookup field.
NOTE ►► When importing to the primary display field of a hierarchy, value mapping allows you to: (1) map source values to the correct hierarchy value (e.g. source value “C” to destination value “A>B>C” rather than “D>E>C”); and (2) add new nodes at specific locations in the hierarchy (without value mapping, new nodes would simply be added by default as sibling children of the root).
NOTE ►► When the current table is the hierarchy table, value mapping to the primary display field is optional because: (1) value mapping would be an unnecessary overhead if you don’t need to resolve ambiguities or add new nodes (as described above); and (2)
when you value map, you cannot record match on just the [Remote Key] field, which must instead be combined with the primary display
field; record matching on the [Remote Key] field alone makes it convenient to populate values of other language layers.
MULTI-VALUED DESTINATION FIELDS
Recall that many of the MDM field and attribute types can be defined as multi-valued; that is, a single field in a single record can be used to store multiple data values.
Field Mapping to Multi-Valued Fields
When you map a source field to a multi-valued destination field, the drop-down edit controls in the Converted Value column of the Source Values grid are multi-valued, as shown in Figure 164.
250 MDM Import Manager Reference Guide
Figure 164. Multi-valued drop-down control
You can then map and convert values as follows: (1) whether or not the mapped destination field requires value mapping, you can use the multi-valued drop-down control to edit or create one or more values for each source value; or (2) if the destination field requires value mapping, you can map each source data value to one or more destination values even without manually expanding it into multiple values.
TIP ►► You can use the multi-valued drop-down control to either create or map to multiple values regardless of whether or not the source data value consists of multiple delimited values.
Field Mapping to Coupled Numeric Attributes
When you a map a source field to a coupled numeric attribute (which is always multi-valued), the MDM Import Manager automatically breaks each coupled numeric value pair in the source data into a primary value and a coupled value based on the setting of the Coupled Numeric delimiter string.
When you edit a source value, the drop-down edit control in the Converted Value column of the Source Values grid is a modified version of the multi-valued drop-down edit control that allows you to edit each value, as shown in Figure 165.
Figure 165. Coupled numeric drop-down control
MDM Import Manager Reference Guide 251
When you apply a value conversion filter, the command for each filter leads to a cascading menu that lets you apply the filter to each value, as shown in Figure 166.
Figure 166. Cascading menu for a coupled numeric attribute
The Multi-Valued Update Option
When creating new destination records or replacing existing destination records, importing source data values into multi-valued fields is exactly the same as importing them into single-valued fields: for each source record, the one or more values are placed into the empty multi-valued field of the new destination record.
By contrast, when updating existing destination records as part of the import – and in particular when updating non-NULL multi-valued fields that already have one or more data values – the update of multi-valued fields can be handled in one of two ways.
Specifically, the set of incoming source data values can either: (1) be appended to; or (2) completely replace the set of existing destination data values, as described in Table 44.
Table 44. The Multi-Valued Update Option
Setting Description
Append The set of incoming source data values is appended to the set of existing destination values, so that the field in the updated record contains both the new and the existing data values.
Replace The set of incoming source data values completely replaces the set of existing destination values already in the multi-valued field, so that the field in the updated record contains only the new data values.
252 MDM Import Manager Reference Guide
NOTE ►► Among the pair of update import actions, the Multi-Valued
Update option only affects the Update (All Mapped Fields) action,
because the Update (NULL Fields Only) action updates only fields that
do not yet have data values, making the Append and Replace settings equivalent. (See “Import Actions” on page 346 for more information about the import actions of the Record Matching tab.)
For each multi-valued destination field or attribute, you can set the Multi-Valued Update option (see “Setting the Multi-Valued Update Option” on page 286) based on the requirements of the situation.
NOTE ►► See “Source Field-Level Settings” on page 224 for more information about the field-level settings for source fields.
Table 45 illustrates the two alternative Multi-Valued Update settings for a multi-valued field with different sets of existing data values and the same set of incoming delimited source values.
Table 45. Multi-Valued Update Examples
Incoming Source Data Values
Existing Destination Data Values
Updated Destination Data Values
Append Replace
Green; Blue
<NULL> Green; Blue Green; Blue
Green Green; Blue Green; Blue
Blue Green; Blue Green; Blue
Green; Blue Green; Blue Green; Blue
Red Red; Green; Blue Green; Blue
Red; Green; Blue Red; Green; Blue Green; Blue
NOTE ►► Append and Replace differ only when the set of existing destination values includes values not in the set of incoming source values.
The Qualified Update Option
Qualified lookup fields pose an additional challenge as it relates to multi-valued destination fields. Specifically, with qualified lookup fields (which are always multi-valued), each of the multiple values is not just a single data value, but rather an entire subrecord consisting of a set of values for both the display fields and the qualifiers of the qualified table record.
MDM Import Manager Reference Guide 253
NOTE ►► Moreover, the source data file will usually contain not one record per product but multiple records per product. Each source record then represents a single value for the multi-valued qualified lookup field (consisting of a set of values both for the display fields of the qualified table record and for the qualifiers) and the multiple source records for each product are treated as multiple instances of the same product, each with a different “value” for the qualified lookup field.
With qualified lookup fields, the Append and Replace settings of the Qualified Update option have precisely the same meaning and behavior as they do for the Multi-Valued Update option with normal multi-valued fields and attributes. Specifically:
Append. Appends to the set of existing qualified lookup values (preserving and appending to the set of existing links to the qualified table records).
NOTE ►► To determine whether to append a new qualified lookup value, MDM compares the data values of an incoming source record against both the values of the display fields of the linked qualified table record and all of the qualifiers taken together as a single “monolithic” value combination.
Replace. Completely replaces the set of existing qualified lookup values (eliminating the existing links to the qualified table records and discarding the qualifier values associated with those links).
This monolithic “all qualifiers” approach to comparing incoming and existing values is sufficient and appropriate in some circumstances (such as those described in the next section) but it provides no ability to update individual qualifier values of existing qualified lookup values rather than replace or append to the set of qualified lookup values.
For those cases where the comparison should instead be made against some subset of rather than all the qualifier values (to permit selective update of the qualifier values of existing qualified links), the Qualified
Update option offers – in addition to the Append and Replace settings – a third setting that is enabled only for qualified lookup fields.
Specifically, the Update setting allows you to specify which particular qualifiers MDM should use as matching qualifiers for matching purposes, and then how to update the values of the remaining qualifiers of existing qualified lookup values, as described in Table 46.
Table 46. The Qualified Update Option for Qualified Lookup Fields
Setting Description
Append Comparing against all of the qualifiers, incoming source data value combinations that do not already exist are appended to the set of existing qualified lookup values.
254 MDM Import Manager Reference Guide
Setting Description
Replace With no comparison against the qualifiers whatsoever, the set of incoming source data value combinations completely replaces the set of existing qualified lookup values.
Update
Comparing against just the matching qualifiers, incoming source data value combinations that already exist update the remaining qualifiers of existing qualified lookup values. Update actions include:
New links (no matching qualified lookup value)
Skip Create
Existing links (matching qualified lookup value)
Skip Update (NULL Qualifiers Only)
Update (All Mapped Qualifiers) Replace
Delete
You can specify a different Update action for source value combinations that do match and those that do not match an existing qualified lookup value based on the matching qualifiers, as described in Table 47.
Table 47. Update Actions for Qualified Lookup Values
Update Action Description
New Links
Skip Skip source value combinations that do not exist.
Create Create a new link to a qualified lookup table record and set the associated qualifier values.
Existing Links
Skip Skip source value combinations that already exist.
Update (NULL Qualifiers Only) Update just the NULL remaining qualifiers of each existing qualified lookup value.
Update (All Mapped Qualifiers) Update all the remaining qualifiers of each existing qualified lookup value.
Replace Replace all the remaining qualifiers of each existing qualified lookup value by first deleting them and then creating a new value combination.
Delete Delete this qualified lookup value during import processing
NOTE ►► The Update setting provides a “mini” record-match-within-a-record-match for the subrecords represented by each qualified lookup value. (See “Import Actions” on page 346 for more information about the import actions of the Record Matching tab.)
For example, assume a main table of product records that includes a qualified lookup field into the Distributors qualified table, shown in Figure 167. The Distributors table contains a list of three distributors, and is used to store distributor-specific part numbers and pricing for
MDM Import Manager Reference Guide 255
each product. Each qualified table record contains a distributor name in the Name field; the table also contains qualifiers Dist No and Price (shown in square brackets), which have values only when a qualified table record is assigned to the value of a main table qualified lookup field.
Distributors:
Name [Dist No] [Price]
Grainger
Applied
Wesco
Figure 167. Qualified table records
Now consider Figure 168, which illustrates two incoming source records (each containing the distributor-specific part number and price for SKU=113) and the corresponding destination record for the same product that has existing links to the three distributors and their associated distributor-specific part number and price values.
Source: Destination:
SKU DISTRIB DIST NO PRICE ► SKU Distrib [l] [Dist No] [Price]
113 Wesco 444-201 $3.50 113 Wesco 444-101 $3.52
113 Applied AP-201 $3.49 Grainger X437-5 $3.55
Applied AP-201 $3.65
Figure 168. Incoming distributor-specific information
Note that the two source records contain part number and price values for two of the three distributors, for which the destination record already contains qualified links and qualifier values.
Unfortunately, neither Append nor Replace is flexible enough to handle this simple import, since neither allows you to simultaneously leave some existing qualified links untouched, update other existing links with new source data, and add new qualified links. In particular:
Append will correctly preserve the link to Grainger, but will create second, duplicate links to Wesco and Applied rather than update the values associated with the existing links.
Replace will update the links to Wesco and Applied with new part number and price values, but will also wipe out the link and associated values for Grainger.
However, with the Update setting, you can specify matching qualifiers
for comparing against existing links, as well as what Update action to take for new and existing qualifier value combinations.
256 MDM Import Manager Reference Guide
Figure 169 illustrates the different effects of the three Qualified Update settings on the resulting set of qualified links to the product record for SKU=113 when incoming source fields are mapped to destination field Distrib and to destination qualifiers Dist No and Price.
Destination: Destination:
SKU Distrib [l] [Dist No] [Price] SKU Distrib [l] [Dist No] [Price]
113 Wesco 444-101 $3.52 113 Wesco 444-201 $3.50
Grainger X437-5 $3.55 Applied AP-201 $3.49
Applied AP-201 $3.65 Replace
Wesco 444-201 $3.50
Applied AP-201 $3.49
Append
Destination:
Destination: SKU Distrib [l] [Dist No] [Price]
SKU Distrib [l] [Dist No] [Price] 113 Wesco 444-101 $3.52
113 Wesco 444-201 $3.50 Grainger X437-5 $3.55
Grainger X437-5 $3.55 Applied AP-201 $3.49
Applied AP-201 $3.49 Wesco 444-201 $3.50
Update Matching Qualifiers = None
New Links = Skip or Create
Existing Links = Update or Replace
Update
Matching Qualifiers = Dist No
New Links = Create
Existing Links = Update or Replace
Figure 169. Qualified Update settings for qualified lookup fields
NOTE ►► As shown above, which qualifiers you specify as matching qualifiers affects the resulting set of qualified links. Moreover, the more qualifiers you specify, the more rigorous the match to an existing qualified link, and the fewer qualifier values you can update.
NOTE ►► The matching qualifiers specified not only for the Update
setting of the Qualified option but also for the Append and Replace settings are also used by the Merge Qualified Links option when merging qualified links, and moreover, affect the resulting set of qualified links (see “Source Record Merging” on page 350).
TIP ►► You can tune the behavior of the Update setting of the Qualified Update option not only by adjusting the set of matching
qualifiers but also by adjusting the particular Update actions for new and existing links.
MDM Import Manager Reference Guide 257
Product Applications and Automotive Parts Data
With automotive parts data, the source data typically represents application records rather than product records, and contains multiple
records per product, where each source record represents a single application of the product (and is again treated as a product record that represents a single value for the multi-valued qualified lookup field).
Consider the table of source application records shown in Figure 170.
PART NO YEAR MAKE MODEL CA EQUIP A/C P/B
A2-444 1998 Toyota Celica Yes Yes Yes
A2-444 1998 Toyota Celica No Yes Yes
A2-444 1996 Toyota Celica No Yes Yes
A2-444 1998 Ford Mustang No Yes Yes
A2-444 1997 Ford Mustang No No No
C6-859 1996 Toyota Celica Yes Yes Yes
C6-859 1996 Toyota Celica No Yes Yes
C6-859 1998 Ford Mustang Yes No Yes
Figure 170. Automotive data and source application records
In this case, using either the Replace setting of the Qualified Update
option, or alternatively, the Append setting to match against all of the qualifiers, is appropriate.
NOTE ►► Recall that qualifiers are part of neither the main table record nor the qualified table record, but rather are defined by each association of a main table record with a qualified table record. When
using the Update setting, the qualifiers on which to match depends upon whether the qualifier values logically represent an extension to: (1) the main table record (such as with multiple pricing or alternate part number data, as described in the previous section, in which case the corresponding qualifiers should be specified as matching qualifiers); or (2) the qualified lookup table record (such as with application records, as described in this section, in which case they should not be specified as matching qualifiers).
This table of application records contains information that corresponds to the following destination fields and qualifiers:
1 field of the Parts main table (PART NO)
3 fields of the Vehicles qualified table (YEAR, MAKE, MODEL)
3 qualifiers of the Vehicles qualified table (CA EQUIP, A/C, P/B)
The records themselves correspond to the following records of the Parts and Vehicles tables:
2 distinct part records (A2-444; C6-859)
258 MDM Import Manager Reference Guide
4 distinct vehicle records (4 for A2-444; 2 for C6-859)
8 distinct application records (5 for A2-444; 3 for C6-859)
NOTE ►► The same vehicle can be linked to each part more than once, with different qualifiers for each link (two links to 1998 Toyota Celica for A2-444; two links to 1996 Toyota Celica for C6-859).
This single table of source data illustrates many of the challenges of dealing with complex source data, and a methodology for doing so using the innovative mechanisms of the MDM Import Manager:
Field mapping. The PART NO field is field mapped to the Part No.
field of the main table, and the CA EQUIP, A/C, and P/B fields are field mapped to the qualifiers of the Vehicles table.
Map-within-a-map. Since nested display fields are automatically
promoted, the YEAR, MAKE, and MODEL fields are field mapped to the Year, Make, and Model fields of the Vehicles table.
Nested lookups-within-lookups. Since Year, Make, and Model are nested lookup fields, YEAR, MAKE, and MODEL are value mapped to the values of the Year, Make, and Model fields (see page 243).
Create Compound Field command. Once YEAR, MAKE, and
MODEL are field and value mapped, the Create Compound Field command creates the qualified lookup field values.
Qualified lookup fields and Automap. The compound source field
is mapped to the qualified lookup field, and the combined values automapped to its values; if necessary, new records are added to the Vehicles table.
Qualified Update option. The Qualified Update option is set to
Append, so the new application records are appended to the existing qualified lookup field values.
NOTE ►► To determine whether an incoming application already exists, it is treated monolithically for the purposes of the comparison as consisting of all of the display field values of the qualified table record and all of the qualifier values for that particular application.
MDM Import Manager Reference Guide 259
The Tuple Update Option
As with qualified lookup fields, a tuple field does not represent a single data value, but rather an entire subrecord (or set of subrecords) that belongs exclusively to its parent record. The value of this subrecord is the combined values of the tuple’s member fields, including both its display and non-display fields.
When importing records into a multi-valued tuple field, Import Manager first checks the tuple’s matching fields to see whether a matching tuple record already exists in the repository. Import Manager’s default behavior is to use all of a tuple’s member fields as matching fields, but you can choose a subset of matching fields on a tuple-by-tuple basis.
Once Import Manager determines whether or not the source record represents a new or existing tuple record, it then performs the corresponding update action specified for the tuple field.
You can use the Tuple Update command to specify a different update action for source records that do match or do not match an existing tuple record based on the tuple field’s matching fields, as described in Table 48.
Table 48. Update Actions for Tuple Fields
Update Action Description
New Records (If no matching tuple record is found)
Skip Do not import the incoming tuple record.
Create Add the incoming tuple record.
Existing Records ( a matching tuple record is found)
Skip Do not import the incoming tuple record.
Update (NULL Fields Only) Update only destination fields which have NULL values.
Update (All Mapped Fields) Update the values of all mapped destination fields.
Replace Delete the entire existing tuple record and replace it with the incoming tuple record.
Delete Delete the entire existing tuple record.
NOTE ►► The Tuple Update command provides a “mini” record-match-within-a-record-match for the tuple records represented by each tuple field value. (See “Record Matching Concepts” on page 338 for more information about record matching.)
CAUTION ►► The Replace and Delete actions permanently delete the entire existing tuple record from the repository, including: (1) the values of any unmapped destination fields on the matching tuple record; and (2) any nested tuple records belonging to the matching tuple record.
260 MDM Import Manager Reference Guide
When setting update actions for nested tuple fields, each multi-valued tuple field can have a update action different than its parent or child tuple field in the tuple hierarchy. This lets you customize update behavior on a tuple-by-tuple basis, rather than forcing you to use a monolithic, one-size-fits-all approach.
NOTE ►► The Tuple Update option is disabled for single-valued tuple fields nested within in a tuple structure. Instead, Import Manager applies the update option set for its parent tuple in the tuple structure.
An important factor to consider when choosing update actions for nested tuple fields is that Import Manager updates tuple records in a “top-down” manner, starting with the tuple field values belonging to the current table record and then progressing to each subsequent tuple field in the tuple hierarchy.
Consider Figure 171, which illustrates a Customers table containing a nested tuple hierarchy (Contact Phone), where:
The Customers table includes the multi-valued Contact tuple field.
The Contact tuple includes the multi-valued Phone tuple field.
Customers
├ Id
├ Company
├ Contact [tuple]
├ Name
└ Phone [tuple]
├ Area Code
├ Number
└ Extension
Figure 171. A Nested Tuple Hierarchy in a Customers table
Figure 172 shows the same Customers table in tabular format, with sample records added.
Customers:
Id Company
Contact [tuple]
Name Phone [tuple]
Area Code Number Ext.
1 Acme John Smith
800 5551212 300
310 777-1234
Mary Hsu 212 444-1234
2 LittleCo Susan Gray 808 333-3333
Figure 172. Nested Tuples in a Customers Table
MDM Import Manager Reference Guide 261
In Figure 172, notice that the Customers table has a single record for the company Acme, and that this Acme record has two private subrecords in its Contact tuple: one for John Smith and one for Mary Hsu. The John Smith tuple record owns two private Phone tuple records; the Mary Hsu tuple record owns one.
Figure 173 represents a source record to import into the Acme record’s Contacts tuple (assume all fields on the source file are mapped to their counterparts on the Customers table):
Source:
Company Name Area Code Number Ext
Acme John Smith 310 777-1234 5
Figure 173. Source Records to Import
Assume also that the source record is imported using the tuple update actions specified in Figure 174.
Tuple Matching Fields Update Action
Contact Name New Record: Create
Existing Record: Replace
Phone Area Code; Number
New Record: Create
Existing Record: Update (NULL Fields Only)
Figure 174. Contact and Phone Tuple Update Actions
To import this source record, Import Manager performs the following actions, in order:
1. Searches for the Acme record on the Customers table.
2. Performs the default import action specified for the record match type in Import Manager’s Match Records tab.
3. In the Acme record, searches for Contact tuple records with the Name value, “John Smith”.
4. Based on the results of the search, performs the corresponding tuple update action specified for the Contact tuple.
5. In the resulting “John Smith” Contact tuple records, searches for Phone tuple records containing the values “310” and “777-1234” in the Area Code and Number fields.
6. Based on the results of the search, performs the tuple update action specified for the Phone tuple.
Note that the import/update action performed after each matching step determines whether or not Import Manager continues to the next step.
262 MDM Import Manager Reference Guide
Based on the tuple hierarchy in the Customers table, the Contact tuple field always gets updated prior to the Phone tuple. The update actions selected for the Contact tuple field, therefore, affect not only the Contact tuple records but also any Phone tuple records they might own.
This is reflected in the results of the import, shown in Figure 175.
Customers:
Id Company
Contact
Name Phone
Area Code Number Ext.
1 Acme John Smith 310 777-1234 5
Mary Hsu 212 444-1234 301
2 LittleCo Susan Gray 808 333-3333
Figure 175. After Importing Source Records
As a result of the Replace action being the Contact tuple’s update option for existing records (Figure 174), when Import Manager matches the source record’s Name field value “John Smith” to the Contact tuple record with the Name value “John Smith”, it deletes the existing John Smith tuple record, including all of the Phone tuple records it owned, and replaces it with the imported record values.
These phone tuple records are deleted even though the Phone tuple
action is set to Update (NULL Fields Only). Because the Phone tuple field belongs to the Contact tuple, all of the phone records for John Smith got deleted with the John Smith Contact tuple record before the Phone tuple’s update options come into play.
Finally, depending on the tuple update options selected, you can update a nested tuple record without having to map any member fields of “higher-up” tuples in the tuple hierarchy (although you always have to map least one field on the current table).
Consider Figure 176, which represents a new source record to import into the Customers table.
Source:
Company Area Code Number Ext
BigCorp 404 987-1234
Figure 176. Source Records to Import
Assume that the source fields are mapped to their destination counterparts, and that the source record is imported using the tuple update actions specified in Figure 177.
MDM Import Manager Reference Guide 263
Tuple Matching Fields Update Action
Contact Name New Record: Create
Existing Record: Replace
Phone Area Code; Number
New Record: Create
Existing Record: Update (NULL Fields Only)
Figure 177. Contact and Phone Tuple Update Actions
The result of the import is shown in Figure 178.
Customers:
Id Company
Contact
Name Phone
Area Code Number Ext.
BigCorp <NULL> 404 987-1234
Figure 178. After Importing Source Records
Note that a new Contact tuple record is added to the BigCorp customer record, even though no Contact tuple member fields were mapped. To preserve the structure of the tuple hierarchy, Import Manager:
1. Created a “placeholder” Contact tuple record, inserting NULL values into the unmapped Contact tuple member field, “Name”.
2. Imported the source record values as a Phone tuple record belonging to the NULL Contact tuple record.
Import Manager creates as many nested placeholder records as are required to maintain a tuple hierarchy structure. That is, if you map only the member fields of the sixth tuple in a six-tuple hierarchy, Import Manager creates and nests the missing five tuple records. You can then overwrite the NULL values in these records using Data Manager or by another import.
Of course, the result of any import to a nested tuple record depends on the tuple update option settings for both the destination tuple field and for all of the tuple fields “above” it in the tuple hierarchy.
For example, assume the Contact tuple field’s New Record update
action was set to Skip instead of Create, as shown in Figure 179.
264 MDM Import Manager Reference Guide
Tuple Matching Fields Update Action
Contact Name New Record: Skip
Existing Record: Replace
Phone Area Code; Number
New Record: Create
Existing Record: Update (NULL Fields Only)
Figure 179. Contact and Phone Tuple Update Actions
The result of the import is shown in Figure 180.
Customers:
Id Company
Contact
Name Phone
Area Code Number Ext.
BigCorp
Figure 180. After Importing Source Records
Note that now, no new Contact tuple record is added to the BigCorp
customer record, even though the Phone tuple’s New Record update
action remains set to Create. Because Import Manager imports from the top to the bottom of a tuple hierarchy, when it cannot find a Contact tuple record to match the source record, it performs the Contact tuple’s
New Record action and “skips” creating the new Contact tuple record. At this point, any tuple fields below the Contact tuple in the tuple hierarchy are also skipped.
Consider what happens instead if the Contact tuple field’s New Record
update action remains set to Create but the Phone tuple’s New record
update action is changed to Skip, as shown in Figure 181.
Tuple Matching Fields Update Action
Contact Name New Record: Create
Existing Record: Replace
Phone Area Code; Number
New Record: Skip
Existing Record: Update (NULL Fields Only)
Figure 181. Contact and Phone Tuple Update Actions
MDM Import Manager Reference Guide 265
The result of the import is shown in Figure 182.
Customers:
Id Company
Contact
Name Phone
Area Code Number Ext.
BigCorp <NULL>
Figure 182. After Importing Source Records
Note that in this scenario, a new NULL Contact tuple record is added even though no Phone tuple record was created. When importing the source record values, Import Manager creates the new Contact tuple record according the Contact tuple’s update actions, and then performs
the Phone tuple update action, which is set to skip the new Phone tuple record. Because the Phone tuple is lower in the tuple hierarchy, its update actions do not change the results of the Contact tuple record’s update action.
MULTILINGUAL DESTINATION FIELDS
When you map a source field to a multilingual destination field, how you map – and if and how you configure MDM to expand the destination field – depends on the structure of the source data. (See “Field Mapping to Multilingual Destination Fields” on page 397 for more information.)
OBJECT LOOKUP DESTINATION FIELDS
When you map a source field to an object lookup destination field, the Converted Value column of the Source Values grid displays either: (1) the object name (for images, PDFs, and binary objects); or (2) the object value itself (for text blocks).
You can then map values in the Source Values grid to the objects of the underlying object lookup table in the Destination Values grid, linking the corresponding object to each source record as part of the import.
The Data Group Display Option
To distinguish among images, PDFs, and binary objects that have the same name, or text blocks that have the same value, you can display the Data Groups hierarchy in the Destination Values grid so that each object name appears as a leaf node beneath its data group, as illustrated in Figure 183 and described in Table 49.
266 MDM Import Manager Reference Guide
Figure 183. Data Groups hierarchy in Destination Values grid
NOTE ►► Object names or values are displayed in black, while data group nodes are highlighted in gray.
Table 49. The Data Group Display Option
Setting Description
Hide Data Group The object names or values are displayed as a list in the Destination Values grid, making it impossible to distinguish among identical names and values.
Display Data Group The object names or values are displayed as leaf nodes beneath their data group in the Data Groups hierarchy in the Destination Values grid.
For each object lookup destination field, you can set the Data” on page 289) based on the requirements of the situation.
NOTE ►► See “Destination Field-Level Settings” on page 225 for more information about all of the destination field-level settings.
WRITEABLE ONCE DESTINATION FIELDS
Recall that a field can be designated in the MDM Console as Writeable Once, so that its value can be set only once and never changed. This can be a problem within the MDM Import Manager if you map a source field to a Writeable Once destination field to populate missing values, but some of the values of the destination field are already set.
Naturally, if the destination value for a record is empty, then MDM properly sets the destination value with the source value without a problem. Similarly, if the destination value for a record is set but is the same as the source value, MDM avoids a potential problem by ignoring
the source value, even when the import action for the Record is Update.
The challenge is what to do with the import record if the destination value is set and is different than the source value.
MDM Import Manager Reference Guide 267
DATA INTEGRITY ►► The “safest” behavior would be to reject source records that attempt to reset the value of a Writeable Once field. However, the MDM Import Manager needs to be able to support automation and high-volume imports without triggering multiple exceptions, especially when ignoring conflicting values is not an issue.
You can use the Writeable Once Field Update configuration option to specify how to handle imports when a source field is mapped to a Writeable Once destination field and the source value is different than the destination value, as summarized in Table 50.
Table 50. Writeable Once Field Update Configuration Option
Setting Description
Ignore Conflicting Value Allow record import and ignore source value if it conflicts with existing destination value.
Reject Record on Conflicting Value
Reject the record import if the source value conflicts with existing destination value.
NOTE ►► This option is relevant only for records where the destination value is set and different than the source value. Further, if the Writeable Once field is used as a record matching field, then this
option is relevant only when the Do Not Update Record Matching Field
configuration option is set to No (see “Record Filtering” on page 356 for more information).
GM TIME FIELDS
When you map a source field to a destination GM Time field, the date values must be in one of the following formats:
YYYYMMDD hh:mm:ss
MM/DD/YYYY hh:mm:ss
DD.MM.YYYY hh:mm:ss
NOTE ►►A space is required between the date and time values.
The format used should be the same as specified in the Configuration Options -> Dates -> Date Format setting.
268 MDM Import Manager Reference Guide
THE MDIS UNMAPPED VALUE HANDLING OPTION
When you run MDIS, it handles the import using an existing map file, which specifies a mapping from each source data value to each destination data value.
When subsequent imports include source data values that were not mapped when you created the map file, you can handle these unmapped values in one of three ways.
Specifically, values that are unmapped either because they do not exist or because Automap Unmapped Values was turned off can be: (1) ignored; (2) added to the set of destination data values; or (3) mapped to existing destination data values, as described in Table 51.
Table 51. The MDIS Unmapped Value Handling Option
Setting Description
Ignore Still unmapped values are ignored, and a NULL inserted for the value of the destination field in the destination record.
Add Still unmapped values are added to the set of destination data values for the destination field.
Map Still unmapped values are mapped to the specified existing destination data values for the destination field.
Fail Still unmapped values cause the import of the underlying records that contain the value to fail
For each destination field, you can set the MDIS Unmapped Value Handling option (see “Setting the MDIS Unmapped Value Handling Option” on page 291) to specify the proper handling of unmapped values during MDIS processing.
NOTE ►► You can use the global default value of the MDIS
Unmapped Value Handling option to Ignore or Add unmapped values, but you must set the option on a destination field-by-field basis if you
want to Map to existing destination data values.
NOTE ►► See “Source Field-Level Settings” on page 224 for more information about the field-level settings for source fields.
NOTE ►► MDIS processing is subject to the following considerations: (1) all configuration regarding the MDIS import should be created using
the MDM Import Manager and saved in an import map; (2) the
Automap Unmapped Values configuration option allows you to specify
whether or not to attempt to automap unmapped values; (3) the Default MDIS Unmapped Value Handling configuration option allows you to specify whether to ignore or add unmapped values, or to map to an existing destination data value.
MDM Import Manager Reference Guide 269
Field Mapping Operations
The following sections describe the operations you can perform on fields in the Field Mapping grids, and are summarized in Table 52.
Table 52. Field Mapping Operations
Operation Description
Buttons
Automap Automatically maps each source field to a destination field with the same name.
Map Maps the selected source field to the selected destination field.
Unmap Unmaps the selected fields.
Add Adds a new attribute to the destination taxonomy.
Source Fields Grid
Find Next Matching Destination Field
Finds the first or next similarly destination field with a name similar to the selected source field.
Create Compound Field
Combines fields that have been mapped to the multiple display fields of a destination lookup field.
Set Field Mapping Delimiter
Sets the Multi-Valued or Coupled Numeric delimiter for the selected source fields.
Set Case Sensitivity Sets the Case Sensitivity option for the selected source fields.
Set Flat NULL Handling
Sets the Flat NULL Handling option for the selected source fields.
Set Hierarchy NULL Handling
Sets the Hierarchy NULL Handling option for the selected source fields.
Set Value Conversion Filter
Sets the field-level value conversion filters for the selected source field.
Destination Fields Grid
Filter Attributes Sets the unlinked attribute filter for the destination fields grid.
Set NULL Interpretation
Sets the NULL Interpretation Option for the selected destination fields.
Set Multi-Valued Update
Sets the Multi-Valued Update option for the selected multi-valued destination fields.
Set Qualified Update Sets the Qualified Update option for the selected qualified lookup destination fields.
Set Tuple Update Sets the Qualified Update option for the selected tuple destination fields.
Set Data Group Display
Sets the Data Group Display option for the selected object lookup destination fields.
Set MDIS Unmapped Value Handling
Sets the MDIS Unmapped Value Handling option for the selected destination field.
270 MDM Import Manager Reference Guide
NOTE ►► See “The NULL Handling Options” on page 231, “The Case Sensitivity Option” on page 237, “Value Conversion Filters” on page 238, “The NULL Interpretation Option” on page 236, “The Multi-Valued Update Option” on page 251, “The Qualified Update Option” on page 252, “The Data Group Display Option” on page 265, and “The MDIS Unmapped Value Handling Option” on page 268 for more information about these field-level settings.
SOURCE FIELD MAPPING
Mapping fields is extremely straightforward using the Field Mapping grids, shown in Figure 185.
Figure 185. Field Mapping grids
For each source field or field combination whose data values you want to import, the field mapping process includes some or all of the following steps:
1. Select a source field.
2. Set the delimiter values.
3. Set the NULL handling.
4. Select the corresponding destination field, attribute, or qualifier.
5. Examine the set of source values and the set of corresponding destination values to determine if the fields should truly be mapped.
6. If necessary, return to one of the earlier steps in the import workflow to further split or partition the field and then attempt to map again.
7. Map the source field to the destination field.
Each of the steps above is described in the following sections.
MDM Import Manager Reference Guide 271
MAPPING FIELDS
You can map an unmapped source field to an unmapped destination field as described in this section. Corresponding source and destination fields do not need to have the same name.
Field Mapping Grid Icons
To make the relationships between fields more understandable, the Field Mapping grids now display source and destination fields in their natural hierarchies. To accurately depict these hierarchies, Import Manager includes both “structural” nodes (such as root nodes and fields containing nested values) as well as data fields in the Field Mapping grids. While you can map any data field from the Source Fields grid to any data field on the Destination Fields grid, you cannot map structural nodes on either grid.
Import Manager use icons to depict which items on the grids can and can not be mapped, as described in Table 53.
Table 53. Field Mapping Grid Icons
Icon Description Mappable?
Field or Attribute Yes
Structural node No; map nested fields instead.
Source Fields
Each of the fields of the current source data table – along with each of the fields that you looked up from other tables – appear in the Source Fields grid.
NOTE ►► Fields that have been partitioned or any of whose values have been partitioned appear as “field [Partition]”.
NOTE ►► Lookup fields may appear as “field <table>” (where “table” is the name of the joined table).
Destination Fields
Each of the fields of the current destination data table – along with any nested fields – appear in the Destination Fields grid.
NOTE ►► For simplicity, the different “types” of destination fields, attributes, qualifiers, and nested display fields are each generically referred to as “fields” unless the context is dealing with a specific type.
The destination field type is identified by the F/D/Q/A column, as summarized in Table 54.
272 MDM Import Manager Reference Guide
Table 54. Destination Field Types Indicated by the F/D/Q/A Column
F/D/Q/A Type Description
F Field Fields of the current destination table.
D Display Field The individual display fields of each lookup field whose value is comprised of multiple display fields.
Q Qualifier
The qualifiers of each of the qualified fields.
With multiple qualified fields, each qualifier appears as “qualifier <qualified field>” (where “qualified field” is the name of the qualified field) (NYI)
A Attribute
The attributes of each of the taxonomy fields.
With multiple taxonomy fields, each attribute appears as “attribute <taxonomy field>” (where “taxonomy field” is the name of the taxonomy field)
MULTILINGUAL ►► In a multilingual repository, the Destination Fields grid may contain multiple instances of each multilingual field (one per language), allowing you to field map each language-specific source field to the applicable language layer of the multilingual destination field. More information about languages and multilingual repositories is provided in “Part 14: Multilingual Support.”
To map a source field to a destination field:
1. In the Source Fields grid, select the unmapped source field that you want to map.
2. In the Destination Fields grid, select the destination field to which you want to map the source field.
TIP ►► If the name of the destination field is the same as the source field, you can automatically locate the destination field by right-clicking
on the source field in the Source Fields grid and choosing Find Next Matching Destination Field from the context menu (see “Finding Matching Destination Fields” on page 274 for more information).
3. Click on the Map button (between the fields grids) to map the selected source field and the selected destination field.
4. MDM indicates that the source and destination fields have been mapped by placing a green thunderbolt ( ) in the Mapped column of both grids, and by placing the names of the source and destination mapped fields into the corresponding column of the other grid.
NOTE ►► You must map a source field to the qualified field if you want to import to any of its qualifiers.
NOTE ►► The Map button is only enabled on main table lookup fields when key mapping is enabled on the lookup table.
MDM Import Manager Reference Guide 273
NOTE ►► You must map a source field to the taxonomy field in order to import to any of its attributes unless: (1) the import action for every
new source record is Skip; (2) the import action for every existing
source record is either of the two Update actions; and (3) the taxonomy field value of every matching destination record is non-NULL.
TIP ►► If the source data does not contain a field that corresponds to the taxonomy field, you can: (1) use the Add Field command to add a new field; (2) field map the new field to the taxonomy field; and (3) value map the <NULL> value of the new field to the corresponding value of the taxonomy field.
DATA INTEGRITY ►► The MDM Import Manager automatically links unlinked attributes to a category as part of the import when: (1) you import into the category records for which the source data field that is mapped to the attribute has non-NULL values; or (2) the category value is changed for an existing record (i.e. an existing record is reassigned to another category) and data values would otherwise be discarded. (See “Configuration Options” on page 62 for more information about the Autolink Attributes options.)
UNMAPPING FIELDS
You can unmap one or more previously mapped source fields as described in this section.
To unmap one or more fields:
1. In the Source Fields grid, select the mapped source fields that you want to unmap.
TIP ►► Alternatively, you can select the mapped destination fields in the Destination Fields grid.
NOTE ►► When you select a mapped source field in the Source Fields grid, the corresponding mapped destination field is automatically selected in the Destination Fields grid (but not the other way around).
2. Click on the Unmap button (between the fields grids) to unmap the source fields from their mapped destination fields.
NOTE ►► The Unmap button is focus-dependent and unmaps the selected fields from the grid that either: (1) currently has the focus; or (2) last had the focus if neither fields grid currently has the focus.
3. MDM indicates that the source and destination fields have been unmapped by removing the green thunderbolt ( ) from the Mapped column of both grids for the unmapped fields.
274 MDM Import Manager Reference Guide
AUTOMAPPING SOURCE FIELDS
If some or all of the source fields have names that are the same or
similar to the corresponding destination fields, you can use the Automap button to automatically map them to their matching destination fields, as described in this section.
TIP ►► The Automap button provides a convenient way to quickly get started with the field mapping process.
To automatically map one or more source fields to the first matching destination fields:
Click on the Automap button (between the fields grids). MDM automatically maps each unmapped source field to the first destination field with a name that is the same or similar, if one exists.
NOTE ►► See “The Find Matching Field Options” on page 275 for more information about how MDM performs the field name comparisons.
FINDING MATCHING DESTINATION FIELDS
Alternatively, if there may be more than one destination field with a name similar to each source field and you want to manually choose to which destination field the source field should be mapped, you can use
the Find Next Matching Destination Field command to quickly locate each matching destination field in sequence for potential mapping, as described in this section.
To find destination fields whose names are similar to a source field:
1. In the Source Fields grid, select the source field for which you want to find a similarly named destination field.
2. Right-click on the field and choose Find Next Matching Destination Field from the context menu, or choose Fields > Find Next Matching
Destination Field from the main menu.
3. MDM moves the selected field in the Destination Fields grid from the current selection to the next similarly named destination field.
TIP ►► You can right-click on the source field and choose Find Next
Matching Destination Field repeatedly (or simply press F3) to find each similarly named destination field one after another.
MDM Import Manager Reference Guide 275
THE FIND MATCHING FIELD OPTIONS
To accommodate subtle differences between source and destination field names and increase the likelihood of finding a matching field with
the Automap or the Find Next Matching Destination Field commands, the MDM Import Manager by default: (1) performs a “contains” comparison against the destination field name; and (2) normalizes (i.e. removes punctuation characters and spaces) from both the source and destination field names before performing the comparison.
You can use the Configuration Options command to change the settings of the Find Matching Field options, as summarized in Table 55.
Table 55. Find Matching Field Configuration Options
Option Value Description
Operator Contains Performs a contains comparison.
Exact Performs an exact comparison.
Special
Normalized First normalizes source and destination field names.
Case Insensitive Performs a case insensitive comparison.
Case Sensitive Performs a case sensitive comparison.
NOTE ►► See “Configuration Options” on page 62 for more information about the Find Matching Field options.
CREATING A COMPOUND SOURCE FIELD
Recall that when a destination lookup field represents the combination of the underlying lookup table’s multiple display fields, each of the individual display fields is promoted and automatically appears in the Destination Fields grid as a target for mapping.
To create a compound source field for mapping to a destination lookup field that consists of multiple display fields: (1) map each source field to one of the display fields; (2) for each source field, either convert or map
its values to the values of the display field; (3) use the Create Compound Field command to create a compound field for mapping to the destination lookup field, as described in this section; (4) map the compound field to the destination lookup field; and (5) automap the compound field’s source value combinations to the lookup field values.
NOTE ►► The source value combinations created by the Create Compound Field command correspond to the destination lookup value combinations created by the lookup table’s multiple display fields.
NOTE ►► Creating a compound source field is possible only if the source data has individual fields that correspond to the individual display fields of the lookup table.
276 MDM Import Manager Reference Guide
To create a compound source field:
1. Right-click in the Source Fields grid and choose Create Compound Field from the context menu, or choose Source > Create Compound Field from the main menu, and select from the cascading menu of destination lookup fields that consist of multiple display fields.
NOTE ►► Destination lookup fields whose individual display fields have not all been mapped and their values all converted or mapped appear in the cascading menu in gray.
2. MDM adds the compound field to the Source Hierarchy tree and to the list of fields in the Source Fields grid.
NOTE ►► The default name for a created compound field is the destination lookup table name.
NOTE ►► Once a compound field is created, the source fields used to create the compound field cannot be unmapped until the compound field is deleted.
SETTING THE FIELD MAPPING DELIMITERS
The current field mapping delimiters determine how the MDM Import Manager interprets source data values as part of the value conversion and mapping process, and must be properly set prior to mapping a
source field to a destination field.
The two field mapping delimiters include:
Multi-Valued delimiter. MDM identifies the individual delimited
values within a single field when mapping to a multi-valued field using the current Multi-Valued delimiter string.
Coupled Numeric delimiter. MDM identifies the coupled numeric
value pairs when mapping to a coupled numeric attribute using the current Coupled Numeric delimiter string.
NOTE ►► The delimiter for compound lookup fields is ", " (comma and space). This delimiter cannot be changed.
Specifically, field mapping uses as the current delimiter the global default delimiter string, if the field is set to inherit; or the field-specific delimiter string, if it has been manually set for the field.
You can set the global default delimiter string for all fields that inherit the
default using the Configuration Options command, or alternatively,
override the default and set it on a field-by-field basis using the Set Field
Mapping Delimiter commands, as described in this section.
NOTE ►► Inherited global default values appear in the Source Fields grid in gray.
MDM Import Manager Reference Guide 277
NOTE ►► A source data field that contains multiple delimited coupled numeric value pairs requires that you set both delimiters. For example, “5hp @ 3500rpm; 6hp @ 4500rpm; 7hp @ 5500rpm” would require that the Coupled Numeric delimiter be set to “@” and the Multi-Valued delimiter be set to “;”.
TIP ►► Be careful not to include leading or trailing spaces in the delimiter string unless your intent is to distinguish between occurrences of the delimiter with and without surrounding spaces (e.g. “X;Y” represents a single value but “X ; Y” represents two values). MDM will automatically trim the leading and trailing spaces from individual delimited values even if you do not include the spaces.
NOTE ►► See “Source Field-Level Settings” on page 224 for more information about all of the source field-level settings; see “Configuration Options” on page 62 for more information about the global default delimiter strings.
To set the Multi-Valued or Coupled Numeric delimiter string for one or more fields:
1. In the Source Fields grid, select the fields for which you want to set the Multi-Valued or Coupled Numeric delimiter string.
2. Right-click on one of the fields and choose Set Field Mapping Delimiter from the context menu, or choose Fields > Set Value Delimiter from the main menu, and then choose from the cascading menu:
Multi-Valued Coupled Numeric
3. MDM open a Set Delimiter dialog which displays the current value for the delimiter string and whether or not the global default is being inherited.
NOTE ►► No value is displayed in the edit control if you have selected multiple fields for which the current values are not all the
same. Similarly, a gray check () appears in the Inherit Default checkbox if some but not all of the selected fields are inheriting the global default.
4. To set the delimiter string and override the global default value, enter the new delimiter string into the edit control, or alternatively, to restore
inheritance, check the Inherit Default checkbox.
5. Click OK when you are done to close the dialog.
278 MDM Import Manager Reference Guide
SETTING THE CASE SENSITIVITY OPTION
The current setting of the Case Sensitivity option determines whether the MDM Import Manager ignores or recognizes case differences in the source data when it determines the set of distinct values for a field, and
must be properly set prior to: (1) displaying values using the Display
Distinct Values command; (2) partitioning the field; (3) splitting the field
using the Split into Hierarchy or Split into Multiple Fields commands; or (4) mapping the source field to a destination field.
Specifically, the MDM Import Manager uses as the current case-sensitivity setting either: (1) the global default setting, if the field is set to inherit; or (2) the field-specific setting, if it has been manually set for the field.
You can set the global default case-sensitivity setting for all fields that
inherit the default using the Configuration Options command, or alternatively, override the default and set it on a field-by-field basis using
the Set Case Sensitivity command, as described in this section.
NOTE ►► Inherited global default values appear in the Source Fields grid in gray.
NOTE ►► See “The Case Sensitivity Option” on page 237 for more information about the Case Sensitivity option; see “Source Field-Level Settings” on page 224 for more information about all of the source field-level settings; see “Configuration Options” on page 62 for more information about the global default Case Sensitivity option.
To set the Case Sensitivity option for one or more fields:
1. In the Source Fields grid, select the fields for which you want to set the Case Sensitivity option.
2. Right-click on one of the fields and choose Set Case Sensitivity from
the context menu, or choose Fields > Set Case Sensitivity from the main menu.
3. MDM displays a cascading menu of three choices:
Case Insensitive Case Sensitive Inherit Default
4. To set the Case Sensitivity option and override the global default
setting, choose either Case Insensitive or Case Sensitive, or
alternatively, to restore inheritance, choose Inherit Default.
NOTE ►► MDM places a check next to the cascading menu choice that corresponds to the current setting unless you have selected multiple fields for which the settings are not all the same.
MDM Import Manager Reference Guide 279
SETTING THE FLAT NULL HANDLING OPTION
The current setting of the Flat NULL Handling option determines whether the MDM Import Manager preserves or ignores NULLs in the source data when you map a field, and must be properly set prior to mapping a source field to a destination field.
Specifically, the MDM Import Manager uses as the current flat NULL-handling setting either: (1) the global default setting, if the field is set to inherit; or (2) the field-specific setting, if it has been manually set for the field.
You can set the global default flat NULL-handling setting for all fields
that inherit the default using the Configuration Options command, or alternatively, override the default and set it on a field-by-field basis using
the Set Flat NULL Handling command, as described in this section.
NOTE ►► Inherited global default values appear in the Source Fields grid in gray.
NOTE ►► See “The Flat NULL Handling Option” on page 232 for more information about the Flat NULL Handling option; see “Source Field-Level Settings” on page 224 for more information about all of the source field-level settings; see “Configuration Options” on page 62 for more information about the global default Flat NULL Handling option.
To set the Flat NULL Handling option for one or more fields:
1. In the Source Fields grid, select the fields for which you want to set the Flat NULL Handling option.
2. Right-click on one of the fields and choose Set Flat NULL Handling
from the context menu, or choose Fields > Set Flat NULL Handling from the main menu.
3. MDM displays a cascading menu of three choices:
Preserve NULLs Ignore NULLs Inherit Default
4. To set the Flat NULL Handling option and override the global default
setting, choose either Preserve NULLs or Ignore NULLs, or alternatively,
to restore inheritance, choose Inherit Default.
NOTE ►► MDM places a check next to the cascading menu choice that corresponds to the current setting unless you have selected multiple fields for which the settings are not all the same.
280 MDM Import Manager Reference Guide
SETTING THE HIERARCHY NULL HANDLING OPTION
The current setting of the Hierarchy NULL Handling option determines how the MDM Import Manager preserves or ignores NULLs in the source data when you map a field, and must be properly set prior to mapping a source field to a destination field.
Specifically, the MDM Import Manager uses as the current hierarchy NULL-handling setting either: (1) the global default setting, if the field is set to inherit; or (2) the field-specific setting, if it has been manually set for the field.
You can set the global default hierarchy NULL-handling setting for all
fields that inherit the default using the Configuration Options command, or alternatively, override the default and set it on a field-by-field basis
using the Set Hierarchy NULL Handling command, as described in this section.
NOTE ►► Inherited global default values appear in the Source Fields grid in gray.
NOTE ►► See “The Hierarchy NULL Handling Option” on page 233 for more information about the Hierarchy NULL Handling option; see “Source Field-Level Settings” on page 224 for more information about all of the source field-level settings; see “Configuration Options” on page 62 for more information about the global default Hierarchy NULL Handling option.
To set the Hierarchy NULL Handling option for one or more fields:
1. In the Source Fields grid, select the fields for which you want to set the Hierarchy NULL Handling option.
2. Right-click on one of the fields and choose Set Hierarchy NULL Handling from the context menu, or choose Fields > Set Hierarchy
NULL Handling from the main menu.
3. MDM displays a cascading menu of three choices:
Preserve NULLs Ignore Non-Leaf Trailing NULLs Ignore All Trailing NULLs Inherit Default
4. To set the Hierarchy NULL Handling option and override the global
default setting, choose either Preserve NULLs, Ignore Non-Leaf Trailing NULLs, or Ignore All Trailing NULLs, or alternatively, to restore
inheritance, choose Inherit Default.
NOTE ►► MDM places a check next to the cascading menu choice that corresponds to the current setting unless you have selected multiple fields for which the settings are not all the same.
MDM Import Manager Reference Guide 281
APPLYING FIELD-LEVEL VALUE CONVERSION FILTERS
When you apply value conversion filters at the field level, the conversions are applied sequentially to each converted value that has not been individually edited or converted at the value level, starting with the original value.
For each field, you can apply one or more value conversion filters at the field level, as described in this section.
NOTE ►► When you first map a field, each value in the Converted Value column of the Source Values grid appears in gray to indicate that it is inheriting value conversion filters from the field level.
NOTE ►► Value conversion filters can also be applied at the individual value level (see “Applying Value-Level Value Conversion Filters” on page 314 for more information).
NOTE ►► See “Value Conversion Filters” on page 238 for more information about the value conversion filters.
To apply a single value conversion filter at the field level:
1. In the Source Fields grid, select the field to which you want to apply the value conversion filter.
2. Right-click on the field and choose Set Value Conversion Filter from the
context menu, or choose Fields > Set Value Conversion Filter from the main menu, and then choose from the cascading menu of value conversion filters.
NOTE ►► Each time you apply a single field-level value conversion filter, you replace any previously applied field-level filters for the field.
3. Follow the instructions for the corresponding value conversion filter, in the subsections of “Applying Value-Level Value Conversion Filters” starting on page 314.
4. MDM sets the filter and applies it to each Converted Value that it is inheriting value conversion filters from the field level.
To cancel field-level value conversion filters for the selected field:
1. In the Source Fields grid, select the field for which you want to cancel the value conversion filters.
2. Right-click on the field and choose Set Value Conversion Filter from the
context menu, or choose Fields > Set Value Conversion Filter from the
main menu, and then choose None from the cascading menu of value conversion filters.
282 MDM Import Manager Reference Guide
To apply multiple value conversion filters at the field level:
1. In the Source Fields grid, select the field to which you want to apply multiple value conversion filters.
2. Right-click on the field and choose Set Value Conversion Filter from the
context menu, or choose Fields > Set Value Conversion Filter from the
main menu, and then choose Multiple from the cascading menu to open the Set Value Conversion Filters dialog shown in Figure 186.
Figure 186. Set Value Conversions dialog
3. Click on the Add button and choose from the cascading menu of value conversion filters.
4. Follow the instructions for the corresponding value conversion filter, in the subsections of “Applying Value-Level Value Conversion Filters” starting on page 314.
NOTE ►► Each time you add a value conversion filter, it is added to the end of the Filters list of previously applied filters for the field.
TIP ►► To remove a filter, select it and click on the Remove button.
TIP ►► To modify a filter, right-click on it and choose Modify from the context menu.
TIP ►► To reorder a filter, drag-and-drop it in the list.
5. Click OK when you are done to close the Set Value Conversions dialog.
TIP ►► To quickly remove all field-level value conversion filters for
the selected field, right-click on the field and choose Set Value Conversion Filter from the context menu, or choose Fields > Set Value Conversion Filter from the main menu, and then choose None from the cascading menu of value conversion filters.
MDM Import Manager Reference Guide 283
FILTERING UNLINKED ATTRIBUTES
Before you map a source field to a destination attribute, it would often be convenient to know precisely which attributes are already linked to a category. This way you can identify a corresponding linked attribute (if one exists) rather than inadvertently map to an unlinked attribute, or worse, map to a new attribute that you create as part of the import.
NOTE ►► The MDM Import Manager automatically links unlinked attributes to a category into which you import records for which the source data field that is mapped to the attribute has non-NULL values. (See “Configuration Options” on page 62 for more information about the Autolink Attributes options.)
To highlight the attributes that are linked to one or more categories:
1. Right-click anywhere in the Destination Fields grid and choose Filter Unlinked Attributes from the context menu, or choose Fields > Filter Unlinked Attributes from the main menu, to open the Filter Unlinked Attributes dialog shown in Figure 187.
Figure 187. Filter Unlinked Attributes dialog
2. Move one or more categories whose linked attributes you want to highlight from Available Categories to the Selected Categories list.
3. Click OK when you are done to close the Filter Unlinked Attributes dialog. MDM highlights in gray the attributes in the Destination Fields grid that are not linked to the categories you selected.
NOTE ►► You can still map to an unlinked attribute even when it is highlighted in gray.
TIP ►► To cancel or change the selected categories, you can perform
the Filter Unlinked Attributes command again.
TIP ►► To sort the linked attributes as a group before the unlinked attributes, sort the Destination Fields grid by the F/D/Q/A column.
Resizable Dialog
284 MDM Import Manager Reference Guide
ADDING A NEW DESTINATION ATTRIBUTE
Sometimes, category-specific fields of source data do not have a corresponding destination attribute. You can create one or more new attributes as part of the import process as described in this section.
To create a new destination attribute to map to from a source field:
1. In the Source Fields grid, select the source field for which you want to create a new corresponding destination attribute.
2. Click on the Add button (between the fields grids) to open the Add Attribute dialog shown in Figure 188.
Figure 188. Add Attribute dialog
NOTE ►► If the current destination table has multiple taxonomy
fields, Add first brings up a cascading menu of taxonomy field names for you to select the one to which you want to add an attribute.
3. MDM automatically populates the Name property with the name of the selected source field.
4. Select the type of attribute from the radio controls, specify the other properties for the attribute, and click OK when you are done to close the dialog and create the attribute.
DATA INTEGRITY ►► The MDM Import Manager automatically links the new attribute to each category into which you import records for which the source data field that is mapped to the attribute has non-NULL values. (See “Configuration Options” on page 62 for more information about the Autolink Attributes options.)
MDM Import Manager Reference Guide 285
SETTING THE NULL INTERPRETATION OPTION
The current setting of the NULL Interpretation option determines whether NULL source values are ignored or used to overwrite existing destination data values.
Specifically, the MDM Import Manager uses as the current NULL Interpretation setting either: (1) the global default setting, if the field is set to inherit; or (2) the field-specific setting, if it has been manually set for the field.
You can set the global default NULL Interpretation setting for all fields
that inherit the default using the Configuration Options command, or alternatively, override the default and set it on a field-by-field basis using
the Set NULL Interpretation command, as described in this section.
NOTE ►► Inherited global default values appear in the Destination Fields grid in gray.
NOTE ►► See “The NULL Interpretation Option” on page 236 for more information about the NULL Interpretation option; see “Destination Field-Level Settings” on page 225 for more information about all of the destination field-level settings; see “Configuration Options” on page 62 for more information about the global default NULL Interpretation option.
To set the NULL Interpretation option for one or more fields:
1. In the Destination Fields grid, select the object fields for which you want to set the NULL Interpretation option.
2. Right-click on one of the fields and choose Set NULL Interpretation
from the context menu, or choose Fields > Set NULL Interpretation from the main menu.
3. MDM displays a cascading menu of two choices:
Ignore Overwrite
4. To set the NULL Interpretation option and override the global default
setting, choose either Ignore or Overwrite, or alternatively, to restore
inheritance, choose Inherit Default.
NOTE ►► MDM places a check next to the cascading menu choice that corresponds to the current setting unless you have selected multiple fields for which the settings are not all the same.
286 MDM Import Manager Reference Guide
SETTING THE MULTI-VALUED UPDATE OPTION
The current setting of the Multi-Valued Update option determines whether incoming source data values either are appended to or completely replace the set of existing destination data values of a multi-valued destination field.
Specifically, the MDM Import Manager uses as the current Multi-Valued Update setting either: (1) the global default setting, if the field is set to inherit; or (2) the field-specific setting, if it has been manually set for the field.
You can set the global default Multi-Valued Update setting for all fields
that inherit the default using the Configuration Options command, or alternatively, override the default and set it on a field-by-field basis using
the Set Multi-Valued Update command, as described in this section.
NOTE ►► The Multi-Valued Update option applies only to multi-valued destination fields.
NOTE ►► Inherited global default values appear in the Destination Fields grid in gray.
NOTE ►► See “The Multi-Valued Update Option” on page 251 for more information about the Multi-Valued Update option; see “Destination Field-Level Settings” on page 225 for more information about all of the destination field-level settings; see “Configuration Options” on page 62 for more information about the global default Multi-Valued Update option.
To set the Multi-Valued Update option for one or more fields:
1. In the Destination Fields grid, select the multi-valued fields for which you want to set the Multi-Valued Update option.
2. Right-click on one of the fields and choose Set Multi-Valued Update
from the context menu, or choose Fields > Set Multi-Valued Update from the main menu.
3. MDM displays a cascading menu of three choices:
Append Replace Inherit Default
4. To set the Multi-Valued Update option and override the global default
setting, choose either Append or Replace, or alternatively, to restore
inheritance, choose Inherit Default.
NOTE ►► MDM places a check next to the cascading menu choice that corresponds to the current setting unless you have selected multiple fields for which the settings are not all the same.
MDM Import Manager Reference Guide 287
SETTING THE QUALIFIED UPDATE OPTION
The current setting of the Qualified Update option determines whether incoming source data values either are appended to or completely replace the set of existing destination data values of a qualified lookup destination field.
Specifically, the MDM Import Manager uses as the current Qualified Update setting either: (1) the global default setting (in conjunction with the setting of the Matching Qualifiers option), if the field is set to inherit; or (2) the field-specific setting, if it has been manually set for the field.
You can set both the global default Multi-Valued Update setting and the Matching Qualifiers setting (which work together in tandem) for all fields
that inherit the default using the Configuration Options command, or alternatively, override the default and set it on a field-by-field basis using
the Set Qualified Update command, as described in this section.
NOTE ►► The Qualified Update option applies only to qualified lookup destination fields.
NOTE ►► Inherited global default values appear in the Destination Fields grid in gray.
NOTE ►► See “The Qualified Update Option” on page 252 for more information about the Qualified Update option; see “Destination Field-Level Settings” on page 225 for more information about all of the destination field-level settings; see “Configuration Options” on page 62 for more information about the global default Qualified Update and Matching Qualifiers options.
To set the Qualified Update option for one or more fields:
1. In the Destination Fields grid, select the qualified lookup fields for which you want to set the Qualified Update option.
2. Right-click on one of the fields and choose Set Qualified Update from
the context menu, or choose Fields > Set Qualified Update from the main menu.
3. MDM displays a cascading menu of four choices:
Append Replace Update Inherit Default
4. To set the Qualified Update option and override the global default
setting, choose either Append, Replace, or Update, or alternatively, to
restore inheritance, choose Inherit Default.
NOTE ►► You can inherit only All or None for Matching Qualifiers since the set of qualifiers is different for each qualified lookup field.
288 MDM Import Manager Reference Guide
5. MDM opens the Set Matching Qualifiers dialog shown in Figure 189.
Figure 189. Set Matching Qualifiers dialog
NOTE ►► The New Links and Existing Links drop-down controls are
disabled when you choose Append or Replace; they are enabled only
when you choose Update.
6. In the dual-list control for qualifier selection, move the qualifiers that you want to update to the Available Qualifiers list and move the qualifiers you want to match to the Matching Qualifiers list.
7. If you chose Update in step 4, in the New Links drop-down control, choose from the list of new link options:
Skip Create
8. If you chose Update in step 4, in the Existing Links drop-down control, choose from the list of existing link options:
Skip Update (NULL Qualifiers Only) Update (All Mapped Qualifiers) Replace
9. Click OK when you are done to close the Set Matching Qualifiers dialog.
NOTE ►► MDM places a check next to the cascading menu choice that corresponds to the current setting unless you have selected multiple fields for which the settings are not all the same.
NOTE ►► The Append and Replace options use matching qualifiers
for qualified link aggregation only, while the Update option uses them both for qualified link aggregation and for matching. (See “Merging Qualified Links and the Qualified Update Option” on page 355 for more information about qualified link aggregation.
MDM Import Manager Reference Guide 289
SETTING THE TUPLE UPDATE OPTION
The current setting of a tuple field’s Tuple Update option determines whether incoming source values are added as new tuple records, used to update an existing tuple record, or skipped entirely.
Specifically, the Tuple Update option allows you to specify:
Which of the tuple’s member fields (if any) MDM should use as matching fields; and then
How to update the values of the remaining member fields for new or existing tuple records.
You can set Tuple Update options on a field-by-field basis using the
Tuple Update command, as described in this section.
NOTE ►► If you do not explicitly set the Tuple Update options for a tuple field, Import Manager updates the field using the default tuple update settings from the Configuration Options dialog (see “Configuration Options” on page 62 for more information).
NOTE ►► The Tuple Update command is disabled for single-valued tuple fields nested within a tuple structure. Instead, Import Manager applies the update option set for its parent tuple in the tuple structure.
NOTE ►► See “The Tuple Update Option” on page 259 for more information about the behaviors of each Tuple Update option.
To set the Tuple Update option for one or more fields:
1. In the Destination Fields grid, select the tuple field for which you want to set the Tuple Update option.
2. Right-click on the field and choose Tuple Update from the context
menu, or choose Fields > Tuple Update from the main menu.
3. MDM opens the Set Matching Tuples dialog shown in Figure 190.
Figure 190. Set Matching Tuples dialog
290 MDM Import Manager Reference Guide
4. In the dual-list control for field selection, move the tuple member fields that you want to update to the Available fields list and move the tuple member fields that you want to match on to the Matching fields list.
NOTE ►► If no fields are in the Matching fields list, then Import Manager will update all of the tuple‘s records with the incoming source data, according to the settings in the New records and Existing records drop-down controls.
5. In the New records drop-down control, choose from the list of options:
Skip Create
6. In the Existing records drop-down control, choose from the list of options:
Skip Update (NULL Fields Only) Update (All Mapped Fields) Replace Delete
7. Click OK when you are done to close the Set Matching Tuple Fields dialog.
SETTING THE DATA GROUP DISPLAY OPTION
The current setting of the Data Group Display option determines whether the Data Groups hierarchy is hidden or displayed in the Destination Values grid for object lookup destination fields.
Specifically, Import Manager uses as the current Data Group Display setting either: (1) the global default setting, if the field is set to inherit; or (2) the field-specific setting, if it has been manually set for the field.
You can set the global default Data Group Display setting for all fields
that inherit the default using the Configuration Options command, or alternatively, override the default and set it on a field-by-field basis using
the Set Data Group Display command, as described in this section.
NOTE ►► The Data Group Display option applies only to object lookup destination fields.
NOTE ►► Inherited global default values appear in the Destination Fields grid in gray.
NOTE ►► See “The Data Group Display Option” on page 265 for more information about the Data Group Display option; see “Destination Field-Level Settings” on page 225 for more information about all of the destination field-level settings; see “Configuration Options” on page 62 for more information about the global default Data Group Display option.
MDM Import Manager Reference Guide 291
To set the Data Group Display option for one or more fields:
1. In the Destination Fields grid, select the object fields for which you want to set the Data Group Display option.
2. Right-click on one of the fields and choose Set Data Group Display
from the context menu, or choose Fields > Data Group Display from the main menu.
3. MDM displays a cascading menu of two choices:
Hide Data Group Display Data Group
4. To set the Data Group Display option and override the global default
setting, choose either Hide Data Group or Display Data Group, or
alternatively, to restore inheritance, choose Inherit Default.
NOTE ►► MDM places a check next to the cascading menu choice that corresponds to the current setting unless you have selected multiple fields for which the settings are not all the same.
SETTING THE MDIS UNMAPPED VALUE HANDLING OPTION
The current setting of the MDIS Unmapped Value Handling option determines whether unmapped values are ignored, added to the set of existing destination values, or mapped to an existing destination value during MDIS processing.
Specifically, the MDM Import Manager uses as the current MDIS Unmapped Value Handling setting either: (1) the global default setting, if the field is set to inherit; or (2) the field-specific setting, if it has been manually set for the field.
You can set the global default MDIS Unmapped Value Handling setting
for all fields that inherit the default using the Configuration Options command, or alternatively, override the default and set it on a field-by-
field basis using the Set MDIS Unmapped Value Handling command, as described in this section.
NOTE ►► Inherited global default values appear in the Destination Fields grid in gray.
To set the MDIS Unmapped Value Handling option for one or more fields:
1. In the Destination Fields grid, select the fields for which you want to set the MDIS Unmapped Value Handling option.
2. Right-click on one of the fields and choose Set MDIS Unmapped Value Handling from the context menu, or choose Fields > Set Unmapped
Value Handling from the main menu.
292 MDM Import Manager Reference Guide
3. MDM displays a cascading menu of four choices:
Ignore Add Map Fail Inherit Default
4. To set the MDIS Unmapped Value Handling option and override the
global default setting, choose either Ignore, Add, Map, or Fail, or
alternatively, to restore inheritance, choose Inherit Default.
NOTE ►► You cannot inherit the Map option since the set of destination values is different for each destination field.
NOTE ►► The Add option is disabled for Boolean fields. You cannot import an unmapped value to a Boolean field.
5. If you chose Map in step 4, MDM opens one of the MDIS Mapped Unmapped Value dialogs shown in Figure 191 and Figure 192, for single and multi-valued destination fields, respectively.
Figure 191. MDIS Map Unmapped Value dialog (single-valued)
Figure 192. MDIS Map Unmapped Value dialog (multi-valued)
Resizable Dialog
Resizable Dialog
MDM Import Manager Reference Guide 293
6. The MDIS Map Unmapped Value dialogs are populated with the set of existing values for the destination field. Choose: (1) one value for a single-valued destination field; or (2) one or more values for a multi-
valued destination field, and click OK.
NOTE ►► MDM places a check next to the cascading menu choice that corresponds to the current setting unless you have selected multiple fields for which the settings are not all the same.
294 MDM Import Manager Reference Guide
Value Conversion and Mapping Operations
The following sections describe the various value conversion and mapping operations you can perform on source and destination values in the bottom pair of Value Conversion and Mapping grids, and are summarized in Table 56.
Table 56. Value Conversion and Mapping Operations
Operation Description
Buttons
Automap Automatically maps each source value to destination values that are equivalent.
Map Maps the selected source values to the selected destination values.
Unmap Unmaps the selected values.
Add Adds the selected source values as new destination values.
Remove Removes the selected new destination values.
Source Values Grid
Apply Value Conversion Filter
Converts the selected source values using the specified value conversion filter.
Find Next Unmapped Source Value
Finds the first or next unconverted or unmapped source value.
Reorder Mapped Destination Values
Reorders the mapped destination values for the selected source value.
Find Next Mapped Destination Value
Finds the first or next destination value to which the selected source value is mapped.
Select Entire Branch Selects the applicable children of the selected internal node.
Destination Values Grid
Add Value Adds a new destination value.
Remove Value Removes the selected new destination values.
MDM Import Manager Reference Guide 295
NOTE ►► NULL handling, case sensitivity, and delimiters must be set before you field map because they affect how original values are interpreted and how converted values are generated; the NULL Handling and Case Sensitivity options also affect the set of original values. By contrast, value conversion filters must be set after you map because they depend on the destination data type. (See “The NULL Handling Options” on page 231, “The Case Sensitivity Option” on page 237, and “Value Conversion Filters” on page 238 for more information about these field-level settings.)
SOURCE VALUE CONVERSION AND MAPPING
Converting and mapping field values is also extremely straightforward using the Value Conversion and Mapping grids, shown in Figure 193.
Figure 193. Value Conversion and Mapping grids
For each mapped source field, the value conversion and mapping process depends upon whether or not value mapping is required:
No value mapping. If value mapping is not required, you must make sure that every source value is properly converted, as indicated by a
green bullet ( ) in the Conv. column of the Source Values grid.
Value mapping. If value mapping is required, you must make sure that every source value is mapped, as indicated by a colored
thunderbolt ( ) in the Mapped column of the Source Values grid
Converting and mapping source values is described in the following sections; the specific meanings of each of the colored thunderbolt icons when value mapping is required are summarized in Table 57.
296 MDM Import Manager Reference Guide
Table 57. Mapped Icons in the Source and Destination Values Grids
Values Grid Icon Description
Source
Value is mapped to an existing destination value.
Value is mapped to multiple existing destination values.
Value is mapped to a new destination value.
Value is mapped to multiple new destination values.
Value is mapped to multiple destination values, some of which are new.
Destination
Single source value is mapped to the value.
Multiple source values are mapped to the value.
Single source value is mapped to the new value.
Multiple source values are mapped to the new value.
Source Values Grid
Each of the values of the current source field appears in the Source Values grid and is available for conversion and/or mapping, as follows:
Value column. Original source values appear with their original data
type in the Value column.
Converted Value column. Converted values appear in the
Converted Value column with their data types automatically converted based on the data type of the mapped destination field, (even when stored in the source data as text values), and the proper grid cell type for editing (e.g. text, numeric, measurement, currency, date, time, Boolean). If necessary, individual values can then be manually edited to override errors or omissions in the source data.
NOTE ►► If the mapped source field or any of its values has been partitioned, then the merged values (combined partition) or the hierarchy of values (nested partition) appear in the Source Values grid.
NOTE ►► When you first map a field, each value in the Converted Value column appears in gray to indicate that it is inheriting value conversion filters from the field level. When you apply a manual edit or a value-conversion filter to an individual value, the Converted Value then appears in black to indicate that you have overridden inheritance
from the field level. You can use the Restore Converted Value command to restore inheritance for a value.
NOTE ►► When value mapping is required, the value in the Converted Value column is most useful for adding new destination values or hierarchy, where it is used as the new item or node value. When mapping to existing destination values, the Converted Value is
just a convenience for Automap, since each value could be manually mapped to any destination value regardless of its value.
MDM Import Manager Reference Guide 297
Destination Values Grid
Each of the values of the current destination field appears in the Destination Values grid and you can map to any of them.
NOTE ►► If the mapped destination field or any of its values has been partitioned, then the merged value combinations (combined partition) or a hierarchy of value combinations (nested partition) appear in the Destination Values grid.
NOTE ►► A hierarchy of values appears in the Destination Values grid also when the current table is a hierarchy table and the mapped destination field is the display field of the hierarchy table.
MULTILINGUAL ►► In a multilingual repository, the Destination Values grid may contain inherited values from language layers other than the current language, which you can set as part of the value mapping. More information about languages and multilingual repositories is provided in “Part 14: Multilingual Support.”
EDITING SOURCE VALUES
Whether or not the mapped destination field requires value mapping, you can use the edit controls in the Converted Value column of the Source Values grid to manually edit each source value.
NOTE ►► When the mapped destination field does not require value mapping, you are done converting values when “n of n values converted” appears on the status bar.
Single-Valued Destination Field
When a source field is mapped to a single-valued destination field, you can manually edit each individual source value using MDM’s standard single-valued edit controls, as described in this section.
NOTE ►► If the destination field requires value mapping, you can map each source data value even without manually editing it.
To edit an individual source value (single-valued destination):
1. In the Source Values grid, select the value that you want to edit.
2. Make sure the current cell is on the Converted Value column, as shown in Figure 194.
298 MDM Import Manager Reference Guide
Figure 194. Single-valued edit control
3. Type a new value, or press Enter to open the cell and edit or overwrite the current value.
NOTE ►► The cell type of the Converted Value cells depends upon the data type of the mapped destination field.
TIP ►► You can also double-click or press F2 to enter the cell and begin editing.
4. Press Enter when you are done to close the cell and save the value.
Multi-Valued Destination Field
When a source field is mapped to a multi-valued destination field, you can manually edit each individual source value using MDM’s special multi-valued drop-down edit controls to edit or create one or more values for each source value, as described in this section.
NOTE ►► If the destination field requires value mapping, you can map each source data value to multiple destination values even without manually expanding it into multiple values.
To edit an individual source value consisting of one or more delimited values (multi-valued destination):
1. In the Source Values grid, select the value that you want to edit.
2. Make sure the current cell is on the Converted Value column.
3. Double-click on the cell or click on the down triangle to open the multi-valued drop-down control, as shown in Figure 195.
MDM Import Manager Reference Guide 299
Figure 195. Multi-valued drop-down edit control
NOTE ►► The cell type of the individual cells within the drop-down control depends upon the data type of the mapped destination field.
4. MDM populates each row of the multi-valued drop-down control with one of the multiple delimited values of the original source value.
5. Edit the values in each row of the drop-down control as follows:
Use the Up and Down arrow keys to move from value to value. Press Enter to open a row and edit or overwrite its current value. Move to the empty row beyond the last value to add a value.
Press Ctrl+Up and Ctrl+Down to reorder the values.
Press Del to delete a value.
6. Click on the up triangle or press Shift+Enter when you are done to close the drop-down control and save the values.
MAPPING SOURCE VALUES
The value mapping operations are described in the following sections and summarized in Table 58.
300 MDM Import Manager Reference Guide
Table 58. Value Mapping Operations
Button Description and Constraints
Map
All cases
Map the selected source item(s) to the selected destination item(s).
Mapping multiple source values to a single destination value eliminates duplicate source values and collapses source value variations into a single, normalized destination value.
Mapping a single source value to multiple destination values creates a multi-value (but is legal only if the destination field is multi-valued); if the source value is delimited, the individual values are irrelevant for the mapping.
Neither Source Values grid nor Destination Values grid can be empty
Hierarchy Flat
At least one of the selected source nodes must be a leaf node (can map source leaf nodes only; cannot map an internal node)
Flat Hierarchy (hierarchy lookup field)
At least one of the selected destination nodes must be a leaf node (can map to destination leaf nodes only; cannot map to an internal node)
Flat Hierarchy (hierarchy table primary display field)
No constraints (can map to both destination internal and leaf nodes)
Hierarchy Hierarchy (hierarchy lookup field)
At least one of the selected source nodes and one of the selected destination nodes must be a leaf node (can map leaf nodes to leaf nodes only; cannot map from or to internal nodes)
Hierarchy Hierarchy (hierarchy table primary display field)
No constraints (can map any leaf or internal node to any leaf or internal node)
Single Selection Multiple Selections
The selected source value must not yet be mapped to the selected destination value
If single-valued destination, the selected source value cannot already be mapped to a destination value other than the selected destination value (otherwise, would create a one-to-many multi-valued mapping)
Multiple Selections Single Selection
At least one of the selected source values must not yet be mapped to the selected destination value (otherwise, no new mappings are represented by the selections)
If single-valued destination, none of the selected source values can already be mapped to a destination value other than the selected destination value (otherwise, would create a one-to-many multi-valued mapping)
Single Selection Multiple Selections
Destination must be multi-valued
The selected source value must not yet be mapped to at least one of the selected destination values (otherwise, no new mappings are represented by the selections)
Multiple Selections Multiple Selections
Disabled (too confusing)
Unmap
All cases
Unmap the selected item(s) in focus that were previously mapped.
The Unmap applies to whichever values grid last had the focus
At least one of the selected items in current or previous focus must already be mapped
MDM Import Manager Reference Guide 301
Button Description and Constraints
Add
Flat / Hierarchy Flat
If necessary, add the selected source item(s) [leaf node(s)] after the last destination item, and then map each source item [leaf node] to each new or existing destination item.
Destination Values grid must be flat
At least one of the selected source items must not already be mapped
Hierarchy Flat
At least one of the unmapped selected source nodes must be a leaf node (can add source leaf nodes only; cannot add from an internal node)
Hierarchy Hierarchy
Not applicable (see Add as Sibling/Child and Add Branch as Sibling/Child)
Add as Sibling
Flat / Hierarchy Hierarchy
If necessary, add the selected source item(s) [node(s)] as siblings after the selected destination node; then map each source item [node] to each new or existing destination leaf node.
Destination Values grid must be a hierarchy
At least one of the selected source items [nodes] must not already be mapped
Source items must be leaf nodes (hierarchy lookup field)
Source items can be internal or leaf nodes (hierarchy table primary display field)
Hierarchy Hierarchy
All the selected source nodes must be leaf nodes or at the same level
Add
as Child
Flat / Hierarchy Hierarchy
If necessary, add the selected source item(s) [node(s)] as children of the selected destination node; then map each source item [node] to each new or existing destination leaf node.
Destination Values grid must be a hierarchy
At least one of the selected source items [nodes] must not already be mapped
Source items must be leaf nodes (hierarchy lookup field)
Source items can be internal or leaf nodes (hierarchy table primary display field)
Cannot add a child to a leaf node that already has records linked
Cannot add a child to a leaf node that has just been added from a source item [leaf node] (hierarchy lookup field)
Can add a child to a leaf node that has just been added from a source item [node] (hierarchy table primary table display field)
Hierarchy Hierarchy
All the selected source nodes must be leaf nodes or at the same level
Add Branch as Sibling
Hierarchy Hierarchy only
If necessary, add the selected branches and leaf nodes in their entirety as siblings after the selected destination node; if necessary, add each of their descendents; then map each source node to each new or existing destination node.
Source and Destination Values grids must both be a hierarchy
All the selected source nodes must be at the same level and at least one must be an internal node
At least one of the source nodes must not already be mapped
Each source leaf node is mapped to each new or existing destination leaf node (hierarchy lookup field)
Each source internal and leaf node is mapped to each new or existing destination internal or leaf node (hierarchy table primary display field)
302 MDM Import Manager Reference Guide
Button Description and Constraints
Add Branch as Child
Hierarchy Hierarchy only
If necessary, add the selected branches and leaf nodes in their entirety as children of the selected destination node; if necessary, add each of their descendants; then map each source node to each new or existing destination node.
Source and Destination Values grids must both be a hierarchy
All the selected source nodes must be at the same level and at least one must be an internal node
At least one of the source nodes must not already be mapped
Each source leaf node is mapped to each new or existing destination leaf node (hierarchy lookup field)
Each source internal and leaf node is mapped to each new or existing destination internal or leaf node (hierarchy table primary display field)
Cannot add a child branch to a leaf node that already has records linked
Cannot add a child to a leaf node that has just been added from a source item [leaf node] (hierarchy lookup field)
Can add a child to a leaf node that has just been added from a source item [node] (hierarchy table primary table display field)
Remove
All cases
Remove the selected destination item(s) that were previously added.
Flat Hierarchy & Hierarchy Hierarchy
Removing an internal node also removes all of its children.
At least one of the selected destination items must be new
NOTE ►► You are done mapping values when “n of n values mapped” appears on the status bar.
NOTE ►► Button actions are selection-dependent, forgiving, and with
the exception of Unmap, focus-independent. “Forgiving” means that any selected value that can’t be operated upon does not disable the button; i.e. at least one but not all the selected items need to participate in order for the button to be enabled.
NOTE ►► Single and multiple selection refers to whether a single item is selected or multiple items are selected in each values grid.
NOTE ►► Many-to-one value mapping of multiple source values to a single destination value collapses source value variations into a single, normalized destination value.
NOTE ►► One-to-many value mapping of a single source value to multiple destination values populates a multi-valued destination field with multiple values.
NOTE ►► Delimiters in delimited source values are most useful for conversion of numeric source values that do not require mapping, but
just a convenience for Automap when value mapping is required.
MDM Import Manager Reference Guide 303
TIP ►► Since you cannot add or map the source root node value, you can instead add and map an entire source hierarchy as follows:
(1) close the source tree to expose only the root;
(2) reopen the root node to display just the first level children;
(3) press Ctrl+A to select all the nodes;
(4) use Ctrl to unselect the root node; and
(5) use the appropriate Add Branch choice to add the hierarchy.
TIP ►► The Add Branch as Sibling/Child buttons only add source internal nodes leading to unmapped source leaf nodes that are added
and mapped as part of the Add. You must use the Add as Sibling/Child buttons to add source internal nodes without adding leaf nodes.
Mapping a Source Value
You can map a source value to a destination value as described in this section. Corresponding source and destination values do not need to be the same.
To map a source value to a destination value:
1. In the Source Values grid, select the source value that you want to map.
NOTE ►► When the source field is a hierarchy, you can only map leaf-node source values to destination values (destination flat or hierarchy lookup field).
2. In the Destination Values grid, select the destination value to which you want to map the source value.
NOTE ►► When the source field is mapped to a destination hierarchy, you can only map to leaf-node destination values (destination hierarchy lookup field).
TIP ►► When a source field is mapped to a multi-valued destination field you can map each source value to multiple destination values, regardless of whether it consists of multiple delimited values. However, if the destination field is a lookup into a table that has key mapping enabled, you can only map each source vale to one destination value.
3. Click on the Map button (between the values grids) to map the selected source value and the selected destination value.
4. If you are mapping to an inherited value in a multilingual repository, choose from the cascading menu:
Set Missing Value Do Not Set Missing Value
304 MDM Import Manager Reference Guide
MULTILINGUAL ►► More information about languages and multilingual repositories is provided in “Part 14: Multilingual Support.”
5. MDM indicates that the source and destination values have been mapped by placing a green thunderbolt ( ) in the Mapped column of both grids, and by placing the source and destination mapped values into the corresponding column of the other grid.
TIP ►► You can map any non-NULL source value to the <NULL> destination value to discard the source value. Alternatively, you can map a <NULL> source value to any non-NULL destination value.
Unmapping Values
You can unmap one or more previously mapped source values as described in this section.
To unmap one or more source values from their mapped destination values:
1. In the Source Values grid, select the mapped source values that you want to unmap.
TIP ►► Alternatively, you can select the mapped destination values in the Destination Values grid.
NOTE ►► When you select a mapped source value in the Source Values grid, the corresponding mapped destination values are automatically selected in the Destination Values grid (but not the other way around).
2. Click on the Unmap button (between the values grids) to unmap the source values from their mapped destination values.
NOTE ►► The Unmap button is focus-dependent and unmaps the selected values from the grid that either: (1) currently has the focus; or (2) last had the focus if neither values grid currently has the focus.
3. MDM indicates that the source and destination values have been unmapped by removing the green thunderbolt ( ) from the Mapped column of both grids for the unmapped values (but only if they are no longer mapped to or from any other values), and by removing the source and destination mapped values from the corresponding column of the other grid.
Adding New Destination Values
You can add one or more new values to the list of destination values in either of two ways, as described in this section.
MDM Import Manager Reference Guide 305
To add and map one or more source values to the list of destination values:
1. In the Source Values grid, select the source values that you want to add.
2. Click on the Add button (between the values grids) to add the source values to the list of destination values.
NOTE ►► The Add button is disabled if the display field of the destination lookup field is compound and the source field is not compound, or if the single destination display field is not of type Text, Text Normalized, Text Block, Integer, or Measurement.
NOTE ►► You must use MDM Data Manager to import new images, PDFs, and binary objects into the repository, since the MDM Import Manager imports links to these objects, not the objects themselves.
NOTE ►► If the object lookup destination field is a text block field and the Data Groups hierarchy is displayed in the Destination Values grid, you can add a new text block value into a specific data group. If the Data Groups hierarchy is not displayed, the text block is added to a Data Group named Imported Text Blocks.
3. MDM: (1) adds the Converted Value of each source value that does not already exist to the list of destination values in blue; (2) maps all of the selected source values to their corresponding destination values; (3) indicates that the source and destination values have been mapped by placing one of the blue thunderbolts ( ) in the Mapped column of both grids; and (4) places the source and destination mapped values into the corresponding column of the other grid.
NOTE ►► Values that do not already exist are added and all values are mapped.
To add one or more values directly to the list of destination values:
1. In the Destination Values grid, right-click anywhere and choose Add Value from the context menu, or choose Values > Add Destination
Value from the main menu.
NOTE ►► The Add Value commands are disabled if the display field of the destination lookup field is compound or if the single destination display field is not of type Text, Text Normalized, or Text Block.
2. MDM adds a new value named “New Value” just after the last value in the list of Destination Values, and highlights it for editing (Figure 196).
306 MDM Import Manager Reference Guide
Figure 196. Adding a new destination value
3. Type the new value you want and press Enter.
4. MDM adds the new value in blue.
Adding New Destination Hierarchy
Recall from the previous parts of this guide that you can create or reconstitute source hierarchy as part of the import process by splitting or partitioning source fields.
One of the most powerful features of the MDM Import Manager is the ability – on the destination side – to not only precisely map to existing destination hierarchy, but also to automatically create new destination
hierarchy as part of the value-mapping process.
The set of destination values appears as a hierarchy in the Destination Values grid in the following cases:
Hierarchy lookup field. When the mapped destination field is a
hierarchy lookup field, the destination values are the hierarchy of the underlying lookup table.
Hierarchy table primary display field. When the current table is a
hierarchy lookup table and the mapped destination field is the primary display field of the hierarchy table, the destination values are also the hierarchy of the lookup table.
Nested partition. When you apply a nested partition to the mapped
destination field or any of its values, the resulting hierarchy of destination value combinations appear as a hierarchy of values in the Destination Values grid. (NYI)
When the Destination Values grid is a hierarchy of values, the Add button (between the values grids) pops up a context menu with four different command choices, as shown in Figure 197.
MDM Import Manager Reference Guide 307
Figure 197. Pop-Up Add Menu
Subject to the constraints identified in Table 58 on page 300, you can add new leaf-node destination values and also internal branch nodes to the destination hierarchy, as summarized in Table 59 and described in this section.
Table 59. Add to Destination Hierarchy Add Operations
Operation Description
Add as Sibling Adds the selected source values as siblings of the selected destination value.
Add as Child Adds the selected source values as children of the selected destination value.
Add Branch as Sibling Adds the selected internal source values and all of their children as siblings of the selected destination value.
Add Branch as Child Adds the selected internal source values and all of their children as children of the selected destination value.
To add one or more internal- or leaf-node source values as branches or as individual nodes to the hierarchy of destination values:
1. In the Source Values grid, select the source values you want to add.
NOTE ►► If the Source Values grid is a hierarchy (destination hierarchy lookup field), the source values must all be leaf nodes or at the same level. As a convenience, you can select and add both
internal and leaf nodes at the same time with the Add commands.
2. Click on the Add button (between the values grids) and choose from the cascading menu:
Add as Sibling Add as Child Add Branch as Sibling Add Branch as Child
308 MDM Import Manager Reference Guide
NOTE ►► All four commands always appear when the Destination
Values grid is a hierarchy. The Add as Sibling and Add as Child
commands are always enabled, while the Add Branch as Sibling and
Add Branch as Child commands are enabled only when the Source Values grid is also a hierarchy.
NOTE ►► The Add button is disabled if the display field of the destination lookup field is compound and the source field is not compound, or if the single destination display field is not of type Text, Text Normalized, Integer, or Measurement.
3. MDM adds the Converted Value of each source value that does not already exist to the hierarchy of destination values in blue, and maps all of the selected leaf-node source values to their corresponding destination values, as described in the previous section.
NOTE ►► Only leaf nodes can be mapped to leaf nodes.
NOTE ►► When the current table is a hierarchy table and the mapped destination field is the primary display field representing the
hierarchy, MDM maps both internal and leaf nodes as part of the Add.
To add one or more values directly to the hierarchy of destination values:
1. In the Destination Values grid, select the value either: (1) after which you want the new value to appear as a sibling; or (2) under which you want the new value to appear as a child.
2. Right-click on the value and choose Add Hierarchy Value from the
context menu, or choose Values > Add Destination Hierarchy Value from the main menu, and then choose from the cascading menu:
As Sibling As Child
NOTE ►► The Add Value commands are disabled if the display field of the destination lookup field is compound or if the single destination display field is not of type Text, Text Normalized, or Text Block.
3. MDM adds a new value named “New Value” either: (1) as a sibling of and just after the selected value; or (2) as the last child of the selected value, and highlights it for editing (Figure 196).
MDM Import Manager Reference Guide 309
Figure 198. Adding a new destination hierarchy value
4. Type the new value you want and press Enter.
5. MDM adds the new value in blue.
Removing New Values
You can remove one or more previously added destination values as described in this section.
To remove one or more values from the set of destination values:
1. In the Destination Values grid, select the previously added values that you want to remove.
2. Click on the Remove button (between the values grids), or right-click on
one of the values and choose Remove Value from the context menu.
3. MDM removes the values from the set of destination values.
NOTE ►► If you remove an internal node, all of its child nodes are automatically removed as well.
Automapping Source Values
If some or all of the source values are exactly the same as the
corresponding destination values, you can use the Automap button to automatically map them to their matching destination values, as described in this section.
NOTE ►► The Automap button uses the Converted Value in the Source Values grid when comparing source and destination values.
NOTE ►► When you map a source field containing remote key values to a destination lookup field for a table that has key mapping enabled, the MDM Import Manager automatically automaps the source values based on the remote keys of the lookup table for that remote system.
310 MDM Import Manager Reference Guide
To automatically map one or more source values to their corresponding destination values :
Click on the Automap button (between the values grids). MDM automatically maps each unmapped source value to the set of identical destination values.
TIP ►► Each time click on the Automap button, MDM attempts to map additional source values based on the current set of Converted Values.
NOTE ►► When you automap from a source hierarchy to a destination hierarchy, MDM compares either the full path or just the leaf node of the source and destination value based on the setting of the Hierarchy Value Automap option (see “Configuration Options” on
page 62 for more information). The Full Path setting ignores the root node of both the source and destination hierarchies.
NOTE ►► When the current table is a hierarchy table and the mapped destination field is the primary display field representing the hierarchy, MDM automaps both internal and leaf nodes.
NOTE ►► If the source values are delimited strings that represent a hierarchy and you want to compare the source values against the full
path of the destination values, then: (1) use the Split into Hierarchy command to create a source hierarchy; and (2) make sure the
Hierarchy Value Automap option is set to Full Path.
NOTE ►► As a convenience, Automap will overmap a source value if it matches more than one destination value, even if the destination field
is single-valued. You can then use the Find Next Mapped Destination Value command to find each mapped destination value and unmap all but one of the multiple mapped destination values (see “Finding Mapped Destination Values” on page 324 for more information). You cannot import the source data if any source values remain overmapped.
Mapping to a Compound Lookup Field
Recall that when the current table is the main table and a destination lookup field consists of multiple display fields, the MDM Import Manager automatically promotes the individual lookup table display fields into the list of main table fields.
NOTE ►► Promoted lookup table display fields appear in the Destination Fields grid as “display field n [lookup field]” and compound lookup fields appear as “lookup field [display field 1; display field n]” (where “display field n” is the name of the nth lookup table display field and “lookup field” is the name of the compound lookup field).
MDM Import Manager Reference Guide 311
Whether you map directly or indirectly to the compound lookup field depends upon how many source fields correspond to it:
Single source field. If the source data has only a single
corresponding field, you must simply field and value map from the applicable source field to the compound destination lookup field. MDM uses the Multi-Valued delimiter to separate each source data value into the multiple delimited values corresponding to the value combinations of the compound lookup field.
NOTE ►► The edit controls in the Converted Value column of the Source Values grid are disabled and cannot be opened for editing any of the individual delimited values.
NOTE ►► The Add button between the Value Mapping grids and the
Add Values commands for the Destination Values grid are also both disabled.
TIP ►► If the source value combinations represented by the single source field do not all correspond to existing values of the destination
lookup field, you must use the Split into Multiple Fields command to split the delimited source values into multiple source fields, and then map them to the compound lookup field as described in this section.
NOTE ►► Whereas a multi-valued field stores multiple values in a single field, a compound lookup field corresponds to a multi-field value combination across each of the multiple lookup table display fields.
Multiple source fields. If the source data has individual fields that
correspond one-for-one to the individual display fields of the lookup table, you can field map the individual source fields to the promoted lookup table display fields and then create a compound source field for mapping to the compound destination lookup field, as described in this section.
To map to a compound lookup field when you have multiple corresponding source fields:
1. Field map from individual source fields to the promoted lookup table display fields.
2. Either convert source field values, or optionally value map to the values of the individual Text display fields.
3. Use the Create Compound Field command to create a source field combination corresponding to the display field combination.
4. Field map from the compound source field to the compound lookup field.
5. Value map from the compound field value combinations to the value combinations represented by the lookup field.
312 MDM Import Manager Reference Guide
Mapping to a Coupled Numeric Attribute
When the mapped destination is a coupled numeric attribute, each source data value consists of one or more primary/coupled value pairs; when you apply a value conversion filter, the command for each filter
leads to a cascading menu for the Primary Value and the Coupled Value, as shown in Figure 199.
Figure 199. Cascading menu for a coupled numeric attribute
NOTE ►► A source data field that contains multiple delimited coupled numeric value pairs requires that you set both the Coupled Numeric and Multi-Valued delimiters. For example, “5hp @ 3500rpm; 6hp @ 4500rpm; 7hp @ 5500rpm” would require that the Coupled Numeric delimiter be set to “@” and the Multi-Valued delimiter be set to “;” (see “Setting the Split Delimiters” on page 150 for more information).
MAPPING TO AN OBJECT LOOKUP DESTINATION FIELD
You can map a source field to an object lookup destination field just like any other lookup field, and then map source values to destination values in either of two ways:
Flat list. MDM displays all of the object lookup values in a list in the
Destination Values grid and you can map to existing destination values (and add new text values).
NOTE ►► See “Mapping a Source Value” on page 303 and “Adding New Destination Values” on page 304 for more information on how to map source values (for all object lookup fields) and add new destination values (for text block lookup fields).
Data Groups hierarchy. MDM displays all of the object names or
values as leaf nodes beneath their data group in the Data Groups hierarchy in the Destination Values grid.
NOTE ►► For each object lookup destination field, you can set the Data Group Display option (see “The Data Group Display Option” on page 265) based on the requirements of the situation.
MDM Import Manager Reference Guide 313
Using the Data Groups Hierarchy
You can map a source field to an object lookup destination field using the Data Groups hierarchy as described in this section.
To map to object lookup fields using data groups:
1. Make sure the Value Map > Hierarchy configuration option is set to Full Path, and the Data Group Display option for the object lookup
destination field is set to Display Data Group.
2. Partition the source data group field by the source object field to create a hierarchy of data groups and objects.
3. Map the partitioned source data group field to the object lookup destination field.
4. Map each of the source values by mapping the source hierarchy leaf-node values to corresponding destination hierarchy leaf-node values.
NOTE ►► You must use MDM Data Manager to import new images, PDFs, and binary objects into the repository, since MDM Import Manager imports links to these objects, not the objects themselves.
NOTE ►► If the object lookup destination field is a text block field and the Data Groups hierarchy is displayed in the Destination Values grid, you can add a new text block value into a specific data group. If the Data Groups hierarchy is not displayed, the text block is added to a Data Group named Imported Text Blocks.
Adding a New Data Group
If the object lookup destination field is a text block field and the Data Groups hierarchy is displayed in the Destination Values grid, you can add a new data group to the Data Groups hierarchy before adding new text block values, as described in this section.
To add a new data group directly to the Data Groups hierarchy:
1. In the Destination Values grid, select the data group either: (1) after which you want the new data group to appear as a sibling; or (2) under which you want the new data group to appear as a child.
2. Right-click on the data group and choose Add Data Group from the
context menu, or choose Values > Add Data Group from the main menu, and then choose from the cascading menu:
As Sibling As Child
3. MDM adds a new data group named “New Value” either: (1) as a sibling of and just after the selected data group; or (2) as the last child of the selected data group, and highlights it for editing.
4. Type the new name you want for the data group and press Enter.
5. MDM adds the new data group in light blue.
314 MDM Import Manager Reference Guide
APPLYING VALUE-LEVEL VALUE CONVERSION FILTERS
When you apply a value conversion filter at the value level, the conversion is applied to the converted value resulting from the prior
application of field-level filters, value-level filters, and manual edits.
For each value, you can apply one or more value conversion filters at the individual value level as described in the following sections.
NOTE ►► If the Converted Value consists of multiple delimited values, the value conversion filter is applied to each of the individual values within the Converted Value.
NOTE ►► Unless noted, each of the value conversion filters can also be applied at the field level (see “Applying Field-Level Value Conversion Filters” on page 281 for more information).
Restoring Converted Values
When you apply a manual edit or a value-level conversion filter to an individual value, the Converted Value changes from gray to black to indicate that you have overridden inheritance from the field level.
You can use the Restore Converted Value filter to discard manual edits and value-level conversions and restore inheritance for one or more Converted Values, as described in this section.
NOTE ►► You can use the Restore Converted Value filter only at the value level and not at the field level.
To restore inheritance for one or more values:
1. In the Source Values grid, select the values for which you want to restore inheritance.
2. Right-click on one of the values and choose Restore Converted Value
from the context menu, or choose Values > Apply Value Conversion
Filter > Restore Converted Value from the main menu.
3. MDM restores inheritance for each value, which once again appears in gray in the Source Values grid.
Setting to Original Values
When you apply field-level conversion filters, they are automatically applied to each source value in the Source Values grid that is inheriting field-level conversions, so that each Converted Value will be different from its corresponding original Value. Moreover, applying any manual edits or value-level conversion filters will also change the Converted Value so that it is different from the original Value.
MDM Import Manager Reference Guide 315
You can use the Set to Original Value filter to set one or more Converted Values back to their original value – and in effect “undo” the effects of the field-level value conversion filters and any manual edits or value-level conversion filters – as described in this section.
NOTE ►► You can use the Set to Original Value filter only at the value-level and not at the field level.
To set one or more values back to their original values:
1. In the Source Values grid, select the values that you want to set back to their original values.
2. Right-click on one of the values and choose Set to Original Value from
the context menu, or choose Values > Apply Value Conversion Filter >
Set to Original Value from the main menu.
3. MDM sets each value back to its original value, which appears in black in the Source Values grid.
Setting Values to NULL
You cannot directly enter a NULL as the Converted Value. Instead, you
can use the Set Value to NULL filter to set one or more Converted Values to NULL, as described in this section.
NOTE ►► When a field does not require value mapping, a source value with a <NULL> Converted Value has a green bullet ( ) in the Conv. column and is deemed to be properly converted.
To set one or more values to NULL:
1. In the Source Values grid, select the values that you want to set to NULL.
2. Right-click on one of the values and choose Set Value to NULL from
the context menu, or choose Values > Apply Value Conversion Filter >
Set Value to NULL from the main menu.
3. MDM sets each value to <NULL>.
Clearing Values
You can use the Clear Value filter to quickly clear one or more Converted Values, so that you don’t inadvertently forget to manually enter a value for each source value, as described in this section.
NOTE ►► When a field does not require value mapping, a source value with an empty Converted Value is deemed to not yet be converted.
316 MDM Import Manager Reference Guide
To clear one or more values:
1. In the Source Values grid, select the values that you want to clear.
2. Right-click on one of the values and choose Clear Value from the
context menu, or choose Values > Apply Value Conversion Filter >
Clear Value from the main menu.
TIP ►► You can also press the Del key to clear a single value when the cell selection is on the Converted Value cell.
3. MDM clears each value.
Applying Operators
You can use the Apply Operator filter to apply a numeric or text operator to one or more Converted Values, as described in this section.
To apply a numeric or text operator to one or more values:
1. In the Source Values grid, select the values to which you want to apply the operator.
2. Right-click on one of the values and choose Apply Operator from the
context menu, or choose Values > Apply Value Conversion Filter > Apply Operator from the main menu, to open the Apply Operator dialog shown in Figure 200.
Figure 200. Apply Operator dialog
3. Choose from the drop-down control of operators (only some of which will appear based on whether the destination is numeric or text):
NOTE ►► If the mapped destination is a coupled numeric attribute,
first choose Primary Value or Coupled Value from the cascading menu.
4. Enter into the Operand2 edit control the value that is to be applied by or to the operator.
5. Click OK to close the dialog. MDM applies the operator to each value and replaces it with the new value.
MDM Import Manager Reference Guide 317
Adjusting Measurement Units
Whenever possible, unit of measure strings are automatically normalized against MDM’s built-in dictionary of over 2000 synonyms for over 750 units in over 70 physical dimensions and appended to each numeric source data value, based on the physical dimension of the mapped destination measurement field or numeric attribute.
However, when the unit is missing entirely or simply unrecognized by MDM, you can use the three measurement filters to set, add, or convert the unit for one or more Converted Values, as summarized in Table 60 and described in this section.
Table 60. Measurement Value Conversion Filters
Filter Description
Set Measurement Unit Sets the unit for the source values, overriding any existing unit.
Affects all the selected values
Add Missing Unit Adds the unit to the source values that are missing a unit.
Only affects selected values without a unit
Convert to Unit Converts the source values to the unit.
Only affects selected values with a unit
TIP ►► The Convert to Unit measurement filter is most useful after
you have already applied either Set Measurement Unit or Add Missing
Unit, so that all source values have a unit and can be converted.
To apply a measurement filter to one or more values:
1. In the Source Values grid, select the values to which you want to apply the measurement filter.
2. Right-click on one of the values and choose the appropriate filter from
the context menu, or choose Values > Apply Value Conversion Filter from the main menu, and choose the appropriate filter from the cascading menu:
Set Measurement Unit Add Missing Unit Convert to Unit
3. Choose the unit from the cascading menu of applicable units based on the dimension of the mapped destination.
NOTE ►► If the mapped destination is a coupled numeric attribute,
first choose Primary Value or Coupled Value from the cascading menu, and then choose the corresponding unit from the second-level cascading menu of applicable units.
4. MDM sets, adds, or converts the unit for each value.
318 MDM Import Manager Reference Guide
Normalizing Values
Normalizing a value returns the capitalized alphanumeric string from which all punctuation – including the underscore (_) – and other “unwanted” characters have been removed. For example, the normalized value of “abc_123-456;N/R” is “ABC123456NR”.
You can use the Normalize filter to normalize one or more Converted Values, as described in this section.
To normalize one or more values:
1. In the Source Values grid, select the values that you want to normalize.
2. Right-click on one of the values and choose Normalize from the context
menu, or choose Values > Apply Value Conversion Filter > Normalize from the main menu.
3. MDM normalizes each value.
Changing the Case of Values
You can use the Change Case filter to change the case of one or more Converted Values, such as from all UPPER CASE to Mixed Case, as described in this section.
To change the case of one or more values:
1. In the Source Values grid, select the values whose case you want to change.
2. Right-click on one of the values and choose Change Case from the
context menu, or choose Values > Apply Value Conversion Filter > Change Case from the main menu, and choose the new case from the cascading menu:
UPPER CASE lower case Sentence case Title Case tOGGLE cASE
3. MDM changes the case of each value.
Replacing Values
You can use the Replace filter to “find and replace” each occurrence of a specified string in one or more Converted Values, as described in this section.
NOTE ►► If the Converted Value consists of multiple delimited values, the value conversion filter is applied to each of the individual values within the Converted Value.
MDM Import Manager Reference Guide 319
To find and replace a text string in one or more values:
1. In the Source Values grid, select the values in which you want to find and replace.
2. Right-click on one of the values and choose Replace from the context
menu, or choose Values > Apply Value Conversion Filter > Replace from the main menu, to open the Find and Replace dialog shown in Figure 201.
Figure 201. Find and Replace dialog
3. In the Find What text box, type the text you want to match.
4. In the Replace With text box, type the text you want to replace the found text.
5. Select or clear the option checkboxes:
Match Case Whole Words Only
6. Click Find Next. MDM displays and highlights the first value (or
delimited value) that matches your text in the Value preview box.
7. Find and replace each occurrence of the value as follows:
Find Next – find the next occurrence without replacing
Replace – replace and move to the next occurrence
Replace All – replace all occurrences without confirmation Cancel – terminate the operation and close the dialog
NOTE ►► If you are applying the Replace filter at the field-level, the
filter is not interactive and there are only two buttons: OK and Cancel.
8. MDM displays an informational dialog when all occurrences of the text you want to match have been found.
NOTE ►► If MDM does not find all the occurrences of the text you type, make sure that you have properly selected all of the applicable rows in the Source Values grid.
NOTE ►► The Regular expression option is always grayed out and will be removed from the Find and Replace dialog in an upcoming release of MDM Import Manager.
320 MDM Import Manager Reference Guide
Accepting Truncated Values
When the mapped destination field is a fixed-width text field, one or more source values may be too long to fit into the field. In this case, the MDM Import Manager automatically truncates each source value to fit into the field, based on its maximum width.
To alert you to this truncation (which may represent the loss of important information), the MDM Import Manager does the following:
Warning dialog. The MDM Import Manager displays a warning
dialog when you first map the field to indicate that one or more source values have been truncated.
Unconverted value. Source values that were truncated do not have
a green bullet ( ) in the Conv. column, and must be manually edited before they are deemed to have been properly converted.
In order to properly convert each unconverted source value so that you can proceed with the import, you must manually edit its Converted Value. You can accept the truncated value without editing it by entering
the Converted Value cell and simply pressing Enter, which converts the value and places the green bullet ( ) in the Conv. column.
DATA INTEGRITY ►► MDM forces you to enter the Converted Value cell for each source value that was truncated to guarantee that you are aware of the truncation and have had a chance to properly address it.
Alternatively, you can use the Accept Truncated Value filter to automatically accept each truncated value, as described in this section.
To accept the truncated value of one or more values:
1. In the Source Values grid, select the values whose truncated values you want to accept.
2. Right-click on one of the values and choose Change Case from the
context menu, or choose Values > Apply Value Conversion Filter >
Change Case from the main menu.
3. MDM accepts and places the green bullet ( ) in the Conv. column of each truncated value.
MDM Import Manager Reference Guide 321
Expanding Year Ranges
Often, multiple years may be represented in source data as a year range, such as when the source value “1995-91” represents the years 1991, 1992, 1993, 1994, and 1995.
You can use the Expand Year Range filter to expand the year ranges in one or more Converted Values into a set of individual years, for more convenient value mapping to a year field, as described in this section.
NOTE ►► The Expand Year Range filter is enabled only if the source field is mapped to a multi-valued field that requires value mapping.
TIP ►► If you have two source fields (e.g. Start Year and End Year), first partition Start Year by End Year and then combine partitions with the Combine Delimiter set to “-“.
To expand the year ranges represented by one or more values:
1. In the Source Values grid, select the values whose year range you want to expand.
2. Right-click on one of the values and choose Expand Year Range from
the context menu, or choose Values > Apply Value Conversion Filter >
Expand Year Range from the main menu.
3. MDM expands each year range into multiple delimited year values by populating the multi-valued drop-down control with each individual value, as shown in Figure 202.
Figure 202. Expanding a year range into multiple delimited year values
322 MDM Import Manager Reference Guide
Expanding Month Qualifications
Often, year ranges in the source data may be qualified with a numeric value representing the value of a month, such as when the source value “1999,6-2001,8” represents June 1999 through August 2001.
You can use the Expand Month filter to expand the month qualifications in one or more Converted Values into a delimited set of qualified values, for more convenient value mapping to automotive qualifiers, as described in this section.
To expand the month qualifiers represented by one or more values:
1. In the Source Values grid, select the values whose month qualifiers you want to expand.
2. Right-click on one of the values and choose Expand Month from the
context menu, or choose Values > Apply Value Conversion Filter >
Expand Month from the main menu.
3. MDM populates the multi-valued drop-down control with the month qualifier corresponding to each year in the expanded year range, as shown in Figure 203.
Figure 203. Expanding month qualifiers into multiple delimited values
FINDING UNMAPPED SOURCE VALUES
Quickly jumping to each unmapped source value can be useful when the Source Values grid contains many values, and especially when it contains a hierarchy, which cannot be sorted by the Mapped column.
You can use the Find Next Unmapped Source Value command to find each unmapped source value, as described in this section.
To find unmapped source values:
1. In the Source Values grid, select any source value.
2. Right-click on the value and choose Find Next Unmapped Source Value
from the context menu, or choose Values > Find Next Unmapped
Source Value from the main menu.
MDM Import Manager Reference Guide 323
3. MDM moves the selected value in the Source Values grid from the current selection to the next unmapped source value.
TIP ►► You can right-click on the source value and choose Find Next Unmapped Source Value repeatedly (or simply press F3) to find each unmapped value one after another.
NOTE ►► The Find Unmapped Source Value command also finds unconverted values.
NOTE ►► The Find Unmapped Source Value command is most useful when the Source Values grid contains a hierarchy; otherwise, you can simply sort by the Mapped column of the grid.
NOTE ►► When the mapped destination field is a hierarchy lookup field, the command ignores internal nodes and finds only leaf nodes; when the current table is a hierarchy table and the mapped destination field is the primary display field representing the hierarchy, the command finds both internal and leaf nodes.
TIP ►► You can also use the Select Entire Branch > Unmapped Leaf Nodes command to select rather than jump to each unmapped leaf node value (see “Selecting Source Values in a Hierarchy” on page 325 for more information).
REORDERING MAPPED DESTINATION VALUES
When you map a source value to the values of a multi-valued destination field, the Pos. column in the Destination Values grid identifies the ordered position of each mapped destination value.
By default, the order of the mapped destination values is as follows:
Automap. The mapped destination values are ordered in the same
order as the delimited source values.
Manual map. The mapped destination values are ordered in the
sequence in which the values were mapped.
After mapping, you can use the Reorder Mapped Destination Values command to change the order of the mapped destination values for each source value.
To reorder the mapped destination values for a source value:
1. In the Source Values grid, select the value whose mapped destination values you want to reorder.
2. Right click on the value and choose Reorder Mapped Destination Values from the context menu, or choose Values > Reorder Mapped
324 MDM Import Manager Reference Guide
Destination Values from the main menu, to open the Reorder Mapped Destination Values dialog shown in Figure 204.
Figure 204. Reorder Mapped Destination Values dialog
3. Drag-and-drop the mapped destination values within the dialog to place them into the order you would like them to be mapped.
4. Click OK when you are done to close the Reorder Mapped Destination Values dialog and reorder the mapped destination values.
5. MDM changes the Pos. values to reflect the new ordering.
FINDING MAPPED DESTINATION VALUES
When a destination field is multi-valued, a source value can be mapped to multiple destination values.
Quickly jumping to each mapped destination value can be useful when the Destination Values grid contains many values, and especially when it contains a hierarchy, which cannot be sorted by the Mapped column of the grid.
You can use the Find Next Mapped Destination Value command to find each mapped destination value for a source value, as described in this section.
TIP ►► Recall that Automap will overmap a source value if it matches more than one destination value, even if the destination field is single-
valued. You can then use the Find Next Mapped Destination Value command to find each mapped destination value and unmap all but one of the multiple mapped destination values.
To find the mapped destination values for a source value:
1. In the Source Values grid, select the source value for which you want to find the mapped destination values.
MDM Import Manager Reference Guide 325
2. Right-click on the value and choose Find Next Mapped Destination Value from the context menu, or choose Values > Find Next Mapped
Destination Value from the main menu.
3. MDM moves the selected value in the Destination Values grid from the current selection to the next mapped destination value.
TIP ►► You can right-click on the source field and choose Find Next
Mapped Destination Value repeatedly (or simply press F3) to find each mapped destination value one after another.
SELECTING SOURCE VALUES IN A HIERARCHY
A Source Values grid that contains a hierarchy rather than a flat list poses two additional challenges when you attempt to identify and select unconverted or unmapped leaf node values for further processing (such as editing the Converted Value or applying a value conversion filter):
Two kinds of nodes. A hierarchy contains two kinds of nodes: (1)
internal nodes (which can often be ignored); and (2) leaf nodes (whose values must be converted or mapped to destination values).
No sorting. The values in the Source Values grid cannot be sorted
by the Conv. or Mapped columns, making it impossible to group all of the unconverted or unmapped values.
You can use the Select Entire Branch commands to conveniently select all or some of the children of an internal node in the Source Values grid, as summarized in Table 61 and described in this section.
Table 61. Select Entire Branch Commands
Command Description
All Nodes Selects the selected internal node and all of its internal and leaf node children.
All Leaf Nodes Selects all of the leaf node children of the selected internal node.
Unmapped Leaf Nodes Selects all of the unmapped leaf node children of the selected internal node.
To select source hierarchy values:
1. In the Source Values grid, select the internal node whose children you want to select.
2. Right-click on the node and choose Select Entire Branch from the
context menu, or choose Values > Select Entire Branch from the main menu, and choose the appropriate selection from the cascading menu:
All Nodes All Leaf Nodes Unmapped Leaf Nodes
326 MDM Import Manager Reference Guide
3. MDM selects the applicable children of the selected node.
TIP ►► You can also use the Find Next Unmapped Source Value command to jump to rather than select each unmapped leaf node value (see “Finding Unmapped Source Values” on page 322 for more information).
TIP ►► To quickly select unmapped values when the Source Values grid is a list rather than a hierarchy, sort the grid by the Mapped column to group all of the unmapped values at the bottom of the list,
and then use Shift selection to conveniently select all of them.
HANDLING MISSING SOURCE DATA
As described in the previous sections, records of source data may be missing data values that are required to proceed with the import. These data values may be missing because they are implied.
When the data is implied, you can automatically create the implied values using various features of MDM Import Manager, so that you can populate the applicable destination fields with the correct data values as part of the import.
The different types of implied data – and the MDM Import Manger features designed to address them – are summarized in Table 62.
Table 62. Handling Implied Source Data
Implied Data Challenge MDM Feature
An entire field is completely missing from the source data because all of the records have the same value. For example, there may be no Manufacturer field in the product data file from the manufacturer itself.
Use the Add Field command to add a new field, map the new field to the applicable destination field, and map the single NULL value to the applicable value of the mapped destination field.
Some or all of the text values of a source field are missing and all the missing values correspond to the same value.
Map the NULL value to the applicable destination value.
Some or all of the numeric values of a source field are missing and all the missing values correspond to the same value.
Convert the NULL value to the applicable numeric value.
All of the numeric values of a source field
mapped to a measurement are missing a unit, which is implied by the field name.
Use the Set Measurement Unit value conversion filter to add the unit to every source data value.
Some of the numeric values of a source field mapped to a measurement are missing a unit, and all the missing units correspond to the same unit.
Use the Add Missing Unit value conversion filter to add the unit to every source data value.
All of the coupled values of a source field mapped to a coupled numeric attribute are missing and all the missing values correspond to the same value.
Use the Apply Operator value conversion
filter with the Equals operator to set the coupled value for every source data value.
MDM Import Manager Reference Guide 327
Some of the coupled values of a source field mapped to a coupled numeric attribute are missing and all the missing values correspond to the same value.
Use the Apply Operator value conversion
filter with the NULL Equals operator to set the coupled value for every source data value that is missing.
Mismatches Between Number of Header and Record Fields
In text files which use headers, the number of header fields may not always match the number of "value" fields in a given record. If an import record has fewer fields than its header, MDM automatically populates the missing fields with blank values during the import. If the record has more fields than the header, MDM issues an error message.
329
PART 11: MATCHING RECORDS
This part of the reference guide explains how to identify one or more matching fields or matching field combinations to match incoming source records to existing destination records, and how you can apply different import actions (create, update, and replace) to groups of source records and on a record-by-record basis.
MDM Import Manager Reference Guide 331
Overview
Once you have completed the field mapping and value conversion and mapping steps, record matching shifts the focus from the field-oriented processing to the more traditional record-oriented processing.
Record matching – often called key field matching – is usually the final step in the import process, in which most import tools allow you to identify the single key field to be used to match source records to existing destination records, so that you can update or replace them rather than create duplicates in the repository.
NOTE ►► This step is called record matching rather than key field matching because you can specify non-key fields as matching fields.
NOTE ►► Record matching shifts the focus from the individual fields (vertical view of the source data) to the individual records (horizontal view of the source data).
Record matching in the MDM Import Manager takes basic key field matching and extends it as follows:
Multiple matching fields. You can specify multiple matching fields,
for a more precise match if no single field is completely reliable as a key field.
Matching field combinations. You can also specify matching field combinations if no individual field identifies each destination record,
such as when MFG and PART NO together comprise the key field.
Matches converted or mapped source values. Rather than using
the original source value, matching is based upon the normalized, converted, or mapped source value, providing for a more accurate and precise match against destination record values.
Case-by-case operations. The matching process breaks the source
records into different groups based on the type of match – exact, partial, or conflict – and then allows you to apply a different default import action – skip, create, update, or replace – to each group.
Record-by-record operations. You can then override the default
import action for each individual source record and each matching destination record on a record-by-record basis, for even more precise import control.
Analytic capabilities. Finally, you can selectively identify for special
processing groups of records that match particular conditions, such as “the SKU field matches but the Description field has been changed” or “the List Price has changed by more than 20%.”
332 MDM Import Manager Reference Guide
THE MATCH RECORDS TAB
The Match Records tab contains several controls, as shown in Figure 205. These controls together allow you to: (1) meticulously match incoming source records to existing destination records; and then (2) efficiently and precisely specify the import action for each group of source records and then for individual records, as follows:
Value matching. You can use the dual-list Value Matching control to
specify a single matching field, multiple matching fields, or matching field combinations.
Default import actions. You can use the Default Import Actions grid
to specify the default import action for each group of source records, with a different import action for each group.
Record matching. You can use the pair of Record Matching grids to
override the default import action for each individual source record, and for each source record, each of its matching destination records.
Figure 205. Match Records tab
The controls of the Match Records tab are described in Table 63.
Table 63. Controls of the Match Records Tab
Control Description
Value Matching Lists Lists the mapped destination fields, and allows you to select one or more as matching fields or matching field combinations.
Import Actions Grid Lists the groups of source records, and allows you to specify the default import action for each group.
Record Matching Grids
Lists the source records, and for each source record, its matching destination records, and allows you to override the default import action for each individual record.
MDM Import Manager Reference Guide 333
NOTE ►► Since you cannot import into the original protected version of a checked out record, the MDM Import Manager displays – and performs record matching against – the checked out version of each checked out record, regardless of whether the import user is already a member of the checkout.
The Value Matching Lists
Value matching is used to identify the one or more fields or field combinations whose values should be used to match each source record to one or more existing destination records.
Specifically, the MDM Import Manager uses the Mapped Destination Fields list on the left to display the destination fields to which you have mapped a source field, as shown in Figure 206.
Figure 206. Value Matching lists
You can then identify one or more mapped destination fields as matching fields or matching field combinations by moving them to the Matching Fields list on the right.
NOTE ►► The Mapped Destination Fields and Matching Destination Fields lists are both disabled if you have not yet selected the current source and destination tables.
The Value Matching lists are described in Table 64.
Table 64. The Value Matching Lists
List Description
Mapped Destination Fields
Lists the destination fields to which source fields have been mapped. 1
Matching Fields Lists the mapped destination fields you have selected as matching fields, and allows you to combine them into matching field combinations.
1 For XML sources, limited to destination fields mapped to source fields in the current table.
NOTE ►► Not all mapped destination types can be used for record matching. Those that appear in the Mapped Destination Fields list and that you can select as Matching Fields are listed in Table 65.
334 MDM Import Manager Reference Guide
Table 65. Destination Types Available in Mapped Destination Fields List
Available as Matching Fields1, 2, 3, 4, 5, 6, 7, 8, 9 Not Available as Matching Fields
AutoID
Integer
Real
Currency
Date
Time
GM Time
Timestamp
Text
Text Normalized
Lookup [Flat]
Lookup [Hierarchy]
Lookup [Main]
Lookup [Taxonomy]
Text Large
Lookup [Text Block]
Lookup [Text HTML]
Lookup [Qualified]
Qualifiers
Attributes
Boolean
1 Cannot be multi-valued 2 Cannot be a promoted display field 3 If hierarchy table, must be the main display field 4 Cannot be a compound field (i.e. lookup to a table with multiple display fields) 5 Can be mapped only from Add, Clone or Split into Multiple Fields extended source fields 6 If mapped from extended source field, must connect to source with read/write permission 7 Must be mapped from a source field which is in the current language layer 8 If not the main table or a hierarchy table, all of the display fields must be selected 9 Record matching on a sortable field for which there is no sort index is extremely slow!
MULTILINGUAL ►► In a multilingual repository, all of the language layer instances of a multilingual Text field appear in the Mapped Destinations Fields list and are available as Matching Fields (where each field appears as “fieldname <language layer>”). More information about languages and multilingual repositories is provided in “Part 14: Multilingual Support.”
The Default Import Actions Grid
Value matching breaks the entire set of source records into several different groups based on the following:
Number of matching destination records. Each source record can
match: (1) no existing destination records; (2) a single destination record; or (3) multiple destination records (see “Matc” on page 340 for more information).
Type of match. There are several different types of possible value
matches between each source record and the corresponding set of matching destination records (see “Match Type” on page 340 for more information).
MDM Import Manager Reference Guide 335
The MDM Import Manager indicate the number of records in each group (and how many of the records in each group are active) within the Default Import Actions grid, as shown in Figure 207.
Figure 207. Default Import Actions grid
You can then specify the Default Import Action for each group of source records, and have the action automatically applied to each active record in the group.
NOTE ►► The Default Import Actions grid is disabled if you have not yet selected the current source and destination tables and at least one matching field.
NOTE ►► See “Active Records and Inheritance” on page 349 for more information about active records.
The columns of the Default Import Actions grid are described in Table 66. All but the Default Import Action column are read-only.
Table 66. Columns of the Default Import Actions Grid
Columns Description
Active The active records in each group of source records as “m of n” (where ‘m’ is the number of active records in the group and ‘n’ is the total number of records in the group) (see page 349).
Match Level
The match level (see page 340):
None Single
Multiple
Match Type
The match type (see page 340):
None
Exact Partial
Conflict
Default Import Action
The default import action (see page 346):
Skip
Create Update (NULL Fields Only) Update (All Mapped Fields)
Replace
336 MDM Import Manager Reference Guide
The Record Matching Grids
The MDM Import Manager uses the two Record Matching grids to display the set of source records, and for each source record, the set of zero or more matching destination records, as shown in Figure 208.
Figure 208. Record Matching grids
For each source record, you can use: (1) the Source Records grid to override the Import Action inherited from the Default Import Actions grid; and (2) the Matching Destination Records grid to override the Import Action inherited from the current source record.
NOTE ►► The Record Matching grids are disabled if you have not yet selected the current source and destination tables and at least one matching field.
The columns of the Record Matching grids include the fields of each record and additional fields of color-coded record matching information, as described in Table 66. All but the Import Action column in each grid are read-only.
Table 67. Columns of the Record Matching Grids
Column Description
Source Records Grid
Record No. The record number of the source record in the original data source.
Active Whether or not the source record is active.
Import Action
The import action for this record (gray if inheriting Default Import Action):
Skip Create Update (NULL Fields Only) Update (All Mapped Fields) Replace
Match Level
The number of matching destination records:
None Single Multiple
MDM Import Manager Reference Guide 337
Column Description
Match Type
The overall type of match:
Exact Partial Conflict
Exact The number of Exact match matching destination records.
Partial1 The number Partial match matching destination records.
Conflict1 The number Conflict match matching destination records.
matching field [Eql]2 The number of Equal value match destination records for matching field.
matching field [Und]1, 2 The number of Undefined value match destination records for matching field.
matching field [Neq]1, 2 The number of Not Equal value match destination records for matching field.
source field(s) The value of source field (displayed with the gray background if the field has not been mapped).
Matching Destination Records Grid
Active Whether or not the destination record is active.
Import Action
The import action for this record (gray if inheriting source Import Action):
None Update (NULL Fields Only) Update (All Mapped Fields) Replace Delete
Match Type
The match type for the destination record with the current source record:
Exact Partial Conflict
Exact1, 2 Whether the match type for the destination record with the current source record is Exact.
Partial1, 2 Whether the match type for the destination record with the current source record is Partial.
Conflict1, 2 Whether the match type for the destination record with the current source record is Conflict.
matching field [All]1, 2
The type of value match for the destination record with the current source record:
Equal Undefined Not Equal
matching field [Eql]2 Whether the value match for the destination record with the current source record is Equal for matching field.
matching field [Und]1, 2 Whether the value match for the destination record with the current source record is Undefined for matching field.
matching field [Neq]1, 2 Whether the value match for the destination record with the current source record is Not Equal for matching field.
destination field(s) The value of destination field.
1 Only present if you have specified multiple matching fields. 2 Hidden by default; unhide to display.
338 MDM Import Manager Reference Guide
RECORD MATCHING CONCEPTS
Record matching layers upon the following concepts in comparing source and destination records to determine if they match:
Value matches
Record matches
Match class
Match type
Each of these concepts is described in the following sections.
NOTE ►► Some of the following record matching concepts are mathematically and logically complex, and are included here only as a reference for interested users. You do not need to fully understand the concepts to make effective use of the record matching functions.
Value Matches
For each matching field or field combination, there are three possible types of value matches, as follows:
Equal. Every pair of corresponding source and destination values is
equal (where some but not all of the matching pairs may be NULL).
Undefined. At least one pair of corresponding source and
destination values consists of a NULL and non-NULL value, and every other pair is equal.
Not Equal. At least one pair of corresponding source and destination
values is not equal.
NOTE ►► For an individual matching field, a single pair of source and destination field values is compared for each source and destination record. For a matching field combination, a pair of values is compared for each individual matching field in the combination.
The three types of value matches are summarized in Table 68.
MDM Import Manager Reference Guide 339
Table 68. Value Matches for a Matching Field or Field Combination
Value Match Individual Matching Field Matching Field Combination
Equal
The single pair of corresponding source and destination values is equal and non-NULL.
Examples:
X and X
Every pair of corresponding source and destination values is equal, and at least one pair is non-NULL.
Examples:
X and X; A and A
X and X; NULL and NULL
Undefined
The single pair of corresponding source and destination values consists of a NULL and non-NULL value or two NULL values.
Examples:
X and NULL
NULL and NULL
At least one pair of corresponding source and destination values consists of a NULL and non-NULL value, and every other pair is equal.
Examples:
X and X; A and NULL
X and NULL; A and NULL
X and NULL; NULL and NULL
Not Equal
The single pair of corresponding source and destination values is not equal.
Examples:
X and Y
At least one pair of corresponding source and destination values is not equal.
Examples:
X and X; A and B
X and Y; A and B
X and Y; A and A
X and Y; A and NULL
X and Y; NULL and NULL
NOTE ►► For the purposes of the value match, NULL and NULL is: (1) treated as Undefined for an individual matching field; and (2) ignored for a matching field combination.
NOTE ►► Tuple matching handles NULL values differently than
described in this section. If the NULL Interpretation option for the
tuple’s matching field is set to Overwrite, source NULL values in the matching field will match any destination NULL value. If the NULL Interpretation option is set to Ignore, source NULL values will not match any destination value.
Record Matches
The destination records that match a particular source record depend upon the value matches for the matching fields or field combinations. An existing destination record matches a source record as follows:
340 MDM Import Manager Reference Guide
Single matching field or field combination. The value match for the single matching field or field combination is Equal
Multiple matching fields or field combinations. The value match for at least one of the matching fields or field combinations is Equal.
NOTE ►► If there is not at least one Equal value match among the one or more matching fields or field combinations (and the underlying pairs of source and destination values), then the source record simply does not have a matching destination record in the repository.
Match Level
The match level for each source record indicates the number of matching destination records. The three match levels include:
None. The source record matches no destination records.
Single. The source record matches exactly one destination record.
Multiple. The source record matches more than one destination.
NOTE ►► When a source record matches multiple destination records, each matching field or field combination may have multiple value matches, each corresponding to a different matching record.
NOTE ►► When you use the Filter command, there is a fourth match level named Filtered that corresponds to both source and destination records (see “Record Filtering” on page 356 for more information).
Match Type
For each source record, the overall match type is based on: (1) the
combination of match types for each individual matching destination record; where (2) the match type for each matching destination record is based on the combination of value matches for each individual matching field or field combination.
For each matching destination record, the match types include:
Exact. All the individual value matches are Equal.
Partial. At least one value match is Equal and at least one
Undefined; no value matches are Not Equal.
Conflict. At least one value match is Equal and at least one value
match is Not Equal.
For a source record that matches a single destination record, the overall match type for the source record is the same as the match type for the single matching destination record.
MDM Import Manager Reference Guide 341
For a source record that matches multiple destination records, the overall match type of the source record is based on the “best” match type among each of the individual matching destination records, as follows:
Exact. At least one Exact match among the matching destination
records.
Partial. No Exact matches but at least one Partial match among the
matching destination records.
Conflict. No Exact or Partial matches; all matching destination
records are Conflict matches.
The match types are summarized in Table 69.
Table 69. Overall Match Type for a Source Record
Match Type Single Matching Destination Multiple Matching Destinations
Exact
All the individual value matches are Equal.
Example:
Equal and Equal
At least one Exact match among the matching destination records.
Examples:
Exact and Exact
Exact and Partial
Exact and Conflict
Partial
At least one value match is Equal and at least one Undefined; no value matches are Not Equal.
Example:
Equal and Undefined
No Exact matches but at least one Partial match among the matching destination records.
Examples:
Partial and Partial
Partial and Conflict
Conflict
At least one value match is Equal and at least one value match is Not Equal.
Example:
Equal and Not Equal
No Exact or Partial matches; all matching destination records are Conflict matches.
Examples:
Conflict and Conflict
NOTE ►► The only match type possible with a single matching field or field combination is Exact, since either a Partial or Conflict match requires an Equal match combined with at least one other value match (and without even a single Equal match, then the destination record simply doesn’t match the source record).
NOTE ►► When you use the Filter command, there are two match types named Source and Destination that correspond to the Filtered match level (see “Record Filtering” on page 356 for more information).
342 MDM Import Manager Reference Guide
Example: Single Matching Field
Figure 209 illustrates the various value match and match type conditions for a single matching field (SKU).
Source Destination Value Match Match Type
SKU SKU SKU [Record]
113-224
113-224 Equal Exact
NULL Undefined None
101-44 Not Equal None
NULL
113-224 Undefined None
NULL Undefined None*
101-44 Undefined None
Figure 209. Single matching field (SKU)
Example: Single Matching Field Combination
Figure 210 illustrates the various value match and match type conditions for a single matching field combination (Mfr. and PartNo.).
Source Destination Value Match Match Type
Mfr PartNo Mfr PartNo Mfr; PartNo [Record]
Baldor 22345
Baldor 22345 Equal Exact
Baldor NULL Undefined None
Baldor 4403 Not Equal None
NULL 22345 Undefined None
NULL NULL Undefined None
NULL 4403 Not Equal None
SKF 22345 Not Equal None
SKF NULL Not Equal None
SKF 4403 Not Equal None
Baldor NULL
Baldor 22345 Undefined None
Baldor NULL Equal Exact
Baldor 4403 Undefined None
NULL 22345 Undefined None
NULL NULL Undefined None
NULL 4403 Undefined None
SKF 22345 Not Equal None
SKF NULL Not Equal None
SKF 4403 Not Equal None
NULL 22345
Baldor 22345 Undefined None
Baldor NULL Undefined None
Baldor 4403 Not Equal None
NULL 22345 Equal Exact
NULL NULL Undefined None
NULL 4403 Not Equal None
SKF 22345 Undefined None
SKF NULL Undefined None
SKF 4403 Not Equal None
Figure 210. Single matching field combination (Mfr. & PartNo.)
MDM Import Manager Reference Guide 343
Example: Multiple Matching Fields
Figure 211 illustrates the various value match and match type conditions for two matching fields (SKU and UPC).
Source Destination Value Match Match Type
SKU UPC SKU UPC SKU UPC [Record]
113-224 22345
113-224 22345 Equal Equal Exact
113-224 NULL Equal Undefined Partial
113-224 4403 Equal Not Equal Conflict
NULL 22345 Undefined Equal Partial
NULL NULL Undefined Undefined None
NULL 4403 Undefined Not Equal None
101-44 22345 Not Equal Equal Conflict
101-44 NULL Not Equal Undefined None
101-44 4403 Not Equal Not Equal None
113-224 NULL
113-224 22345 Equal Undefined Partial
113-224 NULL Equal Undefined Exact
113-224 4403 Equal Undefined Partial
NULL 22345 Undefined Undefined None
NULL NULL Undefined Undefined None
NULL 4403 Undefined Undefined None
101-44 22345 Not Equal Undefined None
101-44 NULL Not Equal Undefined None
101-44 4403 Not Equal Undefined None
NULL 22345
113-224 22345 Undefined Equal Partial
113-224 NULL Undefined Undefined None
113-224 4403 Undefined Not Equal None
NULL 22345 Undefined Equal Partial
NULL NULL Undefined Undefined None
NULL 4403 Undefined Not Equal None
101-44 22345 Undefined Equal Partial
101-44 NULL Undefined Undefined None
101-44 4403 Undefined Not Equal None
Figure 211. Multiple matching fields (SKU and UPC)
344 MDM Import Manager Reference Guide
Example: Multiple Matching Fields with Field Combination
Figure 212 illustrates the various value match and match type conditions for a three matching fields, consisting of an individual matching field (SKU) and a matching field combination (Mfr. and PartNo.).
Source Destination Value Match Match Type
SKU Mfr PartNo SKU Mfr PartNo SKU Mfr; PartNo [Record]
113-224
Baldor 22345
113-224 Baldor 22345 Equal Equal Exact
113-224 Baldor NULL Equal Undefined Partial
113-224 Baldor 4403 Equal Not Equal Conflict
113-224 NULL 22345 Equal Undefined Partial
113-224 NULL NULL Equal Undefined Partial
113-224 NULL 4403 Equal Not Equal Conflict
113-224 SKF 22345 Equal Not Equal Conflict
113-224 SKF NULL Equal Not Equal Conflict
113-224 SKF 4403 Equal Not Equal Conflict
NULL Baldor 22345 Undefined Equal Partial
NULL Baldor NULL Undefined Undefined None
NULL Baldor 4403 Undefined Not Equal None
NULL NULL 22345 Undefined Undefined None
NULL NULL NULL Undefined Undefined None
NULL NULL 4403 Undefined Not Equal None
NULL SKF 22345 Undefined Not Equal None
NULL SKF NULL Undefined Not Equal None
NULL SKF 4403 Undefined Not Equal None
101-44 Baldor 22345 Not Equal Equal Conflict
101-44 Baldor NULL Not Equal Undefined None
101-44 Baldor 4403 Not Equal Not Equal None
101-44 NULL 22345 Not Equal Undefined None
101-44 NULL NULL Not Equal Undefined None
101-44 NULL 4403 Not Equal Not Equal None
101-44 SKF 22345 Not Equal Not Equal None
101-44 SKF NULL Not Equal Not Equal None
101-44 SKF 4403 Not Equal Not Equal None
Figure 212. Multiple matching fields (SKU and Mfr. & PartNo.)
MDM Import Manager Reference Guide 345
VALUE MAPPING AND RECORD MATCHING: A COMPARISON
Recall that value mapping against a destination lookup field validates source data values against the set of legal destination values, that the value of the lookup field corresponds to the value of the lookup table display field or display field combination, and that each lookup field value corresponds to a record of the underlying lookup table.
Value mapping therefore not only validates source data values against the set of lookup field values, but also in effect matches them against existing lookup table records, so that you can map to existing records rather than create duplicates in the lookup table.
In the same way, record matching uses the values of the matching fields you specify to match source records against existing destination table records, so that you can update or replace existing records rather than create duplicates in the current destination table.
Record matching and value mapping thus complement one another to match rather than create duplicates of existing destination records.
NOTE ►► When the current table is the main table, record matching and value mapping prevent duplicates of main table and lookup table records, respectively. When the current table is a lookup table, they prevent duplicates of lookup table records, with the match based on: (1) the mapped value if you are value mapping; and (2) on the record match if you are not value mapping.
Record Matching and Lookup Table Display Fields
When the current table is a lookup table, record matching is required against the display field as a matching field. When the lookup table has multiple display fields, the required record matching is to all of the display fields as a matching field combination. When the lookup table is a hierarchy table, the record matching is required only to its primary display field, regardless of the number of lookup table display fields.
NOTE ►► Value mapping against lookup fields is required to match lookup table records because record matching is reserved for matching main table records. By contrast, when the current table is the lookup table, record matching is available for matching lookup table records, and so value mapping to the display fields is either optional or not required, depending upon the field type (see “Value Mapping and Lookup Table Display Fields” on page 247 for more information).
NOTE ►► When the current table is a lookup table: (1) the Mapped Destination Fields list by default includes all of the display fields of the table; (2) each display field appears as “field [DF]” and appears in gray
until it has been mapped; (3) the Add as Individual command is disabled for display field items; and (4) all of the display fields must be added together as a matching field combination.
346 MDM Import Manager Reference Guide
Consider a main table with a Manufacturer lookup field into the Manufacturers lookup table, which has display fields Name and Code. When the current table is the Manufacturers table, record matching is required against Name and Code as a matching field combination. If the Manufacturers lookup table is a hierarchy table, then record matching would be required to the Name field alone rather than with Code.
NOTE ►► When the current table is a lookup table, the [Remote Key] field can also be used as a matching field, as follows: (1) if you map values for the display field(s) of a flat table or the primary display field
of a hierarchy table, the [Remote Key] field can be used as a matching field only if it is combined with the display field(s) or the primary display field; and (2) if you do not map values for the display field(s), it can be used as the matching field by itself (see “Lookup Table Imports Using
the [Remote Key] Field” on page 414 for more information).
IMPORT ACTIONS
The various import actions that you can apply to source and destination records are described in Table 70.
Table 70. Import Actions for Source and Destination Records
Import Actions Description
Source Records
Skip Skip this source record during import processing.
Create Create a new destination record based on this source record.
Update (NULL Fields Only) Update just the NULL fields of the matching existing destination record(s) based on this source record.
Update (All Mapped Fields) Update the matching existing destination record(s) with all of the mapped fields of this source record.
Replace Replace the matching existing destination record(s) by first deleting them and then creating a new destination record based on this source record.
Destination Records
None Do not perform any action on this destination record during import processing.
Update (NULL Fields Only) Update just the NULL fields of this destination record based on the source record.
Update (All Mapped Fields) Update this destination record with all of the mapped fields of the source record.
Replace Replace this destination record by first deleting it and then creating a new destination record based on the source record.
Delete Delete this destination record during import processing.
MDM Import Manager Reference Guide 347
When you choose to update or replace existing destination records, the appropriate import action depends upon how comfortable you are overwriting existing destination data based on what you believe about the quality and completeness of the incoming source data, as described in Table 71.
Table 71. Different Levels of Import Actions
If the source data is… Import Action
Accurate and complete Use the Replace import action, to replace matching destination records in their entirety.
Accurate and incomplete Use the Update (All Mapped Fields) import action, to preserve unmapped fields but overwrite existing data.
Less accurate than existing destination data
Use the Update (NULL Fields Only) import action, to fill in missing data but not overwrite existing data.
Import Action Inheritance
Import actions are inherited and can be set at several different levels, as follows:
Default Import Actions grid. You can set a different Default Import
Action for each combination of Match Level and Match Type. The Default Import Action is automatically applied to each corresponding source record in the group for which the import action has not been manually set at the source record level, and also to each matching destination record for which the import action has not been manually set at the destination record level.
Source Records grid. When you manually set the import action for
an individual source record, it is applied to the source record and to each matching destination record for which the import action has not been manually set at the destination record level.
Matching Destination Records grid. When you manually set the
import action for an individual destination record that matches a source record, it is applied to the destination record.
NOTE ►► When you set the Default Import Action for a group of source records, each action in the Import Action column in both the Source Records and Destination Records grids appears in gray to indicate that it is inheriting the default. When you manually set the import action for an individual source or destination record, the action then appears in black to indicate that you have overridden inheritance from the Default Import Actions grid or the Source Records grid,
respectively. You can use the Inherit Default command to restore inheritance for a record.
348 MDM Import Manager Reference Guide
The Default Import Action mechanism allows you to quickly set the import action for large groups of source records, and then to precisely override the default on a record-by-record basis.
Source and Destination Record Import Actions
For each combination of Match Level and Match Type, there is a different set of available import actions in the Default Import Actions grid, and a corresponding set of available import actions for each source record in the Source Records grid, and each matching destination record in the Destination Records grid, as summarized in Table 72.
Table 72. Import Actions by Match Level and Match Type
Match Level / Match Type
Source Actions (default in bold)
Corresponding Destination Actions (default in bold)
None
None
Skip n/a
Create n/a
Single
Exact Partial
Conflict
Skip None
Delete
Update (NULL Fields Only) Update (NULL Fields Only)
Update (All Mapped Fields) Update (All Mapped Fields)
Replace Replace
Multiple
Exact Partial Conflict
Skip None Delete
Update (NULL Fields Only)
None Update (NULL Fields Only) Update (All Mapped Fields)
Delete
Update (All Mapped Fields)
None
Update (NULL Fields Only) Update (All Mapped Fields) Delete
Replace None Replace
Filtered
Source
Skip n/a
Create n/a
Filtered
Destination
Skip Skip
Deleted Deleted
NOTE ►► When you use the Filter command, the Source Records grid includes all of the invalid source records, plus a dummy source record that corresponds to the leftover destination records, which appear in the Destination records grid when the dummy is selected.
NOTE ►► The Match Records tab populates the Source Records grid with all source records – including duplicates – regardless of the Merge Source Records setting, which only affects how records are imported.
MDM Import Manager Reference Guide 349
NOTE ►► When a source record is to be imported against multiple matching destination records, the MDM Import Manager applies the default destination record action (inherited either from the default import action or from the source import action) to all of the multiple matching records; if only some of them should be affected, you must override the inherited default on a destination record-by-record basis.
NOTE ►► When the same destination record is a match for more than one source record, each source record may specify a different action for the destination record. In this case, the destination record may need to be cloned so that each of the source record actions can be properly and individually applied to the destination record, as described in Table 73.
Table 73. Cloning Requirements When Importing Source Records
Cloning Requirements for a Destination Record
Destination Action for Second Source Record
None Update Replace Delete
Destination Action for First Source Record
None OK OK OK OK
Update OK Clone Clone Clone
Replace OK Clone Clone Clone
Delete OK Clone Clone Clone
Active Records and Inheritance
Recall from the previous sections that when you set a Default Import Action for a group of records, it is automatically applied to and inherited by each source record in the group, and for each source record, further applied to and inherited by each matching destination record.
Similarly, when you manually set the Import Action for an individual source record, it is applied to the source record, and further applied to and inherited by each matching destination record.
In fact, this inheritance occurs only for those records that are currently active when you set the import action.
By default, every source record in each Match Level / Match Type group of source records is active when you first select the one or more matching fields or matching field combinations.
You can then limit the set of active records within each group based on the individual value matches and other analytic criteria. This breaks the import records into smaller, more precise subgroups, and allows you to apply import actions to the active subgroup of records in each group.
Subgroups of records within each group can be made active based on each of the three value match types, as described in Table 74.
350 MDM Import Manager Reference Guide
Table 74. Setting Active Records
Value Match Tag Active Records
Equal [Eql] Records for which at least one value match is Equal.
Undefined [Und] Records for which at least one value match is Undefined.
Not Equal [Neq] Records for which all value matches are Not Equal.
NOTE ►► The more stringent requirement (“all” rather than “at least one”) is necessary in this last case to avoid making it too broad and causing all records to be active.
SOURCE RECORD MERGING
Sometimes, multiple source records may represent different instances of the same product record (even if some fields have different values). For example, recall that with automotive parts data, the source data typically represents application records rather than product records, and will contain multiple records per product, where each source record represents a single application of the product (see “Product Applications and Automotive Parts Data” on page 257 for more information).
Duplicate source records are more than academic. As part of the source aggregation process, the MDM Import Manager can detect and either: (1) preserve these multiple instances of the same record; or (2) merge them automatically into a single virtual source record prior to importing. It does this based on the matching field, which is used not only for identifying matching destination records, but also for distinguishing between and identifying duplicate source records.
Merging is one of the most powerful aggregation capabilities of the Import Manager, and the ability to either preserve or collapse duplicates provides even finer user control over the aggregation process.
The Key Field
If you have specified multiple matching fields or field combinations, each one might cause different sets of source records to be considered duplicates for the purposes of the import and source record merging.
When there are multiple matching fields or field combinations, you can set any single matching field or field combination to be the key field for
the purpose of identifying duplicate source records (see “Setting the Key Field” on page 367 for more information).
NOTE ►► By default, the first matching field or field combination is automatically designated as the key field.
MDM Import Manager Reference Guide 351
The Merge Source Records Option
Depending upon how you would like the Import Manager to import the source data, you can specify that duplicate source records based on the key field are either preserved or merged, as described in Table 75.
Table 75. The Merge Source Records Option
Setting Description
Off Duplicate source records are preserved as individual records for import as a record group.
On Duplicate source records are collapsed and merged for import as a single virtual source record.
You can use the Merge Source Records configuration option to specify whether duplicate source records should be preserved or collapsed (see “Configuration Options” on page 62 for more information).
Prior to import, the MDM Import Manager groups each distinct set of duplicate source records, and then processes each group as follows:
1. Records in the group whose import action is Skip are ignored and removed from the group.
2. The import action of the last remaining record in the group is used as the import action for the entire group (“last one wins”).
NOTE ►► The "last remaining record" may not necessarily be the last record to appear in the source file.
3. If either: (a) Merge Source Records is On; or (b) the import action of
the group is an Update action, then the duplicate source records in the group are merged into a single record as follows:
Single-valued fields. The values are collapsed, with the last non-NULL value in the group used as the value of the field in the merged record (“last one wins”).
Multi-valued fields. The values are combined, with the set of distinct values across the entire set of duplicate records in the group used as the value of the field in the merged record.
NOTE ►► Even when Merge Source Records is Off, records of each
group for which the import action is an Update action are always merged because multiple sequential updates would lead to the same last-one-wins behavior anyway; by merging first, a single update can instead be applied to the destination record after the duplicate source records have been merged into a single record.
4. If the records were not merged in step (3), identical source records in the group are collapsed, and each of the remaining individual records is preserved for import as a group rather than one-by-one.
352 MDM Import Manager Reference Guide
5. Finally, either the group of duplicate source records or the single merged record is imported, creating one or more new destination records, or updating or replacing one or more existing destination records.
Consider the set of four duplicate source records (based on the SKU key field) and corresponding import actions shown in Figure 213.
SKU Description Color [MV] Price Import Action
#1 203 Widget Blue $3.49 Update (NULL Fields Only)
#2 203 Widget Plus Green $3.51 Replace
#3 203 Blue $3.58 Update (All Mapped Fields)
#4 203 Widget Red $3.69 Skip
Figure 213. Duplicate source records prior to merge
When processed: (1) the last record is ignored because its import action
is Skip; (2) the import action of the group is therefore Update (All Mapped Fields); (3) the three remaining records are therefore merged, regardless of the Merge Source Records setting, as shown in Figure 214; (4) the value of the Description field is taken from the second record because its value in the third record is NULL; (5) the value of the Color multi-valued field is combined from all three records; and (6) the value of the Price field is taken from the third record.
SKU Description Color [MV] Price Import Action
203 Widget Plus Blue; Green $3.58 Update (All Mapped Fields)
Figure 214. Duplicate source records after merge
The Merge Qualified Links Option
Just as the Merge Source Records option determines how to handle duplicate source records, the Merge Qualified Links option determines how to handle duplicate qualified links based on the matching qualifiers (see “The Qualified Update” on page 252).
Specifically, the subrecords representing the qualified links are first combined as a multi-valued field as a result of the source record merging described in the previous section. Then you can specify that duplicate subrecords based on the matching qualifiers are either preserved or merged, as described in Table 76.
MDM Import Manager Reference Guide 353
Table 76. The Merge Qualified Links Option
Setting Description
Off Duplicate subrecords are preserved for import as a set of qualified links.
On Duplicate subrecords are collapsed and merged for import as a single qualified link.
You can use the Merge Qualified Links configuration option to specify whether the duplicate links represented by the subrecords should be preserved or collapsed (see “Configuration Options” on page 62 for more information).
NOTE ►► If a group of duplicate source records is not being merged
(i.e. Merge Source Records is Off and the import action for the group is
not an Update action), then merging qualified links merges duplicate qualified links (represented by a group of delimited subrecords) only within each individual source record, not among and across the entire group of duplicate source records.
After merging source records, the MDM Import Manager groups each distinct set of duplicate subrecords, and then processes each group as follows:
Groups representing the value of a new qualified link:
1. If the import action for new links is Skip, the group is ignored.
2. If Merge Qualified Links is On, then the duplicate subrecords are merged into a single qualified lookup value.
Groups matching the value of one or more existing qualified links:
3. If the import action for existing links is Skip, the group is ignored.
4. If either: (a) Merge Qualified Links is On; or (b) the import action for
existing links is an Update action, then the duplicate subrecords in the group are merged into a single subrecord as follows:
Single-valued qualifiers. The values are collapsed, with the last
non-NULL value in the group used as the value of the qualifier in the merged subrecord (“last one wins”).
Multi-valued qualifiers. The values are combined, with the set of distinct values across the entire set of duplicate subrecords in the group used as the value of the qualifier in the merged subrecord.
354 MDM Import Manager Reference Guide
NOTE ►► Even when Merge Qualified Links is Off, subrecords are
always merged if the import action for existing links is an Update action because multiple sequential updates would lead to the same last-one-wins behavior anyway; by merging first, a single update can instead be applied to the qualified link after the duplicate subrecords have been merged into a single subrecord.
All groups:
1. If the subrecords were not merged in step (2) or (4), identical subrecords in the group are collapsed, and each of the remaining individual subrecords is preserved for import as a group rather than one-by-one.
2. Either the group of duplicate subrecords or the single merged subrecord is imported, creating one or more new qualified links, or updating or replacing one or more existing qualified links.
NOTE ►► When importing application records, each source record is still treated as a product record, one that represents a single value for the qualified lookup field that stores the application. When merged, the group of source records for each product is collapsed into a single virtual record representing the multiple applications for the product.
To Merge or Not to Merge
The primary effects of merging are to: (1) assemble non-NULL values from multiple records or links; (2) create multi-valued field values; and (3) eliminate duplicate or bad data (although “last-one-wins” provides no guarantee that the correct value will be chosen for single-valued fields).
Merging source records is most useful for creating multi-valued fields, particularly when aggregating the qualified links represented by each of multiple source records into the multiple links of a qualified lookup field.
Even when not merging, the grouping that always occurs facilitates the
Create of duplicate source records, and also many-to-many Replace, as follows: (1) with record merging off, you can replace a group of existing destination records with a group of incoming source records (based on the key field); and (2) with record merging on and qualified link merging off, you can replace a group of existing qualified links with a group of incoming qualified links (based on the matching qualifiers).
NOTE ►► With or without merging, duplicate source records that are completely identical across all fields (not just the key field), and duplicate qualified links that are completely identical across all qualifiers (not just the matching qualifiers) are always collapsed.
MDM Import Manager Reference Guide 355
NOTE ►► Merging source records is a distinct concept from matching and updating existing destination records. When merging records, each group of duplicate source records is first merged, and then each single surviving merged record is matched against and its values used to create a new or update or replace an existing destination record.
MULTILINGUAL ►► In a multilingual repository, the source record merging logic collapses multiple records that contain different language layers of data for the same record. More information about languages and multilingual repositories is provided in “Part 14: Multilingual Support.”
Merging Qualified Links and the Qualified Update Option
Recall that the Qualified Update option allows you to specify matching
qualifiers, that the Update setting uses them for matching existing
qualified links, and that the Append setting implicitly uses all of the qualifiers as matching qualifiers. In each case, the setting of the Qualified Update option is used to determine how incoming source data is processed against existing qualified lookup values only when the
import action for the group of records is Update (All Mapped Fields).
However, when merging qualified links, MDM again relies on these matching qualifiers to identify duplicate subrecords for grouping and
merging – even when the import action for the record group is Create,
Replace, or Update (NULL Fields Only) – and the Qualified Update option and corresponding matching qualifiers must therefore be set accordingly for proper merging.
NOTE ►► Merging of records or subrecords occurs when either: (1)
the corresponding Merge option is On; or (2) the import action for the
group of existing links is an Update action.
Specifically, when the Qualified Update option is set to Inherit Default, MDM uses the setting of Matching Qualifiers configuration option to aggregate import records, as follows:
None. Use only the non-qualifier display fields.
All. Use the non-qualifier display fields plus all the qualifiers.
By contrast, when the Set Qualified Update command has been used to override the global default setting of the Qualified Update option, MDM uses the non-qualifier display fields plus the matching qualifiers you specify in the Set Matching Qualifiers dialog.
356 MDM Import Manager Reference Guide
RECORD FILTERING
Record filtering expands the concept of import by complementing the source-centric set of import records with a destination-centric set of records that includes not only matched records but also unmatched records in the filtered destination set.
Specifically, it allows you to specify basic search criteria to filter the source and destination records in conjunction with various operations within MDM Import Manager and MDIS, and when performing the import itself.
Filtering narrows down both the source and destination record sets, but with slightly different effects, as follows:
Source. Narrows down the set of source records so that those that do not meet the filtering criteria can be excluded from the import. The filter is exclusive, and source records that do not meet the filtering criteria are deemed invalid.
Destination. Narrows down the set of destination records, excluding
those that do not meet the filtering criteria from record matching. The filter is inclusive, and expands the set of records on which you can perform import operations to those that do meet the filtering criteria.
NOTE ►► Invalid source records that do not meet the filtering criteria are “filtered out.” By contrast, the leftover destination records that are unmatched but do meet the filtering criteria are “filtered in.”
Full Refresh vs. Net Change Synchronizations
Typically, the MDM Import Manager is used to perform a “net change” synchronization, in which a file of source records is imported to update and replace matching destination records and create new ones, leaving untouched those destination records that do not match source records.
Consider, by contrast, a “full refresh” requirement, in which a file of source records must be imported and should update and replace a corresponding set of destination records in its entirety, whether or not
every destination record matches a source record.
In particular, source records that meet the filtering criteria but do not exist should be created, those that meet the filtering criteria but do exist should be used to update or replace existing destination records, and those that do not meet the filtering criteria should be ignored.
Similarly, destination records that meet the filtering criteria and match source records should be updated or replaced, and those that meet the filtering criteria but do not match source records should be deleted. In effect, all existing destination records that meet the filtering criteria must be affected by the import, not just those matching source records.
MDM Import Manager Reference Guide 357
Consider the example import illustrated in Figure 215, which consists of 120 source records (only 110 of which meet the filtering criteria), and 100 destination records that meet the filtering criteria (only 80 of which match source records), for a total of 80+30+20=130 distinct source and destination records that meet the filtering criteria.
Source: Destination:
Matched (80) ◄ Update/Replace ► Matched (80)
Unmatched (30) ◄ Create Delete ► Leftover (20)
100 total
Invalid (10) ◄ Skip
120 total
Figure 215. Record filtering example
In the example, 80 source records update or replace 80 matching destination records, 30 new source records are created, 10 invalid source records are skipped, and 20 leftover destination records are deleted.
Filtered Match Level and Match Types
Recall that there are three match levels for each source record and three match types, for a total of seven match level / match type combinations:
None/None
Single/Exact
Single/Partial
Single/Conflict
Multiple/Exact
Multiple/Partial
Multiple/Conflict
358 MDM Import Manager Reference Guide
When you use record filtering, two additional match level / match type combinations are enabled in the Default Import Actions grid, comprised of an additional Filtered match level and two additional match types for source and destination records, as follows:
Filtered/Source. Invalid source records that do not meet the filtering criteria.
Filtered/Destination. Leftover destination records that do meet the filtering criteria but are not matched by source records.
You can use the Filter command to filter by one or more lookup values of any lookup field that is used for record matching, and both the source and destination records will be forced to match those lookup value(s).
NOTE ►► Be careful if you set the import action for invalid source
records to Create, because they may correspond to destination records that were not matched because MDM used the filtering criteria to limit the set of destination records that participated in record matching.
Record Filtering Behavior
The record filtering behavior of the MDM Import Manager is summarized in Table 77.
Table 77. Record Filtering Behavior of the MDM Import Manager
Level Description
Filter Command
The filter command brings up a dialog on the selected lookup field that allows you to select one or more lookup values by which to filter.
Record Matching Record matching is performed only against filtered destination records that match the search criteria rather than the entire repository.
Matching Fields Grid
You can add filtering criteria to any lookup field used as a
matching field; MDM adds [Filtered] to the matching field or field combination.
Default Import Actions Grid
Two match level / match type combinations correspond to
filtered records: Filtered/Source (invalid source records) and
Filtered/Destination (leftover destination records).
Filtered Import Actions
Import actions correspond to Filtered/Source records (Skip /
Create) and Filtered/Destination records (Skip / Update (NULL Fields Only) / Update (All Mapped Fields) / Delete).
Source Records Grid Includes all of the invalid source records, plus a dummy source record that corresponds to leftover filtered destination records.
Destination Records Grid Includes the leftover filtered destination records when the dummy source record is selected in the Source Records grid.
MDM Import Manager Reference Guide 359
THE VIRTUAL [MATCH LEVEL/TYPE] FIELD
Typically, the match level / match type value combination for each imported or updated record (including filtered destination records, if you are using record filtering) is transient, used for the duration of the import session for the purposes of assigning and performing import actions, and then discarded by MDM once the import is complete.
Sometimes, however, you may want to actually store each new and updated record’s match level / match type value combination in the repository itself, perhaps for user inspection, or so that a workflow can automatically branch or perform validations based on the value.
To support this requirement, the MDM Import Manager includes for
every source table a virtual field named [Match Level/Type], which appears in: (1) the list of source fields in the Source Hierarchy; and (2) the Source Fields grid of the Map Fields/Values tab.
This field always contains nine values, each corresponding to one of the match level / match type value combinations (see Table 72 on page 348 for more information), which appear in the Value Conversion and Mapping grid of the Map Fields/Values tab when it is selected in the Source Fields grid.
You can then map this field to a destination field in the MDM repository, and then map each of its values to a corresponding value in the destination field, and MDM will then update the destination value for each imported record as part of the import.
NOTE ►► The destination field can be of type Text or Lookup, and you must already have created its nine values prior to the import, since adding source field values to the list of destination field values is not permitted.
UPDATING THE RECORD MATCHING FIELD
The values of the record matching field are used to match incoming source records against existing destination records.
When creating new destination records or replacing existing destination records, the source record value is always used as the value of the record matching field, since in each case, the source record results in an entirely new destination record.
By contrast, when updating existing destination records as part of the
import, the value of the source record may or may not be used to update the value of the record matching field.
You can use the Do Not Update Record Matching Field Value configuration option to specify how the set of incoming source values are treated, as described in Table 78.
360 MDM Import Manager Reference Guide
Table 78. Do Not Update Record Matching Field Value Option
Setting Description
No Update the values of the record matching field with the source values.
Yes Do not update the values of the record matching field with the source values.
Whether or not the value of the record matching field is updated may seem academic, since for a matching destination record, the source and destination values will presumably be the same.
However, there are two circumstances where it does makes a difference whether or not MDM updates the existing value of record matching field:
Time Stamp field. If the field is updated, a Time Stamp field that
tracks the record matching field will see the field value as having been updated; otherwise, the Time Stamp value will not be changed.
Text Normalized field. If the field is updated, a record matching field
of type Text Normalized may be updated with source values that are formatted differently than the existing values.
NOTE ►► The next section discusses this and other considerations for a record matching field of type Text Normalized.
NORMALIZED RECORD MATCHING FIELD
When a record matching field is of type Text Normalized, several challenges relate to how to match and how to update existing values.
You can use the Normalized Record Matching Field configuration option to specify whether to perform normalized or exact matches between source values and the values of the record matching field, as summarized in Table 79.
Table 79. Normalized Record Matching Field Configuration Option
Setting Description
Normalized Match Against Normalized Value
Perform a normalized match of normalized source values against normalized destination values.
Exact Match Against Non-Normalized Value
Perform an exact match of literal source values against non-normalized destination values.
Normalized matches have the advantage of matching existing values even when the source value is formatted differently. For example, with a normalized match, source value 123-ABC will match 123/ABC because the match will compare normalized values 123ABC to 123ABC.
MDM Import Manager Reference Guide 361
Using a normalized match with a record matching field of type Text Normalized increases the likelihood that incoming source records can be properly matched against existing destination records. At the same time, several considerations arise when using a normalized match:
Too many matches. The normalized match may result in matches
against too many destination records, some of which should not match. For example, with a normalized match, 123-ABC will not only match 123/ABC but also 12-3/ABC, 123 [ABC], and 12.3-ABC.
TIP ►► For each source record that matches multiple destination records, you can use the Matching Destination Records grid of the Record Matching tab to identify which of the matching records is the correct match and which should be skipped during the import.
Incorrect source formatting. In first example above, if the correct
format is 123/ABC, you would want the source value 123-ABC to match the destination value 123/ABC but not to overwrite it, since the existing format was correct.
TIP ►► You can use to the Update Record Matching Field Value configuration option to determine whether or not the record matching field value should be updated with incoming source values, as described in the previous section.
MASK UPDATES AND RECORD REPLACEMENT
When the import replaces existing destination records with incoming source records, the masks that contain the replaced records must be updated at the same time.
For each Replace operation, how the masks are updated depends upon how many source records replace how many matching destination records, as summarized in Table 80.
NOTE ►► Recall that source records may be grouped rather than merged, and that each group may match one or more existing destination records.
362 MDM Import Manager Reference Guide
Table 80. Mask Updates and Record Replacement
Number of Replacement Source Records
Number of Matching Destination Records
Description
m 1 Every mask containing the 1 matching destination record will have it replaced by the m replacement source records.
1 1 Every mask containing the 1 matching destination record will have it replaced by the 1 replacement source record.
1 n Every mask containing any of the n matching destination records will have them replaced by the 1 replacement source record.
m n
Every mask containing all of the n matching destination records will have them replaced by the m replacement source records.
Masks containing some but not all of the n matching destination records will have them removed from the mask.
KEY MAPPINGS AND RECORD REPLACEMENT
Existing key mappings are always preserved, even for destination
records set to be replaced with the Replace action, as follows: (1) if a
source field is mapped to the [Remote Key] field, MDM will reject the source record rather than perform the replace; and (2) if a source field is
not mapped to the [Remote Key] field, MDM will perform the replace
after first transferring the [Remote Key] values of the target replaced
records to the new record. In each case, the Replace action will not
result in the loss of [Remote Key] values.
CHECKOUT AND RECORD REPLACEMENT
When an import is set to check out the records of an import, it is not possible to replace existing destination records with incoming source records, and MDM will therefore not allow you to perform the import if
any of the import actions are set to Replace.
Specifically, if the Checkout/Workflow configuration option is set to
Checkout or to a workflow that performs a checkout, you must either: (1) eliminate the Replace import action(s) by changing them to Update; or
(2) set the configuration option to None or to a non-checkout workflow.
NOTE ►► MDM uses the Import Status tab to inform you that the
import cannot proceed because of Replace with checkout.
MDM Import Manager Reference Guide 363
CHECKOUT AND RECORD MATCHING
Recall that an MDM user within the MDM Data Manager can check out a record, and that other users – subject to join permissions – can become members of the checkout. Non-members of the checkout see the original version of the record; members of the checkout see the
checked out version (unless they use the All Versions command).
Notwithstanding the above, regardless of whether or not an import user is already a member of the checkout, the MDM Import Manager always displays – and performs record matching against – the checked out version of each checked out record.
NOTE ►► See “Import, Checkout, and Workflow” on page 378 for more information about checkout- and workflow-related import failures.
RECORD MATCHING ON NON-SORTABLE FIELDS
The MDM Console allows you to specify that no sort index be built for a field that would otherwise be sortable. You can use such fields as record matching fields; however, the record matching process will be extremely slow, and the MDM Import Manager displays a warning dialog when you select such a field for record matching.
364 MDM Import Manager Reference Guide
Record Matching Operations
The record matching operations are described in the following sections, and summarized in Table 81.
Table 81. Record Matching Operations
Operation Description
Matching Field Lists
Add Matching Field
Adds the selected fields to the list of matching fields.
Remove Matching Field
Removes the selected fields from the list of matching fields.
Combine Matching Fields
Combines the selected matching fields into a matching field combination.
Split Matching Fields
Splits the selected matching field combination into individual matching fields.
Set Key Field Sets the selected matching field to be the key field used to identify duplicate source records.
Set Active Records Sets the value matches that should be active.
Filter Records Sets the filtering criteria for source and destination records.
Default Import Actions Grid
Set Default Import Action
Sets the default import action for a Match Level / Match Type group of source records.
Source Records Grid
Set Source Record Import Action
Sets the import action for the selected source records.
Matching Destination Records Grid
Set Destination Record Import Action
Sets the import action for the selected matching destination records.
NOTE ►► Whenever you Add, Remove, Combine, or Split matching fields as described in the following sections, the MDM Import Manager performs record matching on the new set of matching fields and field combinations, and then when record matching is complete, updates: (1) the Default Import Actions grid to reflect the number of source records falling into each Match Level / Match Type group; (2) the Source Records grid to reflect the record matching status of each source record; and (3) the Matching Destination Records grid to reflect the record matching status of each matching destination record.
MDM Import Manager Reference Guide 365
ADDING A SINGLE MATCHING FIELD
You can add a single matching field as described in this section.
To add a single matching field:
1. In the Mapped Destination Fields list, select the field that you want to
make a matching field, and click on the Add button.
2. MDM performs record matching based on the new set of matching fields and displays a progress dialog to indicate the status of the record matching, as shown in Figure 216.
Figure 216. Record matching progress dialog
ADDING MULTIPLE MATCHING FIELDS
When the number of source records and/or the number of destination records is very large, the record matching process can take some time.
If you intend to match on multiple fields – either as individual matching fields or as a matching field combination – it can be more efficient to add them all at once as described in this section.
To add multiple matching fields:
1. In the Mapped Destination Fields list, select the fields that you want to make matching fields.
2. Click on the Add button and choose from the cascading menu:
Add as Individual Add as Combination
3. MDM performs record matching based on the new set of matching fields and field combinations and displays a progress dialog like the one shown in Figure 216 to indicate the status of the record matching.
NOTE ►► When the current table is a lookup table: (1) the Mapped Destination Fields list by default includes all of the display fields of the table; (2) each display field appears as “field [DF]” and appears in gray
until it has been mapped; (3) the Add as Individual command is disabled for display field items; and (4) all of the display fields must be added together as a matching field combination.
366 MDM Import Manager Reference Guide
COMBINING MATCHING FIELDS
You can combine two or more matching fields that were previously added separately into a matching field combination as described in this section.
To combine two or more existing matching fields:
1. In the Mapped Destination Fields list, select the two or more fields that you want to combine into matching field combinations.
2. Click on the Combine button, or right-click on one of the fields and
choose Combine Matching Fields from the context menu.
3. MDM performs record matching based on the new set of matching fields and field combinations and displays a progress dialog like the one shown in Figure 216 to indicate the status of the record matching.
SPLITTING A MATCHING FIELD COMBINATION
You can split a matching field combination into individual matching fields as described in this section.
To split a matching field combination:
1. In the Mapped Destination Fields list, select the matching field combination that you want to split into individual matching fields.
2. Click on the Split button, or right-click on the field combination and
choose Split Matching Fields from the context menu.
3. MDM performs record matching based on the new set of matching fields and field combinations and displays a progress dialog like the one shown in Figure 216 to indicate the status of the record matching.
REMOVING MATCHING FIELDS
Removing matching fields is a matter of simply selecting the fields and removing them from the Matching Fields list as described in this section.
To remove one or more matching fields:
1. In the Matching Fields list, select the one or more matching fields you
want to remove, and click on the Remove button.
2. MDM performs record matching based on the remaining set of matching fields and field combinations and displays a progress dialog like the one shown in Figure 216 to indicate the status of the record matching.
MDM Import Manager Reference Guide 367
SETTING THE KEY FIELD
Recall that if you specify multiple matching fields or field combinations, the first one is, by default, automatically designated as the key field for the purposes of merging duplicate source records. You can designate a different matching field or field combination as the key field, as described in this section.
To change the key field:
1. In the Matching Fields list, select the matching field or field combination that you want to be the key field.
2. Right-click on the matching field and choose Key Field from the context
menu, or choose Records > Set Key Field from the main menu.
3. MDM appends [Key] to the name of the selected matching field or field combination to indicate that it is now the key field.
SETTING THE ACTIVE RECORDS
Recall that by default, every source record in each Match Level / Match Type group of source records is active when you first select the one or more matching fields or matching field combinations.
You can then limit the set of active records within each group based on the individual value matches and other analytic criteria. This breaks the import records into smaller, more precise subgroups, and allows you to apply import actions to the active subgroup of records in each group.
Subgroups of records within each group can be made active based on each of the three value match types as described in this section.
To change the value match criteria for which records are active:
1. In the Matching Fields list, select the matching field or field combination for which you want to change the value match criteria.
2. Right-click on the matching field and choose from the context menu, or
choose Records > Set Active Records from the main menu, and choose from the cascading menu:
All Matches Active [All] Equal Matches Active [Eql] Undefined Matches Active [Und] Not Equal Matches Active [Neq]
3. MDM appends the selected tag to the name of the selected matching field or field combination to indicate the value matches that are active.
NOTE ►► [All] both is replaced by and completely replaces [Eql],
[Und], and [Neq], which are themselves added to one another.
368 MDM Import Manager Reference Guide
SETTING THE RECORD FILTERING CRITERIA
Recall that if you select one or more lookup fields to be matching fields, you can filter the set of source and destination records that participate in an import by their lookup values.
Filtering by lookup values narrows or expands the set of source and destination records by:
Excluding all source records which do not have the filter value.
Including all destination records which do have the filter value.
Sometimes, however, you may not want to set the filter to a specific
lookup value. For example, if you set your map to filter records by the Manufacturer lookup value, BMW, you must then change this filter when you want to import Toyota records, and then save the change to a new map. You then have to repeat this process for each new Manufacturer value you want to import.
To avoid such inconveniences, Import Manager now includes a
[Mapped Values] filter option, which automatically sets the filter criteria to whatever lookup values are mapped in the source file.
NOTE ►► If you are importing via MDIS, the [Mapped Values] filter option can import whatever unmapped lookup values are in the source file as well (see “The MDIS Unmapped Value Handling Option” on page 268 for more information).
Filtering by [Mapped Values] expands the set of source and destination records by:
Including all source records with mapped lookup values.
Including source records with unmapped lookup values (MDIS only).
Including all destination records that have any mapped source value.
With the [Mapped Values] filter option, you can create flexible, reusable maps that automatically filter the lookup values in your source records, freeing you from having to change filters and/or create new maps for each new lookup value you want to filter by.
You can set the record filtering criteria for source and destination records as described in this section.
MDM Import Manager Reference Guide 369
To set the record filtering criteria for a matching field:
1. In the Matching Fields list, select the matching field for which you want to set the filtering criteria.
2. Right-click on the matching field and choose Filter from the context
menu, or choose Records > Set Filter from the main menu to open the Filter dialog shown in Figure 217:
Figure 217. Filter dialog
NOTE ►► If the matching lookup field has previously been combined
with another matching lookup field, Filter first brings up a cascading menu of lookup field names for you to select the one for which you want to set the filtering criteria.
NOTE ►► The [Mapped Values] filter option cannot be combined with any other values.
3. Choose one or more lookup values to use as record filtering criteria.
4. MDM appends [Filtered] to the name of the selected matching field or field combination to indicate that record filtering is in effect.
370 MDM Import Manager Reference Guide
SETTING A DEFAULT IMPORT ACTION
You can set the Default Import Action for a group of source records as described in this section.
To set the Default Import Action for a group of source records:
1. In the Default Import Actions grid, move the cell to the Default Import Action column of the Match Level / Match Type for which you want to set the default import action.
2. Click on the down-triangle to open the drop-down control and choose
the appropriate action from the list, or choose Records > Set Default Import Action from the main menu, choose the Match Level / Match Type from the cascading menu, and choose the appropriate action from the cascading menu:
For None/None (no matching destination records):
Skip Create
For others (matching destination records):
Skip Update (NULL Fields Only) Update (All Mapped Fields) Replace
3. To set the import action for the source records and override the default setting, choose any of the import actions, or alternatively, to restore
inheritance, choose Inherit Default.
NOTE ►► MDM places a check next to the cascading menu choice that corresponds to the current import action unless you have selected multiple records for which the import actions are not all the same.
MDM Import Manager Reference Guide 371
SETTING A SOURCE RECORD IMPORT ACTION
You can set the import action for one or more source records and override inheritance from the Default Import Actions grid as described in this section.
NOTE ►► Inherited import actions appear in the Source Records grid in gray.
To manually set the import action for one or more source records:
1. In the Source Records grid, select the records for which you want to set the import action.
2. Right-click on one of the records and choose the appropriate action
from the context menu, or choose Records > Set Source Import Action from the main menu, and choose the appropriate action from the cascading menu:
Skip Create Update (NULL Fields Only) Update (All Mapped Fields) Replace Inherit Default
3. To set the import action for the source records and override the default setting, choose any of the import actions, or alternatively, to restore
inheritance, choose Inherit Default.
NOTE ►► MDM places a check next to the cascading menu choice that corresponds to the current import action unless you have selected multiple records for which the import actions are not all the same.
372 MDM Import Manager Reference Guide
SETTING A DESTINATION RECORD IMPORT ACTION
You can set the import action for one or more destination records and override inheritance from both the Default Import Actions grid and the matching source record as described in this section.
NOTE ►► Inherited import actions appear in the Destination Records grid in gray.
NOTE ►► A destination record can appear more than once for two different source records.
To manually set the import action for one or more destination records:
1. In the Destination Records grid, select the records for which you want to set the import action.
2. Right-click on one of the records and choose the appropriate action
from the context menu, or choose Records > Set Destination Import Action from the main menu, and choose the appropriate action from the cascading menu:
None Create Update (NULL Fields Only) Update (All Mapped Fields) Replace Delete Inherit Default
3. To set the import action for the destination records and override the default setting, choose any of the import actions, or alternatively, to
restore inheritance, choose Inherit Default.
NOTE ►► MDM places a check next to the cascading menu choice that corresponds to the current import action unless you have selected multiple records for which the import actions are not all the same.
373
PART 12: IMPORTING DATA
This part of the reference guide explains how to actually perform the import, along with the final “pre-flight” checks to make sure that all the necessary steps have been performed prior to import.
MDM Import Manager Reference Guide 375
Import Execution
Every step in the import workflow must be complete before you can perform the import. Moreover, the MDM Import Manager will not permit you to perform the import until two specific conditions have been met:
Value mapping and conversion. For every mapped field, each of
its values must either be properly converted to the destination data type or mapped to an existing or newly added destination data value.
Record matching. You must specify one or more unique fields so
that the MDM Import Manager can determine whether to create new records or update and replace existing destination records.
MONITORING IMPORT STATUS
You can always determine what needs to be done next by checking the Action Items at the top of the Import Status tab, which identifies the next required action, as shown in Figure 218.
Figure 218. Import Status tab
376 MDM Import Manager Reference Guide
The Import Status tab also lists pre-import statistics that indicate what will happen when you perform the import, as summarized in Table 82.
Table 82. Pre-Import Statistics
Item Status Information
Table Mapping
Source table name
Destination table name
# of Mapped fields (m of n)
Table Joins From table/field name
To table/field name
Field Lookups From table/field name
To destination table field name
Field Mapping (without Value Mapping)
Source field name
Destination field name
# of values converted (m of n)
Field Mapping (with Value Mapping)
Source field name
Destination field name
# of values mapped (m of n)
# of values added
# of values overmapped
Matching Fields Source field name
Destination field name
Record Matching
None/None # of records matched
Single/Exact # of records matched
Single/Partial # of records matched
Single/Conflict # of records matched
Multiple/Exact # of records matched
Multiple/Partial # of records matched
Multiple/Conflict # of records matched
Filtered/Source # of invalid source records
Filtered/Destination # of leftover destination records
Import Actions
Skip # of source records
Create # of source records
Update (NULL Fields Only) # of source records
Update (All Mapped Fields) # of source records
Replace # of source records
Delete # of destination records
MDM Import Manager Reference Guide 377
PERFORMING THE IMPORT
When the Action Items on the Import Status tab reads Ready to Import, you can perform the import as described in this section.
To execute the import:
Click the Execute Import toolbar button (shown at left), or choose
Records > Execute Import from the main menu.
NOTE ►► The Execute Import toolbar button and Execute Import command will not be enabled unless the Import Status tab is the active tab, even if the Import Manager is ready to import.
NOTE ►► Import is “forgiving” around individual record failure. In other words, when you import a file of source records, the import will proceed even if some of the records fail to import. (See “Import, Checkout, and Workflow” on page 378 for more information about checkout- and workflow-related import failures.)
NOTE ►► Currently, the MDM Import Manager does not check any security constraints set in the MDM Console. Specifically, with the following exceptions: (1) the import will fail if any record constraints are set for the import user; and (2) you must have write access to the
Family Hierarchy if the Synchronize Family Hierarchy After Import
configuration option is set to Yes.
NOTE ►► If two users are importing to the same repository, the second import will wait for the first to finish, and then proceed; results may be somewhat unexpected since new data imported by the first will not be reflected in value mapping or record matching of the second.
REFRESHING THE RECORD MATCHING
To ensure that the current state of the record matching corresponds to the current state of the records in the MDM repository, you can refresh the record matching prior to performing the import, as described in this section.
To refresh the record matching prior to import:
Click the Refresh Record Matching toolbar button (shown at left), or
choose Records > Refresh Record Matching from the main menu.
NOTE ►► The Execute Import toolbar button and Execute Import command will not be enabled unless the Import Status tab is the active tab, even if the Import Manager is ready to import.
378 MDM Import Manager Reference Guide
Import, Checkout, and Workflow
MDM offers a variety of mechanisms to deal with the challenges around import, checkout and workflow, as described in the following sections.
THE AUTO-JOIN CHECKED OUT RECORDS OPTION
Recall that an MDM user within the MDM Data Manager can check out a record, and that other users – subject to join permissions – can become members of the checkout. Non-members of the checkout see the original version of the record; members of the checkout see the
checked out version (unless they use the All Versions command).
A challenge arises when the MDM Import Manager attempts to import source records into destination records that are checked out, but for which the import user is not already a member of the checkout. In other words, the import user’s view of these destination records would be the original version of the record rather than the checked out version.
DATA INTEGRITY ►► The “safest” behavior would be to reject source records that attempt to modify destination records that have been checked out for which the import user is not already a member of the checkout. However, the MDM Import Manager needs to be able to support automation and high-volume imports without triggering multiple exceptions, especially when the import user has join permissions and joining the checkout group is not an issue.
MDM addresses this challenge in several ways. First, regardless of whether or not the import user is already a member of the checkout, the MDM Import Manager displays, and performs record matching against, the checked out version of each checked out record (which makes sense since the original version is protected anyway).
Second, MDM can join the checkout of checked out records automatically within the MDM Import Manager based on the setting of a configuration option (and subject to join permissions).
Specifically, you can use the Auto-Join Checked Out Records configuration option to specify whether or not to automatically join the checkout of checked out records whenever possible, as summarized in Table 83.
Table 83. Auto-Join Checked Out Records Configuration Option
Setting Description
No Do not automatically join the checkout of checked out records.
Yes Automatically join the checkout of checked out records whenever possible.
MDM Import Manager Reference Guide 379
THE CHECKOUT/WORKFLOW OPTION
Recall that record checkout is a staging mechanism that creates a private “checked out” copy of a record. It allows MDM users to make a series of changes to the checked out version over an extended period of time while the visible original version remains untouched, and then have those changes become visible all at once only after the record has been checked back in (or discarded if the checked out record is rolled back).
Recall also that workflow, in conjunction with checkout, takes staging one step further to make it a guided staging mechanism. Workflow with checkout encapsulates a set of one or more records as a job, checks them out, guides the job through a sequence of steps that represent tasks for the users associated with each step, and then checks them back in at the end of the job (or rolls them back).
Within the MDM Data Manager, checking out records, adding records to workflow jobs, and launching workflow jobs are operations that are all performed manually.
By contrast, MDM can perform these operations automatically within the MDM Import Manager based on the setting of a configuration option (and subject to a variety of limitations that are designed to respect the constraints of checkout and workflow while maintaining data integrity).
Specifically, you can use the Checkout/Workflow configuration option to specify whether or not to check out the imported records or launch a workflow job containing the imported records (which may itself check out the records), as summarized in Table 84.
Table 84. Checkout/Workflow Configuration Option
Setting Description
None Do not check out imported records or launch a workflow job.
Checkout Check out the imported records non-exclusively prior to the import.
Workflowname Launch the specified workflow job that contains the imported records.
NOTE ►► Starting in MDM 7.1 SP3, the Checkout setting results in
either a Check Out Nonexclusive or a Check Out New Record Nonexclusive depending upon whether the imported record already exists or must be created.
NOTE ►► Import checkout and import workflow launch are mutually exclusive. However, regardless of whether imported records are checked out by the import itself or by the workflow job that is launched by the import, they are checked out prior to updating existing destination records or creating new records in the repository.
380 MDM Import Manager Reference Guide
NOTE ►► The Import Manager automatically launches the specified workflow immediately and without delay when the import is complete, regardless of the autolaunch settings of the workflow itself.
CAUTION ►► Importing records into a workflow job has severe performance implications.
CHECKOUT AND WORKFLOW FAILURE CONDITIONS
The previous sections describe MDM Import Manager behaviors – and a pair of configuration options – that support import automation in the face of the various constraints on checkout and workflow. Sometimes, however, these constraints cannot be avoided for all records, and the import of some of the source records may fail.
NOTE ►► Such constraints include: (1) records that are already checked out cannot be added to a workflow; (2) records that are already in a workflow cannot be checked out; and (3) records can be in multiple workflows only if none of them checks out its records.
The conditions under which source records will fail to import for each of
the settings of the Checkout/Workflow configuration option (and
sometimes dependent upon the Auto-Join Checked Out Records configuration option) are summarized in Table 85.
Table 85. Checkout and Workflow Failure Conditions
Setting Fails if Destination Record is…
None
Checked out and Auto-Join=No – or –
Checked out and Auto-Join=Yes and no join permission
Success independent of whether or not record is in workflow
Checkout In workflow – or –
Checked out by another user
Workflow
(w/o checkout)
Checked out
Success independent of whether or not record is in workflow
Workflow
(w/ checkout)
In workflow – or –
Checked out
NOTE ►► The failure to import individual records helps to assure that the integrity of the overall import is respected. For example, if the MDM Import Manager were configured to launch a workflow following the import, and a source record corresponding to an already checked out destination record were allowed to be imported, the record import would succeed but the subsequent workflow launch would fail.
MDM Import Manager Reference Guide 381
A NOTE ABOUT IMPORT AND VALIDATIONS
MDM does not run validations upon import, which might initially seem like a shortcoming of the system. In fact, this is by design; if validations were run on import and invalid records simply rejected, there would be no way for the user to determine which records failed to import and why. By contrast, allowing the invalid records into the repository allows you to use the full search and edit capabilities of MDM on the invalid records.
Moreover, rather than burden the import process with complex validations, a better solution is to make the validations part of a workflow in which import is the first step. Specifically, import launches a workflow job with the records checked out and then performs the validation as part of the workflow to validate the records before checking them back in. In this way, the import can proceed quickly and efficiently but invalid records will never become visible in the repository until and unless they are corrected.
382 MDM Import Manager Reference Guide
Import Error Logs
Records in an import job that contain an import exception are not imported, but do not stop the import process. The list of failed records in an import job is written to an import log file that Import Manager creates for each import job.
The import log file contains the following information for each failed record:
Source record ID
Reason for failure to import
Matching field names
Matching field values
The log file name includes a timestamp (as for MDIS log files), and all the import log files are stored in the Reports folder in the Import Manager installation folder.
You can use the information in the log file to resolve record exceptions.
383
PART 13: IMPORTING RELATIONSHIPS
This part of the reference guide explains how to import relationship links for parent/child relationships.
MDM Import Manager Reference Guide 385
Importing Parent/Child Relationship Links
You can use the MDM Import Manager to import relationship links into parent/child product relationships, as described in this section.
Specifically, the import file that defines the parent/child relationship links is relationship-centric, with one parent/child pair for each link placed on each line or row of the import file, as shown in Figure 219.
SKU <Kits> Part No <Parts> Quantity Required
135-A 1234 3 0
135-A 5678 2 1
135-A 9012 1 1
Figure 219. File format of parent/child relationship pairs
When importing relationship links, the MDM Import Manager behaves as follows:
Relationship selection. The drop-down list of destination tables
includes the parent/child relationships as the last group of items. Choose the relationship into which you want to import relationship links (see “The Current Tables” on page 36 and “Selecting the Current Tables” on page 50).
Destination fields list. The Destination Fields grid of the Map
Fields/Values tab includes all the parent and child table field names, each qualified with the corresponding relationship name in angular brackets (e.g. “SKU <Kits>” and “Part Number <Parts>”), and each appearing twice if the parent and child tables are the same. The list also includes the relationship qualifiers, Quantity and Required.
Field mapping. You must map the source fields to the
corresponding destination fields that uniquely identify each parent relationship record and each child relationship record, as well as the source fields that correspond to the relationship qualifiers.
Value mapping. Since all of the mapped fields correspond to index
fields, no value mapping is required; however, you must map the values in the source field mapped to the Required relationship
qualifier to the Boolean values No and Yes.
Record matching. The Match Records tab is disabled when the
destination table is a relationship; all the matching is done by the Master Data Server during the import itself based on the entire set of mapped source and destination fields as a group.
NOTE ►► Any link records that refer to parent or child records that do not already exist in the repository will be logged as an error and the corresponding link will not be imported.
386 MDM Import Manager Reference Guide
RELATIONSHIP CONFIGURATION OPTIONS
Several configuration options control how the set of relationship links for each record are updated or replaced, as summarized in Table 86 and described in the following subsections.
Table 86. Relationships Configuration Options
Option Value Description
Aggregate By
Parent Groups source records based on the parent record.
Child Groups source records based on the child record.
Links Update
Append Appends to the set of existing links, leaving existing links untouched.
Replace Deletes the set of existing links before adding the imported links.
Update Updates the set of existing links and/or their qualifier values.
New Links (Update Only)
Skip On Update, skips new links.
Create On Update, creates new links.
Existing Links (Update Only)
Skip On Update, skips imported links that correspond to existing links, leaving existing links untouched.
Update (Mapped Qualifiers)
On Update, updates existing links by overwriting just the specified relationship qualifier values with the specified import value.
Replace
On Update, deletes the existing link before adding the imported link, effectively overwriting all the relationship qualifier values either with the specified import value or the default.
Delete On Update, deletes existing links.
NOTE ►► Unlike Import Actions, which includes the Update (NULL Fields Only) choice, Existing Links does not include an Update (NULL
Qualifiers Only) choice because relationship qualifiers cannot be NULL.
NOTE ►► See “Configuration Options” on page 62 for more information about the Relationships options.
NOTE ►► See “Aggregation Option” on page 387 for more information about the Aggregate By option.
NOTE ►► See “Update Options” on page 387 for more information about the Update options.
MDM Import Manager Reference Guide 387
Aggregation Option
Recall that as part of the record matching process for the normal import of source records into the main table or any of the subtables, the MDM Import Manager can group duplicate source records based on the value of the key field (see “Source Record Merging” on page 350), and then either preserve or collapse these duplicate instances based on the setting of the Merge Source Records configuration option.
NOTE ►► When you import relationship links, the Match Records tab is disabled because the entire set of mapped fields collectively defines the record matching field or field combination for both the parent table records and the child table records.
With normal import, multiple source records correspond to the same destination record only in certain circumstances. By contrast, when you import relationship links, there are almost always likely to be multiple instances of each parent table record (identifying multiple linked children) and/or each child record (identifying multiple linked parents).
When the Links Update configuration option is set to Replace, MDM wipes out existing links before importing the new links. In this case, the end result of importing relationship links depends upon whether you group the links based on the parent table records or the child table records, as follows:
Aggregate by parent. For each parent record for which links are
specified in the source file, MDM wipes out the entire set of existing child links before importing the new links.
Aggregate by child. For each child record for which links are
specified in the source file, MDM wipes out the entire set of existing parent links before importing the new links.
You can use the Aggregate By configuration option to specify whether to group by the parent table records or by the child table records (see “Configuration Options” on page 62 for more information).
Update Options
Recall that the Qualified Update option has three settings: Append,
Replace, and Update. Moreover, for the Update setting, you can use the Set Matching Qualifiers dialog not only to identify which qualifiers should
be used for matching, but also to specify a different Update action for new links and for existing links.
Similarly, the Relationships > Links Update configuration option has
three settings: Append, Replace, and Update, and for the Update setting, respects the setting of the New Links and Existing Links configuration options. These three configuration options have the same set of interrelated behaviors as the Qualified Update option.
388 MDM Import Manager Reference Guide
Specifically, the settings of the Links Update configuration option behave as follows:
Append. Appends to the set of existing related records (leaving the set of existing links untouched).
Replace. Completely replaces the set of existing related records (by first eliminating the entire set of existing links).
Update. Updates the set of existing related records based on the settings of the New Links and Existing Links configuration options.
NOTE ►► Updating existing links is really about updating the qualifier values; nothing else about an existing link is updated.
NOTE ►► The New Links option allows you to Skip or Create new links. The Existing Links option allows you to Skip, Update, Replace, or Delete existing links, which only affects the relationship qualifiers.
NOTE ►► MDM does not permit the relationship qualifiers to have NULL values, so if: (1) no qualifier value is specified for an imported link; and (2) an existing link does not already exist or is being replaced,
the default qualifier values are Quantity=1 and Required=No.
NOTE ►► Links Update=Append is equivalent to Links
Update=Update with New Links=Create and Existing Links=Skip.
NOTE ►► Existing Links= Delete allows the import file to specify the links to delete, allowing you to conveniently remove existing links rather than add new links or update existing links as part of the import.
RELATIONSHIP FILE FORMAT
The file format for importing parent/child relationship links using the
MDM Import Manager is the same as that used with the Relationship commands within the MDM Client, and consists of multiple rows, each identifying a single parent/child pair, and for each pair, a value for the Quantity and Required relationship qualifiers, as described in Table 87.
Table 87. File Format for Importing Parent/Child Relationship Links
Relationship Import File Format
Parent/Child
The fields include the display fields of the parent table and the display fields of the child table (both sides of the relationship even if it is a single-table relationship).
The fields also include the Required and Quantity relationship qualifiers (if they are defined for the relationship).
Each row defines a single pair of related parent/child records, listing the field and qualifier values that identify a single parent record and a single child record.
MDM Import Manager Reference Guide 389
Qualifiers
The Required and Quantity fields are optional; if they appear, not all parent/child pairs need to have a value specified in the data file.
If the relationship link being imported does not exist in the repository and no value is specified for a qualifier, the default qualifier values are
Required=No and Quantity=1 (MDM does not permit relationship qualifiers to have NULL values).
If the relationship link being imported already exists in the repository, the existing value for a qualifier is: (1) overwritten if a value is specified in the data file; and (2) unchanged if no value is specified for the qualifier in the data file.
391
PART 14: MULTILINGUAL SUPPORT
This part of the reference guide contains a general overview of multilingual support within the MDM system and a specific description of the multilingual features within the MDM Import Manager. Multilingual support allows you to import multilingual data into a single MDM repository.
MDM Import Manager Reference Guide 393
Multilingual Basics
Basic multilingual concepts are explained in the following sections.
LANGUAGE LAYERS
When you define an MDM repository as multilingual, MDM stores the multilingual data and metadata in multiple language layers, one for each language. A single language repository has a single layer; a multiple language repository has multiple layers.
The best way to understand language layers is to start by considering a typical unstructured approach to storing multiple languages for a field by creating multiple instances of the same field, as shown in Figure 220.
Part Number Product Color (Eng) Color (Fre) Color (Ger)
113 T-Shirt Red Rouge Rot
114 T-Shirt Green Vert Grün
115 T-Shirt Blue Bleu Blau
Figure 220. A typical table with three Color fields for three languages
The table above contains three Color fields side-by-side, one for each language (English, French, and German), and can be successfully used to store the multilingual color data within the table.
Unfortunately, the system would know nothing of the relationship among the fields, so it cannot offer the user a language-centric view of the data, and the user has no way of knowing that the fields are related (except that the field names above have been tagged with the corresponding language). Finally, all the field names themselves exist only in English.
Now consider the MDM approach that uses multiple language layers to represent the multiple languages, as shown in Figure 221.
Figure 221. An MDM table with three language layers
Multiple layers efficiently organize and structure both multilingual data and metadata, with a single Color field above containing multiple data buckets rather than multiple Color fields that are completely unrelated, and multiple language-specific field names for all of the fields.
German
French
English
394 MDM Import Manager Reference Guide
LANGUAGE INHERITANCE
To support convenient user access to multilingual data, the MDM Import Manager provide a language-centric view of data within a multilingual
repository, meaning that data is presented from the point of view of a particular language layer at a time. This single language is called the current language, and you select it when you first connect to a multilingual repository.
NOTE ►► The language-centric view determines not only which language of data is displayed, but also which language of metadata is displayed, including table names, field names, and attribute names.
Now consider a multilingual field that is missing data in the current language. In a single-language repository, the value is shown as empty or NULL. However, MDM uses an innovative inheritance scheme to display – and color-code – data from other language layers for data values that are missing in the current language.
The actual value shown depends on the language inheritance defined for the current language. The language inheritance identifies the priority sequence of language layers from which to find a non-NULL value to display when the current language layer is NULL.
NOTE ►► Language inheritance is a type of layer transparency that allows individual data values to “show through” from other language layers when the current layer is missing data.
Language inheritance is set for each language, and is defined by the administrator as the ordering of all the other languages of the repository, split into: (1) primary inheritance (for languages whose values are close enough to the current language to be acceptable for publishing); and (2) secondary inheritance (for languages whose values are too different from the current language to be acceptable for publishing, but are perhaps useful during data entry and/or translation).
Thus, for the MDM Import Manager, there is the current language and two levels of inheritance, color-coded as follows:
Black. The value is from the current language.
Green. The value is from a primary inherited language.
Red. The value is from a secondary inherited language.
NOTE ►► The MDM Import Manager displays: (1) actual values from the current language; (2) primary inherited values; and (3) secondary inherited values.
MDM Import Manager Reference Guide 395
Consider a repository with three language layers: (1) English [US]; (2) English [UK]; and (3) German [DE]. Both English values are typically the same, so you can set the value for one version of English and allow the other to inherit it. However, you don’t want the English languages to inherit German or vice versa. In this case, inheritance for each language would be as shown in Table 88.
Table 88. Language Inheritance Example for Three Languages
Language Primary Inheritance Secondary Inheritance
English [US] English [UK] German [DE]
English [UK] English [US] German [DE]
German [DE] <none> English [US]; English [UK]
Given the above inheritance, a record with the Size field set to “Small” for English [US] and to NULL for both English [UK] and German [DE] would display and color-code the value from each language-centric view as shown in Table 89.
Table 89. Language-Centric Display Example for Three Languages
Language Actual Value Inherits From Display Value
English [US] Small <actual value> Small
English [UK] NULL English [US] Small
German [DE] NULL English [US] Small
The terminology and behavior around inheritance in the MDM Import Manager is summarized in Table 90.
Table 90. Inheritance in the MDM Import Manager
Inheritance Item MDM Client
Language ordering Language-specific
Actual values Current language
Levels of inheritance Two (primary and secondary)
Type of inherited values Metadata and data
Color coding Black / Green / Red
NOTE ►► The MDM Import Manager displays the current language in the title bar after the MDM repository name. If the language has not been renamed, the language displays as “language [co]” (where “language” is the language name and “co” is the country name).
396 MDM Import Manager Reference Guide
Multilingual Operations
The following sections describe the MDM Import Manager operations for connecting to and importing data into a multilingual MDM repository. There are no explicit commands for any of these operations. Rather, the multilingual data is processed using various extensions to the existing tabs, dialogs, and controls, as summarized in Table 91.
Table 91. Multilingual MDM Import Manager Operations
Operation Description
Language field / Connect to MDM repository dialog
Connects to a multilingual MDM repository.
Destination Fields grid / Map Fields/Values tab
Maps language-specific source fields to the layers of a multilingual field.
Language Field / Destination Fields grid / Map Fields/Values tab
Identifies the applicable language layer of expanded multilingual fields.
Expand Multilingual Fields option / Configuration Options dialog
Whether or not – and how – to expand multilingual fields into multiple fields.
Value Mapping Map button / Map Fields/Values tab
Populates missing values in the current language layer when mapping values.
CONNECTING TO THE MDM IMPORT MANAGER
When you first connect to a multilingual MDM repository using the MDM Import Manager, you must select the current language for that MDM Import Manager session.
You can use the Language field in the Connect to MDM Repository dialog to select the current language from the drop-down list of repository languages, as shown in Figure 222.
Figure 222. Language field in Connect to MDM Repository dialog
MDM Import Manager Reference Guide 397
FIELD MAPPING TO MULTILINGUAL DESTINATION FIELDS
Recall that you can import source data into an MDM repository simply by mapping from a source field to the corresponding MDM destination field in the Field Mapping grids of the Map Field/Values tab.
When the MDM destination field is a multilingual field within a multilingual MDM repository, the source data may be structured in several different ways, including: (1) a single source field with data for the current language layer; (2) multiple source fields, one for each language; or (3) a pair of source fields, one containing the data for all languages and the second identifying the corresponding language for each data value.
MDM supports import into multilingual destination fields based on how the source data is structured, as described in the following sections.
The Expand Multilingual Fields Option
You can use the Expand Multilingual Text Fields configuration option to specify whether or not – and how – to expand multilingual text fields into multiple fields in the Destination Fields grid of the Map Fields/Values tab, as summarized in Table 92.
Table 92. Expand Multilingual Text Fields Configuration Option
Setting Description
No Do not expand multilingual text fields; only display a single field for the current language layer.
As Multiple Fields Expand multilingual text fields into multiple fields, one field per language layer.
As Field and Language Expand multilingual text fields into multiple fields, as the field itself and a language field.
NOTE ►► If you are importing data only for the current language layer and do not wish to expand multilingual text fields, you can turn off the expansion of multilingual text fields in the Destination Fields grid by
setting the Expand Multilingual Text Fields configuration option to No.
As Multiple Fields Setting
When you import source data into a multilingual field, there may be multiple source fields corresponding to each multilingual MDM field, one
for each language.
For multilingual Text and Text Large fields, the MDM Import Manager allows you to import these multiple source fields into the multiple
language layers of the MDM field at the same time using the As Multiple
Fields setting of the Expand Multilingual Text Fields configuration option
398 MDM Import Manager Reference Guide
It does this by expanding each multilingual Text and Text Large field into an instance of the field in the Destination Fields grid for each repository language, as shown for a Description field in Figure 223. You can then field map from each language-specific source field to the applicable language layer instance of the multilingual destination field.
Figure 223. Multilingual Text field expanded into multiple items
NOTE ►► The field name for each instance of the multilingual field is language-specific (e.g. Descrizione for Italian).
NOTE ►► The Language column in the Destination Fields grid: (1) displays the name of the applicable language layer for expanded instances of multilingual fields; and (2) is empty for non-lingual fields.
TIP ►► You can import multilingual lookup data into multiple language layers at the same time by making the lookup table itself the destination table.
As Field and Language Setting
When you import source data into a multilingual field, there may be a pair of source fields corresponding to each multilingual MDM field, one containing the data for all languages and the second identifying the corresponding language for each data value.
For multilingual Text and Text Large fields, the MDM Import Manager allows you to import this pair of source fields into the multiple language
layers of the MDM field at the same time using the As Field and Language setting of the Expand Multilingual Text Fields configuration option
It does this by expanding each multilingual Text and Text Large field into a pair of fields: (1) the field itself; and (2) a Language field, as shown for a Description field in Figure 224. You can then field map from the source field containing data values to the data field and the source field containing language values to the Language field.
MDM Import Manager Reference Guide 399
Figure 224. Multilingual Text field expanded into a pair of items
NOTE ►► The Language field includes a value for each repository language in the Destination Values grid of the Value Mapping pane.
VALUE MAPPING TO MULTILINGUAL DESTINATION VALUES
Recall that the Value Mapping grids of the Map Fields/Values tab allow you as part of the import process to map from source data values to destination data values, to convert the source values before mapping, and to explicitly add new values to the list of destination data values.
In a multilingual MDM repository, the Destination Values grid still displays the list of destination values. However, the list now displays not only actual values from the current language layer, but also for values that are missing in the current language, inherited values from primary language layers in green and from secondary language layers in red.
When you map from a source value to a destination value that is inheriting from another language layer, the MDM Import Manager allows you to populate the missing data value in the current language layer with the converted source value.
Specifically, when the destination value is missing in the current
language layer, the Map button (between the values grids) pops up a context menu with two command choices, as shown in Figure 225.
Figure 225. Pop-up Set menu
400 MDM Import Manager Reference Guide
You can set the missing destination value in the current language layer as summarized in Table 93 and described in this section.
Table 93. Map to Missing Value Map Operations
Operation Description
Set Missing Value Sets the missing destination value in the current language layer.
Do Not Set Missing Value Does not set the missing value in the current language layer.
To map to a destination value that is missing in the current language layer:
1. In the Source Values grid, select the source value you want to map.
2. Click on the Map button (between the values grids) to map the selected source value and the selected destination value and choose from the cascading menu:
Set Missing Value Do Not Set Missing Value
3. MDM maps the source value to the destination value. If you choose Set Missing Value, sets the destination value in the current language layer with the converted source value and highlights it in blue.
IMPORTING MULTILINGUAL DATA INTO LOOKUP TABLES
Importing multilingual data into a lookup table depends upon a number of variables, including whether: (1) the table is a flat or hierarchy table; (2) new records are being added or language layer(s) of existing records are being updated; and (3) just a single language layer or multiple language layers are being imported at the same time.
In particular, import into the hierarchy table itself is now roughly equivalent and as flexible as importing into flat tables thanks to several new features: (1) value mapping is now optional rather than required to the primary display field of the hierarchy table; (2) both internal and leaf
nodes can be mapped; and (3) the [Remote Key] field is now available not only for field mapping but also for record matching.
NOTE ►► Previously, adding new multilingual records and updating the language layer of existing records was not possible with the Import Manager.
Importing multilingual data into flat and hierarchy lookup tables is summarized in Table 94 and described in detail in the following subsections.
MDM Import Manager Reference Guide 401
Table 94. Importing Multilingual Data into Lookup Tables
Flat Hierarchy Notes
To add new multilingual records
n/a If necessary, create the hierarchy from the current language layer Name field that contains the hierarchy information (e.g. using the
Split Hierarchy command).
Hierarchy: Only one language layer Name field contains the hierarchy information.
Map the current language layer Name field.
No value mapping necessary.
Map the current language layer Name field.
Map/Add the entire set of hierarchy values.
Hierarchy: Now supports mapping of internal nodes.
Map the other language layer Name field(s).
No value mapping necessary.
Map the other language layer Name field(s).
No value mapping necessary.
Hierarchy: Map Value button previously could not be unchecked for hierarchy field and so Name field had to be value mapped.
Map the [Remote Key] field. Map the [Remote Key] field.
Record match using: (a) any
Name field; (b) the [Remote
Key] field; or (c) the combination of both.
Record match using: (a) the
Name field; (b) the [Remote
Key] field (if the Name field was not value mapped); or (c) the combination of both.
[Remote Key] previously not available as a record matching field, and so Name field had to be used for record matching.
To update one or more language layers of existing records
Map the other language layer Name field(s).
No value mapping necessary.
Map the other language layer Name field(s).
No value mapping necessary.
Hierarchy: Map Value button previously could not be unchecked for hierarchy field and so Name field had to be value mapped
Map the [Remote Key] field. Map the [Remote Key] field. [Remote Key] previously did not exist, and so current language layer Name field had to be mapped.
Record match using the
[Remote Key] field and set
the import action to Update (All Mapped Fields).
Record match using the
[Remote Key] field and set
the import action to Update (All Mapped Fields).
[Remote Key] previously not available as a record matching field, and so Name field had to be used for record matching.
Hierarchy: Can use [Remote
Key] field for record matching
because Map Value button can now be unchecked for hierarchy field and so Name field no longer needs to be value mapped.
NOTE ►► With value mapping optional for the primary display field of a hierarchy table, MDM automatically figures out the other language layers. When value mapping was required, the primary display field for each language layer had to be individually value mapped.
402 MDM Import Manager Reference Guide
Import into Flat Tables
Multilingual import into a flat table is described in this section.
To add new multilingual records:
1. Map the current language layer Name field. No value mapping is necessary.
2. Map the other language layer Name field(s). No value mapping is necessary.
3. Map the [Remote Key] field.
4. Record match using: (a) any Name field; (b) the [Remote Key] field; or (c) the combination of both.
To update one or more language layers of existing records:
1. Map the other language layer Name field(s). No value mapping is necessary.
2. Map the [Remote Key] field.
3. Record match using the [Remote Key] field and set the import action to
Update (All Mapped Fields).
Import into Hierarchy Tables
Multilingual import into a hierarchy table is described in this section.
To add new multilingual records:
1. If necessary, create the hierarchy from the current language layer
Name field that contains the hierarchy information (e.g. using the Split
Hierarchy command).
2. Map the current language layer Name field and then Map/Add the entire set of hierarchy values.
3. Map the other language layer Name field(s). No value mapping is necessary.
4. Map the [Remote Key] field.
5. Record match using: (a) the Name field; (b) the [Remote Key] field (if the Name field was not value mapped); or (c) the combination of both.
NOTE ►► The Replace import action is disabled for import into the hierarchy table itself; the Import Manager and MDIS will substitute
Update (All Mapped Fields) in old maps that have the Replace import action.
MDM Import Manager Reference Guide 403
To update the language layer of existing records:
1. Map the other language layer Name field(s). No value mapping is necessary.
2. Map the [Remote Key] field.
3. Record match using the [Remote Key] field and set the import action to
Update (All Mapped Fields).
RECORD MATCHING AND MULTILINGUAL DESTINATION FIELDS
In a multilingual repository, all of the language layer instances of a multilingual Text field appear in the Mapped Destinations Fields list and are available as Matching Fields (where each field appears as “fieldname <language layer>”).
This is true even if you have expanded the multilingual Text field into an instance of the field for each repository language in the Destination Fields grid of the Map Fields/Values tab.
SOURCE RECORD MERGING AND MULTILINGUAL DATA
Consider three source records containing different layers of multilingual data for the same record in MDM, as shown in Figure 226
SKU Color (Eng) Color (Fre) Color (Ger)
#1 113 Red
#2 113 Rouge
#3 113 Rot
Figure 226. Three records containing different layers of multilingual data
Prior to import, the source record merging logic of MDM creates the single merged record shown in Figure 227.
SKU Color (Eng) Color (Fre) Color (Ger)
113 Red Rouge Rot
Figure 227. Single merged record of multilingual data
NOTE ►► MDM was fixed to behave as described above in SP2.
405
PART 15: REMOTE SYSTEMS AND MDM
This part of the reference guide contains a general overview of remote systems and MDM, and a specific description of the related features within the MDM Import Manager, including remote systems, key mapping, and ports, which allow MDM to synchronize data between itself and other systems.
407
Introduction
MDM has special features that enable it to synchronize data between itself and other systems, as described in the following sections.
WHAT IS A REMOTE SYSTEM?
Any logical system that can supply data to or receive data from MDM is known as a remote system.
MDM can import data from remote systems and create/update master data objects using that data. Master data objects include main table records, subtable records / lookup values, attribute names, and text attribute text values.
When data is changed in a remote system, the changes can be imported into MDM. Using previously created structural and key mappings, in conjunction with dynamically reconfigurable transformations and mappings, the data is applied to the master data objects. All changes to master data objects are tracked.
At any time, master data objects can then be distributed to all known remote systems through a process known as syndication. This involves determining which master data objects need to be distributed and converting them into a form that the remote system can understand.
NOTE ►► The remote system concept supports a number of features related to distribution by bundling all sorts of useful information related to a particular outside system, including key mapping, import maps, syndication maps, and various timestamps.
KEY MAPPING
A remote system’s objects are mapped to master data objects within MDM using key mapping. A key mapping maintains the relationship
between the remote system’s identifier (or key) for an object and the corresponding master data object in MDM.
A key in MDM is a remote system-specific and object-type-specific
unique identifier. Different remote systems can have their own separate collection of keys. Within a remote system, each type or collection of objects can have its own separate collection of keys as well. Key mappings are subject to the requirement that two different objects of the same type from the same remote system cannot have the same key.
Remote system objects of a particular type can map only to MDM objects of a particular type. A key can map to only one MDM object. However, an MDM object may map to multiple keys from the same remote system. When an MDM object maps to multiple keys, one of the keys is marked as the default key. The default key is the one that is used when syndicating a reference to the mapped MDM object.
408 MDM Import Manager Reference Guide
For example, for a particular remote system, the two color objects Light Red and Dark Red both map to the MDM object Red. An MDM product object Shirt has a Color attribute set to Red. When this Shirt object is syndicated back to the remote system, the default key is used to choose the value to syndicate from the two objects Light Red and Dark Red.
MDM objects that can be mapped to remote system keys include user-defined table records and text attribute text values. Key mapping must be enabled on each collection of objects for MDM to maintain the keys.
[REMOTE SYSTEM] AND [REMOTE KEY] FIELDS
MDM uses the remote systems defined in the Remote Systems table within the MDM Console to store and maintain key mapping information for each record or text attribute. It does this using a virtual “key mapping” field that you never see in the MDM Import Manager.
This virtual key mapping field is very much like a qualified lookup field into a virtual key mapping qualified lookup table. Each record of the virtual lookup table consists of just two fields:
[Remote System]. A single-valued Text field that contains the name of the remote system. This is a normal field.
[Remote Key]. A single-valued Text field that contains a key value for the corresponding remote system. This is a qualifier field.
In effect, each Remote Systems table record becomes a record of the key mapping qualified lookup table, and each actual key mapping becomes a link of the key mapping qualified lookup field, one per
[Remote System] / [Remote Key] value pair, as illustrated in Figure 228.
SKU Name Lookup [Key Mapping]
213 Widget MDM; 112
CRM; 103
CRM; 105
R/3; 55-77
Figure 228. Key Mapping information stored in virtual lookup field
The [Remote System] and [Remote Key] fields are normally not visible; however they do appear in several places in the MDM Import Manager.
Specifically, you choose a [Remote System] when you enter the MDM Import Manager (or you choose one indirectly by selecting a map), and
[Remote Key] appears in the Destination Hierarchy tree, the list of Destination Fields in the Map Fields/Values tab, and the Value Matching lists in the Match Records tab.
MDM Import Manager Reference Guide 409
KEY MAPPING BEHAVIOR
The key mapping behavior of the MDM Import Manager is summarized in Table 95.
Table 95. Key Mapping Behavior of the MDM Import Manager
Level Description
Session Each MDM Import Manager session and each import map is associated with a single remote system, which is treated as the issuing remote system for all remote keys loaded during the session.
Table
If the destination table has Key Mapping enabled in the MDM Console,
then the [Remote Key] virtual field added to every field list for the table.
Appears only for remote system Type=Import/Export
Appears for table Type=Main, Flat, Hierarchy, Taxonomy, or Qualified
Field
[Remote Key] appears in: (1) the Destination Hierarchy tree; (2) the list of Destination Fields in the Map Fields/Values tab; and (3) the Value Matching lists in the Match Records tab.
Field Type=Key Mapping for [Remote Key] in Destination Fields grid
Available for field mapping like any other field
When selected, no values are displayed
Value
Each key in the source data is represented by a value in the list of source values. When a Source Field is mapped to a Destination Field that has key mapping enabled, the MDM Import Manager loads value mapping from the table of remote keys rather than from the map, automatically mapping each Source Value to a Destination Value if the source/destination value pair exists in the key mapping (similar to the way value mappings are restored when a map file is loaded).
Destination fields that have key mapping include: (1) a text attribute that has key mapping enabled; (2) a lookup field for a table that has key mapping enabled; and (3) the display field of a lookup table that has key mapping enabled (for imports to a subtable)
Record
[Remote Key] can be used as a record matching field. Each source record will match one destination record if a key mapping exists for the source remote key, and each remote key is associated with a single destination record during the matching step. If no matching record is
found and the Create option is selected, the remote key will be associated with the newly created record.
During source record aggregation, [Remote Key] is treated like a multi-valued field (i.e. all values from a group of source records combined into a single multi-value). This may result in multiple remote keys for a single source record. In this case, all keys will be mapped to the same single destination record.
When [Remote Key] is used for record matching and a source record
matches multiple destination records, the two Update import action
options are disabled; the only available import action is to Replace all destination records with the source record.
410 MDM Import Manager Reference Guide
Level Description
Key Mapping
Key mappings are saved only for: (1) tables that have key mapping enabled in the MDM Console; and (2) text attributes that have key mapping enabled in Taxonomy mode of the MDM Client.
The key in the source data is designated – and key mapping value pairs are established – as follows: (1) for destination table records, through Record Matching in the Match Records tab, by mapping the key field in
the source to the [Remote Key] field in the Destination Fields list; and (2) for subtables of the destination table and for text attributes, through Value Mapping, by mapping source values to destination values.
The key mapping value pairs are stored in the repository as follows: (1) for destination table records (but not for source records that are skipped), subtables of the destination table, and text attributes, when the import is executed; and (2) for mappings to existing records of subtables of the destination table and to text values of text attributes, when the map is saved
Source keys are stored literally as they appear in: (1) the source field for each record (for the destination table); and (2) the source value list (for subtables of the destination table and text attributes).
Key mappings are saved only for remote system Type=Inbound or Inbound/Outbound
When multiple source records map to the same destination record, one key is designated as the default. Upon subsequent export of the destination record, the single default key will be used.
When an inbound key mapping is already assigned to an existing destination record, whether or not it is reassigned depends upon the
setting of the Reassign Existing Key Mappings configuration option.
KEY MAPPING REASSIGNMENT
Recall that each key mapping value pair consists of a remote system and a remote key value, that each key mapping value pair must be uniquely associated with a single record in an MDM repository, and therefore that each remote key value must be unique for a particular remote system.
A challenge arises when a source record contains a key mapping value pair that already exists in the MDM repository but is associated with a destination record other than the one that matches the source record. Because of the uniqueness of key mapping value pairs, the key mapping could not be added to the matching destination record without first removing it from the existing destination record.
DATA INTEGRITY ►► The “safest” behavior would be to reject source records that attempt to reassign a key mapping to another record. However, the MDM Import Manager needs to be able to support automation and high-volume imports without triggering multiple exceptions, especially when the reassignment of key mappings is not an issue.
MDM can automatically reassign key mapping value pairs within the MDM Import Manager based on the setting of a configuration option.
MDM Import Manager Reference Guide 411
Specifically, you can use the Reassign Existing Key Mappings configuration option to specify whether or not – and how – to reassign existing key mappings during import, as summarized in Table 96.
Table 96. Reassign Existing Key Mappings Configuration Option
Setting Description
No Do not reassign existing key mappings; fail to import source records that contain them.
Into Original Records Only
Reassign existing key mappings, but only into original records.
Into Original and Checked Out Records
Reassign existing key mappings into both original and checked out records.
DATA INTEGRITY ►► When a key mapping is reassigned to a new record: (1) it is immediately removed from the old record to which it was previously assigned; (2) if the old record is a checked out record, the key mapping is removed from both the original and the checked out versions of the record; and (3) if the new record is a checked out record, you will lose the key mapping if the new record is subsequently rolled back rather than checked in, resulting in data loss.
WHAT IS A PORT?
With remote systems as a foundation, an MDM port encapsulates all of
the configuration and logistical information associated with inbound and outbound processing of data from these remote systems.
In so doing, it simplifies the process of: (1) delivery and consolidation of raw data from remote systems into MDM using the MDM Import Manager; and (2) extraction and distribution of data from MDM to remote systems using the MDM Syndicator.
In each case, the port represents the logical point of contact between MDM and the outside world (e.g. XI or a user of the MDM Import Manager). Within the MDM Import Manager and MDM Syndicator, it represents the physical staging location of data and a logical handle by which to identify all of the encapsulated information.
NOTE ►► Along with simplifying the user interaction, ports also lay the foundation for future automation of the consolidation and distribution processes within MDM.
412 MDM Import Manager Reference Guide
Remote System Operations
The following sections describe the MDM Import Manager operations for managing the remote systems, key mapping, and ports of an MDM repository.
CONNECTING TO A REMOTE SYSTEM
Whenever you connect Import Manager to a data source, you must select a remote system as the source of imported data.
You can use the Remote System field in the Connect to Source dialog to select a remote system from the drop-down list of remote systems, as shown in Figure 229.
Figure 229. Connect to Source dialog
NOTE ►► The drop-down list includes: (1) the built-in remote system
named MDM; and (2) user-defined remote systems of type Inbound.
CONNECTING TO A PORT
An inbound port identifies a file system location that contains source data files waiting for import into MDM. All information required by MDM or an MDM Import Manager user to identify and process the inbound data – including not only the location, but also the remote system, the object type, and the MDM Import Manager map to automatically load – is contained in the port’s definition.
When the source Type=Port, you can use the Port field in the Connect to Source dialog to select a port from the drop-down list of ports, as shown in Figure 229.
MDM Import Manager Reference Guide 413
Figure 230. Connecting to a port using the Connect to Source dialog
NOTE ►► The Port drop-down list includes an entry for each inbound port you define in the MDM Console for the selected Remote System; those that do not have files waiting or are locked by another user are disabled in the drop-down list.
NOTE ►► When you select a port, the MDM Import Manager automatically selects the first source data file waiting to be processed;
you can then use the Get Next Source File command to select each succeeding file.
FIELD MAPPING
To map between source records and MDM records of the current destination table, use the Field Mapping grids in the Map Fields/Values tab to create a field mapping between the remote system key field in the
Source Fields list and the [Remote Key] field in the Destination Fields list.
NOTE ►► The [Remote Key] virtual field appears only when Key Mapping is enabled for the destination table and the remote system
Type is Inbound or Inbound/Outbound.
During the import process, source records will be matched against MDM records using the matching rules you define. The result of this matching will be used to create key mappings between the remote system key values and the MDM objects.
VALUE MAPPING
To map between source records and MDM lookup table records or text attribute text values, use the Value Mapping grids in the Map Fields/Values tab to create value mappings.
NOTE ►► The source key used for existing and new MDM destination values is the Original Value in the Source Values list.
414 MDM Import Manager Reference Guide
ATTRIBUTE MAPPING
When displayed as name/value pairs (see “Attribute Data Import as Name/Value Pairs” on page 167 for more information), MDM will automatically perform key mapping for an attribute definition (name) based on the source field mapped to the name/value pair.
The taxonomy [Attributes], [Text Values], and [Links] destination tables include [Attribute Remote Key] fields. Any source field can be mapped to these fields. These fields can also be used as record matching fields.
The taxonomy [Text Values] destination table includes a [Text Value Remote Key] field.
RECORD MATCHING
When importing to a table that has Key Mapping enabled, the [Remote Key] field is available for record matching purposes when it has been mapped. Selecting this field will use the existing key mappings to match the source records.
When mapping to MDM subtable records, existing key mappings are also used to automatically map values when Key Mapping is enabled for the underlying destination records.
NOTE ►► When the [Remote Key] field is used for record matching,
the two Update import action options are disabled for multiple matches.
NOTE ►► If Key Mapping is not enabled, the standard value mapping functionality will try to remember the mapped values. However the import map only remembers the values actually mapped from the last import and discards unused value mappings. The Key Mapping retains mappings whether they are used or not.
Lookup Table Imports Using the [Remote Key] Field
When the current table is a lookup table, the [Remote Key] field can also be used as a record matching field.
However, to enforce the key mapping semantic that a key value can be associated with at most a single record in the MDM repository, MDM
imposes a restriction on the use of the [Remote Key] field as a matching field.
Specifically, if you map values for the display field(s) of a flat table or the
primary display field of a hierarchy table, the [Remote Key] field can be used as a matching field only if it is combined with the display field(s) or the primary display field.
This constraint exists because value mapping allows you to add new values, which correspond to records that do not yet exist in the table but
MDM Import Manager Reference Guide 415
will be created as part of the import. If the [Remote Key] field could be used as the matching field by itself, MDM might find an existing record based on the key value even though a new value was just created based on the value mapping.
Requiring that the [Remote Key] field be used in conjunction with the value mapped display field(s) guarantees that the key field combination will not match an existing or newly created record, allowing MDM to create a new record as part of the record matching process.
NOTE ►► In addition, to maintain the constraint that a key value can
be associated with at most a single record, if the [Remote Key] value is already associated with an existing record, MDM automatically removes it from that record and associates it with the new record.
NOTE ►► Prior to key mapping support within MDM, when the current table was a lookup table, only the display field(s) of a flat lookup table or the primary display field of a hierarchy table could be used as the matching field(s).
Conversely, if you do not map values for the display field(s), there is no
restriction on the use of the [Remote Key] field as a matching field. In
this case, the [Remote Key] field can be used as the matching field by itself because without value mapping, there is no opportunity to create
new values, and therefore no possibility that the [Remote Key] value already exists for a node whose corresponding record was just created.
[REMOTE KEY] VALUES AND LOOKUP FIELD AUTOMAPPING
When you map a source field containing remote key values to a destination lookup field for a table that has key mapping enabled, the MDM Import Manager automatically maps the source values to
destination values based on the remote keys of the lookup table for that remote system.
NOTE ►► Unmapping a source value does not automatically remove the corresponding key mapping from the lookup table (you must instead use the MDM Data Manager’s Edit Key Mapping dialog). The only time MDM does automatically delete a key mapping is if mapping a source value causes the same key for a remote system to be assigned to two destination values. If this happens, MDM resolves the conflict by automatically deleting the key for the “old” destination value.
NOTE ►► Key mapping information for lookup field values is saved whenever you save a map, unlike key mapping information for record matching, which is only saved during import.
416 MDM Import Manager Reference Guide
NOTE ►► For destination hierarchy lookup fields that have key mapping enabled, you can disable the automatic value mapping
feature by setting the Disable automatic hierarchy value mapping according to remote key configuration option to No (see "Configuration Options" on page 62 for more information).
[REMOTE KEY] VALUES AND RECORD REPLACEMENT
Existing [Remote Key] values are always preserved, even for
destination records set to be replaced with the Replace action, as
follows: (1) if a source field is mapped to the [Remote Key] field, MDM will reject the source record rather than perform the replace; and (2) if a
source field is not mapped to the [Remote Key] field, MDM will perform
the replace after first transferring the [Remote Key] values of the target
replaced records to the new record. In each case, the Replace action will
not result in the loss of [Remote Key] values.
MDM Import Manager Reference Guide 417
PART 16: REFERENCE
This part provides reference information in one convenient location, including a menu command reference, glossary, and index.
418 MDM Import Manager Reference Guide
Command Reference
FILE MENU
Main Menu Context Menu Description
Open… n/a Opens the selected map
Close n/a Closes the current map
Save n/a Saves the current map
Save As… n/a Saves the current map to a new name
Save Update n/a Saves an update of the current map
Get Next Source File n/a Selects the next source data file waiting on the current port
Connect to Source… n/a Connects to a new data source
Import… n/a Imports a map file into the repository
Export… n/a Exports a map file from the repository
Exit n/a Quits the application
VIEW MENU
Main Menu Context Menu Description
Select Source Table > table
[Drop-Down List] Selects the current source table
Select Destination Table > table
[Drop-Down List] Selects the current destination table
Switch Layout n/a Switches the main window layout
MDM Import Manager Reference Guide 419
SOURCE MENU
Main Menu Context Menu Description
Add Field Source Hierarchy Tree
Adds an empty field to the hierarchy
Clone Field Source Hierarchy Tree
Clones the selected field and its values
Rename Field Source Hierarchy Tree
Renames the selected field
Delete Field Source Hierarchy Tree
Removes the selected field from the hierarchy
Join > table > field Source Hierarchy Tree
Joins the selected field to a field in another table
Unjoin Source Hierarchy Tree
Unjoins the selected field
Lookup > field Source Hierarchy Tree
Looks up a field from another joined table
Unlookup Source Hierarchy Tree
Removes the selected lookup field from the hierarchy
Split into Hierarchy Source Hierarchy Tree
Splits the data values within the selected field into a hierarchy value
Split into Multiple Fields Source Hierarchy Tree
Splits the data values within the selected field into multiple fields
Create Hierarchy Field Source Hierarchy Tree
Creates a new field based on the selected fields and then splits it into a hierarchy
Create Compound Field > compound field
Source Hierarchy Tree
Combines fields that have been mapped to the multiple display fields of the selected destination lookup field
Create Pivot Table…
Source Hierarchy Tree
Creates a new table by pivoting on the selected fields
Create Reverse Pivot Table…
Source Hierarchy Tree
Creates a new table by reverse pivoting on the selected fields
FIELDS MENU
Main Menu Context Menu Description
Find Next Matching Destination Field
Source Fields Grid
Finds the first or next similarly destination field with a name similar to the selected source field
Set Flat
NULL Handling > setting
Source Fields Grid
Sets the Flat NULL Handling option for the selected source fields
420 MDM Import Manager Reference Guide
Main Menu Context Menu Description
Set Hierarchy NULL Handling > setting
Source Fields Grid
Sets the Hierarchy NULL Handling option for the selected source fields
Set Case Sensitivity >
setting
Source Hierarchy Tree & Source Fields Grid
Sets the Case Sensitivity option for the selected source field
Set Value Delimiter > Split Hierarchy…
Source Hierarchy Tree
Sets the Split Hierarchy delimiter for the selected source field
Set Value Delimiter > Split Multiple Fields…
Source Hierarchy Tree
Sets the Split Multiple Fields delimiter for the selected source field
Set Value Delimiter > Partition Combine…
Partition By List Sets the Partition Combine delimiter for the selected source field
Set Value Delimiter > Multi-Valued…
Source Fields Grid
Sets the Multi-Valued delimiter for the selected source fields
Set Value Delimiter > Coupled Numeric…
Source Fields Grid
Sets the Coupled Numeric delimiter for the selected source fields
Set Value Delimiter > Inherit Defaults
n/a Restores inheritance for all the value delimiters for the selected source fields
Set Value Conversion Filter > filter
Source Fields Grid
Sets the value conversion filter for the selected source field
Set Value Conversion Filter > Multiple…
Source Fields Grid
Sets multiple value conversion filters for the selected source field
Filter Unlinked Attributes Destination Fields Grid
Filters the attributes that are not linked to the selected categories (single taxonomy field)
Filter Unlinked Attributes > taxonomy field
Destination Fields Grid
Filters the attributes that are not linked to the selected categories (multiple taxonomy fields)
Set NULL Interpretation > setting
Destination Fields Grid
Sets the NULL Interpretation option for the selected destination fields
Set Multi-Valued Update > setting
Destination Fields Grid
Sets the Multi-Valued Update option for the selected multi-valued destination fields
Set Qualified Update > setting
Destination Fields Grid
Sets the Qualified Update option for the selected qualified lookup destination fields
Set Data Group Display > setting
Destination Fields Grid
Sets the Data Group Display option for the selected object lookup destination fields
Set MDIS Unmapped Value Handling > setting
Destination Fields Grid
Sets the MDIS Unmapped Value Handling option for the selected destination field
MDM Import Manager Reference Guide 421
VALUES MENU
Main Menu Context Menu Description
Display Distinct Values Source Hierarchy Tree
Displays the distinct values for the selected field
Remove Distinct Values Source Hierarchy Tree
Removes the distinct values for the selected field
Apply Value Conversion Filter > filter
Source Values Grid
Converts the selected source values using the specified value conversion filter
Find Next Unmapped Source Value
Source Values Grid
Finds the first or next unconverted or unmapped source value
Reorder Mapped Destination Values
Source Values Grid
Reorders the mapped destination values for the selected source value
Find Next Mapped Destination Value
Source Values Grid
Finds the first or next destination value to which the selected source value is mapped
Select Entire Branch > All Nodes
Source Values Grid
Selects the selected internal node and all of its internal and leaf node children
Select Entire Branch > All Leaf Nodes
Source Values Grid
Selects all of the leaf node children of the selected internal node
Select Entire Branch > Unmapped Leaf Nodes
Source Values Grid
Selects all of the unmapped leaf node children the selected internal node
Add Destination Value
Destination Values Grid
Adds a new destination value
Add Destination Hierarchy Value > As Sibling
Destination Values Grid
Adds a new destination hierarchy value as a sibling
Add Destination Hierarchy Value >
As Child
Destination Values Grid
Adds a new destination hierarchy value as a child
Remove Destination Value
Destination Values Grid
Removes the selected new destination values
RECORDS MENU
Main Menu Context Menu Description
Set Key Field Matching Fields List
Sets the selected matching field to be the key field used to identify duplicate source records
Set Active Records > value match
Matching Fields List
Sets the value matches that should be active
Set Filter Matching Fields List
Sets the record filtering criteria for the selected matching field
422 MDM Import Manager Reference Guide
Main Menu Context Menu Description
Set Default Import Action > action
Default Import Actions Grid
Sets the default import action for a Match Level / Match Type group of source records
Set Source Import Action > action
Source Records Grid
Sets the import action for the selected source records
Set Destination Import Action > action
Destination Records Grid
Sets the import action for the selected destination records
Refresh Record Matching
n/a Refreshes the record matching results
Execute Import n/a Executes the import
CONFIGURATION MENU
Main Menu Context Menu Description
Options n/a Changes program options
Change Password n/a Changes the connected MDM user’s repository password
HELP MENU
Main Menu Context Menu Description
About SAP MDM Import Manager…
n/a Displays program information, version, and copyright
PARTITION FIELD/VALUE TAB BUTTONS
Button Context Menu Description
Add n/a Adds the selected item to the list of partitions for the selected tree node (single item selected)
Add > Add as Individual
n/a Adds the selected items as individual partitions (multiple items selected)
Add > Add as Combination
n/a Adds the selected items as a combined partition (multiple items selected)
Remove n/a Removes the selected partition items from the list of partitions for the selected tree node
Combine Partition By List Combines the selected partition items for the selected tree node
Split Partition By List Splits the previously combined partition items for the selected tree node
MDM Import Manager Reference Guide 423
MAP FIELDS/VALUES TAB BUTTONS
Button Context Menu Description
Field Mapping Grids
Automap n/a Automatically maps each source field to a destination field with the same name
Map n/a Maps the selected source field to the selected destination field
Unmap n/a Unmaps the selected fields
Add n/a Adds a new attribute to the destination taxonomy (single taxonomy field)
Add > taxonomy field
n/a Adds a new attribute to the selected destination taxonomy (multiple taxonomy fields)
Value Conversion and Mapping Grids
Automap n/a Automatically maps each source value to destination values that are equivalent
Map n/a Maps the selected source values to the selected destination values
Map > Set Missing Value
n/a Sets the missing destination value in the current language layer (missing value)
Map > Do Not Set Missing Value
n/a Does not set the missing destination value in the current language layer (missing value)
Unmap n/a Unmaps the selected values
Add Destination Values Grid
Adds the selected source values as new destination values (flat destination)
Add > Add as Sibling
n/a Adds the selected source values as siblings of the selected destination value (destination hierarchy)
Add > Add as Child
n/a Adds the selected source values as children of the selected destination value (destination hierarchy)
Add > Add Branch as Sibling
n/a Adds the selected internal source values and all of their children as siblings of the selected destination value (destination hierarchy)
Add > Add Branch as Child
n/a Adds the selected internal source values and all of their children as children of the selected destination value (destination hierarchy)
Remove Destination Values Grid
Removes the selected new destination values
424 MDM Import Manager Reference Guide
MATCH RECORDS TAB BUTTONS
Button Context Menu Description
Add n/a Adds the selected field to the list of matching fields (single field selected)
Add > Add as Individual
n/a Adds the selected fields as individual matching fields (multiple fields selected)
Add > Add as Combination
n/a Adds the selected fields as a matching field combination (multiple fields selected)
Remove n/a Removes the selected fields from the list of matching fields
Combine Matching Fields List
Combines the selected matching fields into a matching field combination
Split Matching Fields List
Splits the selected matching field combination into individual matching fields
MISCELLANEOUS CONTEXT MENUS
Command Context Menu Description
Modify… Set Value Conversions Dialog
Modifies the specified value conversion
MDM Import Manager Reference Guide 425
Display and Format Conventions
The MDM Import Manager is like the heads-up display of a cockpit. It uses color and a variety of syntactical format conventions to efficiently represent and convey a tremendous amount of information about the source and destination data and the current state of the import, as listed in Table 97.
Many of these are standard Windows conventions; others correspond to the functional capabilities of the MDM system.
Table 97. Display and Format Conventions of the MDM Import Manager
Where What Description
Source Hierarchy Pane
Source Hierarchy Tree
field [Join] Field is joined
[To: table.field] Field is joined to table.field.
[From: table.field] Field is joined from table.field
Destination Hierarchy Pane
Destination Hierarchy Tree
Node is a text attribute
Node is a text attribute with no links
Node is a numeric attribute
Node is a numeric attribute with no links
Node is a coupled attribute
Node is a coupled attribute with no links
Source Preview Tab
Tables List
table [Lookup] Table has lookup fields
table <Pivot> Table is a pivot table
table <Reverse Pivot> Table is a reverse pivot table
Records Grid field <table> Field is a lookup from table
Partition Field/Value Tab
Partition Field/Value Tab
field Partition item is the current field
field [value] Partition item is the current field value
field Partition item
field1; field2 Combined partition of field1 by field2
Inherit Partitions are inherited
Inherit Partitions are not inherited
Map Fields/Values Tab
Fields Grids value Field is inheriting the global default value
Source Fields Grid
Field is mapped to a destination field
field [Partition] Field has been partitioned
field <table> Field is a lookup from table
426 MDM Import Manager Reference Guide
Where What Description
Destination Fields Grid
A source field is mapped to the destination field, attribute, or qualifier
F Item is a field
D Item is a promoted lookup display field
Q Item is a qualifier
A Item is an attribute
field [DF] Display field of the current table
lookup field [field1; field2] Lookup consists of multiple display fields
field [lookup field] Promoted display field of lookup field
qualifier <qualified field> Qualifier is associated with qualified field
attribute <taxonomy field> Attribute is associated with taxonomy field
attribute Attribute not linked to selected categories
Values Grids
<NULL> Value is NULL
Map Value mapping is required
Map Value mapping not required nor permitted
Map Value mapping is optional
Source Values Grid
Value does not require mapping and is properly converted
Value is mapped to an existing destination value
Value is mapped to multiple existing destination values
Source value is mapped to a new destination value
Source value is mapped to multiple new destination values
Source value is mapped to multiple destination values, some of which are new
converted value Converted value is inheriting value conversion filters from the field level
converted value Converted value has been overridden with manual edits and conversions
Destination Values Grid
A single source value is mapped to the destination value
Multiple source values are mapped to the destination value
A single source value is mapped to the new destination value
Multiple source values are mapped to the new destination value
[n] Order of mapped destination value
value Existing destination value
value New destination value
MDM Import Manager Reference Guide 427
Where What Description
Delimiter Dialog
Inherit default Field is inheriting default
Inherit default Field is not inheriting default
Inherit default Multiple fields are selected, some inheriting the default and some not
Match Records Tab
Matching Fields List
field Matching field
field1; field2 Matching field combination
field [DF] Matching field is a display field
field [Key] Matching field is the key field
field [All] All value matches active for matching field
field [Eql] Equal value matches active
field [Und] Undefined value matches active
field [Neq] Not Equal value matches active
Match Level Column
None Source records with no matches
Single Source records with single match
Multiple Source records with multiple matches
Match Type Column
None Source records with no matches
Exact Source records with Exact match type
Partial Source records with Partial match type
Conflict Source records with Conflict match type
Record Matching Grids
Active
Exact match record
Partial match record
Conflict match record
428 MDM Import Manager Reference Guide
Glossary
Field mapping A mapping at the field level between a source field and a destination field (or attribute).
Join Combines information from two tables by performing a lookup on every record of the primary table.
Lookup Uses one or more pairs of matching fields from two tables, taking the value of the field for a single record in the first (primary) table to “look up” additional information in a single corresponding record in the second (lookup) table.
Mapping Establishes a one-to-one correspondence between a single source item and a single destination item.
Partition A partition creates a set of value combinations based on the value of one field and all of the corresponding values of a second field (or attribute). A single value for a field can be partitioned, resulting in a single set of value combinations, or an entire field can be partitioned, resulting in a set of value combinations for each value of the partitioned field. Partitions can be inherited from parent nodes in a hierarchy, and for a particular node in the hierarchy, can be nested (which creates hierarchy among value combinations) or combined (which merges value combinations).
Table Rectangular row/column arrangement of related data values. Each horizontal row in the table represents a single record and consists of the same set of fields. Each vertical column of the table represents one field that is stored for each row in the table.
Unique field/ unique field combination
Key field (or compound key field combination) for a record that uniquely identifies the record. A table may have more than one unique field (or field combination).
A field mapping to a unique field requires a value match to detect existing records that match each source record.
Value domain The enumerated set of valid values for a destination text field (or attribute). Value domains exist primarily for lookup fields and text-valued attributes, but can also exist for a non-lookup field in a lookup table or when a non-lookup field has a well-defined set of valid values (such as the two-letter abbreviations for state). A field mapping to a destination with a value domain requires a value mapping.
Value mapping A mapping at the field value level between a particular source value and a particular destination value.