Moving Towards Enterprise Data Architecture

Moving towards Enterprise Data Architecture

Kevin Messer, Enterprise Data Architect

Standard Register

• Standard Register (HQ: Dayton, Ohio; NYSE: SR) helps the world‟s leading organizations build and protect their reputations through expert management and precise execution of critical communications

> Secure, confidential print

> Industrial labels and logos

> Fraud prevention solutions

> MSRA-resistant armbands

> www.standardregister.com

• Kevin Messer (Presenter)

> SR Enterprise Data Architect

> Prior: Lockheed, Accenture, NCR

> www.linkedin.com/in/kevinmesser

About . . .

Moving towards Enterprise Data Architecture 2

> Multichannel marketing

> Print outsourcing

> eLearning

> NFC-enabled pokenHEALTH

Background: Improved Data Management

Moving towards Enterprise Data Architecture 3

• New initiatives

> Complex integrations

> Heterogeneous source systems

> How to avoid data duplication?

> Reference sources for data quality?

• Systems documentation challenges

> Multiple repositories

> Variety of documentation tools

> Some documentation ≠ data

> Budget and resource constraints

• Options?

Master Data Management?

Moving towards Enterprise Data Architecture 4

• Brief survey of products > Informatica, Microsoft, Oracle

> Software AG, Talend, Teradata

> Etc, etc, etc.

• Many Challenges > Large upfront cost (>$1M)

> Hidden costs - Internal staffing, training

- Process and culture changes

- How to measure Return on Investment and Total Costs?

- Where to start?

• Lower cost alternatives? > Start small, grow scope incrementally

> Every plan needs an architecture . . .

Simple Master Data Architecture

Moving towards Enterprise Data Architecture 5

Enterprise Data Data structured for reporting and

analysis, based on Reference,

Master, BU data

Business Unit Data Business Unit variations of Master

Data, used for specific markets

and transactions

Master Data Data consistently used by the

business: customer; product

Reference Data External data used by reference:

zip codes; area codes; state; city;

country

Metadata Data about data: source, formats,

business rules

This simple

diagram for

Master Data

Architecture is

easy for people to

remember and

was the basis for

the next steps,

starting with

metadata. . .

Teradata Meta Data Services (MDS)

Moving towards Enterprise Data Architecture 6

• Part of Teradata Utility Pack

• Already licensed, so essentially “free” > No client or seat licenses

• Very low-cost implementation > Windows client utility for

administration

> Microsoft IIS Server (we used laptops)

> Web browser interface (MetaSurf) for users

• Did I mention it was FREE?

Teradata MDS Architecture

Moving towards Enterprise Data Architecture 7

• MDS Repository is a Teradata database

• Versions 13 to V2R6.0

• API supports C++, Microsoft COM, XML, Java, and Excel

• ERwin® Data Modeler r7 includes a wizard

• CWM XMI for Metadata Interchange using other tools

• Supports storing metadata from other DBMS

Teradata

MDS

Repository

MDS Utilities Custom

Applications BI Tools

XML Bridge

MDS Engine

MDS APIs

Auto

DIM

Update

Diagram from “Discover Teradata Meta Data Services” by

Rudy Ezquerro, Copyright 2007 by Teradata Corporation.

Building a “Master Data Catalog”

Moving towards Enterprise Data Architecture 8

• Started “proof of concept” effort in early 2011

• Objectives:

> Use MDS as a central “catalog” of all database metadata

- Capture 100% of Teradata database schemas

- Capture 100% of SQL-compliant non-Teradata sources

- Explore approaches for non-SQL databases

> Evaluate use of MDS as central catalog/portal for both technical and business process documentation

• The slides that follow describe the steps we took and the outcomes

First Steps: Teradata Metadata

Moving towards Enterprise Data Architecture 9

• Capturing from Teradata

> We used QA databases

> Using MetaManager, connect to Teradata

> Select the databases to load

• This approach worked without difficulty

Second Step: Importing non-Teradata Metadata

Moving towards Enterprise Data Architecture 10

• Our original plan:

> Using ERwin, reverse-engineer the databases

> Export schemas from Erwin as XML (CWM XMI) files

> Use MetaBridge to import the XMI

• Software compatibility issues between Erwin and our version of MDS emerged, forcing a different approach

Alternate approach to non-Teradata Metadata

Moving towards Enterprise Data Architecture 11

• The alternate approach used Excel

> Using the MDS API Interfaces

• Microsoft Excel Macros are included with MDS Development Kit sample files

• Macros include loading descriptions for:

> Databases, tables, columns, views

> Business attributes, entities, subject areas

> Macros can be modified

• We worked with tools created by Dan Riehle

Examples

Moving towards Enterprise Data Architecture 12

• Oracle

> Created a query joining All_Tab_Columns and all_tables system views

• SQL Server

> Used INFORMATION_Schema.columns system view

• Supra II (mainframe)

> Flatten the DDL export

• Export the DDL information to Excel format

Important Tip

Moving towards Enterprise Data Architecture 13

• MetaSurf (web browser viewer of MDS) only displays Teradata data types

• When exporting DDL from non-Teradata databases, data types must be converted

• Example

> VARCHAR converted to a code of „CV‟ to display*

*Note: Teradata column type abbreviations are found in the Teradata RDBMS Data Dictionary manual.

MetaSurf Examples

Moving towards Enterprise Data Architecture 14

• Example views of metadata from an Oracle database

Lessons Learned about Teradata MDS

Moving towards Enterprise Data Architecture 15

• MDS is a powerful, flexible tool for > Quickly capturing database technical information

> Providing this database information centrally to - Database Administrators, ETL and Interface Programmers,

Support Staff

> Searching across multiple databases for similar tables, columns, attributes, etc

• MDS is also extensible, allowing it to serve as a portal to information related to databases > Data Model diagrams

> Business Process diagrams and descriptions

> Links to documentation stored in other repositories like SharePoint, Subversion, or the internet

• MDS provides a low-cost way to incrementally implement Enterprise Data Architecture and Master Data Management

Next Steps

Moving towards Enterprise Data Architecture 16

• Plans include:

> Implementing MDS in production

- Staffing

- Support

- Governance

> Position MDS as the company‟s Master Data Catalog

- Single site for documentation about data and systems

- Content to include links to documents in other repositories

Questions?

Moving towards Enterprise Data Architecture 17

?

Kevin Messer

Dan Riehle Dan@GetReals.com

www.getreals.com