Upload
dataversity
View
1.110
Download
0
Tags:
Embed Size (px)
Citation preview
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Welcome!
Date: September 11, 2012Time: 2:00 PM ETPresented by: Dr. Peter Aiken
19/11/2012
Let’s Talk Metadata:Strategies and Successes
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Get Social With Us!
Live Twitter FeedJoin the conversation!
Follow us: @datablueprint
@paikenAsk questions and submit your comments: #dataed
2
Like Us on Facebookwww.facebook.com/
datablueprint Post questions and
commentsFind industry news, insightful
content and event updates.
Join the GroupData Management &
Business IntelligenceAsk questions, gain
insights and collaborate with fellow data
management professionals
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Meet Your Presenter: Dr. Peter Aiken
3
#dataed
• Internationally recognized thought-leader in the data management field with more than 30 years of experience
• Recipient of multiple international awards• Founding Director of Data Blueprint
(http://datablueprint.com)
• Associate Professor of Information Systems at Virginia Commonwealth University (http://vcu.edu)
• President of DAMA International (http://dama.org)
• DoD Computer Scientist, Reverse Engineering Program Manager/Office of the Chief Information Officer
• Visiting Scientist, Software Engineering Institute/Carnegie Mellon University
• 7 books and dozens of articles• Experienced w/ 500+ data management
practices in 20 countries
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060 EDUCATION
Let’s Talk Metadata:Strategies and
Successes
Let’s Talk Metadata: Strategies and Successes9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview2. What is metadata and why is it
important?3. Major metadata types & subject areas4. Metadata benefits, application &
sources5. Metadata strategies & implementation6. Metadata building blocks7. Guiding Principles8. Specific examples9. Take Aways, References and Q&A
7
Tweeting now: #dataed
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
The DAMA Guide to the Data Management Body of Knowledge
8
Data Management
Functions
• Published by DAMA International– The professional
association for Data Managers (40 chapters worldwide)
• DMBoK organized around – Primary data
management functions focused around data delivery to the organization
– Organized around several environmental elements
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
The DAMA Guide to the Data Management Body of Knowledge
9
Environmental Elements
Amazon:
http://www.amazon.com/DAMA-Guide-Management-Knowledge-DAMA-DMBOK/dp/0977140083Or enter the terms "dama dm bok" at the Amazon search engine
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Data Management
109/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Data Management
11
Manage data coherently.
Share data across boundaries.
Assign responsibilities for data.Engineer data delivery systems.
Maintain data availability.
Data Program Coordination
Organizational Data Integration
Data Stewardship
Data Development
Data Support Operations
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
StandardData
Organizational DM Functions and their Inter-relationships
Data Program Coordination
OrganizationalData Integration
DataStewardship
Data SupportOperations
Data Asset Use
Organizational Strategies
Goals
IntegratedModels
BusinessData
Business Value
Application Models & Designs
Feedback
Implementation
Direction
DataDevelopment
Guidance
Leverage data in organizational activities
Data management processes andinfrastructure
Combining multipleassets to produceextra value
Organizational-entity subject area dataintegration
Provide reliable access to data
Achieve sharing of data within a business area
10
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata Management
13
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview2. What is metadata and why is it
important?3. Major metadata types & subject areas4. Metadata benefits, application &
sources5. Metadata strategies & implementation6. Metadata building blocks7. Guiding Principles8. Specific examples9. Take Aways, References and Q&A
14
Tweeting now: #dataed
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Meta-data or metadata• In the history of language, whenever two words
are pasted together to form a combined concept initially, a hyphen links them.
• With the passage of time, the hyphen is lost. The argument can be made that that time has passed.
• There is a copyright on the term "metadata," but it has not been enforced.
• So, term is "metadata"
159/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
DefinitionsMetadata is …• … everywhere in every data management activity and integral
to all IT systems and applications.• … to data what data is to real life. Data reflects real life transactions,
events, objects, relationships, etc. Metadata reflects data transactions, events, objects, relations, etc.
• … the data that describe the structure and workings of an organization’s use of information, and which describe the systems it uses to manage that information. [quote from David Hay's new book, page 4]
• Data describing various facets of a data asset, for the purpose of improving its usability throughout its life cycle [Gartner 2010]
• Metadata unlocks the value of data, and therefore requires management attention [Gartner 2010]
Metadata Management is …• … the set of processes that ensure proper creation, storage,
integration, and control to support associated use of metadata
16
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Analogy: Card catalog in a library • Card catalog identifies what books
are stored in the library and where they are located in the building
• Users can search for books by subject area, author, or title
• Catalog shows author, subject tags, publication date and revision history of each book
• Card catalog information helps determine which books will meet the reader’s needs
• Without this catalog resource, finding books in the library would be difficult, time consuming and frustrating
• Readers may search many incorrect books before finding the right book if a catalog does not exist
17
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Definition (continued)• Metadata is the card catalog in a
managed data environment• Abstractly, Metadata is the descriptive
tags or context on the data (the content) in a managed data environment
• Metadata shows business and technical users where to find information in data repositories
• Metadata provides details on where the data came from, how it got there, any transformations, and its level of quality
• Metadata provides assistance with what the data really means and how to interpret it
18
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Defining Metadata
Metadata is any combination of any circle and the data in the center that unlocks the value of the data!
19
Adapted from Brad Melton
Data
WhereWhy
What How
Who
When
Data
9/11/2012
Library
WhereWhy
What How
Who
When
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Who: AuthorWhat: Title Where: Shelf LocationWhen: Publication Date
A small amount of metadata (Card Catalog) unlocks the value of a large amount of data (the Library)
20
Library Metadata ExampleLibraries can operate efficiently through careful use of metadata (Card Catalog)
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
21
Data
WhereWhy
How
Who
When
EmailMessages
What
Outlook Example
"Outlook" metadata is used to navigate and manage emailImagine how managing e-mail (already non-trivial) would change if Outlook did not make use of metadata
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Who: "To" & "From"What: "Subject" How: "Priority"Where: "USERID/Inbox", "USERID/Personal"Why: "Body"When: "Sent" & "Received”• Find the important stuff/weed
out junk • Organize for future access/
outlook rules
22
Outlook Example, continued
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata practices connect data sources and uses in an organized and efficient manner
• What is the structure of metadata practices?– Storage: repository, glossary, models, lineage - currently multiple technologies
are used– Engineering: identifying/harvesting/normalizing/administer evolving metadata
structures– Delivery: supply/access/portal/definition/lookup search identify/ensure required
metadata supplies to meet business needs– Governance: ensure proper/creation/storage/integration/control to support
effective use
• When executed, engineering and delivery implement governance
23
SourcesMetadata Governance
Metadata Engineering
Metadata Delivery Uses
Metadata Practices
MetadataStorage
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Organized Knowledge 'Data'
Improved Quality Data
Data OrganizaKon PracKces
Metadata Practices will be inextricably intertwined with Data Quality and Master Data and Knowledge Management, (among other functions)
OperaKonal Data
Data Quality Engineering
Master Data ManagementPracKces
Suspected/IdenKfied Data
Quality Problems
RouKne Data Scans
Master Data Catalogs
RouKne Data Scans
KnowledgeManagementPracKces
ExtracKonSources
Data that might benefit from Master Management
249/11/2012
Sources( (Metadata(Governance(
(
Metadata(Engineering(
(
Metadata(Delivery( Uses(
Metadata(Prac8ces((dashed lines not in existence)
Metadata(Storage(
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata History 1990-2008The history of Metadata management tools and products seems to be a metaphor for the lack of a methodological approach to enterprise information management:• Lack of standards and proprietary nature of most managed
Metadata solutions cause many organizations to avoid focusing on metadata
• This limits organizations’ ability to develop a true enterprise information management environment
• Increased attention given to information and its importance to an organization’s operations and decision-making will drive Metadata management products and solutions to become more standardized
• More recognition to the need for a methodological approach to managing information and metadata
259/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata History: The 1990s• Business managers began to recognize the
value of Metadata repositories• Newer tools expanded the scope• Potential benefits identified during this period
include:– Providing semantic layer between company’s system
and business users– Reducing training costs– Making strategic information more valuable as aid in
decision making– Creating actionable information– Limiting incorrect decisions
269/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata History: Mid-to late 1990s• Metadata becomes more relevant to corporations who were
struggling to understand their information resources caused by: – Y2K deadline– Emerging data warehousing initiatives – Growing focus around the World Wide Web
• Beginning of efforts to try to standardize Metadata definition and exchange between applications in the enterprise
• Examples of standardization:– 1995: CASE Definition Interchange Facility (CDIF) – 1995: Dublin Core Metadata Elements– 1994 – 1999: First parts of ISO 11179 standard for Specification and
Standardization of Data Elements were published– 1998: Common Warehouse Metadata Model (CWM)– 1995: Metadata Coalitions’ (MDC) Open Information Model – 2000: Both standards merged into CSM. Many Metadata repositories
began promising adoption of CWM standard
279/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata History: 21st Century• Update of existing Metadata repositories for deployment on the
web• Introduction of products to support CWM• Vendors begin focusing on Metadata as an additional product
offering• Few organizations purchase or develop Metadata repositories• Effective enterprise-wide Managed Metadata Environments are
rare due to:– Scarcity of people with real world skills– Difficulty of the effort– Less than stellar success of some of the initial efforts at some
companies– Stagnation of the tool market after the initial burst of interest in late 90s– Still less than universal understanding of the business benefits– Too heavy emphasis on legacy applications and technical metadata
289/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Polling Question #1
What have been the driving factors in focusing on metadata within the last decade?
a. Recent entry of smaller vendors into the marketb. Challenges related to addressing regulatory requirementsc. Declination to the existing Metadata standards
299/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata History: Current Decade• Focus on need for and importance of metadata• Focus on how to incorporate Metadata beyond
traditional structured sources and include semistructured sources
• Driving factors:– Recent entry of larger vendors into the market– Challenges related to addressing regulatory requirements, e.g.
Sarbanes-Oxley, and privacy requirements with unsophisticated tools
– Emergence of enterprise-wide initiatives, e.g. information governance, compliance, enterprise architecture, automated software reuse
– Improvements to the existing Metadata standards, e.g. RFP release of new OMG standard Information Management Metamodel (IMM), which will replace CWM
– Recognition at the highest levels that information is an asset that must be actively and effectively managed
309/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview2. What is metadata and why is it
important?3. Major metadata types & subject areas4. Metadata benefits, application &
sources5. Metadata strategies & implementation6. Metadata building blocks7. Guiding Principles8. Specific examples9. Take Aways, References and Q&A
31
Tweeting now: #dataed
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Types of Metadata: Process Metadata
• Process Metadata is...– Data that defines and describes the characteristics of
other system elements, e.g. processes, business rules, programs, jobs, tools, etc.
• Examples of Process metadata:– Data stores and data involved– Government/regulatory bodies– Organization owners and stakeholders– Process dependencies and decomposition– Process feedback loop and documentation– Process name
32
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Business Process Metadata
Who: Created the documentation?
What: Are the important dependencies among the processes?
How: Do the business processes interact with each other?
33
Data
WhereWhy
What How
Who
When
Data
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Types of Metadata: Business Metadata• Business Metadata describe
to the end user what data are available, what they mean and how to retrieve them.
• Included are:– Business names and definitions of subject and
concept areas, entities, attributes– Attribute data types and other attribute properties– Range descriptions, calculations, algorithms and
business rules– Valid domain values and their definitions
34
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Types of Metadata: Technical & Operational Metadata
35
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
• Technical and operational metadata provides developers and technical users with information about their systems
• Technical metadata includes…– Physical database table and column names, column properties, other
properties, other database object properties and database storage
• Operational metadata is targeted at IT operations users’ needs, including…– Information about data movement, source and target systems, batch
programs, job frequency, schedule anomalies, recovery and backup information, archive rules and usage
• Examples of Technical & Operational metadata:– Audit controls and balancing information– Data archiving and retention rules– Encoding/reference table conversions– History of extracts and results
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Types of Metadata: Data Stewardship
36
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
• Data stewardship Metadata is about...– Data stewards, stewardship processes, and responsibility
assignments
• Data stewards…– Assure that data and Metadata are accurate, with high
quality across the enterprise. – Establish and monitor data sharing.
• Examples of Data stewardship metadata:– Business drivers/goals– Data CRUD rules– Data definitions – business and technical– Data owners– Data sharing rules and agreements/contracts– Data stewards, roles and responsibilities
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Types of Metadata: Provenance
37
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
• Provenance: – the history of ownership of a valued object or
work of art or literature" [Merriam Webster]– For each datum, this is the description of:
• Its source (system or person or department), • Any derivation used, and • The date it was created.
– Examples of Data Provenance:• The programs or processes by which it was created• Its owner• The steward responsible for its quality• Other roles and responsibilities• Rules for sharing it.
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata Subject Areas
3809/10/12
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
Subject Areas Components
1) Business Analytics Data definitions, reports, users, usage, performance
2) Business Architecture Roles and organizations, goals and objectives
3) Business Definitions Business terms and explanations for a particular concept, fact, or other item found in an organization
4) Business Rules Standard calculations and derivation methods
5) Data Governance Policies, standards, procedures, programs, roles, organizations, stewardship assignments
6) Data Integration Sources, targets, transformations, lineage, ETL workflows, EAI, EII, migration/conversion
7) Data Quality Defects, metrics, ratings
8) Document Content Management
Unstructured data, documents, taxonomies, ontologies, name sets, legal discovery, search engine indexes
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata Subject Areas, continued
3909/10/12
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
Subject Areas Components
9) Information Technology Infrastructure Platforms, networks, configurations, licenses
10) Conceptual data models Entities, attributes, relationships and rules, business names and definitions.
11) Logical Data Models Files, tables, columns, views, business definitions, indexes, usage, performance, change management
12) Process Models Functions, activities, roles, inputs/outputs, workflow, timing, stores
13) Systems Portfolio and IT Governance
Databases, applications, projects, and programs, integration roadmap, change management
14) Service-oriented Architecture (SOA) information:
Components, services, messages, master data
15) System Design and Development Requirements, designs and test plans, impact
16) Systems Management Data security, licenses, configuration, reliability, service levels
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview2. What is metadata and why is it
important?3. Major metadata types & subject areas4. Metadata benefits, application &
sources5. Metadata strategies & implementation6. Metadata building blocks7. Guiding Principles8. Specific examples9. Take Aways, References and Q&A
40
Tweeting now: #dataed
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Benefits of Metadata
41
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
1) Increase the value of strategic information (e.g. data warehousing, CRM, SCM, etc.) by providing context for the data, thus aiding analysts in making more effective decisions.
2) Reduce training costs and lower the impact of staff turnover through thorough documentation of data context, history, and origin.
3) Reduce data-oriented research time by assisting business analysts in finding the information they need in a timely manner.
4) Improve communication by bridging the gap between business users and IT professionals, leveraging work done by other teams and increasing confidence in IT system data.
5) Increased speed of system development’s time-to-market by reducing system development life-cycle time.
6) Reduce risk of project failure through better impact analysis at various levels during change management.
7) Identify and reduce redundant data and processes, thereby reducing rework and use of redundant, out-of-data, or incorrect
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata for Semistructured Data
42
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
• Unstructured data = any data that is not in a database or data file, including documents or other media data
• Metadata describes both structured and unstructured data• Metadata for unstructured data exists in many formats, responding
to a variety of different requirements• Examples of Metadata repositories describing unstructured data:
– Content management applications– University websites– Company intranet sites– Data archives– Electronic journals collections– Community resource lists
• Common method for classifying Metadata in unstructured sources is to describe them as descriptive metadata, structural metadata, or administrative metadata
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Polling Question #2
Which is not an example of descriptive metadata?
a. Catalog informationb. Thesauri keyword termsc. Thesauri keyword labels
439/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata for Unstructured Data: ExamplesExamples of descriptive metadata:• Catalog information• Thesauri keyword terms
Examples of structural metadata• Dublin Core• Field structures• Format (audio/visual, booklet)• Thesauri keyword labels• XML schemas
44
Examples of administrative metadata• Source(s)• Integration/update schedule• Access rights• Page relationships (e.g. site
navigational design)
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Sources of MetadataPrimary Sources:• Virtually anything named in an organization
Secondary sources: • Other Metadata repositories, accessed using
bridge software• CASE tools, ETL tools
Many data management tools create and use repositories for their own use.
459/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Specific ExampleFour metadata sources:
1.Existing reference models (i.e., ADRM)
2.Conceptual model created two years ago
3.Existing systems (to be reverse engineered)
4.Enterprise data model
46
ADRM}9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview2. What is metadata and why is it
important?3. Major metadata types & subject areas4. Metadata benefits, application &
sources5. Metadata strategies & implementation6. Metadata building blocks7. Guiding Principles8. Specific examples9. Take Aways, References and Q&A
47
Tweeting now: #dataed
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata Strategy • Metadata Strategy is…
– … a statement of direction in Metadata management by the enterprise
– … a statement of intend that acts as a reference framework for the development teams
– …driven by business objectives and prioritized by the business value they bring to the organization
• Build a Metadata strategy from a set of defined components• Primary focus of Metadata strategy: gain an understanding of
and consensus on the organization’s key business drivers, issues, and information requirements for the enterprise Metadata program
• Need to understand how well the current environment meets these requirements now and in the future
• Metadata strategy objectives define the organization’s future enterprise Metadata architecture and recommend logical progression of phased implementation steps
489/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
49
Metadata Strategy Implementation Phases
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview2. What is metadata and why is it
important?3. Major metadata types & subject areas4. Metadata benefits, application &
sources5. Metadata strategies & implementation6. Metadata building blocks7. Guiding Principles8. Specific examples9. Take Aways, References and Q&A
50
Tweeting now: #dataed
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Metadata Management
51
üü ü üü ü ü
üü ü üü ü ü
üü ü üü ü ü
üü ü üü ü ü
üü ü üü ü üüü ü üü ü ü
üü ü üü ü ü
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
üü ü üü ü ü
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Goals and Principles
52
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
1. Provide organizational understanding of terms and usage
2. Integrate Metadata from diverse sources
3. Provide easy, integrated access to metadata
4. Ensure Metadata quality and security
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Activities1) Understand Metadata requirements2) Define the Metadata architecture3) Develop and maintain Metadata standards4) Implement a managed Metadata
environment5) Create and maintain metadata6) Integrate metadata7) Management Metadata repositories8) Distribute and deliver metadata9) Query, report and analyze metadata
53
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Activities: Metadata Standards Types
54
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
• Two major types exist:1) Industry or consensus
standards2) International standards
• High level framework shows how standards are related and how they rely on each other for context and usage:
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Common Warehouse Metadata (CWM):• Specifies the interchange of Metadata among data warehousing, BI, KM,
and portal technologies.• Based on UML and depends on it to represent object-oriented data
constructs.
The CWM Metamodel
55
Activities: Noteworthy Metadata Standards Types
Warehouse ProcessWarehouse ProcessWarehouse Process Warehouse Opera=onWarehouse Opera=onWarehouse Opera=on
TransformaKonTransformaKon OLAP Data Mining
InformaKon VisualizaKon
Business Nomenclature
Object Model
RelaKonal Record MulKdimensionalMulKdimensional XML
Business InformaKon Data Types Expression Keys and
IndexesType
MappingSoTware
Deployment
Object ModelObject ModelObject ModelObject ModelObject ModelObject Model
Management
Analysis
Resource
FoundaKon
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Information Management Metamodel (IMM)• Object Management Group Project to replace CWM• Concerned with:
– Business Modeling• Entity/relationship metamodel
– Technology modeling• Relational Databases• XML• LDAP
– Model Management• Traceability
– Compatibility with related models• Semantics of business vocabulary and business rules• Ontology Definition Metamodel
569/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
The Information Management Metamodel...
57
• Based on Core model.• Used to translate from one model to another.
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Primary Deliverables• Metadata repositories
• Quality metadata
• Metadata analysis
• Data lineage
• Change impact analysis
• Metadata control procedures
• Metadata models and architecture
• Metadata management operational analysis
58
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Roles and Responsibilities
59
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
Suppliers:– Data Stewards– Data Architects– Data Modelers– Database Administrators– Other Data Professionals– Data Brokers– Government and Industry
Regulators
Participants:– Metadata Specialists– Data Integration Architects– Data Stewards– Data Architects and Modelers– Database Administrators– Other DM Professionals– Other IT Professionals– DM Executives– Business Users
Consumers:• Data Stewards• Data Professionals• Other IT Professionals• Knowledge Workers• Managers and Executives• Customers and Collaborators• Business Users
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Technology• Metadata repositories• Data modeling tools• Database management systems• Data integration tools• Business intelligence tools• System management tools• Object modeling tools• Process modeling tools• Report generating tools• Data quality tools• Data development and administration tools• Reference and mater data management tools
60
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview2. What is metadata and why is it
important?3. Major metadata types & subject areas4. Metadata benefits, application &
sources5. Metadata strategies & implementation6. Metadata building blocks7. Guiding Principles8. Specific examples9. Take Aways, References and Q&A
61
Tweeting now: #dataed
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Guiding Principles1) Establish and maintain a Metadata strategy and
appropriate policies, especially clear goals and objectives for Metadata management and usage
2) Secure sustained commitment, funding, and vocal support from senior management concerning Metadata management for the enterprise
3) Take an enterprise perspective to ensure future extensibility, but implement through iterative and incremental delivery
4) Develop a Metadata strategy before evaluating, purchasing, and installing Metadata management products
5) Create or adopt Metadata standards to ensure interoperability of Metadata across the enterprise
6) Ensure effective Metadata acquisition for internal and external metadata
7) Maximize user access since a solution that is not accessed or is under-accessed will not show business value
62
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Guiding Principles, continued
63
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
9/11/2012
8) Understand and communicate the necessity of Metadata and the purpose of each type of metadata; socialization of the value of Metadata will encourage business usage
9) Measure content and usage10) Leverage XML, messaging and web services11) Establish and maintain enterprise-wide business involvement
in data stewardship, assigning accountability for metadata12) Define and monitor procedures and processes to ensure
correct policy implementation13) Include a focus on roles, staffing,
standards, procedures, training, & metrics14) Provide dedicated Metadata experts
to the project and beyond15) Certify Metadata quality
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview2. What is metadata and why is it
important?3. Major metadata types & subject areas4. Metadata benefits, application &
sources5. Metadata strategies & implementation6. Metadata building blocks7. Guiding Principles8. Specific examples9. Take Aways, References and Q&A
64
Tweeting now: #dataed
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
6609/10/12
Example: iTunes Metadata
• Example: – iTunes Metadata
• Insert a recently purchased CD
• iTunes can:– Count the number
of tracks (25)– Determine the
length of each track
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
6709/10/12
Example: iTunes Metadata
• When connected to the Internet iTunes connects to the Gracenote(.com) Media Database and retrieves:– CD Name– Artist– Track Names– Genre– Artwork
• Sure would be a pain to type in all this information
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
6809/10/12
Example: iTunes Metadata
• To organize iTunes – I create a
"New Smart Playlist" for Artist's containing "Miles Davis"
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Example: iTunes Metadata
6909/10/12
• Notice I didn't get the desired results
• I already had another Miles Davis recording in iTunes
• Must fine-tune the request to get the desired results– Album contains "The
complete birth of the cool"
• Now I can move the playlist "Miles Davis" to a folder
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Example: iTunes Metadata
7009/10/12
• The same: – Interface– Processing– Data Structures
• are applied to – Podcasts– Movies– Books– .pdf files
• Economies of scale are enormous
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Outline
1. Data Management Overview2. What is metadata and why is it
important?3. Major metadata types & subject areas4. Metadata benefits, application &
sources5. Metadata strategies & implementation6. Metadata building blocks7. Guiding Principles8. Specific examples9. Take Aways, References and Q&A
71
Tweeting now: #dataed
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Summary
721/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
References & Recommended Reading
73
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
References, cont’d
74
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
References, cont’d
75
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
References, cont’d
76
from The DAMA Guide to the Data Management Body of Knowledge © 2009 by DAMA International
1/26/2010 © Copyright this and previous years by Data Blueprint - all rights reserved!
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Questions?
77
It’s your turn! Use the chat feature or Twitter (#dataed) to submit
your questions to Peter now.
+ =
9/11/2012
TITLE
PRODUCED BY
DATA BLUEPRINT 10124-C W. BROAD ST, GLEN ALLEN, VA 23060
CLASSIFICATION
EDUCATIONDATE SLIDE
09/10/12 © Copyright this and previous years by Data Blueprint - all rights reserved!
Upcoming Events
78
October Webinar:Engineering Solutions to Data Quality ChallengesOctober 9, 2012 @ 2:00 PM – 3:30 PM ET(11:00 AM-12:30 PM PT)
November Webinar:Get the Most Out of Your Tools: Data Management TechnologiesNovember 13, 2012 @ 2:00 PM – 3:30 PM ET(11:00 AM-12:30 PM PT)
Sign up here:•www.datablueprint.com/webinar-schedule •www.Dataversity.net
Brought to you by:
9/11/2012