Upload
ibm-new-zealand
View
1.394
Download
1
Tags:
Embed Size (px)
DESCRIPTION
InfoSphere - Leading from the Front - Accelerating Data Integration through Metadata. Presenter: Scott Abbott
Citation preview
Leading from the FrontAccelerating Data Integration through MetadataScott AbbottCertified IT Architect, InfoSphere Software
Make change work for youIBM Insight Forum 09®
C t tContext
Make change work for youIBM Insight Forum 09®
22IBM Insight Forum 09®
Make change work for you
Are you e youconstantly disappointeddisappointed by your Data I t tiIntegration projects?
Make change work for youIBM Insight Forum 09®
Often it’s because we rush in without thinkingthinking what we are d idoing
Make change work for youIBM Insight Forum 09®
Typical Data Integration Project REPORTSTypical Data Integration ProjectOLAP
4WAREHOUSE
DATA INTEGRATIONDATAMARTS
12 3
LEGACYSOURCES
REFERENCE DATA “if we build it they will come”
MASTER DATA
“The custom data model”
“of course our data is good”
“we’ll work it out in the testing”
Make change work for youIBM Insight Forum 09®
Th I f S h S ft E l tiThe InfoSphere Software Evolution
Ch D tDataMirror
LAS Global Name
Change Data Capture
DWLOperational Master Data
Management
Global Name Enrichment
Unicorn
TrigoSRD
Ascential
Transformation, Cleansing, Profiling and metadata integration
Entity Resolution and
Metadata Management
Product Information Management
Entity Resolution and Analysis
Make change work for youIBM Insight Forum 09®
InfoSphere Information Server
Make change work for youIBM Insight Forum 09®
Typical Data Integration Project REPORTSTypical Data Integration ProjectOLAP
4WAREHOUSE
DATA INTEGRATIONDATAMARTS
12 3
LEGACYSOURCES
REFERENCE DATA
MASTER DATA
Make change work for youIBM Insight Forum 09®
METADATA
Pitf ll #1Pitfall #1
“Th C t M d l”“The Custom Model”
Make change work for youIBM Insight Forum 09®
99IBM Insight Forum 09®
Make change work for you
DI Pitfall #1WAREHOUSE
1
“The custom data model”
“ h k i d
data model
NZ Customer Experience
“who knows our industry better than us”
• Project duration 24-36 mths• Model never fully deployed• Complex ETL feeds d t bili d ti BI t“it will only take a couple of
months”
destabilized entire BI system• Users bypass to get required information
Make change work for youIBM Insight Forum 09®
DI Pitfall #1 AcceleratorAccelerator
80:20 rule (20% customization)80:20 rule (20% customization) Months not years
Fully attributed data models across six industries
C l t b i t l t fComplete business templates for industry KPIs
Ke accelerators for migration &Key accelerators for migration & integration projects
A t l ti t l t ithiAct as acceleration templates within Information Server & Cognos 8 BI
Make change work for youIBM Insight Forum 09®
Typical Data Integration Project REPORTSTypical Data Integration ProjectOLAP
4WAREHOUSE
industry
DATA INTEGRATIONDATAMARTS
12 3
LEGACYSOURCES
industry models
REFERENCE DATA
MASTER DATA
Target state
Target state
Make change work for youIBM Insight Forum 09®
METADATA
Pitf ll #2Pitfall #2
if b ild itif we build itthey will come..y
Make change work for youIBM Insight Forum 09®
1313IBM Insight Forum 09®
Make change work for you
14DI Pitfall #2
OLAP
REPORTS
44
“if we build it they will come”
“it is what the business
they will come”
NZ Customer Experience
asked for” • Multiple examples of BI solutions not meeting initial business driversU i BI“the users will understand
the new system”• Users perceive new BI initiatives as burdens rather than assets
Make change work for youIBM Insight Forum 09®
15Missing the PointC t Chi WhiCorporate Chinese Whispers
Identify High Value Customers to support
Call Centre & Web
Monthly Report on Customers Revenue
breakdownCall Centre & Web Personalization
breakdown
DBAsArchitectsSubject Matter Experts
Business Users
DevelopersDataAnalysts
IBM Insight Forum 09®
Make change work for you
16Bridging the Gapl ti th t th ldrelating the new to the old
“item”
“component” “part”?
??
IBM Insight Forum 09®
Make change work for you
Make change work for youIBM Insight Forum 09®
26
Make change work for youIBM Insight Forum 09®
29
U d t di Y D tUnderstanding Your Data
InfoSphere Business Glossary
Captures Business TaxonomiesCaptures and defines shared searchable business glossaryAssigns stewardship to key business termsLinks business terms to technical assets
Make change work for youIBM Insight Forum 09®
InfoSphere Business GlossaryInfoSphere Business GlossaryWeb-based authoring, managing and sharing of business metadataAligns the efforts of IT with the goals of the business Provides business context to
Subject Matter Experts
I f S h B i Gl
Business Users
information technology assetsEstablishes responsibility and accountability
Create and manage business vocabulary and relationships, while
linking to physical sources
InfoSphere Business Glossary
y linking to physical sources
GL Account Database = DB2Number
The ten digit account number. Sometimes referred to as th t ID
Schema = NAACCT
Table = DLYTRANS
C l Technical Business
Business View
the account ID. This value is of the form L-FIIIIVVVV.
Column = ACCT_NO
data type = char(11)
Technical
Make change work for youIBM Insight Forum 09®
Business Glossary Anywhere ANYBusiness Glossary AnywhereReal-time access to business glossary from any desktop application
ANY User
FeaturesFrom any desktop application, click on a term & view its business definition in a pop-up window without any loss of context or focusI t lli t t hi t b t did t i
From Any Application..
.
Intelligent matching returns best candidates in a single searchSearch engine for terms and categoriesAccess steward contact information directlySecurity enforced via the Information Server common security layer
BenefitsIncreased trust and acceptance of information by delivering definitions in contextExpanded adoption of enterprise glossary outside ofExpanded adoption of enterprise glossary outside of Information Platform technologiesImproved information availability with multiple access mechanisms for electronically stored information (ESI)
Pop the Definition!
Typical Data Integration Project REPORTSTypical Data Integration ProjectOLAP
4WAREHOUSE
DATA INTEGRATIONDATAMARTS
12 3
LEGACYSOURCES Correct
REFERENCE DATA
Data Steward
Data Steward
Understood
MASTER DATA
TermsTerms
Target state
Target state
Make change work for youIBM Insight Forum 09®
METADATA
Pitf ll #3Pitfall #3
d t litdata quality
Make change work for youIBM Insight Forum 09®
3636IBM Insight Forum 09®
Make change work for you
DI Pitfall #3
2
LEGACYSOURCES
2
“of course our data is good”
“ h b i h
NZ Customer Experience
“the business owner says the information we need is in there”
• ETL Proof of Concept• Client assured data quality sufficient so
excluded data cleansing from scope• At end of 2wk pilot, project halted due to
unsolvable data quality issues
“the schema’s show they have the same keys”
q y
• Many 15-20 year old systems still in operation in NZ market
Make change work for youIBM Insight Forum 09®
Make change work for youIBM Insight Forum 09®
38
Make change work for youIBM Insight Forum 09®
39
Make change work for youIBM Insight Forum 09®
40
Make change work for youIBM Insight Forum 09®
41
Make change work for youIBM Insight Forum 09®
42
Make change work for youIBM Insight Forum 09®
43
Make change work for youIBM Insight Forum 09®
44
Make change work for youIBM Insight Forum 09®
45
Make change work for youIBM Insight Forum 09®
46
Make change work for youIBM Insight Forum 09®
47
Make change work for youIBM Insight Forum 09®
48
Make change work for youIBM Insight Forum 09®
49
Make change work for youIBM Insight Forum 09®
50
Make change work for youIBM Insight Forum 09®
51
Make change work for youIBM Insight Forum 09®
52
Make change work for youIBM Insight Forum 09®
53
Make change work for youIBM Insight Forum 09®
54
Make change work for youIBM Insight Forum 09®
55
Make change work for youIBM Insight Forum 09®
56
Make change work for youIBM Insight Forum 09®
57
Make change work for youIBM Insight Forum 09®
58
Make change work for youIBM Insight Forum 09®
59
InfoSphere Information AnalyzerInfoSphere Information Analyzer
Data-centric analysis of application, database and file-based sources Data
AnalystsSubject Matter
Experts
Secure, detailed profiling of fields, across fields, and across sources
Analyse source data structures, and monitor adherence to integration and
lit l
InfoSphere Information Analyzer
Creation of metadata from profiling results
Results instantly promotable across
quality rules
Results instantly promotable across IBM InfoSphere Information Server
Physical View
Make change work for youIBM Insight Forum 09®
Typical Data Integration Project REPORTSTypical Data Integration ProjectOLAP
4WAREHOUSE
DATA INTEGRATIONDATAMARTS
12 3
LEGACYSOURCES
Correct
REFERENCE DATA
Data Steward
Data Steward
Understood
MASTER DATA
TermsTerms
Target state
Target stateSource
StateSource State
ETLHints
Make change work for youIBM Insight Forum 09®
METADATA
Pitf ll #4Pitfall #4
It tiIterative Developmentp
Make change work for youIBM Insight Forum 09®
6262IBM Insight Forum 09®
Make change work for you
DI Pitfall #4
DATA INTEGRATION3
“we’ll work it out in the testing”
NZ Customer Experience
• ETL development >75% total project $$P j t t ki 2 3 l th l d• Projects taking 2-3x longer than planned
• Some clients taking 70+% of dev.time doing impact analysis• Impact analysis methods very basic• Largely iterative development method• Unreliable forecast completion dates• Low levels of trust by business in IT ability to achieve BI
outcomes• Substantial cost overruns• Expensive BI maintenance costs
Make change work for youIBM Insight Forum 09®
H d I Fi d O tWhere does the
data for thisHow do I Find Out …Data Analyst
data for this report come
from?
…where this data comes from?
… when the job had been running last time?
… the details for these assets?
IBM Insight Forum 09®
Make change work for you
Pitf ll #4Pitfall #4
D l tDevelopment(Impact Analysis)( p y )
Make change work for youIBM Insight Forum 09®
6565IBM Insight Forum 09®
Make change work for you
Make change work for youIBM Insight Forum 09®
80
What is the InfoSphere Metadata Workbench?What is the InfoSphere Metadata Workbench? Web-based exploration of Information Assets generated and
Data I t ti Developers
gused by Information Server applicationsOut of the box reporting on data
Integration Managers
Developers
Provides IT professionals with a tool for
InfoSphere Metadata Workbench®
p gmovement, data lineage, business meaning, impact of changes and dependencies
Provides IT professionals with a tool for exploring and understanding the assets generated and used by the Information Server suite.
Tracing the data lineage of Business Intelligence Reports to provide basis for compliance with
Slegislation such as Sarbanes-Oxley and Basel II
Typical Data Integration Project REPORTSTypical Data Integration ProjectOLAP
4WAREHOUSE
DATA INTEGRATIONDATAMARTS
12 3
LEGACYSOURCES
Correct
REFERENCE DATA
Data Steward
Data Steward
Understood
MASTER DATA
TermsTermsImpact AnalysisImpact
Analysis
Target state
Target stateSource
StateSource State
ETLHints
Make change work for youIBM Insight Forum 09®
METADATA
Pitf ll #4Pitfall #4
D l tDevelopment(Iterative cycles)( y )
Make change work for youIBM Insight Forum 09®
8989IBM Insight Forum 09®
Make change work for you
Typical Data Integration Project REPORTSTypical Data Integration ProjectOLAP
4WAREHOUSE
DATA INTEGRATIONDATAMARTS
12 3
LEGACYSOURCES
Correct
REFERENCE DATA
Data Steward
Data Steward
UnderstoodRequirements
ETL Code GenerationETL Code
Generation
MASTER DATA
TermsTermsImpact AnalysisImpact
Analysis
Target state
Target stateSource
StateSource State
ETLHints
Make change work for youIBM Insight Forum 09®
METADATA
InfoSphere FastTrack
Business analysts and IT
InfoSphere FastTrackTo reduce costs of integration projects through automation
Business analysts and IT collaborate in context to create project specification
Leverages source analysis
Specification
Leverages source analysis, target models, and metadata to facilitate mapping process
Auto-generation of data transformation jobs and reportsj p
Auto-generates DataStage jobs
Flexible Reporting
Typical Data Integration Project REPORTSTypical Data Integration ProjectOLAP
4WAREHOUSE
DATA INTEGRATIONDATAMARTS
12 3
LEGACYSOURCES
Correct
REFERENCE DATA
Data Steward
Data Steward
UnderstoodRequirements
ETL Code GenerationETL Code
Generation
MASTER DATA
TermsTermsImpact AnalysisImpact
Analysis
Target state
Target stateSource
StateSource State
ETLHints
Make change work for youIBM Insight Forum 09®
METADATA
93Information ServerO ti i i A li ti D l tOptimizing Application Development
IBM Insight Forum 09®
Make change work for you
IBM InfoSphere Information Server94
IBM InfoSphere Information ServerDelivering information you can trust
I f ti SInformation Server
Information Services DirectorInfoSphere
Data Architect
Information AnalyzerInfoSphere
Business GlossaryInfoSphereQualityStageInfoSphere DataStageInfoSphere
Federation ServerInfoSphere
Replication Server / EVPInfoSphereInfoSphere
FastTrackInfoSphere Change Data CaptureInfoSphere
Metadata ServerInfoSphere
Metadata WorkbenchInfoSphere Metadata WorkbenchInfoSphere
Make change work for youIBM Insight Forum 09®
95Bringing It All Togetherg g g
DevelopersSubject Matter Experts
DataAnalysts
Business Users
Architects DBAs
Simplify Integration Increase trust and confidence in informationI li tF ilit t h
Information Server – Common Framework
Increase compliance to standards
Facilitate change management & reuseDesign Operational
IBM Insight Forum 09®
Make change work for you
Leading from the FrontGreater Preparation will yield dramatically lowerGreater Preparation will yield dramatically lower project costs/times
Typical Work Effort for Migration Activities
15-30% of total project budget will be spent on Migration Activities15-30% of total project budget will be spent on Migration Activities15 30% of total project budget will be spent on Migration Activitiesp j g p g
30%Understanding
40%Cleaning, Standardising
30%Conversion, Loading,
DeliverDiscover Prepare
Largely manual effort on small percentage of data. Some manual
This effort is the most unpredictable. The work can vary greatly depending on condition of data, however it is always the largest piece of work in the data initiative.
Largely manual effort on 100% of data. This can mean d f l i t ll t
Coding transformations and loads. Traditionally this effort is plagued with problems related to data quality and it
can easily be pulled by necessity into the
75% Business 50% Business 25% Business
Source Data Harmonizing, Management Interfaces, Connectivity
percentage of data. Some manual coding can review all data . dozens of persons cleaning source systems manually to
correct and augment data and manually aligning records to MRD. Some manual coding can reduce the manual
effort.
can easily be pulled by necessity into the Cleaning, Standardising and Harmonising
area causing timing and budget problems.
75% IT50% IT25% IT
IBM Insight Forum 09®
Make change work for you
97
Th kThank you
Questions?Questions?
IBM Insight Forum 09®
Make change work for you