DataClairvoyance - DAMA Indiana · DataClairvoyance A business approach to data. Quantifying Trust & Confidence of Business Critical Data Designing, Enabling and Growing a Data Centric

DataClairvoyanceA business approach to data.

Quantifying Trust & Confidence of Business Critical

Data

Designing, Enabling and

Growing a Data Centric Culture

Real transparency into shared data responsibilities

A professional services firm that provides a very unique and holistic approach that enables your organization to be successful in traversing the data challenges of today.

Real data practitioners, delivering real improvements to your enterprise data assets.

Agenda

• Data Quality Solution Overview• Human Resource Requirements• Technology Requirements• Example Deliverables• Advanced Solution

4/15/2015 4

Agenda

All rights reserved by Data Clairvoyance Group ©2015.

Data Quality Solution Overview

Data Quality Solution Overview• Data represents the data sources to be analyzed

• Profile Data represents the profiling tool used to infer metadata directly from the data content

• Data Quality Assessment represents the process utilized to analyze the data content

• Monitoring Rules represent the code created to validate the data content over time

• Data Quality Report represents the reports created to validate the data content over time

• Remediation Requirements represents the specifications for correcting the identified data quality problems in the data content

4/15/2015 6All rights reserved by Data Clairvoyance Group ©2015.

Detailed DQ Solution Overview• Data is identified by the flat file and relational data source

and the connections and metadata required• Profile Data represents the profiling tool used to infer

metadata directly from the data content and the inferred metadata produced by the tool

• Data Quality Assessment represents the process utilized to analyze the data content and the review of the assessment results by the lead data quality analyst

• Monitoring Rules represent the code created to validate the data content over time that are reviewed with the subject matter experts and/or the business

• Data Quality Report represents the reports created to validate the data content over time by the subject matter experts and/or the business

• Remediation Requirements represents the specifications for correcting the identified data quality problems


Data Sources

• Flat file data sources that are normalized with a single record type

• The layout for the flat file is required by the profiling tool in order to analyze the data properly

• Relational data sources are databases that contain the data to be analyzed

• Connections are required to the relational database in order to analyze the data content


Data Profiles

• The solution is software agnostic but does require a profiling tool

• The profiling tool analyzes the data content to infer metadata

• The inferred metadata is copied into the assessment template for each attribute in the table/file

• The inferred metadata includes the number of distinct values, null rule, inferred data type, range, value frequencies, and pattern frequencies


Data Quality Assessment

• A step‐by‐step process for analyzing the inferred metadata by column to identify anomalies and/or inconsistencies in the data content

• The inferred metadata is copied from the profiling tool into the assessment template

• The anomalies and/or inconsistencies are categorized as being invalid, inconsistent, duplicate content, mixed content, and/or questionable content

• The assessment results are reviewed and validated by the lead data quality analyst, who also suggests monitoring rules specific to each attribute


DQ Assessment SME Review

• The categorized anomalies are reviewed with the subject matter experts and the business

• The subject matter experts and the business determine if the identified problems are valid or not

• Notes are captured that identify business or data rules specific to the content

• The suggested monitoring rules are also reviewed and validated for coding purposes

• Validated data quality problems are identified for remediation


Data Quality Monitoring Rules• Monitoring rules are suggested by the lead data quality analyst

• Additional rules are identified during the assessment review session

• The rules are coded and tested

• The results of the rules are reviewed with the subject matter expert and the business

• Additional rules and changes to the initial rules are identified and implemented

• The review process is repeated until the subject matter experts and the business are satisfied with the rules

• The subject matter experts and the business sign off on the rules

• The DQ analysts finalize the monitoring rules by creating the appropriate documentation

• The finalized rules are then deployed to begin monitoring the quality of the data over time

• The results of the monitoring rules are also utilized to identify data content for remediation


Data Quality Reports

• Reports are created from the results of the monitoring rules

• Thresholds are set for the rules based on the importance and frequency of change for the data with the help from the subject matter experts and the business

• The reports are scheduled to run based on input from the subject matter experts and the business

• The reports are then deployed to automatically run on a regular basis

• The generated reports are validated by the data quality analysts prior to forwarding to the subject matter experts and the business for review


Data Quality Closed‐Loop Solution• The last step in the process ensures that the quality of both the data content and monitoring rules are maintained over time

• This is accomplished by reviewing the data quality reports and determining if the data problems identified are valid or if the monitoring rules need to be updated to reflect changes in the business and the data content

• Remediation requirements are created for the anomalies identified as true data quality problems

• Changes to the monitoring rules are identified and implemented when the anomalies turn out to be valid data content

• This creates a closed‐loop solution


Human Resource Requirements

Data Quality Team• The Data Quality Project Manager tracks the completion of

the data quality process across data sources and maintains the project plan

• The Lead Data Quality Analyst assigns and manages the work for the administrator and the senior analysts, and also schedules and leads the review meetings with the subject matter experts and the business for the assessments and rule creation

• The Senior Data Quality Analyst distributes the work across the analyst and junior analysts, and performs data quality assessment and creates monitoring rules

• The Data Quality Analyst is responsible for performing assessments and creating monitoring rules

• The Junior Data Quality Analyst is responsible for performing assessments and creating monitoring rules

• The Data Quality Administrator is responsible for gathering metadata for the flat file data sources and creating the connections for the relational data sources


Subject Matter Experts

• The Data Stewards are usually business resources that have expert knowledge about specific subject areas and related data content

• The Data Custodians are usually information technology resources that have expert knowledge about the metadata for specific data content

• The Business or true subject matter experts that utilize the data on a regular basis, ensuring intimate knowledge about the data and how it is used

• Data Modelers that have extensive experience and knowledge about the data from creating data structures to capture and store the data content


Data Governance Team

• The Data Governance Teamparticipates in the review sessions with the subject matter experts

• The Data Governance Enterprise Architect is usually an experienced modeler, analyst, database administrator, and technician

• The Data Governance Business Analyst with a background working with the data from a business perspective


Typical SME Review Session• The review session is similar to a JAD session because you are gathering participants from across the organization to walk through the assessments and monitoring rule results to determine if identified problems are true data quality problems

• The subject matter experts provide intimate knowledge about the data from specific applications

• The data quality team drives the meeting and provides insight as to why specific problems have been identified

• The Data Governance team provides guidance on standards and view the data from an enterprise perspective as opposed to an application perspective


Technology Requirements

Technology Requirements

• Enterprise‐Level Data Profiling tool that profiles at the column, table, and cross‐table levels (column, dependency, and redundancy profiling)

• Enterprise‐Level Data Quality tool for creating and maintaining the data quality monitoring rules

• MS Office – MS Excel specifically• SQL tool for accessing relational data sources

• File editing tool that allows for viewing data in hex format

• Data modeling software capable of reverse engineering relational data sources


Example Deliverables

4/15/2015 23

DQ Assessment Deliverable


4/15/2015 24

Data Quality Monitoring Rules Deliverable


Advanced Solution

Advanced Solution

• Business Glossary• Metadata Repository• Data Quality Repository• Executive Summary• Balance Sheet of Data


Basic Analytics & Data Intelligence

Knowing what assets to invest into, remediate and enrich.

Example Set of Enterprise DQ Metrics

4/15/2015 28

Critical‐to‐Quality Metric Description Level Measurement Method

AccuracyDoes the data accurately represent reality or a verifiable source?

Column then aggregated to cross‐table

TI, TII, TIII Rules (TIII can also include a Manual Audit)

IntegrityDo broken links exist across physical data assets in data that should be related? Cross‐Table


Consistency Is there a single representative of data? Column and Cross‐TableTI, TII, TIII Rules (TIII can also include a Manual Audit)

CompletenessIs there data missing, blank, null, etc when it isnt supposed to? Column and Cross‐Table


Validity

Is the data stored in an acceptable fromat and contain valid values? Where possible, does the data adhere to a known architectural standard? Column


AccessibilityIs the data understandable, easily accessed and match its contextual use cases? Column and Cross‐Table Survey

TimelinessIs information recorded and available to consumers when it is needed? Column and Cross‐Table Survey

Detail Data Quality Analysis

This is an advanced method that is tailored to the specific Data Quality observations identified Column and Cross‐Table Custom Algorithm

Overall Data Quality ScoreWhat is the overall Data Quality considering all quality metrics above?

Aggregation of all data quality scores against a control limit Custom Algorithm

ConformanceDoes the structure or pattern fit to a standard or defined structure?

Overall Data Quality ScoreWhat is the overall Data Quality considering all quality metrics above?

Aggregation of all data quality scores against a control limit Custom Algorithm


Synthesis of the Expected Finished Good(Output of the Machine)Business Data Elementn (i.e. City Name from Address)

Overall MetaQuality Score TM• Conceptual Assets (i.e. Business Data Elements) ‐

Required metatags with degradation• Logical Assetn ‐ Required metatags with degradation• Physical Assetn – Required metatags with degradation

Overall Data Quality Score• Physical Assetn ‐ Column Level

• Completeness ‐ Pass/Fail Percentage• Business Defined Rulen

• Conformance ‐ Pass/Fail Percentage• Business Defined Rulen

• Integrity ‐ Pass/Fail Percentage• Business Defined Rulen

• Validity ‐ Pass/Fail Percentage• Business Defined Rulen

• Accuracy ‐ Pass/Fail Percentage• Business Defined Rulen

4/15/2015 29

BDE: SSNBDC: Items in force

Asset #1

Asset #2

Asset #3

Overall Data Quality Score

Asset #4 Asset #5

The Ultimate Finished Good

Overall Data Certification Score


Social Security Number Confidence Interval 58.33%

Data Quality Metric Score Target % to Target Last Updated Measurement Method

Accuracy 0.7 0.80 88% 7/9/2014 Automated control chart lineageIntegrity 0.5 0.80 63% 7/9/2014 Automated control chart lineageConsistency 4 5 80% 7/9/2014 Automated control chart lineageCompleteness 0.8 0.95 84% 7/9/2014 Automated control chart lineageValidity 0.43 0.95 45% 7/9/2014 Automated control chart lineage

Accessibility 0.75 0.65 85% 5/25/2014 ManualTimeliness 0.33 0.65 51% 6/6/2014 Manual

Overall Data Quality 4.5

MetaQuality Metric Target Score Last Updated Measurement Method

Count of Physical Instances 54 35 46% 6/24/2014 Automated control chart lineage

MetaQuality Score 4 6.00 67% 6/27/2014 Automated control chart lineage

Data Certification Score 4.3

Data Quality Scorecard

4/15/2015 30

data

75%

71%


Enterprise Data Quality Trend Charts

4/15/2015 31

Conformance


Advanced Analytics in Data as an Intangible Asset

Knowing what assets to invest into, remediate and enrich.

Accuracy Score

Consistency Score

Conformance Score

Completeness Score

Integrity Score

Business Data

Element

Business Metadata

Technical Metadata

Core Metadata

People Metadata

Data Quality

Metadata

Copyright Data Clairvoyance Group 2014

Physical instances of

data

Business Data

Element


Business Data Element

(I.e. written premium)

Business Metadata

Technical Metadata

Core Metadata

People Metadata

Data Quality

Metadata

Accuracy Score

Consistency Score

Conformance Score

Completeness Score

Integrity ScorePhysical

instance of data

Systems of Record

Systems of Authority (Access points)

Technical Data Lineage

Where does it live and come from?

Where is creation & consumption happening?


Enriched data elements lead to quantifiable confidence surrounding a given data element

Overall Meta Quality

Overall Data Quality+ = Data

Certification


Leveraging intelligence to manage data similar to how tangible assets are managed

Overall Meta Quality

Overall Data Quality+

Xn ’ s Y

in ’ s

Algorithm

Balance Sheet of Data


TheManagement Mechanism for all Chief Data OfficersBalance Sheet of Data

Data StewardData Certification

Score

Julius Stocking 3.00Juliana Lum 4.25Kayleigh Berryhill 4.30Erica Demma 4.67Allen Hayton 4.67Kimberely Truesdell 4.93


Score


Business Subject Area Business Data ElementData

Certification Score

Data Steward

Customer Active Customer 3.4 Kimberely TruesdellCustomer Social Security Number 2.2 Vivien RebeloCustomer Address Line Text 1.4 Carina SypherCustomer Birth Date 4.1 Dorothy OglesbyCustomer Business Process Channel 3.3 Darline ShearerCustomer Business Process Channel Subtype 4.2 Dorothy OglesbyCustomer City Name 2.0 Vivien RebeloCustomer Party 2.0 Phillip Kroenke

Business Subject Area Business Data ElementData

Certification Score

Data Steward

Customer Active Customer 3.4 Kimberely TruesdellCustomer Social Security Number 2.2 Vivien RebeloCustomer Address Line Text 1.4 Carina SypherCustomer Birth Date 4.1 Dorothy OglesbyCustomer Business Process Channel 3.3 Darline ShearerCustomer Business Process Channel Subtype 4.2 Dorothy OglesbyCustomer City Name 2.0 Vivien RebeloCustomer Party 2.0 Phillip Kroenke


Anna, IT(Technologist)

Bill, Marketing(Consumer)

Sally, Enterprise Architecture(Architect)

John, Marketing(Data Steward)

Customer Name

Capturing & structuring names, roles & functions can be extremely powerful metadata…


Customer Name

3.6Data

Certification Score

94%4.2

65%

Overall Data Quality Score

Overall Data Completeness

Data Accuracy


Name, Function(Role)




Data Element


Analysis on performance & Human Behavior related to data.


0

6

Data Steward

Technologists Consumers Architects DataCustodians


Role or Function


4/15/2015 44


Score



Score


TheManagement Mechanism to drive REAL accountabilityCommunity Cards

Data Steward


4/15/2015 45

TheManagement Mechanism to drive REAL accountabilityCommunity Cards

Data CustodianData Certification

Score

Carol Winters 4Julia Townsand 3Bob Sanders 2Bill Flibert 1


DataClairvoyanceA business approach to data.

Business Data Strategy for Leverage & Management

Business Driven

Real data practitioners, delivering real improvements to your enterprise data assets.

Metadata Design, Management and Implementation

Data Improvement Portfolio Planning & Management Data Quality Analysis & Measurement

Organizational Design for Data Centric Companies Data Improvement SDLC Integration

Technology Enablement

Documents

DataClairvoyance - DAMA Indiana · DataClairvoyance A business approach to data. Quantifying Trust & Confidence of Business Critical Data Designing, Enabling and Growing a Data Centric