Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Does Big Data Mean Big Master Data Management?
An Information Difference Research Study
November 2012
Sponsored by
Does Big Data mean Big MDM? 2
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
TABLE OF CONTENTS
EXECUTIVE SUMMARY ................................................................................................................... 4 BACKGROUND TO THE SURVEY ...................................................................................................... 5 THE APPROACH .............................................................................................................................. 5 ABOUT THE RESPONDENTS ............................................................................................................ 6 HOW IMPORTANT IS BIG DATA? .................................................................................................... 8 HOW BIG IS BIG? ........................................................................................................................... 10 BIG DATA AND MASTER DATA MANAGEMENT .............................................................................. 11 BIG DATA AND DATA QUALITY ...................................................................................................... 15 KEY REQUIREMENTS FOR MDM WITH BIG DATA ........................................................................... 16 ADDITIONAL FEEDBACK ................................................................................................................ 17 CONCLUSIONS ............................................................................................................................... 18
ENTERPRISES ....................................................................................................................................... 18 VENDORS ............................................................................................................................................ 19
ABOUT THE INFORMATION DIFFERENCE ....................................................................................... 20 QUESTIONNAIRE ........................................................................................................................... 21
Does Big Data mean Big MDM? 3
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
LIST OF FIGURES Figure 1 -‐ Respondents by Company Revenue .................................................................................................................................. 6 Figure 2 -‐ Respondents by Job Function .............................................................................................................................................. 7 Figure 3 -‐ Respondents by Industry Sector ........................................................................................................................................ 7 Figure 4 -‐ How Important is Big Data? ................................................................................................................................................ 8 Figure 5 -‐ Status of Big Data Initiatives .............................................................................................................................................. 9 Figure 6 -‐ Technology Used for Big Data Initiative ........................................................................................................................ 9 Figure 7 -‐ Data Volumes in Largest Application .......................................................................................................................... 10 Figure 8 -‐ Annual Growth of most Demanding Application .................................................................................................... 11 Figure 9 -‐ Status of MDM Initiatives .................................................................................................................................................. 12 Figure 10 -‐ Connection Between Analysis of Web Data and MDM ....................................................................................... 12 Figure 11 -‐ Connection Between Big Data and MDM ................................................................................................................. 13 Figure 12 -‐ Nature of Connection Between Big Data and MDM ............................................................................................ 14 Figure 13 -‐ Is Data Quality and issue for Big Data? .................................................................................................................... 15 Figure 14 -‐ Role of Data Governance in Big Data/MDM ........................................................................................................... 16 Figure 15 -‐ Key Requirements for an MDM system for Big Data ........................................................................................... 17
Does Big Data mean Big MDM? 4
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
EXECUTIVE SUMMARY
There has been a great deal of recent interest in the industry about how to deal with so-‐called Big Data, which is defined as follows: “Big Data is the term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. (Source: Wikipedia)”. Despite this media attention there is scant concrete information on how organizations are addressing the issues underlying Big Data. How does this relate, if at all, to master data management (MDM), and what are organizations doing to consider the impact of the Big Data trend on MDM initiatives? In short, what do organizations believe to be the link between Big Data and MDM? This survey, which was sponsored by Kalido and SAP, aims to answer these questions. The main findings from the survey, based on 209 responses, are summarized below: � 78% of organizations regard Big Data as important or higher. This supports the view that Big
Data is currently a hot topic. � Surprisingly, some 24% of organizations already have live Big Data initiatives with a further 15%
planned for completion this year. � Most (33% of the 41% which already have an initiative, see Figure 6 below) have opted to deploy
Hadoop to implement their Big Data initiative. � Big Data is perhaps less big than might at first be thought. The major concentration of sizes
appears to be in the 1 to 10 terabytes (23%) area or 10 to 50 terabytes (17%). Some 21% of respondents appear to have no idea about the size of their data.
� Big Data is growing fast, with 36% of organizations reporting 20% to 25% annual growth rates. A further 13% is experiencing 25% to 50% growth annually.
� 56% of organizations reported that they have live MDM implementations, with a further 9% planning to go live by the year-‐end. This is in line with our other recent studies where figures of 53% having live implementations were reported.
� 55% reported that there is at least a significant connection between the analysis of data collected over the web and MDM.
� 59% told us that there is a significant or greater connection between Big Data and MDM. � The view is that MDM should act as a search index (40%) facilitating much more targeted and
reliable analysis of web data. 27% see the role of MDM as providing feeds of master data to the Big Data applications.
� Only 17% felt that Big Data might be used to feed insights into MDM. � 53% believed that Big Data processing approaches should be used to facilitate processing of
master data. This is a surprising result given that most master data sets are relatively small. � 80% of organizations reported that data quality is of key importance for Big Data. This is a
somewhat paradoxical result given that results from our other surveys reveal that only a third of companies has actually deployed data quality tools.
� Fully two thirds (67%) consider data governance to be essential and a further 27% said it is an important enabler.
� The key requirements for an MDM system for Big Data lie in the ability to auto-‐identify master data elements in Big Data (25%) and to have the MDM closely integrated with the Big Data application (21%). Despite the recent hype by some vendors, only 8% considered increasing the speed of MDM to “real time” to be important.
� In summary, “MDM provides the dimensions for Big Data facts”.
Does Big Data mean Big MDM? 5
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
BACKGROUND TO THE SURVEY
There has been a great deal of recent interest in the industry about how to deal with so-‐called Big Data, which is defined as follows:
Definition: Big Data is the term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. (Source: Wikipedia).
Despite this media attention there is little concrete information on how organizations are addressing the issues underlying Big Data. How does this relate, if at all, to master data management (MDM), and what are organizations doing to consider the impact of the Big Data trend on MDM initiatives? In short, what do organizations believe to be the link between Big Data and MDM? This survey, which was sponsored by Kalido and SAP, aims to answer these questions. In particular, we investigate:
• How do organizations perceive the link between Big Data and MDM? • Just how big is big – how large are real-‐life Big Data applications? • At what rate is the Big Data challenge getting worse? • Is big data linked to data quality? • Do companies perceive data governance to be linked to Big Data?
THE APPROACH
The Information Difference survey, “Does Big Data mean Big MDM?”, was conducted over the Internet during the period September to October 2012. The participants were selected by email invitations originating directly from The Information Difference. Participation was also possible via a link from The Information Difference Ltd. website and via links on LinkedIn. The survey was targeted at senior business and IT leaders worldwide, drawn from larger organizations (with revenues greater than US $1 billion annually). The participants were provided with the following information prior to completing the survey: There has been a great deal of recent interest in the industry about how to deal with so-‐called Big Data, which we define as follows: Definition: Big Data is the term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. (Source: Wikipedia). How does this relate, if at all, to master data management (MDM), and what are organizations doing to consider the impact of the Big Data trend on MDM initiatives? This survey aims to answer these questions. All information provided will be used in aggregate form only and will be kept strictly confidential. The survey should not take more than 10 minutes to complete. In return for a fully completed survey you will receive a free summary of the analysis of the survey results. Additionally your name will be
Does Big Data mean Big MDM? 6
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
entered in a prize draw and the first five winners will receive a free vendor profile (worth $495) of their choice. We will also make a $2 contribution to the Red Cross for each fully completed survey. The questionnaire is appended in the section headed Questionnaire.
ABOUT THE RESPONDENTS
Some 209 respondents from across the world completed the survey. 45% were from North America (including Canada), 39% from Europe and the remainder (16%) fro the rest of the world. 59% of the respondents were from larger organizations with annual revenues in excess of US $ 1 billion. Indeed, 16% were from organizations whose annual revenue last year was greater than US $ 50 billion. 41% were from companies with annual revenues below US $ 1 billion. This represents a balanced mix of both larger and small organizations. The detailed breakdown is shown as Figure 1.
Figure 1 -‐ Respondents by Company Revenue
Only some 37% of the respondents were drawn from a business background with the majority having an IT role (63%). This may well reflect the current focus of Big Data in the media towards the IT community. 20% had job titles at the Director level or above and 32% had the title of Enterprise Architect. The details are shown in Figure 2.
Does Big Data mean Big MDM? 7
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
Figure 2 -‐ Respondents by Job Function
The largest level of participation was from the computing industry (24%) again, perhaps, supporting the observation above that currently this is a topic receiving most focus from the IT community.
Figure 3 -‐ Respondents by Industry Sector
Does Big Data mean Big MDM? 8
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
The proportion of respondents from the high tech (computing) is significantly higher that the norm for our surveys (usually of the order of 5-‐6%). Perhaps this is because high tech companies have to worry more about web traffic than most organizations since Big Data is usually machine generated e.g. web logs, social media, RFID, sensors etc. 16% were from a wide range of other industry sectors ranging from non-‐profit to agriculture and government. The financial sector was the largest vertical of the business respondents (13%) with retail, manufacturing and professional services coming in at around 8%. The remainder represents a wide range of industry sectors. The full results are shown in Figure 3. The analysis of the results from the survey is presented below. The questions referred to in the text are indicated as [Qn] and are set out in full in the appendix headed “Questionnaire”. Analysis of the results from the survey for regional dependencies—for example, comparisons between Europe and North America—did not yield any statistically significant differences or trends.
HOW IMPORTANT IS BIG DATA?
We first wanted to understand the magnitude of the problem and its level of importance in organizations. Respondents were asked to tell us how important Big Data was to their organization [Q1]. The results were surprisingly high, with only some 20% regarding it as unimportant. 78% regard it as at least important or higher. In the context of this result it is interesting to conjecture how many respondents would have been able to even define Big Data three years ago. It has certainly become a hot topic, presumably as a result of the exponential increase in web traffic experienced by many organizations. The full results are shown in Figure 4.
Figure 4 -‐ How Important is Big Data?
Given the considerable interest in Big Data we were curious to discover if organizations already have Big Data initiatives [Q2].
Does Big Data mean Big MDM? 9
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
We were quite surprised to learn that some 24% actually have live (or close to it) Big Data initiatives. Some 15% have initiatives planned for this year and a further 24% are planning to start initiatives next year. The results are summarized in Figure 5.
Figure 5 -‐ Status of Big Data Initiatives
These results suggest some very significant activity in the area of Big Data. Clearly organizations are moving fast to tackle the problem. What approaches are organizations adopting to tackle Big Data? We asked respondents to indicate what their initiative was based upon [Q3]. For example, Hadoop is one such, which is currently receiving much attention in the media. The results are presented in Figure 6.
Figure 6 -‐ Technology Used for Big Data Initiative
It appears that Hadoop wins by a significant margin, with some 33% of the 41% with an initiative deploying Hadoop (from Figure 6 above). This is clearly the dominant way to go at present and vendors would to well to be aware of this market trend.
Does Big Data mean Big MDM? 10
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
What technologies are being used by the 8% not using Hadoop? We asked respondents to share with us the technologies or approaches they were following. Their feedback included the following:
� Exasol. � InfoBright. � Still in process of selection. � Big Data can be categorized in three areas -‐ for us we're working with Big Data in terms of an
appliance and large volumes. So clearly Big Data is regarded as of serious importance by organizations with around two thirds already putting initiatives in place and the overwhelming majority of those deploying Hadoop.
HOW BIG IS BIG?
So how big is big? We were interested to get a view on the size of Big Data storage together with an estimate of the rate at which Big Data is growing. We first asked respondents to tell us what data volumes they had to deal with in their largest individual target application systems [Q12]. The results are shown in Figure 7.
Figure 7 -‐ Data Volumes in Largest Application
The major concentration appears to be in the 1 to 10 terabyte (23%) and 10 to 50 terabyte (17%) areas. Scarily, some 21% recorded that they had no idea! These results are also interesting because the general assumption for many of us is that Big Data is in the 100 terabytes and upwards range whereas here only 23% appear to be in this range. Perhaps we all need to revise downwards our notions of just how big is big.
Does Big Data mean Big MDM? 11
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
So how fast (in percentage terms) are the most demanding key applications growing annually? We asked respondents to share this with us [Q13]. The results are presented as Figure 8.
Figure 8 -‐ Annual Growth of most Demanding Application
This data is indeed growing and how, with 36% of organizations reporting between 20% and 25% annual growth rates with a further 13% experiencing 25% to 50% growth. It’s no wonder that Big Data is rapidly becoming a hot topic. In any event, these results confirm that the current flurry of media attention for Big Data is not all hype.
BIG DATA AND MASTER DATA MANAGEMENT
So, what of Big Data and MDM? Firstly we asked respondents to tell us whether they had an MDM initiative [Q4]. Their responses are summarized in Figure 9.
Does Big Data mean Big MDM? 12
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
Figure 9 -‐ Status of MDM Initiatives
56% of organizations reported that they already had a live MDM implementation. This is fully in line with results from our other recent MDM study1 where a value of 53% was reported. A further 9% plan to go live before the end of the year. Very encouragingly only 13% reported that they had no current plans for MDM. But do organizations perceive a connection between the business reporting and analysis of data collected via the web and MDM [Q5]? The responses are summarized in Figure 10.
Figure 10 -‐ Connection Between Analysis of Web Data and MDM
55% consider that there is at least a significant connection between analysis of data collected over the web and MDM. If the majority of Big Data is driven by web traffic then one might expect this figure to be significantly higher. Interestingly, 25% suggest that there is only a minor link and 13% suggest that there is no link at all. One explanation for this might be that they are deriving large volumes of data via other sources such as RFID.
1 MDM: Real-‐World Experiences, The Information Difference, Research Report, August 2012
Does Big Data mean Big MDM? 13
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
Under the heading “Other” we asked respondents for any additional comments they may like to share. These included:
� Today no connection but there should be in more than 2 years. � Yes, but not too much in the manufacturing business yet, more so in retail. � We are developers for data warehousing with Big Data for specific departments inside
several organizations. They need MDM, and, generally already have it. We then asked whether organizations believe that there is a key connection between Big Data and MDM [Q6]. 59% reported that they believe that there is a significant or greater connection between Big Data and MDM. This is perhaps surprisingly high since we are not currently aware of many examples of current case studies. Perhaps there is an urgent requirement for these to be produced in order to dispel the notion that this connection is merely vendor hype. One comment from a group of respondents who are developers was “Our customers need it!!!” The results are summarized in Figure 11.
Figure 11 -‐ Connection Between Big Data and MDM
We then went on to explore what, in the view of the respondents, should be the nature of the connection [Q7]. Their responses are summarized in Figure 12.
Does Big Data mean Big MDM? 14
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
Figure 12 -‐ Nature of Connection Between Big Data and MDM
The general view is that MDM should act as a search index (40%) allowing much more targeted analysis of Big Data. A further 27% see the role of MDM being to provide “feeds” of master data such as customer/household, product etc to the analytic applications. This is clearly of importance for vendors looking to develop links between their MDM products and Big Data applications. Interestingly, only 17% felt that Big Data could be used to feed insights into MDM systems. So the view is one of very much a one-‐way connection. We again asked respondents to share other options and these included:
� Bidirectional -‐ they provide controls to each other. � ETL. � MDM is the structure against which Big data can be exploited / used. � I want to check multiple. MDM provides the scope for initial Big Data of interest and value
to the organization. � MDM provides the backbone for transactions but with consumer / social media centric
volumes and self-‐management the master data potentially explodes beyond normal operating parameters. In particular the challenge around persistency of data in an analytics / insights environment. This explosion of uncontrolled master data will impact many other areas.
Can and should Big Data processing approaches be applied to MDM data processing [Q8]? Just over half the respondents (53%) believe this to be the case: that Big Data processing techniques should be applied to MDM data processing. This result is surprisingly high given that most master data sets are small. Outside customer the norm is up to about 5 million records, but in the case of customer MDM it is not so unusual for it to be 100 million or more, especially in the US and China. The only case where this might be appropriate as far as we can judge is in situations where you have many customers with billions of customer attributes/entities. In summary, 56% of organizations already have live MDM implementations. Most organizations consider that there is a clear connection both between the analysis of web data (55%) and Big Data and MDM (59%). Their view of the nature of the connection is that it should be very much one way
Does Big Data mean Big MDM? 15
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
with 40% seeing this as some form of search index and 27% believing it should be a feed from the MDM system of appropriate master data.
BIG DATA AND DATA QUALITY
Does data quality have a role to play in Big Data? We asked respondents to tell us whether they considered data quality to be an issue with Big Data [Q9]. Their responses are summarized in Figure 13.
Figure 13 -‐ Is Data Quality and issue for Big Data?
An overwhelming 80% told us that it is of key importance. This is an interesting and somewhat paradoxical result when compared with our previous surveys which have consistently showed that one third of organizations don't have a data quality tool. The feedback under the heading “Other” included:
� Probably, but I guess it's a lost battle if there are too many quality issues with big data. � Yes, but Quality for Big Data is different from Quality for MDM. � Possible depending on business application.
It also begs the question of how do you deploy a data quality tool on such high data volumes. Few data quality tools are good at matching when faced with very high data volumes. We suggest that this poses a massive opportunity for vendors to address. We then turned to enquire about the role of a data governance program for Big Data [Q10]. The results are shown in Figure 14.
Does Big Data mean Big MDM? 16
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
Figure 14 -‐ Role of Data Governance in Big Data/MDM
The above result is encouraging since just 3% consider data governance to be not essential. A resounding two thirds told us that it was essential while a further 27% regarded it as an important enabler. One relevant observation from a respondent was “Establishing business controls around what big data is needs to be harnessed”. This is probably the key missing element currently – what is needed for the business to “govern” Big Data?
KEY REQUIREMENTS FOR MDM WITH BIG DATA
So finally what are the key requirements for an MDM system for Big Data? How can MDM help to provide some structure to Big Data queries and analysis/reporting? We asked our respondents to share their experience/views on this topic [Q11]. Their feedback is summarized in Figure 15.
Does Big Data mean Big MDM? 17
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
Figure 15 -‐ Key Requirements for an MDM system for Big Data
The key requirements appear to lie in the ability to auto-‐identify master data elements in Big Data (25%) and to have the MDM system closely integrated with the Big Data implementation (21%). So effectively using the MDM to provide structure to Big Data. This is key for MDM vendors to take on board. Interestingly, increasing the speed of MDM to “real time” was considered key by only 8% of respondents. Again, an important result for MDM vendors.
ADDITIONAL FEEDBACK
Finally [Q14] we asked respondents to share any additional thoughts on Big Data. Their responses included:
� Big Data methods can help in solving of MDM problems. � Data quality and governance are key in keeping the data integrity going in to MDM. � MDM provides the dimensions for the big data facts. � Data governance and thorough understanding of an organization's needs and strategy as it
relates to data is fundamental before MDM can be successfully implemented. After MDM is implemented and mature, Big Data tools can help with processing/throughput and more advanced, transactional analytics but many of today's organizations have not yet completed the first step (governance and strategy).
� MDM is the foundation for Big Data. So it needs to keep pace with the volume, velocity and variety of big data coming in from internal and external sources.
� MDM and Big Data constructs offer new control capabilities that supersede application capability.
� MDM can't be separated from Big Data. MDM enables targeted Big Data analysis for business derivations. Current MDM systems are not designed to extract and enrich master
Does Big Data mean Big MDM? 18
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
data updates from Big Data. That is major gap that must be filled to automate Big Data processes up-‐to-‐date some extent.
� Big Data tools and technology are changing the entire MDM landscape enabling new approaches to managing data sources that feed master data.
� Understanding the lineage of all datasets is important with respect to consumption and explaining where the data came from in an analytic model.
� Hope MDM becomes more focused. As organizations have just started thinking about Master data management. As we have more tools from past 3-‐4 years focusing more on MDM. And banking industries or retail or any Industry should get good knowledge of their Big Data. And should analyze how much they can save in order of having good MDM tools in hand.
� Big Data does not mean MDM -‐ but it does require MDM be implemented as a foundation to exploit the value of data management for Big Data.
� We need to have meta data associated with master data in a way that the meta data is accessible to source applications and to analytic tools. The meta data values can serve as filters to validate source data and to focus reports/analytics on desired areas. We see the correct meta data enabling different MDM apps to synchronize and correlate.
� Our MDM solutions are legacy and not tailored to handle Big Data. Furthermore Big Data users in our company are not all obvious, i.e. justifiable by a business case. The benefit of their relation is therefore not obvious. We suspect that Big Data might be a buzz: more is not necessarily better. We need to go to "more appropriate" with an emphasis to the latter word. As a consequence a lot of up-‐front thought, organization, governance, strategy exists. This is or should be joint IT-‐business work
The statement that “MDM provides the dimensions for Big Data facts” perhaps best summarizes this.
CONCLUSIONS
Key conclusions and recommendations resulting from the survey analysis are summarized below. These have been split into two groups: those of direct relevance to enterprises and organizations considering or in the process of implementing (or who have already implemented) Big Data and MDM initiatives, and those relating to the software vendors and systems integrators (SIs).
Enterprises � 78% of organizations regard Big Data as important or higher. This supports the view that Big
Data is a hot topic. This is presumably as a result of the increase in web traffic experienced by many large organizations.
� Surprisingly, some 24% of organizations already have live Big Data initiatives with a further 15% planned for completion this year. Clearly organizations are moving fast to tackle the Big Data problem.
� Most (33% of the 41% which already have an initiative) have opted to deploy Hadoop to implement their Big Data initiative. This makes Hadoop the technology of choice at the moment.
� Big Data is perhaps less big than might at first be thought. The major concentration of sizes appears to be in the 1 to 10 terabytes (23%) area or 10 to 50 terabytes (17%). Surprisingly, some 21% appear to have no idea about the size of their data.
Does Big Data mean Big MDM? 19
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
� Big Data is growing fast with 36% of organizations reporting 20% to 25% annual growth rates. A further 13% is experiencing 25% to 50% growth annually.
� 56% of organizations reported that they have live MDM implementations, with a further 9% planning to go live by the year-‐end. This is in line with our other recent studies where figures of 53% live implementations were reported.
� 55% reported that there is at least a significant connection between the analysis of data collected over the web and MDM. Just 25% suggested that there is only a minor connection or none at all (13%). Possibly they are deriving their data from other non-‐web sources such as RFID.
� 59% told us that there is a significant or greater connection between Big Data and MDM. � The view is that MDM should act as a search index (40%) facilitating much more targeted and
reliable analysis of web data. 27% see the role of MDM as providing feeds of master data to the Big Data applications.
� Only 17% felt that Big Data might be used to feed insights into MDM. � Should/could Big Data processing approaches be used to facilitate processing of master data?
53% believed this should be the case. A surprising result given that most master data sets are relatively small.
� 80% of organizations reported that data quality is of key importance for Big Data. A somewhat paradoxical result given that results from our other surveys reveal that only a third of companies has data quality tools.
� Fully two thirds (67%) consider data governance to be essential and a further 27% said it is an important enabler.
� The key requirements for an MDM system for Big Data lie in the ability to auto-‐identify master data elements in Big Data (25%) and to have the MDM closely integrated with the Big Data application (21%). Despite the recent hype by some vendors only 8% considered increasing the speed of MDM to “real time” to be important.
� In summary, “MDM provides the dimensions for Big Data facts”.
Vendors � Some 78% of organizations regard Big Data as important, a potential market opportunity for
vendors. Especially since currently only some 24% actually have live implementations. � Most have opted to deploy Hadoop making this the technology of choice at present. This is a
positive message for those vendors in this technology area, and suggests that vendors with proprietary technology need to supplement and add value to Hadoop rather than trying to compete with it.
� Big Data is currently around the 1 to 50 terabytes (40%) but is growing fast. A third of organizations are experiencing growth rates of 20 to 25% annually. Vendors would do well to focus on this growth area for both MDM and Big Data technologies.
� Over half (56%) of organizations have live MDM implementations with more planned for the next years (26%). This represents an opportunity for MDM vendors.
� 40% believe that the MDM system should act as a search index facilitating more targeted and reliable analysis of web data. 27% see the role of MDM systems as providing feeds of master data to the Big Data applications.
� 53% believed that Big Data processing approaches should be applied to MDM. This is an area which vendors need to address and clarify further.
Does Big Data mean Big MDM? 20
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
� 80% told us that data quality is key for Big Data. This represents an opportunity for data quality vendors given that elsewhere we have consistently found that just one third of organizations have data quality tools.
� Two thirds (67%) say that data governance is essential. This represents an opportunity for vendors in this space.
� The key requirements for MDM systems for Big Data were: o The ability to auto identify master data elements in Big Data (25%), o To have MDM systems closely integrated with Big Data applications (21%). o Only 8% considered increasing the speed of MDM processing to “real time” as
important.
ABOUT THE INFORMATION DIFFERENCE
The Information Difference is an analyst firm focusing primarily on master data management (MDM), data quality and data governance. Our founders are pioneers who helped shape the MDM industry with in-‐depth global project experience. We offer detailed analysis of these industries, in-‐depth profiles of the MDM and data quality vendors, assessments of the marketplace and white papers discussing key issues and best practice. Additionally, we can offer advice on strategy, vendor selection and best practice in these areas. We carry out primary market research and can help you with MDM project justification, building the business case and return on investment.
Does Big Data mean Big MDM? 21
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
QUESTIONNAIRE
The full questionnaire used in the survey is included below. The navigation logic is not shown in the interests of clarity.
Does Big Data mean Big MDM? -‐ Survey 2012 Introduction There has been a great deal of recent interest in the industry about how to deal with so-‐called Big Data, which is defined as follows: Definition: Big Data is the term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. (Source: Wikipedia). How does this relate, if at all, to master data management (MDM), and what are organizations doing to consider the impact of the Big Data trend on MDM initiatives? This survey aims to answer these questions. All information provided will be used in aggregate form only and will be kept strictly confidential. The survey should not take more than 10 minutes to complete. In return for a fully completed survey you will receive a free summary of the analysis of the survey results. Additionally your name will be entered in a prize draw and the first five winners will receive a free vendor profile (worth $495) of their choice. We will also make a $2 contribution to the Red Cross for each fully completed survey.
Please note that questions marked with an asterisk (*) are mandatory.
1) Is Big Data important to your organization?* ( ) Very important ( ) Important ( ) Nether important nor unimportant ( ) Unimportant ( ) Irrelevant ( ) Don't know 2) Do you have a Big Data initiative?* ( ) Currently live ( ) About to go live ( ) Planned for the current year ( ) Planned for next year ( ) No current plans ( ) Don't know 3) If you already have a Big Data initiative is this based upon Hadoop?* ( ) Yes ( ) No (If it is not Hadoop, then what is it based on) [Please specify]: _________________* ( ) We do not have a live Big Data initiative ( ) Don't know
Does Big Data mean Big MDM? 22
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
4) Do you have an MDM (master data management) initiative?* ( ) Currently live ( ) About to go live ( ) Planned for this year ( ) Planned for next year ( ) Currently we have no plans for an MDM initiative ( ) Don't know 5) Do you see a connection between analysis of web data and MDM?* ( ) No ( ) Yes, there is a minor connection ( ) Yes, there is a significant connection ( ) Yes, they are intimately connected ( ) Don't know ( ) Other [Please specify]: _________________* 6) Do you see a connection between MDM and Big Data?* ( ) No ( ) Yes, there is a minor connection ( ) Yes, there is a significant connection ( ) Yes, they are intimately linked ( ) Don't know ( ) Other [Please specify]: _________________* 7) What in your view is the nature of the connection between MDM and Big Data?* ( ) There is no connection ( ) Big data can feed insights into an MDM system ( ) MDM can feed master data such as customer/household into a Big Data system ( ) MDM data can act as a "search index" for Big Data allowing more targeted analysis ( ) Don't know ( ) Other [Please specify]: _________________* 8) Do you believe that Big Data processing approaches could be applied to MDM data processing?* ( ) Yes ( ) No ( ) Don't know ( ) Other [Please specify]: _________________* 9) Is data quality an issue for Big Data?* ( ) Yes, it is of key importance ( ) Yes, but is of minor importance ( ) No ( ) Don't know ( ) Other [Please specify]: _________________* 10) What in your view is the role of data governance in Big Data/MDM?* ( ) Data governance is essential ( ) Data governance is an important enabler
Does Big Data mean Big MDM? 23
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
( ) Data governance is not essential ( ) Don't know ( ) Other [Please specify]: _________________* 11) What is the key requirement for an MDM system for Big Data?* ( ) Need to provide feeds/links to Big Data implementations ( ) Need to increase speed of MDM to "real time" ( ) Need to auto-‐identify master data elements in Big Data ( ) Need to start off from a sound MDM foundation for analytic MDM ( ) MDM for Big Data needs to be closely integrated into Big Data implementations analytic systems ( ) Don't know ( ) Other [Please specify]: _________________* 12) What data volume are you having to deal with in your largest individual application?* ( ) Less than 1 terabyte ( ) 1 to 10 terabytes ( ) 10 to 50 terabytes ( ) 50 to 100 terabytes ( ) 100 to 200 terabytes ( ) 200 to 500 terabytes ( ) 500 to 1000 terabytes ( ) Greater than 1 petabyte (1000 terabytes) ( ) Don't know 13) How fast in (percentage terms) is your most demanding application growing annually?* ( ) Between 0% and 10% ( ) 20% to 25% ( ) 25% to 50% ( ) 50% to 100% ( ) 100% to 200% ( ) 200% to 300% ( ) 300% to 400% ( ) Over 400% ( ) Don't know 14) Please enter below any additional views/comments which you have in regard to the role of MDM in Big Data analysis.
15) What was your company’s total revenue last year?* ( ) $ 50 billion or more ( ) $10 billion to $ 50 billion ( ) $1 billion to $10 billion ( ) $500 million to $1 billion ( ) $100 million to $500 million
Does Big Data mean Big MDM? 24
Copyright © 2012 The Information Difference Company Ltd. All Rights Reserved.
( ) Less than $100 million 16) Please select the main industry in which your company operates.* ( ) Aerospace & Defense ( ) Agriculture ( ) Banking/Insurance/Financial Services ( ) Chemicals/Energy/Utilities ( ) Computing (Hardware and/or Software) ( ) Construction ( ) Education/Training ( ) Government-‐Federal/State/Local ( ) Leisure/Travel/Hospitality ( ) Manufacturing ( ) Media/Publishing/Entertainment ( ) Metals & Mining ( ) Non-‐Profit/Charitable ( ) Pharmaceuticals/Biotech/Healthcare ( ) Professional Services/Consulting ( ) Real Estate ( ) Retail ( ) Telecommunications Services ( ) Transportation Services ( ) Other 17) Which of the following best describes your title or role in your company?* ( ) CxO, SVP or other Executive Role ( ) VP, General Manager, Director ( ) CIO or VP of Information Technology ( ) Enterprise Architect or Chief Architect ( ) Other Business Title ( ) Other IT Title 18) Are you willing to take part in a brief, confidential discussion on this topic with an Information Difference analyst?* ( ) Yes, name of sponsor ( ) No 19) Would you be willing to share your contact information with our research sponsors in order to learn more about their products?* ( ) Yes ( ) No 20) Please provide your brief contact details:* First Name*: ____________________________________________ Last Name*: ____________________________________________ Email Address*: ____________________________________________ 21) Please select your region:* ( ) Africa ( ) Asia