big Data analytics: Future architectures, Skills and …docs.media.bitpipe.com/.../item_575594/bigdata-idc-wp.pdf3 big Data analytiCS: Future architectures, Skills and roadmaps for

September 2011

by philip Carter

w h i t e p a p e r

big Data analytics:

Future architectures, Skills and roadmaps for the CiO

Sponsored by

1

w h i t e p a p e r

big Data analytiCS:


Brave New world oF Big data

the ‘big Data era’ has arrived — multi-petabyte data warehouses, social media interactions, real-time sensory data feeds, geospatial information and other new data sources are presenting organisations with a range of challenges, but also significant opportunities. IDC believes that as CIOs start to adopt the new class of technologies required to process, discover and analyse these massive data sets that cannot be dealt with using traditional databases and architectures, it will become clear that the real value will be derived from the high-end analytics that can be performed on the increasing volumes, velocity and variety of data that organisations are generating – or Big Data analytics.

One of the key differences between analytics in the traditional mode, and what we are dealing with in terms of the big Data era is that we are gathering data that we may or may not need – and from the perspective of analysis, this means ‘we don’t know what we don’t know’ – hence, the variables and models are likely to be entirely new, requiring a different infrastructure strategy and perhaps most importantly, new skill sets.

the objective of this white paper is to explore the initial impact that big Data is having on organisations, particularly the it departments – which is being forced to re-assess architectures, delivery models and future roadmaps. It will explore the following areas in more detail:

DefInIng BIg Data. this is not in the context of the quantity or threshold that actually quantifies Big Data (as this is changing all the time, and will be applied differently, depending on the vertical and market segment), but more in terms of a new generation of technologies and architectures, designed to economically extract value from very large volumes of a wide variety of data, by enabling high-speed capture, discovery and/or analysis.

haDOOp, mapreDuCe, Key Value StOre? there is a lot of hype around the new technologies that are being used by the market to deal with the Big Data phenomenon. We will highlight some of these and their relative importance.

the Value Of BIg Data… In analytICs. the bottom line here is that it is getting more complicated to process and analyse these

2

big Data analytiCS:


large and growing data sets – and it essentially requires a re-assessment of the broader information management strategies for the majority of organisations that have started their business analytics journey.

Why BIg Data analytICs Is ImpOrtant (anD DIfferent). many have asked the question – what is new with this trend? this section will highlight the traditional use of business analytics in the old ‘pre-big Data’ world, versus big Data analytics in the ‘new World’. this will also look at the various use cases that iDC expects to see being most commonly used across a variety of industries.

the SKill FaCtOr – the riSe OF the Data sCIentIst. With the raft of new technologies and organisational structures

that need to be put in place as the big Data phenomenon becomes a reality, there will be increasing demand for ‘data scientists’ – the next-generation analytical professionals who are able to extract information from large data sets and then present value-added content of business value to non-data experts – who also have the unique skill of understanding the new models that need to be put in place.

mapping Out the big Data analytiCS JOurney. the big Data analytics journey will be an iterative one – it is therefore important to map this out in the context of a broader framework. this section aims to do exactly that, and also provide some recommendations to CiOs as they embark on this exciting journey into the brave new world of Big Data analytics.

SituatioN overview the rise of Business analytics

much has been written on how the amount of data in the world is exploding in volume. according to the recent IDC Digital universe study, the amount of information created and replicated will surpass 1.9 zettabytes (1.8 trillion gigabytes) in 2011 – growing by a factor of 9 in just five years.

big data is a dynamic that seemed to appear from almost nowhere. But in reality, Big Data is not new – and it is moving into mainstream and getting a lot more attention. the growth of Big Data is being enabled by inexpensive storage, a proliferation of sensor and data capture technology, increasing connections to information via the cloud and virtualised storage infrastructure, as well as innovative software and analysis tools. It is no surprise then that business analytics as a

technology area is rising on the radars of CiOs and line-of-business (lOB) executives. to validate this, as part of a recent survey of 5,722 end users in the uS market, business analytics ranked in the top five It initiatives of organisations. the key drivers for business analytics adoption remained conservative or defensive. the focus on cost control, customer retention and optimising operations is likely a reflection of the continued economic uncertainty. however,

3

big Data analytiCS:


top drivers vary significantly by organisation size and industry. similarly, IDC surveyed 693 european organisations in February 2011 where 51% of respondents said that bi and analytics are high-priority technologies. In emerging markets such as asia/pacific, the focus is very much on capturing the next wave of growth.

according to more than 1000 CiOs and lOb executives that were interviewed as part of the asia/pacific C-suite Barometer in february 2011, business analytics was rated as the number one technology area that would enable their organisations to gain a competitive edge in the year ahead.

With more businesses in asia investing in It to ride the hyper growth wave in emerging markets, they are harnessing analytics-led solutions to gain better customer insights, manage risk and financial metrics more effectively, and at the same time, strive for unique market differentiation. historically, organisations have made significant investments in applications with the objective of automating business processes and capturing data to improve operational efficiency. many of these projects are still ongoing, but what is becoming increasingly clear to the senior management of these entities is that they (and their business managers) have not been able to get hold of the right information (mainly due to poorly integrated systems and

questionable data quality) at the right time (due to performance and scalability issues) to the right stakeholders within their organisations for the critical decision-making capabilities needed to drive the necessary business impact. and where they are unable to do this, the line of business is procuring and deploying their own solutions in a new wave of ‘shadow it’ investments focusing on business analytics, thereby forcing CiOs to re-examine these issues with a specific focus on driving better It-business alignment. these are taking place even without the ‘big Data’ dynamic in the picture – which when added, creates the ‘perfect storm’ for big Data analytics to take centre stage.

Figure 1: the rise of Business analytics

Source: iDC, 2011

Cloud computing/ services

0 5 10 15 20

Q: you (CIO/CtO) mentioned ‘harnessing ICt to gain competitive advantage’… which of the following technologies or solutions would be your leading choice to better harness iCt?

25 30 35 %

Collaboration (including video, mobility,)

Social media/ online channel

network

business intelligence/ analytics

top 5

4

big Data analytiCS:


a Note oN termiNology: Bi or analytics?

We have some challenges when defining and using terminology for business analytics. Because the bi market is mature, many terms have been around for a long time and have either become obsolete or have been redefined over the years. For example, the term ‘bi’ itself is sometimes used in a narrow sense (only query, reporting, and analysis [Qra] technology) and at times, in a broad sense to refer to the whole of what IDC calls business analytics (including data warehousing and analytic applications in addition to front-end tools). the term ‘analytics’ is relatively new and its meaning is often unclear — does it refer to advanced analytics including predictive analytics, optimisation and forecasting, or analytic applications? In some submarkets, such as Web

analytics, the term ‘analytics’ simply means a dashboard on top of some data.

For the purpose of this white paper, we interpret ‘bi’ to mean either Qra tools or BI across the board (in its narrow definition), or ‘business analytics’ (in its broad definition) in IDC terminology. We interpret ‘analytics’ to mean either advanced analytics (data mining, statistics, optimisation and forecasting) or analytic applications (fpsm, Crm and marketing analytics, supply chain analytics, etc.). Business analytics is a combination of the above (and also includes data warehousing technologies) and this is highlighted by iDC’s business analytics taxonomy for 2011 (see figure 2 below):

Figure 2: idC Business analytics taxonomy

performance management & analytic applications Business intelligence tools

Financial performance & Strategy management

budgeting, planning, Consolidation, profitability, strategy management

Crm analytic applicationsSales, Customer Service,

Contact Centre, marketing, Web site analytics, price Optimisation

Query, reporting, and analysis tools

Dashboards, production reporting, Olap, ad-hoc query

Supply Chain analytic applications

procurement, logistics, inventory, manufacturing

Services operations analytic applications

Financial services, education, government, healthcare,

communications services, etc.

advanced analytics toolsData mining and statistics

production planninganalytic applications

Demand, supply, and production planning

workforce analytic applications Spatial information

analytics tools

Content analysis tools

data warehouse management platform

data warehouse management

data warehouse generationData extraction, transformation, loading; data quality

Source: iDC, 2011

5

big Data analytiCS:


Figure 3: Defining ‘Big Data’

Source: iDC, 2011

unstructured data (video, rich media etc)

time

Data Volumes

the VOlume. One is embodied more in the structured data realm. some of this is held in transactional data stores and is linked to the ever-present electronic trail that individuals and businesses create in the wake of rapidly increasing online activity. sensory data (machine-to-machine) contribute to this area too. the other is in existing data warehouses or data marts, which have over time grown to petabyte scale.

the VarIety. the other aspect of this big Data phenomenon is the need to analyse semi-structured and unstructured data. text, video and other forms of media will require a completely different architecture and technologies to perform for the required analysis. for example, if you look at the social media phenomenon, many marketing departments are looking at ways to do sentiment and brand analysis based on what is being posted on Facebook, twitter and youtube. this dynamic becomes more

complex in asia with local social media sites like renren in China and nate in Korea.

the VelOCIty. there will also be demand to analyse this data on a more regular basis – for example, taking into account all transactions rather than a sample to obtain a more complete view of risk on a trade in real time.

in summary, big Data refers to data sets whose volume, variety, velocity and complexity make it impossible for current databases and architectures to store and manage. IDC intentionally does not define Big Data as larger than a certain threshold (i.e. terabytes), mainly since this threshold would be a moving target depending on the sector, as well as the fact that it will obviously grow over time. more important is the value that organisations can derive from this phenomenon – and the resulting need to rethink their information strategies to extract the value.

Defining ‘Big Data’

big Data is not so much about the content that is created, nor is it even about consumption. It is more about the analysis of the data and how that needs to be done. It is not really a ‘thing’, but instead a dynamic/activity that crosses many it borders. IDC defines Big Data in this way:

“big Data technologies describe a new generation of technologies and architectures, designed to economically extract value from very large volumes of a wide variety of data, by enabling high velocity capture, discovery and/or analysis.”

Semi-Structured (e.g. weblogs, social media feeds)

data =Big, Complex,

high velocity & wide variety

6

big Data analytiCS:


other deFiNitioNS:hadoop, mapreduce, Key value Store

With the focus on Big Data going mainstream, a range of new technologies have hit the market. the table below gives an overview of these technologies, with associated context (note that the list is not exhaustive).

although some of these terms will be used throughout this white paper, the focus is not to examine them in too much detail – because as one it executive recently mentioned – ‘to know the technology is one thing, but to apply it in the right environment is something entirely different’. the new technology needs to be tied back to business requirements as much as possible – not just examining the technology for the sake of

it. having said that, most It executives are not aware of the technologies and trends developing in this area – and where they are aware of it, their strategy is to put a couple of people in their enterprise architecture team to experiment with the new technologies (i.e. in memory, hadoop, mapreduce, Key Value Stores etc) that are being used to deal with the ‘Big Data’ phenomenon.

technology Context

Big tableproprietary distributed database system built on the google file system. Inspiration for hBase.

Cassandraan open source (free) database management system designed to handle huge amounts of data on a distributed system. this system was originally developed at Facebook and is now managed as a project of the apache software foundation.

data warehouse & analytical appliance

Consists of an integrated set of servers, storage, operating system(s), database, business intelligence, data mining and other software specifically pre-installed and pre-optimised for data warehousing.

distributed System

multiple computers, communicating through a network, used to solve a common computational problem. the problem is divided into multiple tasks, each of which is solved by one or more computers working in parallel. Improved price:performance ratio, higher reliability and more scalability.

google File Systemproprietary distributed files system developed by google: part of the inspiration for hadoop.

hadoop

an open source (free) software framework for processing huge data sets on certain kinds of problems on a distributed system. Its development was inspired by google’s mapreduce and google file system. It was originally developed at yahoo! and now managed as a project of the apache software foundation.

hBasean open source (free) distributed, non-relational database modeled on google’s Big table. It was originally developed by powerset and is now managed as a project by the apache Software Foundation as part of hadoop.

mapreducea software framework introduced by google for processing huge data sets on certain kinds of problems on a distributed system. also implemented in hadoop.

Non-relational database/Key value Store

a non-relational database is one that does not store data in tables (rows and columns) – in contrast to a relational database. Key Value stores allow for the management of schema-less (nosQl) entities.

table 1: Big data technologies (terminology)

7

big Data analytiCS:


Big data aNalytiCS: the old world vs. the New era

many have asked the question – what is new with this trend? this section highlights the traditional use of business analytics in the old ‘pre-big Data’ world, versus big Data analytics in the ‘brave new World’. this will also look at the various use cases that iDC expects to see being used most commonly across a variety of industries. the majority of it organisations have progressed in terms of their infrastructure architectures over time; from predominantly mainframe-based environments in the 1980s to a focus on client-server in the 1990s and the Web at the turn of the century, to what is now popularly known as ‘private cloud’. this supposed state of ‘nirvana’ constitutes a consolidated, virtualised set of infrastructure resources (server, storage and network) that can be self-provisioned in an automated fashion by

business users – complete with Slas that have the security, performance, availability and cost profiles transparent to all in the form of a service catalog. Very few organisations, if any, have achieved this state of infrastructure ‘nirvana’, and are still battling with a spaghetti-like tangle of compute resources in their datacenter. and now, we have this external force of big Data as mentioned earlier that is forcing CiOs to re-architect their infrastructure – particularly in the context of how analytics capabilities are deployed in an enterprise-wide fashion.

below is an overview of the changes that iDC sees happening in the infrastructure world that is increasingly impacting the big Data analytics world:

old world New era

tenancy infrastructure Silos pooled resources

architecture performance ‘tuned’linear scalability (linked to distributed parallel processing and ‘in memory’ storage)

delivery model On premise hybrid (with cloud bursting capabilities) and widespread use of the appliance

table 2: old world vs. New era (Big data infrastructure)

8

big Data analytiCS:


based on iDC’s research in this space, here are three suggestions for CiOs in dealing with these issues:

ClOuD BurstIng. the private cloud journey will line up well with the enterprise-wide analytical requirements highlighted earlier, but CiOs need to ensure that workload assessments are conducted rigorously and that risk is mitigated where possible. Critical to this approach will be the evaluation of cloud bursting capabilities from external vendors (i.e. Infrastructure as a service), particularly as organisations start to leverage more real-time analytics environments, to ensure that the use of infrastructure resources maps closely to demand – and that there are no issues in terms of performance and availability.

analytICal applIanCe. in terms of delivery models, IDC has seen significant performance benefits from analytical appliances for customers that are dealing with the impact of Big Data. In addition, since the software is optimised and pre-integrated with appliances, the deployment timeframes are typically shorter. as part of a recent global survey of CiOs, 10% of the respondents indicated that they will be looking at analytical appliances as a delivery model in 2011. IDC also believes that the demand for reference architectures will rise as CiOs look to integrate these appliances within existing data warehousing environments. in line with this increased adoption of the analytical appliance as a delivery model, iDC believes that it departments will allocate less budget towards technical skills (i.e. installation, configuration and management), and more on

the high-end analytical skills needed to help drive the necessary business impact across multiple functions.

enterprIse arChIteCture. enterprise analytics needs an enterprise architecture that scales effectively with growth – and the rise of big Data analytics means that this issue needs to be addressed more urgently. Organisations need to look at creating a ‘high performance analytical environment’ that leverages in-database analytics, parallel processing as well as in-memory storage to deal with the increased volume, velocity and variety of data. particularly, in terms of dealing with unstructured data, more attention needs to be paid to hadoop – an open source software framework set up by apache that allows for the distributed processing of large data sets across clusters of computers. however, there will be an ongoing tension between global standards and local requirements – and the use of hadoop would be a good example of this. another would be the ability to process mixed workloads (e.g. analytical and operational) in the same infrastructure environment such as the appliance that was mentioned earlier. CiOs need to consider ways in which they can deliver value in terms of solving specific business problems, while at the same time, being cognizant of global architecture standards and specifications. While certain global governance models will not allow for the usage of some of these technologies in a production environment, business expectations will force it departments to re-assess the way the enterprise architecture agenda is utilised at a local level.

9

big Data analytiCS:


the bottom line here is that it is getting more complicated to process and analyse these large, complex and growing data sets – and it essentially requires a re-assessment of the broader information management strategy for the majority of organisations that have started their business

analytics journey. But the impact is potentially enormous. If you look at optimising the price on every item in a global retail chain or detecting fraud in real time – you get a sense of the type of problems that big Data analytics can be used to solve.

old world New era

data Sets predefined all-encompassing and iterative

data velocity batch proactive and dynamic (real-time where appropriate)

data analysis predominantly historic predictive, Forecasting & Optimisation

table 3: old world vs. New era (Big data analytics)

video Surveillance/analysis

however, despite the clear potential of such analytics – it is important to understand that it will not necessarily be relevant or applicable to every use case. IDC believes that these use

cases can be best mapped out across two of the big Data dimensions – namely velocity and variety as outlined below:

Figure 4: potential use Cases for Big data analytics

data velocity

real time

batch

unstructuredSemi-structuredStructured

data variety

traditional data warehousing text mining

demand Forecasting in manufacturing

disease analysis on electronic health

records

predictive maintenance in

aerospaceSocial media

Sentiment analysis

Claims and tax Fraud in public Sector

markdown optimization in retail

event-based marketing in Financial Services and telecoms

Fraud detection (Credit Card) & Financial Crimes (aml) in Banks (including social network analysis)

Credit & market risk in Banks

10

big Data analytiCS:


a better sense of the potential impact of deploying big Data analytics to drive high value impact can be derived by exploring these use cases in more detail:

real-tIme frauD DeteCtIOn In BanKs. involves the ability to detect, prevent and manage fraud across multiple products, lines of business and channels for a bank. this requires the ability to capture the history for different types of entities (e.g. card, account, customer, terminal iD or ip address) involved in transactions, amplifying accuracy in detecting customer behaviours that fall outside the norm during point-of-sale (pOs) transactions. this information can be used by multiple predictive models, for fraud detection and credit risk assessment.

marKDOWn OptImIsatIOn In retaIl. the ability for retailers to optimise prices for a

wide range of products in real time based on demand forecasting scenarios (that include the impact of promotions, seasonality and important calendar events) has a major impact on margins. these capabilities can also be augmented by social media sentiment analysis to ascertain customer demand for certain products on a more real-time basis.

DiSeaSe analySiS On eleCtrOniC health reCOrDs. as healthcare services evolve, analysts can get hold of a patient’s entire medical history in electronic format. this will present a major opportunity for big Data analytics. for example, in the case of a disease such as diabetes, the ability to correlate patient medical history with dietary data (potentially from market basket analysis in retail) and optimised exercise schedules will provide medical practitioners with new insights that they had only previously dreamt of.

the SKill FaCtor

as highlighted earlier, iDC believes that the real value from big Data will be derived from the high-end analytics that can be performed on the increasing volumes, velocity and variety of data that organisations are generating. In asia (outside some of the mnCs because this is mainly being driven out of the us and europe), most organisations are not aware of the type and level of skills that are required. IDC also believes that this is linked to the general lack of awareness and skill available historically in the high-end analytics arena (regardless of the Big Data phenomenon).

high-end analytics will require new sets of skills in two key categories:

teChnICal sKIlls. For the new class of technologies required to process, discover and analyse these massive data sets that cannot be dealt with using traditional databases and architectures (i.e. in memory, hadoop, mapreduce, Key Value stores etc). some of these technologies will be delivered as an appliance – and skills to better understand how

the software interacts with the hardware to leverage the data will be required.

the neW type Of BusIness analyst/statIstICIan. One of the key differences between analytics in the ‘Old World’ and what we are dealing in terms of the big Data era is that we are gathering data that we may or may not need – and from the perspective of analysis, this means ‘we don’t know

11

big Data analytiCS:


what we don’t know’ – i.e. there is so much unstructured data that the variables and analytical models are likely to be entirely new. this means that there is a need to re-think the way the analytical power users approach their work by creating a ‘Sandbox mentality’ where discovery is always the starting point. generally, a background in data mining and statistics would be a good starting point for this type of analysis. moving forward, there will be increasing demand for ‘data scientists’ – the next-generation business analyst with strong statistical skills who are able to extract information from large data sets and then present value to non-analytical experts – but with the unique skill of understanding the new algorithms and analytical models that will have the most significant business impact in the short term. globally, IDC is seeing a lot of interest in this more analytically inclined skill set. roles and responsibilities have not been defined – but it basically fits in with the earlier comments in terms of ‘we don’t know what we don’t know’ – i.e. there is so much unstructured data that the variables and analytical models are likely to be entirely new. It requires a very ‘out-of-the-box’ type and creativity in terms of

the analytics that needs to be done on these new data types and structures.

For example, if you look at the social media phenomenon (contributing to the semi-structured and unstructured data part of big Data), many marketing departments are looking at ways to do sentiment and brand analysis based on what is being posted on Facebook, twitter and youtube (massive amounts as you can expect). this dynamic becomes more complex in asia with local social media sites like renren in China and nate in Korea. Currently, It is not the first port of call for the chief marketing officer since it lacks the skills to understand what needs to be done (and in many cases, is still trying to work out what role it should play in the policy or governance of the use of social media). so the make-up of the It department needs to be re-assessed in terms of technical, business and relationship skills.

the maturity model below highlights how iDC sees these skills (both technical and business) mapping out in the context of the organisations that have adopted business analytics over time – with a view to how this could evolve in the era of big Data analytics:

12

phase

impact

old world New era

pilot departmentalanalytics

enterpriseanalytics

Big data analytics

Staff Skills (it)little or no expertise in analytics – basic knowledge of bi tools

Data warehouse team focused on performance, availability and security

advanced data modelers and stewards key part of the it department

business analytics Competency Centre (BaCC) that includes ‘data scientists’

Staff Skills (Business/it)

Functional knowledge of bi tools

Few business analysts – limited usage of advanced analytics

Savvy analytical modelers and statisticians utilised

Complex problem solving integrated into business analytics Competency Centre (BaCC)

technology & tools

Simple historical bi reporting and dashboards

Data warehouse implemented, broad usage of bi tools, limited analytical data marts

in database mining, and limited usage of parallel processing and analytical appliance

Widespread adoption of appliance for multiple workloads. architecture and governance for emerging technologies

Financial impact

no substantial financial impact. no rOi models in place

Certain revenue generating Kpis in place with rOi clearly understood

significant revenue impact (measured and monitored on a regular basis)

business strategy and competitive differentiation is based on analytics

data governance little or none (skunk works) initial data warehouse model

and architectureData definitions and models standardised

Clear master data management strategy

line of Business (loB) Frustrated Visible aligned (including

lOb executives)Cross-departmental (with CeO visibility)

Cio engagement hidden limited involved transformative

% of Customers(idC estimates) 20% 65% 10% 5%

big Data analytiCS:


in terms of capturing and developing the right skills in the era of big Data analytics, the creation of a business analytics Competency Centre that sits across the business and it departments will be critical. IDC believes that this type of structure not only clarifies the roles and responsibilities of key stakeholders for this transformation, it also drives internal visibility, provides a mechanism for education as well as bridging the it/business gap (and the marketing and sales teams in particular – as key individuals from these departments will need to be represented) since improving decision making amongst front-office staff will be the primary focus of these projects.

in conjunction with the skills dimension, iDC believes that this structure should be involved in the following areas:

technology identification/deployment

Business case creation and rOI justification

Data governance frameworks with clear

policies and guidelines around master data management, data quality and data models

ensure it/business alignment by involving the critical stakeholders at the right time

involve the CiO as the supporter of the necessary transformation from an it perspective that will in turn create the necessary business impact

Very few organisations have reached the level of maturity that can truly harness the potential that big Data analytics represents – and practically speaking, it is a major challenge to have ticked off all the relevant boxes, but this transformation is a necessary one in order for organisations to truly differentiate themselves in the current economic environment. the CIO (and the It department) needs to play a critical role in this transformation. the next section highlights some suggestions that iDC believes should be taken into account in the context of this journey.

Figure 5: the Big data analytics maturity model

13

big Data analytiCS:


the CiO ‘Big Data analytics’ Checklist

arChIteCt fOr the future. historically, a lot of work in analytics has been focused on ‘workarounds’ due to the limited scalability of the underlying hardware. as a result, many it departments would create materialised views or pre-calculated data structures so that business users could work off these without impacting the performance of the systems that were processing the underlying data. Clustering, parallel processing and in-memory technologies mean that all that underlying data can now be used in the analytical environment. however, it is important not to fall into the same trap of blindly adding capacity based on availability. there is a need to assess multiple delivery models (i.e. cloud – particularly for bursting capabilities, analytical appliances as well as the traditional client/server or 3-tiered Web architecture approach) on a case by case basis, as one size will definitely not fit all.

Create a ‘SanDbOx mentality’. One of the key differences between analytics in the traditional old-school batch mode and what we are dealing with in terms of the big Data era is that we are gathering data that we may or may not need – and from an analysis perspective, this means ‘we don’t know what we don’t know’ – i.e. there is so much unstructured data that the variables and analytical models are likely to be entirely new. this means that there is a need to re-think the way that analytical power users go about developing their models by creating more of a ‘Sandbox mentality’ where a discovery process is always the starting point, particularly in terms of drawing linkages between unstructured, semi-structured and structured data. as part of this, new types of skills will need to be brought on board to understand social media nuance (i.e. more likely to be from gen y, Z or even the millennials).

nOt tOO muCh ‘tInKerIng’. Whenever a new set of cool technologies hits the market, there is a tendency for it departments

to ‘tinker’ – which impacts the immediate business benefits. so while a certain amount of experimentation is a good thing (as outlined in the context of the ‘Sandbox mentality’ highlighted earlier – hadoop and mapreduce definitely fit into this category), CIOs need to be careful that not too much time is wasted on experimentation versus delivering business value.

get the team rIght. the first step in this process involves the CiO assessing his/her own it department to examine relevant skill levels and organisational structures. In some cases, it will necessitate an internal transformation to get the business to take notice of the change. It then requires that the right people are empowered to execute the it analytics strategy with the relevant processes and governance structures in place to enable them to effectively deliver the business expectations. part of this will require a much deeper understanding of the capabilities of the underlying analytics technology for the CiO, but it will also involve working with lOb executives to hire the right type of analytically minded managers and knowledge workers who can leverage the underlying technological capabilities at the most optimal levels.

taKe analytICs tO the enterprIse. the majority of it projects in this space have been focused on building a data warehouse combined with a variety of bi tools to surface the underlying information to the end users. however, in terms of sophisticated analytics functionality, the lack of it skills meant that these projects have been largely departmental and tactical in nature, leading to a ‘silo-ed’ mentality. as a result, to assess something such as risk-adjusted profitability (combining financial, credit scoring and customer data) would be impossible. this needs to change; and it requires a different level of it/business collaboration to do so, with the CiO personally focused on an enterprise-wide approach in deploying analytics to ensure that these projects are successful.

14

big Data analytiCS:


gOVernanCe anD enaBlement. this is where existing investments made in data warehousing technologies, if done correctly, will pay dividends. the data models and reference architecture that it has in place will ensure that data definitions and standards are consistent across the various business departments. further work needs to be done in the master data management (mDm) space in terms of bridging the operational and analytical gap around data governance – but fundamentally, this platform should provide the necessary management and control that It requires. When it comes to business enablement, iDC sees a new class of projects emerging that combines

business analytics with business process management capabilities – more specifically, decision management software components that include tools for rule management, data mining, query and reporting, complex event processing (Cep), collaboration, Bpm suites, search, and content analysis. IDC believes that it departments that can complement previous investments in data warehousing and business intelligence technologies with a better understanding of the decision-making process in each of their organisations and the underlying decision management software will be best placed to manage the it governance versus business enablement dilemma.

CoNCluSioN

Despite the varying levels of maturity and adoption of business analytics, businesses are definitely gearing up for the utilisation of more advanced solutions and offerings in this space. In line with this, organisations need to plan strategically and build a robust roadmap before adopting business analytics. the new generation of business managers is more aware of the benefits of competing on business analytics and will be looking to drive adoption of this technology area more aggressively. moving forward, IDC believes that a new approach is required to proactively ‘effect’ the necessary change, with a specific focus on the following areas:

elevating the status of the CiO to that of one with more transformative impact on the organisation by playing an integral role in the deployment of the enterprise analytics strategy – and ensuring that these technologies have the expected business impact

an assessment of alternative delivery models (such as the appliance, in memory and hadoop for Big Data)

Capturing higher-level lOb attention and visibility as the next wave of business analytics projects are integrated with complex event processing (Cep) and business activity monitoring (Bam) technologies to drive a new class of projects that IDC defines as ‘decision management’

the role of the CiO is gradually becoming much more important in the boardroom and is playing a key role in the purchase behaviour of advanced applications such as business analytics. moreover, the CIO and the it department need to leverage a broader set of business analytics capabilities to create a new information management strategy that deals with the emerging big Data dynamic as well as delivering improved decision-making capabilities to the business stakeholders across the organisation.

abOut thiS publiCatiOnthis publication was produced by IDC go-to-market services. IDC go-to-market services makes IDC content available in a wide range of formats for distribution by various companies. a license to distribute IDC content does not imply endorsement of or opinion about the licensee.

COpyright anD reStriCtiOnSany iDC information or reference to iDC that is to be used in advertising, press releases, or promotional materials requires prior written approval from IDC. for permission requests, contact the gms information line at 65-6829-7757 or [email protected]. translation and/or localization of this document requires an additional license from IDC.

for more information on IDC, visit www.idc.com. for more information on IDC gms, visit www.idc.com/gms.IDC asia/pacific, 80 anson road, #38-00 fuji Xerox towers, singapore 079970. p. 65.6226.0330 f. 65.6220.6116 www.idc.com.

Copyright 2011 IDC. reproduction is forbidden unless authorized. all rights reserved.

#ap14962u

Documents

big Data analytics: Future architectures, Skills and …docs.media.bitpipe.com/.../item_575594/bigdata-idc-wp.pdf3 big Data analytiCS: Future architectures, Skills and roadmaps for