Report: CIOs & Big Data

#1 Enterprise Big Data Cloud

CIOS & BIG DATACIOCIOCIOSSSSSSSSSSSSSSSSSSSSSS & BIG DATA& BIG DATA& BIG DATAWhat Your IT Team Wants You to Know

CIOS & BIG DATA

Today enterprises face an increasingly competitive and erratic global business environment, and Big Data has become more than just another IT project — it’s truly a finger on the pulse of the business. To say that in 2013 Big Data is “mission critical” is to put it mildly — organizations that ignore the insights that Big Data can deliver are flying blind. So, it is all the more disconcerting that

55% of Big Data projects don’t get completed, and many others fall short of their objectives. While there may be many reasons for this, undoubtedly one of the biggest factors is lack of communication between top managers, who provide the overall project vision, and those charged with actually implementing it. Far too frequently the opinions of the IT staff doing the heavy lifting necessary to develop a Big Data project are taken as an afterthought, and consequently considered only when projects veer off-course. Given the stakes, it’s imperative that CIOs have a 360 degree view of what such an initiative will involve before it leaves the drawing board. In order to provide a complete picture, we partnered with SSWUG.ORG, one of the largest enterprise technology-focused, community-driven sites and a source for answers to IT-related questions and professional growth for more than 570,000 members.

Together, we recieved survey responses from over 300 IT department staffers – 58% of whom have current Big Data projects underway – on what they most wanted their CIOs to know about the process of implementing Big Data projects. The insight we gleaned reveals much about both enterprise technology and enterprise culture. In order for companies to succeed in this era of Big Data, executives will need to rethink long-held notions of how multiple departments should function together. In the past “breaking down silos” was a nice mantra. Now, it is an imperative. Additionally, CIOs and other enterprise executives may find it necessary to educate their organizations on the advantages of new Big Data applications and processes that will give them better customer insights, make their jobs infinitely easier and give their departments the elasticity needed to meet virtually any business need in real-time. We hope this report will serve not only as a source of insight for those responsible for architecting Big Data initiatives, but also as a reminder to seek the invaluable perspective of IT staff as early as possible in the process of developing technology projects. -- Jim Kaskade, CEO, Infochimps

Foreword

Infochimps and SSWUG.ORG queried more than 300 IT department employees for this report in fall of 2012, 58% of whom identify themselves as currently being involved in big data initiatives. Big Data implementation is a concern for companies of all sizes, and we sought the opinions of people from organizations ranging from small business to enterprise level. Of those surveyed,

29%come from companies with more than 1000 employees,

27% are from companies with between 101-1000 employees,

45% identify their companies as having 100 or fewer employees.

In order to hone in on insight from those primarily responsible for implementation, the sample is weighted toward staff-level positions. 86 percent of respondents identify themselves as either directors, managers, or systems administrators/developers, while the remaining 14 percent identify themselves as either VP/SVP or as belonging to the C-suite.

It may also be worth noting that 85 percent of respondents described their data footprint as being larger than one terabyte, and 7 percent of those described it as being larger than one petabyte, indicating the tremendous scope and variety of organizations considering Big Data initiatives.

For the purposes of this report and in order to most accurately represent well-informed viewpoints, the statistics cited are based on the responses of those participants who are currently involved in Big Data initiatives. A breakdown of the opinions of the complete respondent pool may be provided upon request. However, given that a key objective of this report is to inform CIOs regarding potential pitfalls to successfully implementing Big Data projects, we feel that the survey results, as presented, offer the most valuable and strategic insights.

About the Research

Big Data may indeed be a buzzword for 2013, but if so it’s a buzzword with inestimable impact on the organization. At its core, the concept of Big Data is that of supporting executive decision-making with the most accurate, current, comprehensive and comprehensible presentation of all information available regarding a business. Overwhelmingly, respondents indicated that their companies are taking Big Data seriously

81% list “Big Data/Advanced Analytics Projects” in their Top 5 2013 IT priorities.

The survey’s finding that IT staff members prefer to focus on the application layer rather than on infrastructure may provide a strong indication as to where CIOs ought to be directing their investments. When asked about top IT priorities outside of Big Data, respondents ranked mobile, website, security and dashboarding notably above infrastructure-related options. These responses are aligned with a general preference for managing the front-end over the back-end, with participants reporting that processing (43%), analyzing (41%) and ongoing management (42%) of Big Data pose significant difficulties — companies may consider outsourcing these functions to keep staff focused on impacting the bottom line.

Furthermore, teams want to be actively involved in selecting resources and determining how they’re pieced together, but not necessarily in managing and maintaining them. 66% respondents prefer to build in-house, as Big Data represents a huge professional opportunity — but in reality they’re often forced to spend too much time fighting

technologies, and not enough time actually using them to create business value. Interestingly, 61% of respondents said managed, hosted platforms are a viable solution, indicating a willingness to outsource the back-end and focus on delivering value through application.

As a relatively new development, it may come as no surprise that participants list education and the ability to understand the platforms as top challenges on Big Data projects. The report findings also reflect a general talent gap when it comes to data analysis expertise, which companies must address before embarking on new projects. “Lack of Business Context Around the Data” (51%) and “Lack of Expertise to Connect the Dots” (51%) are identified as the top reasons Big Data projects fail. On a similar note, respondents say Data Scientists/Analysts and Business Users are more impactful to the bottom line than the C-Suite, indicating once again that those who can derive business value from data are in demand.

In general, it appears that business goals of Big Data projects are in sync with a general trend toward elasticity and business agility. Respondents listed the ability to scale, ease of management, flexible architecture and speed to deployment/security as the top four most critical requirements for Big Data platforms.

Executive Summary

Executive Summary

Much of what we found speaks to the importance of creating a business culture that supports Big Data. For example, participants rated “Gathering Data from Siloed Sources” as their “Most Significant” Big Data challenge. Breaking down barriers to support a data unification initiative can require executive ‘air cover,’ and a lot of socializing. Also, having the patience to define very specific project parameters also appears to be of paramount importance, with

58% of respondents listing “Inaccurate Scope” as the top reason IT projects fail.

As a best practice, it is advisable to start with a single key business use case with a clearly stated objective that impacts the business.

Finally, while much of today’s Big Data dialogue centers on Hadoop, it’s worth noting that respondents rated Batch (53 %) and Real-time (49%) data processing as nearly equally important components of business decision-making. Clearly, as a batch-oriented system Hadoop isn’t enough.

In general, we find that Big Data projects often fall short or take longer than planned due to challenges related to education, social issues, and integration work. While IT staff may be fully aware of the obstacles to Big Data success, and the commitment a project requires, they may be less aware of options to make their jobs easier, such as partnering with an outside vendor that can help speed time to business value.

If you wanted your CIO to know one thing...

“To benefit fully from Big Data, you must

look beyond the limitation of what you

think can be extracted from the data

available and instead think about and ask

for what you wish you could

derive/learn/extract from the data.”

-- Survey Part icipant

CIOS & BIG DATA : what your it team wants you to know

In general, we found that companies are taking Big Data seriously in 2013, ranking it high on the list of IT priorities. In fact,

81%of companies have Big Data/Advanced Analytics Projects in their Top 5 2013 IT priorities.

Weighing the current business environment — which is growing more competitive, global and erratic by the moment — against the real-time, predictive insight of a well-run Big Data initiative, the remaining 19 percent ought to be asking themselves why they’re not more heavily invested in Big Data.

Emphasizing Application Development Over Infrastructure Maintenance

The survey showed that while IT staff do not want to divorce themselves from having a say in how the various components of the stack are chosen and assembled, they would prefer to focus on the application. Top priorities other than Big Data for IT staff include mobile, website, security and dashboarding, all of which reside in the application layer, not in the infrastructure.

For CIOs, this may provide a guide for 2013 investment strategy. However, such a strategy would not be based solely on work preferences. Responses also indicate that many back-end functions are the very things that devour the most staff time — within Big Data projects, processing (43%), analyzing (41%) and ongoing management (42%) of the data pose significant difficulties for the IT team.

If companies want to keep their IT staff focused on leveraging Big Data to impact the bottom line, they should consider outsourcing these functions.

66% of respondents prefer to build in-house,

which should not be surprising, as Big Data represents a tremendous opportunity for IT staff to build their talents and resumes. To spur innovation, constantly challenging them with career-advancing projects and technologies and empowering them with education is clearly a winning management strategy.

However, consider that this finding may be more indicative of a desire to be actively involved in selecting resources, tools and solutions and determining how they’re pieced together, rather than in managing and maintaining them. Teams are often forced to spend too much time fighting


technologies and not enough time actually using them to create business value. In short, those who could be answering the important questions and producing the greatest insights are stuck wrestling with Hadoop clusters and other Big Data technologies. Relegating the best and brightest to custodial functions that could easily be outsourced may very well cause them to leave in pursuit of more interesting opportunities. In the midst of an enterprise talent shortage, few companies can afford such a brain drain.

In light of this, CIOs may take comfort that

61% of respondents agreed that managed, hosted platforms are a viable solution.

In other words, there may be solid staff support for outsourcing the back-end to a managed, hosted solution, simplifying the process so the IT team can focus on what’s important — the application that delivers business value.

A Need to Educate on the Platforms and a General Lack of Big Data Expertise

As Big Data is a relatively new focus for most IT departments and companies in general, it comes as no surprise that most IT staff are feeling a bit squeezed when it comes to fully grasping the technologies. Respondents listed “Finding Talent” and “Finding the Right Tools” as top “Significant” challenges they face when working with Big Data.

On a related note, other findings are indicative of a general data analysis talent gap, which companies need to address before Bid Data projects can succeed. Participants cite “Lack of Business Context Around the Data” (51%) and “Lack of Expertise to Connect the Dots” (51%) as the top reasons advanced analytics projects fail.

Respondents also say that data scientists, data analysts and business users are more impactful to the bottom line than the C-Suite, once again illustrating the value that they place on those who can derive actionable business insight from data. Given that such skill sets are in short supply, companies may look to bridge this talent gap by offering training to existing staff as well as using the tools that simplify the data analysis process, except turning current staff into data scientists.

80% 76%

finding the

right tools

finding the

talent


“Understanding the hidden trends in all

the data can help us better understand”

-- Survey Part icipant

Identifying Exactly What You Want Big Data to Do for Your Organization

When it comes Big Data, it’s critical that companies clearly define what they want from the project before moving forward — 58% of respondents listed “Inaccurate Scope” as the top reason IT projects fail. While the promise of what can be done with Big Data may seem boundless, implementing a system with the idea that it will meet the entire organization’s needs is most likely a recipe for failure. Rather, start with a very narrow scope focused on one key business use case. Once on course to achieving objectives with this initial use case, expand to the next, building and iterating your system over time.

Additionally, companies need to have a firm idea of what they want their Big Data platform to look like, and how it should perform.

According to survey participants, the four Most Critical Requirements for Big Data platforms are:

• Ability to Scale - as data grows, architecture should keep up

• Ease of Management - it should be straightforward to maintain

• Flexible Architecture - it should be adaptable to the environment and data

• Speed to Deployment - fast time to business value

Hadoop Isn’t Enough

Hadoop may be the most frequently cited component in the era of Big Data, but the survey makes it clear that, on its own, it is not sufficient to offer businesses truly competitive insight. Respondents rated Batch (53 %) and Real-time (49%) data processing as nearly equally important components of business decision-making


“Talk to your Sys

Admins about your big

data project. The biggest

disconnect is between

the Sys Admin and the

C-Level Executive.”

-- Joe Kelly

COO, Infochimps


“The need for a combination of real-time

and batch analysis frameworks.

Discussions tend to emphasize one over

the other, instead of solving both

problems with a single framework.

Priorities tend to be around real-time

access, with data warehousing solutions

dropped.” -- Survey Part icipant

Break Down Silos and the Culture that Produces Them Participant responses shed light not only on technical challenges, but also a corporate culture that stands in the way of Big Data project success by siloing data into the applications of various departments which rarely communicate with each other. If “breaking down business silos” has been a mantra in companies for a while now, perhaps it hasn’t been repeated enough — respondents rate “Gathering Data from Siloed Sources” as their most significant Big Data challenge.

Respondents listed “Budgets for Big Data Projects,” (65%) “Time to Work on the Project (75%),” and “Finding the Right Tools (76%)” as “Significant” challenges they face when working with Big Data.

Big Data is complex, and projects often take longer than planned due to education demands and challenges related to new technologies, corporate culture and integration. However if IT staff has the budget, time and tools to get the job done right, enterprises can accompish their goals. Additionally, keep in mind that while IT staff may be fully aware of the commitment and resources a Big Data project requires, they may be less aware of options to make their jobs easier. Partnering with an outside vendor can speed implementation and accelerate time to business value. Exposing the team to successful Big Data technologies and processes, while offering the expert guidance to turn IT staff into Big Data experts, is a key strategy CIOs should consider for their Big Data products.

About Infochimps

Infochimps was founded by data scientists and cloud computing experts with the belief that by effectively leveraging Big Data, enterprises make better decisions. The company’s solutions make it faster, easier and far less complex to build and manage Big Data projects and quickly deliver actionable insights. With the Infochimps Enterprise Cloud, companies benefit from the fastest way to deploy Big Data environments in public, virtual private and private clouds. Infochimps is a privately held, venture-backed company with offices in Austin, TX and the Silicon Valley.


“Time and resource requirements”... “It

takes time to interact with data”... “the

time it takes to normalize data so that it

can be analyzed properly”... “That it has

inertia, and structural changes don't

happen quickly”...“time to put in place”...

“time to develop”... “Time and effort

required”... “This will take more time not

more people”... “Time”... “nothing is fast”


“In our experience, breaking down barriers to support a data unification

initiative can require executive ‘air cover,’ and a

lot of socializing.”

-- Jim Kaskade

CEO, Infochimps

Contact Us

Infochimps, Inc1214 W 6th St. Suite 202Austin, TX 787031-855-DATA-FUN (1-855-328-2386)[email protected]: @infochimps

Schedule a free demo of Infochimps Enterprise Cloud for Big DataGet a demo with of the leading

solution for your enterprise big data

project. The Infochimps Enterprise

Cloud for Big Data solution makes

it faster, easier and far less complex

for businesses to build and manage

Big Data projects and quickly

deliver actionable insights. With

the Infochimps Enterprise Cloud,

companies benefit from the fastest

way to deploy Big Data environments

in public, virtual private and private

clouds.

www.infochimps.com/demo

© 2013 Infochimps, Inc. All rights reserved. All trademarks are the property of their respective owners. Infomation is subject to change without notice.

http://bigdata.infochimps.com/request-a-big-data-platform-demo/

















www.infochimps.com/demo

Technology

Report: CIOs & Big Data