november 2015 Business Informationcdn.ttgtmedia.com/searchBusinessAnalytics/... · machine learning based on adaptive weighting and approximation. These techniques are just an overview

Editor’s Note Tricks of the Analytics Trade Think Before You Buy

november 2015

Business Information InsIght on managIng and usIng data

2015O N L I N E

specIal Issue

The Big Data Universe Expandslower cost and greater accessibility mean more companies can capitalize on the huge benefits of big data analytics software.

home

edItor’s note

the bIg data analytIcs benefIt

trIcks of the analytIcs trade

thInk before you buy

2 business information • november 2015

BIg DATA ANAlYTIcs requires skilled business analysts and data scientists to be successful, but those gifted profes-sionals also need to be equipped with the proper tools of the trade. And the right tools can be equally as important as the analytics team that’s striving to achieve a project’s intended outcome.

The techniques big data analytics tools provide aren’t all that new. It’s just that, until recently, the arena of tools and adopters has been, for good reason, relatively small. The processing and data storage demands of advanced analytics applications, not to mention the high cost, have limited their adoption primarily to large companies with deep pockets. Fortunately, those once insurmountable barriers have lowered dramatically.

As the barriers have come down, the number of com-panies looking to take advantage of big data has gone up. In a June 2015 Gartner survey, 76% of the 437 respon-dents said their organizations were investing in big data technologies or planned to within two years. But nearly half of those with investment plans—43%—weren’t sure they would see a positive return. Those doubts highlight the need for analytics tools that can effectively support the big data uses a company has in mind.

In the first feature of this special issue of Business Infor-mation, David Loshin, managing director at consultancy

DecisionWorx, explains how the growing availability of big data platforms and less expensive analytics tools is making it possible for more companies to adopt predic-tive and prescriptive analytics applications. In addition, data mining algorithms have been adapted to empower even the mainstream business user to work with massive volumes of data from many sources.

That all sounds good, but is your company ready to make the investment? Loshin reveals in his second fea-ture what you need to consider before taking the plunge. A lot depends on your company’s data and business appli-cations and, most important, whether your organization has what it takes to integrate big data analytics into its technology landscape.

Then turn your focus to the big data analytics market. In his third feature, Loshin details the essential criteria for selecting software products, which include system integration, performance quality and vendor support. Finally, there’s the simplest and most basic consider-ation of all: How much does it all cost? That can only be answered by vendors lucky enough to receive your com-pleted, criteria-laden request for proposal. n

Is your company on the hunt for a big data analytics tool? Write to me at [email protected].

EDITor’s NoTE | ron karjIan

Through the Big Data gauntlet

mailto:[email protected]


homE

gETTINg sTArTED | davId loshIn

ThE BIg DATA ANAlYTIcs BENEfITbig data analytics is a trending practice that many companies are adopting. but before jumping in and buying big data tools, organizations should first get to know the landscape.

The deployment and use of big dataanalytics tools can help companies improve operational efficiency, drive new revenue and gain competitive advantages over business rivals. But analytics applications come in many different varieties.

Descriptive analytics, the lion’s share of the analysis performed in most organizations, describes what has al-ready happened and suggests its root causes. It typically hinges on basic querying, reporting and visualization of historical data.

Alternatively, more complex predictive and prescrip-tive modeling can help companies anticipate business opportunities and make decisions that increase sales and profits. With predictive analytics, historical data sets are mined for patterns indicative of future situations and behaviors, while prescriptive analytics uses the results of predictive analytics to suggest a course of action.

In many environments, the processing and data stor-age demands of advanced analytics applications have

home

edItor’s note





limited their adoption—but fortunately those barriers are beginning to crumble. The growing availability of big data platforms and analytics tools has made it possible for predictive and prescriptive analytics applications to scale so they can handle massive data volumes originat-ing from a wide variety of sources.

Big data analytics tools essentially are software prod-ucts that support predictive and prescriptive analytics applications and run on big data computing platforms. Those platforms typically are parallel processing systems based on clusters of commodity servers, scalable dis-tributed storage and technologies such as Hadoop and NoSQL databases. The tools layered on top of them al-low users to rapidly analyze large amounts of data, often within a real-time window.

Big data tools analyze information, discover patterns and generate analytical models that recognize and re-act to those patterns. And they can enhance business processes by embedding the analytical models within the corresponding operational applications. At a freight carrier, for example, a combination of data on shipping deliveries, traffic, weather and historical vendor perfor-mance can be analyzed. Then a model can be devised for picking the best shipping subcontractors within geo-graphic regions in an effort to limit the risks of late deliv-ery or damaged goods.

Analytics tools supporting big data analytics initiatives ingest a wide variety of data types: structured data with defined and consistent fields, such as transaction data stored in relational databases; semi-structured data, such

as Web server or mobile application log files; and un-structured data, such as text files, documents, email, text messages and social media posts.

Powering AnalyticsSearching online for big data analytics produces a long list of vendors, but many only provide platforms and tools that support the analytics process—data integration, data preparation and other types of data management software. Our focus here is on tools that are designed for analyzing data and that meet the following criteria:

n Provide analysts with advanced analytics algorithms and models.

n Engineered to run on big data platforms such as Ha-doop or specialty high-performance analytics systems.

n Easily adaptable to using structured and unstructured data from multiple sources.

n Able to scale as more data is incorporated into analytical models.

n Have analytical models that easily can be or already are integrated with data visualization and presentation tools.

In addition, the tools must include built-in algorithms and methods that support typical data mining tech-niques, including the following:

n Clustering and segmentation refer to the division of a large collection of entities into smaller groups that


home

edItor’s note





exhibit similarities. This technique can be used, for ex-ample, to analyze a collection of customers and differ-entiate smaller segments for targeted marketing.

n Classification is the process of organizing data into predefined classes based on attributes pre-selected by an analyst or identified through a clustering model. An example is using a segmentation model to categorize a new customer.

n Regression is used to discover relationships between a dependent variable and one or more independent variables. It helps determine how the dependent vari-able’s values change in relation to the independent variable’s values. For example, an analyst working with a construction company could use geographic loca-tion, mean income, average summer temperature and square footage to predict the future value of a property.

n Association and item-set mining look for statistically relevant relationships among variables in a large data set. This could enable call-center representatives, for example, to offer specific incentives based on a caller’s customer segment, duration of relationship with the company and type of complaint.

n Similarity and correlation are used to inform undi-rected clustering algorithms. Similarity-scoring algo-rithms can determine what the entities in a candidate cluster have in common.

n Neural networks are used in undirected analysis for machine learning based on adaptive weighting and approximation.

These techniques are just an overview of the types of analyses used for predictive and prescriptive analytics. Different vendors are likely to provide a variety of algo-rithms supporting each of the methods.

Browsing the marketThe market for advanced analytics tools has evolved, and the types of tools available vary by maturity, capability and ease of use. There are tools with relatively long his-tories from mega-vendors such as IBM, Oracle and SAS Institute. Other large vendors have acquired companies whose tools have a more recent history. Those vendors include Dell, Microsoft, SAP and Teradata. A number of smaller companies provide big data analytics products as well, including Angoss Software, Predixion Soft-ware, Alteryx, Alpine Data Labs, Pentaho, KNIME and RapidMiner.

Some companies have developed their own set of algo-rithms. Others have adopted the open source R statistical analysis language to provide predictive and prescriptive modeling. Commercial products based on R are also available, and users can tap other open source technolo-gies, including Weka and Apache Mahout.

In some cases, especially with major vendors, the big data analytics tools are incorporated into larger big data enterprise suites. In other instances, the tools are sold


home

edItor’s note






as standalone products, and it’s the customer’s job to in-tegrate with the big data platform being deployed. Most tools provide a visual interface to guide the analytics pro-cesses, and, in most cases, the vendors provide guidance and services to get the customer up and running.

getting to Know UsersSome business users are looking to explore and devise new predictive models, some look to embed these models within their business processes and others want to understand the overall impact these tools will have on the business. Therefore, organizations adopting a big data analytics tool need to accommodate a variety of users.

Data scientists perform complex analyses involving complex data types. They’re familiar with the way under-lying models are designed and implemented, so they can assess inherent dependencies or biases.

Business analysts are more casual users. They’re likely to use the tools for proactive data discovery, visualization of existing information and some predictive analytics.

Business managers want to understand the models and conclusions. And IT developers support the data sci-entists, business analysts and business managers.

All of these users typically work together in the model development lifecycle. Data scientists subject sets of big data to undirected analyses and look for patterns of inter-est to the business. Along with the business analysts, they review models and evaluate how each of the discovered models or patterns can affect the company’s operations.

Business managers and IT teams then embed or integrate the models into business processes or devise new pro-cesses around the models.

Many of the early users of big data technologies were Internet companies—such as Google, Yahoo, Facebook, LinkedIn and Netflix—or analytics service providers. These companies relied on operational and analytical applications requiring fast-flowing streams of data that were ingested, processed and analyzed. The results were then used in an effort to continuously improve business performance.

Now, the appetite for data is expanding among compa-nies in mainstream industries. At one time, the financial outlay for a large-scale analytics platform would have been prohibitive to all but the very largest businesses. But barriers to entry have been lowered lately, thanks to the availability of hosted big data platforms like those from Amazon Web Services and the improved ability to install and run big data systems such as Hadoop clusters on premises. In addition, open data sets and accessibility to constant data from social media channels provide the raw material for larger-scale analyses when blended with internal data sets.

And although large businesses may still opt for high-end big data analytics tools, the lower-cost and open source alternatives that have become available are allow-ing small and medium-sized businesses to evaluate and launch analytics programs and potentially achieve the same kinds of business improvements that larger organi-zations are seeing. n


homE

TrIcKs of ThE ANAlYTIcs TrADEbig data analytics tools have a lot to offer and come in many varieties. get to know some of the ways business users and data scientists can use the software.

mANAgEmENT | davId loshIn

Three characteristics differentiatebig data analytics from other forms of analytics: the volume, scale and diversity of the data. Historically, an-alytical models often were built and then “trained” through a test and re-finement process using sample data sets pulled from very large databases.

But with computing platforms now providing scalable storage and computational ability, there are few limita-tions on the volume of data that can be analyzed.

That clears the way for real-time predictive analytics accessing large amounts of data, which can lead to im-proved business performance by enabling companies to respond quickly to market trends, changes in customer sentiment and other developments. It all depends, though, on an organization’s ability to blend and analyze the different types of big data listed here:

Transaction data. A big data platform can capture greater volumes of structured transaction data spanning

home

edItor’s note





much greater periods of time than more conventional data management systems typically can. A broader array of transaction types can be analyzed, extending in the case of a retailer beyond point-of-sale or e-commerce purchases to include “behavioral transactions” such as clicks on a website.

human-generated data. Unstructured data contained in email messages, documents, images, audio files, video files, blog posts, wikis and especially social media chan-nels is fertile ground for text analytics and other forms of analysis.

mobile data. Internet-connected smartphones and tab-lets are becoming ubiquitous, and the apps deployed on the devices are capable of tracking and communicating numerous events. These events range from in-app trans-actions, such as a product search, to demographic or status reporting, such as reporting a new geocode when there’s a change in location.

machine and sensor data. This is created or gener-ated by functional devices such as smart utility meters, intelligent thermostats, factory machinery and net-work-connected home appliances. These devices are prime examples of the Internet of Things, or IoT. The data streaming from the IoT can be used to build ana-lytical models that continuously monitor for predictive behavior and offer prescriptive directives on equipment maintenance.

Business UsesBig data analytics tools make it easier to analyze all the data being captured by companies—and they can be a reasonable investment if your organization considers using analytics in any of these areas:

customer analytics. This includes analyzing customer demographics, behaviors and characteristics to develop models for segmenting customers, predicting attrition and making next-best-offer recommendations to help with retention.

sales and marketing. There are two ways to apply marketing analytics. The first involves using analytical models to improve the way customer-facing applications make direct recommendations to the customer. The models help companies better identify opportunities for cross-selling and upselling, decrease abandoned online shopping carts and generally improve the accuracy of integrated recommendation engines. The second is more reflexive, showing the performance of the marketing group’s processes and campaigns and recommending adjustments to optimize that performance. For example, businesses can analyze which campaign best addressed the needs of specific segments.

social media. Content from social media can be ana-lyzed to determine customer sentiment. Companies can also identify brand risks when negative information is promulgated about a company’s products.


home

edItor’s note





cybersecurity. Massive cybersecurity attacks on com-panies such as Target, Sony, Anthem and Home Depot highlight a growing need for businesses to rapidly recog-nize potential attacks and stop them before they occur. To do this, companies can build analytical models that monitor network activities and access behaviors to iden-tify suspicious patterns.

fraud detection. An adjunct to the growing risk of identity theft is the increase in fraudulent activities and transactions. Financial institutions analyze billions of transactions to identify patterns of fraudulent behavior. Analytical models can also alert a bank customer to a potential fraudulent transaction.

Plant and facility management. As more devices and machines are Internet-enabled, organizations can col-lect and analyze continuous measurements of power usage, temperature, humidity and contaminant particles. Models can be developed to predict equipment failures, schedule pre-emptive maintenance and keep items in working order without interruption.

Pipeline management. Energy pipelines are increasingly fitted with sensors and communications capabilities. Continuous streams of sensor data can be analyzed for lo-cal and global issues that indicate a need for observation or maintenance.

supply chain and distribution channels. Analyzing

warehouse inventory, point-of-sale transactions and shipments for a variety of industries—such as trucking or shipping—produces predictive analytical models. These models can play an important role in inventory management strategies, logistics management and route optimization.

Price optimization. Retailers trying to maximize overall profitability may develop analytical models that combine a variety of data streams, including competitors’ prices, sales transactions across geographic regions and infor-mation on production, inventories and the supply chain. These models can determine when it’s best to raise or lower product prices according to supply and demand.

All these big data uses share characteristics: The anal-ysis involves both structured and unstructured data, the data is being accessed or streamed from a variety of sources and the volumes are potentially massive. Also, the results yield analytical models that can identify pat-terns in real time.


ANAlYTIcAl moDEls ThAT moNITor NETworK AcTIv-ITIEs AND IDENTIfY sUsPI-cIoUs BEhAvIor cAN hElP PrEvENT cYBErATTAcKs.

home

edItor’s note





Easier EntryThree key factors have lowered the barrier to entry and enabled a broader spectrum of organizations to consider adopting big data analytics technology: cost, simplicity and performance.

Early products had high price tags and offered expen-sive after-sale services for integration and deployment. In contrast, many tools now have pricing models that make adoption more economically feasible.

These products are also increasingly designed for nonexperts. Older tools had a limited audience, mostly statisticians and mathematicians who built models and understood the details of how they worked. With today’s products, users don’t need an advanced science degree to implement the analytical models generated by the tools.

In addition, scalable platforms accommodate the data volumes and computational needs of big data analytics. Today’s open source platforms enable massively parallel processing over distributed storage frameworks deployed on commodity hardware. The price and performance ra-tios of these platforms are much lower than they were in the past.

With the barriers to adopting big data analytics soft-ware lowering, forward-thinking organizations are rap-idly trying and integrating these tools. These companies are driven by data and analytics and recognize the poten-

tial value of information. Their key business stakeholders are aware there’s a wide variety of data sources—some static and many others dynamic—that can contribute to analytics processes. They’re also open-minded and flexi-ble when considering new technologies. If your company exhibits some or all of these characteristics, it may be poised and ready to take advantage of big data analytics tools. n


wITh ToDAY’s BIg DATA ANAlYTIcs Tools, UsErs DoN’T NEED AN ADvANcED scIENcE DEgrEE To ImPlE-mENT ANAlYTIcAl moDEls.


homE

Big data analytics tools let users analyze a wide variety of information, from structured transaction data to social media posts, Web server log files and other forms of unstructured and semi-structured data.

When an organization decides to buy a big data ana-lytics tool, the decision makers must create a process for evaluating the available products and then find the one that best fits their needs and requirements. There are certain features and attributes that need to be considered in assessing tools. To start, companies should evaluate the breadth and depth of modeling techniques, integra-tion and accessibility and how easy a tool is to use.

Analytics capabilities have diverse levels of sophisti-cation, or breadth. Analytics models include regression techniques, classification and regression trees, time series that predict variable values based on an analysis of past trends and neural networks. Two aspects charac-terize the depth of the modeling techniques: the algo-rithmic sophistication and the flexibility of the modeling techniques.

ThINK BEforE YoU BUYbefore companies purchase a big data analytics tool, they must identify their specific needs and match them to available products.

EvAlUATIoN | davId loshIn

home

edItor’s note





So that begs the following questions: What level of expertise in data mining and predictive analytics is nec-essary to know what kind of model can be developed, and how can a model be built with a particular tool? Less experienced data analysts may be interested in products that provide a broad range of analytical capabilities, while more expert analysts and statisticians may prefer tools with greater depth of analytical models.

feature of the weekBig data analytics applications often rely on a growing number of internal and external data sources containing structured and unstructured data. That drives a need for data accessibility and systems integration. Take the fol-lowing steps when considering product features:

n Verify that tools can ingest the different types of unstructured data, including documents, email, images, videos, presentations and streams from social media channels.

n Compare how they connect to big data architectures, such as distributed data stored in Hadoop, as well as files managed within other types of scale-out storage—for example, NoSQL databases such as MongoDB or Apache Cassandra.

n Ensure interoperability is possible, especially when there’s an expectation to blend more traditional data management and business intelligence practices with

advanced analytics methodologies. Many analytics tools allow analytical models to be invoked through traditional SQL queries, which may be more familiar to many data analysts.

n Assess how well a tool can access other systems and feed results to established platforms for reporting and analysis.

Some big data analytics products have been built from the ground up by vendors. Others are based on the open source statistical analysis language R. In either scenario, companies should evaluate how easy a product is to use for analyzing data, developing models and determin-ing the efficacy and accuracy of the models. Evaluation should include the following:

Usability for business analysts. Check if the product provides visual methods that facilitate development and analytics, especially for business analysts without a statis-tical background.

flexibility in deployment for different business uses.

If the kinds of analyses your organization plans to do are somewhat limited and centered on more general uses—such as customer-lifetime value analysis, fraud analysis or attrition prevention—you may be able to tolerate less flexible techniques. But if your organization desires a broader and less constrained approach to analytics, look for a greater degree of modeling flexibility.


home

edItor’s note





model scoring. Products, sometimes augmented by ad-ditional tools, offer features that can help analysts auto-matically compare the accuracy, efficacy and predictive value of different models intended for similar business scenarios.

collaboration. Isolated analysis and development can lead to replicated efforts and uncoordinated results. Be-ing able to collaborate and share analytical models helps lower development costs while increasing consistency.

Performance EnhancersThe practical aspects of integrating a new technology into an organization must also be considered. It’s im-portant to understand system requirements and de-pendencies for installation, configuration and ongoing management. For example, the big data analytics tools that take advantage of the statistical models in R require that the open source environment be installed at the same time as the product. Understanding requirements includes identifying the platforms on which the product may be installed and determining the platforms that can embed the developed models and applications.

Other considerations are security for the designation of roles and access rights for the analytics process and for incorporating developed models into business appli-cations. So be sure to explore what options the product provides for authentication, authorization and access control.

Most high-end Hadoop platforms and specialty

appliances provide multiple compute nodes for parallel processing and distributed computing. If a high level of performance is required, it’s critical the products you evaluate take advantage of massively parallel processing (MPP). The selected tool must optimize MPP features.

Parallel systems work best when compute processes execute independently on data sets distributed to min-imize network bandwidth and maximize data locality. Therefore, you should review how a product’s paralleliza-tion dovetails with your data distribution strategy.

Analytics algorithms may also be able to take advan-tage of the inherent capabilities of the system stack’s other components. An example would be if a database management system provides parameterized modeling utilities that use the system’s architectural features. In this case, it’s wise for the analytics tool to use the native capability rather than attempt to replicate it.

Scalability and elasticity are also important. Assess how the different analytics products are intended to scale as data volumes expand and increased processing and storage capacity are needed.

Prices and ProposalsIn most cases, pricing understandably influences the buy-ing decision. Some big data analytics tools can be costly, but others are low-cost or sometimes even free. Pricing obviously will affect the features, capabilities and con-straints of the tool.

Another consideration is special services. For each product being evaluated, assess whether it’s necessary to


home

edItor’s note





depend on the software vendor or external experts for installation, training or specialty development services. Also, be sure to consider the long-term total cost of own-ership, which can include annual maintenance fees as well as associated costs for the system stack, operations and maintenance staff, data center space, cooling mecha-nisms and other utilities.

Once you’ve narrowed down your choice of vendors and products, it’s time to develop a request for proposal (RFP). You should create an RFP that—aside from the standard set of questions about integration, interopera-bility and corporate details—focuses on quantifying how well the product conforms with your expectations for factors such as analytical modeling, data volumes, neces-sary levels of expertise and data accessibility. Armed with your list of requirements, you will be ready to find the right big data analytics tool for your company. n

DAvID loshIN is a managing director at data management and analytics consultancy DecisionWorx. Loshin also is a frequent conference speaker, and he has written numerous books, including Big Data Analytics: From Strategic Planning to Enterprise Integra-tion with Tools, Techniques, NoSQL, and Graph. He can be reached through the DecisionWorx website, www.decisionworx.com.

Business Information is a SearchDataManagement.com e-publication.

ron Karjian, Managing Editor

moriah sargent, Associate Managing Editor

scott wallask, News Director

David Essex, Executive Editor

lauren horwitz, Executive Editor

Jan stafford, Executive Editor

craig stedman, Executive Editor

marty moore, Production Editor

linda Koury, Director of Online Design

Doug olender, Publisher, [email protected]

Annie matthews, Director of Sales, [email protected]

TechTarget, 275 Grove Street, Newton, MA 02466 www.techtarget.com

© 2015 TechTarget Inc. No part of this publication may be transmitted or reproduced in any form or by any means without written permission from the publisher. TechTarget reprints are available through The YGS Group.

About TechTarget: TechTarget publishes media for information technology professionals. More than 100 focused websites enable quick access to a deep store of news, advice and analysis about the technologies, products and pro-cesses crucial to your job. Our live and virtual events give you direct access to independent expert commentary and advice. At IT Knowledge Exchange, our social community, you can get advice and share solutions with peers and experts.

COVER PHOTOGRAPH: ALExSL/ISTOCKstay connected with us on social media.

linkedInfacebook


http://www.decisionworx.com

http://searchdatamanagement.techtarget.com/

http://reprints.ygsgroup.com/m/techtarget

http://www.linkedin.com/company/3002580

https://www.facebook.com/businessinformationmagazine

Documents

november 2015 Business Informationcdn.ttgtmedia.com/searchBusinessAnalytics/... · machine learning based on adaptive weighting and approximation. These techniques are just an overview