2016 DSG Webinar Azure HDInsight 2 V4

How to Act on Big Data in Real Time – Part 2

Name · Title · Dunn SolutionsJanani EshwaranRenata Simanjuntak

Today’s Agenda

Introduction

Impact of Real Time Analytics in Telecom Industry

Microsoft Azure HDInsight

Demo –Telecom Real Time Fraud Detection

What’s Next

Dunn Solutions is a Full-Service IT Consulting FirmFounded in 1988

Raleigh, NCDelivery Training

Bangalore, IndiaDelivery

MinneapolisDelivery Training Chicago

Delivery

Practice Areas

Application Development• Portals• eCommerce &

Content Managed Websites

• Mobile App Development

• Custom App Development

Training• Certified

SAP/Liferay• Classroom, On-

site, Computer Based & Virtual

• Mentoring & Custom Training

Frameworks• Accountable

Care Orgs (ACO’s)

• Corporate Legal • Higher

Education• Optical Shop

Solutions

Analytics• Analytics & BI

Platforms• Data

Warehouse & Data Integration

Big DataPredictive Analytics

Selected Clients

PartnershipsPartnerships

Analytics Practice

Business Intelligence

Big Data

Data IntegrationBusiness Analytics

Data Warehousing

• KPI’s and Metrics• Dashboards• Data Exploration

and Visualization• Ad Hoc Analysis &

Reporting

• Data Mining• Predictive

Analytics• Prescriptive

Analytics• R, AzureML

• Hadoop, MapReduce

• AWS and Azure• Hive, Sqoop, Spark• NoSQL

• Data Lake• Columnar• In-memory• EIM (Data

Integration and Data Quality

• Dimensional Modeling

Today’s Agenda

Introduction

Impact of Real Time Analytics in Telecom Industry

Microsoft Azure HDInsight

Demo –Telecom Real Time Fraud Detection

What’s Next

Real Time Big Data Analytics

• It is not only to store and analyze streaming BIG data• It is more about making better decision

and taking meaningful action at the right time

Traditional Enterprise Data Warehouse plus analytics are no longer enough

• Fraud detection while a credit card is swiped

• Triggering an offer while a shopper is standing on a checkout line

• Placing an ad on a website while someone is reading a specific article

Real Time Business Benefit

• Service improvement• Cost savings• Fraud detection• Keep up with customer trends• Sales insights enhancement• Instantly errors detection• Immediate new strategies of your

competition notification

Power of Real Time Analytics

• Financial Loss• External Confidence• Company Morale• Increased Audit Costs

How Fraud Hurts You & Your Organization

Impact of Fraud in Telecom Industry

Top 5 Fraud Type in Telcom Industry

Communications Fraud Control Association (CFCA) Global Fraud Loss Survey

Telecom Fraud cost the industry 2015 over 38 Billion USD annually

Today’s Agenda

Introduction

Impact of Real Time Analytics in Telecom Industry

Microsoft Azure HDInsight

Demo –Telecom Real Time Fraud Detection

What’s Next

Azure Event Hub

Customer Name / 16

• Benefits:• Stream millions of events per second• Process events with variable load profiles• Connect millions of devices across platforms

• How much data? Throughput units• To scale the traffic coming in or out• Key pricing parameter• In (Publisher): 1mb or 1000 events/sec• Out (Consumer): 2mb/sec

Azure HDInsight - Ecosystem

• Big Data with No Hassle• Open and

Flexible• Insight in MS

Excel• Build Big Data

Apps your Way

• Scalable• High-throughput• Fault-tolerant • Stream processing of live data streams• Data collected can be later post-processed • Code and business logic can be shared and reused

Spark Streaming for Real Time Analytics

SparkStreaming

SparkEngine

Input Data Stream

Batches of input data

Batches of processed data

Less time learning, implementing, and maintaining different frameworksMore focus on developing smarter applications

Hive

Customer Name / 19

• Data warehouse in Hadoop• Project structure on largely unstructured

data• Work with structured and semi-structured

data

• Hive QL

• Low cost data storage

• Empower user• Q&A function• Dashboard visualization• Innovative technology• In memory engine• Columnar database

• You own your data• Faster turn around• Lower cost

Power BI

Customer Name / 20

Enterprise-level data is yours for free or at a very low monthly cost

• Email• Link• Website• Phone: text or call• Application

Real Time Alert

Customer Name / 21

You are informed in real time when errors or frauds or anomalies existTake action in real time for real results

Today’s Agenda

Introduction

Impact of Real Time Analytics in Telecom Industry

Microsoft Azure HDInsight

Demo –Telecom Real Time Fraud Detection

What’s Next

Big Data Real Time Use Case for Telecom (1)

• Real-time fraud prevention• Can be passed on to customer bills• Prevent revenue loss and additional expense to

correct • Visibility of service performance, costs and

discounts to the customer• Cannot monitor customer bills to provide services• Analyze and offer products and discounts

• Optimization of Least Cost Routing (LCR) • Choose lost cost network in real time • Select optimized and high performing network

quickly

Big Data Real Time Use Case for Telecom (2)

• Call performance monitoring • Cannot prevent dropped calls and issues• Can identify issues to resolve immediately

• Real-time profitability analysis • Make use of long term trend data offline• Can learn service provided to customer for

understanding gross margin

What if I wanted to…

• Capture data from any application in real time• Store the data• Perform analysis on the streamed data• Visualize the information interactively

Demonstration: Setting the Stage

Big Data, Real time project checklist• Azure Event Hub• Azure HDInsight cluster

• Spark Streaming• Hive

• Azure SQL database• Power BI• Real Time Notification (email)

What Do I Need?

Our Demo Architecture

Streaming data sources

(Call Records)

Azure Event Hub

Consume Store

Streaming data

Enrichment data

Step 1: Start The Event Hub Event To Collect The Events

Streaming data sources (Call Records)

Azure Event Hub

Step 2: Prepare Your Receiver To Receive Events

Streaming data sources (Call Records)

Azure Event Hub

Consume

Step 3: Persist The Events In Hive Table

Streaming data sources (Call Records)

Azure Event Hub

Consume Store

Step 4: Alert /Notify the anomaly

Streaming data sources (Call Records)

Azure Event Hub

Consume Store

Step 5: Visualization in Power BI

Streaming data sources

(Call Records)

Azure Event Hub

Consume Store

Streaming data

Enrichment data

Today’s Agenda

Introduction

Impact of Real Time Analytics in Telecom Industry

Microsoft Azure HDInsight

Demo –Telecom Real Time Fraud Detection

What’s Next

Recap

Streaming data sources

(Call Records)

Azure Event Hub

Consume Store

Streaming data

Enrichment data

1. Capture streaming data

2. Process and store data

4. Analyze

3. Alert by email

• What is your Big Data strategy?• Do you have a Big Data project in mind?• Are you wondering how you can use Big Data

for real time data analysis to benefit your company?• Should you do it on premise or in the cloud?

• Contact us and we’ll help you execute!• info@dunnsolutions.com

Let’s Get You Started with Real-Time Big Data

Thank YouJanani Eshwaran· Analytics Consultant · Dunn Solutions

jeshwaran@dunnsolutions.com

Renata Simanjuntak· Analytics Manager· Dunn Solutionsrenatas@dunnsolutions.com

• http://cfca.org/fraudlosssurvey/2015.pdf

• http://www.cfca.org/pdf/survey/CFCA2013GlobalFraudLossSurvey-pressrelease.pdf

• Microsoft blogs and tutorials

• Other analytics webinars:• http://www.dunnsolutions.com/content/webinar-white-

paper

Reference