CTO Perspective on Capturing the Potential of Big Data in a Service Provider

8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider

1/17

CTO Perspective on Capturing tPotential of Big Data in a ServicProviderJrgen UrbanskiBoard Member Big Data & Analytics of BITKOM (German IT Industry Views and opinions expressed are not necessarily those of his emplo

T-Systems, the enterprise arm of Deutsche Telekom.


2/17

Big Data 1956

IBM 305

RAMAC5 MB!


3/17

Volume

Velocity

Variety

Big Data 1956

IBM 305

RAMAC5 MB!

Big Data 2013


4/17

Big Data Use Cases for Telcos

Business Intelligence Marketing & Sales Service & Operat

! 360 degree view ofcustomer value

! Targeted TV advertising,serving the mostprofitable TV ad to

individual viewers basedon analysis of their

viewing behavior (e.g.,what ads prompt theuser to switch channels)

! Personalized marketingcampaigns, integratinganalytics across various

marketing channels

! Big data as a product

! Network maintenance anoperational intelligence for planning and customer exp

! Analytics and archiving of c(CDRs) on Hadoop for com

disputes, and congestion m

! Real time analytics of CDRand in-memory technologyon pre-paid services (e.g.,

! Security analytics (e.g., intr! M2M device telemetry anal

security and assisted living

! Log analytics for customer

! Enterprise datawarehouse offload

- Data landing zone- Active archive- Enterprise-wide data

lake augmenting theEDW

! Mainframe offload

1 3

2

= HighN

* A large mobile carrier might reach 1 billion new CDRs, ingesting 20 TB per day


5/17

The Situation

! Many EDWs are at capacity! Running out of budget before running out of

relevant data

! Older data archived in the dark, notavailable for exploration

Enterprise Data Warehouse Offload

Operational (44%)

ETL Processing (42%)

Analytics (11%)

DATAWAREHOUSE

Storage & Process

HADOOP

1

Operational (50%

Analytics (50%)

DATAWAREHOUS

The Solution

Cost is

1/10th

! Hadoop for data storage and pparse, cleanse, apply structuretransform

! Free EDW for valuable queries! Retain all data for analysis!


6/17

! Europes leading real estatemarketplace with data on...

! 1m properties listed currently! 20m properties cumulative! 6m saved searches! Geographical coordinates! Enriched by socio-demographic data on

19m properties

! Team! Product manager! Data scientists! 2 scrum teams

Data Products: ImmobilienScout (a DT subsidiary)2

! Market Navigator serv! Supports realtors in acquiri

customers

! Local market analysis helpssetting for rent and buy

! Integrates third-party data! Functionality includes! Price heat maps and trendi! Demand- and supply-side i! Local area information! Comparable transactions

The Situation The Solution


7/17

Turning Big Data into Products!2


8/17

! Analyze node utilization against customer experience indicators to secorrelates to experience

! Increase of dropped packets (IPDR data)! Increase of calls into the call center for customers associated to the

(customer experience data)

! Increase of requests to drop service (work order data)

Network Maintenance and Upgrades to Improve the

Customer Experience The Situation3

Approach

Challenge!

Poor visibility into how cable network congestion affects churn, and wnetwork upgrades produce the most incremental revenue

Hypothesis! Nodes considered to be causing customer experience issues can be p

maintenance and upgrade based on the value of the customers serve

node

Source: Zaloni project for a large cable provider


9/17

! Analysis that integratessubscriber and network

node data to seecorrelations betweennetwork congestion andcustomer experience

! 11 different data sources! 4m subscriber records, 12m

work orders, 9m calls, 42mIPDRs1, 20m Tivoli NPMs2

! Finding:Only smallnumber of nodes areresponsible for majority ofthe negative customerexperience

3

Source: Zaloni project for a large cable provider

1 IPDR = Internet Protocol Detail Record, provides information about IP-based service usage, usually to inform OSS and2 NPM = NetView Performance Monitor Messages.

NetworkNode

TNMPCMTS

Performanc

CompetitiveSpendData

SubscriberSubscriber

MasterSubscriber

Record

MarketingDemo-

graphics

CallerExperience

WorkOrders

Products

Equipment

Network Maintenance and Upgrades to Improve the

Customer Experience The Solution


10/17

Approach to Execution

Implications:

! Technology st! Target archite! Vendor select! New processe! New skills!

Privacy consid! Etc.

Architecture& Design Proof ofConcept Pilot

Production

Implemen-tation

Training1

1 Architect, Developer, Data Science and Admin

Agile

learningon eachproject

Programmaticsteering

Supply-side requirements:

! Higher capital efficiency! Lower upfront investment! Faster time to value! More rapid innovation! Market-facing differentiation!

Security & compliance

+A

= HighN

B

C


11/17

Technology Strategy: From Data Puddles to Data Lak

AVOID:

Systems separated by workloadtype due to contention

BigData

BU1

BigData

BU2

BigData

BU3

GOAL:

Platform that natively sumixed workloads as share

Big DataTransactions, Interactions, Obs

Refine Explore

Batch Interactive

A


12/17

Target Architecture: Modular against a Fragmented E

Physical Infrastructure

Data Processing

Batch ProcessingStreaming & Complex

Event ProcessingSearch

DataIntegration &Governance

Extract,Transform,

Load

Real Time &Batch

Ingestion

DataConnectors

Life CycleManagement

DEnc

Data I& Mten

IdenAc

Mana

Data

CusGat

1 Includes key value, document, graph and object data bases.

Application

Data Mining& Predictive

Geo-spatialVideo &Audio

Web &Social Media

Text &Semantics

OLAP

Presentation

AdvancedVisualization

ClientsReal-TimeMonitoring

Reports &Dashboards

Data Management

DistributedStorage &

Processing

(Scale-out)Relational DB

NoSQL DB1In-memory

DB(MPP) EDW

B


13/17

Hadoop Is the Foundation for Much of the Innovation

Physical Infrastructure

Data Processing

Batch ProcessingStreaming & Complex

Event ProcessingSearch2

DataIntegration &Governance

Extract,Transform,

Load

Real Time &Batch

Ingestion

DataConnectors

Life CycleManagement

DEnc

Data I& Mten

IdenAc

Mana

Data

CusGat

1 Includes key value, document, graph and object data bases.2 Solr and Lucene open source projects, also applicable outside Hadoop.

Application

Data Mining& Predictive

Geo-spatialVideo &Audio

Web &Social Media

Text &Semantics

OLAP

Hadoop Projects & Ecosystem Ad

Presentation

AdvancedVisualization

ClientsReal-TimeMonitoring

Reports &Dashboards

Data Management

DistributedStorage &

Processing

(Scale-out)Relational DB

NoSQL DB1In-memory

DB(MPP) EDW

Store first, ask

questions later (HDFS)

Parallel scale-out

processing (MapReduce)

Much cheaper

storage

Any data type,

including unstructured


14/17

Vendor Selection ConsiderationsCategory Select requirements

DataManagement

! Support of batch, interactive, online & streaming use cases! Full data lifecycle management

Operations! Rolling upgrades without service disruption and fallback capability! Support for end-to-end management and automation frameworks (e.g., P

Security! Granular role-based access control via AD, LDAP, Kerberos! Tenant, data, network and namespace separation in all services! Auditability

Infrastructure! Deployment flexibility: virtual on-premise environments, public clouds, ap

Strategic Fit

! Relevance of ISV ecosystem (notably EDW & BI) that is certified! Avoidance of vendor lock-in (open source vs. proprietary)

EXTC


15/17

! In 3 years, 50% of new data for enterpriseworkloads will land on Hadoop! Big Data can deliver value in every function of a telco! Big Data has a high return-on-investment, if you master the learning curve! Operators who embrace Hadoop today will see their business performance

away from those who are late to join the new world of Big Data

Our perspective


16/17

! Start Small, Grow Tall: Debunking Three Big Data Myths (Link)! WIRED Innovation Insights! October 4, 2013! Enterprises dont need petabytes of data, a small army of data scientists, not even a big budget to get a me

start with Big Data -- thanks to Hadoop.

! Hadoops Second Generation Offers More To Enterprises (Link)! Information Week! October 2, 2013! The first Hadoop tools weren't easy to deploy or manage. But the second-wave tools deliver great advances

! Hadoop! Coming soon to an enterprise data warehouse near you (Link)! TDWI! June 2013! Deutsche Telekoms perspective on how the open-source Hadoop ecosystem delivers powerful innovation

databases and business intelligence at a fraction of the cost of legacy systems.

! Been there, forked that: What the Unix-Linux schism can teach us about Hadoops futu! GigaOm! June 26, 2013! Concerned about proprietary and expensive forks of Hadoop, T-Systems Juergen Urbanski explains how to

are buying an open version of Hadoop or something you might later regret.

Further Reading


17/17

CTO Perspective on Capturing tPotential of Big Data in a ServicProviderJrgen UrbanskiBoard Member Big Data & Analytics of BITKOM (German IT Industry Views and opinions expressed are not necessarily those of his emplo

T-Systems, the enterprise arm of Deutsche Telekom.

Documents

CTO Perspective on Capturing the Potential of Big Data in a Service Provider