Big Data - Fresh Ideas for Your Business Value

0 Copyright 2014 FUJITSU

Human CentricInnovation

Fujitsu Forum2014

19th – 20th November


Big Data – Fresh Ideas for Your Business Value

Gernot FelsGlobal Services & Solutions Marketing, Fujitsu


The Promise of Big Data

Discover hidden secrets

Predict opportunities

Identify and minimize unknown risks

Take better and faster decisions

Accelerate business processes

Increase performance and productivity

Improve efficiency and effectiveness

Profitability and competitive advantage

Better utilize our planet‘s resources

A convincing value proposition – For business and society.


Entering new dimensions …

Versatile data sources –internal and external

DB

Text

Mail

Multi-media

Machine logs

Web logs

Social media

Internet

Sensors (GPS, RFID, …, IoT)

…

Versatility




DB

Text

Mail

Multi-media

Machine logs

Web logs

Social media

Internet


…

Various data types

Structured

Unstructured

Semi-structured

Poly-structured Variety

Versatility




DB

Text

Mail

Multi-media

Machine logs

Web logs

Social media

Internet


…

Various data types

Structured

Unstructured

Semi-structured

Poly-structured

Large data volumes

TB to PB

Volume

Variety

Versatility




DB

Text

Mail

Multi-media

Machine logs

Web logs

Social media

Internet


…

Various data types

Structured

Unstructured

Semi-structured

Poly-structured

Large data volumes

TB to PB

High velocity

Generation and processing

Real-time demands

Volume

Variety

Versatility

Velocity




DB

Text

Mail

Multi-media

Machine logs

Web logs

Social media

Internet


…

Various data types

Structured

Unstructured

Semi-structured

Poly-structured

Large data volumes

TB to PB

High velocity

Generation and processing

Real-time demands

Generate new value by correlation of data and new ways of analytics.

Volume

Variety

Versatility

Velocity

Value


Big Data is the new oil

Volume

Variety

Versatility

Velocity

Value

Discover

Extract

Refine

Value



Volume

Variety

Versatility

Velocity

Value

Discover

Extract, collect

Transform

Analyze, visualize

Decide,act

Discover

Extract

Refine

Value



Volume

Variety

Versatility

Velocity

Value

Discover

Extract

Refine

Value

Infr

astr

uct

ure

Serv

ices

You need an infrastructure and services to make it happen.

How to get from Big Data to Big Value?

A lot of questions …

Which use case (business priority)?

Which data from which sources?

Is external data trustworthy, and always (free) available?

How to transform data into high quality?

How to explore data and discover its meaning?

What to look for? Which questions to ask?

Which actions to trigger? Are actions executable?

Which analytic methods?

How to visualize results effectively?

Which tools? How to use them?

How to ensure data security and privacy?

What about data retention and deletion?

Which skills?

Fresh ideas, deep knowledge of data, tools and intended results.


Example: Retail

Fresh ideas step by step – to become more attractive and improve business.


Example: Retail – Which data sources what for?

Transactional DB and DW Customers, partners, suppliers, products, revenues, margins

Too little insight.

http://mediaportal.ts.fujitsu.com/pages/view.php?ref=30183&search=!collection3&offset=48&order_by=field12&sort=DESC&archive=0&k=


http://mediaportal.ts.fujitsu.com/pages/preview.php?ref=30181&ext=jpg&k=&search=!collection3&offset=48&order_by=field12&sort=DESC&archive=0




Transactional DB and DW

Mail, notes, call records

Customers, partners, suppliers, products, revenues, margins

Customer interaction, problems

New data sources, more insight.









Social media

Web logs, click streams



Customer sentiment, preferences, behavior

Foresee opportunities, prevent churn, web site optimization.









Social media





Future demand, stock requirements

Always the right goods, at the right quantity in stock – in every store.









Social media






Optimized store layout

Optimum combinations of products in shelves.









Social media


Weather data






Weather affects purchase.









Social media


Weather data

Internet






Market, competition

Recognize general trends, opportunities, risks.









Social media


Weather data

Internet

Location data, RFID cards






Market, competition

Personalized coupons and ads in real-time

Improve customer experience.









Social media


Weather data

Internet


Machine logs






Market, competition


Preventive maintenance

Foresee and prevent problems.









Social media


Weather data

Internet


Machine logs

Sensors (temperature)






Market, competition



HVAC control

Improve customer experience. Avoid spoiled goods.









Social media


Weather data

Internet


Machine logs

Sensors (temperature)

Cameras






Market, competition



HVAC control

Surveillance

Customer flow according to gender, age, time.






Example: Retail – Not the end of the story …

… steadily looking for new creative ideas to move the business forward.

Insatiable appetite to analyze new sources …


Big Data infrastructure challenges

Big Data requires new solution approaches.

Exponential growth of

data

Keep processing time constant while volumes increase

Store and process large data volumes at affordable cost

Ad-hoc queries andreal-time demands

Process continuously generated event streams


Distributed Parallel Processing

DataNode

TaskTracker

DataNode

TaskTracker

DataNode

TaskTracker

NameNode

JobTracker

Clie

nt

Master

Slaves DFSConcept

Distribute data and I/O to many nodes

Local server storage

Move computing to where data resides

Shared nothing architecture

Data replication to several nodes

Benefits

High performance, fast results

Unlimited scalability

Fault-tolerance

Cost-effective (standard servers with OSS)

Examples


Applications

Distilled Essence

What it is

Consolidated result of data transformation

Usually much smaller than initial data volumes

All essential information for use case included

Base for analytics

Where it is stored

Local disks in server cluster

…

DB allow better queries than file systems.

DistilledEssence


Databases – Designed for Big Data

Flexible data model

Designed to be distributed

Schema-less (simple structure)

Easily cope with new data types

Change format of data without app disruption

Ease of app development

Data automatically spread across servers

Support distributed query execution

Designed for scale-out

Add server to cluster on demand

Standard server HW

Integrated caching

Fault tolerance

Reduce latency, increase throughput

Improve app responsiveness

Data replication (across cluster / DC)

Automatic recovery from failure

Various flavors

Graph DB

Key-value stores

Document stores

Columnar stores

NoSQL databases help overcome the limitations of RDBMS.


Column-oriented DB

Row-oriented data store

Column-oriented data store

Millions of rows (records)

Many queries relate to small number of columns

Inefficient to read all rows

Reduced I/O, fast analysis

Access only relevant columns

Split column for parallel processing

High compression

Column contains only 1 data type

Often only few values per column (to store)

Typical compression factor 10-50

NoSQL databases help overcome the limitations of RDBMS.

Row-oriented storeKey Region Product Sales Vendor

0 Europe Mice 750 Tigerfoot1 America Mice 1100 Lionhead2 America Rabbits 250 Tigerfoot3 Asia Rats 2000 Lionhead4 Europe Rats 2000 Tigerfoot

Column-oriented storeKey Region Product Sales Vendor

0 Europe Mice 750 Tigerfoot1 America Mice 1100 Lionhead2 America Rabbits 250 Tigerfoot3 Asia Rats 2000 Lionhead4 Europe Rats 2000 Tigerfoot

SELECT SUM (Sales) GROUP BY Region

Data required

Data read, but not required


Applications

Distilled Essence

What it is

Consolidated result of data transformation

Usually much smaller than initial data volumes

All essential information for use case included

Base for analytics

Where it is stored

Local disks in server cluster

HDD storage

AFA

…

What if immediacy matters?

DistilledEssence


In-Memory Data Grid (IMDG)

Concept

Distributed in-memory store between apps and data

Any data object (suitable for apps)

Any data size (even > RAM capacity)

Scale-up, scale-out

Benefits

Optimal response of apps by reducing I/O

Performance as you grow

How to ensure business continuity?

Applications

DistilledEssence

RAM

RAM

RAM

IMDG



Concept




Scale-up, scale-out

Persistent storage for automatic recovery

Benefits



Meet real-time demands, ensure business continuity.

Applications

DistilledEssence

RAM

RAM

RAM

IMDG

Snapshots

Change log

Persistence



Concept




Scale-up, scale-out


Data mirror on other server for ultra-fast recovery

Use apps unmodified (optimization possible)

Pre-defined queries

Benefits



Meet real-time demands, ensure business continuity.

Applications

DistilledEssence

RAM

RAM

RAM

IMDG

Snapshots

Change log

Persistence

RAM

RAM

RAM

Mirror


Benefits

In-Memory Database (IMDB)

Meet real-time demands for ad-hoc queries, …

Concept

Entire DB and operation in RAM of server(s)

No disk access for analysis

Scale-up, scale-out

Data access 1K-10K x faster than disk

Ad-hoc queries in real-time

Fast decisions by eliminating I/O

Applications

RAM

RAM

RAM

IMDB

DistilledEssence


In-Memory Database (IMDB)

Concept

Entire DB and operation in RAM of server(s)

No disk access for analysis

Scale-up, scale-out


Data mirror on other server for ultra-fast recovery

Condition: Data size < RAM capacity

Benefits

Data access 1K-10K x faster than disk

Ad-hoc queries in real-time

Fast decisions by eliminating I/O

Meet real-time demands for ad-hoc queries, ensure business continuity.

Applications

RAM

RAM

RAM

RAM

RAM

RAM

Snapshots

Change log

Persistence

IMDBMirror


Complex Event Processing (CEP)

Concept

Critical parameters

Collect and analyze continuously generated data

Rules for logical and temporal correlation

Event pattern matching

Trigger action in real-time

Throughput: 10K’s to M’s of events per sec

Latency: µs to ms (time from event input to output)

Event input

Ru

les

State transition

Apply Event output

ControlCEP Engine

Sensor


Complex Event Processing (CEP)

Concept

Critical parameters

Collect and analyze continuously generated data

Rules for logical and temporal correlation

Event pattern matching

Trigger action in real-time

Throughput: 10K’s to M’s of events per sec

Latency: µs to ms (time from event input to output)

Streaming

Event input

Ru

les

State transition

Apply Event output

ControlCEP Engine

Routing

Options for acceleration

Data in RAM / IMDG (long time window)

Parallelization (large streams, many / complex rules)

CEP requires parallel processing and in-memory technologies.

Sensor Sensor Sensor


μsecmsecsecminhrs

CEP (Complex Event Processing)

Response time

Conventionaltechnologies(e.g. RDBMS)

In-Memory(IMDB, IMDG)

GB

TB

PB

Dat

a vo

lum

eBig Data technologies at a glance

Select the right technology depending on volume and time requirements.

DistributedParallelProcessing


Un- / Semi-/Poly-

structureddata

Big Data infrastructure

IMDB

Data Sources Analytics Platform Access

Extract, Collect Clean, Transform Decide, ActAnalyze, Visualize

Consolidated data Distilled essence Applied knowledgeVarious data

Dat

a at

res

t

DB / DW


Un- / Semi-/Poly-

structureddata


IMDB




Dat

a at

res

t

DB / DW

DistributedParallel

Processing


Un- / Semi-/Poly-

structureddata


IMDB




Dat

a at

res

t

DB / DW

DistributedParallel

Processing

FS

NoSQL

DataReservoir


Un- / Semi-/Poly-

structureddata


IMDB




Dat

a at

res

t

DB / DW

DistributedParallel

Processing

FS

NoSQL

AppsServicesQueries

….

VisualizationReporting

Notification

DataReservoir


Un- / Semi-/Poly-

structureddata


IMDB




Dat

a at

res

t

DB / DW

DistributedParallel

Processing

FS

NoSQL

AppsServicesQueries

….


Notification

DB / DW

DataReservoir


Un- / Semi-/Poly-

structureddata


IMDB




Dat

a at

res

t

DB / DW

DistributedParallel

Processing

FS

NoSQL

AppsServicesQueries

….


Notification

DB / DW

IMDGDataReservoir


Un- / Semi-/Poly-

structureddata


IMDB




Dat

a at

res

t

DB / DW

DistributedParallel

Processing

FS

NoSQL

AppsServicesQueries

….


Notification

DB / DW

IMDG

IMDB

DataReservoir


Un- / Semi-/Poly-

structureddata


IMDB




Dat

a at

res

t

DB / DW

DistributedParallel

Processing

FS

NoSQL

AppsServicesQueries

….


Notification

DB / DW

IMDG

IMDB

DataReservoir


Un- / Semi-/Poly-

structureddata


IMDB




Dat

a at

res

t

DB / DW

DistributedParallel

Processing

FS

NoSQL

AppsServicesQueries

….


Notification

DB / DW

IMDG

IMDB

NoSQL

DataReservoir


Un- / Semi-/Poly-

structureddata


IMDB




Dat

a at

res

t

DB / DW

DistributedParallel

Processing

FS

NoSQL

AppsServicesQueries

….


Notification

DB / DW

IMDG

IMDB

NoSQL

DataReservoir


Un- / Semi-/Poly-

structureddata


IMDB




Dat

a at

res

tD

ata

in m

otio

n

DB / DW

DistributedParallel

Processing

FS

NoSQL

AppsServicesQueries

….


Notification

DB / DW

IMDG

IMDB

NoSQL

ComplexEvent

Processing


Un- / Semi-/Poly-

structureddata


IMDB




Dat

a at

res

tD

ata

in m

otio

n

DB / DW

DistributedParallel

Processing

FS

NoSQL

AppsServicesQueries

….


Notification

DB / DW

IMDG

IMDB

NoSQL

ComplexEvent

Processing


Un- / Semi-/Poly-

structureddata


IMDB




Dat

a at

res

tD

ata

in m

otio

n

DB / DW

DistributedParallel

Processing

FS

NoSQL

AppsServicesQueries

….


Notification

DB / DW

IMDG

IMDB

NoSQL

CEP

IMDG


Un- / Semi-/Poly-

structureddata


IMDB




IMDB

AppsServicesQueries

….


Notification

DistributedParallel

Processing

IMDG

Dat

a at

res

tD

ata

in m

otio

n

FS

IMDG

DB / DW

DB / DWNoSQL NoSQL

IMDG

CEP


How can help

Optimum solution for every situation

Combination of technologies

Infrastructure products

SW / MW (OSS, Fujitsu, ISV)

End-to-end services

Assessment, consulting

Solution design

Deployment, integration, maintenance

Attractive financing options

Sourcing options

Self-managed (on-premise)

Managed services (on- / off-premise)

Fujitsu Cloud

One-stop shop for Big Data: Reduce complexity, time and risk.

…




Integrated Systems for Big Data Infrastructures

PRIMEFLEX for Hadoop PRIMEFLEX for Terracotta BigMemory

PRIMEFLEX for SAP HANA

Fujitsu Hadoop Services

Pre-integrated, ready-to-run: Fast deployment. Short time to production.

Analytical apps / templates (free)

Datameer (Visual Analytics)

Hadoop (HDFS, MapReduce)

Linux OS

PRIMERGY RX / CX

Brocade switches

Fujitsu Optimization Services

Terracotta BigMemory

Java Runtime Environment

Linux OS

PRIMERGY RX200 / CX4770

Local SSD

Network (1 / 10Gb Ethernet)

Fujitsu HANA Services

SAP HANA DB

Linux OS

PRIMERGY / PRIMEQUEST

Persistent storage

NW switches (multi-node only)


Integrated Systems for Big Data Infrastructures

PRIMEFLEX for Hadoop PRIMEFLEX for Terracotta BigMemory

PRIMEFLEX for SAP HANA

Pre-integrated, ready-to-run: Fast deployment. Short time to production.

Datameer (Visual Analytics)

Hadoop (HDFS, MapReduce)

Linux OS

PRIMERGY RX / CX

Brocade switches

Fujitsu Optimization Services

Terracotta BigMemory

Java Runtime Environment

Linux OS

PRIMERGY RX200 / CX4770

Local SSD

Network (1 / 10Gb Ethernet)

Fujitsu HANA Services

SAP HANA DB

Linux OS

PRIMERGY / PRIMEQUEST

Persistent storage

NW switches (multi-node only)

Fujitsu Hadoop Services

Analytical apps / templates (free)


Big Data – Important pillar of Fujitsu’s vision

Create innovationthrough people

Power business and society with information

Optimize ICT Systems from end to end

Big Data makes the Human-Centric Intelligent Society come true.


Big Data in the Internet

http://www.fujitsu.com/fts/solutions/high-tech/bigdata/

Global Internet site under construction

Umbrella around topic and all building blocks

Links to individual building blocks and offerings

http://www.fujitsu.com/fts/solutions/high-tech/bigdata/


Summary

Big Data offers enormous potential for business opportunities and value

If you ignore it, your competitor won’t !

It changes the way companies make decisions, do business, succeed or fail

New ideas and use cases

New types of analytics

New skills

New tools, SW and MW

New infrastructure concepts

New value

Fujitsu – A one-stop shop for Big Data

Enabler, making your dreams come true

Fujitsu – Turning fresh ideas into business value.

Thank you for listening


Technology

Big Data - Fresh Ideas for Your Business Value