Data-driven Innovations supported by the Discovery Lab

Data-driven Innovations – supported by the Discovery Lab Harald Erb Oracle Business Analytics

DOAG 2015 Business Intelligence München, 23. April 2015

Copyright © 2015, Oracle and/or its affiliates. All rights reserved.


» Harald Erb

» Principal Sales Consultant

» Business Analytics Architect Domain Lead - DE/CH Cluster

» Kontakt

+49 (0)6103 397-403

» [email protected]

Referent

http://www.linkedin.com/in/haralderb


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

4


Program Agenda

5

Digital Business, Data-driven Decisioning

Discovery Lab

Oracle Big Data Discovery

Monetizing new Insights

Unified Big Data Management and Analytics Architecture

Digital Business with Oracle

1

2

3

4

5

6


Digital Business, Data-driven Decisioning

6

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. 7


Example: „Hijack“ – A Campaign that starts to sell in competitive stores Digital Business

Video: „HIJACK - MEAT PACK GUATEMALA“Cannes Lions Winner of Bronze & Silver (Mobile Category)

https://vimeo.com/44351185





They ‘Reframe’ Challenges

Looking at them from new perspectives and multiple angles

They Sprint

They work at pace - researching, testing and evaluating current ideas while generating new ones

They Appreciate That

Failure Can Be Good

and are not afraid of new ideas

They Convert Data Into Value

They invest heavily in analyzing their own data and data from external sources to establish patterns and un-noticed opportunities

Characteristics of Digital Business Leaders


Data-driven Decisions

10

Ide

nti

fy (

bu

sin

ess

) q

ue

stio

n

Become clear about all aspects of the decision to be taken or the problem to be solved.

Try to identify alternatives to your percep-tion

Ve

rify

ear

lier

fin

din

gs

Find out who has investi-gated such or a similar problem in the past and the approach that has been taken

De

sign

of

a so

luti

on

mo

de

l Formulate a detailled hypothesis how specific variables might influence the result of the chosen model

Gat

he

r al

l ne

cess

ary

dat

a

An

alys

e t

he

dat

a

Pre

sen

t & im

ple

me

nt

resu

lts

Gather all available information about the variables of your hypo-thesis. The relevance of a dataset might address your business question directly or needs to be derived

Apply a statistical model and evaluate the correctness of the approach. Repeat this procedure until the right method has been identified.

Source: Thomas H. Davenport, Harvard Business Manager 2013

Frame the results obtained in a comprehensible story. This kind of presentation intends to motivate decision makers and relevant stake-holders to take action

Non-Analysts & Executives should take a closer look on steps 1 and 6 of the analysis process if they plan to make use of statistical analysis.

Knowledge Discovery


Vertical and Horizontal Data Scientists

Data Warehouse Horizontal Vertical

Deep technical skills

Eigenvalues, Lasso-related regressions

Experts in Bayesian networks, R

Support Vector Machine

Hadoop, NoSQL, Data Modeling, DW

Cross-discipline knowledge

Machine Learning & Statistics

Visualization skills

Domain expertise

Storytelling experts

Programming experience

Aware of pitfalls

& rules of thumb

The Specialist The Unicorn

Look for the individual unicorn

or build a Data Science Team?


Enabling Data-driven Innovations in Organizations

12

Perf.

Mgmt.

Knowledge Discovery

Dynamic Dashboards and Reports

Volume and Fixed Reporting

Knowledge Driven Business Process

Analytical Competence Center (ACC)

» Separate group reporting to CxO

» Not part of a Business Intelligence Competence Center (BICC)

» Mission: broadening the adoption of Analytics across the organization

» Skilled resource pool of Data Scientists, Statisticians and Business Experts with privileged access to the internal Enterprise Data Sources

» Will be assigned to projects for a limited time

» Using scientific methods to solve business problems using available data.

Executive: Decisions effecting

strategy and direction

Business Analyst: Day-to-Day performance

of a business unit

Information Consumer: Reporting on

individual transactions

Automated Process: Decisions effecting

execution of an indiv. transactions

Insight Data Scientist:

Information analysis to meet strategic goals

BICC

ACC


Discovery Lab

13


Information Management – Conceptual View

Discovery Lab

Innovation

Discovery Output

Events & Data

Actionable

Events

Event Engine Data Reservoir

Data Factory Enterprise Information Store

Business

Intelligence

Actionable

Information

Actionable

Insights

Data

Streams

Execution

Structured

Enterprise

Data

Other

Data

Line of governance

Source: Oracle White Paper “Information Management and Big Data – A Reference Architecture”


» Event Engine: Components which process data in-flight to identify actionable events and then determine next-best-action based on decision context and event profile data and persist in a durable storage system.

» Data Reservoir: Economical, scale-out storage and parallel processing for data which does not have stringent requirements for formalisation or modelling. Typically manifested as a Hadoop cluster or staging area in a relational database.

» Data Factory: Management and orchestration of data into and between the Data Reservoir and Enterprise Information Store as well as the rapid provisioning of data into the Discovery Lab for agile discovery.

» Enterprise Information Store: Large scale formalised and modelled business critical data store, typically manifested by an (Enterprise) Data Warehouse. When combined with a Data Reservoir, these form a Big Data Management System.

» Reporting: BI tools and infrastructure components for timely and accurate reporting.

» Discovery Lab: A set of data stores, processing engines, and analysis tools separate from the everyday processing of data to facilitate the discovery of new knowledge of value to the business. This includes the ability to provision new data into the Discovery Lab from outside the architecture.

» Execution: Flow of data for execution are tasks which support and inform daily operations

» Innovation: Flow of data for innovation are tasks which drive new insights back to the business

» Arranging solutions on either side of this division (as shown by the red line) helps inform system requirements for security, governance, and timeliness.

Information Management – Conceptual View

15

Source: Oracle White Paper “Information Management and Big Data – A Reference Architecture”


Discovery Lab: Design Pattern

» Specific focus on identifying commercial value for exploitation

» Wide range of tools and techniques applied

» Iterative development approach – data oriented NOT development oriented

» Data provisioned through Data Factory or own ETL processes

» Typically separate infrastructure but could also be unified Reservoir if resource managed effectively

» Small group of highly skilled individuals (aka “Data Scientists” or Analytical Competence Center, ACC)


Discovery Lab: Activity Cycles


Discovery Lab: Sandbox Provisioning

18

Analysis Processing & Delivery

Discovery Lab & Development Environment

Data

Science

(Primary

Toolset)

Statistics Tools

Data & Text Mining Tools

Faceted Query Tools

Programming & Scripting

Data Modelling Tools

Query & Search Tools

Pre-Built

Intelligence

Assets

Intelligence

Analysis

Tools

Ad Hoc Query & Analysis Tools

OLAP Tools

Forecasting & Simulation Tools

Reporting Tools

ACC

Virtu

alis

atio

n &

Info

rma

tion S

erv

ices

Data Factory flow

ACC may quickly develop new reporting through mashups from any available internal and external sources and may used advanced analytical tools for innovative analysis

Data Quality & Profiling

Graphical rendering tools

Dashboards & Reports

Scorecards

Charts & Graphs

Sandbox – Project 3



Data store Analytical Processing

General BI flow

1

2

BICC

The majority of BI development activity will be from existing sources – performed by the BICC developing new reports to existing or new channels

External Data


Discovery Lab: Need To Get Analytic Value Fast

19

Tool Complexity

» Early Hadoop tools only for experts

» Existing BI tools not designed for Hadoop

» Emerging solutions lack broad capabilities

80% effort typically spent on evaluating and preparing data

Data Uncertainty

» Not familiar and overwhelming

» Potential value not obvious

» Requires significant manipulation

Overly dependent on scarce and highly skilled resources


Oracle Big Data Discovery: The Visual Face of Hadoop

find explore transform discover share


Oracle Big Data Discovery. The Visual Face of Hadoop

21

find explore transform discover share See the potential in big data


Catalog

22

» Access a rich, interactive catalog of all data in Hadoop

» Familiar search and guided navigation for ease of use

» See data set summaries, user annotation and recommendations

» Provision personal and enterprise data to Hadoop via self-service


Explore

23

» Visualize all attributes by type

» Sort attributes by information potential

» Assess attribute statistics, data quality and outliers

» Use scratch pad to uncover correlations between attributes



24

find explore transform discover share Quickly make big data better

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. 25 25

» Intuitive, user driven data wrangling

» Extensive library of powerful data transformations and enrichments

» Preview results, undo, commit and replay transforms

» Test on sample data then apply to full data set in Hadoop

Transform



26

find explore transform discover share Unlock big data not only for Data Scientists


» Join and blend data for deeper perspectives

» Compose project pages via drag and drop

» Use powerful search and guided navigation to ask questions

» See new patterns in rich, interactive data visualizations

Discover


» Share projects, bookmarks and snapshots with others

» Build galleries and tell big data stories

» Collaborate and iterate as a team

» Publish blended data to HDFS for leverage in other tools

Share


Oracle Big Data Discovery: Deployment

29

Diagram Source: RittmannMead Blog, 2015


Oracle Big Data Discovery: Components

30

Oracle Big Data Discovery Workloads

Hadoop Cluster (Oracle Big Data Appliance or Commodity Hardware with

Cloudera CDH 5.)

BDD node

data node

data node

data node

data node

name node Data Processing, Workflow & Monitoring • Profiling: catalog entry creation, data type &

language detection, schema configuration • Sampling: dgraph (index) file creation • Transforms: >100 functions • Enrichments: location (geo), text (cleanup,

sentiment, entity, key-phrase, whitelist tagging)

Self-Service Provisioning & Data Transfer

• Personal Data: Upload CSV and XLS to HDFS

In-Memory Discovery Indexes • DGraph: Search, Guided Navigation, Analytics

Studio

• Web UI: Find, Explore, Transform, Discover, Share

Hadoop 2.x

Filesystem (HDFS)

Workload Mgmt (YARN)

Metadata (HCatalog)

Other Hadoop Workloads

MapReduce

Spark

Hive

Pig

Oracle Big Data SQL (Oracle Big Data Appliance only)


Oracle Big Data Discovery: Data Ingestion Workflow Overview

1M of 100M

Diagram Source: RittmannMead Blog, 2015


User friendly,... Oracle Big Data Discovery: Data Preparation

Preferred method for the Business Analyst


...and flexible (based on Groovy Programming Language) Oracle Big Data Discovery: Data Preparation

Preferred Method for IT / Data Engineer / Data Scientist / …


Working the Data Analyzing the Data

34

Data Discovery & Analytics Lifecycle

Data Select Data

Prepare Data

Transform Data

See Patterns

Interpret & Evaluate Knowledge

Oracle Advanced Analytics


Time

f(x)

?

a = A

80% 20% 20% 80%


More Data Variety available – Better Results

35

Response Modelling Example: Getting „lift“ on responders

Data Mining-based Prediction results with Big Data and hundreds of input variables including:

Naïve Guess or Random

100 0 Population Size (% of Total Cases)

% o

f P

osit

ive R

esp

on

ders

Model with 20 variables



» Demographic data » Purchase POS

transactional data » Polystructured data,

text & comments » Spatial location data » Long term vs. recent

historical behaviour » Web visits » Sensor data » …

100


Oracle R Enterprise (ORE)

» Allows distributed processing of huge data volumes

» Benefits from DB features, e.g. Security and SQL access

» R Studio = GUI for Data Analysts

36

Oracle Data Mining (ODM)

» Implemented in the Oracle Database kernel

» Direct access via PL/SQL API and SQL operators

» Oracle Data Miner GUI embedded in SQL Developer

Oracle Advanced Analytics Native SQL Data Mining/Analytic Functions + High-performance R Integration


Discovery Lab: End of Research Phase



Apply statistical & predictive models

No Data Movement; Bring algorithms to the data

Utilize Oracle R and Data Mining for Massive Computing Scalability on Hadoop or Oracle

Integrated with SQL and BI tools

Find data for analytics & data science projects

Explore the shape and quality of the data

Transform data for analytics

Discover and visualize insights in data sets

Share insights with analysts and downstream systems

Share Insight

Interpret & Evaluate

Select, Prepare & Transform


Acceptable Results Probability: A Clearer View

38

Storytelling / Infographics

Discovery Lab: Explanation & Validation of the Results

Individuals of the Analytical Competence Center need to frame the results obtained in a comprehensible story. This kind of presentation intends to

motivate decision makers and relevant stake-holders to take action

Result of 1000 simulations of a $100 million investment in a new factory: Estimation expects an annual return of 20% over a 10-year lifespan, but the risk to loose invested money is still 8% Big Data Discovery – Gallery feature documents all discovery

steps taken to achieve new insights

Individually created infographic explaining the key findings


Monetizing New Insights

39


Discovery and monetising steps have different requirements Making sense from diverse data

Research & Development

» Unbounded discovery

» Self-Service sandbox

» Wide toolset

» Agile methods


Discovery and monetising steps have different requirements Making sense from diverse data

Promotion to Data-driven Services

» Commercial exploitation

» Narrower toolset

» Integration to operations

» Non-functional requirements

» Code standardisation & governance


Usage of Data-driven Innovations: Predictive BI

Oracle Business Intelligence: Dashboards, Alerts,...

» Understandable prediction results, Self-service BI

» Making analysis results available to every business user, i.e. potential cross-selling effects to responsible Buyers

» Operated by Business Analysts / BICC, etc.

Predictive Query Example

SELECT cust_income_level, cust_id

, ROUND(probanom,2) AS probanom

, ROUND(pctrank,3)*100 AS pctrank

FROM (SELECT cust_id, cust_income_level, probanom

, PERCENT_RANK()

OVER (PARTITION BY cust_income_level

ORDER BY probanom DESC) AS pctrank

FROM (SELECT cust_id, cust_income_level

, PREDICTION_PROBABILITY(OF ANOMALY,0 USING *)

OVER (PARTITION BY cust_income_level)

AS probanom

FROM customers

)

)

WHERE pctrank <= .05

ORDER BY cust_income_level, probanom DESC;

Oracle 12c In-Database Mining / Statistics

» Operationalize Data Mining Models as part of Oracle BI Dashboards, calculated on-the-fly

» Available query types: Classification & regression (incl. Multi-target problems), clustering, anomaly detection, feature extraction


Example: Oracle Human Capital Management Usage of Data-driven Innovations: Forward-looking Apps

Characteristics:

» Includes Oracle Advanced Analytics factory-installed Predictive Analytics:

» Employees likely to leave & predicted performance

» Top reasons, expected behavior

» Real-time "What if?" analysis


Example: Next Best Offer Usage of Data-driven Innovations: Real-Time Decisoning

1 Channel: Web 1

2 Placement: Homepage 2

3 Creative-Content: "Expert Tennis Tips" 3

4 Slot Type: Articles 4

5 Slot: Center Middle 5

7 Tags: Tennis | Tips | Pros 7

6 Offer: Discount on Tennis Lessons 6

Leads to multiple model updates and discovery of associated correlations across the graph

Rules & Predictive Models

Performance Goals


Intelligent User Experience Usage of Data-driven Innovations: Event Processing

iBeacons

» Bluetooth Low Energy (BLE)

» Optimized for small bursts of data.

» Impressive battery Life

» Ideal for sensors

Requirements

» Find purchase pattern from data of shopper’s purchase history

» Leverage all the data, including real-time context from Beacon, CRM data, purchase history data, to improve the relevance of the offer

» Leverage predictive models to alleviate the reliance on the rule based models

» Being able to understand customer’s feedback on Beacon marketing


Usage of Data-driven Innovations: Event Processing Solution Architecture

Analysis and Offering

Decision Engine

Unstructured Text Analysis (VOC analysis)

Rule Based Statistical

Model-based

Modeling Processing

Real Time Offering

Qualitative indices

Text Mining

Data Dictionary

Text Analysis

Collection

Batch collection

Real Time Collection

Web Crawling

Open API

Storage and processing Utilization

ETL

Treatment Store

Hadoop File

Reduce Map

HDFS

Datafile#1 HDFS

Datafile#2 HDFS

Datafile#n HDFS

NoSQL DB Transaction (Key-Value)

Stores

Big Data Connectors

Mobile Apps

Unstructured Data Visualization

Coupon

Mileage

…..

New information

Keywords Visualization

Search Vigan Visualization

Dash Board

Mobile

Real Time

Formal & Informal

Integration

Source system

Other internal and external systems

Beacon

Time

Phone Number

Distance

Beacon MAC

Customer

…..

Martial Status

Customer Type

Customer ID

…..

Num of Children

Occupation

Gender

Purchase

Amount

Product

Customer ID

…..

Quantity

Date

Smart App

Web

VOC

SNS

ODS DW

Advanced Analytics on

Purchase Pattern


Usage of Data-driven Innovations: Event Processing Solution Architecture – Product View

Analysis and Offering

Decision Engine

Unstructured Text Analysis (VOC analysis)

Rule Based Statistical

Model-based

Modeling Self-Learning

Real Time Offering

Qualitative indices

Text Mining

Data Dictionary

Text Analysis

Collection

Batch collection

Real Time Collection

Web Crawling

Open API

Storage and processing Utilization

ETL

Treatment Store

Hadoop File

Reduce Map

HDFS

Datafile#1 HDFS

Datafile#2 HDFS

Datafile#n HDFS

NoSQL DB Transaction (Key-Value)

Stores

Big Data Connectors

Mobile Apps

Unstructured Data Visualization

Coupon

Mileage

…..

New information

Keywords Visualization

Search Vigan Visualization

Dash Board

Mobile

Real Time

Formal & Informal

Integration

Source system

Other internal and external systems

Beacon

Time

Phone Number

Distance

Beacon MAC

Customer

…..

Martial Status

Customer Type

Customer ID

…..

Num of Children

Occupation

Gender

Purchase

Amount

Product

Customer ID

…..

Quantity

Date

Smart App

Web

VOC

SNS

ODS DW

Advanced Analytics on

Purchase Pattern

Oracle Big Data Appliance

Oracle Event

Processing

Endeca

Information Discovery


Ora

cle

Dat

abas

e

Oracle Big Data Connectors

Oracle Data Integrator

Oracle Golden Gate

Oracle Data Integrator


Unified Big Data Management and Analytics Architecture

48


Productize, Secure & Govern

Experiment, Prototype & Collaborate

Data Reservoir

Poly

stru

ctu

red

D

ata

Data Warehouse

Oracle Database

Stru

ctu

red

Dat

a


Oracle Big Data SQL

Hadoop (HDFS)

Oracle R for Hadoop

Oracle Advanced Analytics (Data Mining, Oracle R Enterprise)

Tables in Hadoop

Tables in DB

SQL join

In-Memory Appliance

Oracle BI Foundation Suite (ROLAP/MOLAP, Mobile,…)

Oracle SQL Queries

Exalytics

Exadata

BDA

Unified: Big Data Management and Analytics…

Experiment, Prototype, Collaborate

» Quickly find, explore, transform, analyze and share discoveries in Big Data Discovery

» Publish results to the Hadoop Distributed File System (HDFS)

» Use to build predictive models with Oracle R for Hadoop

Productize, Secure, Govern

» Connect published HDFS files to secure Oracle DB using Oracle Big Data SQL

» No data movement required

» Seamlessly extends existing DWH and BI investments with non-traditional data in Hadoop

49


Example: „Pattern Matching“ Feature over Hadoop and DWH Data Sources Oracle Big Data SQL – Enables Full Use of Oracle 12c

Example Table Definition

CREATE TABLE movieapp_clicks

(click VARCHAR2(4000))

ORGANIZATION EXTERNAL

(TYPE ORACLE_HIVE

DEFAULT DIRECTORY Dir1

ACCESS PARAMETERS

(com.oracle.bigdata.tablename logs

com.oracle.bigdata.cluster mycluster

)

)

REJECT LIMIT UNLIMITED

Oracle Business Intelligence Dashboard

External Database Table accessing a Hive Table logs in Hadoop Cluster

named mycluster


…powered by Oracle Engineered Systems


Complete: Oracle Unified Information Architecture

52

Data Center Example


Digital Business with Oracle

53


Market, Sell & Service

Choice Of Deployment

Digital Product Development

End-to-end Visibility

Marketing Systems

Commerce Systems

Sales Systems

Service Systems

Private Cloud IaaS / Paas / SaaS

On_Premise

Public Cloud

Moblle & Device Security Identity Mgmt. Authorization Gateways

Digital Content

Process Integration Analytics

Service/Events Integration

Data Integration

Web / Mobile / IoT Development Framework

Real Time Data & Analytics

Platform Access APIs

Social Things Mobile Web

Learn

& A

dap

t Consume & Access

Multi-Channel

Protect & Secure

Oracle’s Digital Business Platform

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 13 55

Nike A Digital Example of an

Oracle Customer

Product & Digital Innovation is a Constant State for Nike

2012

2012

2009

2008

2006

2005

2002

2000

1987

1982

1979

1974

Flyknit upper technology introduced

Nike’s Fuelband is launched

Pro Combat apparel is launched

Lunar Foam and Flywire Technology launched

Nike+ (8 million+ members)

Nike Free launched

Golf clubs introduced

Shox technology introduced

Air Max shoe launched

Air Force 1 Basketball shoe

Tailwind – 1st Nike Air shoe

Waffle Trainer introduced

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 13 57

“Where most initiatives focus on enticing

consumers to complete a purchase, Nike+

continues to engage the consumer long

after the transaction has occurred, keeping

Nike+ runners motivated & connected, with

each other & with the Brand.”

Nick Law, R/GA EVP/Chief Creative Officer

Digital is enabling relationships &

community

+ +

Product + Content + Community = Premium Consumer

Experience


Show Me

Ext. Data Analytics

Connect

Context

Content

Configure

Always Connected, Sharing & Aware

C l o u d

Digital Business Platform - Overview

https://secure-nikeplus.nike.com/plus/


Documents

Data-driven Innovations supported by the Discovery Lab