Big Data at Oracle - Strata 2015 San Jose

Take it to the Limit:An Information Architecture for Beyond HadoopJeff PollockVP, Product ManagementFebruary, 2015

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

#StrataHadoop - Oracle Big Data Architecture


Jeff Pollock speaker bio:

• Product Lead for Big Data Integration & Governance• Oracle Middleware• IBM InfoSphere

• Chief Technical Officer• Cerebra – Semantic technology startup• Modulant – Semantic technology startup

• Software Engineer• Ernst & Young LLP – Java and E-Commerce• Amgen – Information services

• Author• Semantic Web for Dummies, 2006• Adaptive Information, 2004

In this session you’ll learn about how to apply Data Discovery and Deep Data Storage for new breakthroughs in data warehousing. We’ll discuss the benefits of using Hadoop technologies like Spark, Kafka, and Hive together with enterprise information architecture and data governance best practices.

Come hear the latest about Oracle’s Big Data architecture with special emphasis on new product updates in Oracle Big Data Discovery and Oracle Data Integration offerings – including the ability to replicate data in real-time with Hadoop as well as new support for Flume, Kafka, Pig, Oozie, and Spark. You’ll also get the latest information on Oracle Big Data SQL which can access all sources in NoSQL, HBase, HDFS, and relational databases in a single query.

The gravity of big data on the information management industry is evolving the way we collect, store, move, and analyze information. It is both pulling and pushing the world of business analytics to new breakthroughs in real-time streaming, predictive capabilities, virtualized data access, and self-service modes of operation. Join us in this session as we help bridge the gap between old and new by explaining how to apply data discovery, data governance, and SQL interoperability in this brave new world of Hadoop technology.

Take it to the Limit: An Information Architecture for Beyond Hadoop



CustomerExperience

OperationalImprovement

New BusinessModels

44% 30% 26%



TraditionalCurrency

PhotoFilm

PostalServices

PrintingPress

RecordIndustry

DigitalCurrency

WebPublishing

Email DigitalCamera


DigitalDownload

Entire Industries are Being Disrupted

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

BiggerSmarter

Easier

CheaperBetter

Faster VALUEFROM DATA



Connected by Data



Business Data as a Platform


Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | #StrataHadoop - Oracle Big Data Architecture

New Concepts for a Modern Big Data Platform Architecture

Polyglot

Fit for Purpose Data

Lambda

Speed Layer

Batch Layer

DataSources

DataServices

Kappa

DataServices

Data PipelineDataSources


Revenge of the Data Nerds!

Lambda

Fit for Purpose Data

Lambda

Speed Layer

Batch Layer

DataSources

DataServices

Lambda

DataServices

Data PipelineDataSources


Execution

Innovation


4th Generation Data Architecture for Big Data

WarehouseData FactoryReservoir

Data Streaming

Data Platform

Discovery Lab

Analytics

APIs

Enterprise Data

Other Data Sources

Data Streams

BusinessData

Social/LogData

Model FirstAnalytics

• Reporting-oriented• Often enterprise wide

in scope, cross LoB• “you know the

questions to ask”

Reports & Dashboards

Data FirstAnalytics

• Data Exploration• Highly visual and/or

interactive• “you don’t know the

questions to ask”

Discovery

• Telematics• Industry Services• Internet of Things• Sentiment

DataServices


Execution

Innovation


Comprehensive Oracle Solution for Big Data

WarehouseFactoryReservoir

Data Streaming

Data Platform

Discovery Lab

Analytics

APIs

Enterprise Data

Other Data Sources

Data Streams

BusinessData

Social/LogData




questions to ask”


Data FirstAnalytics



questions to ask”

Discovery


DataServices

ApacheOracleNoSQL

OracleCAF & OEP

Oracle Data Integration & Governance

Oracle Database& Big Data SQL

OracleR

OracleBig Data

Discovery

OracleBusiness

Intelligence

OracleBig Data

Discovery

Apache


Execution

Innovation


Integrated Oracle Engineered Systems for Big Data

Data Streaming

Data Platform

Discovery Lab

Analytics

APIs

Enterprise Data

Other Data Sources

Data Streams

BusinessData

Social/LogData




questions to ask”


Data FirstAnalytics



questions to ask”

Discovery


DataServices

APIs

Analytics


Execution

Innovation


Visionary Oracle Cloud for Big Data

Data Platform

Discovery Lab

Analytics

APIs

Enterprise Data

Other Data Sources

Data Streams

BusinessData

Social/LogData




questions to ask”


Data FirstAnalytics



questions to ask”

Discovery


DataServices

Data Streaming


Oracle Brings Business Value to Big Data

Enterprise-Grade Capabilities

Discover and Predict – Fast

Govern and Secure All Data

Simplify Access to All Data

Performance Integration Availability Scalability Manageability


EnterpriseProductivity

DisruptiveTechnology


BIG DATA DISCOVERY



Data Discovery Accelerates Time-to-Value From Data

80% effort typically spent on evaluating and preparing data

Data Uncertainty

• Not familiar and overwhelming

• Potential value not obvious

• Requires significant manipulation

Overly dependent on scarce and highly skilled resources

Tool Complexity

• Early Hadoop tools only for experts

• Existing BI tools not designed for Hadoop

• Emerging solutions lack broad capabilities


Announcing: Oracle Big Data DiscoveryThe Visual Face of Hadoop

find explore transform discover share



Find

20

• Access a rich, interactive catalog of all data in Hadoop

• Familiar search and guided navigation for ease of use

• See data set summaries, user annotation and recommendations

• Provision personal and enterprise data to Hadoop via self-service


Explore

21

• Visualize all attributes by type

• Sort attributes by information potential

• Assess attribute statistics, data quality and outliers

• Use scratch pad to uncover correlations between attributes

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 2222

• Intuitive, user driven data wrangling

• Extensive library of powerful data transformations and enrichments

• Preview results, undo, commit and replay transforms

• Test on sample data then apply to full data set in Hadoop

Transform


• Join and blend data for deeper perspectives

• Compose project pages via drag and drop

• Use powerful search and guided navigation to ask questions

• See new patterns in rich, interactive data visualizations

Discovery


Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 24

• Share projects, bookmarks and snapshots with others

• Build galleries and tell big data stories

• Collaborate and iterate as a team

• Publish blended data to HDFS for leverage in other tools

Share


Open Platform: Core Technical Innovations on Hadoop

Oracle Big Data Discovery Workloads

Hadoop Cluster(BDA or Commodity Hardware)

BDD node

data node

data node

data node

data node

name nodeData Processing, Workflow & Monitoring

• Profiling: catalog entry creation, data type & language detection, schema configuration

• Sampling: dgraph (index) file creation• Transforms: >100 functions• Enrichments: location (geo), text (cleanup,

sentiment, entity, key-phrase, whitelist tagging)

Self-Service Provisioning & Data Transfer

• Personal Data: Upload CSV and XLS to HDFS

In-Memory Discovery Indexes

• DGraph: Search, Guided Navigation, Analytics

Studio

• Web UI: Find, Explore, Transform, Discover, Share

Hadoop 2.x

Filesystem

Workload Mgmt(YARN)

Metadata(HCatalog)

Other Hadoop Workloads

MR

Spark

Hive

Pig

Oracle Big Data SQL (Oracle BDA)


Radical Simplicity: Discovery Lab in a Box

Big Data Discovery with the Big Data Appliance X5

• Ready to Go – just add Data

• Pre-configured, Tuned

• Integrated Management

• Run any 3rd party software

• Install Exalytics into a BDA Starter Rack– In-memory Database & Analytics

– OBIEE Certified with 12c & Big Data SQL

– Big Data Discovery Certified

– Oracle Data Integration Certified


BIG DATA APPLIANCE



18 Sun Oracle X5-2L Servers with per server:

• 2 * 18-core Intel® Xeon® Haswell-EP - fastest Intel processor ever shipped

• 128 GB Memory – DDR4 upgradeable from 128GB (8x16GB) to 768 GB

• 48TB Disk space

Integrated Software (4.1):

• Oracle Linux6.5, Oracle JDK 7u72

• Cloudera Distribution of Apache Hadoop 5.3 – EDH Edition, Cloudera Manager 5.3

• Oracle Big Data SQL 1.1, Oracle R Distribution 3.1.1-2, Oracle NoSQL Database CE 3.2.4

Announcing: BDA X5-2 UpdateFaster Processors, More Cores, More Memory, Same Price

BDA X5-2 vs. X4-2 Full Rack

2.25x More Cores 648 cores – Intel® Xeon® E5-2699 v3

2x More Memory Default 3.2TB DDR4-2133MHz

50% More DIMM Slots Up to 13.2TB DDR4-2133MHz



Easier Adoption of New Big Data Technologies

INTEGRATION SKILLS SECURITY

EngineeredSystems

SQL onAll Data

Database Security on

All Data

SQL


Standard, Modular and Highly Extensible Appliance

Starter Rack is a fully cabled

and configured for growth with 6

servers

In-Rack Expansion delivers 6

server modular expansion block

Full Rack delivers optimal blend

of capacity and expansion

options

Grow by adding rack – up to 18

racks without additional switches



Driving Business Value from Technology Innovation

Use the Right Tool for the Job and benefit from the Power of “AND”

RelationalHadoop NoSQL Graph

Run the Business Integrate existing

systems

Mission-critical

tasks

Use existing

investments

Ensure skills

relevance

Change the Business

Disrupt competitors

Disintermediate

supply chains

Leverage new

paradigms

Exploit new analyses

Scale the Business

Serve data

faster

Persist data streams

Meet mobile and

device challenges

Scale-out

economically

Link the Business

Associate complex

business entities

Link to Open Data

Share data sets via

Ontology

Evolve data and

schema together


Proven, Cost Effective Solution

“Oracle Big Data Appliance is an excellent choice for customers looking to work with the full suite of Cloudera’s leading Hadoop-based technology. It’s more cost-effective and quicker to deploy than a DIY cluster.”

⁻ Mike Olson, Cloudera founder, Chief Strategy Officer, and Chairman of the Board

Source:ESG White Paper

21%Cost Savings

33%Faster Time

to Value

http://www.oracle.com/us/corporate/analystreports/corporate/esg-bigdata-intel-2193858.pdf


BIG DATA INTEGRATION



Comprehensive Integration and Governance

FastLoad

Speed Layer

Batch Layer

Oracle Data Integrator(Transform)

Oracle GoldenGate(Move & Ingest)

DataGovernanceFoundation

Enterprise Data Quality(Profile & Cleanse)

Enterprise Metadata Management & Business Glossary(Business Glossary, Data Lineage, Impact Analysis and Data Provenance)

Veridata(Verify)

Data Enrichment(Prepare)

Real-Time Data Movement– Low impact capture, stage in Hadoop– Continuous data availability

Data Transformation– Bulk data movement– Pushdown data processing

Data Governance– Prepare unstructured data– Profile data with sampling– Clean data in real time or batch– Verify data for consistency– Trace lineage of all data– Define glossary of business terms


Staging


Open Lambda Architecture with Oracle Big Data Integration

Sqoop

HDFS

Hive

Flume

Cap

ture

Trai

l

Ro

ute

De

live

r

Pu

mp

Transformation

Data StreamingKafka (MPP Pub/Sub)

Storm and Trident

Spark Streaming

HBase

Discovery Sandbox/s

ROracle GoldenGate

Oracle Data Integrator

Oracle Data Governance

Oracle Data Enrichment




questions to ask”

Data FirstAnalytics



questions to ask”



Discovery

DataServices


Cap

ture

Trai

l

Ro

ute

Del

iver

Pu

mp


Announcing: Oracle GoldenGate for Big Data

New DB/HW/OS/APP

Zero Downtime Upgrades & Data Migration

Fully Active Distributed DB

High Availability& Disaster Recovery

ApplicationOffloading

Query & Report Offloading

Big Data, DW & Marts

Real-time BI, Hadoop Data Staging, Data Ingestion

Event Driven Architecture,SOA/JMS, Coherence

Message Bus& Data Grid

Data SynchronizationAcross the Enterprise

Global Data Centers

Real-time Analytics& Massive Parallelization

DataStreaming

GoldenGate

Real-timeData Delivery


Oracle Data Integrator for Transformation on Big Data

Flume

Hive on MR, Tez, Spark

Logs

OLTP DB

SQOOP

OGG

Pig on MR, Tez, Spark

Oracle Data Integrator

SQOOP

Any DW

OGG

Spark

OEMM

Metadata Mgmt& Lineage

API/File

Hive/HCat,HDFS,HBase

Hive/HCat,HDFS,HBase

NoSQL

Kafka

http://hadoop.apache.org/



Business Value of ODI: Low Cost and High Dev Efficiency

Oracle Confidential, under Non-Disclosure 38

No ETL engine is required

Separation of Logical and

Physical design

Physical exec on SQL, Hive, Pig, or

Spark

Runtime exec in Oozie or via ODI

Java Agent

Rich set of pre-built operators

User defined functions


Load to Oracle

OLH/OSCH


Oracle Data Integration on Engineered Systems

Transform

ODI

Hive/HDFS

Federate Hive/HDFS to Oracle

Big Data SQL

Oracle DB OLTP

Load from Oracle

CopyToBDA

Hive/HDFS

OGGOGG

Hive/HDFS

SQOOP

FlumeKafka




BIG DATA GOVERNANCE



Data Governance is Not Easy, Hadoop is No Silver Bullet!

Data Governance

Metadata Management

Business Glossary

Data Profiling

Data Cleansing

Data Archiving

Data Privacy

PEOPLE

PROCESS TECHNOLOGY

…people and process first, …tools and capabilities next, …and, there is no magic!

“…the overall impact of poor-quality data on the whole dataset remains the same. In addition, much of the data that organizations use in a big data context comes from outside, or is of unknown structure and origin. This means that the likelihood of data quality issues is even higher than before. So data quality is actually more important in the world of big data."

- Ted Friedman, Gartnerhttp://www.gartner.com/newsroom/id/2854917


Operational Data Preparation for Data Without Schema

Data Discovery& Visualization

EnterpriseReporting

Internet

Logs

Unstructured & Structured Data

90% of time is spent WRANGLING DATA

MONTHS of effort spent on each new

dataset

PROGRAMERS writing scripts or complex ETL

EnterpriseETL & DataIntegration


Internet

Logs

Unstructured & Structured Data

Data Discovery& Visualization

EnterpriseReporting

EnterpriseETL & DataIntegration

Operational Data Preparation for Data Without Schema


Industry Proven Data Privacy for All Data Types

extend OracleDatabase Security to Hadoop, NoSQL and

Graph data

authentication and authorization, audit and encryption on

Hadoop

end to end data transparency and

lifecycle management

Oracle Database Security

Industry Standard Access Controls and Lifecycle Controls

Relational

Hadoop

NoSQL

Graph


Comprehensive Metadata Management for Big Data

ETL

BIDashboards

App

ETL

ETL

Sys Admin

Executive

BI Developer

Application User

CDC

Data Steward

ETLDeveloper

Data Scientist

GG

Oracle Enterprise Metadata Management

Glossary & Catalog Harvest & Stitch Lineage




Most Open & Heterogeneous Big Data Solution

Hadoop HBase Hadoop Hive/Flume HP Enscribe HP NonStop HP Neoview Hypersonic SQL IBM DB2 i Series IBM DB2 UDB IBM DB2 z Series IBM Informix IBM Netezza JMS / MQ Microsoft Access Microsoft SQLServer MySQL Pivotal Greenplum PostgreSQL Salesforce.com SAP BW / BI SAP ERP / ECC SAS SQL/MP SQL/MX Sybase ASE Sybase IQ Teradata

Adaptive Altova Apache Hcatalog Apache Hive/HQL Borland CA ERwin Cloudera Impala COBOL Copybook DataStax Embarcadero EMC ProActivity GentleWare Google BigQuery Grandite Hadapt Hive Hortonworks Hive IBM Cognos IBM DB2 IBM DataStage IBM Discovery IBM Federation Server IBM Lotus Notes IBM Netezza IBM Rational Rose IBM Rational Architect Informatica Metadata Mgr. Informatica PowerCenter

CoSORT ISO SQL Standard (DDL) MapR Hadoop Hive MicroFocus Microsoft Access Microsoft Office Excel Microsoft Visio Microsoft SQL Server Microsoft SSIS Microsoft Visual Studio Microstrategy Magic Draw OMG CWM Standard OMG UML Standard Oracle BI Answers Oracle BI Enterprise Edition Oracle BI Server Oracle DAC Oracle Data Integrator Oracle Data Modeler Oracle Database Oracle Designer Oracle Hyperion Applications Oracle Hyperion Essbase Oracle Warehouse Builder Pivotal Greenplum PostgreSQL

QlikView SAP BO Crystal Reports SAP BO Designer SAP BO Desktop Intelligence SAP BO Repository SAP BO Data Integrator SAP BO Data Steward SAP Master Data Management SAP Sybase PowerDesigner SAP Sybase ASE Database SAS Data Integration Studio SAS BI Server SAS Information Map SAS Metadata Management SAS OLAP Server Select Sparx Architect Syncsort Tableau Talend Teradata Tigris Visible W3C DTD & XSD Schema

Operational Integration (Movement / Transformation) Metadata Harvesting (Glossary, Lineage & Impact Analysis) Oracle Database Oracle Exadata Oracle Big Data Appliance Oracle TimesTen Oracle OLAP Oracle Business Intelligence Oracle BI Applications Oracle E-Business Suite Oracle JD Edwards Enterprise One Oracle JD Edwards World Oracle Fusion Applications Oracle Governance Risk and Compliance Oracle Fusion AIA Oracle Retail Applications Oracle Agile BI / DW Oracle Agile PLM for Process Oracle iFlex FlexCUBE Oracle iFlex Mantas Oracle Hyperion Applications Oracle PeopleSoft Oracle Siebel CRM / OnDemand Oracle Communications Oracle WebLogic Server Oracle Coherence Data Grid Oracle SOA Suite Oracle Enterprise Service Bus

+ open APIs and standards based meta-model


Land Grab of the Future isData Capital

47Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | #StrataHadoop - Oracle Big Data Architecture


It’s yours for the taking
