[email protected] Dan Simchuk · SPSS Modeler Gold Predictive Analytics and Modeling SPSS Modeler Reporting and Analysis COGNOS BI COGNOS TM1 Discovery and Exploration Watson Explorer

© 2015 IBM Corporation

Introduction to the PureData for Analytics System (PDA) +

Details on the N3001 Family

Dan [email protected]

© 2015 IBM Corporation2

Legal Disclaimer

• © IBM Corporation 2015. All Rights Reserved.

• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

• Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

• All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.


PureData for Analytics Basics


Built-in Expertise • No indexes or tuning• Data model agnostic• Fully parallel, optimized In Database Analytics

Integration by Design• Server, Storage, Database in one easy to use package• Automatic parallelization and resource optimization to scale

economically • Enterprise-class security and platform management

Simplified Experience • Up and running in hours • Minimal ongoing administration• Standard interfaces to best of breed Analytics, BI, and data integration

tools• Built-in analytics capabilities allow users to derive insight from data

quickly• Easy connectivity to other Big Data Platform components

IBM PureData System for AnalyticsThe Simple Appliance for Serious Analytics

SpeedSimplicityScalability

Smart


Evolution of Netezza & PureData System for Analytics

World’s FirstData Warehouse

Appliance

World’s First100 TB DataWarehouse Appliance

World’s FirstPetabyte Data

Warehouse Appliance

World’s FirstAnalytic Data Warehouse Appliance

NPS®

8000 Series

TwinFin™ with i-Class™ Advanced Analytics

NPS®

10000 Series

TwinFin™

2003 2006 2009 2010 2012 2014

World’s Fastest and Greenest Analytical

Appliance

PureData System for AnalyticsN300x

PureData System for AnalyticsN200x

World’s First appliance with no cost encryption


PureData System for Analytics Family

▪ 10-100x faster than custom systems1

▪ 3.3x faster I/O scan rate2

▪ Load and go, no tuning▪ Designed to run

complex analytics in minutes, not hours

▪ Rich set of in-database analytics

1Based on IBM customers' reported results. "Traditional custom systems" refers to systems that are not professionally pre-built, pre-tested and optimized. Individual results may vary.2Comparing N1001 scan rate of 145 TB/hour to N2002 scan rate of 478 TB/hour

…plus▪Rack mountable appliance▪Ideal for small and medium business with up to 16 TB of user data

...plus▪“In the box” capability for real-time analytics, Hadoop data services, data movement and business intelligence▪Advanced security▪Partial rack to 8-rack configurations

▪ The hybrid computing platform integrating Netezza technology with zEnterprise technology

▪ Supports transaction processing and analytic workloads concurrently, efficiently & cost effectively

▪ Accelerates complex queries, up to 2000x faster

▪ Required security compliance with Data-at-Rest Encryption


Actionable Insight

Information Integration & GovernanceInformation Server, MDM, Guardium, Optim, Federation Server, Replication

All Data

Landing, Exploration and

Archive Data Zone

BigInsights

Operational Data Zone

DB2, Informix,PureData System for

Transactions

Real-time Data Processing & Analytics

Streams, Data Replication

Deep Analytics Modeling

PureData System for Analytics

Decision Management

SPSS Modeler Gold

Predictive Analytics and Modeling

SPSS Modeler

Reporting and AnalysisCOGNOS BI

COGNOS TM1

Discovery and ExplorationWatson Explorer

Reporting & Interactive Analysis

DB2 BLU,PureData System for

Analytics

Machine and sensor data

EnterpriseContent

Image andVideo

Social Media

Third-party data

Transaction and application data

Next Generation Architecture for Big Data and Analytics


Warehousing and Analytics, The PDA Way


▪ Too complex an infrastructure▪ Too complicated to deploy▪ Too much tuning required

▪ Too inefficient at analytics▪ Too many people needed to maintain▪ Too costly to operate

9

Traditional Data Warehouses

They do NOT to meet the demands of advanced analytics on big data.

are just too complex

Too long to get answers


Appliances Make It Simple

transforming the user experience.▪ Dedicated device

▪ Optimized for purpose

▪ Complete solution

▪ Fast installation

▪ Very easy operation

▪ Standard interfaces

▪ Low cost


Simplify …Move Analytics into the Data Warehouse

▪ Integrate the server, storage and database into one optimized package

▪ Move complex analytics into the database

▪ Leverage proven technology that accelerates analytics with no tuning or storage administration

Database AnalyticsStorageServer

Server

Storage

Database

Analytics


Data Warehouse WorkloadFewer requests, lots of data manipulation

CPU

Request

General Purpose Storage

Request

Transactional System used for BI


Data Warehouse WorkloadTransaction systems are inefficient for data shuffling

Results

Transactional System used for BI

Request

General PurposeStorage

CPU


Results

PureData for Analytics Performance Server™ System

Data Warehouse BladesDesigned for Tera-scale Business Intelligence

Intelligent StorageCPU

Request

Asymmetric Massively Parallel Processing


Results

PureData for Analytics Performance Server™ System

Data Warehouse BladesHighly efficient data movement

Intelligent StorageCPU

Request

1% of network

traffic

2% of CPU

requirements

Asymmetric Massively Parallel Processing


The PureData System for Analytics AMPP Architecture

PureData System for Analytics Appliance

FPGA

Memory

CPU

FPGA

Memory

CPU

FPGA

Memory

CPU

S-Blades NetworkFabric

Field Programmable Gate Array = a blank canvas until it’s configured

Advanced Analytics

Loaders

ETL

BI

Applications

Disk Enclosures

“Lite”Host

(IBM xSeries,Red Hat Linux)


Select State, Age, Gender, count(*) From MultiBillionRowCustomerTable Where BirthDate < ‘01/01/1960’ And State in (’FL’, ’GA’, ‘SC’, ‘NC’) Group by State, Age, Gender Order by State, Age, Gender

S-Blade Data Stream Processing

FPGA Core CPU Core

Decompress Project RestrictVisibility

SQL &Advanced Analytics

From MultiBillionRowCustomerTableWhere BirthDate <‘01/01/1960’Group by State, Age, Gender

Select State, Age, Gender, count(*)And State in (‘FL’, ‘GA’, ‘SC’, ‘NC’) Order by State, Age, Gender

From Select Where Group by

Stream viaZone Map

From


Asymmetric Massively Parallel Processing™

Massively Parallel Intelligent Storage

1

2

3

920

.

.

.

Network FabricSMP Host

DBOSFront End

IBM PureData System for Analytics Appliance

High-Speed Loader/Unloader

ODBC 3.XJDBC Type 4

OLE-DBSQL/92

Execution Engine

SQL Compiler

Query Plan

Optimize

AdminHigh-PerformanceDatabase EngineStreaming joins,

aggregations, sorts

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

SourceSystems

Client

High Performance

Loader

3rd PartyApps

DBA CLI

ETL Server

SOLARIS

LINUX

HP-UX

AIX

WINDOWS

System Z


High-PerformanceDatabase EngineStreaming joins,

aggregations, sorts

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

Execution Engine



1

2

3

920

.

.

.


DBOSFront End

IBM PureData System for Analytics Appliance


SQL Compiler

Query Plan

Optimize

Admin

SQL

1 2 3

1 2 3

1 2 3

1 2 3

Snippets

SQL

SourceSystems

Client

High Performance

Loader

3rd PartyApps

DBA CLI

ETL Server

SOLARIS

LINUX

HP-UX

AIX

WINDOWS

System Z

1 2 3


S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic



1

2

3

920

.

.

.


DBOSFront End

IBM Pure Data System for Analytics Appliance


SQL Compiler

Query Plan

Optimize

Admin

1 2 3

1 2 3

1 2 3

1 2 3

Consolidate

Execution Engine

ODBC 3.XJDBC Type 4

OLE-DBSQL/92

High-PerformanceDatabase EngineStreaming joins,

aggregations, sorts

SourceSystems

Client

High Performance

Loader

3rd PartyApps

DBA CLI

ETL Server

SOLARIS

LINUX

HP-UX

AIX

WINDOWS

System Z


Spend Less Time Managing and More Time Innovating Simplicity andEase of

Administration

▪ No dbspace/tablespace sizing and configuration▪ No redo/physical/Logical log sizing and configuration▪ No page/block sizing and configuration for tables▪ No extent sizing and configuration for tables▪ No Temp space allocation and monitoring▪ No RAID level decisions for dbspaces ▪ No logical volume creations of files▪ No integration of OS kernel recommendations▪ No maintenance of OS recommended patch levels▪ No JAD sessions to configure host/network/storage

Data Experts, not Database Experts

✓ Easy Administration Portal✓ No software installation✓ No indexes and tuning✓ No storage administration


Data Management in Legacy Databasescreate multiset table wcrm.f_monthly_billing_schedule, no fallback , no before journal, no after journal ( per_key integer not null, exposure_detail_key integer not null, billing_schedule_char_key integer not null, source_system_limit_key char(10) not null, charge_type_key smallint not null, effective_from_date date format 'yy/mm/dd', effective_to_date date format 'yy/mm/dd', amount_due decimal(18,2) compress (0.00 ,10000.00 ,50000.00 ,250000.00 ,100000.00 ), amount_due_ccy decimal(18,2) compress (0.00 ,200000.00 ,10000.00 ,50000.00 ,250000.00 ,

100000.00 ,150000.00 ), total_installments integer compress (0 ,1 ,36 ,38 ,48 ,51 ,52 ,55 ,56 ,60 ,180 ), current_installments integer compress (0 ,1 ,2 ,3 ,4 ,5 ,6 ,7 ,8 ,9 ,10 ), percent_due decimal(9,6) compress (0.000000 ,100.000000 ,10.000000 ,15.000000 ), as_of_date date format 'yy/mm/dd', last_update_event_ts timestamp(6), last_update_user_id char(8) , source_rec_id integer)primary index pmy_idx ( exposure_detail_key )partition by range_n(per_key between 200001 and 200012 each 1 ,200101 and 200112 each 1 ,200201 and 200212 each 1 ,200301 and 200312 each 1 ,200401 and 200412 each 1 ,200501 and 200512 each 1 ,200601 and 200612 each 1 ,200701 and 200712 each 1 ,200801 and 200812 each 1 );

• Journaling

• Compression

• Indexes

•Partitions

PLUS:

•Logs

•Tablespaces

•Extents

•Bit maps

•Etc.


Table conversion example – PureData for Analyticscreate multiset table wcrm.f_monthly_billing_schedule, no fallback , no before journal, no after journal ( per_key integer not null, exposure_detail_key integer not null, billing_schedule_char_key integer not null, source_system_limit_key char(10) not null, charge_type_key smallint not null, effective_from_date date format 'yy/mm/dd', effective_to_date date format 'yy/mm/dd', amount_due decimal(18,2) compress (0.00 ,10000.00 ,50000.00 ,250000.00 ,100000.00 ), amount_due_ccy decimal(18,2) compress (0.00 ,200000.00 ,10000.00 ,50000.00 ,250000.00 ,

100000.00 ,150000.00 ), total_installments integer compress (0 ,1 ,36 ,38 ,48 ,51 ,52 ,55 ,56 ,60 ,180 ), current_installments integer compress (0 ,1 ,2 ,3 ,4 ,5 ,6 ,7 ,8 ,9 ,10 ), percent_due decimal(9,6) compress (0.000000 ,100.000000 ,10.000000 ,15.000000 ), as_of_date date format 'yy/mm/dd', last_update_event_ts timestamp(6), last_update_user_id char(8) , source_rec_id integer)primary index pmy_idx ( exposure_detail_key )partition by range_n(per_key between 200001 and 200012 each 1 ,200101 and 200112 each 1 ,200201 and 200212 each 1 ,200301 and 200312 each 1 ,200401 and 200412 each 1 ,200501 and 200512 each 1 ,200601 and 200612 each 1 ,200701 and 200712 each 1 ,200801 and 200812 each 1 ) DISTRIBUTE ON (exposure_detail_key);

• Logical model only

• No indexes/partitioning

• Compression is automatic

• No physical tuning/space considerations

• Significantly reduced administration

The only consideration is how

you spread your data across all

the disks in the system


Distribution

▪ Good distribution is a fundamental element of performance!▪ A data slice is an individual element of parallelism (1000-12 = 94 data slices)▪ If all data slices have the same amount of work to do, a query will be 94 times

quicker than if one data slice was asked to do the same work▪ Bad distribution is called data skew▪ Skew to one data slice is the worst case scenario ▪ Skew affects the query in hand and others as the data slice has more to do ▪ Skew also means that the machine will fill up much quicker ▪ Simple rule. Good distribution – Good performance


A Good Distribution: 2.2 Trillion Records


Synergy with Data Integration and Reporting & Analysis Tools

▪ Ab Initio▪ Cloudera▪ Composite Software▪ IBM BigInsights▪ IBM Information

Server▪ IBM InfoSphere

Streams▪ Informatica▪ Oracle Data

Integrator▪ Oracle GoldenGate▪ SAP Business

Objects

SQL

O

DB

C

JD

BC

O

LE-D

B ▪ IBM Cognos▪ IBM SPSS▪ IBM Unica▪ Actuate▪ Information

Builders▪ Kalido▪ KXEN▪ Microsoft ▪ MicroStrategy▪ Oracle ▪ SAP Business

Objects▪ SAS▪ Tableau

SQL

O

DB

C

JD

BC

O

LE-D

B

Data In Data Out

Data IntegrationReporting &

Analysis


✓ No data movement

✓ Analyze deep and wide data

✓ High performance, parallel computation

Transformations

Mathematical

Geospatial

Predictive

Statistics

Time Series

Data Mining

PureData System for Analytics

In-Database

PureData System for Analytics: In-Database

IBM INTERNAL USE ONLY


New extensions for ESRI, Spatial and “R”

ESRI functions Open Source “R” Spatial extensions

IBM Netezza Analytics v3.2

Open Source “R”


• Basic Math*

• Permutation and Combination*

• Greatest Common Divisor and Least Common Multiple*

• Conversion of Values*

• Exponential and Logarithm*

• Gamma and Beta Functions

• Matrix Algebra+

• Area Under Curve*

• Interpolation Methods*

Transformations MathematicalTime Series

• Linear Regression+

• Logistic Regression+

• Classification

• Bayesian

• Sampling

• Model Testing

• Geospatial Data Type

• Geometric Functions

• Geometric Analysis

Predictive Geospatial* Fuzzy Logix

DB Lytix capabilities

+ Netezza Analytics and Fuzzy Logix DB Lytix capabilities

• Data Profiling / Descriptive Statistics+

• General Diagnostics

• Statistics+

• Sampling

• Data prep

Pre-Built In-Database Analytics

• Descriptive Statistics+

• Distance Measures*

• Hypothesis Testing*

• Chi-Square & Contingency Tables*

• Univariate & Multivariate Distributions+

• Monte Carlo Simulation*

• Autoregressive+

• Forecasting*

• Association Rules+

• Clustering+

• Feature Extraction+

• Discriminant Analysis*

Data Mining

Statistics


PureData System for Analytics Optimization With Other IBM Products

▪ InfoSphere Streams▪ InfoSphere BigInsights▪ System ML (Machine Learning)

▪ IBM DB2 Analytics Accelerator (IDAA)▪ zLinux ODBC driverSystem Z

Big DataPlatform

Business Intelligence / PerformanceManagement

DataIntegration

▪ Information Server v9.1▪ InfoSphere Discovery v4.5 ▪ InfoSphere Data Architect v8.1▪ InfoSphere CDC Heterogeneous Replication▪ InfoSphere Optim Data Archive 9.1▪ Industry Models v8.4 – Banking, Insurance, Healthcare▪ Industry Model Packs – Supply Chain, Customer, Market & Campaign▪ Tivoli Storage Manager▪ Vivismo Data Explorer v8.2

Coming Soon:

▪PureData System for Operational Analytics▪Guardium▪ Informix Data Warehouse Edition▪SPSS v16

▪ Cognos v10.2▪ Cognos TM1 v9.5▪ Guardium DB Monitoring v9▪ SPSS Modeler v15▪ Unica EMM Marketing Analytics 8.6▪ Unica NetInsights 8.6


PureData System for Analytics Delivers

Faster information delivery

Easy access to required data

Analytical tools that are easy to use

“Making decisions based on data instead of intuition or gut feeling is better. There is already a greater demand from users for data to support day-to-day operations – solutions such as the InfoSphere Business Glossary empower them with this information so that they can work more autonomously and efficiently.” - Philippe Chartier, BI Team Lead, Information Delivery, Canadian National Railway Company

“With the IBM PureData System for Analytics, we can reduce the time to analyze complex GIS data from days to minutes—a more than 98 percent improvement.”- Steve Trammell, Strategic Alliances Marketing Manager, Esri

“We knew that our IBM SPSS Modeler software could scale to meet our needs; the limitation was on the hardware and data warehousing side. Instead of having separate databases and servers for each client, we wanted to build a single, multi-tenant platform that could support a cloud-based service for the entire business. In the IBM PureData System for Analytics, we found the answer.”- Patrick Ritto, CTO, FleetRisk Advisors


Mini appliance early beta test resultsAvnet beta test using customer workload

IBM PureData System for AnalyticsMini Appliance (N3001-001)

MS SQL Server

3seconds

1Avnet beta test performed using customer workload on PureData System for Analytics N3001-001 compared to MS SQL Server 2008

384seconds

What could you do if your queries were 127x faster?

To hear more, come to Insight 2014, October 26-30

vs.


Comparing PureData System for Analytics with Teradata

1ITG: Comparing Costs and Time to Value with Teradata Data Warehouse Appliance, May 2014.

2.6x higherpersonnel costs1

3.4x moreDBAs required1

33% higher3-year TCO1

3.8x higherdeployment costs1

Teradata has …

…than the IBM PureData System for Analytics


Comparing PureData System for Analytics with Oracle

1ITG: Comparing Costs and Time to Value with Oracle Exadata Database Machine X3, June 2014.

3x moreDBAs required1

45% higher3-year TCO1

3.5x higherdeployment costs1

Oracle has …

…than the IBM PureData System for Analytics


The new PureData System for Analytics N3001 Family


The PureData System for Analytics N3001

Big Data and Business Intelligence readywith capabilities to unlock data’s true potential

Advanced security in an insecure worldat no extra cost

An even broader family of appliance modelsto fit a broad range of data capacity needs

Changing the game for data warehouse appliances (again)

and yes, simple is STILL better!


Big Data and Business Intelligence ReadyUnlocking Data’s True Potential

Data Warehouse Appliance

Built-in, In-Database analytic capability and integration with

a variety of 3rd party toolsReal-time AnalyticsInfoSphere Streams Developer Edition 2 users, non-production licenses

Business Intelligence Cognos software, 5 Analytics User licenses, plus 1 Analytics Administrator license

Hadoop Data ServicesInfoSphere BigInsights Software licenses to manage ~100 TB of Hadoop data

Exceptional value

provided

Included with the PureData System for Analytics N3001

Industry Process & Data ModelsModels for Banking, Financial Markets, Healthcare, Insurance, Retail, Telco

For additionalvalue

• Advanced security• New rack-mountable appliance for midsize organizations

• New 8-rack system for Petabyte+ capacity

Data Integration & TransformationInfoSphere DataStage 280 PVUs, 2 concurrent Designer Client licenses and InfoSphere Data Click

IBM InfoSphere Data Privacy and Security for Data Warehousing


IBM Netezza AnalyticsIn-database Analytics For Every Role in Your Enterprise

Bring the analytics to the data not the data to the analytics

Included

Use cases

Features

▪ Built-in, in-database analytic functions- Data mining, prediction, transformations, statistics,

geospatial, data preparation▪ Full integration with tools for BI & visualization

- IBM Cognos, Microstrategy, Business Objects, SAS, MS Excel, SSRS, Kognitio, Qlikview

▪ Full integration with tools for model building & scoring- IBM SPSS, SAS, Open Source R, Fuzzy Logix

▪ Full integration for custom analytics- Open Source R, Java, C, C++, Python, LUA

▪ Reduce hospital admissions or personalize disease treatments

▪ Achieve an order of magnitude improvement in manufacturing quality

▪ Better understand the risk of catastrophic events▪ …and many more

Data Preparation

Predictive Analytics

Geospatial Analytics

Advanced Statistics


Use cases

Features

Business IntelligenceThe Power of IBM Cognos with PureData System for Analytics

▪ Leading Business Intelligence- Interactive analysis- Compelling visualizations - web, mobile or email- Enterprise scalability

▪ Optimized for PureData for Analytics- Offers high performing OLAP over relational

experience- Cognos Dynamic Query Mode extends benefits of

PureData by adding in-memory & caching on top of already fast appliance performance

- Exploits Netezza analytic in-database functions

Rapid deployment of answers to key business questions

Included with PureData for Analytics:IBM Cognos Business Intelligence 10.2.1

5 Analytics User licenses, 1 Analytics Administrator license1

Included

▪ Reporting, analysis, scorecards, dashboards▪ Data visualization▪ Mobile business intelligence▪ … and many others

1PureData System for Analytics N3001 must be the data source for Cognos.


Data Integration & TransformationInfoSphere DataStage, Designer Client and Data Click

Rich capabilities for data integration

Included

Use cases

Features▪ Ease of Use

- Provides an easy-to-use, top-down, work-as-you-think design interface that enables users to design once and deploy anywhere—batch or real time; extract, transform, load (ETL); or extract, load, transform (ELT)

- Self-service data integration to enhance business agility

▪ Accelerate time to value- Includes a comprehensive library of

transformation components for easily defining common integration processes

▪ Integration, transform and deliver trustworthy information to your data warehouse

▪ Analysts, data scientists or even line-of-business users can easily retrieve data and populate the PureData System for Analytics

▪ Move data from the data warehouse into a subject area data mart

Included with PureData for Analytics:IBM InfoSphere DataStage 11.3 (280 PVU

Information Server Engine Tier)1, Designer Client (2 concurrent users),

InfoSphere Data Click1

1PureData System for Analytics N3001 must be the source or target database.


Hadoop Data ServicesIncluded Capability with IBM InfoSphere BigInsights

▪ Big data analytical platform- Best of open source + IBM technologies- Big SQL

- High performance SQL access of Hadoop- Federation across many data sources -

combine information from Hadoop and PureData for Analytics

- BigSheets visualization tool▪ Built-in analytics

- Text analytics, Big R

Bringing the power of Hadoop to your enterprise

Included with PureData for Analytics:InfoSphere BigInsights 3.0 software licenses for 5 enterprise nodes to

manage up to ~100 TB of Hadoop data1

Included

▪ Federated SQL access across Hadoop and your PureData System for Analytics

▪ Pre-processing and landing zone for all data types prior to loading to data warehouse

▪ Queryable backup for cold data

Use cases

Features

1Based on 4 data nodes + 1 master node. 12 TB uncompressed per data node with 4 TB drives. 12 TB x 4 nodes = 48 TB uncompressed. Using 2-2.5x compression yields 96-120 TB compressed data. Capacity will depend on hardware configuration selected.


Use cases

Features

Real-Time AnalyticsIncluded Capability from IBM InfoSphere Streams

▪ Analyze data in motion- Provides sub-millisecond response times,

allowing you to view information and events as they unfold

- Analyze all kinds of data: simple & advanced text, geospatial, acoustics, images, video, sensors

- Eclipse-based development environment

Deploy analytic models on data-in-motion to enable real-time

decisions and land data in the warehouse to build the analytic models

Included with PureData for Analytics:InfoSphere Streams Developer Edition 3.2.12 developer users, non-production licenses

Included

▪ Fraud detection▪ Predict customer churn▪ Telco real-time mediation and analysis▪ Real-time monitoring of medical sensors to improve

healthcare outcomes▪ Defect detection in manufacturing▪ Traffic pattern analysis and management


1N3001-001 does not have Hardware Acceleration (FPGA)Inside the IBM PureData System for Analytics N30011

Optimized Hardware + Software

▪Hardware accelerated AMPP

▪Purpose-built for high performance analytics

▪Requires no tuningSnippet Blades ™

▪Hardware-based query acceleration with FPGAs

▪Blistering fast results

▪Complex analytics executed as the data streams from disk

Disk Enclosures

▪User data, mirror, swap partitions▪High speed data streaming

SMP Hosts

▪SQL Compiler▪Query Plan▪Optimize▪Admin


Hardware Overview: Model N3001

▪ User Data Capacity: 192 TB1

▪ Data Scan Speed: 478 TB/hr*▪ Load Speed: 10 TB/hr

▪ Power Requirements: 7.5 kW▪ Cooling Requirements: 27,000

BTU/hr

1Assuming 4X compression

Scales up to 8 full Racks

Terabyte to Petabyte+ Capacity

Up to 10TB/hr load rate in multi-rack

configurations

2 Hosts (Active-Passive)▪ 2 Intel Ivy Bridge CPUs▪ 5X600 GB SAS Self Encrypting Drives▪ Red Hat Linux 6 64-bit

7 PureData for Analytics S-Blades™▪ 2 Intel 10 Core Ivy Bridge CPUs▪ 2 8-Engine Xilinx Virtex-6 FPGAs▪ 128 GB RAM + 8 GB slice buffer▪ Linux 64-bit Kernel

12 Disk Enclosures▪Total 288 600 GB SAS2 Self Encrypting Drives

• 240 for User Data• 14 for S-Blades• 34 Spare

▪ RAID 1 Mirroring


Self Encrypting Drive (SED) Feature Overview▪ Protecting Sensitive Data at Rest

− All data encrypted – user and temp− Local key management out of the box− 2-Tier key management

• Uses AEK (Authentication Encryption Key) • 256 bit AES key• One key for SPUs and one for Hosts

− Keys can be initialized or changed at any time even after loading data • No need to reinitialize system for setting keys

− Supports Instant Cryptoerase functionality to re-purpose the drives− nzkeybackup or nzhostbackup utilities to backup AEKs1

▪ Encrypts/Decrypts all user data at full interface speed using dedicated encryption engine− AEK locks the drive to protect data at rest− One time up front setup, No overhead to pass the key

▪ Requirements− Available on all N3001 models− NPS 7.2.0+

1Refer Netezza System Administration Guide for details


PureData System for Analytics with NPS 7.2New database features, Improved performance and predictability

Databasefeatures

Performance and Predictability

▪ WLM throughput and latency optimization

▪ Faster load rates up to 10 TB/hr

▪ Faster restore rates

▪ Enhanced security enables single sign-on and centralized management

▪ New built-in functions and SQL updates

▪ Portal enhancements

Resiliency and Serviceability

▪ Enhanced Health Check capabilities

▪ Enhanced storage topology and communication fabric

▪ Call Home via https and SOAP


Netezza support following GPFS versions:

GPFS V3.5 x86_64 on RHEL 5 (N1000 series)GPFS V3.5 x86_64 on RHEL 6 (N2000 series)GPFS V4.1 x86_64 on RHEL 6 (N2000 series)

GPFS client / server cluster is independent of NPS

Extend the logical warehouse!✓Add a Netezza node to your GPFS cluster✓Setup GPFS client for automated failover✓Use for unload / load ETL operations✓Run nzbackup / nzrestore to GPFS cluster✓Create external table and access ✓Join a FPO configured GPFS cluster

Mount and leverage the GPFS server cluster!

• Seamless capacity• High availability• 3-way mirroring• High performance• Policy-driven• Simple administration• Cost-effective

Netezza Support for GPFS


Connect to PDA without requiring a password!

Benefits Kerberos and PDA▪ Identity federation to provide user

convenience via single sign-on (SSO)

▪ Reduce security administration and costs through a federated approach

▪ Better accountability and regulatory compliance

▪ Requires kerberos v1.12.1 for best results

▪ Currently allows one method of authentication

▪ Only ADMIN will have LOCAL authentication

▪ Cross-realm authentication and multi-user are supported

▪ Supports nzSQL, ODBC, JDBC, and OLEDB

▪ Working on ability to delegate credentials and support for dual authentication (local and kerberos)

Kerberos Support


Workload Management: GRA+PQE+SQB+Job Limits

▪ Query priorities managed in the context of GRA allocations and job limits▪ Short Query Bias (SQB)

− Short queries prioritized ahead of longer running queries ▪ Powerful mechanisms for managing workloads, partitioning resources and implementing

chargeback in complex multi-user environments

LCNCLHH

Departmental User – 40 job limit

Admin Tasks – 3 job limit

Power User – 10 job limit

H CH

L L

N C

C

NC

H

HL L

Minimum Resource Guarantees with Prioritized Execution

Prioritized User Requests

Request Queues

Guaranteed Resource Allocation (GRA)

Priority Queue Execution (PQE) Job Limits


The new Latency Based Scheduler can substantially improve latency and throughput. Perfect for busy systems running high concurrency.

❶❷ ❸

Throughput – scheduling conflicts when queuing is heavy

Latency - gives preference to shorter running queries

GRA accuracy - minimizes “bursts” by predictively avoiding over-serving or under-serving specific resource groups through GRA

Short (< 2s) Medium (2s to 60s) Long- Behaves just like 7.1- Cost estimate configurable- SQB applies to Shorts- Shorter latency on average

-Cost estimate configurable-SQB is not applicable-Selected by a blend of ‘arrival’ and ‘estimate’ order

-Latency metrics available from schedqueues and logs

-Not configurable -SQB is not applicable-Minimizes “bursts”-Better average latency-Higher average throughput-Some queries will be faster and others the same

WLM – Latency Based Scheduler


PureData System for Analytics N3001-001: The Mini-Appliance

Bringing speed and simplicity to midsize organizations for big outcomes

• Rack mountable• Production ready• Full function appliance• User data capacity 16 TB* • High availability - All redundant

hardware, 4 disk spares, hot swap power supply

• Self encrypting drives, Kerberos support, LDAP/Active directory

Solution Highlights

*Assumes 4x compression

▪ Simple− Same user experience as all PureData System for

Analytics appliances• Full function Netezza Platform Software with IBM

Netezza Analytics• Support tools and Netezza Performance Portal• ODBC/JDBC/OLE-DB/SQL Driver integration

− Load and go with no tuning or administration▪ Speed

− 10-100x faster than traditional custom systems1

▪ Smart− Rich set of in database analytic functions− Protection of all data from unauthorized access− Includes starter kits for Big Data and Business Intelligence

▪ Agile− Easily incorporated into the data center with simplified

installation into an existing rack▪ Affordable

− Purchase or lease

1Based on IBM customers’ reported results. “Traditional custom systems” refers to systems that are not professionally pre-built, pre- tested and optimized. Individual results may vary.


PureData System for Analytics N3001-080 -- 8-rack System

▪ 1.5 PB of user data capacity1

▪ Hosts: 2x x3750M4 and 600 GB Self Encrypting Drives▪ Blades: 56x HS23 with 20 core IvyBridge processors▪ Storage: 96 EXP2524 disk enclosures with 24x 600 GB Self Encrypting Drives

1Assumes 4x compression


The PureData System for Analytics N3001 Family

Specification N3001-001 N3001-002 N3001-005 N3001-010 N3001-020 N3001-040 N3001-080

Racks n/a, 2 x 2U 1 (1/4 full) 1 (1/2 full) 1 2 4 8

Active S-Blades

n/a 2 4 7 14 28 56

CPU cores 40 40 80 140 280 560 1,120

User data (TB) *

16 32 96 192 384 768 1,536

* Assuming 4x compression

Single rack systems Multiple rack systems

Linear Scalability!


Business Benefits of Simplicity

▪ Lower total cost of ownership (4 DBAs -> 1 part time)▪ Faster delivery (no physical design)▪ More flexible (no need for tuning)▪ Lower risk

− Ease of Use → Fewer mistakes− Little Downtime

• Redundancy throughout the system• Maintenance and updates/upgrades included in

service contract and can be scheduled to meet workload demands.

54


THINK

55



Customer Successes


Targeted advertisingto promote products that customers want at the price they want them

Understand what customers want, when they walk into a Bon-Ton store

Freeing the time of Bon-Ton buyers and plannersfrom the mundane task of gathering & compiling customer data so they can spend their time making informed decisions to drive the business

“I need some way to understand what they're thinking, what they're feeling, without having to have contact with them. PureData for Analytics is what's going to help us understand what the customers want when they walk into my stores”

- Paula Post, Vice President Merchandising Optimization.

Bon-Ton Optimizes Their Customer’s Experience Using IBM PureData System for Analytics

Video: https:/www.youtube.com/watch?v=0gsWOL6gciw

https://www.youtube.com/watch?v=0gsWOL6gciw


Carphone Warehouse Increases Profitability Through New Revenue Streams & Reduced Costs

Case Study: http://www-03.ibm.com/software/businesscasestudies?synkey=M183113U13038J58

“The PureData System, powered by Netezza technology, provided huge technical advantages & big business advantages. We can now insure devices on behalf of a bank in the UK, which we couldn’t have done before.”

- Paul Scullion, Head of Business Intelligence

Up to 1200Xfaster performance; reports that once took an hour to run now take seconds

50% reductionin time to market for new business intelligence services

http://www-03.ibm.com/software/businesscasestudies?synkey=M183113U13038J58





96% decreasein query run times (from 1 hour to 2 minutes)

100% increasein subscriber base

Reduced spendingOn low-return promotional activities

"Through the entire subscription lifecycle, the company tracks everything members do on the website. This process generates an enormous amount of data, which would be completely wasted without the ability to extract hidden insights about how members behave.”

- eHarmony C-Level executive

eHarmony Attracts New Members by Understanding Behavior and Fine-tuning Matching Algorithm

Video: https://www.youtube.com/watch?v=_0wffNyHn8s

https://www.youtube.com/watch?v=_0wffNyHn8s


Canadian National Railway Company leverages the power of predictive analytics to run trains on time

Reduction in time spent on running reports, some reports that took 10-20 minutes earlier now run in 5 seconds

Enhanced confidence in data driven decision-making

Accelerated analytics for faster insight, the company is moving to near real time report generation compared to monthly reports earlier

“The performance of PureData is very good, most reports we have are running in less than 5 seconds where as with other databases we had reports running for 10-20 minutes”

- Philippe Chartier, BI Team Lead, Information Delivery, Canadian National Railway Company

Video: https://www.youtube.com/watch?v=yyZu5seKbLI

https://www.youtube.com/watch?v=yyZu5seKbLI


Promotes self-service business intelligence & insights throughout the hospital

98% reduction in time spent on some queries

“We’re getting deeper into the data in multiple ways . . . When we see new commonalities in treatments for children, we can design new protocols to provide the best possible care”

- Wendy Soethe, Enterprise Data Warehouse Manager

More effective diagnosis & treatment by enabling faster, more accurate insights, on-demand

Seattle Children’s Optimizes Business Intelligence & Insight into New Treatment Protocols to Enrich Patient Care

Video: https://www.youtube.com/watch?v=bjGWIectvkI

https://www.youtube.com/watch?v=bjGWIectvkI

Documents

[email protected] Dan Simchuk · SPSS Modeler Gold Predictive Analytics and Modeling SPSS Modeler Reporting and Analysis COGNOS BI COGNOS TM1 Discovery and Exploration Watson Explorer