Upload
eleanore-anderson
View
223
Download
6
Tags:
Embed Size (px)
Citation preview
© 2013 IBM Corporation
DataWarhouse rješenje za banke - IBM NetezzaLeader in Data Warehouse Appliances
Robert Božič[email protected]
© 2013 IBM Corporation
Information Management
The IBM Netezza Appliance: Revolutionizing Analytics
What is Netezza?
© 2013 IBM Corporation
Information Management
3
completely transforming the user experience.
Appliances make it simple,
Dedicated device
Optimized for purpose
Complete solution
Fast installation
Very easy operation
Standard interfaces
Low cost
© 2013 IBM Corporation
Information Management
The IBM Netezza Appliance: Revolutionizing Analytics
Purpose-built analytics engine
Integrated database, server & storage
Standard interfaces
Low total cost of ownership
Speed: 10-100x faster than traditional systems
Simplicity: Minimal administration
Scalability: Peta-scale user data capacity
Smart: High-performance advanced analytics
© 2013 IBM Corporation
Information Management
Some of the customers
Zavarovalnica Maribor
Zavarovalnca Triglav
Telekom Slovenije
Tuš Mobil
Petrol
Informatika
NLB
MKZ
Fabrika Duvana Sarajevo
© 2013 IBM Corporation
Information Management
Page 6
Digital Media
Financial Services
Government
Health & Life Sciences
Retail / Consumer
Products
Telecom
Other
© 2013 IBM Corporation
Information Management
Business case – financial institutionEvolution of DWH environment
Year 2000 – first “real” data warehouse– 1 power user, 20 common users, 1 IT developer
Years 2001 – 2005– Number of users increased – 3 IT professionals, 100 common users, 10 power
users– Enlargment of the data warehouse by factor 3– DWH/BI becomes invloved in all crucial processes
Years 2005 – 2007– Enlargment of the data warehouse by 100– Increasing the number of common users to 250– By end of 2007 DWH/BI declared as a key business process at ZM
© 2013 IBM Corporation
Information Management
Users expectations
High demand for information from the DWH system – the users expect that the system contains everything
“Immediate” responsiveness• Daily or even shorter frequencies for data loading• Real time BI• Sophisticated iterative analysis: marketing analysis, product or
customer profitability analysis etc.• Quick ad-hoc reporting for various purposes
© 2013 IBM Corporation
Information Management
Main issues with existing DWH
• More and more administration: database servers, application servers, BI tools, BI applications...
• Higher and higher ownership costs (HW; SW licences...)
• Smaller and smaller time window for ETL
• More and more end users
• Continously increased complexity of the reports and analysis - killer queries which can only be run in the evening hours , “freeze” the server, even on DWH server.
© 2013 IBM Corporation
Information Management
Main challenges with building fully mature DWH/BI infrastructure
• Cost boost• Entering in the upper cost-range of DWH servers• Unfavorable licensing policy for database SW (number of users,
CPU licensees) • Complex administration of database servers• Required 1 - 4 employees to manage DWH
© 2013 IBM Corporation
Information Management
Benefits
• Considered by the management as one of the best IT investments in the last 10 years
• The same DWH/BI team is capable to manage even larger BI infrastructure.
• Vast simplification of many complex queries and batch jobs
• Development of new BI solutions
• Users completely changed the way of thinking what they can get out of DWH
© 2013 IBM Corporation
Information Management
Smart
“Because of (Netezza’s) in-database technology, we believe we'll be able to do 600 predictive models per year (10X as many as before) with the same staff."
Eric Williams,
CIO and executive VP
Coupon redemption rates as high as 25%
Predicts what shoppers are likely to buy in future visits
© 2013 IBM Corporation
Information Management
Managing The Netezza Appliance
No storage administration
No database tuning
No software installation
© 2013 IBM Corporation
Information Management
Data Integration
Data In
The Netezza Appliance – Loading
Ab Initio
Business Objects/SAP
Composite Software
Expressor Software
GoldenGate Software (Oracle)
Informatica
IBM Information Server
Sunopsis (Oracle)
WisdomForce
SQ
L
OD
BC
J
DB
C
OLE
-DB
© 2013 IBM Corporation
Information Management
The Netezza Appliance – Querying
Reporting & AnalysisActuate
Business Objects/SAP
Cognos (IBM)
Information Builders
Kalido
KXEN
MicroStrategy
Oracle OBIEE
QlikTech
Quest Software
SAS
SPSS (IBM)
Unica (IBM)
SQ
L
OD
BC
J
DB
C
OLE
-DB
Data Out
© 2013 IBM Corporation
Information Management
Simple to Deploy and Operate
17
Operations Simply load and go .… it’s an appliance Installation to Business Value in ~2 days Ease of Evaluation and Perform As Advertised
BI Developers Data model agnostic No configuration or physical modeling No indexes or tuning – out of the box performance Focus on business value, not physical design
ETL Developers Faster load and transformation times No aggregate tables needed – simpler ETL logic In-database transformation – ‘ELT’
Business Analysts On-Stream processing by 100’s of nodes Train of thought analysis – 10 to 100x faster True ad hoc queries Lower latency – load & query simultaneously
© 2013 IBM Corporation
Information Management
Server
CACHE
IBM Netezza True Appliance Architecture
SQL
DATA
Source Systems
Client
High Performance
Loader
3rd PartyApps
DBA CLI
ETL Server
SOLARIS
LINUX
HP-UX
AIX
WINDOWS
TRU64
SQL Data
Storage
CACHE
Database
CACHEI/O I/O
© 2013 IBM Corporation
Information Management
IBM Netezza True Appliance Architecture
Source Systems
Client
High Performance
Loader
3rd PartyApps
DBA CLI
ETL Server
SOLARIS
LINUX
HP-UX
AIX
WINDOWS
TRU64
ODBC 3.XJDBC Type 4
SQL-92SQL-99 Analytics
Database, Server, Storage - in one
Storage
CACHE
Server
CACHE
Database
CACHE I/O I/O
© 2013 IBM Corporation
Information Management
IBM Netezza True Appliance Architecture
21
Optimized Hardware+Software
Purpose-built for high performance analytics; requires no tuning
Streaming Data
Hardware-based query acceleration for blistering fast results
True MPP
All processors fully utilized for maximum speed and efficiency
Deep Analytics
Complex analytics executed in-database for deeper insights
© 2013 IBM Corporation
Information Management
IBM Netezza True Appliance Massively Parallel Processing
Massively Parallel Intelligent Storage
1
2
3
960
ŸŸŸ
Network FabricSMP Host
DBOSFront End
High-Speed Loader/Unloader
ODBC 3.XJDBC Type 4
OLE-DBSQL/92
Execution Engine
Execution Engine
SQL Compiler
Query Plan
Optimize
Admin
SQL Compiler
Query Plan
Optimize
Admin
Source Systems
Client
High Performance
Loader
3rd PartyApps
DBA CLI
ETL Server
SOLARIS
LINUX
HP-UX
AIX
WINDOWS
TRU64
High-PerformanceDatabase EngineStreaming joins,
aggregations, sorts
S-Blade
Processor &
streaming DB logic
S-Blade
Processor &
streaming DB logic
S-Blade
Processor &
streaming DB logic
S-Blade
Processor &
streaming DB logic
© 2013 IBM Corporation
Information Management
High-PerformanceDatabase EngineStreaming joins,
aggregations, sorts
S-Blade
Processor &
streaming DB logic
S-Blade
Processor &
streaming DB logic
S-Blade
Processor &
streaming DB logic
S-Blade
Processor &
streaming DB logic
Massively Parallel Intelligent Storage
1
2
3
960
ŸŸŸ
Network FabricSMP Host
DBOSFront End
High-Speed Loader/Unloader
SQL Compiler
Query Plan
Optimize
Admin
SQL Compiler
Query Plan
Optimize
Admin
SQL
Snippets
1 2 3
SQL
Source Systems
Client
High Performance
Loader
3rd PartyApps
DBA CLI
ETL Server
SOLARIS
LINUX
HP-UX
AIX
WINDOWS
TRU64
IBM Netezza True Appliance Massively Parallel Processing™
Execution Engine
Execution Engine
2 3
2 3
2 3
2 3
© 2013 IBM Corporation
Information Management
High-PerformanceDatabase EngineStreaming joins,
aggregations, sorts
S-Blade
Processor &
streaming DB logic
S-Blade
Processor &
streaming DB logic
S-Blade
Processor &
streaming DB logic
S-Blade
Processor &
streaming DB logic
IBM Netezza True Appliance Massively Parallel Processing™
Massively Parallel Intelligent Storage
1
2
3
960
ŸŸŸ
Network FabricSMP Host
DBOSFront End
High-Speed Loader/Unloader
SQL Compiler
Query Plan
Optimize
Admin
SQL Compiler
Query Plan
Optimize
Admin
2 3
2 3
2 3
2 3
Consolidate
Source Systems
Client
High Performance
Loader
3rd PartyApps
DBA CLI
ETL Server
SOLARIS
LINUX
HP-UX
AIX
WINDOWS
TRU64
Execution Engine
Execution Engine
© 2013 IBM Corporation
Information Management
Our Secret Sauce
FPGA Core CPU Core
Uncompress Project Restrict,Visibility
Complex ∑Joins, Aggs, etc.
select DISTRICT,
PRODUCTGRP,
sum(NRX)
from MTHLY_RX_TERR_DATA
where MONTH = '20091201'
and MARKET = 509123
and SPECIALTY = 'GASTRO'
Slice of table
MTHLY_RX_TERR_DATA
(compressed)
Slice of table
MTHLY_RX_TERR_DATA
(compressed)where MONTH = '20091201'
and MARKET = 509123
and SPECIALTY = 'GASTRO'
where MONTH = '20091201'
and MARKET = 509123
and SPECIALTY = 'GASTRO'
sum(NRX)sum(NRX)select DISTRICT,
PRODUCTGRP,
sum(NRX)
select DISTRICT,
PRODUCTGRP,
sum(NRX)
© 2013 IBM Corporation
Information Management
Appliance family for data life-cycle management
26
Skimmer N1001/N2001 CruiserDev & Test System Data Warehouse
High Performance Analytics
Queryable ArchivingBack-up / DR
1 TB to 10 TB 1 TB to 1.5 PB 100 TB to 10 PB
© 2013 IBM Corporation
Information Management
Advanced Analytics the Netezza Way
SPSS
R, S+
SQL
SQL
Fraud Detection
Fraud Detection
Demand Forecasting
Demand Forecasting
complex analyticsSAS, SPSS, R, Java, etc
implicit parallelism petabyte scalability appliance simplicity
© 2013 IBM Corporation
Basic Math* Permutation and
Combination* Greatest Common
Divisor and Least Common Multiple*
Conversion of Values* Exponential and
Logarithm* Gamma and Beta
Functions Matrix Algebra+ Area Under Curve* Interpolation Methods*
Transformations MathematicalTime Series
Linear Regression+
Logistic Regression+
Classification
Bayesian
Sampling
Model Testing
Geospatial Data Type
Geometric Functions
Geometric Analysis
Predictive Geospatial* Fuzzy Logix
DB Lytix capabilities
+ Netezza Analytics and Fuzzy Logix DB Lytix capabilities
Data Profiling / Descriptive Statistics+
General Diagnostics
Statistics+
Sampling
Data prep
Pre-Built In-Database Analytics
Descriptive Statistics+
Distance Measures*
Hypothesis Testing*
Chi-Square & Contingency Tables*
Univariate & Multivariate Distributions+
Monte Carlo Simulation*
Autoregressive+
Forecasting*
Association Rules+
Clustering+
Feature Extraction+
Discriminant Analysis*
Data Mining
Statistics
© 2013 IBM Corporation
Information Management
We prove we are simpler
We prove we deliver performance
We prove we work within your environment
We prove we integrate with your 3rd party tools
We prove we are “easy to do business with”
We prove we have the lowest TCO
We prove business value
Bold Claims, but...We Prove Them!