Fred Ho | Informix Development Sept 4, 2013coiug.weebly.com/uploads/7/4/3/6/7436955/iwa-overview-v14.pdfAnalytic Integration into OLTP”, July 2012 “…Supporting both operational

© 2013 IBM Corporation1

Fred Ho | Informix Development

Sept 4, 2013

IBM Informix Warehouse Accelerator (IWA)

© 2013 IBM Corporation

Informix 12.1

Agenda

• Data Warehouse Trends

• IWA Technology Overview

• IWA Customers and Partners

• IWA Reference Architecture and Competition

• IWA Roadmap and 12.10 Features


Informix 12.1

TRENDSDatabase and Data Warehousing Industry

3


Informix 12.1

Data Warehousing Workload & Optimizations

� Data Warehousing/OLAP workload

are inherently more complex than

OLTP transactions and reasons are well-documented.

� Ways to overcome that include:– Building Indexes– Partitioning of data– Building cubes, MOLAP / ROLAP / HOLAP– Query tuning– Appliances that add a new layer of hardware

to perform I/O for DBMS

� Mixed-Workload always a challenge

� DBMS needs to be built to handle such a workload

4


Informix 12.1

Third Generation of Database Technology

According to IDC’s Article (Carl Olofson) – Feb. 2010

� 1st Generation:

– Vendor proprietary databases of IMS, IDMS, Datacom

� 2nd Generation:– RDBMS for Open Systems– Dependent on disk layout, limitations in scalability and disk I/O

� 3rd Generation: IDC Predicts that within 5 years:– Most data warehouses will be stored in a columnar fashion– Most OLTP database will either be augmented by an in-memory database

(IMDB) or reside entirely in memory

– Most large-scale database servers will achieve horizontal scalability through

clustering

5


Informix 12.1

Data Warehouse Trends for the CIO, 2011-2012

� Data Warehouse Appliances:– DW appliances are not a new concept.

– Most vendors have developed an appliance offering or promote certified configurations.

– Main reason for consideration is simplicity.

� The Resurgence of Data Marts:– Data marts can be used to optimize DW by offloading part of the workload, returning

greater performance to the warehousing environment.

� Column-Store DBMSs:– CIOs should be aware that their current DBMS vendor may offer a column-store solution.

– Don’t just buy a column-store-only DBMS because a column store was recommended by

your team.

� In-Memory DBMSs:– IMDBMS technology also introduces a higher probability that analytics and transactional

systems can share the same database.

Source:TheState of DataWarehousing in 2011; 1/31/2011; GartnerMarkBeyer,Roxane Edjlali,Donald Feinberg(IDNumber: G00209643)

6


Informix 12.1

In-Memory DBMSs (IMDBMS) Have Plenty of Promises

� Promise that they will transform the way data is provisioned and consumed

� The extreme performance enable true real-time decision making on any shape

and volume of data

� Powers unique business innovation and competitive differentiation in today’s

high-speed business environment

� Gartner estimates that IMDBMS will replace 25% of traditional data warehouse

and OLTP systems by 2016

� Examples are: Real-time pricing for airlines or banks evaluating assets for rebalancing of portfolio

� Lower TCO by reducing the need for separate OLTP and data warehouse copiesof data and by eliminating need for cubes, aggregates and indexes.

Gartner ID Number: G002215735, Publication Date: 8 Sept 2011, Authors: Roxane Edjlali, Donald Feinberg

7


Informix 12.1

IDC WW RDBMS 2012-2016 Forecast

� “ As a result of the near-ubiquitous adoption of 64-bit processors, precipitous declines in the price of main memory, and the need for greater transaction throughput, database technology is migrating from a disk-based paradigm to a memory-based one.

� Data is increasingly “stored” in memory, protected by redundant replication and asynchronous logging (for recovery), and organized for highly efficient retrieval and update.”

8


Informix 12.1

Does the 21st-Century "Big Data" Warehouse Meanthe End of the Enterprise Data Warehouse?

Key Findings:

� Organizations that deploy an EDW almost all create second and third data warehouses or marts to support additional user needs (judging from up to 90% of the data warehouse inquiries received from Gartner clients), despite strict instructions to use the EDW.

� The architectural style of a data warehouse is usually determined by the available skills and tools, and secondarily by time-to-delivery.

Source: Gartner; Mark Beyer, Donald Feinberg (ID Number G00213081

9


Informix 12.1

Market Research on IWA: What do the Analysts Say?

� White Paper by Bloor Research: “IBM Informix in Hybrid

Workload Environments”, August 2012“…for hybrid environments, Informix has a number of unique capabilities that

cannot be matched by either conventional data warehouse vendors or traditional data warehouses…”

� White Paper by Ovum Research: “Informix Accelerates

Analytic Integration into OLTP”, July 2012“…Supporting both operational and analytic workloads with the same system is

a relatively unique idea that is also being pursued by rivals such as Oracle. Ovum believes that IBM now has a strong story to tell with IWA”.

� Magic Quadrant for Data Warehouse Database Management

Systems by Gartner, Jan 2013“IBM has a vast number of products that use in-memory computing, including

solidDB, but its only in-memory solution for the data warehousing and analytical market is Informix Warehouse Edition.”

10

To download these and other papers, go to: http://www-01.ibm.com/software/data/informix/warehouse/


Informix 12.1

IWA TECHNOLOGY OVERVIEWIBM Informix Warehouse Accelerator (IWA)

11


Informix 12.1

TC

P/IP

Query

Processor

CompressedDB partition

InformixQuery Router

SQL

Results

(via DRDA)

IDS Database

Informix:

Routes SQL queries to the Accelerator

User need not change SQL or applications

Can always run query in Informix if not accelerated

Bulk Loader

Informix Warehouse Accelerator:

Connects to Informix via TCP/IP

Analizes, compresses and loads In-Memory…

…a copy of (portion of) Informix warehouse

Proceses routed SQL queries with extraorinary speed

Returns results/answer back to Informix/IDS

SQL Queries (from apps)

Informix Warehouse Accelerator

Informix Warehouse Accelerator (IWA):Overview and Seamlessly Integration with Informix/IDS

Informix Warehouse Accelerator (IWA) transparentlyaccelerates Informix warehouse/analytic queries up to

100 times or more!

Linux x86_6464-bit


Informix 12.1

Intelligent Frequence Paritioning

64 bit Intel/AMD

Prcessors

TB of RAM Memoria

Predicates Evaluationon Compressed Data

CommonValues

RareValues

Nu

mb

er

of

occu

rren

ces

SIMD

No Need for Aggregate Tables

Row and Column Store

Compresion

IWA Technology Innovations provide:Extreme unparallel analytics speed for fast business decisions


Informix 12.1

IWA: Breakthrough Technologies for Extreme Performance

14

1

2

34

5

6

7 1

2

34

5

6

7

Row & Columnar DatabaseRow format within IDS for transactional workloads

and columnar data access via accelerator for OLAP queries.

Extreme CompressionRequired because RAM is the limiting factor.

Massive ParallelismAll cores are used within used for queries

Predicate evaluation on compressed data

Often scans w/o decompression during evaluation

Frequency PartitioningEnabler for the effective parallel access of

the compressed data for scanning. Horizontal and Vertical Partition

Elimination.

In Memory Database3rd generation database technology avoids I/O. Compression allows huge databases

to be completely memory resident

Multi-core and Vector Optimized Algorithms

Avoiding locking or synchronization


Informix 12.1

How Fast is IWA?Columnar In-Memory Analytics with Unprecedented Performance

© 2013 IBM Corporation15


Informix 12.1

IBM Informix 12.10 editions: New Value-Added Software Bundles

16


Informix 12.1

CUSTOMERS AND PARTNERSInformix Warehouse Accelerator (IWA)

17


Informix 12.1

Some IWA Customers by Sector:Retail, Government, Transportation, E&U

18


Informix 12.1

Some IWA Customers by Sector:Telecommunications, Insurance, Financial, IT Services

19


Informix 12.1

Real-Life Productive IWAin Government Agency in LATAM

20


Informix 12.1

IWA Architecture and Use (same Government Agency in LATAM)

21

Real-time Analytics

IWA

37x to 456x faster !!

IDS 10.00 -> 11.70.FCxAIX on pSeries

Cubes built in

Microsoft (MSAS)IWA 11.70.FCxLinux x86_64

BI Tools

ETL: Consolidate and aggregate data in IDS to later source and build MSAS cubes

ETL

Sources/OLTP


Informix 12.1

Europe’s Largest Power company tackles the Smart Meter Big Data challenge with Informix TimeSeries + In-Memory Accelerator (IWA)

� E.ON Metering (EMTG) is the centre of excellence for the development and

commercialization of smart energy solutions and technologies and part of Europe’s largest

Power and Gas company E.ON

� EMTG operates a sophisticated Smart Meter data infrastructure based on IBM Informix

TimeSeries technology in combination with Informix In-Memory Warehouse Accelerator

� IBM Information Management products currently used:

– Informix 11.70 Ultimate Warehouse Edition

– Cognos Business Intelligence 10

22


Informix 12.1

Some of our IWA Business Partners

23


Informix 12.1

REFERENCE ARCHITECTURE & COMPETITION

Informix Warehouse Accelerator (IWA)

24


Informix 12.1

25

Current Customer BI Architecture with Informix

Source If Applicable

IBM Presentation Template Full Version

ETL

Sun T3

Solaris

IUE v11.50

Intel serversMS Windows

MS SQLserver

Cubes

BIETL

IUE : Informix Ultimate Edition

Informix

DW

Data Mart 1

Data Mart 3

Data Mart 2

Prod A

Prod E

Prod D

Prod C

Prod B


Informix 12.1

26

Target Referenced Architecture with IWA

Source If Applicable

IBM Presentation Template Full Version

ETL

Sun T3Solaris

IUWE v11.70

BI

IUWE : Informix Ultimate Warehouse Edition

Informix

DW

Prod A

Prod E

Prod D

Prod C

Prod B

Warehouse

Accelerator


Informix 12.1

IWA’s Industry Positioning and Competitors

27

DW Appliance

DataAllegro (Microsoft)

Dataupia

Greenplum (EMC)

Kognito

Netezza (IBM)

Columnar Database

Calpont

Exasol

Infobright

ParAccel

Sand Technology

Vertica (HP)

Sybase IQ (SAP)

In-Memory OLAP Tools

QlikTech/QlikView

Applix TM-1 (IBM-Cognos)

Exalytics (Oracle)

PALO

In-Memory DataWarehouse

HANA (SAP)

IWA (IBM)


Informix 12.1

ROADMAP & NEW FEATURESInformix Warehouse Accelerator (IWA)

28


Informix 12.1

Informix 11.70.xC2

� IWA 1st Release

� On SMP

Informix 11.70.xC3

� Workload Analysis Tool

� More Locales

� Data Currency

Informix 11.70.xC4

� IGWE

� IWA on Blade Server

IBM Informix Warehouse/Analytics Present Roadmap

20112011 20122012

Informix 11.70.xC5

� Partition Refresh

� Load from Secondary

� Solaris on Intel

Informix 11.70.xC6


� Load from Secondary


Informix 11.70.xC7


� Load from Secondary node in Cluster


Informix 12.10.xC1

� Bundled Cognos BI & SPSS

� Automatic incremental refresh

� Trickle feed (continuous refresh)

� Accelerate new SQL & OLAP queries

� Admin IWA using OAT & built-in functions

� Right/Real-Time In-Memory Analytics

� Big Data on Sensor data (TimeSeries+IWA)

� Informix 12.10.xC2

� Coming soon…

20132013


Informix 12.1

Summary: Key Value Propositions for the Informix Market

• State-of-the-art query accelerator for current OLTP customers

• Ideal for an embedded database in a single machine environment

•Leverage partner-based solutions & sales

• Fully compatible with existing Informix architecture

•Deploy in HA configurations

•Integration with OAT

• Ideal for SMB

•Commodity based hardware, configurable memory size, no expensive interconnect required

•Low cost entry, scaling via cluster

•Appliance not always a fit

• Seamless integration with TimeSeries for Big Machine data


Informix 12.1

Informix 12.10: Simply Powerful

28


Informix 12.1

SAPs

0

10000

20000

30000

40000

50000

60000

70000

80000

3850

(CRANFORD,SC)

3850 (Paxville, DC) 3850 (TULSA DC) 3850 (TIGERTON

QC)

3850 DUNNINGTON

(HC)

3850 x5 (NEHALEM-

EX, 8C)

SAPs

SAPs 7.0 UNICODE

SAP ECC 6.0 Basic

IWA HW & SW Breakthrough Technology Innovations

� Row and Column storage

� Compression & queryprocessing on compressed data

� Intelligent Partitioning

� No Aggregate Tables, No Indexes

� Insert Only on Delta

� Multi-Core Parallelism & Intel 64 SIMD tech.

� Massive Parallel Scaling for Loads & Queries

� In-Memory storage & query

� Extreme performance (10x up to 200x faster complex queries) using Low cost commodity HW, transparent integration with Informix ORDBMS

� Fast Storage Backup for Recoverability

In 12.10, we made IWA even more accessible by providing:

• Automatic incremental (partition-level) refresh and trickle feed (continuous refresh)

•Support for smart sensors/meters data (Time Series) –Big Data (on sensors’ data)

•Additional SQL capabilities for common OLAP queries

• Integrated administration via Open Admin Tool (OAT) and SQL API functions

IWA 12.10: Highlights of New Features and Benefits


Informix 12.1

IBM Informix 12.10: Performance for Real-Time Operational Analytics and Big Data on Sensor Data

� NEW! IWA Support for UNION queries and additional SQL support

� NEW! Much faster operational analytics and enhanced OLAP capabilities

– Enhanced integration with Cognos for much faster Cognos BI

– IDS and IWA support for OLAP functions and windowed aggregates

33


Informix 12.1


� NEW! Trickle Feed (Continuous Refresh)

– Automated, continuous updates/refresh of changed data from Informix into IWA for

speed-of-thought analysis of real-time data

34

ifx_setupTrickleFeed

Tracks changes in Dimensions

Tracks inserts in Fact tables

Automated updates in IWA

datamart


Informix 12.1


� NEW! Automatic Partition-Level Refresh

� NEW! IWA administration through OpenAdmin Tool (OAT) and SQL API functions

35


Informix 12.1


� NEW! Informix TimeSeries + Informix Warehouse Accelerator Integration

– Provides real-time analytics of stored sensor data (Big Data for sensors/meters)

36


Informix 12.1

([email protected])


Informix 12.1

Documents

Fred Ho | Informix Development Sept 4, 2013coiug.weebly.com/uploads/7/4/3/6/7436955/iwa-overview-v14.pdfAnalytic Integration into OLTP”, July 2012 “…Supporting both operational