Nls

SAP BW Near-line Storage (NLS)

Dr. Michael Hahne

Hahne Consulting GmbH

February 27, 2013

2012 SAP AG. All rights reserved. 2

Disclaimer

This presentation outlines SAPs general product direction and should not be relied on in making a purchase decision. This presentation is not subject to any license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.

26.02.13 Copyright Hahne Consulting GmbH 2013 2


Agenda SAP BW Near-line Storage (NLS)

n What is Near-line Storage (NLS)? n History, Strategy, Definition, Differentations

n What is the NLS Interface in SAP Business Warehouse (BW)? n Architecture, Functionality, Modeling Objects, Designtime vs. Runtime ...

n What is an Implementation for the SAP BW NLS Interface? n Architecture, Partner Solutions, Differentiators, License Model ...

n Whats new with the SAP owned BW NLS Implementation for Sybase IQ?

n Benefits, Realization, Availability, Piloting, Licensing, Roadmap ...



Data-Aging Strategies resp. Information Life Cycle Definition of Near-line Storage (NLS) for SAP BW

Storage Type / Data Category Online Database

Near-Line Storage (read only)

Classic Archive (read only)

Frequently read / changed data

(actual)

Infrequently read data (mature)

Very rarely read data (aged)

Near-line Storage (NLS) nalternative storage type with SQL based direct access

capabilities for reporting, ETL, and look-ups nextracted NLS partitions are deleted in RDBMS (NLS

storage and Online Storage together consistently reflect the BW data persistency of an InfoProvider)

nNLS storage is much more cheaper than DB storage nNLS partitions of an InfoProvider are write-protected

(!) nNLS is primarily partitioned by time/age



ADK Archive

RDBMS

Design Aspects for a Classic BW Environment Near-line Storage (NLS) vs. BW Accelerator (BWA)

InfoMarts (InfoCube)

Near-line Storage

BWA

Acquisition

Acceleration Archiving

BW

Access - very frequently frequently not frequently rarely



Performance Data volume

Data is read and/or written frequently In memory No restrictions, all features available

Infrequent access On disk, no need to keep in memory all the time No restrictions, all features available

Sporadic access Not stored in HANA DB; stored in Near-line Storage Restricted to NLS capabilities

Multi Temperature Data

Non-Active Data Concept

Providing lower TCO by optimized data volume management

hot

warm

cold NLS Management for read-

only data



Near-line Storage Implementation Challenges

n Analysis and Reporting are operating on a combination of Online-, and Near-line data whereby data consistency is an indispensable requirement.

n Transition processes touching several storage types have to guarantee consistency:

n Archiving and Deleting of online data has to be one Logical Unit of Work (LUW). n Rollback mechanisms for single archiving steps have to be available. n The Archive is getting the attidude of a database.

For this reason the processing of an Data Archiving Process (DAP) is structured in different phases:



NLS Interface Design and Runtime Objects

DBMS

InfoProvider

NLS Implementation

BW

DB

DAP Design Time

n Define partitioning criteria

n Generate DAP

DAP Scheduling / Monitoring / Run- Time: n Describe partition

to be extracted n Generate request n InfoProvider

Manage Menu n Process Chain n Archive / Extract (DAP

menu, Process Chain n Restore (DAP menu)

Data Archiving Process (DAP)

Q3/2010

NLS - Interface

Add. Features

n Query access to NLS n DTP access to NLS n Look up during Transforms

Q4/2010

Q1/2010 Q2/2010

Q1/2011 Younger



Data Management with Near-line Storage Implementation Aspects

Create a Data Archiving Process Create and schedule archiving requests Restore archiving requests Load data to subsequent Data Targets

LSA

Data Propagation Layer Corporate Memory

Data Acquisition Layer

DataSource

InfoSource

InfoPackage

Nearline Storage

Reporting Layer (Architected Data Marts)

MultiProvider SAP Sales InfoCube

DAP

DTP DTP

PSA

DTP

1

2 3

4

Nearline Storage

nLook-up during Transformation

nQuery Settings

nMultiprovider Settings

6

5

7

Nearline Storage

1 2

3

4

5

6

7

DTP



SAP BW - Key facts about NLS Interface

n NLS Interface Features n Supports archiving of InfoCubes and DataStore Objects nMainly time-based archiving, yet can also be based on other characteristics nLock of the archived data slice in the original InfoProviders nNLS Process Steps in Process Chain Handling nCopes with changes in the meta data to the BW objects of the archived data nDeletion of NLS partitions supported nDirect load from ADK archive files into NLS supported

n NLS should be a part of an Information Lifecycle Management (ILM) strategy n Based on well-established SAP / SAP BW archiving concepts n Data consistency guaranteed before deleting the data from source n Data archived in NLS can be incorporated into reporting n Saves storage costs and other system resources n High compression rate (up to 95% depending on implementation) n Increases retention period for analysis data n Included in the query statistic data collection (RSRT)


2012 SAP AG. All rights reserved. 12 26.02.13 Improve Performance and lower Costs for SAP BW 12

Advantages of a Data Management Strategy

Performance Faster Data Loads Faster Queries

Costs Less storage costs Less administrative effort controlled system growth

Availability of data: faster rollups and change runs etc. of system: less downtime for backups, upgrades etc.

Enterprise Data Warehousing Data Marts on Demand: more flexibility, faster rollout


The Near-line Storage Solution for SAP BW

Based on the NearLine Storage Interface Development Partners can implement their Solutions for Archiving and NLS into SAP BW 3rd Party NLS Solutions are implemented within the SAP BW ABAP Stack in partner

specific namespaces have to pass a certification process can offer specific Application Area in the SAP Support Portal have to be licensed in addition to SAP licenses can have a different release cycle compared to

SAP NetWeaver BW NLS

Partner Solution



SAP Defined Certification Category: Nearline Storage

(see also http://www.sap.com/ecosystem/customers/directories/SearchSolution.epx )

Certified Partners (January 2013)



Informatica ILM Nearline for SAP BW (Former SAND Nearline)

Reduce production storage costs by moving infrequently accessed data to a nearline data store. Does not require any database administration. Data is stored in encrypted files on standard file system, simplifying backup and restore processes as

well as replication to other environments. Reduce non-compliance risk with continued access to data in immutable, compressed files -

compression rates as high as 98%, with average of 90%+ compression rate. Contact: [email protected]



SAP NetWeaver BW Nearline Storage on DataVard OutBoard

DataVards OutBoard is the all-in-SAP nearline storage solution Built in ABAP, runs in your existing SAP landscape, no additional database or hardware required OutBoard is priced per productive system landscape. No matter how much data you load onto it. Compresses all InfoProviders, PSA and Changelogs up to 95% Contact: [email protected]

OutBoard Storage Management

OutBoard

Out

Boa

rd

N

ear-

Line

Sto

rage

Business Warehouse

SAP

NLS

In

terf

ace

Storage Mgmt.

SAP cluster tables

File / Cloud

External / Remote

DB

All SAP-certified RDBMS

Big data ready (e.g. Apache Hadoop)

File Apache Hadoop Corporate Cloud

End of lifecycle Deletion

HANA DB or



SAP NetWeaver BW Nearline Storage on PBS CBW

With CBW PBS provides an add-on solution for the data retrieval of archived/nearlined BW data.

On the basis of the SAP ADK files, the CBW solution creates aggregates on the archive data and (optionally, if needed) index structures to enable a fast retrieval option for the archived data directly from the end-user's original query.

PBS CBW NLS IQ for Sybase Analytics Server IQ utilizes the Sybase Analytics Server IQ as a nearline database and offers extremely fast query response times.

Contact: [email protected]



SAP NetWeaver BW Nearline Storage on IBM DB2

Takes full advantage of DB2 compression Fully exploits DB2s self managing capabilities Included in SAP DB2 OEM licnese at no extra cost SAP NLS (Near Online Storage) solution for DB2 has been

downported to NW 7.0 EhP1 and will be released for SAP customers with SPS06 end 2009. The solution is also contained in NW 7.0 EhP2 and NW 7.20

Contact: [email protected]

Install Near-line Storage data base



SAP owned MaxDB based Sample Solution of NLS Demo and verification implementation of the interface

MaxDB schema for user SAPR3 with NLS objects

Table structure

Not for productive use SAP BW

Because the implemention of the Nearline Interface is completely delegated to partners a test and reference implementation is necessary. See SAP Notes 1154026 and 1025273 (with attachment specifying the NLS API)



Overview SAP BW NLS Partner Solutions

n Different storage types supported n BW Database inbound

n Underlying BW DB (from Datavard) n Databases

n IBM DB6 (from IBM) n Sybase IQ (from PBS) n Sybase IQ (from DataVard) n MaxDB ( for SAPs T&E implementation) n Informatica Nearline (former SAND DNA) (from Informatica) n Apache Hadoop (from Datavard)

n File System n ADK storage systems (from PBS) n File or Apache Hadoop (from Datavard)

n Different license models n # of systems based n Volume based n User based n Mixed approach



Data analysis and assistance for ROI analysis

n Sizing of NearLine-Storage solutions: n Hardware sizing of the NearLine-Storage solution has to be done by the vendor

n Different NearLine-Storage technologies on the market n From database solutions, to file-based solutions, to column-based storage solutions

n System Analysis e.g. with DataVard Fitnesstest n Deliver a thorough analysis of BW objects distribution n Can help on estimating the data volume that may be archived / transferred to NLS for

the largest InfoProviders within the system n Considers only technical facts (and not the customers business requirements)



BWFT Result: Best case 1,7 TB savings

0

500

1000

1500

2000

2500

3000

3500

Heute mit OutBoard und ERNA

OutBoard

Other data

ODS data

Temporary data

Cube data

Master data



ERNA - Features Overview

n Application log n Batch log n IDoc tables (EDI40, EDIDS) n qRFC, tRFC n Job-Tables (TBTCO, TBTCP etc.) n Change & Transportsystem n Spool data (TST03) n Table Change Protocols n Batch Input Folders n Alert Management Data (SALRT*) n ...

System (Netweaver)

Scope of Housekeeping

n PSAs n Change Logs n Request logs & tables (RSMON* and RS*DONE) n Unused DIMIDs, Unused SIDs n Unused Masterdata n Cube & Aggregate compression n NRIV buffering n Table buffering n BI-Statistics n Process Chain Log n Errorlog for DTPs (RSBERRORLOG) n ...

Application (e.g. BW)

Central access for scheduling and monitoring Process Automation

Automated recommendations



Motivation

Main aspects of NLS Separation of frequently accessed (hot) data from less frequently accessed

(cold) data keeping the HANA in memory costs in a reasonable range Complementary to active / non-active classification within HANA

Data in NLS is highly compressed in cost-efficient storage with fewer backups

reducing operational costs



Design Aspects for BW powered by HANA Near-line Storage (NLS) vs. SAP HANA DB

RAM

Filer Persistent data

Processing data

Temporary data

IQ Persistent data

hot

warm

cold

Active Data

in HANA DB

no restrictions, all features available

active + not-active partitions

in HANA


(mostly) not-active data

in HANA


NLS

in Sybase IQ

restricted to [BW] NLS capabilities

BW on HANA

Sybase

Displacement Strategy

NLS

data volume performance



SAP BW powered by HANA and Sybase IQ NLS Architecture Overview

BI Clients

SAP HANA

Sybase IQ

MultiProvider Transient Provider InfoCube/DSO

Near-Line SDK

SAP Netweaver BW 7.3x

PBS OEM

BW NLS4IQ SAP Native

An SAP owned BW NLS Implementation for Sybase IQ offers a fully integrated solution from one provider Main aspects Deliver an ABAP based implementation of the BW

NLS interfaces

Deliver an Sybase IQ DBSL light that covers all needs for the above mentioned NLS implementation

Sybase IQ to deliver reliable and performing execution of the DBSL driven loads and queries

Planned availability Q1/2013

No longer required

new



ODBC Driver from IQ

HANA DB

BW NLS Implementation for Sybase IQ (DBSL based solution)

InfoProvider

NLS Interface

Nearline Adapter (ABAP based )

BW

DBSL for Sybase ASE (/w IQ ext.)

(Kernel based; Open SQL + Native IQ SQL via ADBC)

BW Porting

General Porting

DB Client



Preliminary Results from ongoing Performance Tests

Environment BW 7.30 on HANA Rev. 42 (Online DB)

IBM M-Box Intel Xeon X7560 64 Cores + HT 2,27 GHz 512 GByte RAM Sybase IQ 15.4 (Near-line Storage)

Nehalem BWA-Blade Intel Xeon X5670 12 Cores + HT 2,93 GHz 72 GByte RAM

Setup SPO partioned by 10 InfoCubes with 10 Mio. rows each Single target table on IQ side partitioned by Archiving Request (i.e. Time-/Dataslice)

Results 64 Mio. rows/h (26 GByte/h) for single relocation run (1 Cube) 238 Mio. Rows/h (97 GByte/h) for 4-fold parallel relocation run (4 Cubes) Still potential for improvements in single process mode and scalability (e.g. IQ 16.x)



Example for 3(4)-fold Archiving Process on single Object



Further Information SAP Public Web

SAP Public Web SAP NetWeaver Business Warehouse SCN Space http://scn.sap.com/community/data-warehousing/netweaver-bw BW Powered by HANA Overview on saphana.com https://www.saphana.com/docs/DOC-1522/ SAP NetWeaver BW 7.3 powered by SAP HANA Release Notes on SAP Help http://help.sap.com/nw73bwhana/#section1 SAP NetWeaver BW 7.3 Documentation Section Using the SAP HANA Database http://help.sap.com/saphelp_nw73/helpdata/en/AF/81F38183CF46B9B919F82D7BB7EC47/frameset.htm



How to get started


Analysis Workshop / Fitnesstest Objective: identify Quick Wins / low hanging fruits Benchmarking and Recommendations in areas of

System Performance and Robustness Data Management Housekeeping

ROI in a couple of months Proof of Concept

Test installation Proof of Concept

Project and usage Analysis and design Implementation Operation Check-up with Fitnesstest

Thank You!

Hahne Consulting GmbH Dr. Michael Hahne Managing Director [email protected] +49 671 9203662

Documents

Nls