62
Lower Maintenance Costs and Sustain System Performance with the SAP NetWeaver BI Data Aging Toolkit Bryan Katis SAP Labs

Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

  • Upload
    others

  • View
    3

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

Lower Maintenance Costs and Sustain System Performance with the SAP NetWeaver BI Data Aging ToolkitBryan KatisSAP Labs

Page 2: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 2

Learning Objectives

As a result of this session, you will be able to:Explain data aging and Information Lifecycle Management (ILM)Understand how SAP NetWeaver BI supports ILMKnow the most important aspects of implementing a data aging strategy

Page 3: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

Data Aging and Information Lifecycle ManagementMotivation to Develop a Data Aging StrategyData Aging ScenariosNear-line Storage and SAP NetWeaver 2004s BIData Archiving and SAP NetWeaver 2004s BISummary

Page 4: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

Data Aging and Information Lifecycle ManagementMotivation to Develop a Data Aging StrategyData Aging ScenariosNear-line Storage and SAP NetWeaver 2004s BIData Archiving and SAP NetWeaver 2004s BISummary

Page 5: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 5

What is Data Aging?

Data warehousing is a very powerful concept for creating a unified and consistent view of the business

In a data warehousing environment, it is typical that:Data is amassed and analyzed at an increasing rateAs time progresses, companies face the dilemma of storing more and more historical data Over time, data tends to lose its “day-to-day” relevance and is therefore accessed less frequentlyThe costs associated with maintaining historical data are high

Data aging is a strategy for managing data over time, balancing data access requirements with TCO

Each data aging strategy is uniquely determined by the customer’s data and the business value of accessing the data

The SAP BI solution provides alternatives for the typical “cost vs. business data availability” conundrum

Page 6: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 6

The challenges cannot only be addressed by purchase of additional memory. An effective administration of the data is necessary.

Challenges Facing the Information Lifecycle

Data growthEmails, attachments, Web sites, audio and video content, voice recordings …Constantly increasing data volumes in BI

Direct access capabilitiesPredictable residence time for ERP dataLong-term direct accessibility appreciated, ad-hoc analysis needs

Legal requirementsSEC, FDA, HIPPA, SOA, GDPdU, Basel II for ERP data

Data value during lifecycleConstantly decreasing in ERP environmentsMore long lasting in BI environments

CostsPersonal costs, technology costs, process costs

Technological innovationsATA disks, blue laser, etc.Write-once file system, NLS, etc.

Page 7: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 7

Information Lifecycle Management (ILM)

Everyone’s talking about ILM

ILM is not only a product

ILM is not only a toolILM is not only

software

ILM is a combination of processes and technologies with a goal of

providing the right information at the right time in the right place with the lowest possible costs

ILM is a storage management concept that actively manages all information objects during their

entire lifecycle

Page 8: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 8

Archiving Requirements of the Various Layers in a BI Environment

ERP

Entities

Relationships

Extracts

BI

EDW Layer

Operational Reporting

Strategic Analysis

Data MiningBusiness Planning

Classic archiving • Archiving objects• Legal storage regulations• Cost reduction

Description of extraction formats • DART (Data Retention Tool)• Service API for BI Extraction

Persistent BI inbound layer• High-volume, highly granular data• Direct availability not required• Data archiving processes

Persistent BI analysis horizon• High-volume data sets• Direct availability required• Data archiving process

-A

ggre

gati

on

+

+

Gra

nula

rity

-

Page 9: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 9

Information Lifecycle Management at SAP – One View

Analyze business objects, determine data categories and layers, define attributes,

identify parameters

Determine policies for attributes and

categories such as retention period,

legal compliance,data extracts

Apply policiesto business objects and

data categories

Automation, archiving,migration, deletion,accessibility

AnalyzeCategorize

Define

Policy

Apply Policy

Rea

lize

ILM

Page 10: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 10

Data Aging: The Corporation Information Factory

DSS Applications Departmental Data Marts

EDW

MarketingAcctg Finance

Sales ERPERP

ERP

CRM

eComm.

Bus. Int.

ETL

GlobalODS

Oper.Mart

Exploration warehouse/data mining

Source: Bill Inmon

Stag

ing

Are

a

LocalODS

DialogueManager

CookieCognition

Preformatteddialogues

Cross mediaStorage mgr

Near-lineStorage

Web Logs

SessionAnalysis

Internet

ERPCorporate

Applications

ChangedData

GranularityManager

Archives

SAP BWPowered by SAP NetWeaver

Page 11: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

Data Aging and Information Lifecycle ManagementMotivation to Develop a Data Aging StrategyData Aging ScenariosNear-line Storage and SAP NetWeaver 2004s BIData Archiving and SAP NetWeaver 2004s BISummary

Page 12: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 12

Typical Data Warehouse Problems

End-User ChallengesMaking timely, informed business decisions

Users cannot wait for historical data to be restored Transparent access to data for regular reporting and ad-hoc analysis

IT Management ChallengesMeeting end-user data demand while managing cost

High costs of adding/managing online disk storageHigh costs of backup and recovery – especially when data is accessed infrequently Data protection and availability

Page 13: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 13

Meta Group

“70% of all corporate data is currently located in databases”

Gartner Group

“Databases with multiple terabytes are already a reality –in the near future these will have hundreds of TBs”

“Business Applications – such as those from SAP – play a significant role in these growth in memory needs”

“Growth rate is 64%”

Constantly Increasing Database Volumes

Page 14: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 14

Distribution of Memory Costs

“The costs for data media don’t even make up a quarter of memory costs” (Giga Information Group)

“The administrative expense for 1 terabyte of memory is five to seven times as high as the memory cost itself” (Dataquest/Gartner)

Misc(Purchasing,

training)

10%

Environment(Electricity, Space)

3%

Storage-Management

(Soft & Hardware)

19%

Hard disk 23%

Personel45%

Page 15: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 15

Motivation for a Data Aging Strategy: Benefits

PerformanceFaster data load timesFaster query execution times

Cost Storage costs: High availability, high IO disks, etc.Resource and Administration overhead

System: CPU, Memory, etc.Headcount: Number of full-time employees, etc.

Control of system growth

AvailabilityData availability – faster rollups, change runs, etc.System availability – less downtime for backups, upgrades, etc.

Page 16: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 16

How to Avoid High Data Volumes in a DW Environment?

Is datastill needed?

Can the data be

aggregated?

Can the data be archived?

Yes Aggregatedata!

No

Yes

No

Data remains in the

database!

Archivedata!

Yes

Yes

Maybe

We need to addressthe no longer neededAND the maybe scenario!

Page 17: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 17

Data Aging Strategy Implementation – Initial Steps

Data aging is a strategy for managing data over time to balance data access requirements with TCO

Each data aging strategy is uniquely determined by the customer’s data and the business value of accessing the data

Which tools should I consider using?

Frequently read /updated data

Very rarelyread data

Infrequentlyread data

BW data archiving

Near-line Storage

Online DatabaseStorage

Page 18: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 18

23

1

4

Benefits of a Fundamental ILM Strategy for BI

Increase VolumeManage and use even larger amounts of information more effectivelyInformation available for any time frame for ad-hoc analyses and rebuilds

Reduce Resource ConsumptionReduction of hardware costs for hard drive on the BI sideMain memory and CPU as well as costs for system administration

Increase AvailabilityQuicker, simpler software and release management in BIReduced backup and recovery timesIntelligent data access

Optimize PerformanceSpeed up loading processes in SAP NetWeaver BISAP NetWeaver BI Query answer times in the dialog

Page 19: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

Data Aging and Information Lifecycle ManagementMotivation to develop a Data Aging StrategyData Aging ScenariosNear-line Storage and SAP NetWeaver 2004s BIData Archiving and SAP NetWeaver 2004s BISummary

Page 20: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 20

Data Model Design and Strategies: Definition

Include data aging early on in the blueprinting phase

Data retention should be determined during requirements gatheringDetermine retention for all data layers including transactional and master data

Transactional data should be evaluated from both a DataSource and an individual DataTarget perspective

Evaluate legal reporting requirements

Evaluate regulatory reporting and retention requirements

Future business data analysis requirements should also be considered

Observation: SAP typically observes data retention of three to five years in InfoCubes and DataStore objects

Data model sizing should be included in overall BW capacity planningData volume and growth should be determined

Data “change” activity profiles should also be determined

Frequency of data deletion and data updates should be included

BW capacity plans and TCO should be revisited regularly!

Page 21: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 21

Data Model Design and Strategies: Definition (cont.)

Data warehouse data retention should be integrated with your OLTP system’s archiving plans

Data warehouses normally retain more historical data than operational transaction processing systems Do you need to archive BW data if your ERP system also archives it?Does your OLTP archiving strategy limit future data warehouse developments?SAP R/3 archives can be read from BW via SAP Archive InformationSystem functionality

Keys to a successful data aging strategy development:Define data retention for all dataProfile your data activity and access Determine the capacity impact of ongoing data storageIf possible, determine a cost model for your data storage/accessChoose and implement technology that does not limit your business

Page 22: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 22

ILM in SAP NetWeaver BI

1. OnlineData persistent in the databaseData modeling aspects important Use multiple layers to control data growthFrequent cleanup necessary

2. Near-lineNear-line Storage (NLS)ADK (Archiving Development Kit) and non-ADK-basedSet up proper archiving policy Transparent access for reporting

3. OfflineClassic archiving as known in SAP BW 3.xVery cheap storage medium can be used ADK-based No access for reporting

Page 23: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 23

PSA deletion: Temporary inbound data load layer, exists for recovery or reload purposes

Retain daily loads one week, monthly retain 60 daysCustomer example: DB growing at 450 GB/month without PSA deletion, 150 GB/month with deletion

ODS object data deletionOptimize ODS Object activation performance

Change log deletion: It’s like a PSAWhen data consolidated in subsequent data targetWhen no delta at all is provided by this ODS

Active tableWhen data is consolidated in active table

Online: Data Volume Minimization Strategies

Page 24: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 24

Data consolidation: Eliminate redundant dataMerge data from similar data martsOptimize aggregatesAggressively archive and delete from InfoCube and ODS object layersSummarize at the InfoCube and aggregate level (RRI)

Remote access: Use RRI, Virtual InfoProviders, UDI, etc.: Operational processing possibly might remain in OLTP system

Online: Data Volume Minimization Strategies (cont.)

Page 25: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 25

Data has to be reloaded before availability in reporting

20042003 2005

Use Offline Archiving with MultiProvider

MultiProviders are used …

To combine actual data with reloaded data from archiveDefining data target-independent queries

BEx or Web Reporting

20052006200520042003

ADKArchive

20042003

MultiProvider

MultiProvider

ADKArchive

200520042003

Page 26: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 26

Use NLS Archiving

MultiProviders are used for …Defining data target-independent queries

BEx or Web Reporting

MultiProvider

MultiProvider

2005

NLSArchive

20042003

2006

NLSArchive

200520042003

Page 27: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 27

2002

ADKArchive

Use Offline Archiving with MultiProvider

MultiProviders are used for …Splitting data from one business scenario into separate basic InfoCubes (“logical partitioning”)

Possible partitioning criteria: Year, plan/actual, regions, business areaUse MultiProviders to cut large amounts of data into chunks

2003 2004 2005

BEx or Web Reporting

2004 2005 200620022003

ADKArchive

MultiProvider

MultiProvider

Page 28: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 28

BenefitsSingle InfoProviders smaller, less complex, and less sparsely filledthan one big InfoProvider(Parallel) data load into individual InfoProvidersBetter performance in administration on basic InfoProvidersIn most cases, queries are split automatically and distributed to InfoProviders (for more details on parallel execution see SAP Service Marketplace Note 629541)Dedicated query split for archive-related data portion as wellCentral interface for querying even if underlying data models change

DisadvantagesAdministration (with aggregates)Data has to be reloaded before available in reportingIncreasing number of InfoProviders to be administratedRemodeling at year end (are you using a new InfoCube in the new year?)New process design at year end (unload, new archive object)

Use Offline Archiving with MultiProvider (cont.)

Page 29: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 29

Single Basic InfoCube Approach with NLS

200520042003

Archive20022001

200620052004

Archive200320022001

Year Change

Data Archiving ProcessArchive Type

Near-line storage

Selection SchemaTime slices(Archive everything older than 2 years)

Schedule:January, 01

BI accelerator

Zero

Administration

Page 30: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

Data Aging and Information Lifecycle ManagementMotivation to Develop a Data Aging StrategyData Aging ScenariosNear-line Storage and SAP NetWeaver 2004s BIData Archiving and SAP NetWeaver 2004s BISummary

Page 31: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 31

Data Aging Strategy Implementation Toolkit

SAP BI’s data aging toolkit:Near-line data storage

The relocation of data from online storage to a secondary storage systemSecondary storage is typically utilizing more cost-effective online storage (e.g., IDE disks) or removable media (e.g., tape, optical)Seamless online BW Query access to data via direct SQL access

BW data archivingThe relocation of data from online storage to offline storageNo direct access to archived data for online queryingIntegrated with SAP’s Archive Development Kit

Frequently read/updated data

Very rarelyread data

Infrequentlyread data

BW data archivingNear-line storageOnline databasestorage

Page 32: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 32

Where Is Archiving and Near-line Storage Applicable?

Access frequency

reads

Data age

Archiving with SAP BW

Access frequency

reads

Data age

Near-line storage

Archiving (BW 3.x)For analysis, archived data must be reloaded first again into the BW databaseReduction in costs of data retention on alternative media

Near-line storage (> BW 3.x*)Direct access to data in alternative storage media for queries Performance and data retention coststo access aged data can be minimized

* Pilot project possible for BW 3.x

Decreasing TCO

Decreasing TCO

Page 33: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 33

TimeDays Months Years

Access Frequency

DASD

Central Access Tape

Central Capacity Tape

Virtual Tape

RAM

Costs Performance

Near-line Storage (NLS): Basics

OnlineMemory

OfflineArchiveDirect

SQL Access

MemoryManagement

Near-line Storage/Archives

Page 34: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 34

Modern NLS Memory Management System

Performance CharacteristicsPerformance- and cost-optimized management to keep data in multi-layer, transparent storage systemsCombined storage systems of disk, tape, and optical storage media in different variationsDirect row-oriented data access – also to compress archive data using all types of storage mediaAccess strategies and aging pattern for logical grouping of dataDynamic migration on cost-effective storage media for data files with regressive access frequencyAutomated process for backup, shadowing, mirroring, recovery, etc.

Page 35: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 35

Classic Archiving in SAP NetWeaver BI

Access Frequency

Age of Data

Archiving (SAP BW 3.x)ADK-based archiving solution for InfoCubes and ODS objectsCost reduction due to storing data on alternative storage mediaADK indexes for BI Query access ineffective, index structures too inflexible

Archived data must be reloaded into the SAP NetWeaver BI database for analysis purposes

Online

Archiving Archiving

Reload Previously

Online

Page 36: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 36

SAP NetWeaver BI: Near-line Storage Strategy

Age of Data

Near-line Storage (SAP NetWeaver 2004s BI)

SAP NetWeaver BI analyses have direct access to archived data in various storage mediaAvailability of historic data whilereducing costs

Physical decoupling of frequently, less frequently, or rarely used dataReloading of data into the InfoCube or DataStore object only necessary in exceptional cases

From now onOnline

Near-line Storage

Access Frequency

Page 37: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 37

The Near-line Storage Solution for SAP NetWeaver BI

Separation of frequently used (kept in database) and rarelyused (in NLS) dataNLS support for InfoCubes and DataStore objectsTransparent access to “non-archived” and “archived” data for queriesIntelligent data access

Analysis/feedback data selectionHigh-level index in SAP NetWeaver BILow-level index in NLS

Open interface for certified partnersPresent development partners

PBS Software – CBW®FileTek – StorHouse®OuterBay – LiveArchive®SAND-Technologies – Searchable Archive®

Page 38: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 38

DB Interface

SAP NetWeaver 2004s BI – Generic NLS Interface

Near-line Storage Partner Solution

OpticalLibrariesRobot.

TapeLibrary

Network-Attached Storage or Cost-Effective Data Medium

InfoCube/DataStorewith NLS

Analysis

Combination

Distribution and Sending

Data Flow

Control Flow

High-Level Index

BI Database

Low-Level IndexData Manager

Near-line Storage AdapterArchiving/Restoring

Page 39: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 39

DB Interface

SAP NetWeaver 2004s BI – Generic NLS Interface (cont.)

Near-line Storage Partner Solution

OpticalLibrariesRobot.

TapeLibrary

Network-Attached Storage or Cost-Effective Data Medium

InfoCube/DataStorewith NLS

Analysis

Distribution and Sending

Data Flow

Control Flow

High-Level Index

BI Database

Low-Level IndexData Manager

Near-line Storage AdapterArchiving/Restoring

Page 40: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 40

SAP NetWeaver 2004s BI: NLS-Based Archiving

InfoProvider InfoCubesDataStore Objects

Data Archiving ProcessArchive Type

Offline, ADK-basedNear-line, ADK-basedNear-line storage, standalone

Selection SchemaTime slices (Relative archiving periods, delta oriented, compressed data only)Request-orientedFree selection criteria

ArchiveOnline DB

Query PropertiesNear-line storage should be read as well Scheduling

viaProcess Chain

Page 41: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

Data Aging and Information Lifecycle ManagementMotivation to Develop a Data Aging StrategyData Aging ScenariosNear-line Storage and SAP NetWeaver 2004s BIData ArchivingSummary

Page 42: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 42

BW Data Archiving Technology

SAP BW 3.0B/3.1 Content provides the following functionality:BW objects data archiving

Selection of data based on any criteria (time- or data-specific)Automatically scheduled on a periodic basis

Data deletion (without archiving)Restoring (reloading) of archived data Consistent archiving processes for BW objects

“Future proof” data archives for reuse

Archive objects can be defined for the following DataTargets:InfoCubesODS objectsIt is planned to include PSA and master data archiving in future BW releases IDoc archiving has been available since BW 2.0B

Page 43: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 43

BW Data Archiving Technology: ADK Features

SAP BI data archiving is built on the Archive Development Kit (ADK). The ADK is part of the underlying WebAS technology on which mySAP applications such as SAP BW, R/3, CRM, etc., are built.

ADK Features:Open to CSP archiving and storage systems (example: IXOS)Widely used by mySAP application componentsHandles structural changesPlatform independency (code-page, number format)Data compressionStatistics and indexing supportAdministration of archiving via transaction SARAProvide logging function for archive administrationSupports indexing with Archive Information System (SAP AS)(BW does not use this functionality)Archiving objects are transport-enabled

Page 44: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 44

BW database

Technical Architecture: BW and ADK Integration

ODSobject

InfoCubesOpticalstorage system

HSMSystem

Storagesystem

ArchiveFiles

ArchiveFiles

SAP Functionality – ADK Third-party products

File system External Storage Systems

Archiving Object

Archiving Object

Data Objects

Data Objects

Page 45: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 45

DataM

anagerD

ataManager

BW Data Archiving Technology: Process Overview

Archiving ObjectArchiving Object

Write

Delete

Read

ADKADK

DatamartExtractor

DatamartExtractor

File system, CMS, HSM

InfoCube

ODS object

Scheduledread

delete

Verification

generates

activatesBW RepositoryBW Repository

ArchiveAdministration

(SARA)

ArchiveAdministration

(SARA)

Page 46: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 46

BW Data Archiving Technology: Archive Admin

Archiving object meta data is maintained at the DataTarget levelGenerated on DataTarget activation

Archiving object name is generated from DataTarget name

Administration and executionAccessed via link from Admin Workbench or transaction code SARAMaintain variants for archive execution

Data selection criteria, Archive processing and scheduling options

Scheduling and monitoring of archiving sessionsManagement of archive runsInformation systems/statistics

1:1 Relationship

Archiving ObjectInfoCube :

ODS objects

File

InfoCubes

Archiving Object

Archiving Object

Page 47: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 47

BW Data Archiving Technology: Archiving Process

Archiving Process:

Select DataTarget to be archived

Configure and generate archive

object

Configure archiving process

Test archiving process

Archive data

Delete Data

Move archive data to storage subsystem

(optional)

Successful?

Validate archive?

Yes No

Yes

No

Page 48: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 48

BW Data Archiving Technology: Object Definition

Populated after activation!

InfoCube Example

Page 49: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 49

BW Data Archiving Technology: Object Definition (cont.)

Time Slots option:Time characteristics to be used for selection criteria for archiving. “Relative time” archiving can be achieved using Time Slots.

Record selection option:Possible to choose fields usedfor selection criteria for archiving. All DataTarget fields are availableto be chosen.

Protected Archived Areas option:Ability to lock archived “time selections”from further data load activations. Archive data would have to be reloaded first before new data is activated.Note: Only available for Time Slots archiving of ODS objects.

ODS Object Examples

Page 50: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 50

BW Data Archiving Technology: Object Definition (cont.)

Archive will create a new file when max file size is reached

Number of data records written to a file before a new file is created

Defines field layout for archive file

Available (unassigned) fields for record

Page 51: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 51

BW Data Archiving Technology: Object Definition (cont.)

Logical path for archive filedefined via transaction FILE

Only required if you have an archive

storage subsystem(e.g., IXOS, Filenet,

Storhouse, etc.) Determines how the subsequent data deletion

job will be scheduled and executed

Page 52: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 52

BW Data Archiving Technology: Menu Access

Page 53: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 53

“Write to archive file” and “Data deletion” are separatedArchived data files are read for verification

CompletenessAccessibility

One-step deletion if all session files are verified (different from OLTP archiving)

Using selection criteria of archiving session Note: NOT the contents of the archive file like SAP R/3 archivesODS objects: Selective deletion from active tableInfoCube: Selective deletion from fact tablesOptimized deletion strategies

Adjust or rebuild aggregates –determined by data volumeNote: DataTarget is locked for data loads until data is deleted or the archive run is invalidated

BW Data Archiving Technology: Data Deletion

Writejob

W2

W1 Deletejob 1

Deletejob 2

V1

V2

Wn Write to archive file n

Vn Verify archive file n

D Selective deletion with selection criteria of archiving session

ODS objects

File

InfoCubes

Archiving Object

Archiving Object

Deletion

Page 54: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 54

BW Data Archiving Technology: Write Process

The write process archiving parametersRelative or absolute time-slot archiving Time granularity for data

selection for relative archiving

Ability to restrict archiving to previous closed time periods from

relative archiving,e.g., closed fiscal years,

posting periods

Ability to exclude additional specified time periods from

relative archiving

Constant time period archive specification (hard-coding)

Archive write process control

Page 55: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 55

Administration of the Archiving Process

The write process archiving parametersSelective criteria archiving (non-time slot)

Archive write process control

Ability to restrict archiving via selection criteria for all fields selected during archive object definition; individual values, ranges, exclude values, etc.

Page 56: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 56

InfoPackage is extended by an option “archive selection”

Selection options for available archive sessions and filesOnly full extraction supportedArchive files are scanned with selection criteria of the request

Reload to original DataTarget is possible but not recommendedReload recommendation:

Extract to a copy of original DataTarget insteadUse MultiProvider to combine remaining data with reloaded data

BW Data Archiving Technology: Extraction/Reloading

InfoCube Archive Files

Export DataSource

ODS object

Update Rules

InfoCube ODS object

InfoPackage

Page 57: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

Data Aging and Information Lifecycle ManagementMotivation to Develop a Data Aging StrategyData Aging ScenariosNear-line Storage and SAP NetWeaver 2004s BIData Archiving and SAP NetWeaver 2004s BISummary

Page 58: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 58

Summary

1The “Healthy” SystemDon’t start thinking about data archiving when your system is about to crash!

2Timely PlanningProactive preparation for sustainable system performance

3Interdisciplinary ProcessData archiving requires a large amount of coordination between IT and those who are responsible for applications

Page 59: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 59

For more information: Access the SAP Developer Network – www.sdn.sap.com

The central hub for the SAP technology community

Everyone can connect, contribute and collaborate- consultants, administrators and developersFocus around SAP NetWeaver and SAP xApps

High quality of technical resourcesArticles, how-to guides, weblogs, collaborative areas, discussion forums and downloads, toolkits and code-samples

A collaboration platform, not a one-way street

SAP experts from customers, partners and SAP

SDN is powered by SAP NetWeaver™Built on the SAP Enterprise PortalFeaturing collaboration capabilities of SAP Knowledge Management

Page 60: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 60

7 Key Points to Take Home

TCO can be dramatically decreased for your EDW by implementing a data aging strategy

Successful data warehouse implementations are cost-effective solutions that continue to drive the business

It is imperative that you consider data retention early on during the blueprinting phase of your project

Data retention requirements should be routinely assessed and adjusted based on business demands and costs

SAP NetWeaver BI provides a rich set of tools for implementing your data- aging strategy

SAP NetWeaver BI’s Near-line Storage solution provides a compelling, cost-effective alternative to online database storage for less frequently accessed data

Data archiving is a very flexible and powerful tool for managing infrequently accessed data in your data warehouse

Page 61: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 61

Q&A

Questions?

[email protected]

Bryan Katis

Page 62: Lower Maintenance Costs and Sustain System Performance with … · 2017-02-23 · Lower Maintenance Costs and Sustain System Performance with ... ATA disks, blue laser, ... Corporate

© SAP AG 2006, 62

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.Microsoft, Windows, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries, pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, and Informix are trademarks or registered trademarks of IBM Corporation in the United States and/or other countries.Oracle is a registered trademark of Oracle Corporation.UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc.HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology. Java is a registered trademark of Sun Microsystems, Inc.JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. MaxDB is a trademark of MySQL AB, Sweden.SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may varyThe information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG.This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP® product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/or development. Please note that this document is subject to change and may be changed by SAP at any time without notice.SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement.SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence.The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages

Copyright 2006 SAP AG. All Rights Reserved