BW - Data Extraction - V2

India SAP CoE, Slide 1

BI1003 – Data Extraction – BI1003 – Data Extraction – v2.0v2.0


Data Extraction Data Extraction 1 PrepareMe

2 TellMe

3 ShowMe

4 LetMe

5 HelpMe


Data ExtractionData Extraction1 PrepareMe

2 TellMe

3 ShowMe

4 LetMe

5 HelpMe


PrepareMePrepareMe

In this course, we will cover 3 main sub-topics :In this course, we will cover 3 main sub-topics :

- Purpose- Purpose

- Use- Use

- Challenges- Challenges


PurposePurpose

• This course demonstrates the various data extraction This course demonstrates the various data extraction methods.methods.

• Extraction Methods covered Extraction Methods covered – Business Content Extraction from SAP R/3 Business Content Extraction from SAP R/3 – LIS & LO Extraction from SAP R/3 LIS & LO Extraction from SAP R/3 – CO-PA and FI-SL Extraction CO-PA and FI-SL Extraction – Generic Extractors Generic Extractors – Flat File Extraction Flat File Extraction – UD Connect, DB Connect, SOAP Based and Open Hub UD Connect, DB Connect, SOAP Based and Open Hub

ExtractionExtraction

• DataSource EnhancementDataSource Enhancement


UseUse

• In many organizations, data is fragmented and spread In many organizations, data is fragmented and spread across many databases and applications.across many databases and applications.

• To be useful, data must be integrated, standardized, To be useful, data must be integrated, standardized, synchronized, and enriched – typically through ETL synchronized, and enriched – typically through ETL (extraction, transformation, and loading) processes. (extraction, transformation, and loading) processes.

• SAP BI provides a broad set of ETL capabilities that support SAP BI provides a broad set of ETL capabilities that support data extraction.data extraction.

• With the open interfaces of SAP BI, data can be loaded With the open interfaces of SAP BI, data can be loaded from virtually any source and can handle the huge amounts from virtually any source and can handle the huge amounts of transactional data typical of the enterprise landscape.of transactional data typical of the enterprise landscape.

• Thus extraction deals with extracting the data into the BW Thus extraction deals with extracting the data into the BW system for analysing and reporting.system for analysing and reporting.


ChallengesChallenges

• To extract large volumes of data from SAP R/3 by defining To extract large volumes of data from SAP R/3 by defining delta managementdelta management

• Modifying the already available Business content and Modifying the already available Business content and satisfying user needssatisfying user needs

• Integrating data from non SAP systems into BWIntegrating data from non SAP systems into BW


Data Extraction Data Extraction

1 PrepareMe

2 TellMe

3 ShowMe

4 LetMe

5 HelpMe

ShowMeShowMe


Data Extraction : OverviewData Extraction : Overview

BI Content Extractors

LO Cockpit

FI-SL Extractors

CO-PA ExtractorsGeneric Extractors

Flat File Extraction

DB Connect, UD Connect and SOAP

based

Datasource Enhancement


Update RulesTransfer RulesUpdate Rules

Transfer Rules

ODSODS

Process FlowProcess Flow

PSAPSA

DataSourceDataSourceDataSourceDataSource

Update RulesUpdate Rules

InfoCube

DeliveryOrder

Order Delivery

OLTP SystemData extract

Data extractData extract

Data extract

Order-Deliv.


Process FlowProcess Flow

• The data flows from the OLTP system to the PSA tables The data flows from the OLTP system to the PSA tables without any transformation.without any transformation.

• If any transformation or filtration is required in the data it can If any transformation or filtration is required in the data it can be done in the update rules and transfer rules.be done in the update rules and transfer rules.

• Then the data is loaded in the data targets like ODS. An Then the data is loaded in the data targets like ODS. An ODS object acts as a storage location for consolidated and ODS object acts as a storage location for consolidated and cleaned-up transaction data.cleaned-up transaction data.

• The data from multiple ODS is then loaded into InfoCube The data from multiple ODS is then loaded into InfoCube and thus available to generate reports.and thus available to generate reports.

• In this example, Order and Delivery Data are extracted In this example, Order and Delivery Data are extracted separately, and populate separate PSA and ODS tables. separately, and populate separate PSA and ODS tables. The data is then merged within the ODS to form a new The data is then merged within the ODS to form a new ODS object storing combined Order Delivery data. The ODS object storing combined Order Delivery data. The data then updates the InfoCube with Order Delivery data.data then updates the InfoCube with Order Delivery data.


Multiple Source SystemsMultiple Source Systems


Types of ExtractorsTypes of Extractors

FI

SAP CRM

Types of Extractors

Application Specific Cross Application

BW Content Extractors

Customer Generated Extractors

Generic Extractors

LO COCKPIT

HR COLIS

FISL

COPA

DB View

Infoset

Function Module

Application Specific DB Tables

Various SAP DB tables


FI

SAP CRM


Types of ExtractorsTypes of Extractors

• Extractors mainly are of two types:Extractors mainly are of two types:– Application SpecificApplication Specific– Cross Application Cross Application

• Application Specific Extractors consist ofApplication Specific Extractors consist of– BI Content Extractors (LO Cockpit)BI Content Extractors (LO Cockpit)– Customer Generated Extractors (CO-PA, FI-SL and LIS)Customer Generated Extractors (CO-PA, FI-SL and LIS)

• Cross Application Extractors consist of generic extractors Cross Application Extractors consist of generic extractors (based on Database table, Infoset and Function module)(based on Database table, Infoset and Function module)


Business Content ExtractionBusiness Content Extraction

FI

SAP CRM

Types of Extractors




Generic Extractors

LO COCKPIT

HR COLIS

FISL

COPA

DB View

Infoset

Function Module




FI

SAP CRMBusiness Content Extractors:

Application Specific Extractors that are activated when

business content datasources are transferred


Business Content ExtractorsBusiness Content Extractors• SAP provides extractors for almost all applications. They SAP provides extractors for almost all applications. They

are provided as an add on with the Business Content.are provided as an add on with the Business Content.• For each application like FI,CO and HR there are specific For each application like FI,CO and HR there are specific

tables in the OLTP system. tables in the OLTP system. • BI Content extractors consist of extract structures which are BI Content extractors consist of extract structures which are

based upon these tables and thus can be used to fetch the based upon these tables and thus can be used to fetch the data in BW.data in BW.

• Hence when any company implements application like FI, Hence when any company implements application like FI, CO it can directly use the already available BI Content CO it can directly use the already available BI Content Extractors without the need to create their own.Extractors without the need to create their own.

• SAP delivers these objects in Delivery version (D) and we SAP delivers these objects in Delivery version (D) and we need to convert them in the active version (A).need to convert them in the active version (A).

• Business content DataSources from a Source System are Business content DataSources from a Source System are available in BW for transferring data only after you have available in BW for transferring data only after you have converted them into the active versions in the source converted them into the active versions in the source system and replicated them.system and replicated them.


Uses of BI Content ExtractorsUses of BI Content Extractors

• Built in ExtractorsBuilt in Extractors

• High coverage (Applications and Industries)High coverage (Applications and Industries)

• Available for both Transaction and Master dataAvailable for both Transaction and Master data

• Ready to useReady to use

• Reduce implementation Costs and effortsReduce implementation Costs and efforts

• Delta capableDelta capable


LO Data Extraction : Overview LO Data Extraction : Overview

FI

SAP CRM

Types of Extractors




Generic Extractors

LO COCKPIT

HR COLIS

FISL

COPA

DB View

Infoset

Function Module




FI

SAP CRM


LO CockpitLO Cockpit

• Logistics Customizing Cockpit provides a simplified way to Logistics Customizing Cockpit provides a simplified way to extract logistics data and transfer it to SAP Business extract logistics data and transfer it to SAP Business Information Warehouse.Information Warehouse.


LO Data Extraction : Data FlowLO Data Extraction : Data Flow• As shown in the figure the document data for various As shown in the figure the document data for various

applications like Customer order, delivery and billing is first applications like Customer order, delivery and billing is first transferred to various Communication Structures.transferred to various Communication Structures.

• These communication structures are used to form various These communication structures are used to form various extract structures in R/3 which consist of various DataSources extract structures in R/3 which consist of various DataSources like 2LIS_11_VAHDR, 2LIS_11_VAITM etc.like 2LIS_11_VAHDR, 2LIS_11_VAITM etc.

• These DataSources then can be replicated to BW and These DataSources then can be replicated to BW and assigned to Infosources.assigned to Infosources.

• Then by creating transfer rules and update rules and defining Then by creating transfer rules and update rules and defining the transformation the data is loaded into various data targets the transformation the data is loaded into various data targets like InfoCube and ODS, thus available for reporting. like InfoCube and ODS, thus available for reporting.


V1 and V2 UpdatesV1 and V2 Updates• V1 Update – Synchronous UpdateV1 Update – Synchronous Update

– If you create/change a purchase order (me21n/me22n), If you create/change a purchase order (me21n/me22n), when you press 'SAVE' and see a success message when you press 'SAVE' and see a success message (PO.... changed..), the update to underlying tables (PO.... changed..), the update to underlying tables EKKO/EKPO has happened (before you saw the EKKO/EKPO has happened (before you saw the message). This update was executed in the V1 work message). This update was executed in the V1 work process.process.

• V2 Update – Asynchronous UpdateV2 Update – Asynchronous Update– If you create/change a purchase order (me21n/me22n), If you create/change a purchase order (me21n/me22n),

when you press 'SAVE' it takes few seconds to update to when you press 'SAVE' it takes few seconds to update to underlying tables EKKO/EKPO depending on system underlying tables EKKO/EKPO depending on system load. This update was executed in the V2 work process.load. This update was executed in the V2 work process.

• V3 Update has now replaced V1 and V2 updates.V3 Update has now replaced V1 and V2 updates.


V3 Update Modes in LO CockpitV3 Update Modes in LO Cockpit

• There are four Update Modes in LO ExtractionThere are four Update Modes in LO Extraction– Serialized V3 UpdateSerialized V3 Update– Direct DeltaDirect Delta– Queued DeltaQueued Delta– Un-serialized V3 UpdateUn-serialized V3 Update


Serialized V3 UpdateSerialized V3 Update

Transaction tables

Application specific custom

er exits

Update Tables

BW Delta queue

Staging Engine

Periodic Job

Delta Load

SM 13

RSA7

R3 system BW System


Serialized V3 UpdateSerialized V3 Update

• Transaction data is collected in the R/3 update tableTransaction data is collected in the R/3 update table• Data in the update tables is transferred through a periodic Data in the update tables is transferred through a periodic

update process to BW Delta queueupdate process to BW Delta queue• Delta loads from BW retrieve the data from this BW Delta Delta loads from BW retrieve the data from this BW Delta

queuequeue• Transaction postings lead to:Transaction postings lead to:

– Records in transaction tables and in update tablesRecords in transaction tables and in update tables– A periodically scheduled job transfers these postings into the A periodically scheduled job transfers these postings into the

BW delta queueBW delta queue– This BW Delta queue is read when a delta load is executed.This BW Delta queue is read when a delta load is executed.


Direct DeltaDirect Delta

Transaction tables


er exits

BW Delta queue

Staging Engine

Delta Load

R3 system BW System


Direct DeltaDirect Delta

• Each document posting is directly transferred into the BW Each document posting is directly transferred into the BW delta queuedelta queue

• Each document posting with delta extraction leads to Each document posting with delta extraction leads to exactly one LUW in the respective BW delta queues exactly one LUW in the respective BW delta queues

• Transaction postings lead to:Transaction postings lead to:– Records in transaction tables and in update tablesRecords in transaction tables and in update tables– A periodically scheduled job transfers these postings into the A periodically scheduled job transfers these postings into the

BW delta queueBW delta queue– This BW Delta queue is read when a delta load is executed.This BW Delta queue is read when a delta load is executed.


Queued DeltaQueued Delta

Transaction tables


er exits

Extraction queue

BW Delta queue

Staging Engine

Periodic Job

Delta Load

LBWQ

RSA7

R3 system BW System


Queued DeltaQueued Delta

• Extraction data is collected for the affected application in an Extraction data is collected for the affected application in an extraction queueextraction queue

• Collective run as usual for transferring data into the BW Collective run as usual for transferring data into the BW delta queuedelta queue

• Transaction postings lead to:Transaction postings lead to:– Records in transaction tables and in extraction queueRecords in transaction tables and in extraction queue– A periodically scheduled job transfers these postings into the A periodically scheduled job transfers these postings into the

BW delta queueBW delta queue– This BW Delta queue is read when a delta load is executed. This BW Delta queue is read when a delta load is executed.


Un-Serialized V3 UpdateUn-Serialized V3 Update

Transaction tables


er exits

Update Tables

BW Delta queue

Staging Engine

Periodic Job

Delta Load

SM 13

RSA7

R3 system BW System


Un-Serialized V3 UpdateUn-Serialized V3 Update

• Extraction data for written as before into the update tables Extraction data for written as before into the update tables with a V3 update modulewith a V3 update module

• V3 collective run transfers the data to BW Delta queueV3 collective run transfers the data to BW Delta queue• In contrast to serialized V3, the data in the updating In contrast to serialized V3, the data in the updating

collective run is without regard to sequence from the update collective run is without regard to sequence from the update tablestables

• Transaction postings lead to:Transaction postings lead to:– Records in transaction tables and in update tableRecords in transaction tables and in update table– A periodically scheduled job transfers these postings into the A periodically scheduled job transfers these postings into the

BW delta queueBW delta queue– This BW Delta queue is read when a delta load is executed. This BW Delta queue is read when a delta load is executed.


New LO Data Extraction : Why ?New LO Data Extraction : Why ?


New LO Data Extraction : Why ?New LO Data Extraction : Why ?

• Performance and Data VolumePerformance and Data Volume– Detailed extraction: You can deactivate the extraction of,

for example, scheduling data ('thin extractors' lead to small upload volumes)

– Document changes: only BW-relevant data changes are

updated (smaller upload volumes)

– LIS tables are not updated: Reduced data volumes due to removed redundant data storage

– Update with batch processes: No overload of everyday work'


New LO Data Extraction : Why ?New LO Data Extraction : Why ?• Simple Handling

– BW Customizing Cockpit: Central, single, maintenance tool for Logistics applications

– No LIS functionality: No knowledge of LIS Customizing update settings, etc, required

– Function enhancement: Creating simply and without modifications

• Others– Delta tables (SnnnBIW1/-2): duplicated update,

duplicated data storage


Customer Generated ExtractorsCustomer Generated Extractors

FI

SAP CRM

Types of Extractors




Generic Extractors

LO COCKPIT

HR COLIS

FISL

COPA

DB View

Infoset

Function Module




FI

SAP CRM


Customer Generated ExtractorsCustomer Generated Extractors

• In response to demands from the customers, SAP has In response to demands from the customers, SAP has designed the reporting facility in certain parts of the R/3 designed the reporting facility in certain parts of the R/3 system in such a way that it can be easily customized to system in such a way that it can be easily customized to meet customer’s requirements.meet customer’s requirements.

• The Business Information Warehouse must be able to The Business Information Warehouse must be able to process this data, with the result that it must be possible to process this data, with the result that it must be possible to define generic extractors in certain applications.define generic extractors in certain applications.

• These applications are the These applications are the – Logistics Information SystemLogistics Information System– Profitability analysisProfitability analysis– Special LedgerSpecial Ledger


LIS ExtractionLIS Extraction

FI

SAP CRM

Types of Extractors




Generic Extractors

LO COCKPIT

HR COLIS

FISL

COPA

DB View

Infoset

Function Module




FI

SAP CRM


LIS Data ExtractionLIS Data Extraction



• The LIS data flow can be described as follows:

• Operative transaction data (sales orders, billing) is stored in application data tables (for example VBAK/VBAP for sales orders)

• The information is transferred from an operative document to the Communication Structures (for example MCVBAK,MCVBAP) to update the data in the LIS).

• These R/3 Communication perform the same function as the BW Communication Structures, but they fill LIS InfoStructures in R/3 with data and not cubes in BW.

• The update rules write the information from the Communication Structure to the corresponding Information Structures.



• Although the term InfoStructure is used in R/3, the Although the term InfoStructure is used in R/3, the InfoStructure is a real transparent table and not the InfoStructure is a real transparent table and not the structure which contains data only at run time.structure which contains data only at run time.

• LIS has many reporting tools like ABC analysis, correlation, LIS has many reporting tools like ABC analysis, correlation, graphical displays but the limitations of the schema with graphical displays but the limitations of the schema with only one table makes reporting slower than it would be in only one table makes reporting slower than it would be in BW.BW.

• Thus in the LIS Extractor the data is transferred from the Thus in the LIS Extractor the data is transferred from the LIS InfoStructure in BW.LIS InfoStructure in BW.

• LIS extractors are now obsolete and replaced by LO LIS extractors are now obsolete and replaced by LO Cockpit.Cockpit.


FI-SL ExtractorsFI-SL Extractors

FI

SAP CRM

Types of Extractors




Generic Extractors

LO COCKPIT

HR COLIS

FISL

COPA

DB View

Infoset

Function Module




FI

SAP CRM


FI-SL ExtractorsFI-SL Extractors• FI-SL is an application in which data (planned and actual) FI-SL is an application in which data (planned and actual)

from different levels of OLTP applications is combined to from different levels of OLTP applications is combined to measure business performance.measure business performance.

• FI-SL includes planning functions and reporting tools. FI-SL FI-SL includes planning functions and reporting tools. FI-SL reporting in SAP R/3 is however restricted by the following :reporting in SAP R/3 is however restricted by the following :– Cross application reporting is not diverse Cross application reporting is not diverse – The OLTP system is optimized for transaction processing and The OLTP system is optimized for transaction processing and

a high reporting workload would have a negative impact on a high reporting workload would have a negative impact on the overall performance of the systemthe overall performance of the system

• The solution to the above limitation is FI-SL Reporting in The solution to the above limitation is FI-SL Reporting in BW.BW.


Uses of FI-SL ExtractorsUses of FI-SL Extractors• In FI-SL one is able to use alternate chart of accounts In FI-SL one is able to use alternate chart of accounts

(operative, group specific, country specific charts of (operative, group specific, country specific charts of accounts)accounts)

• Adjustment postings (direct data entry) can be made in the Adjustment postings (direct data entry) can be made in the FI-SL systemFI-SL system

• Various fiscal year variants enables one to create a weekly Various fiscal year variants enables one to create a weekly or monthly reportor monthly report

• Validations or substitutions allow one to check or modify the Validations or substitutions allow one to check or modify the data whenever it enters the FI-SL system.data whenever it enters the FI-SL system.

• Up to three currencies and two quantities can be run in Up to three currencies and two quantities can be run in parallel in FI-SLparallel in FI-SL

• Selective data retrieval - the assignment of transactions to Selective data retrieval - the assignment of transactions to particular company code or ledgers determines which particular company code or ledgers determines which ledgers are to be updated.ledgers are to be updated.


FI-SL Data Flow FI-SL Data Flow


CO-PA ExtractorsCO-PA Extractors

FI

SAP CRM

Types of Extractors




Generic Extractors

LO COCKPIT

HR COLIS

FISL

COPA

DB View

Infoset

Function Module




FI

SAP CRM



• Profitability Analysis (PA) is an integrated component in the Profitability Analysis (PA) is an integrated component in the SAP R/3 system.SAP R/3 system.

• All of the data related to profitability from the other SAP R/3 All of the data related to profitability from the other SAP R/3 applications is mapped in CO-PA in accordance with the applications is mapped in CO-PA in accordance with the corresponding business transactions.corresponding business transactions.

• This allows you to transfer into CO-PA billing document This allows you to transfer into CO-PA billing document data from SD, cost object costs from CO-PC, and overall data from SD, cost object costs from CO-PC, and overall costs from overall cost controlling.costs from overall cost controlling.

• CO-PA collects all of the OLTP data for calculating CO-PA collects all of the OLTP data for calculating contribution margins (sales, cost of sales, overhead costs)contribution margins (sales, cost of sales, overhead costs)

• CO-PA also has powerful reporting tools and planning CO-PA also has powerful reporting tools and planning functionsfunctions




CO-PA ExtractorsCO-PA Extractors• During billing SD, revenues and payments are transferred to During billing SD, revenues and payments are transferred to

profitability segments in Profitability Analysis. At the same time, profitability segments in Profitability Analysis. At the same time, sales quantities are valuated using the standard cost of goods sales quantities are valuated using the standard cost of goods manufactured, as specified in the cost component split from manufactured, as specified in the cost component split from CO-PC.CO-PC.

• One of the typical questions that can be answered with CO-PA One of the typical questions that can be answered with CO-PA module is what are the top products and customers in our module is what are the top products and customers in our different divisions.different divisions.

• The wide variety of analysis and planning functions in CO-PA The wide variety of analysis and planning functions in CO-PA allow you to plan, monitor and control the success of your allow you to plan, monitor and control the success of your company in terms of product-oriented, customer-oriented and company in terms of product-oriented, customer-oriented and organizational dimensions of multidimensional profitability organizational dimensions of multidimensional profitability segments.segments.


Generic ExtractorsGeneric Extractors

FI

SAP CRM

Types of Extractors




Generic Extractors

LO COCKPIT

HR COLIS

FISL

COPA

DB View

Infoset

Function Module




FI

SAP CRM



• Generic Extractors are Cross Application Extractors used in Generic Extractors are Cross Application Extractors used in scenarios where the other type of extractors are scenarios where the other type of extractors are unavailable.unavailable.

• Generic extractors are of three types:Generic extractors are of three types:– Based on Database view/TableBased on Database view/Table– Based on InfoSetBased on InfoSet– Based on Function ModuleBased on Function Module



• When should you use generic extractors?When should you use generic extractors?– Business Content does not contain a DataSource for Business Content does not contain a DataSource for

your application.your application.– The application does not feature its own generic delta The application does not feature its own generic delta

extraction methodextraction method– You are using your own programs in SAP R/3 to You are using your own programs in SAP R/3 to

populate the tablespopulate the tables• The tools for generic data extraction can be used to The tools for generic data extraction can be used to

generate an extractor for all applications. generate an extractor for all applications.


Flat file ExtractionFlat file Extraction

• BW provides facility to load data from flat files (CSV or BW provides facility to load data from flat files (CSV or ASCII files).ASCII files).

• It supports the following types of data :It supports the following types of data :– Transaction dataTransaction data– Master data, either directly or flexiblyMaster data, either directly or flexibly

• AttributesAttributes• TextText• HierarchiesHierarchies

• Flat file can be stored either on a local system or Flat file can be stored either on a local system or Application Server.Application Server.

• But from performance point of view we store the file on But from performance point of view we store the file on the applications server and then load it into BW.the applications server and then load it into BW.


UD Connect OverviewUD Connect Overview

• With the help of the SAP Web AS J2EE connectivity, UDWith the help of the SAP Web AS J2EE connectivity, UD Connect (Universal Data Connect) enables reporting andConnect (Universal Data Connect) enables reporting and analysis of both SAP and non-SAP data. analysis of both SAP and non-SAP data.

• Using UD Connect, you can access just about all relational Using UD Connect, you can access just about all relational and multi-dimensional data sources. and multi-dimensional data sources.

• UD Connect transfers the data as flat data. Multi-UD Connect transfers the data as flat data. Multi- dimensional data is converted to a flat format when UDdimensional data is converted to a flat format when UD Connect is used. Connect is used.


UD Connect OverviewUD Connect Overview

• For the connection to DataSources (in BW), UD Connect For the connection to DataSources (in BW), UD Connect uses the J2EE Connector Architecture.uses the J2EE Connector Architecture.

• BI Java Connectors that are available for various drivers, BI Java Connectors that are available for various drivers, protocols and providers as resource adapters.protocols and providers as resource adapters.– BI JDBC ConnectorBI JDBC Connector– BI ODBO ConnectorBI ODBO Connector– BI SAP Query ConnectorBI SAP Query Connector– BI XMLA ConnectorBI XMLA Connector


DB Connect OverviewDB Connect Overview

Unix

ForeignApplication

ORACLEDataBase

Purchasing

SAP BW

MS SQLDataBase

NTDB-Connect


DB Connect OverviewDB Connect Overview

• A purchasing application runs on a legacy system based on an ORACLE database.

• To analyze the data from the purchasing application, the data needs to be loaded into the BW System (possibly installed on a different database, e.g. MS SQL-Server).

• DB-Connect can be used to connect the DBMS of of the purchasing application and extract data from tables or views.


Data Transfer with DB Connect



• By default, when a BW application server is started by the SAP kernel, the system opens up a connection to the database on which the SAP system runs.

• All SQL commands (irrespective of whether they are Open or Native SQL) that are submitted by the SAP kernel or by ABAP programs, relate automatically to this default connection, meaning that they run in the context of the database transaction that is active in this connection.



• Connection data, such as database user name, user password, or database name are taken either from the profile parameters or from the corresponding environment variables (this is database-specific).

• Thus DB Connect can be used to open up other database connections in addition to the default connection and use

these connections to transfer data from tables or views into a BW system.


SOAP-Based Transfer of Data


SOAP-Based Transfer of Data

• As a rule, data transfer in BW takes place using a data request that is sent from BW to the Source System (pull from the scheduler).

• You can also send the data to SAP BW using external control. This is a data push in the SAP BW.

• Data push is possible for multiple scenarios:– Transferring Data Using the SOAP Service SAP Web AS– Transferring Data Using Web Service– Transferring Data Using SAP XI

• In all three scenarios, data transfer takes place via transfer mechanisms that are sufficient for Simple Object Access Protocol (SOAP) and are XML based.


Open Hub Service


Open Hub Service

• The open hub service enables you to distribute data from an SAP BW system into external data marts, analytical applications, and other applications.

• With this, you can ensure controlled distribution using several systems.

• The central object for the export of data is the InfoSpoke. Using this, you can define the object from which the data comes and into which target it is transferred.

• Through the open hub service, SAP BW becomes a hub of an enterprise data warehouse. The distribution of data becomes clear through central monitoring from the distribution status in the BW system.


Datasource EnhancementDatasource Enhancement• Need for DataSource enhancementNeed for DataSource enhancement

– When we require an additional field from a database table and it is not directly available from the datasource, then we can append the field in the extract structure of the datasource.

– Then the logic to populate that field can be written in the customer exit.


1 PrepareMe

2 TellMe

3 ShowMe

4 LetMe

5 HelpMe

Data Extraction Data Extraction


BI Content ExtractorsBI Content Extractors


LO Cockpit

FI-SL Extractors




based



BI Content Extractors – StepsBI Content Extractors – Steps

1. Activate Datasources

2. Replicate the Datasource in BW System

3. Assign InfoSource and maintain communication structure and transfer rules

4. Maintain InfoCube and Update Rules

5. Create InfoPackage

6. Initialize Delta Process

7. Carry Out Delta Update


Step 1 - Activate DataSourcesStep 1 - Activate DataSources

In SAP R/3, go to transaction Code SBIW Display IMG Business Content DataSources ExecuteTransfer BusinessContent DataSources


Step 1 - Activate DataSourcesStep 1 - Activate DataSources

Select the DataSource from Application Component Hierarchy.

Click on the button Transfer DataSources/Activate the DataSource.

Assign to a transport request.

Activate datasources


Step 2 - Replicate datasourcesStep 2 - Replicate datasources Tcode RSA1Tcode RSA1 ModellingModelling Source SystemsSource Systems Context Menu Context Menu

ReplicateReplicateDataSources.DataSources.


Step 3- Assign the InfoSourceStep 3- Assign the InfoSource

Maintain the Maintain the communication communication structure and Activate structure and Activate the transfer rules.the transfer rules.


Step 4 - Maintain InfoCube and Step 4 - Maintain InfoCube and Update rulesUpdate rules

Infocube

Update rules


Step 5 - Create InfoPackageStep 5 - Create InfoPackage

Go to InfoSources, Select the InfoSource

Right click on the Source System and say create InfoPackage

Scheduler screen

Monitor


Step 6 - Initialize the delta Step 6 - Initialize the delta processprocess

Go to InfoPackage and select the update type as initialize delta process.

And click on start to schedule the process

Monitor

Scheduling

Initialize the delta process


Initializing the Delta Process Initializing the Delta Process (Scheduler)(Scheduler)

• If you want to transfer data in the delta process, the process must be initialized in the first data request. • In this process, the selection conditions for connecting delta uploads are set and its complete dataset is loaded into BW from the R/3 System.• To do this, you must select the checkbox Initialize delta process for the Info Package under the update parameters of the Scheduler.• Once the data is requested message appears after scheduling, Go to Monitor to view the status and details of the data load.


Monitoring the Upload ProcedureMonitoring the Upload Procedure


Delta Update (Scheduler)Delta Update (Scheduler)

Only the data since last upload is requested


Delta Update (Scheduler)Delta Update (Scheduler)

• For high volume of transaction data, a full update here is mostly only justified for the first time data is transferred or if the statistics data has to be rebuilt following a system failure.

• Delta update mechanisms that restrict the volume of data to within realistic limits, therefore, are required to implement a performance-oriented, periodical data transfer strategy. – For e.g. when sales figures are updated every week in the

Business Information Warehouse, only the sales document information that has been added or changed in the last week should be extracted.




LO Cockpit

FI-SL Extractors




based



Logistics Customizing CockpitLogistics Customizing Cockpit

• The LO Cockpit contains the following functionality, as specified in the procedure sequenced below:

– Maintaining extract structures

– Maintaining DataSources

– Activating the update

– Controlling V3 update


LO Data Extraction (SBIW)LO Data Extraction (SBIW)

LO customizing cockpit lbwe

BW Logs (LBWF)

Perform set up Application

wiseDelete set up tables (LBWG)


Logistics Extract Structure – Logistics Extract Structure – Cockpit (LBWE)Cockpit (LBWE)

Maintain extract

structure

Set up V3 update

Activate datasources


Individual Steps in LO ExtractionIndividual Steps in LO Extraction

1. Maintain Extract Structure






5. Activate Extract Structure

6. Delete setup tables/Perform Extraction

7. Set Up periodic V3 update


Step 1 - Maintaining the Extract Step 1 - Maintaining the Extract StructureStructure

Maintain extract

structure


Maintaining the Extract Maintaining the Extract StructureStructure

• The extract structure is filled with the assigned communication structures. You can only use selected fields from the communication structures (SAP internal control fields, for example, are not offered).

• SAP already delivers extract structures, which you can enhance (by connecting to the communication structure). Every extract structure can be maintained by you and by SAP.

• After you set up the extract structure, the system automatically generates it. This completes missing fields (their units and characteristics). The extract structure is created hierarchically according to the communication structures. Every communication structure leads to the generation of a substructure that belongs to the actual extract structure.


Maintaining DataSourcesMaintaining DataSources

Maintain datasources


Maintaining DataSources : Maintaining DataSources : ProcedureProcedure• There exists a DataSource

(for e.g. 2LIS_11_VAITM) for each extract structure that is made available (for example, MC11VA0ITM) in the OLTP System.

• A maintenance screen is displayed in which you can to assign other properties to the fields of the extract structure– Selection fields– Hide fields– Cancellation fields

• (field is inverted when canceled (*-1))


Step 2 – Replicating Step 2 – Replicating DataSourcesDataSources


Step 3 - Maintaining Step 3 - Maintaining Communication StructureCommunication Structure

All the fields in the extract structure are shown with the assigned info objects

in the template


Step 4 - Maintaining Transfer Step 4 - Maintaining Transfer RulesRules


Step 5 - Maintaining InfoCubes Step 5 - Maintaining InfoCubes and Update Rulesand Update Rules


Step 6 - Activating Extract Step 6 - Activating Extract StructuresStructures

Activate extract structure


Step 7 - Delete set up tablesStep 7 - Delete set up tables

In transaction LBWG enter the application number for which you want to delete the set up tables.

For e.g. For 2LIS_11_VAITM enter the number 11 and execute.


Step 8 - Initialization/Simulation Step 8 - Initialization/Simulation (OLI*BW)(OLI*BW)

Give a name to the job

Initializing ('build') the dataset for the respective extract structures is to be done before data can be extracted from the R/3 System.The initialization is set up in the OLTP. The restructuring process fills restructuring tables that are read during the initialization.It is recommended to carry this out before, in the simulation mode. The option of checking the extract structure update, without actually updating the data is available. The results are recorded in a detailed simulation log.


Step 8 - Creating InfopackagesStep 8 - Creating Infopackages


Step 9 - Initializing the Delta Step 9 - Initializing the Delta ProcessProcess


V3 Update- SettingsV3 Update- SettingsSet up V3 updateSet the start time

and print parameters.

Schedule batch jobs


V3 Update SettingsV3 Update SettingsIn Start Date

Select immediate scheduling

Give the print parameters

Schedule the job and check the job

overview in SM37


Delta Queue MaintenanceDelta Queue Maintenance

Go to transaction RSA7.

You can see that the delta queue has been successfully filled.


Step 10 -Delta Update Step 10 -Delta Update (Scheduler)(Scheduler)

Only the data since last upload is

requested




LO Cockpit

FI-SL Extractors




based



FI-SL extraction Step-by-StepFI-SL extraction Step-by-Step

2. Define or Assign a DataSource for Ledger







1. Generate an extract structure for summary table


Step 1 - Setting up an FI-SL Step 1 - Setting up an FI-SL DataSourceDataSource


Go to SBIW Go to SBIW Settings Settings for for Application Application specific data specific data sources sources Generate Generate transfer transfer structure for structure for totals tabletotals table





Step 2 - Setting up an FI-SL Step 2 - Setting up an FI-SL datasource :datasource :

Go to SBIW Go to SBIW Settings for Settings for application application specific specific datasources datasources Financial Financial accounting accounting create create infosource for infosource for LedgerLedger


Step 2 - Setting up an FI-SL Step 2 - Setting up an FI-SL DataSource DataSource

The status The status column shows column shows whether a whether a DataSource is DataSource is assigned to the assigned to the ledger already. ledger already. (Traffic light (Traffic light green)green)

You select the You select the ledger to which ledger to which you want to you want to assign the assign the DataSourceDataSource


Step 2 - Setting up an FI-SL Step 2 - Setting up an FI-SL DataSource DataSource



• After generating the FI-SL DataSource rest of the steps are After generating the FI-SL DataSource rest of the steps are same as explained earlier.same as explained earlier.

• Replicate the DataSource and then assign it to the Replicate the DataSource and then assign it to the infosource.infosource.

• Create transfer rules and update rulesCreate transfer rules and update rules

• Create InfoPackage and initialize the delta processCreate InfoPackage and initialize the delta process




LO Cockpit

FI-SL Extractors




based



CO-PA Extraction (Steps)CO-PA Extraction (Steps)







1. Generate DataSource in OLTP


Step 1 - Generate CO-PA Step 1 - Generate CO-PA DataSourceDataSource

Go to Transaction KEB0.Define the DataSource for the current client of your SAP R/3 System on the basis of one of the operating concerns.A standard name is always generated. For e.g. 1_CO_PA%CL%ERK with prefix 1_CO_PA is mandatory.

Select the operating concern


Step 1 - Generate DataSourceStep 1 - Generate DataSource


Maintain CO-PA datasourceMaintain CO-PA datasource

You can then specify which fields are to be applied as the selection for the CO-PA extraction.

Providing required fields for selection



• After generating the CO-PA DataSource rest of the steps After generating the CO-PA DataSource rest of the steps are same as explained earlier.are same as explained earlier.

• Replicate the DataSource and then assign it to the Replicate the DataSource and then assign it to the infosource.infosource.

• Create transfer rules and update rulesCreate transfer rules and update rules

• Create InfoPackage and initialize the delta processCreate InfoPackage and initialize the delta process




LO Cockpit

FI-SL Extractors




based



Generic Data ExtractionGeneric Data ExtractionSteps for transaction DataSteps for transaction Data

1. Create DataSources for transaction Data







2. Create InfoSet/enhance table


Generic Data ExtractionGeneric Data ExtractionSteps for Master DataSteps for Master Data

1. Create DataSources for Master Data (attr)



5. Create InfoPackage for Master Data

6. Create InfoPackage for texts

7. Schedule InfoPackages

2. Create DataSources for texts


Step1 - Create generic Step1 - Create generic DataSourceDataSourceGo to Go to Transaction Transaction RSO2 RSO2 Here Here you can select you can select whether you whether you want to create a want to create a datasource for datasource for transaction data, transaction data, Attributes or Attributes or texts.texts.

Give a technical Give a technical name to the name to the datasource and datasource and choose createchoose create


Create Generic DataSourceCreate Generic DataSource

Create a Generic DataSource.

Choose an application component to which the DataSource is to be assigned.

Enter the description texts. You can choose whether

you want to create the DataSource based on View, Function Module or InfoSet query.

Choose Generic delta


Generic Data ExtractionGeneric Data Extraction Specify the delta specific field and its type. Maintain the settings for generic delta. Specify the safety interval.

NOTE: safety interval should be set so that no document is missed even if it was not stored in the DB table when extraction took place.

Select delta type : New status for changed records (i.e. after image) This can be used with data target ODS (AIE).Additive delta i.e. aggregated data records (ADD) and then choose save.

Only one attribute can be defined as delta

attribute


Generic Data ExtractionGeneric Data Extraction

.

Now choose save again. This will generate the DataSource. After generating the DataSource you will see the delta update flag selected.

Select the fields in the selection which you want to appear in the Infopackage selection.


Generic Data ExtractionGeneric Data Extraction

Transport the DataSource to BW in Tcode RSA6(Post Transport the DataSource to BW in Tcode RSA6(Post Processing of DataSources).Processing of DataSources).The steps in BW :The steps in BW :• Replicate the DataSources.Replicate the DataSources.• Create necessary Info objectsCreate necessary Info objects• Generate Info source in BW, Maintain Communication Generate Info source in BW, Maintain Communication

Structure and Transfer RulesStructure and Transfer Rules• Create Info package and schedule it.Create Info package and schedule it.


Flat File ExtractionFlat File Extraction


LO Cockpit

FI-SL Extractors




based




1. Identify the data

3. Define fields of InfoSource as InfoObjects




7. Schedule Flat file using extractor

2. Define Source System in AWB


Step 1 – Identify the DataStep 1 – Identify the DataGo to MS Excel.

Create a flat file and enter the records to be updated in the file.

Save the file as a .CSV file.

NOTE: The order of the fields in the flat file should be same as that in the communication structure of the InfoSource.


Step 2 - Create a Source SystemStep 2 - Create a Source System

RSA1RSA1

Modelling Modelling Source SystemsSource SystemsContext MenuContext Menu

CreateCreateSelect the SourceSelect the Source

System as FileSystem as File

System .System .



• After creating the Source System, create the Infoobjects in After creating the Source System, create the Infoobjects in the BW system.the BW system.

• Create an InfoSource and assign the flat file Source System Create an InfoSource and assign the flat file Source System to it and maintain the transfer rules.to it and maintain the transfer rules.

• Create and ODS and maintain update rules.Create and ODS and maintain update rules.

• Now create an infopackage and load the data.Now create an infopackage and load the data.


Hints while loading flat fileHints while loading flat file

Data separator

File in CSV format

Give the name of the CSV file which you have created in the infopackage.

Use the data separator as ;


Hints while loading flat fileHints while loading flat file

• Be sure the delimiter in the InfoPackage is given as ‘ ; ’Be sure the delimiter in the InfoPackage is given as ‘ ; ’

• You might get an error on alpha conversion. Take in mind You might get an error on alpha conversion. Take in mind the option in transfer rule to convert the data from external the option in transfer rule to convert the data from external format into internal.format into internal.


Hints while loading flat fileHints while loading flat file• Calendar day should be of Calendar day should be of

the format yyyymmdd the format yyyymmdd (SAP requires this (SAP requires this format). format).

• Sometimes you might get Sometimes you might get an error like the system an error like the system cannot open the file. Try cannot open the file. Try keeping the file in My keeping the file in My Documents or in C drive.Documents or in C drive.

• View the preview option in View the preview option in the InfoPackage before the InfoPackage before loading the fileloading the file

• Set the number of header Set the number of header rows to be ignored to 1. rows to be ignored to 1. This ignores the heading This ignores the heading row in the flat filerow in the flat file


DB Connect, UD Connect and DB Connect, UD Connect and SOAP BasedSOAP Based


LO Cockpit

FI-SL Extractors




based



Steps involved for DB ConnectSteps involved for DB Connect

1. Create a DB Connect Source System

3. Create InfoSource and assign Source System

4. Maintain communication structure and transfer rules



7. Initialize Delta Update

2. Create a DataSource


Steps involved for DB ConnectSteps involved for DB Connect

• Create a DB Connect Source SystemCreate a DB Connect Source System

• Create a DataSourceCreate a DataSource

• Create InfoSource and assign the Source System created Create InfoSource and assign the Source System created to it.to it.

• Create InfoPackage and load the data.Create InfoPackage and load the data.


Step 1 - Create a DB Connect source system

DB Connection : Give the name of the connection.

DBMS : Enter the database type to which you want to connect in this field. (i.e. Oracle, DB2 etc)

Enter the username and password.

Conn. Info : This field contains the technical information that is analyzed in native SQL in the CONNECT statement in order to open the database connection. e.g. In Oracle enter the name of the database you want to connect


Taking over Metadata via DB Connect

You use the context menu for a database Source System Select Database Tables to generate a DataSource for database source systems.

First, you choose a selection of tables for a database source system and create a connection to the database source system.

Next, you select the table fields for a specific table of the database Source System, and specify whether you want these table fields to be available for selection in the BW system.

Step 2 - Generate DataSource


Based on the generated DataSources for external DB views/tablesInfoSources in BW can be established as usual.Dataflow is now possible!

Step 3 - Assign the DataSource to the InfoSource


DataSource EnhancementDataSource Enhancement


LO Cockpit

FI-SL Extractors




based



Datasource Enhancement via Datasource Enhancement via Customer ExitCustomer Exit

• The different customers exits for enhancements of The different customers exits for enhancements of transaction data, master data attributes, texts and hierarchy transaction data, master data attributes, texts and hierarchy DataSources areDataSources are– EXIT_SAPLRSAP_001 : Transaction data DataSourceEXIT_SAPLRSAP_001 : Transaction data DataSource– EXIT_SAPLRSAP_002 : Master data DataSourceEXIT_SAPLRSAP_002 : Master data DataSource– EXIT_SAPLRSAP_003 : Text DataSourceEXIT_SAPLRSAP_003 : Text DataSource– EXIT_SAPLRSAP_004 : Hierarchy DataSourceEXIT_SAPLRSAP_004 : Hierarchy DataSource


Datasource enhancement Steps Datasource enhancement Steps involvedinvolved

1. Check whether required field is available in extract structure

3. If not available proceed with enhancement

4. Enhance extract structure with append structure

5.Unhide the custom field

6. Write appropriate customer exit for enhancement

2. If available include the same in extract structure


Step 1 - Check for availability in Step 1 - Check for availability in Extract StructureExtract Structure

• Applicable for DataSources where required field is available but Applicable for DataSources where required field is available but not included by defaultnot included by default

• Pre-requisite is empty delta queue and set-up tablesPre-requisite is empty delta queue and set-up tables


Step 2 - Enhance Extract Step 2 - Enhance Extract StructureStructure

• Find the name of the extract structure for DataSourceFind the name of the extract structure for DataSource

• Namespace for custom field is ‘YY or ZZ’Namespace for custom field is ‘YY or ZZ’

• Activate append structureActivate append structure

• For currency/quantity fields, the reference field should also For currency/quantity fields, the reference field should also be present in the extract structurebe present in the extract structure


Step 3 - Unhide the added fieldStep 3 - Unhide the added field

• Note that ‘Field Only’ is checkedNote that ‘Field Only’ is checked

• Incase field is required for selection, check ‘Selection’Incase field is required for selection, check ‘Selection’


Step 4 - Write the logicStep 4 - Write the logic• Go to CMODGo to CMOD

• Enter Project name and select componentEnter Project name and select component

• Select appropriate Customer ExitSelect appropriate Customer Exit


Write the LogicWrite the Logic• Write the ABAP code for populating the custom fieldWrite the ABAP code for populating the custom field

• Test Enhancement in RSA3Test Enhancement in RSA3

• Replicate DataSource in BWReplicate DataSource in BW


Data ExtractionData Extraction1 PrepareMe

2 TellMe

3 ShowMe

4 LetMe

5 HelpMe



• Create the InfoObjects document number, order no, name Create the InfoObjects document number, order no, name of the material, cost, currency, order creation date in the of the material, cost, currency, order creation date in the BW system.BW system.

• Create a flat file containing data for above InfoObjectsCreate a flat file containing data for above InfoObjects• Save the file in the CSV format.Save the file in the CSV format.• Now create a flat file source system and InfoSource and Now create a flat file source system and InfoSource and

assign the Source System to the InfoSource.assign the Source System to the InfoSource.• Create an ODS with document number as key field and Create an ODS with document number as key field and

include the above InfoObjects.include the above InfoObjects.• Create the update rules from the InfoSource you have Create the update rules from the InfoSource you have

created.created.• Create an InfoPackage and load the file.Create an InfoPackage and load the file.


BI Content ExtractionBI Content Extraction

• In the Source system go to transaction RSA5, activate the In the Source system go to transaction RSA5, activate the DataSource 2LIS_11_VAITM.DataSource 2LIS_11_VAITM.

• Go to BI Content select the grouping as data flow before Go to BI Content select the grouping as data flow before and install the InfoSource 2LIS_11_VAITM.and install the InfoSource 2LIS_11_VAITM.

• Now replicate the DataSource in BW and assign it to the Now replicate the DataSource in BW and assign it to the InfoSource and maintain transfer rulesInfoSource and maintain transfer rules

• Create an ODS and update rulesCreate an ODS and update rules

• Create the InfoPackage and load the ODSCreate the InfoPackage and load the ODS



• Go to transaction SBIW. Check out the various options to Go to transaction SBIW. Check out the various options to delete and fill the set up tables and simulate the batch jobsdelete and fill the set up tables and simulate the batch jobs

• Go to transaction LBWE, check the option to maintain the Go to transaction LBWE, check the option to maintain the datasources and to activate them as well as to simulate the datasources and to activate them as well as to simulate the V3 updateV3 update


TransactionsTransactions

RSA1RSA1 Administrator WorkbenchAdministrator Workbench

RRMX RRMX Business Explorer ( BEX )Business Explorer ( BEX )

RSO2RSO2 Maintain Generic DatasourceMaintain Generic Datasource

RSA3RSA3 Extractor Checker Extractor Checker

RSA7RSA7 Delta Queue StatusDelta Queue Status

RSA6RSA6 Maintain DatasourcesMaintain Datasources

RSQ02RSQ02 Maintain InfoSetsMaintain InfoSets

RSBOH1RSBOH1 Open Hub DestinationOpen Hub Destination


TransactionsTransactionsRSMORSMO MonitoringMonitoring

SBIWSBIW BW Config.BW Config.

OLI*BWOLI*BW Statistical SetupStatistical Setup

SM50SM50 Process OverviewProcess Overview

RSA5 Install Business Content

RSA6 Post Processing of DataSources

LBWE Logistics Cockpit

LBWG Deleting Setup tables

LBWF BW Log

SM37SM37 Job OverviewJob Overview

KEB0 Create the COPA Data Source


Course NameCourse Name1 PrepareMe

2 TellMe

3 ShowMe

4 LetMe

5 HelpMe


Tips & TricksTips & Tricks

• Before starting an SAP BW project, analyze the reporting requirements against the standard DataSources available.

• Zero-in on the standard DataSources satisfying the requirement.

• If data that cannot be supplied by the standard sources, we can:

- Create a generic DataSources. - Enhance the standard DataSource.


Some Useful sitesSome Useful sites

• www.sdn.sap.com

• www.help.sap.com

• www.service.sap.com