SAP NetWeaver Business Intelligence Unicode-compliance

SAP NetWeaver Business Intelligence

Unicode-compliance

Product Management NetWeaver BI, SAP AG

Version 6.0 – April 4 , 2006

SAP AG 2004, Unicode@BW, Product Management NetWeaver BI, 2

Unicode in GeneralUnicode in General

Excursus: MDMPExcursus: MDMP

Unicode support of SAP NetWeaver BIUnicode support of SAP NetWeaver BI

Interfaces to SAP SystemsInterfaces to SAP Systems

Interfaces to non-SAP SystemsInterfaces to non-SAP Systems

Content

Contents


Unicode Essentials

What is Unicode? Character encoding schema for (nearly) all characters used world

wide

Each character has a unique number („Unicode code point“) Notation U+nnnn (where nnnn are hexadecimal digits)

See http://www.unicode.org for complete code charts


E3 91 B9 79 3434 79U+3479

CE B1B1 0303 B1U+03B1

C3 A4E4 0000 E4U+00E4ä

6161 0000 61U+0061a

UTF-8UTF-16little endian

UTF-16big endian

Unicodecode point

Character

Representation of Unicode Characters

UTF-16 – universal transformation format, 16 bit encoding Fixed length, 1 character = 16bit. Surrogates using 2*16 bit Platform dependent byte order 16 bit alignment restriction (needs even addresses)

UTF-8 – Unicode transformation format, 8 bit encoding Variable length, 1 character = 1...4 bytes Platform independent No alignment restriction


Character expansion model Separate Unicode and non-Unicode versions of SAP R/3

No explicit Unicode data type in ABAP Single ABAP source for Unicode and non-Unicode systems Automatic conversion of character data for communication between Unicode

and non-Unicode systems

1 character = 2 bytes (UTF16), (types C, N, D, T, STRING) Unicode kernel Unicode database

1 character = 1 byte (types C, N, D, T, STRING) Non-Unicode kernel Non-Unicode database

ABAPsource

Non-Unicode

R/3

UnicodeR/3

ABAPsourceABAPsource

Transparent Unicode Enabling of SAP R/3


Implications

You are not interested in Unicode?You are not interested in Unicode?

No customer is forced to convert to Unicode Possibly minor changes for not Unicode enabled

programs (more restrictive syntax check as of WebAS 6.10 on)

You are interested in Unicode?You are interested in Unicode?

Major part of ABAP coding is ready for Unicode without any changes

Minor part of ABAP coding, i.e. some customer-specific ABAP coding, has to be adapted to comply with Unicode restrictions

SAP will deliver powerful tools to convert your existing system


Resource Requirements for Unicode-compliant BI What are the additional resource requirements after a Unicode What are the additional resource requirements after a Unicode conversion?conversion?

Text fields are usually bigger in a Unicode environment than in a Non-Unicode environment. Enhanced functionality requires additional resources, which strongly depend on the given customer scenario. In the following, we give a rough estimation what a Unicode conversion could mean for the resources.

CPU’s We expect additional SAP NetWeaver BI requirements of roughly 30% more CPU

power - like the SAP R/3 requirements.

Main Memory We expect additional SAP NetWeaver BI requirements of roughly 50% more memory

- like the SAP R/3 requirements.

Disk Storage Disk Storage depends strongly on the underlying DBMS and the given data

model/volume. For a significant share of InfoCube data (only numeric keys!), there might not be a significant increase of the DB size.

For SAP R/3 on ORACLE, tests have resulted in roughly 35% additional disk space. For SAP NetWeaver BI, we expect – depending on the scenario – less additional disk space.

Note that after a conversion, the disk size may even decrease because of DB reorganisation.


The Unicode Upgrade Project

PreparationPreparationPreparationPreparation

Set up the Unicode Set up the Unicode upgrade projectupgrade project

Analyze data to Analyze data to minimize downtimeminimize downtime

ConversionConversionConversionConversion

To be done during To be done during system downtimesystem downtime

Unload /reload process Unload /reload process for small databasesfor small databases

Minimum downtime tool Minimum downtime tool for large databases for large databases

(Incremental Migration (Incremental Migration IMIG)IMIG)

Post-ConversionPost-ConversionPost-ConversionPost-Conversion

Set up the system Set up the system for Unicodefor Unicode

Start up the Unicode Start up the Unicode systemsystem







Content

Contents


MDMP – Definition

What does MDMP mean?What does MDMP mean? In the past, SAP supported multiple languages by MDMP

MDMP = Multiple Display – Multiple Processing Mixed code page technology depending on logon language

Several 100 satisfied customers

MDMP was an SAP workaround for accessing multiple code pages Defined users where accessing the solution using SAPgui frontend Only little integration into the world wide web

MDMP should not be used anymore for new installations MDMP is not supported for SAP NetWeaver BI


MDMP – Connection

How can I connect SAP NetWeaver BI to an How can I connect SAP NetWeaver BI to an MDMP SAP source system?MDMP SAP source system?

Language-dependent source tables Most text tables include the language key. In an SAP system, the

code page can be derived from the language key. This enables the correct conversion of the texts from the source system to SAP NetWeaver BI.

Language-independent source table If a text table does not include the language key (e.g. customer

texts), these texts have to be converted “manually”. The code page (or language) has to be defined for EACH record in these language-independent tables either by appending a new column into the table or by defining the extraction job in the corresponding codepage.

If there is no language information given, the source tables will be converted based on the logon language of the user.


MDMP – Project Based Solution

Why is there no generic solution for the Why is there no generic solution for the connection of MDMP source systems?connection of MDMP source systems?

All language-independent source tables must be checked and for each record, the code page which it was created in must be determined

In order to derive this code page information, attribute values, organizational assignments or heuristics may help, but there is no generic approach

Examples: in order to determine the code page of a customer name, the country

where she/he lives in may be used usually not correct, if e.g. a Japanese customer lives in Russia

Alternatively, the assignment to a sales organization may help

For more information see How-To Guide in SAP Service Marketplace


Design Principles for Source Systems

How can I ensure a smooth Unicode conversion?How can I ensure a smooth Unicode conversion? The BI system receives the following information from source systems

Date and timestamp information Key figures Characteristics by key Texts

How should I design the data in the source system? Date and timestamp information as well as key figures are just figures; the

conversion is not critical Characteristic keys (e.g. material number, customer number, brand key, country

key) must be designed as 7-bit-US-ASCII characters (common characters). Don’t use other characters as keys or attributes.

As a benefit, the permitted character setting in the BW customizing need not be maintained. See OSS note 173241 for more information.

Note that maintaining the permitted characters makes you code page-dependent. There might be problems with the conversion to Unicode (e.g. characters like ‘µ’ are a lower case letter in Unicode – and thus not allowed)

Texts are ONLY loaded into the text tables of the InfoObjects; transaction data and master data attributes must NOT contain texts. Texts are usually language-dependent and must be designed in the source system with an associated language key (even if e.g. the R/3 datamodel is not providing this).







Content

Contents


Unicode-Compliant SAP NetWeaver BI

Unicode-compliance means:Unicode-compliance means:

SAP NetWeaver BI can interpret and display Unicode characters User interface will be shown in local language Business data can use all languages in parallel

SAP NetWeaver BI can extract data from source systems with specific code pages (Non-Unicode or Unicode)

SAP NetWeaver BI can extract data from SAP source system running mixed code pages (MDMP)

Interfaces to 3rd party systems support correct code page conversion


Unicode Release Planning

Which versions of SAP BW are released for Unicode and Which versions of SAP BW are released for Unicode and when are they available?when are they available?

SAP NetWeaver BI Unicode releases SAP BW 3.5 as part of SAP NetWeaver ’04 and all subsequent releases [SAP BW 3.1 Content has been piloted, but it is not shipped anymore]

Are there any platforms that do not support Unicode?Are there any platforms that do not support Unicode? Yes, the following DB/OS platforms do not support Unicode

Informix Reliant Unix


Supported Scenarios – SAP NetWeaver 2004s

Which scenarios does SAP NetWeaver 2004s support?Which scenarios does SAP NetWeaver 2004s support?

BI in SAP NetWeaver 2004s is fully Unicode-compliant Load of data in different code pages Display of all data if front end is configured in Unicode code page All MS Windows-based client tools are Unicode-enabled


Supported Scenarios in SAP NetWeaver ‘04

Which scenarios does SAP NetWeaver ’04/SAP BW 3.5 support?Which scenarios does SAP NetWeaver ’04/SAP BW 3.5 support? BI Server (backend)

Load of data in different code pages Display of all data in SAP Gui environment

Unicode Web front end Display of all data if front end is configured in Unicode code page

Restrictions for MS Windows-based client tools (BEx Excel front end, Web Application Designer, Query Designer) in SAP NetWeaver ’04 – see also OSS note 588480

Object keys must be in US-7-bit-ASCII to ensure access to keys in all logon languages / code pages

The restrictions only apply to texts, transactional data is not affected. The client is configured in one Non-Unicode code page. Note that also the Windows default

code page must be switched to this Non-Unicode code page. Only the data in the logon code page are displayed correctly: I.e. a Russian employee can see

all Russian texts, but not the German ones (‘#’ characters will be displayed for special German characters), whereas the German employee can see the German texts (but not the Russian ones)

English texts (US-7-bit-ASCII) are displayed correctly A user must not change and save queries, work books, query elements or web templates with

Non-US-7-bit-ASCII (English) texts that have been originally created in a different code page. For example, a query containing Russian text elements should only be changed in Russian code page.


Front end Restrictions in SAP BW 3.5 in Detail – 1 – Restrictions for MS Windows-based client tools – example Restrictions for MS Windows-based client tools – example BEx Excel FrontendBEx Excel Frontend

Logon Language Russian German characters corrupted

Logon Language German Russian characters corrupted

No restrictions in Web Frontend Correct characters from both code pages – independent from logon

language


Front end Restrictions in SAP BW 3.5 in Detail – 2 –

BEx Analyzer Excel front end Note that the corrupted text is a pure display matter; the data in the

database is always correct!

Query Designer / Web Application Designer In order to avoid corrupted text written back to the database,

organizational means must be installed. There are basically two options Queries, query elements, work books, web templates must only contain

English texts (US-7-bit-ASCII) Queries, query elements, work books, web templates containing texts (also

hidden ones in e.g. restricted key figures or variables) in a specific language / code page must not be changed by users in a different code page

RTL (Right-to-Left) Languages The Windows frontend components (BEx Analyzer, Query Designer, Web

Application Designer) do not support RTL languages (like Hebrew, Arabian). However BEx Web does support them.

Keys in US-7-bit-ASCII Keys (of any InfoObject) must be in US-7-bit-ASCII to enable users in

every code page to enter the key in a search criterion; if Russian characters were allowed in keys, a German user could never enter these and, hence, never select these


Supplementation

Business Scenario You run your SAP NetWeaver BI in different languages, but the availability of

translations for all languages cannot be guaranteed. Hence, you want to define a standard language. This standard language is used to generate all missing translations.

Example: you want to use English material texts in case the Japanese ones are not available.

Supplementation The text tables can be supplemented with a standard language.

Usage Start in transaction SMLT and check by double-clicking on the (target) language

(e.g. Japanese), if the supplementation language has been maintained (e.g. English).

Then use the menu path Language Special Actions Supplementation (Expert) in order to select all affected tables (usually all language-dependent /BIC/T* and /BI0/T*-tables – those tables are cross-client class A tables)

If you want to run supplementation periodically, you can use report RSTLAN_SUPPLEMENT_PERIODIC (parameter: preceding supplementation run) and include it into your process chain.

Check OSS note 111750 for supplementation of German language


BW Unicode Installation and Conversion

How can I make my BI Unicode-compliant?How can I make my BI Unicode-compliant?

Delivery You can choose between Non-Unicode and Unicode installation

Note: Unicode installation requires more hardware resources (depending on DB platform)

Installation modes New Install Conversion of an existing SAP NetWeaver BI

Before the conversion, upgrade your SAP BW to at least SAP BW 3.5 / SAP NetWeaver ‘04

R3LOAD converts an existing SAP NetWeaver BI (’04 or 2004s) automatically by exporting the DB, realigning the DB and importing the DB again

Note that the Unicode Conversion is a pilot project as part of a BI System Copy (see note 543715)

Customer-developed programs (variable exits, virtual characteristics/key figures, transformation rules, table interface etc.) must be in line with the Unicode rules

The duration of a conversion depends on the size of the existing database







Content

Contents


Interfaces to SAP Source Systems

Which SAP Source System Types exist ?Which SAP Source System Types exist ?

Single code page systems One single code page can include several languages (e.g., Latin-1 contains

German, French, Spanish, Italian etc.) A Unicode source system is also a single code page system

Mixed code page systems SAP source systems with several code pages (usually: MDMP)

How does the conversion work?How does the conversion work? RFC-Calls are enhanced by source and target code page parameters; they

transform characters „on the fly“ to the correct code page The RFC call derives the source code page information from the language

key Note: English texts (US-7-bit-ASCII) is converted correctly in ALL code

pages For mixed code page systems, language information on RECORD level

defines the conversion PROJECT based solution


Connection Scenarios – 1 –

Which systems can I connect to ?Which systems can I connect to ?

Overview on the supported connections

SAP BW 3.5 Unicode and following releases

SAP BW 3.x Non-Unicode

SAP BW 2.x Non-Unicode

SAP source Unicode

ok ok, but some information might be lost not recommended

Project! Some information might be lost not recommended

SAP source non-Unicode

ok ok ok

SAP source MDMP

Project! ok, but some information might be lost not recommended

ok, but some information might be lost not recommended



What are the prerequisites for a proper connection ?What are the prerequisites for a proper connection ?

Unicode SAP BW 3.5 systems and Unicode / Non-Unicode SAP source systems

SAP NetWeaver BI-Service API 3.0B SP3 resp. PI_BASIS 2002_1_620 SP3 is required to connect the Unicode source system with SAP NetWeaver BI

e.g. R/3 BI

Unicode / Non-Unicode

Service API 3.0B SP3 or PI_BASIS 2002_1_620 SP3

Unicode BI



Non-Unicode SAP BW 3.x systems and Unicode SAP source systems During conversion some information might be lost. The following

restrictions apply Characters might be corrupted as the target system does not know all Unicode

characters In multi-byte code pages (Asian languages), suffix characters could get cut off

during the extraction, as the bigger Unicode containers must be mapped to the target non-Unicode containers

Non-Unicode SAP BW 2.x systems and Unicode SAP source systems SAP BW 2.x as non-Unicode BW system is not generally released to

extract data from Unicode source systems; in special scenarios, this connection might be possible on a project basis

e.g. R/3 BI

Unicode Non-Unicode SAP BW 3.x

e.g. R/3 BI

Unicode Non-Unicode SAP BW 2.xProject Solution

Restrictions apply



What about other interfaces to SAP source systems ?What about other interfaces to SAP source systems ?

The same connection scenarios apply for transient interfaces to SAP source systems

Report-Report-Interface RemoteCubes Virtual InfoProviders to SAP source systems


Technical Settings for Unicode

Be sure to set up your system according to the following Be sure to set up your system according to the following recommendationsrecommendations

Languages You can only extract languages to SAP NetWeaver BI, which are defined in

the basis parameter ZCSA/INSTALLED_LANGUAGES (language vector)







Content

Contents


Unicode-compliance of Interfaces

Which source system code pages are supported for Non-Which source system code pages are supported for Non-SAP sources?SAP sources?

For Non-SAP sources, it is difficult to determine the source code page. In most cases, only English texts (7-bit-US-ASCII) or Unicode texts are

converted correctly Examples

Flat File DB Connect (Same restrictions apply like for all MultiConnect connections –

independent from Unicode) UD Connect Virtual InfoProviders to Non-SAP sources


Unicode-compliance of BAPIs and 3rd Party Tools

What about Unicode with respect to BI-BAPI’s and 3What about Unicode with respect to BI-BAPI’s and 3rdrd Party Party Tools?Tools?

Loading data into BW via Staging-BAPI (and 3rd party ETL-tools) This interface is released for the following sources: (see OSS note 765543)

Single-byte code pages Unicode UTF-16 All other code pages, but only for transfer structure of size 250 characters and

smaller

This interface is not released for the following sources: Multi-byte code pages with variable byte length (I.e. Asian code pages and UTF-

8) AND Transfer structure is larger than 250 characters

Displaying data in a 3rd party front end via OLAP BAPI, OLE DB for OLAP or XMLA

These interfaces are currently released as a pilots for Unicode

ETL or frontend partners Ask our partners for Unicode-compliance of their products







Content

Contents

Documents

SAP NetWeaver Business Intelligence Unicode-compliance