Cupertino, CA, USA / September 11, 2000First ICU DeveloperWorkshop : Build/Installation... 1 ICU Build / Installation Issues and Beyond Yves Arrouye Steven

Cupertino, CA, USA / September 11, 2000 First ICU DeveloperWorkshop : Build/Installation ...

1

ICU Build / Installation Issues and Beyond

Yves Arrouye<[email protected]>

Steven R. Loomis<[email protected]>

Cupertino, CA, USA / September 11, 2000 First ICU DeveloperWorkshop : Build/Installation ...2

Topics

• Configuring and building ICU• Integrating ICU into your product• Porting ICU to a new platform• Migrating an existing product to ICU• Adding new data to ICU• Packaging application data


Build Process – Directory Structure

• icu– data – mostly text source files for data

– source• common - core unicode functionality• config – build configuration files• data – data files used during the build• extra - ustdio (file handling, used by tools)• i18n – higher level resource-based localization• samples• test

– cintltst (C), intltest (C++)

• tools – for processing and packaging data files


File Name Conventions

• Library– Win32: icuuc.dll, icui18n.dll, icudata.dll

– Unix: libicu-uc.so, libicu-i18n.so, libicudata.so

• (Could be .a, .sl, .x depending on platform)

• Include file– Public API’s under “unicode/” directories– Example: search path icu/source/common #include “unicode/ucnv.h”


Dependencies

• Required– ANSI C & C++ compiler

• NO: templates, exceptions, RTTI, STL

– Working math library– stdio (or wrapper, especially for tools)

• Nice to have– Dynamic loading– Memory mapping– Threads– IEEE 754 numerics


Unix – “configure” script

• GNU Autoconf system– Detects platform capabilities– Generates Makefiles

• Install prefix: --prefix=/usr/local– /usr/local/lib/libicu-uc.so– /usr/local/share/icu (data)– /usr/local/include/unicode– /usr/local/sbin (tools)– /usr/local/bin (samples)


Unix – configure options

• Things to turn off or on (use --help)Threads, hpuxcma, Mapped, Tests, Extras, Samples

• Out of source build (--srcdir)– Share one read-only source base with multiple targets

• mkdir /build/hpux11-hppa-gcc/• cd /build/hpux11-hppa-gcc/• /icu/source/configure –srcdir=/icu/source

• Example call:sh ./configure --enable-threads=no --prefix=/opt


Unix – Building

gmake all all code, data, tests, tools

gmake check run all tests

gmake install optional: install to $(prefix)

(Use just ‘make’ if your make is GNU..)


Unix – configure variables

• Shell variablesCC=/usr/opt/bin/ccCXX=/usr/opt/bin/CC (C++)

– CC and CXX can need full paths

CPPFLAGS= -Dmydefine=3 -I/path(both C++ and C compiler)

LDFLAGS=-L/usr/local/lib –lotherlibFORCE_LIBS=-lm –lpthread –lstdc++ PATH=…• sh CC=/usr/opt/bin/cc ./configure …


Unix – runConfigureICU

• For a specific set of platforms– Pre-selects compiler and options for reference platforms and then

some.– Use instead of ‘configure’ sh runConfigureICU AIX4.3xlC …

• Supported platforms:AIX4.1.5xlC SOLARISCC SOLARISGCC LinuxRedHat5.2

LinuxRedHat6.x HP/UX10.2CC HP/UX10.2ACC HP/UX11CC HP/UX11ACC AIX4.3xlC SOL2.7CC PTX SOLARISX86 AIX4.3VA


Win32

• Use ‘allinone’ project• ICU_DATA – sets target location for icudata.dll

(use a trailing backslash \)– (Default: same as other DLL locations)

• Target: Debug vs. Release• Makedata project builds and updates ICU data• Hit F7!


Running the Tests

• Set ICU_DATA (trailing slash or backslash) and set TZ=PST8PDT .. (or “gmake check”)

• Intltest – shows errors after each level– Intltest LIST list possible tests– Intltest format– Intltest rbbi/LIST– Intltest –v format

• Cintltest– Cintltest –l - list possible tests– Cintltest /tsvconv– Cintltest –v /tsformat


Integrating ICU – Compiling

• UNIX: can include …/share/icu/Makefile.inc or, -I$(prefix)/include

• Win32: add icu\include\ to search path


Integrating ICU – Linking

• May not need to link in “i18n”

• Unix: use $(ICULIBS) or –licu-uc –licu-i18n

• Win32: icu\lib\Debug\icuuc.lib, icu\lib\Debug\icui18n.lib (or Release)


Integrating ICU – Data

• u_setDataDirectory() or ICU_DATA– Set these before any ICU calls!

• Modes:– DLL - copy libicudata.so or icudata.dll with other libs– Common – set a path to dir containing icudata.dat– Linked – Link app with data dll, call u_setCommonData()

before any ICU calls are made. (see udata.h )


Port ICU to new platform

• Keep us in the loop!– (You will be glad, the next time you don’t have to back-port)

• Bring up incrementally, prioritize (if needed)• If you fix a general bug, don’t #ifdef it• Test for features, not platforms (except putil.c)

– #if U_HAVE_XXX instead of #if MYPLATFORM

• Read and respect the ICU coding guidelines


Port – Sane order

• Common– umachine.h platform include.h

• Not needed under autoconf• Base types (int32_t,..)

– cmemory.h, cstring.h, putil.c, umutex.c• Platform utilities: strcpy() through default time zone.• putil.c has POSIX routines for many of these tasks

– udata.c – dynamic loading, memory map..

• I18n• samples/date – test linking an executable!• Toolutils


Port – Sane order, cont’d

• Tools– Can skip pkgdata/genccode/gencmn if need be– Makeconv & gencnval – converter files– Ustdio – UFILE* library, used in..– Genrb – Resource Bundle compiler– Ctestfw – used in cintltest

• Test– Intltest – C++ [tsmthred.cpp: requires threads]– Cintltest – C


Port – Sane order, #3

• Other tools (gentz, gencnval, ..)• Data tools

– Determine data loading options (dll, memory map, ..)– Go back and update udata.c if needed– Pkgdata/genccode/gencmn

• Gotchas:– Sometimes C++ linkage is needed for icudata.– Exporting symbols (see U_EXPORT macros)– Tests sensitive to limits of floating point capability


Port – keeping autoconf happy

• autoconf 2.13 depends on GNU m4– Only edit “.in” files [if possible!]

• Add compiler choice in runConfigureICU• Find a ‘close’ mh- file in source/config, copy to mh-yourplatform• Platform detection:

– Aclocal.m4, config.guess, config.sub

• Custom tweaks in configure.in– Search for pthreads, headers, math lib– rpath separator

• Test other platforms!


Migrating to ICU - Context

• What does the “international contexts” of your application consist of?– Locale

• System/Default Locale – for OS interface• Locale of an end user• Locale of external services

– Codepage• System/Default• File format or communications protocol• Internal processing

– Timezone


Migrating – Binding

• To what are these contexts bound?– User (may be the same as system settings)– Connection, thread, transaction (server)

• What type of international data does my application need?– Localized messages– Collation sequences– New converters

• See ICU Design Principles & Services Guide– Converters, Formatters,etc. are per-thread


Migrating – Data Path

• Either widen the data path…– Unicode everywhere– Direct manipulation easier

• …Or build wrappers– Knowledge of Unicode only at certain levels– Pass data as opaque ‘blobs’ of text elsewhere


Migrating – Input/Output

• Application data input and output– Conversion APIs to obtain data in Unicode, output in

legacy codepages– ICU locale suport to manipulate dates, times,

currencies– ICU collation support for all comparison, break

iterators instead of scanning for whitespace to separate words and sentences.


Migrating - Messages

• Use Resource Bundles to externalize locale-depentant data such as messages.

• Use MessageFormat, ustdio

• Translate the messages.


Adding Data to ICU

• Locale language _ country _ variant– Determine where data belongs in hierarchy

• Language– Don’t set specific currency (format with ¤ OK)– Base translations: country/language names, days of weeks, months

• Country– Currency codes– Formatting variations

• Variant (e.g. Euro)

– Add to: data/index.txt, tools/genrb/genrbfiles.mkReslocal.mk for temporary additions

– Use Locale Explorer for examples

http://oss.software.ibm.com/icu/demo


Add – Converter Alias

• Alias – convrtrs.txtibm-367 us-ascii {MIME} ANSI_X3.4-1968 {IANA} ascii #

Table file: ibm-367.ucm– Standard name follows alias (above: us-ascii is the MIME name)– Algorithmic types listed in ucnv_bld.c

– Locale argument: ISO_2022,locale=jp

• Rebuild alias table with gencnval– (“gmake all” in tools, or rebuild “makedata”project)

http://oss.software.ibm.com/cvs/icu/~checkout~/icu/data/convrtrs.txt

http://oss.software.ibm.com/cvs/icu/~checkout~/icu/data/convrtrs.txt

http://oss.software.ibm.com/cvs/icu/~checkout~/icu/source/common/ucnv_bld.c





Add – Converter Table

• Format: Modified IBM CDRA• Choose a type

– SBCS – single byte<U0127> \xB1 # comment |0<UFF01> \x20 # fallback |1

– DBCS – double byte<U2537> \x43\xEC # |0

– MBCS / EBCDIC_STATEFUL• Mixed single and double byte. See documentation

• The LAST entry overrides if there is a conflict


Add – Converter Table to Build

• Add .ucm file to tools/makeconv/ucmfiles.mk ucmlocal.mk for temporary additions


Packaging Data

What is a package?• 1.6: 160+ converters, 170+ bundles = ~7.6Mb• udata: unified API to create, locate, and manage data• Raw form of data (usable in this way)

– Single directory (no subdirs)– Individual files (.res, .dat, .cnv, ..)– Each file has versioning, endianness, etc..– Created with unewdata.h (toolutils library)


Pkgdata – Modes

• Common (icudata.dat)– Single file, with internal table of contents– Memory mapped– Portable among platforms which share endianness and base charset (ASCII vs

EBCDIC)

• Shared Library (icudata.dll)– Data converted into static byte arrays– One symbol per library: table of contents– Can be used with u_setCommonData() for compile time linkage– DLL search path finds your data!


Pkgdata – List File

• Manifest for package creation– Root.res

ja_jp.resc:\otherfiles\otherdata.dat

• Relative to –s source directory


Pkgdata – Options

• -m mode (DLL or Common)• -O buildoptions (Unix: path to Makefile.inc, Win32: path to ICU dir)• -p shortname (e.g. ‘icudata’ or ‘myappdata’)• -I installdir

• Examples:pkgdata –m dll –p xyz –O /opt/icu/Makefile.inc xyz.lstpkgdata –m dll –p xyz –O DD:E:\Work\icu xyz.lst

(Win32 Debug)


Q&A

Documents

Cupertino, CA, USA / September 11, 2000First ICU DeveloperWorkshop : Build/Installation... 1 ICU Build / Installation Issues and Beyond Yves Arrouye Steven