Upload
shona-miller
View
276
Download
4
Tags:
Embed Size (px)
Citation preview
UNICODE
• Character Sets and Coding Standards
• Han Unification and ISO10646
• Encoding Evolution and Unicode
• Programming Unicode
Character Sets
• Character Sets - a complete group of characters for one or more writing system.
• Coded character set – a mapping from a set of abstract characters to a set of integers.
• Character encoding scheme- a mapping from a coded character set to a set of octets.
Standards
• International standards – - RFC 2130, - ISO
• RFC standards come out of the Internet Engineer Task Force(IETF), it stands for Request For Comments.
• ISO is a worldwide federation of national standards bodies such as ANSI, JISC, KISI, CSA(GB} and ECMA.
Coded character sets (1)
• ASCII ( American Standard Code for Information Interchange) or ISO 646 defined by 7-bit
- Latin alphabet;
- Arabic numerals;
- Punctuation marks;
- Some computer control codes;
Coded character sets (2)
• ISO-8859 Standards- ISO-8859-1 to 15• ISO-8859-1(Latin1) - the most widely used
standard, it contains the characters necessary for writing in Western European and Scandinavian languages. It is also an ANSI Standard and known as the ANSI character set.
• The first 128 positions are identical to ASCII• All use 8-bit byte• Windows use ISO-8859-1 as its character set.
Coded character sets (3)
• Asian Language character setsTraditional Chinese(Taiwan)- BIG5: 8-16 bit, 94x157 matrix :13002 characters;- CNS 11643 X5012: 16 bit, 48000 characters.Simplified Chinese(China)- GB2312-80: 8-16 bit, 94x94 matrix: 6763
characters.- GBK (Extended GB2312-80): 8-16 bit, 94x94
matrix, 21003 characters.
Coded character sets (3)
Japanese(Japan)- JIS x0208: 16 bit, 94x94 matrix, 6879 characters.- JIS x0212: 16 bit, 5801 supplemental kanji.- JIS x0201: 7 bit, JIS-Roman plus half-width
katakana.Korean(Korea)- KSX-1001(KSC 5601): 8-16 bit, 94x94 matrix,
4888 hanja.- KSC-5636: 7 bit Korean version of ASCII.
Coded character sets (4)• EUC (Extended UNIX Code);• ISO 2022 Escape sequence• Windows Code pages: 8-16 bit,
- cp1252 English is ISO8859-1 or Latin1or Windows ANSI;- cp1200 Unicode code page;- cp932 Japanese is shift-JIS;- cp936 Simplified Chinese is close to GB2312-80;- cp949 Korean is in KSC 5601-1992 order;- cp950 Traditional Chinese is the same as Big 5.
Han Unification
• The Han standards consist of :JIS X0208(6349 Kanji), GB 2312-80 (6763 Hanzi), CNS 11543 (13951 Hanzi) and KSC 5601 (4888 Hanji). All characters in these standards must be included in 10646
• Without unification, more than 100,000 characters are separately encoded
• Characters from these standards should be unified in ISO 10646 that is identical characters from two or more of these standards may have the same code point in 10646. The unification of Han characters allows for approximately 40,000 characters.
Coded character sets (5)• Unicode: 16 bit, most languages are covered, developed by the
Unicode Consortium.• Unicode 1.0 Unicode 2.0 contains 38,885 characters. It will be
Unicode 3.0 in this year.• Unicode is similar to ISO 10646, the UCS(Universal Character Set)
encoding, - UCS-2: i.e.Unicode, 16 bit, a fixed with 2 byte format scheme; - UCS-4: also called ISO 10646, 32 bit, 4 byte format, 32,000 planes each with 65,000 characters capacities, for tall 2,080 million characters. The 1st plane is in use(that’s Unicode)- UTF-7: 7-40 bit, UCS Transformation Format, a Unicode character encoding scheme using a 7 bit.- UTF-8: 7-48 bit, UCS Transformation Format, a Unicode character encoding scheme using a 8 bit.- UTF-16(UCS-2E): 16-32 bit.
• Windows NT, use Unicode, has a single character set.
• UCS Transformation Format;• A variable-width or multi-byte encoding format;• In UTF-8, the standard ASCII characters occupy
only one byte, other Unicode characters occupy two or three bytes.
Start Character End Character Required Data Bits Binary Byte Sequence (x = data bits)
\u0000 \u007F 7 0xxxxxxx
\u0080 \u07FF 11 110xxxxx 10xxxxxx
\u0800 \uFFFF 16 1110xxxx 10xxxxxx 10xxxxxx
Table The UTF-8 Encoding
UTF-8
Unicode
ISO 10646 (UCS2)
16 bits
8 bits
EUC
PC CPs
Shift-JIS
ISO2022
ISO8859
EBCDIC
Apple CPs
7 bits ASCIIGB JIS KSC
Encoding Evolution
Programming Unicode
• How does Win32 support it
• How to write apps for Unicode
• How to be backward compatible
Win32 Server
Windows NT Base
Win32 App(Unicode)
Win32 AppNon-Unicode
ANSI to Unicode Conversion
Win32 Client
Client-Server Boundary
System fully Unicode internally
Unicode in Win32
Unicode in Win32
• Separate Unicode Datatype
- Wide character: Unicode
- 8-bit char:ANSI, DBCS
• Parallel Unicode and ANSI APIs
- Unicode and ANSI windows classes
- Implicit code conversion
• Resources always in Unicode
Programming Unicode
• Basic techniques
- How to migrate existing code base• Special topics
- Interaction with non-Unicode apps
- Untyped file system• Foundation for script/language specific
functionality• Migration example: Win32
Windows Unicode Programming
CodeConversion
DataExchange
Common Source
Win32 EXENon-Unicode
Win32 EXEUnicode
Generic data types
TCHAR LPTSTR
wchar_t wchar_t *char char *
Explicit data types
WCHAR CHAR LPWSTR LPSTR
wchar_t char wchar_t * char *
Generic data types in C
Macros and literals
• String literals
TEXT(“hello”);
“hello”; L”hello”;
• numeric equivalence
‘A’ = 0x41 0x0041 = L’A’
Dual function prototypes
SetWindowText (HWND, LPTSTR) ;
SetWindowTextA (HWND, LPSTR) ;
SetWindowTextW (HWND, LPWSTR) ;
#ifdef UNICODE
• Generic API prototypes
• Resolved to explicit prototypes
Basic conversion steps
• Use generic data types:
TCHAR, LPTSTR for Text
• Use explicit types for BYTE pointers (data buffers)
• Use TEXT() macro for literal constants
• Adjust pointer arithmetic
• Use generic function prototypes
Conversion metrics
• About 10% of source lines need to be modified by global replace
LPSTR LPTSTR
strstr() wcswcs()• About 2-5% need small modification
lstrlen(s) lstrlen(s)/sizeof(TCHAR)• Less than 1% need revised algorithm
OpenFile() SearchPath(); CreateFile()
Summary: Migrating Windows-based Program to Unicode
1. Modify your code to use generic data types2. Modify your code to use generic function prototypes3. Surround any character or string literal with the TEXT macro.4. Create generic version of your data structures5. Change your make process.6. Adjust pointer arithmetic.7. Check for any code that assumes a character is always 1 byte long.8. Add Unicode-specific code if necessary.9. Add code to support special Unicode characters.10. Determine how using Unicode will affect file I/O.11. Double check the way in which you retrieve command line
arguments.12. Debug your port by enabling your compiler’s type-checking