Localization and Language Technology Standards Kavi Narayana Murthy University of Hyderabad ELITEX - 2007 New Delhi, 10-11 January 2007

Localization and Language Technology Standards

Kavi Narayana MurthyUniversity of Hyderabad

ELITEX - 2007New Delhi, 10-11 January 2007

Kavi Narayana Murthy UoH

2

Outline Character Encoding Standards Fonts, Glyphs, Mapping Standards OS/Browser Support, Drivers Transliteration, Romanization Translation, Linguistic Resources Speech and OCR Technologies Enforcement


3

Goals Functionality

Whatever we can do with English, we must be able to do with our own languages and scripts with equal ease

Inter-operability, Platform Independence All Applications must work seemlessly on all

hardware and software platforms Language and Script Independence

Multi-lingual, Multi-Script Support


4

Standards Even a poor standard is better than no

standard Standards save us a lot in the long run Commercial forces promoting non-

standard, proprietary, secret systems must not be allowed to succeed Let us not say “Let the Market Decide”!!!


5

Character Encoding Standards ISCII and Unicode ISCII is a BIS Standard, Unicode is

not Unicode is based on ISCII In some sense, Unicode is a step in

the backward direction Let us understand ISCII first


6

Language and Script Do not confuse one for the other Many-to-Many Script is neither language nor font Script and SuperScript Phonetic Basis

Common SuperScript for all ILs Script Grammar


7

Language and Script Sanskrit is written in Devanagari,

Telugu, Kannada, Bangla etc. scripts

Devanagari is used for writing Sanskrit, Hindi, Marathi, etc.

English words are often written (transliterated) in local language scripts


8

Phonetic Basis Words: Meanings, Sounds, Written

Symbols Meanings are supreme but difficult

to quantify and encode Sounds are the next best

A ‘ka’ sound is a ‘ka’ sound, whatever be the language – Hence ‘Universal’

No need for ‘Spellings’ What is write is what we speak - directly


9

Orthography Written symbols correspond with

phonemes – basic sound units Minor variations in sounds

(allophones, co-articulation effects etc.) are not depicted in orthography t: Mountain, tea, truck, spilt, little

Special Symbols not to confused with basic Characters


10

What is a Character? Indian Languages:

No ‘alphabet’, not letters, no spellings Phoneme-based Units are syllable-like: called ‘akshara’-

s akshara-s very large in number

Corpus studies not sufficient Made up of vowels, consonants etc. Not all sequences valid


11

Script Grammar A Grammar for Scripts Allows all valid sequences, only valid

sequences No need to code all possible akshara-s Script grammar must be part of

standards: ISCII includes. UNICODE? Script Grammar to be enforced by s/w


12

SuperScript ILs: 10 Scripts with a nearly common

sound system – all derived from the ancient ‘braahmi’ script

=> SuperScript Super Set of all Phonemes

Common encoding: ISCII Extendable to all languages of the

world


13

ISCII: (BIS – 1991: IS 13194) 128 codes more than sufficient Uses second half of ASCII, first half

untouched – allows mixing with English

SuperScript: Transliteration built-in Long Standing: ISCII 1988, 1991 Well thought and well designed


14

Why did ISCII fail to catch on? Silent on Character-to-Font mapping

A complex many-to-many mapping Fonts not standardized, fonts not available

Not registered, no OS/Browser Support (BIS – 1991: IS 13194) Rationale not explained Not publicized, not enforced


15

History Proprietary, non-standard, secret

font based encoding schemes Promoted by commercial companies Near Zero Inter-operability Ad-hoc ISCII-to-font mapping schemes Mapping schemes not made public To be made Illegal and Punishable

Put India back by at least a decade!


16

Improving ISCII Register - To get OS/Browser Support

Remove encoding of allophones, allographs Script Grammar: FSM enough, CFG - not needed

Include Rationale, explanatory notes Remove Attribute/Extension codes Standardize ISCII-to-Font Mapping Scheme Promote, Enforce


17

Character-to-Font Mapping Complex scripts – not linear Glyphs: shape units convenient for

rendering Poor correspondence with sound

units Many-to-Many mappings

Glyph selection, scaling, positioning No Glyph Encoding Standard


18

From Character to Font Must be provably complete and

100% consistent Current systems are all ad-hoc –

neither complete nor consistent Finite State Transducers:

Necessary and Sufficient Without restricting Creativity and

Flexibility Simple, Efficient, Re-Usable


19

Encoding Standards: Unicode For Language/Script/SuperScript?

CJK. Why not for ILs? Script Grammar? Character-to-Font:

relegated to font level font effects

ISCII-88 Based, Has Errors Once added, cannot be deleted!


20

ISCII or Unicode? Unicode:

To be with the World, to know and be known ‘Correcting’ Mistakes, Improving Standards Support (OS, Fonts, etc.), Education, Training Converting Legacy Data – A Huge Task

ISCII-to-Unicode is not trivial Ignore BIS Standard and embrace what is not

yet ‘standardized’? Why not co-exist? – Internal and External

Views


21

Keyboard Layouts, Drivers Several de-facto standards and

many variations in use To select a few and standardize

So called Roman Phonetic Typing ILs through English! OK for oldies, not for future!

INSCRIPT: ISCII Standard, Good for new comers

To strictly enforce Script Grammar


22

Document Encoding Standards Plain Text: pure ISCII/UNICODE

Mono-lingual Plain Text? Annotated Text (Ex. Word

Processors) XML Style, Open, Readable formats to

be encouraged Proprietary, secret, non-standard

encodings must be discouraged


23

Transliteration Widely used, part of our Tradition

Sanskrit texts in local scripts English, Hindi, Urdu words in local

scripts Music Compositions

Automatic in ISCII. Unicode? Quality of transliteration

To and From English?


24

Romanization Need:

Where there is no support for local languages English dailies, posters, advertisements etc. Lack of support: OS/Browser/Fonts etc.

Where users prefer Roman A variety of ad-hoc schemes in use

iTRANS, RTS, W-X, etc. Standards badly wanted


25

Romanization Multi-dimensional optimization problem

Case Mix-up 26 Letters not sufficient 52 nearly sufficient Not always supported

Storage space, Ease of Typing, Aesthetics Scientific/Logical Design/Naturalness

English-like – for the oldies: a, ee, oo, a, oa ??? Futuristic: aa/ii/uu/ee/oo


26

Romanization Clashes: a+u/au, k+h/kh, s’

Two way conversion, cyclic check Ex. Long Vowels:

a: -clashes with colon diacritic –not supported ipa –not understood –not supported A +single char. +saves space –ugly –

difficult to type –case-mix-up aa +logical (like ee) +easy to type


27

Romanization: An Example a aa i ii u uu R RR e ee ai o oo au M H k kh g gh n~ c ch j jh n` T TH D DH N t th d dh n p ph b bh m y r l v s’ S s h L


28

Translation Create Material Afresh Translate by Hand Automatic/Machine Translation Machine Aided Translation English – Local Language

Translation Local – Local Language Translation


29

Translation Resource Intensive

Manpower, Time, Cost Quality/Uniformity

Standards, Bench-Mark Data, Testing and Evaluation Procedures

Dictionaries, Terminology Databases Pan-Indian Terms/Sanskritize/Localize


30

Linguistic Resources Dictionaries – General, Domain Specific Terminological Databases Thesauri, WordNets, Ontologies Morphological Analyzers, Generators Spell/Grammar/Style Checkers Annotated Text and Speech Corpora


31

India: Future is in Speech One Billion People, A Sixth of the World More than 150 Languages, 22 Recognized 95 % not comfortable with English Computers, Current, Connectivity Info Revolution benefits: Majority

Deprived 10 M Computers, 100 M Phones Future is in Speech


32

Speech Natural Easy, Fast Hands-Free No need to Learn

Technology Language

Available to all


33

Text and Speech Speech is Natural Reading/Writing is learnt, Artificial Some never learn – Illiterates Oral Tradition Speech is more permanent than Text! “I did not steal that ring of gold” Trust Yourself!


34

Speech Technologies Speech Recognition: Speech to Text Speech Synthesis: Text to Speech Speaker Recognition,Verification,ID Speech Coding/Decoding,

Compression Slow down, Speed up Speech as Evidence


35

Applications Telephone Dialing Form Filling Dictation Machine Command and Control Voice enabled Web OCR+WP+TTS MT: Cross-Lingual IR, S2S


36

OCR OCR in Local Scripts Needed

To digitize and save legacy data To compile/process/edit/refine data

For Printed Texts/Manuscripts Old Data

deterioration of paper old type fonts, problems of type-

setting


37

Multi-Modal Interfaces

To Reach out to 1 Billion People, we must get the best of many worlds: Speech Recognition and Synthesis Graphics and iconic Interfaces OCR Technologies Translation, CLIR Camera, Gestures, Touch Screen


38

Balance Between Backward Compatibility

and Future-Proof Designs Quick Fix Solutions and Long Haul One Standard or Several? Economics and Business Sense

versus Social Responsibilities Acceptance versus Enforcement


39

The 3 Most Important Things1. Develop/Refine/Update Standards

Detailed Documentation Including rationale, issues, evaluation,

etc.

2. Education and Training3. Enforcement

Make use of non-standard methods illegal and punishable under law

Technical Workshops for detailing

Thank You!

Visitwww.LanguageTechnologies.a

c.in

Documents

Localization and Language Technology Standards Kavi Narayana Murthy University of Hyderabad ELITEX - 2007 New Delhi, 10-11 January 2007