1

Wen-Hsiang Lu (盧文祥 )Department of Computer Science and Information Engineering

National Cheng Kung University2014/02/17

Multilingual and Crosslingual Information System

2

Contact Information

• Room: 4261, Monday 09:10 - 12:00 AM

• Instructor: Prof. Wen-Hsiang Lu (盧文祥 )– Office: 4216

– Office hours: Monday 12:10 - 2:10PM

– Phone: 62545

– Web page: http://myweb.ncku.edu.tw/~whlu/mis.htm

– Email: whlu@mail.ncku.edu.tw

– Teaching assistant: 王廷軒• Email: playif@gmail.com

3

Course Grading

• Class participation/presentation: 30% • Tests: 25% • Project: 25% • Homeworks: 20%

4

Source Textbooks

• Christopher D. Manning and Hinrich Schutze, Foundations of Statistical Natural Language Processing, The MIT Press, 1999. ( 全華科技圖書 : 02-23717725)

• Daniel Jurafsky and James H. Martin, Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, Prentice Hall, 2000.

• James Allen, Natural Language Understanding, Benjamin/Cummings Publishing Co, 1995.

• Gregory Grefenstette, Cross-Language Information Retrieval, Kluwer, 1998.

• Jean Veronis, Parallel Text Processing: Alignment and Use of Translation Corpora, Kluwer, 2000.

5

Other Useful Sources (1)

• Reference Books– Charniak, E. Statistical Language Learning. – Cover, T. M., Thomas, J. A. Elements of Information Theory.– Jelinek, F. Statistical Methods for Speech Recognition.

• Major Conferences:– ACL (Association of Computational Linguistics)– COLING (International Conference on Computational Linguistics )– HLT (Human Language Technology Conference)– IJCNLP (International Joint Conference on Natural Language Processing )

• Journals– Computational Linguistics– Natural Language Engineering– TALIP (ACM Transactions on Asian Language Information Processing)– TSLP (ACM Transactions on Speech and Language Processing)

6

Other Useful Sources (2)

• Resource URL– http://www.aclclp.org.tw/res_other_c.php ( 中華民國計算語言學學

會 )

– http://nlp.stanford.edu/software/index.shtml (Stanford NLP Group)

– http://www.phontron.com/nlptools.php (Graham Neubig)

• Tools/Software– Online Dictionary

• WordNethttp://wordnet.princeton.edu/

• HowNethttp://www.keenage.com/html/c_index.html

• The Academia Sinica Bilingual Ontological Wordnet (BOW)http://bow.sinica.edu.tw/

7

CKIP ( 中研院詞庫小組 )(Chinese Knowledge and Information Processing)

• Parser: http://140.109.19.112/main.exe?id=6833

• POS (part of speech) tagger: http://ckipsvr.iis.sinica.edu.tw/

8

Eric Brill's POS Tagger

• Website: http://cst.dk/online/pos_tagger/uk/

This/DT is/VBZ a/DT book/NN ./.

9

Stanford Parser

• Website – http://nlp.stanford.edu/software/lex-parser.shtml

• Tools– Online version

• Stanford Parser version 1.5.1

• English & Chinese

• http://josie.stanford.edu:8080/parser/

10

Stanford Parser

11

[Homework 1]

• Using CKIP POS (part of speech) tagger, Eric Brill’s POS tagger, and Stanford parser to tag and parse at least three sentence.

12

Course Topics

• Probability and Information Theory– basics: definitions, formulas, examples.

• Language Modeling– n-gram models, parameter estimation– smoothing (EM algorithm)

• Some Linguistics– phonology, morphology, syntax, semantics, discourse

• Words and the Lexicon– word classes, mutual information, lexicography.

13

Course Topics (cont.)

• Hidden Markov Models– background, algorithms, parameter estimation

• Tagging: methods, algorithms, evaluation– tag sets, HMM tagging, transformation-based, feature-based

• Grammars and Parsing: data, algorithms– statistical parsing: algorithms, parameterization, evaluation

14

Course Topics (cont.)

• Applications– Machine Translation (MT)– Acoustic Speech Recognition (ASR)– Information Retrieval (IR)– Cross-Language Information Retrieval (CLIR)– Question Answering (QA)– Cross-Language Question Answering (CLQA)– Summarization– Information Extraction– …

15

Course Introduction

• Lecture1: Introduction

• Lecture2: Mathematical Foundations

• Lecture3: Linguistics Essentials

• Lecture4: Corpus-based Work

• Lecture5: Collocations

• Lecture6: Statistical Inference: n-gram Models over Sparse Data

• Lecture7: Word Sense Disambiguation

• Lecture8: Statistical Alignment and Machine Translation

• Lecture9: Markov Models

• Lecture10: Term Translation Extraction & Cross-Language Information Retrieval

• Lecture11 : Statistical/Probabilistic Models for Word Alignment & CLIR

• Lecture12: Part-of-Speech Tagging

• Lecture13: Probabilistic Context Free Grammars

• Lecture14: Question Answering

16

The Ultimate Research Goal in Natural Language Processing (NLP) • To develop an automated language understanding

system• Why is this important?

– Easy for everyone to use language

– Natural Human interface for a variety of applications (e.g., database access, on-line tutor, robot control, etc.)

– Language seems fundamental for developing an intelligent system

• iPhone Siri

• IBM's DeepQA project

17

Natural Language is VERY Useful

18

OCR Problems

19

20

21

Aspects of Computational Linguistics

• Description of the Language: universals, cross-linguistic research

• Implementation of Computer Model: algorithms and data structures, formal models to represent knowledge, model of the reasoning process

• Psycho-Linguistic Aspect: humans are an existence proof of the computability of language comprehension; psychological research can be used to justify a computer model; obtain human processing parameters

22

NLP Issues• Why is NLP difficult?

– Many “words”, many “phenomena”, many “rules”• OED (Oxford English Dictionary): 400k words;

Finnish lexicon (of forms): ~2 ×107

• sentences, clauses, phrases, constituents, coordination, negation, imperatives/questions, inflections, parts of speech, pronunciation, topic/focus, and much more!

– irregularity (exceptions, exceptions to the exceptions, ...)• potato potato es (tomato, hero,...); photo photo s, and even:

both mango mango s or mango es

• Adjective / Noun order: new book, electrical engineering, general regulations, flower garden, garden flower

23

Difficulties in NLP (cont.)– Ambiguity

• books: NOUN or VERB?

– you need many books vs. she books her flights online

• Thank you for not smoking, drinking, eating or playing radios without earphones. (MTA bus)

– Thank you for not eating without earphones??

– Thank you for drinking?? …

• Fred’s hat was blown off by the wind. He tried to catch it.

– ...catch the wind or ...catch the hat ?

24

Rules or Statistics?• Preferences:

– context clues: she books books is a verb– rule: if an ambiguous word (verb/nonverb) is preceded by a

matching personal pronoun word is a verb

– pronoun reference:– she/he/it often refers to the most recent noun or pronoun (but

there are certainly exceptions)

– selectional restrictions:– catching hat is better than catching wind (but not always)

– semantics: – We thank people for doing helpful things or not doing annoying

things

25

Solutions

• Don’t guess if you know:• morphology (inflections)• lexicons (word information)• unambiguous names• perhaps some (really) fixed phrases• syntactic rules?

• Use statistics (based on real-world data) for preferences (only?)

• No doubt: but this is an important question!

26

Types of Linguistic Knowledge

• Acoustic/Phonetic Knowledge: How words are related to their sounds. (transliteration)– E ri c sson <=> 易利信

• Morphological Knowledge: How words are constructed out of basic meaning units.un + friend + ly unfriendly

love + past tense loved

object + oriented object-oriented

27

More Types of Linguistic Knowledge

• Lexical Knowledge (or Dictionary): This should include information on parts of speech, features (e.g., number, case), typical usage, and word meaning.

• Syntactic Knowledge: How words are put together to make legal sentences (or constituents of sentences).

28

More Types of Linguistic Knowledge

• Semantic Knowledge: Word meanings, how words combine into sentence meaning, – e.g., Fred tossed the ball.

Semantic roles

29

More Types of Linguistic Knowledge

• Pragmatic Knowledge: How context affects the interpretation of a sentence. Examples:– Louise loves him.

[Context 1:] Who loves Fred?[Context 2:] Louise has a cat. 

– What time is it?[Context 1:] Fred is fidgeting (坐立不安 )

and staring at his watch.[Context 2:] Louise has no watch. 

30

More Types of Linguistic Knowledge

• World Knowledge: How other people‘s minds work, what a listener knows or believes, the etiquette ( 成規 ) of language. Examples:– Will you pass the salt?

– I read an article about the war in the paper.

– Fred saw the bird with his binoculars.

– Tim was invited to Tom's birthday party. He went to the store to buy him a present.

31

Multilingualism Issues in Web Age

• Language barrier– There are about 6,700 languages listed in the Ethnologue

(http://www.ethnologue.com/)

• Information overloading– Scaling up of language resources

• Webpages

• News

• Weblogs

• Microblogs

32

Multilingual Understanding??

33

Multilingual Understanding??

34

Multilingual Understanding??

35

Real World Situation• Use statistical model based on REAL WORLD DATA and care

about the best sentence only • Imagine:

– Each sentence W = { w1, w2, ..., wn } gets a probability P(W|X) in a context X

– For every possible context X, sort all the imaginable sentences W according to P(W|X):

– Ideal situation:

best sentence (most probable in context X)

P(W)

Wbest Wworst

36

Real World Situation

• Unable to specify a set of grammatical sentences using fixed “categorical” rules

• (disregarding the “grammaticality” issue)

best sentence (most probable in context X)

P(W)

Wbest Wworst