Natural Language Processing - VUB · Syntax 8 Words This is a simple sentence WORDS Philipp Koehn...

Natural Language Processing

Introduction

Pieter Wellens2012-2013

These slides are partially based on the course materials from the ANLP course given at the School of Informatics, Edinburgh and the online coursera Stanford NLP course.

Saturday 16 February 13

Welcome

• Lecturer

• Pieter Wellens

• pieter@ai.vub.ac.be

• room: 10G725 (make appointment by e-mail or after class)

• Room: 10G725 (AI-Lab meeting room)

• Time: Each friday, 16-18h

Welcome

• Examination:

• assignments: 50%

• traditional exam (written and oral): 50%

• Pointcarré: used for urgent communication

• let me know if you are not yet registered

• http://pointcarre.vub.ac.be/run.php?application=weblcms&go=course_viewer&course=9108

Welcome

• Course website:

• contains slides, background reading, assignments, etc

• http://ai.vub.ac.be/courses/2012-2013/natural-language-processing

Welcome

• Textbook:

• Daniel Jurafsky and James H. Martin, Speech and Language Processing, second edition (2008/2009), Pearson International Edition.

• Steven Bird, Ewan Klein, and Edward Loper, Natural Language Processing with Python, first edition 2009.

• http://nltk.org/book/

What is NLP: Applications

• Speech Recognition

• Machine Translation

• Information Retrieval

• Dialog Systems

• Question Answering

• Information Extraction

• Summarization

• Sentiment Analysis

• ...

Overview of this class

• Linguistic terminology

• Tokenization

• Normalization and stemming

• Sentence segmentation

Linguistic terminology 8

This is a simple sentence WORDS

Philipp Koehn ANLP Lecture 1 19 September 2012

Linguistic terminology 8

Morphology

This is a simple sentencebe3sg

present

MORPHOLOGY

Morphology

present

MORPHOLOGY

Parts of Speech

present

DT VBZ DT JJ NN

MORPHOLOGY

PART OF SPEECH

Part of Speech

Linguistic terminology

Syntax

Morphology

present

MORPHOLOGY

Syntax

present

DT VBZ DT JJ NN

MORPHOLOGY

SYNTAX

PART OF SPEECH

Syntax

Morphology

present

MORPHOLOGY

Syntax

present

DT VBZ DT JJ NN

MORPHOLOGY

SYNTAX

PART OF SPEECH

Semantics

present

DT VBZ DT JJ NN

SENTENCE1string of words satisfying the

grammatical rules of a languauge

SIMPLE1having

few parts

MORPHOLOGY

SYNTAX

PART OF SPEECH

SEMANTICS

Semantics

present

DT VBZ DT JJ NN

SIMPLE1having

few parts

MORPHOLOGY

SYNTAX

PART OF SPEECH

SEMANTICS

Syntax

Morphology

present

MORPHOLOGY

Syntax

present

DT VBZ DT JJ NN

MORPHOLOGY

SYNTAX

PART OF SPEECH

Semantics

present

DT VBZ DT JJ NN

SIMPLE1having

few parts

MORPHOLOGY

SYNTAX

PART OF SPEECH

SEMANTICS

Semantics

present

DT VBZ DT JJ NN

SIMPLE1having

few parts

MORPHOLOGY

SYNTAX

PART OF SPEECH

SEMANTICS

Discourse

present

DT VBZ DT JJ NN

SIMPLE1having

few parts

But it is an instructive one.

CONTRAST

MORPHOLOGY

SYNTAX

DISCOURSE

PART OF SPEECH

SEMANTICS

Discourse

present

DT VBZ DT JJ NN

SIMPLE1having

few parts

CONTRAST

MORPHOLOGY

SYNTAX

DISCOURSE

PART OF SPEECH

SEMANTICS

Linguistic terminology today 8

Morphology

present

MORPHOLOGY

Words and Morphology:Tokenization

Word tokenization

• Demo time!

Word tokenization

• Demo time!

Word tokenization

• Definition: strings of letters separated by spaces

• But:

• punctuation: commas, periods, etc. (tokenization)

• hyphens: top-down

• clitics: Luc’s

• compounds: website, postbodetasbeschermhoes

• no spaces: 伦敦每日快报指出,两台记载黛安娜王妃一九九七年巴黎死亡车祸调查资料的手提电脑,被从前大都会警察总长的办公室里偷走. (chinese and japanese)

• Most frequent words in the English Europarl corpus (www.statmt.org/europarl/)

• http://www.edict.biz/textanalyser/wordlists.htm

Word CountsMost frequent words in the English Europarl corpus

any word nouns

Frequency in text Token1,929,379 the

1,297,736 ,

956,902 .

901,174 of

841,661 to

684,869 and

582,592 in

452,491 that

424,895 is

424,552 a

Frequency in text Content word129,851 European

110,072 Mr

98,073 commission

71,111 president

67,518 parliament

64,620 union

58,506 report

57,490 council

54,079 states

49,965 member

Word tokenization: counting

• But: many words occur only once and even more words never appear in the corpus

• Zipf’s law: f x r = k

• f = frequency of a word

• r = rank of a word (if sorted by frequency)

• k = a constant

• 10000 sentences from the Europarl corpus

• Why the difference?

How Many Di↵erent Words?10,000 sentences from the Europarl corpus

Language Di↵erent words

English 16k

French 22k

Dutch 24k

Italian 25k

Portuguese 26k

Spanish 26k

Danish 29k

Swedish 30k

German 32k

Greek 33k

Finnish 55k

Why the di↵erence? Morphology.

• 10000 sentences from the Europarl corpus

• Why the difference? Morphology

How Many Di↵erent Words?10,000 sentences from the Europarl corpus

Language Di↵erent words

English 16k

French 22k

Dutch 24k

Italian 25k

Portuguese 26k

Spanish 26k

Danish 29k

Swedish 30k

German 32k

Greek 33k

Finnish 55k

Why the di↵erence? Morphology.

Word Tokenization

• Lemma: the canonical, dictionary form

• throw, threw, throws, throwing are forms of the same lexeme, with throw as lemma.

• throw, threw, throws, throwing are forms of the same lexeme, with throw as lemma

• Stem: the part of a word that is common to all its inflected variants

• stem of delete and deletion is delet

Word Tokenization

• throw, threw, throws, throwing are forms of the same lexeme, with throw as lemma

• Stem: the part of a word that is common to all its inflected variants

• stem of delete and deletion is delet

• Wordform: Full inflected surface form

• cat and cats are different wordform

Word Tokenization

• Fragments: I uh am teaching you some uh interest - interesting stuff

• Filled pauses: I uh am teaching you some uh interest - interesting stuff

Word Tokenization

Word Tokenization: Types and tokens

• The shuttle bus dropped me at the wrong hotel in New York.

• Type: an element of vocabulary (vocabulary == set of types)

• Token: instance of a type in running text

• How many types and tokens?

• 12 tokens and 11 types

Tokens'='N' Types'='|V|'

Switchboard,phone,conversa2ons, 2.4,million, 20,thousand,

Shakespeare, 884,000, 31,thousand,

Google,NBgrams, 1,trillion, 13,million,

Morphology

present

MORPHOLOGY

Words and Morphology:Normalization and Stemming

Word Normalization: Morphemes

• Morphemes: smallest meaningful units that make up words

• Two types of morphemes

• stems: dog, fast, study

• affixes: +ed, un+, +s

• Four types of affixes

• suffix: cat+s (plural), small+er, great+ly, walk+ed

• prefix: un+seen, dis+entangled, re+phrase

• infix: op+ge+kropt (dutch)

• circumfix: ge+vraag+d (dutch)

• Creates lots of possibilities:

• nerd

• nerdify (14400 hits)

• nerd

• nerdify (14400)

• nerdifier (2050 hits)

• nerd

• nerdify (14400)

• nerdificiation (8860 hits)

• nerd

• nerdify (14400)

• nerdificiation (8860 hits)

• nerdificationism (117 hits)

• Difficulties:

• Consonants of the lemma may be changed to removed

• walk+ed, frame+d, transmit+ted

• tiny, tinier

• irregular forms

• Most frequent words have irregular forms

• to be: am, are, is, been, was, ...

• to eat: eat, ate, eaten, ...

• to go: go, goes, went, ...

Word Normalization: Irregular forms

• Differences in the definite article (e.g. the):

• http://wals.info/feature/37A?tg_format=map&v1=c00d&v2=c99f&v3=cd00&v4=dfff&v5=cfff

• Past tense:

• http://wals.info/feature/66A?tg_format=map&v1=cff0&v2=cf60&v3=cd00&v4=cfff

• Case affixes

• http://wals.info/feature/51A?s=20&z4=3000&z3=2999&z8=2998&z5=2997&z7=2996&z2=2995&z6=2994&z9=2993&z1=2992&tg_format=map&v1=c00d&v2=cd00&v3=c000&v4=c

Word Normalization: Cross-linguistic variation

Word Normalization: what is it?

• Information retrieval

• U.S.A -> USA

• window -> window, windows

• windows -> Windows, windows, window

• Windows -> Windows

• reducing all letters to lower case

• exceptions: General Motors, New York, ...

• can be important for sentiment analysis: US vs us

Lemmatization

• Reduce inflections or variant forms to base form

• am, are, is -> be

• car, cars, car’s cars’ -> car

• Finding the dictionary headword form

• Important in Machine translation

Stemming

• Reduce terms to their stems in information retrieval

• It is chopping of affixes from word forms

• Is a language dependent process

Porter’s algorithm: English stemmer

• Simple replacement algorithm, based on ordered replacement rules

!!!Step!1a!sses → ss ! caresses → caress!ies → i ! ponies → poni!ss → ss ! caress → caress!s → ø!!!!!!!!!cats → cat!

!!Step!1b!(*v*)ing → ø!!!!walking → walk! sing → sing!(*v*)ed → ø!!!!plastered → plaster!…!

!!!Step!1a!sses → ss ! caresses → caress!ies → i ! ponies → poni!ss → ss ! caress → caress!s → ø!!!!!!!!!cats → cat!

!!Step!1b!(*v*)ing → ø!!!!walking → walk! sing → sing!(*v*)ed → ø!!!!plastered → plaster!…!

!!!Step!2!(for!long!stems)!ational→ ate relational→ relate!izer→ ize ! digitizer → digitize!ator→ ate ! operator → operate!…!

!!!!Step!3!(for!longer!stems)!al → ø!!!!!!revival → reviv!able → ø!!!!!!adjustable → adjust!ate → ø activate → activ!…!

• M.F. Porter, 1980, An algorithm for suffix stripping, Program, 14(3) pp 130−137.

• You can download versions for many programming languages at: http://tartarus.org/martin/PorterStemmer/

•Demo!

Stemming: Language dependent

• Turkish:

• Uygarlas(ramadiklarimizdanmissinizcasina

• `(behaving) as if you are among those whom we could not civilize’

• Uygar `civilized’ + las `become’ + tir `cause’ + ama `not able’ + dik `past’ + lar ‘plural’+ imiz ‘p1pl’ + dan ‘abl’ + mis ‘past’ + siniz ‘2pl’ + casina ‘as if’

Morphology

present

MORPHOLOGY

Words and Morphology:Sentence segmentation

Sentence segmentation

• Segmenting out sentences from running text.

• !, ? are relatively unambiguous

• Period “.” is ambiguous

• Sentence boundary

• Abbreviations like Inc. or Dr.

• Numbers like .02 or 3.14

• How to solve the ambiguous “.”?

• Build a binary classifier that decides: EndOfSentence/NotEndOfSentence.

• possibilities: hand-written rules, regular expressions, or machine learning

• machine learning: SVM, neural network, Memory-based, decision tree, ...

Sentence segmentation

Sentence segmentation: decision trees

• More sophisticated features:

• Case of word with “.”: Upper, Lower, Cap, Number

• Case of word after “.”: Upper, Lower, Cap, Number

• Numeric features:

• Length of word with “.”

• Probability(word with “.” occurs at end-of-s)

• Probability(word after “.” occurs at beginning-of-s)

Sentence segmentation: decision trees

Syntax

Morphology

present

MORPHOLOGY

Syntax

present

DT VBZ DT JJ NN

MORPHOLOGY

SYNTAX

PART OF SPEECH

Semantics

present

DT VBZ DT JJ NN

SIMPLE1having

few parts

MORPHOLOGY

SYNTAX

PART OF SPEECH

SEMANTICS

Semantics

present

DT VBZ DT JJ NN

SIMPLE1having

few parts

MORPHOLOGY

SYNTAX

PART OF SPEECH

SEMANTICS

Discourse

present

DT VBZ DT JJ NN

SIMPLE1having

few parts

CONTRAST

MORPHOLOGY

SYNTAX

DISCOURSE

PART OF SPEECH

SEMANTICS

Discourse

present

DT VBZ DT JJ NN

SIMPLE1having

few parts

CONTRAST

MORPHOLOGY

SYNTAX

DISCOURSE

PART OF SPEECH

SEMANTICS

Conclusion

• Tokenization, lemmatization, stemming and sentence segmentation are almost inevitable when working with raw text.

• They are not rocket science but also not trivial. Can be very tedious.

• Often highly language dependent

• Use of machine learning when necessary but often regular expressions or iterative rules suffice

assignments by next week!

• Install python and NLTK, the natural language toolkit

• Follow instructions on: http://nltk.org/

• For python optionally use: http://epd-free.enthought.com/?Download=Download+EPD+Free+7.3-2

• Read the first chapter of the NLTK book and do the “Your turn” items.

• http://nltk.org/book/ch01.html

• Collect all your python commands and the output for the “Your turn” questions in a document and send it to pieter@ai.vub.ac.be

• deadline: Friday 22 February at 12.00h midday.

assignments by next week!

Natural Language Processing - VUB · Syntax 8 Words This is a simple sentence WORDS Philipp Koehn...

Documents

SENTENCE PATTERN - educationtn.com€¦ · SENTENCE PATTERN English- . A group of words which gives complete sense, is called a sentence. The basic sentence patterns are given below:

Sentence fluency transition words

SG at CTU · 3SG at CTU 3sg.czacm.org Cerny T., Chalupa V., Rychtecky L., Linhart T. 3sg-jf@googlegroups.com Vývoj enterprise aplikací

Presented by Diane Frame. Pre-KKinderFirst A sentence is made up of words. Orally separate a 4 word sentence into individual words Syllables: combine/delete

Sentence Structure *A sentence is a group of words expressing a complete thought

Example of a Pattern 1 Sentence: Sample Copy - Shurley · PDF filelearn sentence cores is to identify only the sentence pattern without classifying all the words in the sentence. To

SENTENCE MAKING SPELLING & SENTENCE … for students to identify words and groups, ... sentences to the class, ... Sentence Making, Spelling, Sentence Writing 5

Identifying correct sequence and reading known words ...€¦ · Instructions: Read each sentence. Then circle the picture that matches the sentence. Instructions: Read each sentence

files.schudio.com · Web viewChoose one word and hold a sentence with that word in (think of your sentence, count the words, say the sentence again, when you know your sentence

Speaking Words - Northwestern Universityfaculty.wcas.northwestern.edu/~mag938/rappGoldrickCNreviewMS.pdf · Speaking Words: Contributions of ... domains of sentence production and

Third Grade Vocabulary Words 2013-2014. Quiz 1 Declarative Sentence

classes of words, basic sentence patterns, sentence formulas

sentence. Please underline your spelling words! Write

Editing Messages. Readability Number of words per sentence / number of sentences Number of words per sentence / number of sentences Number of long words

SAT related words, familiar sayings, roots, prefixes, suffixes. Sentence Completion Read the entire sentence, saying “blank” for the blanks. Introductory and transitional words

Sentence Essentials Warm-up: What are the two type of words a sentence must have?

SYNTAX. Syntax = Sentence Structure The way that words are organized

Key words linguistics, syntax, and complex sentence

PARTICLES The words that hold a Japanese sentence together

Photosynthesis. Sentence Please write a sentence using at least 3 of the following words: Please write a sentence using at least 3 of the following words: