Testing Functional Explanations of Word Order...

Testing Functional Explanations of Word Order Universals

Michael Hahn Richard FutrellStanford UC Irvine

(Greenberg 1963)

U3: ‘Languages with dominant VSO order are alwaysprepositional.’

U4: ‘With overwhelmingly greater than chancefrequency, languages with normal SOV order arepostpositional.’

U3: ‘Languages with dominant VSO order are alwaysprepositional.’

U4: ‘With overwhelmingly greater than chancefrequency, languages with normal SOV order arepostpositional.’

`Relative position of adposition & noun ~relative position ofverb & object’

OV languages with postpositions

VO languages with prepositions

Why do these universals hold?

Innate constraints on language, ‘Universal Grammar’? (Chomsky 1981)

Facilitation of human communication? (Dryer 1992, Hawkins 1994)

Make languages learnable? (Culbertson 2017)

Why do these universals hold?

Approach: Test functional explanations by implementing efficiency measures, optimizing grammars, and checking whether universals hold in optimized grammars.

Three Efficiency Measures

Dependency Length Minimization (Rijkhoff, 1986; Hawkins, 1994, 2003)

Surprisal (Gildea and Jaeger, 2015; Ferrer-i Cancho, 2017)

Parsability (Hawkins, 1994, 2003)

21 1+ + = 4

Three Efficiency MeasuresSurprisal

Surprisal(w1...wi-1) = -Σi log P(wi|w1...wi-1)

Three Efficiency MeasuresSurprisal

Surprisal(w1...wi-1) = -Σi log P(wi|w1...wi-1)

Estimated using recurrent neural networks, the strongest existing methods for estimating surprisal and predicting reading times.

Three Efficiency MeasuresParsability

Mary has two green books.

Parsability(utterance) := log P(tree | utterance)

Estimated using a neural network model (Dozat and Manning 2017)

with extremely generic architecture.

Utility Informativity Cost-=

Amount of Meaning that can be extracted from utterance

Cost of processing utterance

Combining Parsability + Surprisal

Long tradition as an explanation of language (Gabelentz 1903, Zipf 1949, Horn 1984, …)

Amount of Meaning that can be extracted from utterance ~ Parsability

~ Surprisal

~ SurprisalLong tradition as an explanation of language (Gabelentz 1903, Zipf 1949, Horn 1984, …)

Formalized in Rational-Speech Acts models (Frank and Goodman 2012)

Related to Signal Processing (Rate-Distortion Theory, Information Bottleneck)

~ Surprisal

Why do the universals hold?

Approach: Test processing explanations by implementing efficiency measures, optimizing grammars, and checking whether universals hold in optimized grammars.

Testing Functional Explanations

Approach: Optimize the word orders of languages for the three objectives, keeping syntactic structures unchanged

Languages have word order regularities ⇒ Not sufficient to optimize the word orders of individual sentences

Instead: optimize word order rules of entire languages

That is: optimized languages have optimized but internally consistent grammatical regularities in word order, and agree with an actual natural language in all other respects.

Mary has two green books

nummod

Dependency Corpus

nummod

hastwo

greenbooks

Tree Topologies

Dependency Corpus

nummod

hastwo

greenbooks

Tree Topologies

Dependency Corpus Ordering GrammarNOUN ADJamod

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

nummod

hastwo

greenbooks

Tree Topologies

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

“Object follows verb”

nummod

hastwo

greenbooks

Tree Topologies

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

“Adjective precedes noun”

nummod

hastwo

greenbooks

Tree Topologies

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

“Adjective precedes noun”

“Numerals follow adjectives & precede nouns”

nummod

hastwo

greenbooks

Tree Topologies

Maryhastwogreenbooks

Counterfactual Corpus

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

nummod

hastwo

greenbooks

Tree Topologies

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

Each parameter setting generates a different counterfactual corpus.

nummod

hastwo

greenbooks

Tree Topologies

Maryhastwogreen books

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

nummod

hastwo

greenbooks

Tree Topologies

Maryhas twogreenbooks

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

Dependency Length Surprisal

Parsability

2.35.81.8

We compute processing measures on counterfactual corpora.

Parsability

2.35.81.8

Each parameter setting results in different values for the processing measures.

Parsability

2.94.52.9

Parsability

3.47.81.2

Parsability

3.47.81.2

Which settings optimise the measures?

Parsability

3.47.81.2

Which settings optimise the measures?

Do the optimised settings replicate the Greenberg correlations?

For each objective, find parameters that optimise it.

NOUN ADJamod0.1

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

NOUN ADJamod0.1

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

Minimize Dep. Length Minimize Surprisal

NOUN ADJamod0.1

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

NOUN ADJamod0.21

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

Maximize Parsability Optimize Pars.+Surp.

Repeat this for corpora from 51 real languages from Universal Dependencies Project.

NOUN ADJamod0.1

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

NOUN ADJamod0.1

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

NOUN ADJamod0.1

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

NOUN ADJamod0.21

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

Repeat this for corpora from 51 real languages from Universal Dependencies Project.

NOUN ADJamod0.1

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

NOUN ADJ 0.1

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

NOUN ADJ 0.1

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

1. How do the objectives compare?2. Which universals are predicted?

Surprisal and Parsability minimize Dependency Length

Communicative Utility predicts Dependency Length Minimization.

Better Parsability

Lower Surprisal

z-transformed on the level of languages

Language optimizes Surprisal and Parsability

Better Parsability

Lower Surprisal

Random Grammars

Better Parsability

Lower Surprisal

Random Grammars

Grammars fit to Real Orderings

Better Parsability

Lower Surprisal

Random Grammars

Optimized for Surprisal

Optimized for Parsability

Optimized for Parsability+Surprisal

Grammars fit to Real Orderings

(Dryer 1992 in Language)

`Relative position of adposition & noun ~relative position ofverb & object’

We formalize the correlations in the Universal Dependencies format.

(Dryer 1992 in Language)

We formalize the correlations in the Universal Dependencies format.

For any word order grammar, we can then check which correlations it satisfies.

Are the universals satisfied by models fit to the actual orderings for our 50 languages?

Prevalence of SVO (Dryer 1992)

Limitation of formalisation

Percentage of grammars optimized for each objective satisfying the universal

Assessing Significance:X = “Object precedes verb”Y = “Object-patterner precedes verb-patterner”

Logistic model:Y ~ X + (1+X|family) + (1+X|language)

Predictions largely complementary

Predictions mostly agree

Communicative Utility replicates predictions of Dependency Length Minimization.

Predictions mostly agree

Communicative Utility replicates predictions of Dependency Length Minimization.Both measures predict most of the correlation universals.

Conclusion

● Tested explanations of Greenberg correlation universals in terms of efficiency of human processing and communication

● Using corpora from 50 languages, constructed counterfactual optimized languages

● Most of the correlations can be derived from pressure to shorten dependencies, decrease surprisal, or increase parsability

● Clear evidence for functional explanations of word order universals

Optimized grammars are easier to parse even when sentences are presented in orders very different from natural language

ACEBD ADBEC ACEDBABCDE

Random grammarOptimized grammar

Random grammars remain hard to parse even as training data increases.

Formalizing ParsabilityNeural parser (Dozat and Manning 2017):

Mary met John

hn 1. BiLSTM reads the sentence2. Identify heads by

computing score for each pair of words

Generic architecture, no assumptions beyond sequential nature of input.

Formalizing ParsabilityInformation about syntactic tree that can be extracted from sentence:

Mary met John

Formalizing Dependency Length

Distance between word and its syntactic head

summing over all words in sentence

sentence w = w1...wn

Formalizing Surprisal

summing over all words in sentence

per-word surprisal

Surprisal depends on the probability model P.

Right choice of P depends on the entire language!

Given a word order grammar θd choose the model that minimizes surprisal on the resulting sentences.

Use LSTM recurrent neural networks, the SOTA in probabilistic modelling of natural language and predicting reading times.Very general sequence models, arguably minimizing architectural biases.

Formalizing Informativity

Information about the syntactic tree that can be extracted from the sentence:

Formalizing Informativity

Information about the syntactic tree that can be extracted from the sentence:

Use a recent neural model (Dozat and Manning 2017) with generic architecture and SOTA performance on many languages.

Word Order Grammars

For each dependency type, there are two parameters:a. α: probability that whether dependent precede headb. β: determines distance

two green

booksαverb-object = 0.1

αverb-subject = 0.95

αnoun-numeral = 0.99 αnoun-adjective = 0.8

two green

booksαverb-object = 0.1

αverb-subject = 0.95

αnoun-numeral = 0.99 αnoun-adjective = 0.8

Maryhas

Word Order Grammars

For each dependency type, there are two parameters:a. α: probability that dependent precede headb. β: determines distance

βNoun-Adjective = -0.3

βNoun-Numeral = 0.8

softmax(βNoun-Adjective , βNoun-Numeral ) ~ (0.1, 0.9)

adjective first

numeral first

softmax(βNoun-Adjective , βNoun-Numeral ) ~ (0.1, 0.9)

adjective first

numeral first

Maryhas

twogreen

Word Order Grammars

For each dependency type, there are two parameters:a. α: probability that dependent precede headb. β: determines distance

This specifies the space of possible grammars, within which we optimize.

nummod

hastwo

greenbooks

Tree Topologies

NOUN NUMnummod

VERB NOUNnsubj

VERB NOUNdobj

nummod

Will be working with trees in the Universal Dependencies format:

nummod

Will be working with trees in the Universal Dependencies format:

To optimize grammars, we need a space of possible grammars.

SOV and VSO support correlationSVO does not

Support SVO(Gibson et al 2013)

Dependency Length MinimizationShort syntactic dependencies ease processing (Gibson, 1998; Grodner and Gibson, 2005; Demberg and Keller, 2008; Bartek et al., 2011)

Quantitative corpus evidence from many languages confirms that languages have shorter dependencies than would be expected at random (Futrell et al., 2015).

Dependency Length MinimizationShort syntactic dependencies ease processing (Gibson, 1998; Grodner and Gibson, 2005; Demberg and Keller, 2008; Bartek et al., 2011)

Quantitative corpus evidence from many languages confirms that languages have shorter dependencies than would be expected at random (Futrell et al., 2015).

Argued to explain several of the Greenberg correlations (Rijkhoff, 1986; Hawkins, 1994, 2003)

Two Objectives for Optimization

Dependency Length Minimization

Communicative Utility

Utility Informativity Cost-= λ

Two Objectives for OptimizationCommunicative Utility

Informativity(utterance) := log P(tree | utterance) - log P(tree)

Informativity(utterance) := log P(tree | utterance) - log P(tree)We use a neural network model (Dozat and Manning 2017) with extremely generic architecture.

Surprisal(wi|w1...wi-1) = -log P(wi|w1...wi-1)

We use recurrent neural networks, the SOTA in probabilistic modelling of natural language and predicting reading times.

Parsability

2.35.81.8

(1) For each objective, find parameters that optimise it.

(2) Which universals do the resulting counterfactual languages satisfy?

Parsability

2.35.81.8

(1) For each objective, find parameters that optimise it.

(2) Which universals do the resulting counterfactual languages satisfy?

Testing Functional Explanations of Word Order...

Documents

Final Amod

AMOD GPS Photo Tracker AGL3080units.flwg.us/resources/site2298/General/airborne... · AGL3080 GPS Photo Tracker User Manual V2.2 May 2009 V2.2 AMOD Technology 3 Overview AGL 3080

AMOD GPS Photo Tracker AGL3080module.amod.com.tw/files/Product/AGL 3080 GPS Photo Tracker User... · AGL3080 GPS Photo Tracker User Manual V2.2 May 2009 V2.2 AMOD Technology 8 Import

My Dna Amod Kumar Ias Sitapur E Governance Initiative P1a

ENG 109 - efy.ksu.edu.sa · All the questions are MCQs with four options. ... blood noun book noun breastbone noun ... pathology noun patient noun

pace noun/verb Pacific noun/adj. pack verb/noun package noun … · piss verb/noun 中高級 pistol noun 中高級 pit noun/verb 中級 pitch noun/verb 中級 pitcher noun 中高級

Chapter 11 Allie Frink 7C. Noun Abstract noun Common noun

aMod.40 - Fund. Metod. e Pratica Do Ensino de Ciencias

Amod (G-19 Extn.) Lignite Mine

still life noun noun: still life ; plural noun: still lifes ; modifier noun: still-life

Unit 1 Social Studies Vocabulary & Essential …bridges-sifeproject.com/Course_Materials/06_SS/SS_U01/...noun country país noun globe globo noun map mapa noun city ciudad noun town

Noun and Noun Phrase

Grammars and Parsing. Sentence Noun Verb Noun Noun boys Noun girls Noun dogs Verb like Verb see Grammars Grammar: set of rules for generating

การแปล Countable-noun-uncountable-noun

Indian forests amod saini

Lecture 4 Noun and Noun Phrase

Merry Christmas! Mad Libs - Happiness is Homemade€¦ · Merry Christmas! Mad Libs adjective adjective adjective adjective Noun Noun Noun Noun Noun Noun Noun Noun Holiday Holiday

WaterColor Amy Tanner. wa·ter·col·or ˈwôdərˌkələr,ˈwädərˌkələr / noun noun: watercolour; plural noun: watercolours; noun: watercolor; plural noun: watercolors;

1. Possesor Phrase Possessee Noun (Genitive Noun phrase) (Head Noun) 2.Possessee Noun Possesor Phrase (Head Noun) (Genitive Noun phrase) GENITIVE PHRASE

The Madhya Pradesh Vilasita, Manoranjan, Amod Evam