Paper Presentation: A Pendulum Swung Too Far

Embed Size (px)

DESCRIPTION

A paper presentation made by me for the paper 'A Pendulum Swung Too Far' by Kenneth Church at IIT Bombay as a part of preparation for the MTech Seminar. Get the paper on which this presentation is based here: http://languagelog.ldc.upenn.edu/myl/ldc/swung-too-far.pdf

Text of Paper Presentation: A Pendulum Swung Too Far

  • 1. Paper Presentation A Pendulum Swung Too Far (2011) by Kenneth Church Sagar Ahire [133050073]

2. Roadmap Introduction History of NLP Objections to Empiricism Chomsky Minsky Pierce Reasons for the Problem and Solutions 3. Roadmap: We Are Here Introduction History of NLP Objections to Empiricism Chomsky Minsky Pierce Reasons for the Problem and Solutions 4. Introduction The paper deals with the oscillation between the predominance of theory-driven approaches vs data-driven approaches in the history of NLP and its reasons. Specifically, it predicts a surge in rationalism in the 2010s and explains why and how researchers need to be prepared for it. 5. Rationalism vs. Empiricism RationalismEmpiricism1. Emphasizes on theory 2. Assumes an innate language faculty 3. Aims at discovering the language of the human mind (linguistic competence) 4. Assigns categories to language units 5. Major advocates: Chomsky, Minsky1. Emphasizes on data 2. Assumes all knowledge gathered only via senses 3. Aims at analysing language as it actually occurs (linguistic performance) 4. Assigns probabilities to language units 5. Major advocates: Shannon, Norvig 6. Rationalism vs. Empiricism RationalismEmpiricism1. Emphasizes on theory 2. Assumes an innate language faculty 3. Aims at discovering the language of the human mind (linguistic competence) 4. Assigns categories to language units 5. Major advocates: Chomsky, Minsky1. Emphasizes on data 2. Assumes all knowledge gathered only via senses 3. Aims at analysing language as it actually occurs (linguistic performance) 4. Assigns probabilities to language units 5. Major advocates: Shannon, Norvig 7. Rationalism vs. Empiricism RationalismEmpiricism1. Emphasizes on theory 2. Assumes an innate language faculty 3. Aims at discovering the language of the human mind (linguistic competence) 4. Assigns categories to language units 5. Major advocates: Chomsky, Minsky1. Emphasizes on data 2. Assumes all knowledge gathered only via senses 3. Aims at analysing language as it actually occurs (linguistic performance) 4. Assigns probabilities to language units 5. Major advocates: Shannon, Norvig 8. Rationalism vs. Empiricism RationalismEmpiricism1. Emphasizes on theory 2. Assumes an innate language faculty 3. Aims at discovering the language of the human mind (linguistic competence) 4. Assigns categories to language units 5. Major advocates: Chomsky, Minsky1. Emphasizes on data 2. Assumes all knowledge gathered only via senses 3. Aims at analysing language as it actually occurs (linguistic performance) 4. Assigns probabilities to language units 5. Major advocates: Shannon, Norvig 9. Rationalism vs. Empiricism RationalismEmpiricism1. Emphasizes on theory 2. Assumes an innate language faculty 3. Aims at discovering the language of the human mind (linguistic competence) 4. Assigns categories to language units 5. Major advocates: Chomsky, Minsky1. Emphasizes on data 2. Assumes all knowledge gathered only via senses 3. Aims at analysing language as it actually occurs (linguistic performance) 4. Assigns probabilities to language units 5. Major advocates: Shannon, Norvig 10. Roadmap: We Are Here Introduction History of NLP Objections to Empiricism Chomsky Minsky Pierce Reasons for the Problem and Solutions 11. History of NLP 12. 1950s: Empiricism Empiricism dominated across several fields Words were classified on the basis of their co-occurrence with other words (You shall know a word by the company it keeps Firth, 1957) 13. 1970s: Rationalism Several authors such as Chomsky, Minsky, etc criticized the Empirical approach Failure of the Empirical approach led to funding cutbacks (winters) 1966: Machine Translation Failure 1970: The abandonment of connectionism 1971-75: Speech Recognition Failure 14. 1990s: Empiricism Large amounts of data became available Several specialized problems could be solved by statistical frameworks, without concentration on the general problems 15. 2010s: Rationalism? Most of the low-hanging fruit has been picked up But the original criticisms of the empirical approach are still as valid 16. Roadmap: We Are Here Introduction History of NLP Objections to Empiricism Chomsky Minsky Pierce Reasons for the Problem and Solutions 17. Objections to Empiricism Several common empirical frameworks were opposed by rationalists in the 70s, including: Linear Separators (Machine Learning) Vector Space Model (Information Retrieval) n-grams (Language Modeling) HMMs (Speech Recognition) Many of these are mere approximations of complex phenomena 18. Chomskys Objections n-gram Language Modeling Finite State Methods 19. Chomskys Objections: n-gram Language Modeling Chomsky showed that n-grams cannot learn long-distance dependencies (dependencies spanning more than n words) For practical purposes n needs to be a small value (3 or 5) However, such small values fail to capture several interesting facts 20. Chomskys Objections: Finite State Methods Examples of Finite State Methods include Hidden Markov Models (HMMs) Conditional Random Fields (CRFs) Finite State Methods can capture dependencies beyond n words However, they may require infinite memory to process certain sentences 21. Chomskys Objections: Center Embedded Grammars A center embedded grammar is of the form: A -> x A y Chomsky proved that a center embedded grammar will require infinite memory and thus cannot be handled by finite state methods Center embedding is common in English, for example: A man that a woman that a child that a bird that I heard saw knows loves 22. Minskys Objection Linear Separators 23. Minskys Objections: Perceptrons Minsky showed that perceptrons (and linear separators in general) cannot learn functions that are not linearly separable such as XOR. 24. Minskys Objections: Perceptrons This has implications for several tasks including: Word Sense Disambiguation Information Retrieval Author Identification Sentiment Analysis For instance, this is the reason why sentiment analysis ignores loaded terms 25. Minskys Objections: Sentiment Analysis Loaded terms can be either positive or negative depending on whom it is addressed to. This is an XOR dependency: Loaded TermAddressed to usSentimentPositiveYPositivePositiveNNegativeNegativeYNegativeNegativeNPositive 26. Pierces Objections Evaluation by Demos Pattern Matching 27. Pierces Objections: Evaluation by Demos According to Pierce, evaluation of projects should be based on scientific principles rather than laboratory demos. Projects give good results in laboratory conditions, but have much higher error rates in real-world conditions. 28. Pierces Objections: Pattern Matching Pierce stated that pattern matching is artful deception, i.e. it is based on heuristics rather than scientific theory. Examples: The ELIZA effect The Chinese Room argument 29. Pierces Objections: Pattern Matching While pattern matching produces better results in the short term, it does so only by ignoring real scientific questions. While ambitious approaches may require time to deliver, they are backed by hard science. 30. Roadmap: We Are Here Introduction History of NLP Objections to Empiricism Chomsky Minsky Pierce Reasons for the Problem and Solutions 31. Reason for the Oscillations: Gaps in Teaching The losing side of the debate (currently Rationalism) is never mentioned in textbooks/courses Leads to reinventing the wheel by each generation of NLP researchers 32. Reason for the Oscillations: Gaps in Teaching Currently most courses concentrate on Statistical methods, ignoring linguistic and scientific questions This prepares students only for low-hanging fruit but not the real scientific questions 33. Solution Introduce the following in NLP courses: Syntax Morphology Phonology Phonetics Historical Linguistics Language Universals Create parallels between computational linguistics and formal linguistics 34. Solution Teach both sides of the rationalism vs. empiricism debate Educate students about the challenges ahead of the low-hanging fruit 35. Major References A Pendulum Swung Too Far by Kenneth Church, 2001 36. Other References Papers In Linguistics 1934-1951 by JR Firth, 1957 Syntactic Structures by Noam Chomsky, 1957 Whither Speech Recognition by John Pierce, 1969 ELIZA - A Computer Program for the Study of Natural Language Communication between Man and Machine by Joseph Weizenbaum, 1966 Minds, Brains and Programs by John Searle, 1980