Natural Language Processing in Digital Humanities: …Presentation at IXA, January 2016 Pablo Ruiz...

Natural Language Processing in Digital Humanities: application examples

Presentation at IXA, January 2016

Pablo Ruiz Fabo — LATTICE Lab

Summary

• Digital Humanities’ needs in terms of text

analysis tools

• Two examples of pipelines to address

these needs:

– Entity Linking and the PoliInformatics corpus

– Proposition Extraction and the Earth

Negotiations Bulletin Corpus

• Implementation choices, demo, evaluation

2

Digital Humanities (DH)

• Application of computational methods to

questions in fields like Humanities or

Social Sciences.

– A goal is to allow for asking questions and

attaining findings that would be impossible

without computational means (Berry, 2012)

– Critical reflection: “as much of a focus on what

the computational techniques obscure as

reveal” (Meeks and Weingart, 2012)

3

DH and Text: Topic Models

• Very popular tool in DH: LDA (MALLET) (discussion in Meeks and Weingart, 2012)

4

Europarl Corpus Topics for all speeches by the French Green Party (Les Verts), 1999-2004 session

DH and Text …

• But there are many other Text Analysis /

Language Technology / Natural Language

Processing applications potentially useful

for DH questions …

5

DH methods: Network analysis

• Transforming a collection of texts into a

network (a graph)

• Network’s nodes:

– Actors in the corpus: People, Institutions …

– Concepts in the corpus

6

DH methods: Network analysis

• Transforming a collection of texts into a

network (a graph)

• Network’s nodes:

– Actors in the corpus: People, Institutions …

– Concepts in the corpus

7

DH methods: Network analysis

• Transforming a collection of texts into a

network (a graph)

• Network’s nodes:

– Actors in the corpus: People, Institutions …

– Concepts in the corpus

Named Entity Recognition / Disambiguation

Entity and Concept Linking

8

Text => Network via EL: User Needs

9

Venturini et al. (2012) Once Upon a Text

[médialab at Paris SciencesPo]

The careful use of natural language processing algorithms could provide better filtering metrics and support in expression merging

The manual filtering is crucial because it allows entities to be reduced to a set size appropriate for analysis, but also recovering important entities that could have been excluded by the automatic filtering.

Though AlchemyAPI offers a trustworthy

service, we don’t like relying on it. In

particular, we don’t like that the service is

offered as a “black box” and that the exact

extraction algorithm is secret.

10

Venturini and Guido (2012). Once upon a text:

An Actor-Network Theory tale in text analytics.

Text => Network via EL: User needs

https://github.com/medialab/ANTA

Entity Linking in DH: User Needs

• Variety of corpora needs to be treated

• EL literature shows that tools’ performance

varies according to the corpus:

11

TOOL CORPUS

AIDA/CoNLL (news, sports) IITB (web, various topics)

P R F1 P R F1

Spotlight 31.2 40.4 35.2 46.2 50.0 48.0

TagMe 61.4 55.5 58.3 45.2 42.0 43.6

WikipediaMiner 46.9 52.8 49.7 56.8 48.2 43.6

AIDA 63.3 29.1 39.8 65.7 4.1 7.6

Data from Cornolti et al. (2013) Metric: Weak Annotation Match

• Correlations between number of

occurrences of a textual feature and tool

performance (Usbeck et al. 2015)

12

CORRELATIONS Nbr. PER Nbr. ORG Nbr. LOC Nbr. OTHER

Babelfy 0.769 -0.376 0.254 -0.431

Spotlight 0.217 -0.480 -0.461 0.26

TagMe 0.257 -0.272 -0.194 0.036

WikipediaMiner 0.082 -0.679 -0.632 0.497

Data from 20 Nov 2015, GERBIL platform (gerbil.aksw.org/gerbil/overview), A2KB/Ma task

Entity Linking in DH: User Needs

Entity Linking approach given User needs

NEED APPROACH

• Avoiding “black boxes” • Open source tools

• Treating a variety of corpora, knowing that tools’ performance varies with the corpora

• Combining tools to get complementary results

• Manual filtering of entities

• Information to guide the filtering

• Providing annotation quality metrics to users

• Simultaneous acess to metrics and text to validate annotations

• Optional automatic annotation selection 13

Open Source Tool Combination

• Open-source Public-domain tools which link

to generic ontologies (DBpedia, YAGO, Babelnet)

[P.S.: OK maybe Babelfy is not exactly open source … we

could perhaps use AGDISTIS:

https://github.com/AKSW/AGDISTIS]

14

2010 2011 2008 2011 2014

Metrics to guide filtering?

15

• Confidence scores

• Coherence scores

EL: Annotation confidence

SOCCER –JAPAN GET LUCKY WIN,

CHINA IN SURPRISE DEFEAT

16

CONFIDENCE

EL: Annotation coherence

• Wikipedia Link Based Measure: Relies on

common inbound links

• Milne & Witten (2008) [original proposal]

• Ferragina et al. (2010) [optimizations]

• Hoffart et al. (2011) [among other measures]

• Moro et al. (2014) [other measures]

17

Milne-Witten coherence between entities e1 and e2 (as in Hoffart et al. 2011)

Entity Linking Demo

18

Demo: PoliInformatics Corpus

19

Demo: PoliInformatics Corpus

20

• NLP Unshared Task in PoliInformatics

2014

– Who was the financial crisis?

• Participants: individuals, industries, …

– What was the financial crisis?

• Causes, proposals for reform, ….

• Heterogeneous corpus containing

– Congress Hearings transcripts

– Official reports on the crisis by Congress

– Bills

– Etc.

Demo: PoliInformatics Corpus

21

• NLP Unshared Task in PoliInformatics

2014

– Who was the financial crisis?

• Participants: individuals, industries, …

– What was the financial crisis?

• Causes, proposals for reform, ….

• Heterogeneous corpus containing

– Congress Hearings transcripts

– Official reports on the crisis by Congress

– Bills

– Etc.

EL Demo: Corpus

22

Congress Hearings: Interviews to witnesses

EL Demo: Corpus

23

Official report by Congress about the causes

for the crisis

Entity Linking Demo

2010 2011 2008 2011 2013

results displayed on demo not displayed

Description: Ruiz, Poibeau & Mélanie (Demo at NAACL 2015).

Evaluation

• … as an NLP-related system

• … in terms of Digital Humanities

25

• Does a selection among the annotations

provided by several systems outperform

each of those systems’ annotations taken

individually?

• Combination method: ROVER. Each

system is weighted by its precision on a

test-corpus – Fiscus, 1997 for ASR

– De la Clergerie et al. 2008, for parsing

– Ruiz and Poibeau, 2015 (*SEM poster)

26

Evaluation as an NLP system

ROVER: system weights

• Assume two application scenarios:

– User wants Entities only

– User wants Entities and Concepts

• For Entities only:

– Weight according to Precision on

AIDA/CoNLL B corpus (no concepts in gold)

• For Entities and Concepts:

– Weight according to P on IITB corpus (many

concepts in gold)

27

Corpora: Cornolti et al. 2013, BAT Framework https://github.com/marcocor/bat-framework/tree/master/benchmark/datasets

ROVER: testing

• Entities only was tested on the MSNBC

corpus (reference set has no concepts)

• Entities and Concepts was tested on the

AQUAINT corpus (ref set has concepts)

28

29

Evaluation as an NLP system M

icro

-ave

rage

d S

tro

ng

An

no

tati

on

Mat

ch

t =

too

l’s o

pti

mal

co

nfi

de

nce

th

resh

old

*p

<0

.05

(ra

nd

om

per

mu

tati

on

)

30

Evaluation as an NLP system M

icro

-ave

rage

d S

tro

ng

An

no

tati

on

Mat

ch

t =

too

l’s o

pti

mal

co

nfi

de

nce

th

resh

old

*p

<0

.05

(ra

nd

om

per

mu

tati

on

)

31

Evaluation as an NLP system M

icro

-ave

rage

d S

tro

ng

An

no

tati

on

Mat

ch

t =

too

l’s o

pti

mal

co

nfi

de

nce

th

resh

old

*p

<0

.05

(ra

nd

om

per

mu

tati

on

)

Evaluation for DH

• Possible researcher objectives:

– Finding evidence about a research question

– Doing that faster or with less manual work

– Obtaining networks that confirm (or challenge)

previously available knowledge

– Obtaining quantitative evidence where only

qualitative one was available

• Do the following elements help ?

– The UI and workflows it allows

• The confidence and coherence scores

• The automatic annotation selection

32

Evaluation for DH

• Possible researcher objectives:

– Finding evidence about a research question

– Doing that faster or with less manual work

– Obtaining networks that confirm (or challenge)

previously available knowledge

– Obtaining quantitative evidence where only

qualitative one was available

• Do the following elements help ?

– The UI and workflows it allows

• The confidence and coherence scores

• The automatic annotation selection

33

34

35

36

Proposition Extraction

• Daily reports on international climate

conferences (Conference of the Parties or

COP), like COP-21 which took place in

December 2015.

• Summary of participant countries’

proposals.

37

Prop Ext: Corpus

• Identify different countries’ participations

• Identify negotiation points supported or

opposed by participants

• Help researchers compare countries’

positions via keyphrase and entity

extraction on the negotiation points

• Provide more detailed analysis than

prior work on this corpus, based on

word co-occurrence methods (Venturini et al., 2014)

38

Pipeline Objectives

Typical corpus sentence

The EU, with NEW ZEALAND and opposed

by CHINA, MALAYSIA and BHUTAN,

supported including the promotion of natural

regeneration within the definitions of

"afforestation" and "reforestation."

39

Actors (Countries)

The EU, with NEW ZEALAND and opposed

by CHINA, MALAYSIA and BHUTAN,

supported including the promotion of natural

regeneration within the definitions of

"afforestation" and "reforestation."

40

Messages (negotiation points)

The EU, with NEW ZEALAND and opposed

by CHINA, MALAYSIA and BHUTAN,

supported including the promotion of

natural regeneration within the

definitions of "afforestation" and

"reforestation."

41

Predicates (support/opposition)

The EU, with NEW ZEALAND and opposed

by CHINA, MALAYSIA and BHUTAN,

supported including the promotion of

natural regeneration within the definitions of

"afforestation" and "reforestation."

42

Actor + Predicate + Message =

Proposition

43

Propositions

ACTORS PREDICATES MESSAGES

European_Union supported including the promotion of

natural regeneration within the definitions of "afforestation" and "reforestation."

New_Zealand

China

~supported Malaysia

Bhutan

44

Propositions

ACTORS PREDICATES MESSAGES

1 European_Union supported including the promotion of

natural regeneration within the definitions of "afforestation" and "reforestation."

2 New_Zealand

3 China

~supported 4 Malaysia

5 Bhutan

45

Propositions

ACTORS VERBAL

PREDICATES MESSAGES

1 European_Union supported including the promotion of

natural regeneration within the definitions of "afforestation" and "reforestation."

2 New_Zealand

3 China

~supported 4 Malaysia

5 Bhutan

46

ACTORS NOMINAL

PREDICATES MESSAGES

1 Group_of_77 / China

proposal

to include research and development in the transport and energy sectors in the priority areas to be financed by the SCCF.

Sources of proposition info?

• Open Relation Extraction

– OLLIE (Mausam et al., 2012 EMNLP)

• https://github.com/knowitall/ollie

– Open Information Extraction 4.0

• https://github.com/knowitall/openie

• Traditional sources

– Syntactic dependency parsing

– Semantic Role Labeling

(CONLL 08/09 for both)

47

Sources of proposition info?

• Open Relation Extraction

– OLLIE (Mausam et al., 2012 EMNLP)

• https://github.com/knowitall/ollie

– Open Information Extraction 4.0

• https://github.com/knowitall/openie

• Traditional sources

– Syntactic dependency parsing

– Semantic Role Labeling

• IXA Pipes wrapper for MATE-tools

48

Can we do this with patterns?

The EU, with NEW ZEALAND and

opposed by CHINA, MALAYSIA and

BHUTAN, supported including the

promotion of natural regeneration within

the definitions of "afforestation" and

"reforestation."

49

Can we do this with patterns?

[ The EU, with NEW ZEALAND ] and

[ opposed by CHINA, MALAYSIA and

BHUTAN ], [ supported including the

promotion of natural regeneration within

the definitions of "afforestation" and

"reforestation." ]

50

Cf. Salway et al., 2014, ACL, Grammar Induction approach exploiting the ADIOS algorithm from Solan et al. 2005, PNAS

Can we do this with patterns?

• Maybe, but …

– What about anaphora resolution?

– What about negation?

• An NLP pipeline deals with these

phenomena in a uniform way (unlike

“linguistically-agnostic” patterns)

51

Can we do this with patterns?

• Maybe, but …

– What about anaphora resolution?

– What about negation?

• An NLP pipeline deals with these

phenomena in a uniform way (unlike

“linguistically-agnostic” patterns)

• IXA Pipes provides SRL info, coreference

chains (and syntactic dependencies in

case needed)

52

Using SRL info

• We have propositions (events) involving:

– speakers

– reporting verbs/reporting-related nouns

– messages communicated by the speakers

53

The EU, with NEW ZEALAND and opposed by

CHINA, MALAYSIA and BHUTAN, supported

including the promotion of natural

regeneration within the definitions of

"afforestation" and "reforestation."

Using SRL info

• We have propositions (events) involving:

– speakers: generally in the predicate’s A0 role

• List of countries and other actors created manually

from specialized sources (UNFCC site)

– reporting verbs/reporting-related nouns

• Created a list based on VerbNet and NomBank

(using NLTK interface).

– messages communicated by the speakers:

generally the predicate’s A1 role

(complemented with adjunct roles …)

• What about “opposed by …”? (below)

54

Using SRL Info: Generic rule

55

“Opposed by” spans

• A rule verifies if roles contains a span

introduced by “opposed by”. If yes:

• The main verb, and the negotiation point

related to it are found

• A proposition is created with:

– Each actor in the “opposed by” span

– A negated form of the main verb

– The main verb’s message

56

Treating Pronominal anaphora

• IXA Pipes’ CorefGraph coreference chains

A small subset of possible cases is treated

• Note: He/She can refer to a country (acc.

to country representative’s gender)

- Pronoun antecedents only searched in

their same sentence or the preceding one

- Actor in main verb’s subject (dep-

parsing) is the antecedent of a sentence-

initial he/she in the following sentence.

• Evaluation: Accurate. But coverage … ? 57

Ru

les

Treating negation

• AM-NEG roles from SRL

• Surface cues: negative items in a small

window preceding a predicate

– Not, no, lack of, …

• Problems …

– There was no lack of acceptance by …

58

Overall goal

• Help researchers to compare countries’

positions, via keyphrases and linked

entities in the messages supported or

opposed by countries

• Who agrees with whom?

59

Proposition Extraction Demo

60

KeyPhrase Extraction

• YaTeA (Aubin et Hamon, 2006)

61

Evaluation as an NLP system

• 311 propositions from ENB corpus: F1 .69

• 631 propositions from IPCC corpus (official

scientific report creation negotiations): F1 .72

• Exact match of all three proposition elements

62

ENB

res

ult

s

• Is this pipeline (and UI) helping

researchers analyze climate negotiations?

– Comparing actors’ positions

– Looking for answers to research questions

– Drawing attention to overlooked evidence

– Confirming one view on a controversial point

– …

63

Evaluation for DH

Future work

• More implementation

• Actual domain-expert evaluation for DH

purposes

• Assessing contribution of the work based

on domain-expert evaluation

64

Other technologies

• Discourse Analysis (e.g. Xue et al., 2015 (CONLL Task))

– Testing Lin et al. 2014 PDTB parser on the

subcorpora of the IPCC corpus

• Assessment Report

• Summary for Policymakers

• Technical summary

• Semantic Textual Similarity (Agirre et al., 2012

onwards, SemEval)

– Testing TakeLab system (from SemEval 2012)

– Comparing to “non-semantic” similarity

• Perhaps Interpretable STS (Agirre et al. SemEv

2015 Pilot) 65

Summary

• Proposing to researchers in Humanities or

Social Sciences:

– Language technologies that help move

beyond word co-occurrence methods

• Making researchers in other domains

familiar with a broader variety of

Language Technologies

• Evaluating impact of the tools on those

researchers’ activities

66

Some references (1)

Sophie Aubin and Thierry Hamon. (2006) Improving Term

Extraction with Terminological Resources. In Advances

in Natural Language Processing: 5th International

Conference on NLP, FinTAL 2006, pp. 380-387. LNAI

4139. Springer.

David Berry (2012). Understanding Digital Humanities.

Palgrave

Marco Cornolti, Paolo Ferragina, and Massimiliano

Ciaramita. (2013). A framework for benchmarking

entity-annotation systems. In Proc. of WWW, 249–260.

Éric V. De La Clergerie, Olivier Hamon, Djamel Mostefa,

Christelle Ayache, Patrick Paroubek, and Anne Vilnat.

(2008). Passage: from French parser evaluation to

large sized treebank. In Proc. of LREC 2008

Paolo Ferragina and Ugo Scaiella. (2010). Tagme: on-the-fly

annotation of short text fragments (by wikipedia

entities). In Proc. of CIKM’10, 1625–1628.

Jonathan G. Fiscus. (1997). A post-processing system to

yield reduced word error rates: Recognizer output

voting error reduction (ROVER). In Proc. of the IEEE

ASRU Workshop, 1997, 347–354.

Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino,

Hagen Fürstenau, Manfred Pinkal, Marc Spaniol,

Bilyana Taneva, Stefan Thater, and Gerhard Weikum.

(2011). Robust disambiguation of named entities in

text. In Proc. of EMNLP, 782–792.

Heng Ji, Joel Nothman and Ben Hachey. (2014). Overview

of TAC-KBP2014 Entity Discovery and Linking Tasks. In

Proc. Text Analysis Conference.

Lin, Ziheng, Hwee Tou Ng, and Min-Yen Kan. 2014. A PDTB-

Styled End-to-End Discourse Parser. Natural Language

Engineering 20 (02): 151–84.

Mausam, Schmitz, Bart, Soderland, Etzioni, and others.

(2012). Open Language Learning for Information

Extraction. In Proc. EMNLP / CoNLL, 523–34.

Elijah Meeks and Scott B. Weingart. (2012). The Digital

Humanities Contribution to Topic Modeling. Journal of

Digital Humanities, 2:1

Pablo N. Mendes, Max Jakob, Andrés García-Silva, and

Christian Bizer. (2011). DBpedia spotlight: shedding

light on the web of documents. In Proc. of the 7th Int.

Conf. on Semantic Systems, I-SEMANTICS’11, 1–8.

David Milne and Ian H. Witten. (2008a). An effective, low-

cost measure of semantic relatedness obtained from

Wikipedia links. In Proc. of AAAI Workshop on

Wikipedia and Artificial Intelligence: an Evolving

Synergy, 25–30.

Andrea Moro, Alessandro Raganato, and Roberto Navigli.

(2013). Entity Linking meets Word Sense

Disambiguation: A Unified Approach. Transactions of

the ACL, 2, 231–244.

Thierry Poibeau, Horacio Saggion, Jakub Piskorski, and

Roman Yangarber (eds.). (2012). Multi-source,

Multilingual Information Extraction and Summarization.

Springer Science & Business Media.

P. Ruiz, T. Poibeau, F. Mélanie. (2015). Entity Linking with

corpus coherence combining open source annotators.

In Proc. NAACL-HLT Demos

P. Ruiz, T. Poibeau. (2015). Combining Open Source

Annotators for Entity Linking through Weighted Voting.

In Proc. of *SEM. Denver, U.S..

67

Some references (2)

Satoshi Sekine, Kiyoshi Sudo and Chikashi Nobata. (2002).

Extended Named Entity Hierarchy. In Proc. LREC.

Eric F. Tjong Kim Sang and Fien De Meulder. (2003).

Introduction to the CoNLL-2003 Shared Task:

Language-Independent Named Entity Recognition. In

Proc. CoNLL. (ACL)

Ricardo Usbeck et al. (2015). GERBIL – General Entity

Annotator Benchmarking Framework. In Proc. of

WWW.

Venturini, Tommaso, and Daniele Guido. 2012. Once Upon a

Text: An ANT Tale in Text Analysis. Sociologica 6 (3).

[Note: ANT=Actor-Network Theory]

Venturini, T., N. Baya Laffite, J.-P. Cointet, I. Gray, V.

Zabban, and K. De Pryck. 2014. Three Maps and Three

Misunderstandings: A Digital Mapping of Climate

Diplomacy. Big Data & Society 1 (2).

… plus several references to work by IXA

68

Thank you !

pablo.ruiz.fabo@ens.fr http://www.lattice.cnrs.fr/Pablo-Ruiz-Fabo,541

Supplemental Slides

70

Cooccurrence methods:

Wordfish

71

Wikipedia-Link-Based Relatedness

72

Witten, I., and David Milne. 2008. “An Effective, Low-Cost Measure of Semantic Relatedness Obtained from Wikipedia Links.”

Coherence: other examples

73

Thomas and Mario are strikers playing in

Munich (Moro and Navigli, 2014)