Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and ComputationalNatural Language Learning, pp. 122–131, Prague, June 2007. c©2007 Association for Computational Linguistics

Characterizing the Errors of Data-Driven Dependency Parsing Models

Ryan McDonaldGoogle Inc.

76 Ninth AvenueNew York, NY 10011

ryanmcd@google.com

Joakim NivreVaxjo University Uppsala University

35195 Vaxjo 75126 UppsalaSweden Swedennivre@msi.vxu.se

Abstract

We present a comparative error analysisof the two dominant approaches in data-driven dependency parsing: global, exhaus-tive, graph-based models, and local, greedy,transition-based models. We show that, inspite of similar performance overall, the twomodels produce different types of errors, ina way that can be explained by theoreticalproperties of the two models. This analysisleads to new directions for parser develop-ment.

1 Introduction

Syntactic dependency representations have a longhistory in descriptive and theoretical linguistics andmany formal models have been advanced (Hudson,1984; Mel’cuk, 1988; Sgall et al., 1986; Maruyama,1990). A dependency graph of a sentence repre-sents each word and its syntactic modifiers throughlabeled directed arcs, as shown in Figure 1, takenfrom the Prague Dependency Treebank (Bohmova etal., 2003). A primary advantage of dependency rep-resentations is that they have a natural mechanismfor representing discontinuous constructions, aris-ing from long distance dependencies or free wordorder, through non-projective dependency arcs, ex-emplified by the arc from jedna to Z in Figure 1.

Syntactic dependency graphs have recentlygained a wide interest in the computational lin-guistics community and have been successfully em-ployed for many problems ranging from machinetranslation (Ding and Palmer, 2004) to ontology

Figure 1: Example dependency graph.

construction (Snow et al., 2004). In this work wefocus on a common parsing paradigm called data-driven dependency parsing. Unlike grammar-basedparsing, data-driven approaches learn to produce de-pendency graphs for sentences solely from an anno-tated corpus. The advantage of such models is thatthey are easily ported to any domain or language inwhich annotated resources exist.

As evident from the CoNLL-X shared task on de-pendency parsing (Buchholz and Marsi, 2006), thereare currently two dominant models for data-drivendependency parsing. The first is what Buchholz andMarsi (2006) call the “all-pairs” approach, where ev-ery possible arc is considered in the construction ofthe optimal parse. The second is the “stepwise” ap-proach, where the optimal parse is built stepwise andwhere the subset of possible arcs considered dependon previous decisions. Theoretically, these modelsare extremely different. The all-pairs models areglobally trained, use exact (or near exact) inferencealgorithms, and define features over a limited historyof parsing decisions. The stepwise models use localtraining and greedy inference algorithms, but definefeatures over a rich history of parse decisions. How-ever, both models obtain similar parsing accuracies

122

McDonald NivreArabic 66.91 66.71

Bulgarian 87.57 87.41Chinese 85.90 86.92

Czech 80.18 78.42Danish 84.79 84.77Dutch 79.19 78.59

German 87.34 85.82Japanese 90.71 91.65

Portuguese 86.82 87.60Slovene 73.44 70.30Spanish 82.25 81.29Swedish 82.55 84.58Turkish 63.19 65.68Overall 80.83 80.75

Table 1: Labeled parsing accuracy for top scoringsystems at CoNLL-X (Buchholz and Marsi, 2006).

on a variety of languages, as seen in Table 1, whichshows results for the two top performing systems inthe CoNLL-X shared task, McDonald et al. (2006)(“all-pairs”) and Nivre et al. (2006) (“stepwise”).

Despite the similar performance in terms of over-all accuracy, there are indications that the two typesof models exhibit different behaviour. For example,Sagae and Lavie (2006) displayed that combiningthe predictions of both parsing models can lead tosignificantly improved accuracies. In order to pavethe way for new and better methods, a much moredetailed error analysis is needed to understand thestrengths and weaknesses of different approaches.In this work we set out to do just that, focusing onthe two top performing systems from the CoNLL-Xshared task as representatives of the two dominantmodels in data-driven dependency parsing.

2 Two Models for Dependency Parsing

2.1 Preliminaries

Let L = {l1, . . . , l|L|} be a set of permissible arclabels. Let x = w0, w1, . . . , wn be an input sen-tence where w0=root. Formally, a dependency graphfor an input sentence x is a labeled directed graphG = (V,A) consisting of a set of nodes V and aset of labeled directed arcs A ⊆ V × V × L, i.e., if(i, j, l) ∈ A for i, j ∈ V and l ∈ L, then there is an

arc from node i to node j with label l in the graph.A dependency graph G for sentence x must satisfythe following properties:

1. V = {0, 1, . . . , n}

2. If (i, j, l) ∈ A, then j 6= 0.

3. If (i, j, l) ∈ A, then for all i′ ∈ V − {i} andl′ ∈ L, (i′, j, l′) /∈ A.

4. For all j ∈ V −{0}, there is a (possibly empty)sequence of nodes i1, . . . , im∈V and labelsl1, . . . , lm, l∈L such that (0, i1, l1),(i1, i2, l2),. . . , (im, j, l)∈A.

The constraints state that the dependency graphspans the entire input (1); that the node 0 is a root(2); that each node has at most one incoming arcin the graph (3); and that the graph is connectedthrough directed paths from the node 0 to every othernode in the graph (4). A dependency graph satisfy-ing these constraints is a directed tree originating outof the root node 0. We say that an arc (i, j, l) is non-projective if not all words k occurring between i andj in the linear order are dominated by i (where dom-inance is the transitive closure of the arc relation).

2.2 Global, Exhaustive, Graph-Based ParsingFor an input sentence, x = w0, w1, . . . , wn considerthe dense graph Gx = (Vx, Ax) where:

1. Vx = {0, 1, . . . , n}2. Ax = {(i, j, l) | ∀ i, j ∈ Vx and l ∈ L}

Let D(Gx) represent the subgraphs of graph Gx

that are valid dependency graphs for the sentencex. Since Gx contains all possible labeled arcs, theset D(Gx) must necessarily contain all valid depen-dency graphs for x.

Assume that there exists a dependency arc scoringfunction, s : V × V × L → R. Furthermore, definethe score of a graph as the sum of its arc scores,

s(G = (V,A)) =∑

(i,j,l)∈A

s(i, j, l)

The score of a dependency arc, s(i, j, l) representsthe likelihood of creating a dependency from wordwi to word wj with the label l. If the arc score func-tion is known a priori, then the parsing problem canbe stated as,

123

G = arg maxG∈D(Gx)

s(G) = arg maxG∈D(Gx)

∑(i,j,l)∈A

s(i, j, l)

This problem is equivalent to finding the highestscoring directed spanning tree in the graph Gx origi-nating out of the root node 0, which can be solved forboth the labeled and unlabeled case in O(n2) time(McDonald et al., 2005b). In this approach, non-projective arcs are produced naturally through theinference algorithm that searches over all possibledirected trees, whether projective or not.

The parsing models of McDonald work primarilyin this framework. To learn arc scores, these mod-els use large-margin structured learning algorithms(McDonald et al., 2005a), which optimize the pa-rameters of the model to maximize the score mar-gin between the correct dependency graph and allincorrect dependency graphs for every sentence in atraining set. The learning procedure is global sincemodel parameters are set relative to the classificationof the entire dependency graph, and not just over sin-gle arc attachment decisions. The primary disadvan-tage of these models is that the feature representa-tion is restricted to a limited number of graph arcs.This restriction is required so that both inference andlearning are tractable.

The specific model studied in this work is thatpresented by McDonald et al. (2006), which factorsscores over pairs of arcs (instead of just single arcs)and uses near exhaustive search for unlabeled pars-ing coupled with a separate classifier to label eacharc. We call this system MSTParser, which is alsothe name of the freely available implementation.1

2.3 Local, Greedy, Transition-Based Parsing

A transition system for dependency parsing defines

1. a set C of parser configurations, each of whichdefines a (partially built) dependency graph G

2. a set T of transitions, each a function t :C→C

3. for every sentence x = w0, w1, . . . , wn,

(a) a unique initial configuration cx

(b) a set Cx of terminal configurations

1http://mstparser.sourceforge.net

A transition sequence Cx,m = (cx, c1, . . . , cm) for asentence x is a sequence of configurations such thatcm ∈ Cx and, for every ci (ci 6= cx), there is a tran-sition t ∈ T such that ci = t(ci−1). The dependencygraph assigned to x by Cx,m is the graph Gm definedby the terminal configuration cm.

Assume that there exists a transition scoring func-tion, s : C × T → R. The score of a transitiont in a configuration c, s(c, t), represents the likeli-hood of taking transition t out of configuration c.The parsing problem consists in finding a terminalconfiguration cm ∈ Cx, starting from the initialconfiguration cx and taking the optimal transitiont∗ = arg maxt∈T s(c, t) out of every configurationc. This can be seen as a greedy search for the optimaldependency graph, based on a sequence of locallyoptimal decisions in terms of the transition system.

Many transition systems for data-driven depen-dency parsing are inspired by shift-reduce parsing,where configurations contain a stack for storing par-tially processed nodes. Transitions in such systemsadd arcs to the dependency graph and/or manipu-late the stack. One example is the transition systemdefined by Nivre (2003), which parses a sentencex = w0, w1, . . . , wn in O(n) time, producing a pro-jective dependency graph satisfying conditions 1–4in section 2.1, possibly after adding arcs (0, i, lr)for every node i 6= 0 that is a root in the outputgraph (where lr is a special label for root modifiers).Nivre and Nilsson (2005) showed how the restric-tion to projective dependency graphs could be liftedby using graph transformation techniques to pre-process training data and post-process parser output,so-called pseudo-projective parsing.

To learn transition scores, these systems use dis-criminative learning methods, e.g., memory-basedlearning or support vector machines. The learningprocedure is local since only single transitions arescored, not entire transition sequences. The primaryadvantage of these models is that features are not re-stricted to a limited number of graph arcs but cantake into account the entire dependency graph builtso far. The main disadvantage is that the greedyparsing strategy may lead to error propagation.

The specific model studied in this work is that pre-sented by Nivre et al. (2006), which uses labeledpseudo-projective parsing with support vector ma-chines. We call this system MaltParser, which is also

124

the name of the freely available implementation.2

2.4 Comparison

These models differ primarily with respect to threeimportant properties.

1. Inference: MaltParser uses a transition-basedinference algorithm that greedily chooses thebest parsing decision based on a trained clas-sifier and current parser history. MSTParserinstead uses near exhaustive search over adense graphical representation of the sentenceto find the dependency graph that maximizesthe score.

2. Training: MaltParser trains a model to makea single classification decision (choose the nexttransition). MSTParser trains a model to maxi-mize the global score of correct graphs.

3. Feature Representation: MaltParser can in-troduce a rich feature history based on previ-ous parser decisions. MSTParser is forced torestrict the score of features to a single or pairof nearby parsing decisions in order to makeexhaustive inference tractable.

These differences highlight an inherent trade-off be-tween exhaustive inference algorithms plus globallearning and expressiveness of feature representa-tions. MSTParser favors the former at the expenseof the latter and MaltParser the opposite.

3 The CoNLL-X Shared Task

The CoNLL-X shared task (Buchholz and Marsi,2006) was a large-scale evaluation of data-driven de-pendency parsers, with data from 13 different lan-guages and 19 participating systems. The officialevaluation metric was the labeled attachment score(LAS), defined as the percentage of tokens, exclud-ing punctuation, that are assigned both the correcthead and the correct dependency label.3

The output of all systems that participated in theshared task are available for download and consti-tute a rich resource for comparative error analysis.

2http://w3.msi.vxu.se/users/nivre/research/MaltParser.html3In addition, results were reported for unlabeled attachment

score (UAS) (tokens with the correct head) and label accuracy(LA) (tokens with the correct label).

The data used in the experiments below are the out-puts of MSTParser and MaltParser for all 13 lan-guages, together with the corresponding gold stan-dard graphs used in the evaluation. We constructedthe data by simply concatenating a system’s outputfor every language. This resulted in a single out-put file for each system and a corresponding singlegold standard file. This method is sound because thedata sets for each language contain approximatelythe same number of tokens – 5,000. Thus, evalu-ating system performance over the aggregated filescan be roughly viewed as measuring system perfor-mance through an equally weighted arithmetic meanover the languages.

It could be argued that a language by languagecomparison would be more appropriate than com-paring system performance across all languages.However, as table Table 1 shows, the difference inaccuracy between the two systems is typically smallfor all languages, and only in a few cases is thisdifference significant. Furthermore, by aggregatingover all languages we gain better statistical estimatesof parser errors, since the data set for each individuallanguage is very small.

4 Error Analysis

The primary purpose of this study is to characterizethe errors made by standard data-driven dependencyparsing models. To that end, we present a large set ofexperiments that relate parsing errors to a set of lin-guistic and structural properties of the input and pre-dicted/gold standard dependency graphs. We arguethat the results can be correlated to specific theoreti-cal aspects of each model – in particular the trade-offhighlighted in Section 2.4.

For simplicity, all experiments report labeledparsing accuracies. Identical experiments using un-labeled parsing accuracies did not reveal any addi-tional information. Furthermore, all experiments arebased on the data from all 13 languages together, asexplained in section 3.

4.1 Length Factors

It is well known that parsing systems tend to havelower accuracies for longer sentences. Figure 2shows the accuracy of both parsing models relativeto sentence length (in bins of size 10: 1–10, 11–20,

125

10 20 30 40 50 50+Sentence Length (bins of size 10)

0.7

0.72

0.74

0.76

0.78

0.8

0.82

0.84

Depe

nden

cy A

ccur

acy MSTParser

MaltParser

Figure 2: Accuracy relative to sentence length.

etc.). System performance is almost indistinguish-able. However, MaltParser tends to perform betteron shorter sentences, which require the greedy in-ference algorithm to make less parsing decisions. Asa result, the chance of error propagation is reducedsignificantly when parsing these sentences. The factthat MaltParser has a higher accuracy (rather thanthe same accuracy) when the likelihood of errorpropagation is reduced comes from its richer featurerepresentation.

Another interesting property is accuracy relativeto dependency length. The length of a dependencyfrom word wi to word wj is simply equal to |i− j|.Longer dependencies typically represent modifiersof the root or the main verb in a sentence. Shorterdependencies are often modifiers of nouns such asdeterminers or adjectives or pronouns modifyingtheir direct neighbours. Figure 3 measures the pre-cision and recall for each system relative to depen-dency lengths in the predicted and gold standard de-pendency graphs. Precision represents the percent-age of predicted arcs of length d that were correct.Recall measures the percentage of gold standard arcsof length d that were correctly predicted.

Here we begin to see separation between the twosystems. MSTParser is far more precise for longerdependency arcs, whereas MaltParser does betterfor shorter dependency arcs. This behaviour canbe explained using the same reasoning as above:shorter arcs are created before longer arcs in thegreedy parsing procedure of MaltParser and are lessprone to error propagation. Theoretically, MST-Parser should not perform better or worse for edgesof any length, which appears to be the case. Thereis still a slight degradation, but this can be attributedto long dependencies occurring more frequently inconstructions with possible ambiguity. Note that

even though the area under the curve is much largerfor MSTParser, the number of dependency arcs witha length greater than ten is much smaller than thenumber with length less than ten, which is why theoverall accuracy of each system is nearly identical.For all properties considered here, bin size generallyshrinks in size as the value on the x-axis increases.

4.2 Graph Factors

The structure of the predicted and gold standard de-pendency graphs can also provide insight into thedifferences between each model. For example, mea-suring accuracy for arcs relative to their distance tothe artificial root node will detail errors at differentlevels of the dependency graph. For a given arc, wedefine this distance as the number of arcs in the re-verse path from the modifier of the arc to the root.Figure 4 plots the precision and recall of each sys-tem for arcs of varying distance to the root. Preci-sion is equal to the percentage of dependency arcs inthe predicted graph that are at a distance of d and arecorrect. Recall is the percentage of dependency arcsin the gold standard graph that are at a distance of dand were predicted.

Figure 4 clearly shows that for arcs close to theroot, MSTParser is much more precise than Malt-Parser, and vice-versa for arcs further away from theroot. This is probably the most compelling graphgiven in this study since it reveals a clear distinction:MSTParser’s precision degrades as the distance tothe root increases whereas MaltParser’s precision in-creases. The plots essentially run in opposite direc-tions crossing near the middle. Dependency arcs fur-ther away from the root are usually constructed earlyin the parsing algorithm of MaltParser. Again a re-duced likelihood of error propagation coupled witha rich feature representation benefits that parser sub-stantially. Furthermore, MaltParser tends to over-predict root modifiers, because all words that theparser fails to attach as modifiers are automaticallyconnected to the root, as explained in section 2.3.Hence, low precision for root modifiers (without acorresponding drop in recall) is an indication that thetransition-based parser produces fragmented parses.

The behaviour of MSTParser is a little trickier toexplain. One would expect that its errors should bedistributed evenly over the graph. For the most partthis is true, with the exception of spikes at the ends

126

0 5 10 15 20 25 30Dependency Length

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Depe

nden

cy P

recis

ion MSTParser

MaltParser

0 5 10 15 20 25 30Dependency Length

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Depe

nden

cy R

ecal

l MSTParserMaltParser

Figure 3: Dependency arc precision/recall relative to predicted/gold dependency length.

of the plot. The high performance for root modifica-tion (distance of 1) can be explained through the factthat this is typically a low entropy decision – usu-ally the parsing algorithm has to determine the mainverb from a small set of possibilities. On the otherend of the plot there is a sharp downwards spike forarcs of distance greater than 10. It turns out thatMSTParser over-predicts arcs near the bottom of thegraph. Whereas MaltParser pushes difficult parsingdecisions higher in the graph, MSTParser appears topush these decisions lower.

The next graph property we will examine aims toquantify the local neighbourhood of an arc withina dependency graph. Two dependency arcs, (i, j, l)and (i′, j′, l′) are classified as siblings if they repre-sent syntactic modifications of the same word, i.e.,i = i′. Figure 5 measures the precision and recallof each system relative to the number of predictedand gold standard siblings of each arc. There isnot much to distinguish between the parsers on thismetric. MSTParser is slightly more precise for arcsthat are predicted with more siblings, whereas Malt-Parser has slightly higher recall on arcs that havemore siblings in the gold standard tree. Arcs closerto the root tend to have more siblings, which ties thisresult to the previous ones.

The final graph property we wish to look at is thedegree of non-projectivity. The degree of a depen-dency arc from word w to word u is defined hereas the number of words occurring between w and uthat are not descendants of w and modify a word thatdoes not occur between w and u (Nivre, 2006). Inthe example from Figure 1, the arc from jedna to Zhas a degree of one, and all other arcs have a degreeof zero. Figure 6 plots dependency arc precision andrecall relative to arc degree in predicted and goldstandard dependency graphs. MSTParser is more

precise when predicting arcs with high degree andMaltParser vice-versa. Again, this can be explainedby the fact that there is a tight correlation between ahigh degree of non-projectivity, dependency length,distance to root and number of siblings.

4.3 Linguistic Factors

It is important to relate each system’s accuracy to aset of linguistic categories, such as parts of speechand dependency types. Therefore, we have madean attempt to distinguish a few broad categoriesthat are cross-linguistically identifiable, based on theavailable documentation of the treebanks used in theshared task.

For parts of speech, we distinguish verbs (includ-ing both main verbs and auxiliaries), nouns (includ-ing proper names), pronouns (sometimes also in-cluding determiners), adjectives, adverbs, adposi-tions (prepositions, postpositions), and conjunctions(both coordinating and subordinating). For depen-dency types, we distinguish a general root category(for labels used on arcs from the artificial root, in-cluding either a generic label or the label assignedto predicates of main clauses, which are normallyverbs), a subject category, an object category (in-cluding both direct and indirect objects), and variouscategories related to coordination.

Figure 7 shows the accuracy of the two parsersfor different parts of speech. This figure measureslabeled dependency accuracy relative to the part ofspeech of the modifier word in a dependency rela-tion. We see that MaltParser has slightly better ac-curacy for nouns and pronouns, while MSTParserdoes better on all other categories, in particular con-junctions. This pattern is consistent with previousresults insofar as verbs and conjunctions are ofteninvolved in dependencies closer to the root that span

127

2 4 6 8 10Distance to Root

0.74

0.76

0.78

0.8

0.82

0.84

0.86

0.88

0.9

Depe

nden

cy P

recis

ion MSTParser

MaltParser

2 4 6 8 10Distance to Root

0.76

0.78

0.8

0.82

0.84

0.86

0.88

Depe

nden

cy R

ecal

l MSTParserMaltParser

Figure 4: Dependency arc precision/recall relative to predicted/gold distance to root.

0 2 4 6 8 10+Number of Modifier Siblings

0.5

0.6

0.7

0.8

0.9

Depe

nden

cy P

recis

ion MSTParser

MaltParser

0 2 4 6 8 10+Number of Modifier Siblings

0.5

0.6

0.7

0.8

0.9

Depe

nden

cy R

ecal

l MSTParserMaltParser

Figure 5: Dependency arc precision/recall relative to number of predicted/gold siblings.

longer distances, while nouns and pronouns are typ-ically attached to verbs and therefore occur lower inthe graph, with shorter distances. Empirically, ad-verbs resemble verbs and conjunctions with respectto root distance but group with nouns and pronounsfor dependency length, so the former appears to bemore important. In addition, both conjunctions andadverbs tend to have a high number of siblings, mak-ing the results consistent with the graph in Figure 5.

Adpositions and especially adjectives constitutea puzzle, having both high average root distanceand low average dependency length. Adpositions dotend to have a high number of siblings on average,which could explain MSTParser’s performance onthat category. However, adjectives on average occurthe furthest away from the root, have the shortestdependency length and the fewest siblings. As such,we do not have an explanation for this behaviour.

In the top half of Figure 8, we consider precisionand recall for dependents of the root node (mostlyverbal predicates), and for subjects and objects. Asalready noted, MSTParser has considerably betterprecision (and slightly better recall) for the root cat-egory, but MaltParser has an advantage for the nomi-nal categories, especially subjects. A possible expla-nation for the latter result, in addition to the length-based and graph-based factors invoked before, is that

60.0%

65.0%

70.0%

75.0%

80.0%

85.0%

90.0%

95.0%

Verb Noun Pron Adj Adv Adpos Conj

Part of Speech (POS)

Labe

led

Atta

chm

ent S

core

(LAS

)

MSTParserMaltParser

Figure 7: Accuracy for different parts of speech.

MaltParser integrates labeling into the parsing pro-cess, so that previously assigned dependency labelscan be used as features, which may be important todisambiguate subjects and objects.

Finally, in the bottom half of Figure 8, we dis-play precision and recall for coordinate structures,divided into different groups depending on the typeof analysis adopted in a particular treebank. The cat-egory CCH (coordinating conjunction as head) con-tains conjunctions analyzed as heads of coordinatestructures, with a special dependency label that doesnot describe the function of the coordinate structurein the larger syntactic structure, a type of categoryfound in the so-called Prague style analysis of coor-dination and used in the data sets for Arabic, Czech,

128

0 1 2 3 4 5 6 7+Non-Projective Arc Degree

0.55

0.6

0.65

0.7

0.75

0.8

0.85De

pend

ency

Pre

cisio

n MSTParserMaltParser

0 1 2 3 4 5 6 7+Non-Projective Arc Degree

0.6

0.65

0.7

0.75

0.8

0.85

Depe

nden

cy R

ecal

l MSTParserMaltParser

Figure 6: Dependency arc precision/recall relative to predicted/gold degree of non-projectivity.

65.0%

70.0%

75.0%

80.0%

85.0%

90.0%

95.0%

Root Subj Obj

Dependency Type (DEP)

Depe

nden

cy P

recis

ion

MSTParserMaltParser

72.0%

74.0%

76.0%

78.0%

80.0%

82.0%

84.0%

86.0%

88.0%

90.0%

Root Subj Obj

Dependency Type (DEP)

Depe

nden

cy R

ecal

l

MSTParserMaltParser

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

CCH CCD CJCC CJCJ

Dependency Type (DEP)

Depe

nden

cy P

recis

ion

MSTParserMaltParser

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

CCH CCD CJCC CJCJ

Dependency Tyle (DEP)

Depe

nden

cy R

ecal

l

MSTParserMaltParser

Figure 8: Precision/recall for different dependency types.

and Slovene. The category CCD (coordinating con-junction as dependent) instead denotes conjunctionsthat are attached as dependents of one of the con-juncts with a label that only marks them as conjunc-tions, a type of category found in the data sets forBulgarian, Danish, German, Portuguese, Swedishand Turkish. The two remaining categories con-tain conjuncts that are assigned a dependency labelthat only marks them as conjuncts and that are at-tached either to the conjunction (CJCC) or to an-other conjunct (CJCJ). The former is found in Bul-garian, Danish, and German; the latter only in Por-tuguese and Swedish. For most of the coordinationcategories there is little or no difference between thetwo parsers, but for CCH there is a difference in bothprecision and recall of almost 20 percentage pointsto MSTParser’s advantage. This can be explained by

noting that, while the categories CCD, CJCC, andCJCJ denote relations that are internal to the coor-dinate structure and therefore tend to be local, theCCH relations hold between the coordinate struc-ture and its head, which is often a relation that spansover a greater distance and is nearer the root of thedependency graph. It is likely that the difference inaccuracy for this type of dependency accounts for alarge part of the difference in accuracy noted earlierfor conjunctions as a part of speech.

4.4 Discussion

The experiments from the previous section highlightthe fundamental trade-off between global trainingand exhaustive inference on the one hand and ex-pressive feature representations on the other. Errorpropagation is an issue for MaltParser, which typi-

129

cally performs worse on long sentences, long depen-dency arcs and arcs higher in the graphs. But this isoffset by the rich feature representation available tothese models that result in better decisions for fre-quently occurring arc types like short dependenciesor subjects and objects. The errors for MSTParserare spread a little more evenly. This is expected,as the inference algorithm and feature representationshould not prefer one type of arc over another.

What has been learned? It was already known thatthe two systems make different errors through thework of Sagae and Lavie (2006). However, in thatwork an arc-based voting scheme was used that tookonly limited account of the properties of the wordsconnected by a dependency arc (more precisely, theoverall accuracy of each parser for the part of speechof the dependent). The analysis in this work not onlyshows that the errors made by each system are dif-ferent, but that they are different in a way that can bepredicted and quantified. This is an important stepin parser development.

To get some upper bounds of the improvementthat can be obtained by combining the strengths ofeach models, we have performed two oracle experi-ments. Given the output of the two systems, we canenvision an oracle that can optimally choose whichsingle parse or combination of sub-parses to predictas a final parse. For the first experiment the oracleis provided with the single best parse from each sys-tem, say G = (V,A) and G′ = (V ′, A′). The oraclechooses a parse that has the highest number of cor-rectly predicted labeled dependency attachments. Inthis situation, the oracle accuracy is 84.5%. In thesecond experiment the oracle chooses the tree thatmaximizes the number of correctly predicted depen-dency attachments, subject to the restriction that thetree must only contain arcs from A ∪ A′. This canbe computed by setting the weight of an arc to 1 ifit is in the correct parse and in the set A ∪ A′. Allother arc weights are set to negative infinity. One canthen simply find the tree that has maximal sum ofarc weights using directed spanning tree algorithms.This technique is similar to the parser voting meth-ods used by Sagae and Lavie (2006). In this situa-tion, the oracle accuracy is 86.9%.

In both cases we see a clear increase in accuracy:86.9% and 84.5% relative to 81% for the individualsystems. This indicates that there is still potential

for improvement, just by combining the two existingmodels. More interestingly, however, we can usethe analysis to get ideas for new models. Below wesketch some possible new directions:

1. Ensemble systems: The error analysis pre-sented in this paper could be used as inspirationfor more refined weighting schemes for ensem-ble systems of the kind proposed by Sagae andLavie (2006), making the weights depend on arange of linguistic and graph-based factors.

2. Hybrid systems: Rather than using an ensem-ble of several parsers, we may construct a sin-gle system integrating the strengths of eachparser described here. This could defer toa greedy inference strategy during the earlystages of the parse in order to benefit from arich feature representation, but then default toa global exhaustive model as the likelihood forerror propagation increases.

3. Novel approaches: The two approaches inves-tigated are each based on a particular combina-tion of training and inference methods. We maynaturally ask what other combinations mayprove fruitful. For example, what about glob-ally trained, greedy, transition-based models?This is essentially what Daume III et al. (2006)provide, in the form of a general search-basedstructured learning framework that can be di-rectly applied to dependency parsing. The ad-vantage of this method is that the learning canset model parameters relative to errors resultingdirectly from the search strategy – such as errorpropagation due to greedy search. When com-bined with MaltParser’s rich feature represen-tation, this could lead to significant improve-ments in performance.

5 Conclusion

We have presented a thorough study of the dif-ference in errors made between global exhaustivegraph-based parsing systems (MSTParser) and lo-cal greedy transition-based parsing systems (Malt-Parser). We have shown that these differences canbe quantified and tied to theoretical expectations ofeach model, which may provide insights leading tobetter models in the future.

130

ReferencesA. Bohmova, J. Hajic, E. Hajicova, and B. Hladka.

2003. The PDT: A 3-level annotation scenario. InA. Abeille, editor, Treebanks: Building and UsingParsed Corpora, chapter 7. Kluwer Academic Publish-ers.

S. Buchholz and E. Marsi. 2006. CoNLL-X shared taskon multilingual dependency parsing. In Proc. CoNLL.

Hal Daume III, John Langford, and Daniel Marcu. 2006.Search-based structured prediction. In Submission.

Y. Ding and M. Palmer. 2004. Synchronous dependencyinsertion grammars: A grammar formalism for syntaxbased statistical MT. In Workshop on Recent Advancesin Dependency Grammars (COLING).

R. Hudson. 1984. Word Grammar. Blackwell.

H. Maruyama. 1990. Structural disambiguation withconstraint propagation. In Proc. ACL.

R. McDonald, K. Crammer, and F. Pereira. 2005a. On-line large-margin training of dependency parsers. InProc. ACL.

R. McDonald, F. Pereira, K. Ribarov, and J. Hajic. 2005b.Non-projective dependency parsing using spanningtree algorithms. In Proc. HLT/EMNLP.

R. McDonald, K. Lerman, and F. Pereira. 2006. Multi-lingual dependency analysis with a two-stage discrim-inative parser. In Proc. CoNLL.

I.A. Mel’cuk. 1988. Dependency Syntax: Theory andPractice. State University of New York Press.

J. Nivre and J. Nilsson. 2005. Pseudo-projective depen-dency parsing. In Proc. ACL.

J. Nivre, J. Hall, J. Nilsson, G. Eryigit, and S. Marinov.2006. Labeled pseudo-projective dependency parsingwith support vector machines. In Proc. CoNLL.

J. Nivre. 2003. An efficient algorithm for projective de-pendency parsing. In Proc. IWPT.

J. Nivre. 2006. Constraints on non-projective depen-dency parsing. In Proc. EACL.

K. Sagae and A. Lavie. 2006. Parser combination byreparsing. In Proc. HLT/NAACL.

P. Sgall, E. Hajicova, and J. Panevova. 1986. The Mean-ing of the Sentence in Its Pragmatic Aspects. Reidel.

R. Snow, D. Jurafsky, and A. Y. Ng. 2004. Learningsyntactic patterns for automatic hypernym discovery.In Proc. NIPS.

131