Proceedings of LAW X – The 10th Linguistic Annotation Workshop, pages 188–198,Berlin, Germany, August 11, 2016. c©2016 Association for Computational Linguistics

Creating a Novel Geolocation Corpus from Historical Texts

Grant DeLozier?∗, Ben Wing?∗, Jason Baldridge∗, Scott Nesbit†University of Texas at Austin∗

University of Georgia†

grantdelozier@gmail.com, ben@benwing.comjasonbaldridge@gmail.com, snesbit@uga.edu

? The first two authors contributed equally to the content of the paper.

Abstract

This paper describes the process of an-notating a historical US civil war corpuswith geographic reference. Reference an-notations are given at two different textualscales: individual place names and docu-ments. This is the first published corpusof its kind in document-level geolocation,and it has over 10,000 disambiguated to-ponyms, double the amount of any priortoponym corpus. We outline many chal-lenges and considerations in creating sucha corpus, and we evaluate baseline andbenchmark toponym resolution and docu-ment geolocation systems on it. Aspects ofthe corpus suggest several recommenda-tions for proper annotation procedure forthe tasks.

1 Introduction

Geographic information is an important compo-nent of a number of areas including informationretrieval (Daoud and Huang, 2013), social me-dia analysis, and historical research (Nesbit, 2013;Grover et al., 2010; Smith and Crane, 2001). Todate however, very few corpora exist for text ge-olocation tasks, and those which do exist haveflaws or are very small in size. This is particu-larly true for tasks seeking to do geolocation workwith historical texts. In the realm of document ge-olocation, there exist no historical corpora what-soever; in the realm of toponym resolution histor-ical corpora exist, but are flawed in important re-spects (Speriosu and Baldridge, 2013; DeLozier etal., 2015).

This paper describes the process of annotatinga set of American Civil War archives commonlyknown as the Official Records of the War of theRebellion (officially titled The War of the Rebel-

Docgeo subset Topo subset Full dataTotal tokens 1,743,331 447,703 57,557,037# volumes 118 15 126# documents 7,533 1,644 254,744Avg. tokens/document 231.43 272.32 225.94

Table 1: Statistics on WOTR, annotated subsetand full data (using documents predicted based ona sequence model derived from the annotated data,as described in §3).

lion: a Compilation of the Official Records of theUnion and Confederate Armies and henceforth ab-breviated as WOTR), arguably the most importantand widely used corpus in this area of historicalstudy1.

Document geolocation and toponym resolutionenable work on the specific content of individualdocuments and themes contained within this cor-pus, revealing the ways in which content is dis-tributed in the corpus over time and space (Ayersand Nesbit, 2011; Thomas III, 2011). Themes inthis corpus pertinent to the study of Civil War lit-erature include the rise of irregular warfare, theend of slavery in Confederate and Union states, theuse of railroads by United States and Confederatearmies in the war, and the destruction of the war-making capacity of the Confederate states. Theannotation process and geolocation tools also en-able historians to reexamine the process by whichthe archive was produced, an area which has re-cently seen growing interest (Sternhell, 2016).

We develop geolocation corpora for two relatedbut separate tasks: document geolocation (doc-geo) and toponym resolution (TR). Statistics onthe full WOTR corpus and the annotated docu-ment geolocation and toponym subsets are shownin Table 1 and Table 2.

Geographic summaries of the annotations aregiven in Figure 1 (documents) and Figure 2 (to-ponyms). The docgeo annotations are concen-

1http://ehistory.osu.edu/books/official-records

188

Docgeo SubsetDocuments 8,121Documents with geometries 5,035 (62%)Documents with only points 4,811 (59%)Documents with polygons 224 (3%)

Topo SubsetAvg. toponyms/document 7.17Toponyms 11,795Toponyms with geometries 10,380 (88%)Toponyms with points 8,130 (69%)Toponyms with polygons 2,296 (19%)People 7,994Organizations 2,591

Table 2: Statistics on WOTR, annotated subset(using documents predicted based on a sequencemodel derived from the annotated data, as de-scribed in §3).

trated in a number of areas that saw heavy fighting,such as in Virginia, South Carolina and NorthernGeorgia. The toponym annotations are more con-centrated around the western theater of the CivilWar. In both corpora, almost all US states arerepresented by at least some references. The to-ponym annotations contain more full-state poly-gons, while the docgeo annotations are primarilypoints, leading to the differing appearances of thetwo maps.

2 Geolocation tasks

Both toponym resolution and document geoloca-tion involve assigning geographic reference, usu-ally latitude-longitude coordinates, to spans oftext, but differ as to the size of the span. Toponymresolution involves assigning such reference to in-dividual, potentially ambiguous toponyms (e.g.Springfield or Dallas), while document geoloca-tion assigns geographic reference to larger spansof text (documents, broadly construed).

Among the key difficulties associated with bothtasks are ambiguity of reference, fluidity in thedefinition of the tasks, and lack of sufficientand/or appropriate training material. As an exam-ple of the issues surrounding ambiguity, considerthe toponym Springfield. Dominant place namegazetteers indicate at least 236 unique senses ofthe term (and these underestimate the true total),with possible references spanning the globe. TRsystems must choose referents in these highly am-biguous scenarios, even when correct referents arenot listed in gazetteers. In document geolocation,

the problem is even more acute, as a document canpotentially be assigned a location anywhere on theglobe.

Another issue affecting both domains is fluidityin how one defines the task itself. In toponym res-olution, metonymy—the ability of a place name torefer to something closely related to a place (e.g.a government)—and demonymy—names for thepeople who inhabit an area (e.g. Americans)—are properties that must be considered. All exist-ing TR corpora include metonymic uses of placenames. The Local Global Lexicon (LGL) corpus(Lieberman and Samet, 2012) includes demonymsas toponyms and georeferences them, while allother corpora do not. An additional issue pertainsto the range of entity types a system is expected toresolve. Many corpora limit their expectations tolarger entities—e.g. TR-CoNLL (Leidner, 2008)is limited to cities, states, and countries), whileothers focus more on highly local entities (e.g. busstops) (Matsuda et al., 2015). A final issue relatesto whether systems ought to resolve places whichare embedded inside other named entities. For ex-ample, the LGL corpus expects New York in theexpression New York Times to be resolved to thestate of New York. Many of the characteristics ofexisting TR corpora are summarized in Table 3.

In document geolocation, different researchershave interpreted the task differently, dependingon the corpus: typically as either as the loca-tion of the document’s author when the docu-ment was created, or as the geographic theme (i.e.topic) of the content of the document. The for-mer interpretation has usually been used whenworking with social-media corpora such as Twit-ter (Han et al., 2014; Schulz et al., 2013) andFlickr (O’Hare and Murdock, 2013; Bolettieri etal., 2009), and the latter with encyclopedic corporasuch as Wikipedia (van Laere et al., 2014) andhistorical corpora such as the unpublished BeadleCorpus (Wing, 2015). Another difficulty with us-ing the geographic-theme interpretation is that thisreference may not be easily identifiable for sometexts. (For example, only about 10% of the articlesin the English Wikipedia have document-level an-notations assigned to them.)

An additional issue related to the definition ofboth tasks is the scope of the geographic refer-ence. Smaller geographic entities, such as citiesand neighborhoods, can be reasonably approxi-mated as a point in latitude-longitude space, while

189

Figure 1: Distribution of References in WoTR-DocGeo

Figure 2: Distribution of Toponyms in WOTR-Topo

190

it is more difficult to do so for larger entities suchas states or countries. Various solutions have beenused for this problem, depending on the corpus.Wikipedia and most gazetteers take the simplestapproach of assigning a point to all entities, re-gardless of size. However, for large entities suchas countries, this necessitates choosing a singlerepresentative point (e.g. the geographic centroidor the capital city), which leads to many problems(e.g. the geographic centroid of the UK is a pointin the Irish Sea).

Toponym resolution, especially, currently suf-fers from a lack of sufficient training material.Existing training corpora fixate around very nar-row ranges of geographic entities. One major cor-pus used in toponym resolution, TR-CoNLL, hasonly 800 unique strings and 6259 toponyms, whilegazetteers such as GeoNames list over 8 millionunique places (which still greatly underestimatesthe true number of toponyms). Such mismatchesdo more than underscore the need for larger andmore domain-diverse corpora; they point to fun-damental issues associated with learning to re-solve geographies from language. Geographic en-tities, like all named entities, are fiat objects; nam-ing them dictates their existence (Kripke, 1980).Many systems have attempted to alleviate paucityproblems by splicing corpora with latent annota-tions inferred from a more general resource likeWikipedia (Speriosu and Baldridge, 2013; Santoset al., 2014; DeLozier et al., 2015).

In document geolocation, the amount of train-ing material available is crucially tied in with howthe task is defined (as described above). Abun-dant training material is available from the vari-ous language-specific versions of Wikipedia andfrom social-media sites such as Twitter and Flickr,but the variations in language and task definitionmake the corpora highly domain-specific. Thismeans that cross-corpus generalization is fraughtwith difficulty, particularly in domains where nopreviously-published corpora exist, such as his-torical documents. Nonetheless, researchers haveachieved some success from docgeo domain adap-tation, using Wikipedia as out-of-domain train-ing material for historical documents under aco-training setup (Wing, 2015) and Flickr as asource of language-model data for geolocation ofWikipedia (De Rouck et al., 2011).

3 Data preparation

The source data available was in the form of textscanned directly from the published books us-ing OCR (optical character recognition), and thenhand-corrected. The digital form of the collectionwe accessed included page breaks which some-times occur in the middle of a word, footnotes andheaders undifferentiated from body text, and noformal delimiting of where particular records be-gan and ended. Figure 3 is an example of part ofthe source text of a volume in the collection, afterpreprocessing to stitch up page breaks and removefootnotes, headers, footers, etc., but before split-ting into individual documents.

To alleviate some of these issues in workingwith this form of the text, the following steps weretaken to improve our annotated version of the cor-pus:

1. Remove page breaks and stitch up paragraphsdivided across the breaks.

2. Create a GUI annotation tool to allow anno-tators to quickly note the extent of documents(which we term spans) and indicate the doc-ument locations on a map.

3. Create a sequence model to automaticallysplit up the continuous text into docu-ments, training it on the documents manuallymarked up by the annotators.

Stitching up page breaks As mentioned above,the source text is in the form of individual pagesscanned from the published books, with pagebreaks, footnotes, stray headers, etc. often inter-rupting a paragraph in the middle of a word, fre-quently in an inconsistent fashion. A program waswritten that used various heuristics to do the ma-jority of work, although several more steps and agood deal of hand editing were required to achievesatisfactory results.

Automatically locating document spans Thereis no indication in the source text where one doc-ument ends and another one begins. In a letter,for example, sometimes the destinee appears nearthe beginning of the letter, following a headingdescribing the location and date, while in othercases the destinee appears at the very end, after thesalutation. Both examples can be seen in the textbox in the annotation tool screen shot in Figure 4,

191

Table 3: Toponym CorporaCorpus Domain Entity Types Reference

TypesMetonyms Demonyms Nested NE Toponyms

TR-CoNLL ContemporaryInternational News

Cities,States,Countries

Point only Yes No Most En-compassingNE

6259

LGL Contemporary Lo-cal Newspapers

Few Locales, cities,states, countries

Point only Yes Yes AnnotatesEmbeddedPlaces

5088

LRE Tweets from Japan Highly local ’facili-ties’ and above

Point only ? No ? 951

WOTR US Civil War Let-ters + Reports

Locales, Cities, andStates

Point andPolygon

No No Most En-compassingNE

10380

along with the way that successive documents di-rectly abut each other. Because the unit of analy-sis is a single document, it is necessary to locatethe beginning and end of each document, and thismust be done automatically since only a fractionof the text was manually annotated.

To do this, a CRF (conditional random field) se-quence model was created using MALLET (Mc-Callum, 2002). Each successive paragraph wasconsidered a unit in the sequence labeling task,and labeled with one of the following: B (begin-ning), I (inside), L (last), or O (outside), similar tohow named entity recognition (NER) sequence la-beling is normally handled. CRF’s have the advan-tage over HMM’s (hidden Markov models) thatthey can be conditioned on arbitrary features of thevisible stream of paragraphs, including the neigh-bors of the actual paragraph being labeled. This al-lowed for various features to be engineered, suchas (1) the presence of a date at the end of a line,possibly followed by a time; (2) the presence ofcertain place-related terms typically indicating aheader line, such as HEADQUARTERS, HDQRSor FORT; (3) the presence of a rank-indicatingword (e.g. Brigadier, General or Commanding) atthe beginning of or within a line; (4) the presenceof a line beginning with a string of capital letters,typically indicating a header line; (5) the presenceof certain words (e.g. obedient servant) that typi-cally indicate a salutation; (6) the combination ofthe above features with certain punctuation at theend of the line (comma, period, or colon); (7) thelength of a line; (8) all of the above features for theactual paragraph in question as well as the previ-ous, second-previous, next, second-next, and com-binations thereof; and (9) the first and last wordsof the paragraph, after stripping out punctuation.

The resulting model performed well, but did notconsistently handle the cases where the destinee isat the end of the letter, and so a postprocessingstep was added to adjust the spans whenever such

a situation was detected.

4 Annotation process

4.1 Annotation tool

A GUI annotation tool was written that allowsdocument spans to be selected in a text box andpoints or polygons added on a map. Figure 4shows a screen shot of the tool at work. Spans oftext are indicated with inward-pointing red arrowsat their edges and are colored yellow (a markedspan without geometry), green (a span with geom-etry) or cyan (currently selected span for addingor changing the geometry). Points and polygonscan be added by drawing directly on the map,by using the list of recent locations below themap, or (in the case of points) by entering a lat-itude/longitude coordinate into the text box andclicking Set Lat/Long.

The annotation tool is written in HTML andJavaScript using the OpenLayers2 and Rangy li-braries3, with data stored using Parse, a backend-as-a-service which allows for free data storagewithin certain storage and bandwidth limits.

4.2 Document geolocation annotation

The docgeo annotation process took 280 hoursover two months. Five annotators were hired, al-though in practice most of the work was done bya single annotator. 25-page subsections of 118 of126 volumes were annotated with geographies. Afew of the volumes had an additional 75 pages an-notated.

4.2.1 Document annotator guidelinesAnnotators were hired to note the individual doc-uments within the archives and attach document-level geometries to them, which are intended toencode the geographic theme of the content of the

2http://openlayers.org/3https://github.com/timdown/rangy

192

...

2. While congratulating the troops on their glorious success,the commanding general desires to impress upon all officers aswell as men the necessity of greater discipline and order. Theseare as essential to the success as to the victorious; but with themwe can march forward to new fields of honor and glory, till thiswicked rebellion is completely crushed out and peace restoredto our country.

3. Major-Generals Grant and Buell will retain the immediatecommand of their respective armies in the field.

By command of Major-General Halleck:

N. H. McLEAN,

Assistant Adjutant-General.

HEADQUARTERS DEPARTMENT OF THE MISSISSIPPI,Pittsburg, Tenn., April 14, 1862.

Major General U. S. GRANT,

Commanding District and Army in the Field:

Immediate and active measures must be taken to put your com-mand in condition to resist another attack by the enemy. Frac-tions of batteries will be united temporarily under competentofficers, supplied with ammunition, and placed in position forservice. Divisions and brigades should, where necessary, bereorganized and put in position, and all stragglers returned totheir companies and regiments. Your army is not now in con-dition to resist an attack. It must be made so without delay.Staff officers must be sent out to obtain returns from divisioncommanders and assist in supplying all deficiencies.

H. W. HALLECK,

Major-General.

NEW MADRID, April 14, 1862.

J. C. KELTON:

General Pope received message about Van Dorn and Price. Doyou want his army to join General Halleck’s on the Tennessee?His men are all afloat. He can be at Pittsburg Landing in fivedays. Fort Pillow strongly fortified. Enemy will make a de-cided stand. May require two weeks to turn position and reducethe works. Answer immediately. I wait for reply.

THOMAS A. SCOTT,

Assistant Secretary of War.

SPECIAL ORDERS, HDQRS. DIST. OF WEST TEN-NESSEE,No. 54. Pittsburg, Tenn., April 14, 1862.

II. Brigadier General Thomas A. Davies, having reported forduty to Major-General Grant, is hereby assigned to the com-mand of the Second Division of the army in the field.

By order of Major-General Grant:

[JNumbers A. RAWLINS,]

Assistant Adjutant-General.

CAIRO, ILL., April 14, 1862.

H. A. WISE, Navy Department:

...

Figure 3: Example of WOTR source text, afterstitching up text across page breaks, removing ex-traneous headers/footers/footnotes, etc.

document. The theme of a document is the pri-mary location or locations that the document con-cerns. For example, if the document describes abattle, skirmish or other military action, the loca-tion of that action is the document’s geography.Most correspondence is headed by the location atwhich it was written, which is often the same asthe geographic theme, depending on what the con-tent of the correspondence says. Annotators wereallowed to mark multiple locations or to draw apolygon around an area of the map, which is use-ful when for example the geographic theme is log-ically a body of water or a section of a state ratherthan a single point. However, in the interests ofachieving as many annotations as possible, anno-tators were encouraged to not overly make use ofpolygons or multiple points, preferring a singlepoint when possible. In particular, the mere men-tion of a place name in a document is not sufficientfor it to be included in the geographic theme; itmust be of primary relevance to the subject of thedocument.

Annotators were encouraged to look up to-ponyms found within the text to retrieve theirlatitude/longitude coordinates, with helpful rele-vant keywords attached as necessary, such as CivilWar or the region or commander mentioned in thelarger document context. Annotators were shownhow to retrieve the geocoordinate from Wikipediapages, which was by far the most-frequently usedresource, although Google Maps and niche USCivil War websites were used as well.

4.2.2 Document annotation challengesGeographically diverse documents A largefraction of documents mention multiple places,and our annotators frequently struggled with deter-mining the geographic theme of these documents,preferring to mark multiple points in questionablecases. These cases are common, with an averageof 1.84 points per annotated document. The sys-tems whose results are described in Table 5 are de-signed to work with documents annotated with asingle point; to handle multiple-point documents,the centroid of the points was taken.

Difficult to geolocate documents The geo-graphic theme of many documents is difficult todetermine because they don’t mention any easilyidentifiable locations. Some documents containonly ad-hoc names (e.g. McCullan’s Store or tem-porary army camps named after individual com-

193

Figure 4: Screen shot of the toponym annotation tool. Place names highlighted in yellow, place nameswith geoemetries in green.

manders). Many documents mention only a loca-tion relative to a previously-specified location ina different document, making the theme discover-able only by looking at the whole series of cor-respondence. In some cases no clear geographictheme exists at all. In all such cases (amountingto about 38% of the total), the annotators assignedno geometry to the documents.

4.3 Toponym annotation

To begin the toponym annotation procedure, weidentified a subset of the volumes which had beenannotated with document geolocations (subsec-tions of 15 volumes, selected in part for geo-graphic and topic diversity). Stanford’s NamedEntity Recognizer (NER) was then run on the col-lection of documents, using the standard MUC,CoNLL trained models (Finkel et al., 2005). Theplace annotations that Stanford NER producedwere used as a pre-annotated set, which annota-tors were then asked to correct and add geographicreference to.

The toponym annotation process, whichspanned 4 months and occupied 290 hours,resulted in the annotation of 11,795 toponyms(10,389 with geometries) spanning 1,644 anno-tated documents across 100 page subsections of15 volumes. Originally all toponym annotationswere done by a single annotator. After this processall of the original annotations were reviewed by asecond team of three annotators. These annotatorswere asked to correct a number of problemswith the annotations that were not realized untilafter the initial annotation process had finalized.

Corrections to the original annotation mostlyfocused on building consistent approaches to thechallenges outlined in §4.3.2.

4.3.1 Toponym annotator guidelinesAnnotators were asked to quickly scan the doc-uments and look for place names. Place nameswhich were not detected by Stanford NER shouldbe added, and other entities incorrectly classifiedas places should be deleted. We directed anno-tators to include point, multi-point, polygon, andmulti-polygon geometries where appropriate.

They key guidelines annotators were given forthe task concerned three aspects of toponyms:metonymy, demonymy, and nested named enti-ties. Annotators were asked to exclude metonymicand demonymic names from annotation. NamedEntity Classification researchers have typicallyadopted the stance of annotating the most encom-passing named entity (Finkel and Manning, 2009),though there are exceptions to this trend as is thecase in the LGL corpus. Following the majorityof related work, we ask annotators to only marktoponyms which constitute the most encompass-ing named entity (e.g. 44th Virginia Cavalry ismarked as an organization, and in this case theword Virginia would not be marked). Not includedamong nested named entities are toponym hier-archies, or disambiguators such as in the phraseRichmond, VA, CSA. In these cases each toponymis annotated with separate reference. To find thereference of places, annotators were allowed ac-cess to Internet search. As with document ge-olocation, annotators were encouraged to look up

194

troublesome toponyms on the Internet, and mostlymade use of Wikipedia.

4.3.2 Toponym annotation challengesConjunctive toponyms (toponyms that arejoined by conjunctions) are a problem when theyare in the form of Varnell’s and Lovejoy’s Sta-tions. Here we assumed two toponyms should beadded. However, due to how our GeoAnnotatetool worked, we could not annotate overlapping,discontinuous spanning place names. In thesecases we asked annotators to mark Varnell’s as aplace separate from Lovejoy’s Stations, includingthe Stations term only with the second toponym.

Possessive toponyms (toponyms partially con-sisting of a person’s name) appeared in the cor-pus, e.g. Widow Harrow’s house. Originally, weasked annotators to avoid annotating these as to-ponyms. We later amended our guidelines to askannotators to mark these as toponyms only whenthe possessed entity was capitalized (e.g. Varnell’sStation would be annotated).

Difficult Toponyms (toponyms that could notbe geographically referenced) made up about 12%of the overall toponyms in Wotr-Topo. This wastypical of toponyms that described the locationsof ferries, bridges, railroads, and mills. These fea-tures usually no longer exist, so discovering theirexact reference even with access to Google is verydifficult.

Rivers, and physical features are difficult toreference geographically because their geometricdefinitions are often highly complex, vague, andpoorly defined in gazetteers. Rather than ask an-notators to annotate the full extent of rivers, weasked them to mark a point on the river that theyfelt was most relevant to the context. Annota-tors tended however to opt for whichever point theriver’s Wikipedia page indicated, though this wasnot always the case.

Geographically vague toponym regions ap-pear in the texts. Some of the common examplesappearing in the text are the North, the South, theWest, and Northern Mississippi. We asked anno-tators to mark these as toponyms, and attempt todraw their reference given the context.

Referring Expressions (e.g. the stone bridge)are common. We originally asked annotators not

to annotate them, yet we failed to anticipate refer-ring expressions which were partially constitutedof place names (e.g. the Dalton road). Given thatthese expressions contain proper place names, andare places themselves, we decided to ask annota-tors to try and reference the whole expression (i.e.the location of the road). Unfortunately though,discovering the georeference of such roads is verydifficult, and annotators tended to mark the loca-tion as a point near one of the embedded city to-ponyms.

Embedded Named Entities : We gave our an-notators a rule to only annotate the entity type ofthe most-encompassing named entity. Using thisrule expressions like 44th Virginia Cavalry be-came annotated as one single organization, ratherthan a place inside an organization. We did not an-ticipate however the range of semantically equiv-alent expressions such as 44th Cavalry of Virginiaor 44th Cavalry from Virginia. The former formwe tended to mark as an organization, while thelatter we marked as an organization 44th Cavalryplus a toponym Virginia.

5 Baseline and benchmark systemevaluation

In order to gain an understanding of the difficul-ties of the corpus and encourage its adoption, weevaluate the performance of a number of baselineand benchmark systems on the dataset.

For docgeo, two methods are used for construct-ing grid cells: Uniform and adaptive (KD), whichadjusts cell sizes to equalize the number of doc-uments in each cell (Roller et al., 2012). LRuses flat logistic regression while Hier constructsa coarse-to-fine hierarchy of grids with a beamsearch (Wing and Baldridge, 2014)4.

For TR, Population selects a matchinggazetteer referent with the highest population.WISTR is a bag of words multinomial logisticregression model trained on Wikipedia (Speriosuand Baldridge, 2013). SPIDER is a weighteddistance minimization approach that prefersselecting gazetteer referents that occupy minimalarea (Speriosu and Baldridge, 2013). TopoClus-ter uses a geographic density estimation of thetoponym and context words; TopoClusterGaz5

additionally ’snaps’ to the nearest gazetteer refer-ent (DeLozier et al., 2015). All TR systems were

4https://github.com/utcompling/textgrounder

5https://github.com/grantdelozier/TopoCluster

195

Table 4: WoTR Toponym Resolution ResultsSystem A@161 Mean P R F-1Random 22.2 2216 14.8 6.4 8.9Population 63.1 1483 42.2 18.2 25.4SPIDER 67.1 482 37.8 16.3 22.7WISTR 65.5 895 54.9 15.6 24.4WISTR+SPIDER 67.0 489 37.9 16.4 22.9TopoCluster 57.0 604 31.8 25.9 28.6TopoClusterGaz 71.5 468 37.7 30.7 33.8

Table 5: Doc Geolocation ResultsSystem Acc@161km Median MeanRandom/Uniform 3.4 1009.5 865.6Random/KD 8.3 828.8 753.2NaiveBayes/Uniform 74.8 194.7 53.1NaiveBayes/KD 72.2 204.4 80.2LR/Uniform 77.2 189.8 53.6LR/KD 74.4 182.1 59.8Hier/Uniform 76.8 185.5 49.6Hier/KD 76.2 171.8 47.2

trained using out of domain resources, but someweights and parameters (e.g. context windowsize) were optimized using the WOTR dev set.

Table 5 shows the results of a number of currenttext-only document geolocation systems (Wing,2015) on WOTR. Compared with Naive Bayes,both flat (LR) and hierarchical logistic regression(Hier) produce additional benefits. Hier producesthe best mean and median despite the fact thatit is designed primarily for larger corpora thanWOTR. Uniform grids do slightly better over-all, a result we have seen before in similar-sizedcorpora, but adaptive (KD) grids do better withHier, which is able to compensate somewhat forthe larger adaptive grid cells found in low-densityareas through its use of multiple grid levels.

Table 4 shows the resolution results of manystate-of-the-art toponym resolution systems on thetest split of WOTR. As can be seen, TopoClus-terGaz outperforms all resolvers on all metricswhen oracle NER is used, and outperforms otherson Recall and F-1 Score when predictive NER isincluded in the evaluation. Key to the TopoClus-terGaz’s success is the ability to predict on bothnon-gazetteer and gazetteer matched entities, di-rectly boosting Recall and F-1 Score by largemargins. When evaluating on a development setof the data, we observed that most differencesin system performance could be sourced to howthe respective systems dealt with place namesthat do not have specific GeoNames entries, orare spelled differently than their GeoNames en-try (e.g. Camp Lapwai, Colo. Terr.). TopoClus-ter often produced correct predictions on theseentities, while the gazetteer dependent systems

like Population, WISTR, and SPIDER were un-able to make predictions. NER inclusive scores(P, R, F-1) are generally much lower for WoTR-Topo than other datasets because the NER sys-tems utilized (Stanford-NER and openNLP-NER)are trained on very different domains. Never-theless, strongly superior recall on the gazetteer-independent TopoCluster systems leads to higherF-1 scores on the dataset.

6 Conclusion

The War of the Rebellion corpus represents aunique domain for geolocation research. From theperspective of toponym resolution, the corpus isinnovative in many respects: richness of geomet-ric annotation (annotations with multi-point, poly-gon geometries), corpus size (with roughly twicethe toponyms of other corpora), and place namesnot in gazetteers. Baseline system resolution re-sults indicate that the corpus is the most difficultof the corpora surveyed, with A@161 km scores–and especially NER-inclusive scores–being signif-icantly lower than the next most difficult corpus,LGL (DeLozier et al., 2015). The corpus is thefirst published document-geolocation corpus fo-cusing on historical texts, the first based on run-ning text, the first that was annotated specificallyfor the task of theme-based document geoloca-tion, and the first annotated with multi-point andpolygon geometries. Finally, the availability oftext marked both with toponym and docgeo anno-tations presents new opportunities for joint infer-ence.

7 Corpus availability

The corpus is freely available at our github page6

under an MIT License. We hope others may ex-pand and improve on the annotations.

8 Acknowledgements

We would like to thank David Staley and OhioState University’s Department of History for ac-cess to their high quality version of the War of theRebellion corpus. This research was supported bya grant from the Morris Memorial Trust Fund ofthe New York Community Trust.

6https://github.com/utcompling/WarOfTheRebellion

196

ReferencesEdward L. Ayers and Scott Nesbit. 2011. Seeing

emancipation: Scale and freedom in the americansouth. Journal of the Civil War Era, 1(1):3–24.

Paolo Bolettieri, Andrea Esuli, Fabrizio Falchi, Clau-dio Lucchese, Raffaele Perego, Tommaso Piccioli,and Fausto Rabitti. 2009. CoPhIR: a test col-lection for content-based image retrieval. CoRR,abs/0905.4627.

Mariam Daoud and Jimmy Xiangji Huang. 2013. Min-ing query-driven contexts for geographic and tem-poral search. International Journal of GeographicalInformation Science, 27(8):1530–1549.

Chris De Rouck, Olivier Van Laere, Steven Schockaert,and Bart Dhoedt. 2011. Georeferencing wikipediapages using language models from flickr. In Seman-tic Web, 10th International conference, Proceedings,page 8.

Grant DeLozier, Jason Baldridge, and Loretta London.2015. Gazetteer-independent toponym resolutionusing geographic word profiles. In Twenty-NinthAAAI Conference on Artificial Intelligence.

Jenny Rose Finkel and Christopher D Manning. 2009.Nested named entity recognition. In Proceedingsof the 2009 Conference on Empirical Methods inNatural Language Processing: Volume 1-Volume 1,pages 141–150. Association for Computational Lin-guistics.

Jenny Rose Finkel, Trond Grenager, and ChristopherManning. 2005. Incorporating non-local informa-tion into information extraction systems by Gibbssampling. In Proceedings of the 43rd Annual Meet-ing on Association for Computational Linguistics,pages 363–370. Association for Computational Lin-guistics.

Claire Grover, Richard Tobin, Kate Byrne, MatthewWoollard, James Reid, Stuart Dunn, and JulianBall. 2010. Use of the Edinburgh geoparserfor georeferencing digitized historical collections.Philosophical Transactions of the Royal Society A:Mathematical, Physical and Engineering Sciences,368(1925):3875–3889.

Bo Han, Paul Cook, and Tim Baldwin. 2014. Text-based Twitter user geolocation prediction. Journalof Artificial Intelligence Research, 49(1):451–500.

Saul A. Kripke. 1980. Naming and Necessity. HarvardUniversity Press.

Jochen L Leidner. 2008. Toponym Resolution in Text :Annotation, Evaluation and Applications of SpatialGrounding of Place Names. Universal Press, BocaRaton, FL, USA.

Michael D Lieberman and Hanan Samet. 2012.Adaptive context features for toponym resolution in

streaming news. In Proceedings of the 35th interna-tional ACM SIGIR conference on Research and de-velopment in information retrieval, pages 731–740.ACM.

Koji Matsuda, Akira Sasaki, Naoaki Okazaki, and Ken-taro Inui. 2015. Annotating geographical entitieson microblog text. In The 9th Linguistic AnnotationWorkshop held in conjuncion with NAACL 2015,page 85.

Andrew Kachites McCallum. 2002. MAL-LET: A machine learning for language toolkit.http://mallet.cs.umass.edu.

Scott Nesbit. 2013. Visualizing emancipation: Map-ping the end of slavery in the american civil war.In Justyna Zander and Pieter J. Mosterman, editors,Computation for Humanity: Information Technol-ogy to Advance Society, pages 427–435. New York:Taylor & Francis.

Neil O’Hare and Vanessa Murdock. 2013. Modelinglocations with social media. Information Retrieval,16(1):30–62.

Stephen Roller, Michael Speriosu, Sarat Rallapalli,Benjamin Wing, and Jason Baldridge. 2012. Super-vised text-based geolocation using language modelson an adaptive grid. In Proceedings of the 2012 JointConference on Empirical Methods in Natural Lan-guage Processing and Computational Natural Lan-guage Learning, EMNLP-CoNLL ’12, pages 1500–1510, Stroudsburg, PA, USA. Association for Com-putational Linguistics.

Joao Santos, Ivo Anastacio, and Bruno Martins. 2014.Using machine learning methods for disambiguatingplace references in textual documents. GeoJournal,pages 1–18.

Axel Schulz, Aristotelis Hadjakos, Heiko Paulheim,Johannes Nachtwey, and Max Muhlhauser. 2013.A multi-indicator approach for geolocalization oftweets. In Emre Kiciman, Nicole B. Ellison, BernieHogan, Paul Resnick, and Ian Soboroff, editors,ICWSM’13: Proceedings of the 7th InternationalAAAI Conference on Weblogs and Social Media. TheAAAI Press.

David A Smith and Gregory Crane. 2001. Disam-biguating geographic names in a historical digitallibrary. In Research and Advanced Technology forDigital Libraries, pages 127–136. Springer.

Michael Speriosu and Jason Baldridge. 2013. Text-driven toponym resolution using indirect supervi-sion. In ACL (1), pages 1466–1476.

Yael A. Sternhell. 2016. Afterlives of a confederatearchive: Civil war documents and the making of sec-tional reconciliation. Journal of American History,102(4):1025–1050.

William G. Thomas III. 2011. The Iron Way: Rail-roads, the Civil War, and the Making of ModernAmerica. Yale University Press.

197

Olivier van Laere, Steven Schockaert, Vlad Tanas-escu, Bart Dhoedt, and Christopher B. Jones. 2014.Georeferencing wikipedia documents using datafrom social media sources. ACM Trans. Inf. Syst.,32(3):12:1–12:32, July.

Benjamin Wing and Jason Baldridge. 2014. Hierar-chical discriminative classification for text-based ge-olocation. In Proceedings of the 2014 Conferenceon Empirical Methods in Natural Language Pro-cessing (EMNLP), pages 336–348, Doha, Qatar, Oc-tober. Association for Computational Linguistics.

Benjamin Wing. 2015. Text-Based Document Geolo-cation and its Application to the Digital Humanities.Ph.D. thesis, University of Texas at Austin.

198