Web Searching with Multiple Correct

Answers

Steve O’Hara, Google Inc., CaliforniaTom Bylander, Univ. of Texas San Antonio

June 2 - 4, 2014, Thessaloniki, GreeceWeb Intelligence, Mining and Semantics

“LONGMONT, CO—The Information Age was dealt a stunning blow Monday, when a factual error was discovered on the Internet. The error was found on TedsUltimateBradyBunch.com, a Brady Bunch fan site that incorrectly listed the show's debut year as 1968, not 1969.”

http://www.theonion.com/articles/factual-error-found-on-internet,102/

Factual Error Found On Internet!

“What is the atomic mass of Copper?” “What is the diameter of Pluto?”

Factoid Questions

# Value Units Miles Delta Min Max1 2300 KM 1,429 31.1 1,398 1,4602 2390 KM 1,485 3.1 1,482 1,4883 2170 MILES 2,170 5.0 2,165 2,1754 1416 MILES 1,416 0.5 1,416 1,4175 1444 MILES 1,444 0.5 1,444 1,4456 1370 MILES 1,370 5.0 1,365 1,3757 1500 MILES 1,500 50.0 1,450 1,5508 2390 KM 1,485 3.1 1,482 1,4889 2390 KM 1,485 3.1 1,482 1,488

10 1429 MILES 1,429 0.5 1,429 1,43011 2300 KM 1,429 31.1 1,398 1,46012 2274 KM 1,413 0.3 1,413 1,41313 1865 MILES 1,865 2.5 1,863 1,868

# Value Units Miles Delta Min Max14 2274 KILO-

METERS 1,413 0.3 1,413 1,41315 1860 MILES 1,860 5.0 1,855 1,86516 1860 MILES 1,860 5.0 1,855 1,86517 2320 KM 1,442 3.1 1,438 1,44518 1430 MILES 1,430 5.0 1,425 1,43519 2274 KILO-

METERS 1,413 0.3 1,413 1,413

20 2274 KILO-METERS 1,413 0.3 1,413 1,413

21 2270 KM 1,411 3.1 1,407 1,41422 1485 MILES 1,485 2.5 1,483 1,48823 1400 MILES 1,400 50.0 1,350 1,45024 300000

0 METERS 1,864 310.7 1,553 2,17525 1400 MILES 1,400 50.0 1,350 1,450

Select “Best” Answer

The two candidate answers of 1400 miles receive the most Support

The four candidate answers of 2274 km (1413 miles) give the most Support

What is the Diameter of Pluto?

zoom

Numeric Correlation

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%0

50

100

150

200

250

300

350

400

450

Reliability for Numeric Questions

Rightiness

Ran

ked

Sup

port

What is the population of New York

City? What is the area of Canada? When did Elizabeth Taylor get

married? What is the capital of Bolivia? Who coaches the Hoyas men’s

basketball team?

Motivating Questions

Numeric Hypothesis

99

200

301

400

500.1

100

400

403

497

500

300

300.9398.6

601 600

City Populations

City, State City MSA

Abilene, Texas 118,117 166,416Akron, Ohio 198,402 701,456

Albuquerque, New Mexico 552,804 898,642Wichita Falls, Texas 103,931 150,261

Wichita, Kansas 384,445 625,526

Wilmington, North Carolina 108,297 369,685

Some Candidate Answers

(for the population of Akron, Ohio)

th a population of 217,074 (2000 census).<div/

n, OH. Population: 214,556. Compare Akron to T

ent from 69,000 to 210,000. By the mid 20&#39;

. 1920, Population 208,435 . Warren G. Harding

with approximately 207,934 people within the c

as a population of 207934 people this year. Ak

Total population, 200,181 ...<div/><div><span

as a population of 200,000. Through the 1900s,

any) was producing 200,000 "Diamond" brand lam

ad a population of 199,110. The Akron Metropol

OH, population is 199,110.<div/><div><span st

Candidate Answer Distribution

0

5

10

15

20

25

Num

ber o

f Can

dida

te A

nsw

ers

Population of Akron, Ohio (zoomed)

Target answers: 198,402 city, 701,456 MSA

Abilene Results

Rank

CandidateAnswer Support City

RightinessMSA

Rightiness1 112,205 4.600 0.431 0.1622 114,757 4.572 0.510 0.169

Rank CandidateAnswer Pair

Paired Suppo

rt

City Rightines

s

MSA Rightine

ss1 106,654 - 108,157 4.273 0.356 0.1482 975,619 -

1,005,656 3.394 0.018 0.027

Rank

CandidateAnswer Support

CityRightine

ss

MSA Rightine

ss1 116,966 6.889 0.656 0.1752 1,005,656 5.021 0.018 0.026

Single Score(0.431/1 + 0.169/2) / 2

0.258

Paired Score(0.356 + 0.148) / 2

0.252

Combined Score(0.656 + 0.026) / 2

0.341

Target: 118,117 Cityand 166,416 MSARan

kCandidate

Answer Support CityRightiness

MSA Rightiness

1 117,063 2.000 0.668 0.1752 11,928,500 2.000 0.001 0.002

Baseline Score(0.668 + 0.002) / 2

0.335

City Population Results

 Baselin

e Results

Single Result

s

Paired Result

sCombined Results

Mean Score 0.233 0.297 0.291 0.319Improveme

nt - 31% 29% 41%

Multi-Valued Named Entities

Ann

Bob

Carl

Dan

Eva

Anna

Dana

Dana

Eve

Eve

Carl

CarlDana

Fred Fred

Adapt Levenshtein’s Algorithm to whole words

Count #insertions, #deletions and #changes Instead of character matching, do whole word

matching Which uses regular Levenshtein’s edit distance

Handles typos Handles middle names Does not handle Bill / William

Proper Name Matching

Two Levels of Searches

SearchMovie

Directors

SearchMovieStars

MovieDirectors

MovieStars

Directorsand Stars

Movie Target Answers

Movie Director(s) Stars

Ben-Hur William Wyler Charlton Heston, Jack Hawkins,Stephen Boyd, Haya Harareet

Gone with the Wind Victor Fleming Clark Gable, Vivien Leigh,

Thomas Mitchell, Barbara O'Neil

Time of Violence Ludmil StaikovIossif Surchadzhiev, Rousy

Chanev,Ivan Krystev, Anya Pencheva

The English Patient

Anthony Minghella

Ralph Fiennes, Juliette Binoche,Willem Dafoe, Kristin Scott

ThomasSchindler's List Steven Spielberg Liam Neeson, Ralph Fiennes,

Ben Kingsley, Caroline Goodall

GWTW Results

Support Candidate Answer Result37 Victor Fleming correct21 George Cukor wrong12 Sam Wood (skip)

Support Candidate Answer Result10 Vivien Leigh correct5 Clark Gable (skip, correct)3 Victor Fleming (skip, director)3 Evelyn Keyes (skip)2 Rhett Butler (skip)2 Rand Brooks (skip)

MovieDirectors

MovieStars

Of 500 movies, 32 instances of co-occurring

pairs 23 additional correct answers 5 duplicate correct answers 4 incorrect answers:

Movie Results

Movie Errors and RationalizationJFK Tom Hanks is the director of a second JFK

movie.Lee Harvey Oswald was the assassin.

The Fighter The movie is about Mickey Ward.V for

VendettaAlan Moore is the author of the book.

Is Wikipedia Always Right?

“The number of factual errors shows just how unreliable it can be to use the online resource as a sole means of digging up information.

Yet millions base everything from school homework to corporate presentations using facts and figures they have gleaned from the site.

A study into, specifically, company information on the massively popular website discovered 60 per cent of articles had factual errors.”

Source: www.dailymail.co.uk/sciencetech/article-2131458/Up-articles-Wikipedia-contain-factual-errors.html#ixzz2sDrWl4RH