Leveraging Wikipedia-based Features for Entity Relatedness and Recommendations

Leveraging Wikipedia-based Features for Entity Relatedness

and Recommendations

Nitish AggarwalSupervised by Dr. Paul Buitelaar

PhD Viva

Brad Pitt

Motivation

MotivationSemantic Web

Technologies:1. RDF2. SPARQL3. Ontology4. Linked data5. Turtle (syntax)

Entity Recommendation

Companies:1. Metaweb2. Ontoprise GmbH3. OpenLink Software4. Ontotext5. Powerset (company)

MyosinProteins and cells:1. Actin2. Muscle contraction3. Sarcomere4. Myofibril5. Cytoskeleton

Biologists:1. Hugh Huxley2. James Spudich3. Ronald Vale4. Manuel Morales5. Brunó Ferenc Straub

Determine the degree of relatedness between two entities

Brad Pitt Tom Cruise

Entity Relatedness

Person, location, organization

Time, date, money, percent

Event, movie, disease, symptom, side effect, law, license and more

Background

Entity• Many such types are covered

in Wikipedia

• More than 2K classes in DBpedia

• More than 350k classes in Yago

• Every Wikipedia article is considered about an entity

Motor vehicle

Car Motorcycle

Automobile

AutoCar seat

Car windows

Background

Relatedness

Synonyms Similar

• Distributional Semantics for Entity Relatedness (DiSER)• Evaluation

• Entity Recommendation• Wikipedia-based Features for Entity Recommendation (WiFER)• Evaluation

• Text Relatedness• Non-Orthogonal Explicit Semantic Analysis (NESA)• Evaluation

• Application and Industry Use Cases• Conclusion

Wikipedia Features for Entity Recommendation

(WiFER)Feature

Extraction

Thesis Overview

Distributional Semantic for Entity Relatedness

(DiSER)

DistributionalRepresentatio

Non-Orthogonal Explicit Semantic Analysis

(NESA)

Chapter V

Chapter IV

Chapter VI

Motivation Entity Relatedness

Text Relatedness Application Conclusion

Thesis Overview

(WiFER)Feature

Extraction

(DiSER)

(NESA)

Chapter IV

Entity Relatedness

Entity Relatedness: State of the Art• Graph-based methods

• Path distance in Wikipedia graph (Strube and Ponzetto, 2006)• Normalized Google Distance on Wikipedia graph (Witten and Milne,

2008)• Personalized pagerank on Wikipedia graph (Agirre et. al, 2015)• Path-based measures on DBpedia graph (Hulpus et. al, 2015)

• Corpus-based methods• Key-phrase Overlap for Related Entities (KORE): partial overlaps

between key-phrases in corresponding Wikipedia articles (Hoffart et. al, 2012)

• Text relatedness measures: use colocation information in text

Explicit Semantic Analysis (ESA)Uses explicit (manually defined) concepts like Wikipedia articles where every article is considered describing a single concept (Gabrilovich and

Markovitch, 2007)

Entity Relatedness: State of the ArtDistributional Semantics

word1 W11 W12 W13 W14 …....... W1n

word2 W21 W22 W23 W24 …....... W2n

word3 W31 W32 W33 W34 …........ W3n

wordm Wm1 Wm2 Wm3 Wm4 …... Wmn

doc1 doc2 doc3 doc4 ….... docn

word1 W11 W12 W13 W14 …....... W1n

word2 W21 W22 W23 W24 …....... W2n

word3 W31 W32 W33 W34 …........ W3n

wordm Wm1 Wm2 Wm3 Wm4 …... Wmn

Entity Relatedness: State of the ArtDistributional Semantics

Implicit/Latent Semantic Analysis (LSA)Transforms sparse document space into a dense latent topic space

Latent Dirichlet Allocation (LDA)(Blei et al., 2003)

Latent Semantic Analysis (LSA)(Deerwester et al., 1990)

Neural Embeddings(Word2Vec)(Mikolov et al., 2013)

n ~ 1M

word1 W11 W12 ……..... W1k

word2 W21 W22 ……..... W2k

wordm Wm1 Wm2 ……..... Wmk

topic1 topic2 … topick

Dimensionality

Reduction

k < 1000

Limitation of Text Relatedness Measures

• Compositionality • Most of the entities are multiword expressions• Vector(Brad Pitt) = Vector(Brad) + Vector(Pitt) ?

• Ambiguity • Vector of an entity with ambiguous name like “Nice” (French city)

Chapter IV

Distributional Semantics for Entity Relatedness (DiSER)

entity1 W11 W12 W13 W14 …....... W1n

entity2 W21 W22 W23 W24 …....... W2n

entity3 W31 W32 W33 W34 …........ W3n

entityn Wn1 Wn2 Wn3 Wn4 …... Wnn

Wikipedia-based Distributional Semantics for Entity Relatedness In: AAAI-FSS-2014

[Steve Jobs] co-founded Apple in 1976 to sell Wozniak’s [Apple I] [Personal Computer]. [Steve Jobs | Jobs] was CEO of [Apple Inc. | Apple] and largest shareholder of [Pixar]. Jobs is widely recognized as a pioneer of the [Microcomputer Revolution], along [Steve Wozniak | Wozniak].

Annotated Wikipedia with entities

One sense per document

Wikipedia entities[Steve Jobs] [Apple Inc.| Apple] [Steve Wozniak | Wozniak]’ [Apple I] [Personal Computer]. [Steve Jobs | Jobs] was CEO of [Apple Inc. | Apple] and largest shareholdef [Pixar]. [Steve Jobs | Jobs] is widely recognizpioneer of the [Microcomputer Revolution], along [Steve Wozniak | Wozniak].

The Tree of Life (film)Falmouth, CornwallWorld War Z (film)What Just HappenedA Mighty Heart (film)Plan B EntertainmentJamaican PatoisRichard: A NovelSobriquetI Want a Famous Face

Brad Pitt (DiSER)Damiani (jewelry

company)University of Pittsburgh

BandBrad PittMake It Right FoundationPittsburgh men’s basketballBrangelinaPittsburgh Panthers baseballPitt (Comics)Pitt RiverBrad Pitt filmography

Brad Pitt (ESA)

ESA vs DiSER Vector

Chapter IV

Entity Relatedness: Evaluation

• Absolute relatedness score• Relatedness between “Apple Inc.” and “Steve Jobs”• Very low inter-annotator agreement

• Relative relatedness score• Is “Steve Jobs” more related with “Apple Inc.” than “Bill Gates”• High inter-annotator agreement

• KORE (Hoffart et al., 2012)• 21 seed entities• Every entity has list of 20 entities with their relatedness score• 420 entity pairs in total

Entity Relatedness: DatasetMotivation Entity

RelatednessEntity

RecommendationText Relatedness Application Conclusion

ApproachesSpearman

Rank Correlation

Graph-based measures

Path-DBpedia (Hulpus et al., 2015) 0.610WLM (Witten and Milne, 2008) 0.659PPR (Agirre et al., 2015) 0.662

Corpus-based measures

Word2Vec (Mikolov et al., 2013) 0.181GloVe (Pennington et al., 2014) 0.194LSA (Landauer et al., 1998) 0.375KORE (Hoffart et al., 2012) 0.679ESA (Gabrilovich and Markovitch, 2007)

DiSER 0.781

Results: KORE DatasetMotivation Entity

RelatednessEntity

DiSER Vector for non-Wikipedia Entities

BBC: http://www.bbc.com/news/world-europe-22204377

Article about Savita

Context-DiSER

Noun phrase extraction: StanfordNLP

Entity linking: Prior probability

AbortionAbortion-

rights movement

The Irish Times

United States pro-life

movement

Vincent Browne

Michael D.

Higgins

Context-DiSER

Irish abortion lawDeath of SavitaGalway University HospitalMiscarriageCatholic Country…….

Savita Halappanavar

Approaches Spearman Rank Correlation

KORE (state of the art) 0.679Context-ESA 0.684Context-DiSER (Manual linking)

Context-DiSER (Automatic linking)

Context-DiSER: Results on KORE Dataset

Thesis Overview

(WiFER)Feature

Extraction

(DiSER)

(NESA)

Chapter V

Chapter IV

• Classical Recommendation Systems• Focus on personalized recommendation• Require user-item preferences

• Entity Recommendation in Web Search (Blanco et al., 2013)• Co-occurrence features: query logs, query session, Flickr tags,

tweets• Graph-based features: shared connections in Yahoo knowledge

graph and others domain specific knowledge bases• Entity and Relation type in Knowledge graph• More than 100 features• Combines features using learning to rank

Entity Recommendation: State of the Art

Features:

Prior Probability of Entity1

Prior Probability of Entity2

Joint ProbabilityConditional ProbabilityReverse Conditional ProbabilityCosine SimilarityPointwise Mutual InformationDistributional Semantic Model

Learning to Rank

Leveraging Wikipedia Knowledge for Entity Recommendations In: ISWC 2015

Wikipedia-based Features for Entity Recommendation (WiFER)

Prior Probability of Entity1Prior Probability of Entity2Joint ProbabilityConditional ProbabilityCosine Similarity Pointwise Mutual InformationReverse Conditional ProbabilityDistributional Semantic Model (ESA)

Wikipedia Text Wikipedia EntitiesPrior Probability of Entity1Prior Probability of Entity2Joint ProbabilityConditional ProbabilityCosine Similarity Pointwise Mutual InformationReverse Conditional ProbabilityDistributional Semantic Model (DiSER)

Wikipedia-based Features for Entity Recommendation (WiFER)

• Learning to Rank• Gradient Boosted Decision Trees (GBDT) (Li Hang, 2011)• It builds the model in a stage-wise fashion

• Dataset: Entity recommendation in web search• 4,797 web search queries (entities)• Every entity query has a list of entity candidates (47,623 entity-

pairs)• All candidates are tagged on 5 label scales: Excellent, Prefer, Good,

Fair, and Bad

Combining Features

Type Total instances Percentage

Location 22,062 46.32People 21,626 45.41Movies 3,031 6.36

TV Shows 280 0.58Album 563 1.18Total 47,623 100

• Evaluation• Normalized discounted cumulative gain (NDCG@10) • 10 fold cross validation

Features All Person LocationSpark

(Blanco et al., 2013)

0.9276 0.9479 0.8882

WiFER 0.9173 0.9431 0.8795Spark+WiFE

R0.9325 0.9505 0.8987

Insights into Entity Recommendation in Web SearchIn: IESD at ISWC, 2015

Entity Recommendation: Results Motivation Entity

RelatednessEntity

Insights into Entity Recommendation in Web SearchIn: IESD at ISWC, 2015

Entity Recommendation: Feature Analysis in Spark+WiFER

Relation type

Cosine similarity over Flickr tagsProbability of target entity over Wikipedia text corpusCF7 over Flickr tagsDSM over Wikipedia entities corpus (DiSER)Conditional user probability over query termsDSM over Wikipedia text corpus (ESA)Probability of source entity over Wikipedia entities corpusProbability of target entity over Flickr tagsProbability of target entity over Wikipedia entities corpus

Thesis OverviewMotivation Entity

RelatednessEntity

(WiFER)Feature

Extraction

(DiSER)

(NESA)

Chapter V

Chapter IV

Chapter VI

Text Relatedness:Non-Orthogonal Explicit Semantic Analysis (NESA)

ESA assumes that related words share highly weighted concepts in their distributional vector

Chapter VIImproving ESA with Document SimilarityIn: ECIR-2013

“soccer”History of Soccer in the United

StatesSoccer in the United States

United States Soccer Federation

North American Soccer League

United Soccer Leagues

“football”

FootballHistory of association

footballFootball in England

Association football

ESA(football, soccer) = 0.0

Orthogonality in ESAMotivation Entity

RelatednessEntity

Chapter VIImproving ESA with Document SimilarityIn: ECIR-2013

“soccer”History of Soccer in the United

StatesSoccer in the United States

United States Soccer Federation

North American Soccer League

United Soccer Leagues

“football”

FootballHistory of association

footballFootball in England

Association football

NESA(football, soccer) = (FIFA x Soccer in the United States + FIFA x United Soccer Leagues ….) = 0.38

Non-Orthogonal Explicit Semantic Analysis (NESA)

• ESA: v1 and v2 are the n-dimensional vectors for words w1 and w2

• relESA (w1, w2) = v1T . v2

• NESA: Correlation between vector dimensions

• relNESA (w1,w2) = v1T . C . v2

• C(n,n) = ET . E

• Dimension correlation methods• DiSER scores between corresponding Wikipedia article

Non-Orthogonal Explicit Semantic Analysis (NESA)

• WN353• 353 word pairs annotated by 13-15 experts on a scale of 1-10.

• RG65• 65 word pairs annotated by 51 experts on scale of 0-4

• MC30• 30 word pairs annotated by 38 experts on scale of 0-1

• MT287• 287 word pairs annotated by 10-12 experts on scale of 0-1

Word Relatedness DatasetsMotivation Entity

RelatednessEntity

Non-Orthogonal Explicit Semantic AnalysisIn: *SEM-2015 Chapter VI

WN353 MC30 RG65 MT287

LSA 0.579 0.667 0.616 0.555

LSA (Wiki) 0.538 0.744 0.697 0.353

Word2Vec 0.663 0.824 0.751 0.560

ESA 0.66 0.765 0.826 0.507

NESA 0.696 0.784 0.839 0.572

Spearman rank correlation with word similarity gold standard datasets

NESA: ResultsMotivation Entity

RelatednessEntity

Non-Orthogonal Explicit Semantic AnalysisIn: *SEM-2015 Chapter VI

NESA: Results

• Word similarity vs relatedness (Agirre et al., 2009)• WN353Rel: 202 word pairs from WN353• WN353Sim: 252 word pairs from WN353Spearman rank correlation with word similarity vs relatedness

datasets WN353Rel

WN353Sim

LSA 0.521 0.662

LSA (Wiki) 0.506 0.559

Word2Vec 0.601 0.741

ESA 0.643 0.663

NESA 0.663 0.719

Outline• Motivation• Entity Relatedness

• Distributional Semantics for Entity Relatedness (DiSER)• Evaluation

• Entity Recommendation• Wikipedia-based Features for Entity Recommendation (WiFER)• Evaluation

• Text Relatedness• Non-Orthogonal Explicit Semantic Analysis (NESA)• Evaluation

• Application and Industry Use Cases• Conclusion

Chapter VIIhttp://enrg.insight-centre.org/

EnRG SPARQL Endpoint

National University of Ireland, Galway

Industrial Use Cases

Medical entity linking for question-answering and relationship explanation in Knowledge Graph

Entity Recommendation in Web Search

Company name disambiguation for social profiling

• Entity Relatedness• Distributional Semantics for Entity Relatedness (DiSER)• Outperformed state of the art entity relatedness measures

• Entity Recommendation• Wikipedia-based Features for Entity Recommendation (WiFER)• Effective features for entity recommendation in web search

• Text Relatedness• Non-Orthogonal Explicit Semantic Analysis (NESA)• Outperformed other existing word relatedness measures

• Entity Relatedness Graph (EnRG)• Contains all Wikipedia entities and their pre-computed relatedness

scores• Contains distributional vectors for all Wikipedia entities

ConclusionMotivation Entity

RelatednessEntity

• Relationship explanation for recommended entities• Best path in knowledge graph• Best natural language description

• Knowledge discovery• Analogy querying over knowledge graph

e.g. Google to Motorola => Microsoft to ?• Example based querying

e.g. Google to Motorola => ? to ?

Future Research Directions

Related Queries?

Leveraging Wikipedia-based Features for Entity Relatedness and Recommendations

Data & Analytics

Relatedness 1: IBD and coefficients of relatedness

Institutional relatedness behind product …mikepeng/documents/Peng17_APJM_SunTan.pdfInstitutional relatedness behind product diversification and international diversification Sunny

Relatedness = recency of common ancestry

Designing Smart Specialization Policy: relatedness

Relatedness part 2 - Familias

Multinational enterprises, industrial relatedness and

Institutional relatedness behind product diversification ...mikepeng/documents/Peng17_APJM_… · Institutional relatedness behind product diversification and international diversification

Leveraging distant relatedness to quantify human mutation ... · Leveraging distant relatedness to quantify human mutation and gene conversion rates Pier Palamara ... Albert Hofman

Controlling for Phylogenetic Relatedness and Evolutionary

Semantic Relatedness for All (Languages): A Comparative Analysis of Multilingual Semantic Relatedness using Machine Translation

Content About Wikipedia About Wikipedia How to assess Wikipedia??? How to assess Wikipedia??? GOOGLE SEARCH – WIKIPEDIA GOOGLE SEARCH – WIKIPEDIA

Motivation and Relatedness

A Random Graph Walk based Approach to Computing Semantic Relatedness Using Knowledge from Wikipedia Presenter: Ziqi Zhang OAK Research Group, Department

Mending Nets of Relatedness

Roots and Event Structure I - Linguistic Society of America · 2019-09-24 · Subject-verb relatedness 2.3 2.2 Verb-object relatedness 2.9 3.0 Subject-object relatedness 2.7 2.7 McKoon

relatedness - University of Technology Sydney

Pedigree relatedness and pseudo-phenotypes as a first ... relatedness and pseudo...2 Pedigree relatedness and pseudo -phenotypes as a first approach to assess and maintain genetic

A wikipedia-based semantic relatedness framework for ...Keywords Online reputation management · Semantic relatedness · Wikipedia · Reputation dimensions 1 Introduction The area

Knowledge Graph Embeddings for Recommender Systems · 2020. 9. 6. · book that they like, leveraging an item-item relatedness measure based on entity2rec. In addition to defining

Relatedness, Complexity and Local Growthmotu- · Relatedness, Complexity and Local Growth ii Abstract We derive a measure of the relatedness between economic activities based on weighted