27
Linking Organization al Social Networking Profiles Research Wrap-Up – 28 August 2015 1

Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

Embed Size (px)

Citation preview

Page 1: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

1

Linking OrganizationalSocial Networking ProfilesResearch Wrap-Up – 28 August 2015

Page 2: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

2

Develop a systemto find an organization’s profiles across different social networks.

Objective

Page 3: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

3

Brands

Regional

Affiliates

Affiliate Profiles

Page 4: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

4

System

Overview

Organization Name

Official

Affiliate

Unrelated

Page 5: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

5

OfficialProfiles representing the company as a whole. e.g. @Microsoft, @Dell (respectively)

AffiliateProfiles representing a brand or regional affiliate.

e.g. @Surface, @Windows, @MicrosoftAsia

UnrelatedProfiles that aren’t run by the company itself.

Includes employees, other companies.

Page 6: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

6

Introduction

Introduction

Implementation

Evaluation

Results/Discussion

Future?

Page 7: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

7

Input Processing

QueryGET /company/Microsoft Corporation

Profile Acquisition

Twitter/Facebook Search API

DuckDuckGo Instant Answers

API

Processed Querye.g. “Microsoft”

Profile Conversion

Profile Classification

Twitter/Facebook Profiles

Feature Vectors

Labelled Profilesjson

Pipeline

Page 8: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

8

Page 9: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

9

Input Processing

QueryGET /company/Microsoft Corporation

Profile Acquisition

Twitter/Facebook Search API

DuckDuckGo Instant Answers

API

Processed Querye.g. “Microsoft”

Profile Conversion

Profile Classification

Twitter/Facebook Profiles

Feature Vectors

Labelled Profilesjson

Pipeline

Page 10: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

10

Input ProcessingQuery DuckDuckGo Instant Answers API, which gives

a “topic summary”.Take the name from that summary.

Page 11: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

11

Profile AcquisitionQuery Twitter/Facebook’s search API and retrieve 20

candidate profiles.

Page 12: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

12

Name-based

(5)

• N1: Normalized Edit Distance: Query to Username• N2: Normalized Edit Distance: Query to Display Name• N3: Length of Query• N4: Length of Username• N5: Length of Display Name

Description-

based (3)

• D1: Occurrences of Query in Description• D2: Cosine Similarity: Query and Description• D3: Cosine Similarity: Profile Description and

DuckDuckGo Description

Language Model-

based (6)

• LM1: “Official” Description LM Probability• LM2: “Affiliate” Description LM Probability• LM3: “Unrelated” Description LM Probability• LM4: “Official” Post LM Probability• LM5: “Affiliate” Post LM Probability• LM6: “Unrelated” Post LM Probability

Profile Conversion - Features

Page 13: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

13

Name-based FeaturesN1 - Normalized Edit

Distance: Query to UsernameN2 - Normalized Edit Distance: Query to Display NameN3 - Length of QueryN4 - Length of UsernameN5 - Length of Display Name

1−𝑒𝑑𝑖𝑡 _𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒

𝑚𝑎𝑥 (𝑙𝑒𝑛 (𝑠1 ) , 𝑙𝑒𝑛 (𝑠2 ))0 when completely different, 1 when identical

Username: GMDisplay Name: General Motors

QuirksAbbreviations: GM versus General Motors

Stopwords: “Corporation”, “Company”, etc.

Imposters!

Page 14: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

14

Description-based FeaturesD1 - Occurrences of Query

D2 - Cosine Similarity: Query and DescriptionD3 - Cosine Similarity: DuckDuckGo Description and Profile Description

Page 15: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

15

Language Model-based FeaturesProbability that description/posts

appear in each language model:

Description

• LM1 - Official Profiles• LM2- Affiliate Profiles• LM3 - Unrelated

Profiles

Recent Posts

• LM4 - Official Profiles• LM5 - Affiliate Profiles• LM6 - Unrelated

Profiles

Page 16: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

16

Official; 232; 7%Affiliate;

675; 20%

Unrelated; 2474; 73%

3381 labels from 228 organizations

Twitter Labels

Official; 145; 4% Affil-iate; 491; 14%

Unrelated; 2767; 81%

3403 labels from 216 organizations

Facebook Labels

Ground Truth Breakdown

Page 17: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

17

Per-Fold Evaluation Process

Official Profiles

Affiliate Profiles

1. Training set is used to train the classifier.

Classifier

Unrelated

Profiles

2. Test set is filtered for official and affiliate profiles.

Official Profiles

Affiliate Profiles

Test Set

3. Obtain list of organizations that own these profiles.

Official Profiles

Affiliate Profiles

Organization Names

System

4. Names used to query system, results used to calculate performance.

Organization Names

Classifier

Classified

Official

Classified

AffiliateClassifie

d Unrelate

d

Page 18: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

18

BaselineSimulates manually judging profiles by name alone.

N1 - Normalized Edit Distance: Query to Username

N2 - Normalized Edit Distance: Query to Display Name

Page 19: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

19

F1 Precision Recall0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

60.4%

80.1%

48.5%

93.5%97.5%

89.9%

Baseline Final

Official

F1 Precision Recall0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

67.0%74.0%

61.1%

92.3% 94.9%89.8%

Baseline Final

Affiliate

Results - Twitter

Page 20: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

20

F1 Precision Recall0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

81.0% 79.4% 82.6%

93.3% 96.2%90.6%

Baseline Final

Official

F1 Precision Recall0%

10%

20%

30%

40%

50%

60%

70%

80%

48.9%

59.0%

41.8%

67.0%

75.3%

60.4%

Baseline Final

Affiliate

Results - Facebook

Page 21: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

21

Profile TypesFacebook has multiple profile types: people, pages,

places, groups, etc.Twitter has just one: people.

Affiliates?Why don’t FB affiliates score as well? Page usernames are

optional./pages/Netflix-Latinoamérica/553454298124413

Display Name ID

Page 22: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

22

Profile TypesFacebook has multiple profile types: people, pages,

places, groups, etc.Twitter has just one: people.

Affiliates?Why don’t FB affiliates score as well? Page usernames are

optional./pages/Netflix-Latinoamérica/553454298124413

Display Name ID

Page 23: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

23

Affiliates?Why don’t FB affiliates score as well? Page usernames are

optional./pages/Netflix-Latinoamérica/553454298124413

Display Name ID

Auto-generated pages also follow the same pattern!

Page 24: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

24

Future?

Focus on affiliates – unique to the domain.

Page 25: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

25

Future?

Focus on affiliates – unique to the domain.

Drill down into the various different types: (e.g.) outreach, regional, brand, business unit.

Page 26: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

26

Future?

Focus on affiliates – unique to the domain.

Drill down into the various different types: (e.g.) outreach, regional, brand, business unit.Improve ground truth: crowd-source labels.

Page 27: Linking Organizational Social Networking Profiles Research Wrap-Up – 28 August 2015 1

27

DoneObjective: develop a system to find an organization’s profiles across different social networksUsed network-specific classifiers to do soEvaluated performance using modified cross-validation

FUture

Dive deeper into affiliates, which are unique to organizations