Embed Size (px)
DESCRIPTION
Contextual Image Search. Wenhao Lu , Jingdong Wang , Xian- Sheng Hua , Shengjin Wang , Shipeng Li Tsinghua University, Beijing, P. R. China, Microsoft Research Asia, Beijing, P. R. China,. MM 2011. Traditional image search. MM 2011. Contextual image search. company. iPhone. - PowerPoint PPT Presentation
Citation preview
Contextual Image SearchContextual Image Search
Wenhao LuWenhao Lu , Jingdong Wang , Xian-Sheng Hua, Shengjin Wang , Shipeng Li , Jingdong Wang , Xian-Sheng Hua, Shengjin Wang , Shipeng Li
Tsinghua University, Beijing, P. R. China, Tsinghua University, Beijing, P. R. China,
Microsoft Research Asia, Beijing, P. R. China, Microsoft Research Asia, Beijing, P. R. China,
Wenhao LuWenhao Lu , Jingdong Wang , Xian-Sheng Hua, Shengjin Wang , Shipeng Li , Jingdong Wang , Xian-Sheng Hua, Shengjin Wang , Shipeng Li
Tsinghua University, Beijing, P. R. China, Tsinghua University, Beijing, P. R. China,
Microsoft Research Asia, Beijing, P. R. China, Microsoft Research Asia, Beijing, P. R. China,
MM 2011
MM 2011
Traditional image search
3
Contextual image search
company
iPhone
MM 2011
4
Commercial image search engine
query: Funny George Bush
MM 2011
5
System overview
Text input
MM 2011
6
System overview
Image input
2. Annotating images by mining search result(2008)
MM 2011
7
Image input example
Input image
Candidate queries:“Blue mosque”, “Istanbul”, “Turkey travel”, “Istanbul turkey”
The mosque is one of several mosques known as the BlueMosque for the blue tiles adorning the walls of its interior
Search Result
Similar image
MM 2011
8
Contextual Image Search WithText Input
1. Context Capturing
visual contexts: vision-based page segmentation algorithm (VIPS)
textual contexts: page title / document title local context
MM 2011
9
Contextual Image Search WithText Input
2. Contextual Query Augmentation
Goal: remove possible ambiguities Augmented query = query + textual context
Candidate augmented query
evaluate the relevance betweenthe context and augmented query (Okapi BM25)MM 2011
10
3. Image Search by Text: Microsoft Bing image search4.Contextual Reranking:
Combine textually and visually context
MM 2011
Contextual Image Search WithText Input
11
Quantitative Evaluation
0.95
0.65
MM 2011nDCG curves
12
YouPivot: Improving Recall with Contextual Search
YouPivot: Improving Recall with Contextual Search
SIGCHI 2011
Joshua Hailpern 1, Nicholas Jitkoff 2, Andrew Warr 2Joshua Hailpern 1, Nicholas Jitkoff 2, Andrew Warr 2 Karrie Karahalios 1,Robert Sesek 3, Nikita Shkrob 4Karrie Karahalios 1,Robert Sesek 3, Nikita Shkrob 4
1 University of Illinois Urbana, IL USA 61801 1 University of Illinois Urbana, IL USA 61801 2 Google Mountain View, CA USA 940432 Google Mountain View, CA USA 94043
3 Boston University Boston, MA USA 022153 Boston University Boston, MA USA 022154 University of Waterloo 4 University of Waterloo Waterloo, Ontario, Canada N2L 3G1Waterloo, Ontario, Canada N2L 3G1
Joshua Hailpern 1, Nicholas Jitkoff 2, Andrew Warr 2Joshua Hailpern 1, Nicholas Jitkoff 2, Andrew Warr 2 Karrie Karahalios 1,Robert Sesek 3, Nikita Shkrob 4Karrie Karahalios 1,Robert Sesek 3, Nikita Shkrob 4
1 University of Illinois Urbana, IL USA 61801 1 University of Illinois Urbana, IL USA 61801 2 Google Mountain View, CA USA 940432 Google Mountain View, CA USA 94043
3 Boston University Boston, MA USA 022153 Boston University Boston, MA USA 022154 University of Waterloo 4 University of Waterloo Waterloo, Ontario, Canada N2L 3G1Waterloo, Ontario, Canada N2L 3G1
13
SIGCHI 2011
“what was that website I was looking at when Yesterday by The Beatles was last playing? ”
Why Contextual Search?
14
SIGCHI 2011
Improving Recall ?
Contextual cues: temporally related activities Cognitive science: leveraging context improves speed and accuracy in recall tasks
Loses car keys: “retrace your steps since the last time you know you had them.”
15
SIGCHI 2011
Interface
16
SIGCHI 2011
Features and Functionality
?title?domain?when
Mr. Richfield
graphic designer
SarahWhere is the website?
17
SIGCHI 2011
Features and Functionality
Websites Layout
18
Context Sensitive Paraphrasing with a Single Unsupervised
Classifier
Context Sensitive Paraphrasing with a Single Unsupervised
Classifier
Michael Connor and Dan RothMichael Connor and Dan RothDepartment of Computer ScienceDepartment of Computer Science
University of Illinois at Urbana-ChampaignUniversity of Illinois at Urbana-Champaign
Michael Connor and Dan RothMichael Connor and Dan RothDepartment of Computer ScienceDepartment of Computer Science
University of Illinois at Urbana-ChampaignUniversity of Illinois at Urbana-Champaign
ECML2007
Context Sensitive ParaphrasingContext Sensitive Paraphrasing
19
‘X commanded Y’ ‘X spoke to Y’
When can ‘speak to’ replace ‘command’ in the original sentence and not change the meaning of the sentence?
ECML2007
Polysemous Nature of VerbsPolysemous Nature of Verbs
20
- command
ECML2007
Definition of ContextDefinition of Context
21
derived from parsing informationderived from parsing information subject and and object of the verbof the verb
Marshall Formby of Plainview suggested a plan to fill byappointment future vacancies in the Legislature andCongress, eliminating the need for special elections.
Local Context: obj:plan , subj:NE:PER
ECML2007
Modeling Context Sensitive Paraphrasing
Modeling Context Sensitive Paraphrasing
22
1. Context Sensitive Decisions
v: original verbu: substitute verb
type c contextual features of v/u
obj:plan , subj:NE:PER
creating, breaking or presenting
c: sub / obj
ECML2007
23
Unsupervised Training: Bootstrapping Local Classifiers
ECML2007
Experimental ResultsExperimental Results
AQUAINT Corpus (News Articles)
• test set has 721 S, v, u examples with 57 unique v verbs and 162 unique u.
(random selection of polysemous verbs that occur in WordNet 2.1)
AQUAINT Corpus (News Articles)
• test set has 721 S, v, u examples with 57 unique v verbs and 162 unique u.
(random selection of polysemous verbs that occur in WordNet 2.1)
24
ECML2007
Experimental ResultsExperimental Results
25
ECML2007
Hierarchical summarization for delivering information to mobile devices
Hierarchical summarization for delivering information to mobile devices
26
SIGIR 2007
Jahna Otterbacher a,*, Dragomir Radev b, Omer Kareem bJahna Otterbacher a,*, Dragomir Radev b, Omer Kareem ba University of Cyprus, Nicosia, Cyprusa University of Cyprus, Nicosia, Cyprus
b University of Michigan, Ann Arbor, MI 48109, United Statesb University of Michigan, Ann Arbor, MI 48109, United States
27
SIGIR 2007
17歲天才寫 APP 幫新聞摘要精華 (2012/11/3)
一名英國的 17歲男生，設計出一款可以幫新聞摘要出「精華版」的 APP-Summly，這款應用程式在業界廣受好評。
Recent News
28
SIGIR 2007
Limitation of Mobile Device
small screens
constrained wireless bandwidth
29
SIGIR 2007
Architecture of summarization method
30
SIGIR 2007
Sentence scoring
Centroid value: the importance of the sentence
:TF*IDF values of word w in
Positional value: More weight is given to sentences that appear earlier in the document than those that appear later.(news articles)
:first sentence centroid value
31
SIGIR 2007
Sentence scoringSentence scoring First sentence overlap value: The first sentence in a text is likely to convey information about its main theme or topic.
32
SIGIR 2007
Hierarchical nestingHierarchical nesting
33
SIGIR 2007
ExperimentExperiment
─ 39 subjects in the experiment(student studying information and computer science)─10 articles 10 questions
A Maximum Entropy Web Recommendation System:
Combining Collaborative and Content Features
A Maximum Entropy Web Recommendation System:
Combining Collaborative and Content Features
34
SIGKDD 2005
Xin Jin, Yanzan Zhou, Bamshad MobasherXin Jin, Yanzan Zhou, Bamshad MobasherCenter for Web IntelligenceCenter for Web Intelligence
School of Computer Science, Telecommunication, School of Computer Science, Telecommunication, and Information Systemsand Information Systems
DePaul University, Chicago, Illinois, USADePaul University, Chicago, Illinois, USA
35
SIGKDD 2005
Web Recommendation System
36
SIGKDD 2005
─ Goal: help users locate information on the Web
About Web Recommendation
─ Input: Web users’ navigation or rating data content features of the items
─ Approach:
Data mining or Machine Learning to discover usage patterns that represent aggregate user models.
37
SIGKDD 2005
Maximum Entropy Recommendation Model
P( 飛行 | fly) + (P( 搭機 | fly) + P( 蒼蠅 | fly) = 1
─ Maximum Entropy:
s.t. P( 飛行 | fly) + (P( 搭機 | fly) = 4/5 P( 蒼蠅 | fly)=1/5
38
SIGKDD 2005
Maximum Entropy Recommendation Model
─ Offline: 1.accept constraints to form the model 2.estimate the model parameters
─ Online: 1.reads an active session 2.runs the recommendation algorithm
39
SIGKDD 2005
Maximum Entropy Recommendation Model
─ Distribution form:
: a page being visited next
: user’s recent navigational history
: weight of
=
40
SIGKDD 2005
Maximum Entropy Recommendation Model
─ identification of features (navigation data):
if ( )
41
SIGKDD 2005
Maximum Entropy Recommendation Model
─ identification of features (rating data):
select highly correlated item pairs
42
SIGKDD 2005
Maximum Entropy Recommendation Model
─ identification of features (content information):
Use Latent Dirichlet Allocation(LDA) to find the class ofeach item.
In a movie site, high ratings for “Indiana Jones”and “Air Force One” may suggest that a user is a Harrison Ford’s fan and enjoys Action-Adventure movies.
43
SIGKDD 2005
Maximum Entropy Recommendation Model
─ identification of features (content information):
44
SIGKDD 2005
Experiment─ Realty data, 24,000 user sessions from 3,800 unique users
![Contextual Based Image Inpainting: Infer, Match and Translate€¦ · Contextual-based Image Inpainting: Infer, Match, and Translate Yuhang Song*1[0000−0003−4990−2964], Chao](https://img.dokumen.tips/doc/110x75/6005fd10d8769943c60a4ddc/contextual-based-image-inpainting-infer-match-and-translate-contextual-based-image.jpg)
