View
226
Download
3
Category
Tags:
Preview:
Citation preview
Descriptive Semantic Image Retrieval
David Norton
Derral Heath
Motivation
Retrieve an image based on a descriptive query: “Find me an image that is red, dark, scary, and
beautiful”
Content-Based Image Retrieval
Retrieve an image strictly from image features color texture shape
General semantic based image retrieval is hard “Find me a picture of a piranha”
Emotional Semantic Image Retrieval
Query images matching emotional words or word-pairs “Find me a happy picture”
Usually adjectives
Usually a small orthogonal subset of terms
Query via single words (or pairs)
Descriptive Semantic Image Retrieval
Open to all descriptive words
Query via any number of words “Find me an image that is red, dark, scary, and
beautiful”
Three Components
Extraction of image features
Semantic representation of image
Mapping between visuals and language
Extraction of Image Features
Color (12) Average RGB values Color count
Texture (50) Entropy
Shape (10) Eccentricity
Extraction of Image Features
Average R Average G Average B Hue Count Hue Percent Entropy Eccentricity
114.0887 99.0219 118.648 20 0.1332 8.924632 0.109940296
113.5899 98.5232 118.145 18 0.1411 8.200488 0.101902564
113.6192 98.5358 118.153 20 0.1511 8.015762 0.082569348
Extraction of Image Features
Average R Average G Average B Hue Count Hue Percent Entropy Eccentricity
146.4833 75.8135 1.547 5 0.5259 7.805696 0.064152
145.9641 75.2931 1 4 0.5317 7.654775 0.073264
146.0377 75.3628 1.054 4 0.5496 7.502806 0.054083
Semantic Representation of Image
How do we obtain a description?
What is a descriptive word?
What are the features?
User Input Interface
Bright has 11 Senses 1. (17) bright -- (emitting or reflecting light readily or in large amounts; "the sun was bright and hot"; "a
bright sunlit room") 2. (6) bright, brilliant, vivid -- (having striking color; "bright dress"; "brilliant tapestries"; "a bird with vivid
plumage") 3. (5) bright, smart -- (characterized by quickness and ease in learning; "some children are brighter in one
subject than another"; "smart children talk earlier than the average") 4. (3) bright -- (having lots of light either natural or artificial; "the room was bright and airy"; "a stage bright
with spotlights") 5. (1) bright, burnished, lustrous, shining, shiny -- (made smooth and bright by or as if by rubbing;
reflecting a sheen or glow; "bright silver candlesticks"; "a burnished brass knocker"; "she brushed her hair until it fell in lustrous auburn waves"; "rows of shining glasses"; "shiny black patents")
6. (1) bright -- (splendid; "the bright stars of stage and screen"; "a bright moment in history"; "the bright pageantry of court")
7. undimmed, bright -- (not made dim or less bright; "undimmed headlights"; "surprisingly the curtain started to rise while the houselights were still undimmed")
8. bright, brilliant -- (clear and sharp and ringing; "the bright sound of the trumpet section"; "the brilliant sound of the trumpets")
9. bright -- (characterized by happiness or gladness; "bright faces"; "all the world seems bright and gay") 10. bright, shining, shiny, sunshiny, sunny -- (abounding with sunlight; "a bright sunny day"; "one shining
morning"- John Muir; "when it is warm and shiny") 11. bright, promising -- (full or promise; "had a bright future in publishing"; "the scandal threatened an
abrupt end to a promising political career")
Narrowing Down the Feature Space
Interface: Adjectives from WordNet
Restrict characters
Reduce available senses
Narrowing Down the Feature Space
Post Processing: Use Synsets
Frequent synsets
Fit ORM ontology lexicons
Image ORM Ontology
Mapping between visuals and language
Series of Neural Networks
Bayes Net
Fuzzy Logic
Evaluation
Let machine label images
Let humans label images
Let different humans evaluate machine and human labels
Compare evaluations
Related Work
Aesthetic Visual Quality Assessment of Paintings (2009) Congcong Li and Tsuhan Chen
Labeled impressionistic style landscape paintings as ‘high’ or ‘low’ quality using machine learning.
Algorithmic Inferencing of Aesthetics and Emotion in Natural Images: An Exposition (October 2008) Ritendra Datta, Jia Li, and James Z. Wang
Overview of research involving predicting the quality class, score, and emotional label of photographs.
Related Work
A Survey on Emotional Semantic Image Retrieval (2008) Weining Wang and Qianhua He
Surveys ongoing Emotional Semantic Image Retrieval research.
Image Retrieval by Emotional Semantics: A Study of Emotional Space and Feature Extraction (October 2006) Wang Wei-ning, Yu Ying-lin, and Jiang Sheng-ming
Labeled paintings with 12 emotional word pairs. Psychological research involved in choosing word pairs.
Further Motivation
Augment the study of human perception and cognition.
Establish a linguistic-visual foundation for an artificially creative artist.
Questions?
Recommended