Embed Size (px)
Citation preview
A picture’s worth a thousand hashtags:
How image recognition will power the future of analyticsDavid BerkowitzChief Strategy [email protected]@sysomos / @dberkowitz
About this presentation
This talk was presented to Marketing Week Live in London in March 2017. A Texan version of this was delivered to W2O Group’s Pre-Commerce Summit during SXSW that month. If you prefer Frito pie to bangers and mash, I will gladly send you the W2O edition.
Sources for information shown in slides are presented as links in the bottom-left corner. Further details about sources, where applicable, are also included in the notes field when downloading this presentation.
To share your feedback or discuss this further, please contact me at [email protected].
Thank you.
David
And now for something completely standard: an agenda
• A brief history of nearly everything visual search
• Why visual search matters
• How Google, Pinterest, and others are deploying it
• How marketers can use it
• Numerous gratuitous references to all things British
?
Not British, but just a Chunnel ride away
How would you #HASHTAG
the Mona Lisa?
#YOLO
Mona Lisa #Hashtags
• #MonaLisa
• #art
• #painting
• #woman
• #lady
• #smile
• #smug
• #Italian
• #DaVinci
• #epic
• #outdoors
• #masterpiece
• #sky
• #France
• #LaGioconda
• #badhairday
• #beautiful
• #Louvre
• #Renaissance
• #portrait
This is a futile exercise. One can’t simply capture the Mona Lisa in hashtags. It points to the need for better ways to
identify and analyze visual content. Text and hashtags alone don’t cut it.
Textual Media
Visual Media
We are here.
Drop in a bucket visual
The Visual Data GapRecall how pictures are worth a thousand words? There is
so much more data in images (‘the sun’) than there is in text (represented metaphorically by the planets).
A Short History of Nearly Everything Visual
Dual British/ American citizen
A challenge of Shakespearean proportions:
“His reasons are as two grains of wheat hid in two bushels of chaff: you shall seek all day ere you find them, and when you have them, they are not worth the search.”-Bassanio, The Merchant of Venice
Number of object categories out there
15,000There are 15,000
object categories
Source: IEEE
Source: Computer Vision by Richard Szeliski
For fun, I included a few examples of early attempts at machine-
powered object recognition.
Source: Computer Vision by Richard Szeliski
An inflection point waiting to happen
Jason Goldberg, Razorfish: “I’m strongly bullish on visual search. It solves a real problem consumers have… In the not-too-distant future, it’ll become a heavily used mainstream feature. I think the inflection point is at least a year away, but not two years." We’re approaching the
inflection point, but it has taken longer than expected.
This report is from November 2014.
Scanning products = Cool
Facial scanning = Creepy
This is simply an enlarged, cropped
version of the highlights from the
previous chart.
You can’t always get what you want (with text search)
• 74% of consumers say text-based keyword searches are inefficient for helping them find the right products online
• 67% of consumers say quality of product images is very important in selecting and purchasing products
• 90% of information transmitted to the brain is visual
• Visual information is processed 60,000x faster than text
Source: Slyce
The dress that inspired Google Image Search in 2001
The dress that inspired Google Image Search in 2001
“…People wanted more than just text. This first became apparent after the 2000 Grammy Awards, where Jennifer Lopez wore a green dress that, well, caught the world’s attention. At the time, it was the most popular search query we had ever seen. But we had no surefire way of getting users exactly what they wanted: J Lo wearing that dress. Google Image Search was born.”-Eric Schmidt, Executive Chairman, Google
Source: Project Syndicate
Solving the Clarissa problem My wife gave me this reference. As a kid, she always wanted to identify and shop for whatever Clarissa wore.
Applications for image recognition
Source: Facebook
Source: The Verge
An introductory framework for visual search
Layers of image recognition
A Deep Learning algorithm is presented with the images made up of simple pixels.
The algorithm discovers simple regularities that are present across many/all images, like curves & lines.
The algorithm discovers how these regularities are related to form higher-level concepts
The system gains a high-level understanding of the original image… all automatically
Source: GazeMetrix
A framework for visual search
Scene
Identification Intelligence
ObjectIdentification Intelligence
LogoIdentification Intelligence
ImageIdentification Intelligence
CategoryIdentification Intelligence
This notes some of the most important processes within visual search. Also note that identification and intelligence
are two separate approaches. Examples follow.
What follows is an example using a real photo from Agnes, a
Chinese tourist to the UK.
Here’s her photo. In each subsequent slide, you can see how the framework
plays out and a sample finding that can be derived. Note the intelligence
examples that follow are for illustrative purposes only; feel free to cite the framework, but not the data itself.
Category Identification:This is food and drink
Category Intelligence:7.2% of images posted at museums include food or
drinks
Logo Identification:There is a Fanta logo, and
the text in the top-right says Starbucks
Logo Intelligence:Fanta logos are rarely paired with Starbucks; Fanta logos
are most often seen with Coca-Cola and Adidas
Object Identification:This looks like fish and chips with a can of Fanta Lemon
Object Intelligence:Fanta Lemon is the fourth most popular soda when paired with fish & chips
Image Identification:This is the same photo that
appears on Agnes_Cin’s Flickr and public Facebook
pages
Image Intelligence:This image hasn’t been
shared in any media outlets and hasn’t been shared
publicly
Scene Identification:This photo seems to be taken
outdoors during the day
Scene Intelligence:94% of photos at the British Museum are shot indoors,
compared to 87% of museum photos worldwide
Spotlight: Google
This section is drawn (no pun intended) from quickdraw.withgoogle.com.
Spotlight: Pinterest
Pinterest: one of the world’s biggest search engines
• 150 million monthly users• 75 billion pins• 2 billion searches/month• 97% of searches are unbranded
Source: Pinterest
Pinterest: search pins from real-world images
Source: Pinterest
Examples from Pinterest’s new Visual Discovery follow. In the downloadable
version of this talk, the next few visuals play as GIFs, and you can read more at
the source below.
Pinterest Visual Discovery
Source: Pinterest
Browse images and buy from them
Source: Pinterest
I really liked the circled image here: Pinterest draws a connection between Big Ben and the Blue Mosque. Such
errors are often more revealing.
Spotlight: Emerging Technologies
Toys R Us offers Slyce image detection for its catalog
Source: Slyce
Visual search to complement textual search
eMarketer: Do you think [visual search] will replace some types of searches, or do you think it will augment existing searches?
Gierhart: It will probably augment. It’s adding a new utility to what was there before... There will still be contexts for both.
Source: eMarketer (see a related video on YouTube)
Blippar: scan images for surprises about Planet Earth
Source: Blippar
Source: Houzz
Houzz has a Visual Match offering akin to
Pinterest.
TheTake uses AI to identify products, locations in video
Source: TheTake
What’s Possible with Image Analytics
The images that follow are sample reports drawn from Sysomos. The data is again for illustrative purposes. Reach out if you want to dive deeper into any of
this.
Visual analysis: understanding visual characteristics
Logo recognition
Object recognition
Scene recognition
Food recognition
Color detection
OCR: Search text within images
Visual analysis: understanding visual characteristics
Logo recognition
Object recognition
Scene recognition
Food recognition
Color detection
OCR: Search text within images
Visual analysis: understanding visual characteristics
Logo recognition
Object recognition
Scene recognition
Food recognition
Color detection
OCR: Search text within images
Audience analytics show growth and spikes
Brand affinities can highlight cross-promotion opportunities
Identify most popular objects, scenes
B2B applications for visual search
Creative optimization
Influencer marketing
Rights manageme
nt
Crisis manageme
nt
Partnership ideation
Competitive intelligence
Customer service
Any questions?
So long, and thanks for all the fish.Let’s take tea!
David BerkowitzChief Strategy [email protected]@sysomos / @dberkowitz
