Smart Image And Video SearchEnterprise Image and Video Analytics
The problem
• Government has piles of video, analysts miss things after looking at it for 20 minutes !
• What to do?
Help!
We are collecting video and images EVERYWHERE !!!!!!!!
How to search image/video for what you are looking for
identify people suspected IED sites
changes in geospatial data
geographic features
IED circuit boardsyellow ships
trains
planesmini vans
known terrorists
activity …
The Enterprise Solution• Only piXserve by piXlogic offers a generalized enterprise
class image/video search capability• Enterprise class 64bit Java application for Windows Server
2003+• Indexes all objects in images and frames of video and also
attempts to classify those objects• Supports 90+ image or video formats• Generates a light weight XML index and leaves the original
information where it lives and can place index in RDBMS• Special focus on text in image and facial recognition• Offers a JMS based event and alerting system• Supports a web services REST API
piXlogic A New Approach
Background
Foreground
Connected Background
Face
Tree
Sky
“Seeing”
Synthesize descriptions
Discern shapes
Vectorize shapes
LocationSizeColorGeometryRelationshipsEtc.
“Comparing” Direct ComparisonsRelationshipsContext
“Understanding”
Think About Text Indexing
By three methods we may learn wisdom: First, by reflection, which is noblest; Second, by imitation, which is easiest; and third by experience, which is the bitterest. Confucius
At their most Basic they Index Text ….
By three methods we may learn wisdom….
Confucius
<Keywords>
Enter Query:
Search
Common Text Indexing Engines:
<Phrases>“three methods”“learn wisdom”“by imitation”“by experience”
</Keywords>
</Phrases>
Its All About Metadata
Text Result Set
Rank Date Description . 98% 01/01/2011 Confucius dreams .. 87% 04/12/2010 Eastern philosophy and culture84% 07/23/2009 Harvard study on ancient China 73% 02/12/2009 Fortune cookies run amok65% 03/01/2008 Medical studies calm fears in …58% 01/01/2007 Confucius dreams .. 47% 04/12/2007 Eastern philosophy and culture44% 07/23/2004 Harvard study on ancient China 43% 02/12/2000 Fortune cookies run amok35% 03/01/1999 Medical studies calm fears in …
Image/Full Motion Video Indexing
Foreground
Background
Background
WE LOOK FOR SHAPES AT THE PIXEL LEVEL
How Pixel Indexing WorksEnterprise Pixel Indexing Engine:
Indexes Key Objects …
…
Key Objects
people" id="61" <ImageIndex ImageURL="C:\Data\Sample\quebec2008-1.jpg" ImageDate=" 2 6 11016" ImageSize="57489323"><ImageText Name="1" Sz="4" X="66" Y="1" W="2" H="2" L="0" U="0" V="0" R="400852085d086608" FC="41002e002e005600" FL="2e002e002e007400" FN="2e00310031003400" <ImageText Name="2" Sz="4" X="67" Y="1" W="2" H="2" L="0" U="0" V="0" R="7c08890891089808" FC="52002e0055002e00" FL="2e0074002e002e00" FN="2e0031002e003100" <ImageText Name="3" Sz="3" X="11" Y="2" W="4" H="3" L="0" U="0" V="0" R="65018801a401" FC="48004c005a00" FL="6e002e007a00" FN="2e0031003200" …
Synthetic Metadata
Enter Query:
Search
Query By ImageEnter Text
Date Result 01/01/2011
04/12/2010
07/23/2009
02/12/2009
03/01/2008
01/01/2007
08/14/2005
Image/Video Result Set
04/14/2009
How Do We Search
• Select a database or databases to search• Select search term type– Image only– Keyword– Text in image– Text and image– Coordinates– Faces
How Does It Work
Segment Images
…
Break up images into smaller and smaller pieces
Fill In Missing Information• Uses an ontology called Notions:– Apriori information about the world– Attempts to classify objects in an image or frame
of video.– Reasons using the notions to identify objects– System has over 70 notions
PeopleTree
Helicopter
Beach
Sky
Auto-Tagging in piXserve• Add as much meaningful metadata as possible at the
time of indexing (“Auto Tagging”)• Several mechanisms:– “Name Faces”: name a person and autotag when the
person is seen in the image/video frame– “Keyword Recommendations”: add keywords based on
how similar images have been tagged in the past– “Objects-2D”: autotag for a specific item based on a
single example (can use multiple views for 3D objects) – “Notions”: autotag generic classes of items (conceptual
tagging)
Advanced Capabilities
• Text in image or video– Not OCR– Uses the same techniques as we use to find any
object in image or video frames– Tuned for languages – English, European, Some
Chinese, Japanese, Korean, Russian• Faces (biometric)– No pre-registration – just uses images– Very high precision and recall using the NIST standard
test >98%
2D Objects Library
• Topic specific filtering– Specific library can be built to filter for specific
information – as images/frames of video arrive they are
compared to this information and specific tagging / matching occurs
– E.g. light arms, ships, clothing, hair styles etc.
Text In ImageEnter Query:
Search
Query By Text in Image
HEALTH
Enter Text
RESTful API
• All Server features & functions • All User interface functions• Data access•Machine to machine integration• API Guide
Alerting Engine
• Create standing queries• When an event in an image triggers an alert
two things happen:– A JMS message is sent – to whatever program or
system designated– An email message is sent to whomever has been
designated to receive the alert
SUPPORTING MATERIALS
The Industry
• Video monitoring software packages integrated with DVR support such as Object Video, Image IQ, 3VR, Virage and others can recognize certain objects, perimeters, and activities ..
• They are security focused .. Often with some ability to search faces and possibly license tags
The Industry (Cont’d)• Other packages (COTS & GOTS) are starting to address generalized image and
video search• Numenta – offers an image search/pattern matching toolkit (not enterprise .. and
each solution must be hand crafted)• Pixsy – offers image search to match images – for internet advertising and
shopping online• IDEE Tin-Eye – offers image on the internet – more advertising based approaches• KEVA (Sierra Nevada Corp/PNNL) is a GOTS parallel processing engine for
searching video – government only and fairly early stage• Google and Facebook – Both have image and video tagging algorithms in their
general search and/or analytics suites, with some specific object search capabilities – but web focused only
• There are others .. But NONE are enterprise class generalized Google like search engines
• NERVVE
Notions/Classesartificial" id="1" white" id="2" flower" id="3" sky" id="4" gray" id="5" vegetation" id="6" lawn" id="7" skin" id="8" road" id="9" vertical" id="10" landscape" id="11" red" id="12" green" id="13" blue" id="14" aqua" id="15" yellow" id="16" gun" id="65"
smooth" id="19" circle" id="20" polygonal" id="21" sparse" id="22" face" id="23" horizontal" id="24" document" id="25" map" id="26" diagram" id="27" text" id="28" bikini" id="29" exposed body" id="30" line" id="31" adult" id="32" mountain" id="60" people" id="61" fire" id="62"
rectangle" id="37" pentagon" id="38" clutter" id="39" star" id="40" zebra" id="41" silver" id="42" pumpkin" id="43" bride" id="44" sea" id="45" snow" id="46" sand" id="47" beach" id="48" fireworks" id="49" sunset" id="50" soccer ball" id="59" airplane" id="66"
purple" id="17" orange" id="18" child" id="33" drawing" id="34" flat" id="35" triangle" id="36"flash" id="51" indoor" id="52" outdoor" id="53" near" id="54" dark" id="55" helicopter" id="56" building" id="57" car" id="58" crowd" id="63" 3D drawing" id="64"
Segment the Image or Video Frame
Indexing breaks the image up into smaller and smaller pieceswhich are more easily mathematically described
Medical Uses
Ductal CarcinomaPerspective 1
Ductal CarcinomaPerspective 2
Medical Uses
Frontal Lobe InjuryPerspective 1
Frontal Lobe InjuryPerspective 2
Physical CONOP
video/imagesource
continuousstreams or files
contentStore
db
Store inDAM system
indexingservers
direct indexing from source
indirect indexingfrom DAM system
Web Server
Web Server
Web Server
analyst
analyst
analyst
analyst
analystnetwork
Index held in memory
searching
downstreamanalytical systems
analysis
Alert Server
application
faces
Logical CONOPS 1Watching The Border
continuousvideo/image
source
automatic continuous
indexing
indexal
erts
reporting
metadata• name• watch list• vehicle make• vehicle model• tag state• tag number
analysts
downstreamanalyticalsystems
correlationFilter
indexes
vehicles
license plates
Filter indexesFilter
indexes
Conceptual CONOPS 1Watching The Border
vehicle border crossing
cameras
Tag - state & text extracted Vehicle - make/model extractedFace - biometrics
vehicle
tag face
Indexes-watch list-vehicle types-state license plates
-State: Florida-Tag: SAMPLE
-Daniel San Diego-U.S. Most Wanted
-Make: Honda-Model: Civic
NET EMULATED UAV
use video filespiXlogic index
video and searches
piXlogic matches and
clips video on target
event
piXlogic apitransfers clip
to flumedirectory
flume transmits matches over
MITRE network emulator
1
activity
2 3 4 5
RADAR INFRARED VIDEO VIDEO IMAGERY
Query:• Buildings• Ships• Aircraft
TNT - CONOPS
Alert Results
network
pixserve
flume
net emulator
commsLo
cal N
etw
ork
FIELD ANALYSTS
Recommended
