b l hA ti Q S i f M bil L ti S hActive Query Sensing for Mobile Location SearchActive Query Sensing for Mobile Location SearchActive Query Sensing for Mobile Location SearchActive Query Sensing for Mobile Location SearchActive Query Sensing for Mobile Location SearchQ y gl h h hFeli X Y Ron ron Ji Shih F ChanFelix X Yu Rongrong Ji Shih Fu ChangFelix X Yu Rongrong Ji Shih‐Fu ChangFelix X. Yu    Rongrong Ji Shih Fu Changg g g

l d d l d b l b k dDigital Video and Multimedia Laboratory Columbia University 10027 New York United StatesDigital Video and Multimedia Laboratory Columbia University 10027 New York United StatesDigital Video and Multimedia Laboratory, Columbia University, 10027, New York, United Statesg y, y, , ,

Abstract P bl J tifi ti System Architecture E i tAbstract Problem Justification System Architecture ExperimentAbstract Problem Justification System Architecture Experimenty pWh h fi f il fi d h i h ( 50%• When the first query fails to find the right target (up to 50%• When the first query fails to find the right target (up to 50%q y g g ( plikelihood) how should the user form his/her search strategylikelihood), how should the user form his/her search strategylikelihood), how should the user form his/her search strategyin the subsequent interaction?in the subsequent interaction?in the subsequent interaction?

• We propose a novel Active Query Sensing system to suggest• We propose a novel Active Query Sensing system to suggestWe propose a novel Active Query Sensing system to suggestthe best way for sensing the surrounding scenes while formingthe best way for sensing the surrounding scenes while formingthe best way for sensing the surrounding scenes while formingth d f l ti hthe second query for location searchthe second query for location search.q y

• Most locations are recognizable only with a subset of the• Most locations are recognizable only with a subset of theMost locations are recognizable only with a subset of theie sviewsviews.

Each location has a unique subset of preferred views for• Each location has a unique subset of preferred views forEach location has a unique subset of preferred views forg itirecognitionrecognition.

Location Dependence: different locations have different• Location Dependence: different locations have differentLocation Dependence: different locations have differentd f “diffi lt ”degrees of “difficulty”degrees of difficulty .g yLocations of the same search difficulty do not significantly• Locations of the same search difficulty do not significantlyLocations of the same search difficulty do not significantlyl t t thcluster together Failure rates over successive query iterationscluster together. Failure rates over successive query iterations.g Failure rates over successive query iterations.

There is no single dominant view that can successfully• There is no single dominant view that can successfully W i l t th fi t ith d l h i i lThere is no single dominant view that can successfully • We simulate the first query with a randomly chosen viewing anglei ll l ti

We simulate the first query with a randomly chosen viewing angle.recognize all locations O l 47% f h d i d l i i 53%recognize all locations. • Only 47% of the random query images succeed resulting in a 53%g • Only 47% of the random query images succeed, resulting in a 53%

A li ti S iy q y g , g

Application Scenario failure rate after the first queryApplication Scenario failure rate after the first query.• Offline analysis: to discover the best query view(s) for each location indexed by the system

q yU I t f • Offline analysis: to discover the best query view(s) for each location indexed by the system. • We evaluate the performance of reducing the failure rates inUser InterfaceNAVTEQ 0 3M NYC D t S t

Offline analysis: to discover the best query view(s) for each location indexed by the system. • We evaluate the performance of reducing the failure rates inUser InterfaceNAVTEQ 0 3M NYC Data Set We evaluate the performance of reducing the failure rates inNAVTEQ 0.3M NYC Data Set • Online process: to estimate the most likely view of the first query (which has failed); to suggest optimal view change to the user s bseq ent q eries b sing different acti e q er strategiesQ • Online process: to estimate the most likely view of the first query (which has failed); to suggest optimal view change to the user. subsequent queries by using different active query strategies.Online process: to estimate the most likely view of the first query (which has failed); to suggest optimal view change to the user. subsequent queries by using different active query strategies.h d i f b 300 000 i f 0 000 l i Th f i hi d b th li b d AQS h i• The data set consists of about 300 000 images of 50 000 locations • The performance gain achieved by the saliency based AQS scheme is• The data set consists of about 300,000 images of 50,000 locations

ffli l iThe performance gain achieved by the saliency based AQS scheme is, g ,

h ll d b h A Q Offline AnalysisOnline Analysis i i i 12% f l ddi i lin Manhattan collected by the NAVTEQ street view imaging system Offline AnalysisOnline Analysis quite impressive – 12% error rate after only one additional queryin Manhattan collected by the NAVTEQ street view imaging system. Offline AnalysisOnline Analysis quite impressive – 12% error rate after only one additional query.y Q g g y q p y q y• The offline measures of saliency can also be used to “grade” the• The offline measures of saliency can also be used to grade they g

searchability of each location indicating the locations where thesearchability of each location, indicating the locations where thesearchability of each location, indicating the locations where thet f b tlsystem performs more robustly.system performs more robustly.

First queryFirst query q y

l i d kview Conclusions and Future Worksview  Conclusions and Future Works( d) Conclusions and Future Works(estimated)(estimated)( )W h d l d l A ti Q S i t th t ti l• We have developed a novel Active Query Sensing system that activelyWe have developed a novel Active Query Sensing system that actively

h b if/ h h fi i l f ilsuggests the best query strategy if/when the first visual query failssuggests the best query strategy if/when the first visual query fails.gg q y gy q yf ffP bl F l ti • This the first effort in actively guiding users to achieve moreProblem Formulation • This the first effort in actively guiding users to achieve moreProblem Formulation y g g

MostGeographical Distribution of NAVTEQ Data Set satisfactory experience in using mobile visual searchMost Geographical Distribution of NAVTEQ Data Set satisfactory experience in using mobile visual search.g p QA ti U i lik l t t k “j k i ” ith h f

satisfactory experience in using mobile visual search.salient viewAssumption: User is unlikely to take “junk image” with no hope for W d l li t b d di t ib ti tsalient viewAssumption: User is unlikely to take junk image with no hope for • We develop saliency measurement based on score distributions top y j g p

fi di th t t t Th fi t f l b dWe develop saliency measurement based on score distributions to

finding the true target The first query even unsuccessful can be used di t th b t f h i d th h diffi lt ffinding the true target. The first query, even unsuccessful, can be used predict the robustness of each query view and the search difficulty ofg g q y“ b ” t d th l ti

predict the robustness of each query view and the search difficulty ofas “probe” to narrow down the solution space h l ias probe to narrow down the solution space. each locationp p each location.

S ti t i ht 90 d h f ll h l b hSuggestion: turn right 90 degrees • In the future we will extend the idea to multi view object searchSuggestion: turn right  90 degrees • In the future, we will extend the idea to multi‐view object search., j

Th t f ti ti th fi t i• There are two ways for estimating the first query viewThere are two ways for estimating the first query view. y g q yExamples of “junk” images Examples of query imagesExamples of  junk  images Examples of query imagesa p es o ju ages p q y g

A id l di ib i i h h h i l i• An ideal score distribution is the one that has maximal separation• A subsequent query should maximize the discriminability • An ideal score distribution is the one that has maximal separation• A subsequent query should maximize the discriminability pq q y y( ) between the scores of the positive results and those of the negative(uncertainty reduction) over candidate locations narrowed down by between the scores of the positive results and those of the negative

C fi ti f C (uncertainty reduction) over candidate locations, narrowed down by p gConfiguration of Cameras Whi h i ill b th b t ?

( y ) , yh l d b k b l i h b i onesConfiguration of Cameras Which view will be the best query?the query already been taken by selecting the best query view ones.Train view classifiers offline Which view will be the best query?the query already been taken, by selecting the best query view. ones.Train view classifiers offline yq y y , y g q y

• Predict online search performance b sing the reference image of• Predict online search performance by using the reference image ofPredict online search performance by using the reference image ofh {l ti i } t t i th l tieach {location view} to retrieve the same location:each {location, view} to retrieve the same location:

AcknowledgementAcknowledgementT i l h bil l i h i ll Acknowledgement• To simulate the mobile location search scenarios we manually Wi h h i h i l d il f d h bg• To simulate the mobile location search scenarios, we manually • Without showing mathematical details we found the above process, y

d i f G l S Vi i 226 d l h• Without showing mathematical details, we found the above process

cropped queries from Google Street View in 226 randomly choseng , p

b ll i t d b View alignment based on the image matchingcropped queries from Google Street View in 226 randomly chosen can be well approximated by View alignment based on the image matchingpp q g yl i d b h b i d i NYC W th k NAVTEQ f idi g th NYC i g d t t D Xi Ch

can be well approximated by e a g e t based o t e age atc glocations covered by the above mentioned routes in NYC We thank NAVTEQ for providing the NYC image data set Dr Xin Chen

pp ylocations covered by the above mentioned routes in NYC. We thank NAVTEQ for providing the NYC image data set, Dr. Xin Chen y

d D J ff B h f th i g h l d T gt Zh g f • We use a majority voting mechanism to estimate the optimal viewF h l ti i i g d f i i g and Dr. Jeff Bach for their generous help and Tongtao Zhang for • We use a majority voting mechanism to estimate the optimal view• For each location six query images are cropped from viewing and Dr. Jeff Bach for their generous help, and Tongtao Zhang for j y g pFor each location, six query images are cropped from viewingd ig i g th bil i t fangle change and suggest the user turns to the most salient viewgl i il t th i i t ti d i th d t b Thi designing the mobile interfaceangle change and suggest the user turns to the most salient view.angles similar to the view orientations used in the database This designing the mobile interface.angle change and suggest the user turns to the most salient view.angles similar to the view orientations used in the database. This Llt i 1 356 i ith l d d t th l ti t • is the candidate locations narrowed down by the first queryLresults in 1 356 images with angles and ground truth locations tags. • is the candidate locations narrowed down by the first query.NLresults in 1,356 images with angles and ground truth locations tags. is the candidate locations narrowed down by the first query.N

RESEARCH POSTER PRESENTATION DESIGN © 2011RESEARCH POSTER PRESENTATION DESIGN © 2011

P t P t tiwww.PosterPresentations.com