CrowdSearch: Exploiting Crowds for Accurate Real-time Image Search on Mobile Phones

Tien Sheng Wen

Department of Electronic Engineering

Chung-Yuan Christian University, Taiwan

Tingxin Yan, Vikas Kumar, Deepak Ganesan

Department of Computer Science

University of Massachusetts, Amherst, MA 01003

{yan, vikas, dganesan}@cs.umass.edu

OUTLINE Introduction

System Architecture

Crowdsearch for Search

Crowdsearch Algorithm

Image Search Engine

System Implementation

Experimental Evaluation

Conclusions

Introduction

Image search system for mobile phones

Real-time validation

Beyond Image Search

System Performance

Payment

System ArchitectureCrowdSearch is implemented on

Apple iPhone and Linux servers.Requires three pieces of information prior to initiating

search:

(a) A image query

(b) A query deadline

(c) A payment mechanism

for human validators

Crowdsearch for Search

Amazon Mechanical Turk (AMT) Constructing Validation Tasks Minimizing Human Bias and Error Pricing Validation Tasks

Crowd Search Algorithm (1/2)

Optimizing Delay and Cost

Crowd Search Algorithm (2/2)

Delay Prediction Model

Case 1 - Delay for the first response:

Case 2 - Inter-arrival delay between responses:

The image search process contains two

major steps:

(1) Extracting features from a query image good features:

Scale-Invariant Feature Transform (SIFT)

(2) Search through database images with features of

query image.

Image Search Engine (1/2)

Image Search Engine (2/2)

A SeqTree to Predict Validation Results.

The received sequence is ‘YNY’, the two sequences

that lead to positive results are ‘YNYNY’

and ‘YNYY’. The probability that ‘YNYY’ occurs

given receiving ‘YNY’ is 0.16/0.25 = 64%

System Implementation

CrowdSearch Implementation Components Diagram

Experimental Evaluation (1/3)

Datasets Improving Search Precision Accuracy of Delay Models

Experimental Evaluation (2/3)

CrowdSearch Performance

Experimental Evaluation (3/3)

Varying user-specified deadline

Conclusions Multimedia search presents a unique challenge.

Because image search system is still far from reality.

Humans are excellent at distinguishing images, thus human validation can greatly improve the precision of image search. However, human validation costs time and money, hence we need to dynamically optimize these parameters to design an real-time and cost-effective system.