Automatic Photo Automatic Photo AnnotatorAnnotator

Bryan KlimtBryan Klimt

May 10, 2005May 10, 2005

Photo AnnotatorPhoto Annotator

Photos must be labeled with keywords Photos must be labeled with keywords to enable effective searchto enable effective search

Hand labeling photos is a tedious Hand labeling photos is a tedious processprocess

Automatic Photo Annotator labels Automatic Photo Annotator labels incoming photos based on statistical incoming photos based on statistical learninglearning

Annotator then learns from user Annotator then learns from user corrections corrections

Most common keywordsMost common keywords

Colors or PatternsColors or Patterns sky, water, trees, cloud, hazysky, water, trees, cloud, hazy

PeoplePeople bryan, amy, shyambryan, amy, shyam

LocationsLocations china, italy, pisa, tiantan, florence,china, italy, pisa, tiantan, florence,

firenze, beijing, luccafirenze, beijing, lucca

Types of ClassifiersTypes of Classifiers

Color-based methodsColor-based methods color histogramscolor histograms cumulative color histogramscumulative color histograms

Face-based methodsFace-based methods Face detectionFace detection Face recognitionFace recognition

Time-based methodsTime-based methods k Nearest Neighborsk Nearest Neighbors

Color-based PerformanceColor-based Performance

Cumulative ColorCumulative Color

Why does “color” fail?Why does “color” fail?

Photos that are Photos that are mislabeledmislabeled (because (because the feature is not the focus of the the feature is not the focus of the photo).photo).

Photos with similar color-patterns, Photos with similar color-patterns, but of completely different subjects.but of completely different subjects.

Hazy?Hazy?

Water?Water?

Face-based PerformanceFace-based Performance

FacesFaces

Faces?Faces?

Time-based ResultsTime-based Results

Final ResultsFinal Results

ConclusionsConclusions

Time-based annotator performed bestTime-based annotator performed best

Color-based annotator has inherent Color-based annotator has inherent limitslimits

Face-based annotator may become Face-based annotator may become important with more training dataimportant with more training data

Future WorkFuture Work

Hybrid MethodsHybrid Methods

Pattern-based annotatorsPattern-based annotators

Improving face recognizer with Improving face recognizer with unlabeled data?unlabeled data?