A Graphical User Interface for a Fine-Art Painting Image Retrieval System October 15-16, 2004 October 15-16, 2004

A Graphical User A Graphical User Interface for a Fine-Interface for a Fine-Art Painting Image Art Painting Image Retrieval SystemRetrieval System

October 15-16, 2004October 15-16, 2004

Multimedia Information Retrieval 2004

IntroductionIntroductionStudents of art history learnStudents of art history learn

three primary skills:three primary skills:

Formal analysisFormal analysis ComparisonComparison ClassificationClassification

How can computer science How can computer science

contribute to the contribute to the developmentdevelopment

of these skills?of these skills? Girl with a Pearl Earring, Jan Vermeer, 1665


Working HypothesisWorking Hypothesis

An Interactive Indexing and Image An Interactive Indexing and Image Retrieval System (IIR) for fine-art Retrieval System (IIR) for fine-art paintings can aid students in these paintings can aid students in these endeavors by providing:endeavors by providing: a mathematical summarization of an imagea mathematical summarization of an image a measurable basis for comparing two a measurable basis for comparing two

imagesimages an elementary way to classify an image an elementary way to classify an image

relative to those in a database relative to those in a database


Previous WorkPrevious Work

We synthesize the goals of two research areas:We synthesize the goals of two research areas: Classification of paintings which often requires Classification of paintings which often requires

special images (brush stroke detection) or special images (brush stroke detection) or features with little semantic relevance to art features with little semantic relevance to art studentsstudents

Image retrieval which aims to bridge the semantic Image retrieval which aims to bridge the semantic gapgap

Can we find a feature set that satisfies the Can we find a feature set that satisfies the objectives of objectives of

both areas while providing analytically relevant data both areas while providing analytically relevant data to students?to students?


System OverviewSystem Overview

The system consists of two major The system consists of two major components:components:

Image Database Image Database stores images, thumbnail images, and stores images, thumbnail images, and

extracted features for later retrieval extracted features for later retrieval and analysis.and analysis.

Graphical User Interface Graphical User Interface provides interactive query capabilities provides interactive query capabilities

to the end userto the end user


Database ConstructionDatabase Construction

An XML index file stores extracted An XML index file stores extracted features and control informationfeatures and control information

A file system stores images and A file system stores images and thumbnail imagesthumbnail images


Database Construction – Database Construction – Cont.Cont.

XML Index File File System


Global Feature Global Feature ExtractionExtraction

Two different kinds of features are extracted:Two different kinds of features are extracted: Palette features Palette features

concern the set of colors in an image (color concern the set of colors in an image (color map)map)

examples: palette scopeexamples: palette scope Canvas features Canvas features

concern the spatial and frequency distribution concern the spatial and frequency distribution of colors in an image (image index)of colors in an image (image index)

examples: max, min, median, mean (for each examples: max, min, median, mean (for each color channel)color channel)


Example: Palette ScopeExample: Palette Scope

Palette ScopePalette Scope -- the total number of unique colors -- the total number of unique colors used in an image.used in an image.

We expect Dali’s piece to have a higher palette We expect Dali’s piece to have a higher palette depth than Mondrian’s work.depth than Mondrian’s work.

Hallucinogenic ToreadorSalvador Dali, 1970

Composition with Large Blue Plane,Red, Black, Yellow, and GrayPiet Mondrian, 1921


Example: Palette Scope – Example: Palette Scope – Cont.Cont.

ArtistArtist RGB Raw Palette ScopeRGB Raw Palette Scope NormalizedNormalized

MondrianMondrian 3000530005 0.001788430.00178843

DaliDali 7661376613 0.004566490.00456649

We see that Dali uses twice as much of the color spectrum as Mondrian.

Palette scope is an important feature for artist and period style identification because many styles are defined by color, i.e. Picasso’s Blue Period and fauvism.


Graphical User InterfaceGraphical User Interface


Test ResultsTest Results

Two types of tests were Two types of tests were conducted:conducted:

Feature testsFeature tests Interactive testsInteractive tests


Test Results – Cont.Test Results – Cont.

Training SetTraining Set Test SetTest Set Percent Percent CorrectCorrect

3636 3636 9494

200200 200200 8888

200200 200200 8383

Les Demoiselles d’Avignon,Pablo Picasso, 1907.

Road with Cypress and Star,Vincent Van Gogh, 1890.

Feature test to distinguish the work of Picasso and Van Gogh.


Initial Interactive TestInitial Interactive Test

Database of 10 works of each of the following Database of 10 works of each of the following ten artists: ten artists:

Braque, Cezanne, De Chirico, El Greco, Braque, Cezanne, De Chirico, El Greco, Gauguin, Gauguin,

Modigliani, Mondrian, Picasso, Rembrandt, Modigliani, Mondrian, Picasso, Rembrandt, and Van and Van

Gogh.Gogh.Training SetTraining Set Testing SetTesting Set Percent Percent CorrectCorrect

100100 9090 8181


Interactive Test on Web Museum DatabaseArtistArtist Training Training

SetSetQueriesQueries SuccessSuccess PercentPercent

AertsenAertsen 99 88 77 87.587.5

El GrecoEl Greco 1010 88 44 50.050.0

HopperHopper 1010 88 11 12.512.5

MalevichMalevich 1010 1111 66 54.554.5

MonetMonet 1010 1010 66 60.060.0

MorisotMorisot 1010 1111 55 45.545.5

RembrandtRembrandt 1010 3232 2323 71.971.9

RenoirRenoir 1010 3838 1212 31.631.6

TurnerTurner 1010 1010 33 30.030.0

VelazquezVelazquez 1010 88 77 87.587.5

OverallOverall 500500 299299 147147 49.249.2


EvaluationEvaluation ofWeb Museum Test ResultsTest Results

Overall result: 49.2% accuracy Overall result: 49.2% accuracy 29.2% better than blind guessing (10 29.2% better than blind guessing (10

guesses/50 artists = 20%)guesses/50 artists = 20%) Dissecting the classification Dissecting the classification

mistakes reveals some intelligent mistakes reveals some intelligent mistakesmistakes Rembrandt is most often confused with Rembrandt is most often confused with

Caravaggio, Ast, and VermeerCaravaggio, Ast, and Vermeer


ConclusionsConclusions Simple palette and canvas features are Simple palette and canvas features are

sufficient for an interactive classification sufficient for an interactive classification systemsystem

A single feature set can serve for A single feature set can serve for classification and image retrieval classification and image retrieval applicationsapplications

A general feature set can adequately serve A general feature set can adequately serve for educational applicationsfor educational applications

Although showing promise, we currently Although showing promise, we currently have a low confidence system have a low confidence system


Future WorkFuture Work

Can computer science provide an Can computer science provide an empirical framework for the empirical framework for the study of painting?study of painting?

Quantitative descriptionQuantitative description Falsifiable statementsFalsifiable statements Hypothesis verificationHypothesis verification

Documents

A Graphical User Interface for a Fine-Art Painting Image Retrieval System October 15-16, 2004 October 15-16, 2004