Workpackage 4 Image Analysis Algorithms

Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment

Review Meeting N.1: Paris, C2RMF, November 28, 2000

Workpackage 4Workpackage 4Image Analysis AlgorithmsImage Analysis Algorithms

Kirk Martinez, Paul Lewis and Stephen ChanIntelligence, Agents and MultimediaDepartment of Electronics and Computer ScienceUniversity of SouthamptonUK



Task 4.1Task 4.1User requirements AnalysisUser requirements Analysis

• First step was to identify the requirements of the users (note overlap with other workpackages)

• Required Output: Collation of scenarios and functionality

• Output achieved in collaboration with other participants and delivered in the form of some initial scenarios and a set of 16 goals. These are published as part of the System design document.



Task 4.3 Task 4.3 Recognition Algorithm DevelopmentRecognition Algorithm Development

• PM 8-24

• Aim is to develop image analysis algorithms to meet user requirements

• Required outputs: Image content analysis software and report

• Consider 4.1 and 4.3 together in this presentation



GoalsGoals

• G1 Matching of similar images (includes “have you got this picture”)

• G2 Automatic search using synonyms• G3 Search based on features oriented to the

restoration framework - uv spotmeasures - x-ray and reverse pic views of frames - craquelure classifier -search based on “butterfly” supports in the frame



Goals Cont.Goals Cont.

• G5 Access information quickly and easily• G6 Search based on colour• G7 Query by low quality images (especially faxes)• G8 Query by sketch• G9 Query refinement• G10 Joint retrieval by content and by text• G11 Use of publishing products built on the

Artiste system• G12 Detail finding



Goals Cont.Goals Cont.

• G13 Search using multilingual vocabulary

• G14 Respect installation site privacy and security policy

• G15 Produce a sustainable system after the end of the project

• G16 Be consistent with partners’ predefined technical constraints



AnalysisAnalysis

• The goals were analysed in terms of the implications for image analysis

• Possible image processing (IP) approaches were identified for goals requiring IP

• Six distinct groups of algorithms were identified together with the goals to which they could contribute.



1.1. Algorithms which will find similarity matches based on Algorithms which will find similarity matches based on global histograms (colour or grey scale) between a query global histograms (colour or grey scale) between a query

image or sub-image and images in the database image or sub-image and images in the database collections. collections.

• Could contribute to goals 1,3,7

• Useful for basic image matching

• May contribute to style search and classification

• Potentially faster than spatial-colour matching methods

• Status: Implemented at IAM



Example of Global Colour Example of Global Colour Histogram SearchHistogram Search

Query image

Best Matches



2. 2. Algorithms which will find similarity matches between a queryAlgorithms which will find similarity matches between a query image or sub-image and images or sub-images in the database image or sub-image and images or sub-images in the database

using spatial colour distributions.using spatial colour distributions.

• Goals: 1,3,6,10,12

• Takes into account the spatial arrangement of colours

• Finds similar colour patterns at similar locations

• Or similar colour patterns at any location

• Status: Implemented a hierarchical colour coherence vector based matcher in IAM



Example of H-CCV Matching Example of H-CCV Matching

Query sub-image

Best Matches



3. Algorithms which will segment image into regions of similar 3. Algorithms which will segment image into regions of similar texture and record feature vectors representing texture for each texture and record feature vectors representing texture for each

main regions. User can then indicate a query texture either by main regions. User can then indicate a query texture either by indicating a region in a particular image or selecting a texture indicating a region in a particular image or selecting a texture

from a texture palette. Images in the database containing texture from a texture palette. Images in the database containing texture regions matching the query are then retrieved. regions matching the query are then retrieved.

• Goals: 1,3,12

• Status: Previously implemented texture extraction algorithms

• Not yet implemented automatic texture segmentation



4.4. Algorithms which will match an outline query Algorithms which will match an outline query shape with similar shapes within database images.shape with similar shapes within database images.

• Goals: 1,3,4,7.8,10,12• Pre extracting all shapes of all objects in all images is impossible i.e.

can not pre-index shapes!• Techniques like the Generalised Hough Transform (GHT) use evidence

accumulation to find a shape in an image and are related to template matching.

• They are computationally intensive• Status: Not yet implemented for this project



5. Algorithms which will detect and in some 5. Algorithms which will detect and in some cases analyse specific image featurescases analyse specific image features

• Goals: 4

• May be able to use e.g. the GHT

• Will also require specially tailored algorithms

• Status: Not yet implemented



6. Algorithms to provide basic image manipulation6. Algorithms to provide basic image manipulation

• Goals: All involving image handling

• Include operations like image conversion, compression, scaling, rotation etc

• Most are widely available but may need re-implementing or tailoring in context of Artiste

• Status: Partial implementation



Environment for Algorithm Testing Environment for Algorithm Testing

• Test-beds in the IAM lab include VIPS and MAVIS 2• Algorithms developed as “stand alone” modules which deliver feature

vectors (FVs) and modules for calculating similarity between FVs• MAVIS 2 is a multimedia retrieval and navigation environment• It associates media content and the concepts they represent• Concept layer equivalent to a thesaurus• Allows integrated content and concept based searching with query

scope expansion

Documents

Workpackage 4 Image Analysis Algorithms