Geometric and Semantic Matching for Cultural …gravitate-project.eu/sites/default/files/ResearchSpace...Similarity Search Similarity matching of artefacts Start with search criteria

Copyright University of Southampton IT Innovation Centre 2017 1 www.gravitate-project.eu

Geometric and Semantic Matching for Cultural

Heritage Artefacts

Paul Walland

IT Innovation

ResearchSpace Symposium

28-07-2017


Initial Ambition of GRAVITATE Project

Re-Assembly

Re-Unification

Re-Association

------------------------

Similarity Search



British Museum, London Cyprus Museum, Cyprus

Ashmolean Museum, Oxford Fitzwilliam Museum, Cambridge


Re-Unification

Head in the British Museum

Statue in the Cyprus Museum

Re-unify 3D models of the statue

3D print copies for re-unification

Re-unify statue in a virtual museum


Re-Association

Artefact with semantic description Material, decoration, size, shape, period, style, glaze, …

Data in another museum catalogue


Re-Association

Find similarities leading to new insights about past cultures



Data on Artifacts is in many different forms:

Physical Descriptors: Image (2D and 3D) Geometrical Material Pattern Colour … etc

Abstract (non-physical, semantic) Age Condition Style Location Part (e.g. foot or ear) Specialised curatorial terms … etc

And in many different places!


Fragments of Terracotta Statues from Salamis


Problem Statement British Museum, London Cyprus Museum, Cyprus

Ashmolean Museum, Oxford Fitzwilliam Museum, Cambridge


Problem Statement

Jigsaws

in 3D

most pieces missing

all the edges worn off

many puzzles jumbled together

pieces spread across many countries


Problem Statement

Front Back

Faded

Cracked

Eroded


Problem Statement Re-Assembly

You have some (eroded) pieces.

The computer fits them together.

This is hard in 2 dimensions, never mind in 3D!

Localised geometry

3D descriptors

Localised detail

Colour descriptors


We need to Gather all the Knowledge

Catalogue data with text

descriptions

Archaeological papers

Excavation notes

Chemical analysis

X-ray fluorescence

3D scanning models




descriptions


Excavation notes

Chemical analysis

X-ray fluorescence

3D scanning models




descriptions


Excavation notes

The British Museum has 2.5M objects described in CIDOC CRM Catalogue data from Ashmolean, Fitzwilliam and Cyprus museums is now mapped to the same data model

Object hasNote “…lots of free-form text added by the curator”

Using Natural Language Processing to extract and encode meaning from this text: • References to papers, to catalogue entries • Parts and features • Conservation condition • Measurements • … CIDOC CRM / CRMarchaeo / etc



Chemical analysis

X-ray fluorescence

3D scanning models

Volume, area thickness,

curvature, …

Colour, texture, distance from convex hull, …

Feature detection

Faceting: front / back

/ fracture

Part annotation

CRMdig CH Artefact Partonomy

All fragments (211) are scanned, in a

variety of resolutions and with a variety of

scanners


What is our raw data?

Semantic data

From catalogues

From curators

Measurements of the fragments

3D scanning 3D models

Chemical analysis, pigment composition

X-ray fluorescence

…etc


Fundamental Geometric Properties and Descriptors

Library of tools for the analysis of the geometric and colorimetric properties of a fragment’s shape Curvature-based

Distance-based

Volume, area, thickness

..

Library of descriptors and signatures to evaluate similarity geometry

color

textures

Mean Shape index

Distance from convex hull

Average geodesic distance


Faceting Faceting partitions a fragment into

distinct facets by sharpness and roughness

Each facet will be classified into: Skin (original) boundary surface of

the unbroken object

Fracture created when the object broke

Re-Assembly modules use the facets differently: Skin facets local continuity

in color and shape

Fracture facets complementarity of abraded local shape

(SotA faceting from predecessor PRESIOUS)


NLP Example Head from Toumba, Ashmolean Museum, Oxford, Inv. no. 1909.837=1891.468 (Tubbs and Munro 1891, 149, fig. 7; SCE IV:2, pl. IX bottom; Hermary 1991, pl. XXXIX:a). The lower part of the beard is restored. Restored height of the head: 39 cm. Upper part of (conical?) helmet missing; upturned cheek-pieces; preserved part of helmet is covered with stamped circles. The face is rectangular; almond-shaped eyes, ridged eyelids, broad 'feathered' eyebrows. The beard is neatly separated from the rest of the face by a 'rope' line as on Cat. no. 73. The restored lower part of the beard must originally have been square with rounded corners, like that of Cat. no. 73. It is grooved vertically with each groove terminating in a curl (cf. a beard fragment from Samos, no. T324, Schmidt 1968, pl. 60). A fringe of curly hair appears on the forehead below the lower edge of the helmet and at the temples, as on Cat. no. 73. Trapezoidal moustache with grooved strokes. The ears are rendered in the same way as on Cat. no. 73. Black paint on beard, hair, moustache, eyebrows and irises; red paint on lips. This head was assigned by Gjerstad (SCE IV: 2, 108) to the Western Neo-Cypriote style. It has several stylistic similarities with Tamassos heads Cat. nos 68-69, though the expression of Cat. no. 74 is more smiling (Karageorghis 1993, 32, Cat. no. 74).

Ashmolean reference AN1909.837

The discourse styles for Salamis artefacts in BM, Ashmolean, Fitzwilliam and Cyprus museums are similar, but different

We attempt to automate annotation of the artefact semantics from such sources


ResearchSpace

• British Museum’s platform impementing CIDOC CRM as framework

• Components: Search, Clipboard, Data Annotation, Data Argumentation, Image Annotation/ Argumentation, Workflow, User Defined Pages/ Publication • Shows faceted relationships

• Combines results from multiple datasets

• Supports contextual search


Similarity Search

Similarity matching of artefacts Start with search criteria or set of example objects

Geometric descriptors » size, part, colour, texture, edge/surface » Geometric matching algorithms

Semantic metadata » type, style, location, size, part, colour, texture, damage, hypothesis/conjecture » Graph matching

Fusion of match results Aggregation of the result space + Uncertainty analysis

Not a black box approach

Iterative expansion / refinement of match results with expert feedback


Demonstration of GRAVITATE Platform

http://gravitate-1.cytera.cyi.ac.cy/





















Desktop app


GRAVITATE-PROJECT.EU

Thank you for your attention!

Documents

Geometric and Semantic Matching for Cultural …gravitate-project.eu/sites/default/files/ResearchSpace...Similarity Search Similarity matching of artefacts Start with search criteria