The Versus Comparison Framework

National Center for Supercomputing ApplicationsUniversity of Illinois at Urbana-Champaign

The Versus Comparison Framework

Kenton McHenry, Ph.D.Research ScientistNational Center for Supercomputing Applications

The Problem

• The abundance of file formats is a problem when preserving electronic records

• Why?• Will there be software to load the file in the future?• If not will the specification for the format still exist?• Was the specification ever available to begin with

(closed/proprietary formats)?

*.dwg *.max, *.3ds*.blend

*.k3d

*.w3d

*.ma, *.mb, *.mp

*.iam*.lwo *.c4d

*.pdf (*.prc, *.u3d)

*.vtk, *.vtp

*.skp

Available 3D File Formats…

Converting Formats

• In order to preserve content for future use one option is to convert the file to an open/standardized format that is likely to be supported for some time.• Store both this file and the original for provenance

• Ideally with one file format for a particular content type it will be easy for users to view/use the data.

Converting Formats (continued…)• How and which format!?

• Fully supporting the many available formats is an enormous undertaking

• If a file format is closed/proprietary it may be difficult to retrieve the data directly from the file• May be possible to reverse engineer and recover some of

the content• Vendor file formats sometimes store application feature specific

pieces of information that’s not supported in other formats• Examples include: animations, physics, …• When converting to a format that doesn’t have a place for

such information we must drop it.• Information loss…

Converting Formats (continued…)

• There are different ways of storing the 3D content itself• Faceted:

• Comprised of vertices and faces • Popular within the graphics community

• Boundary Representation: • Comprised of vertices, edges, edge loops, and primitive

surfaces• Popular among CAD users

• Constructive Solid Geometry• Comprised of boolean operations on primitive volumes

• …

Converting Formats (continued…)

• Translating geometry representation may not be trivial• B-Rep to Faceted

• Translating involves triangulating the surfaces created from the bounded primitives (tesselation)

• The resulting sampled surface will suffer from aliasing at high viewing resolutions

• Can accommodate by performing a finer triangulation (i.e. more triangles and a larger file)

• Faceted to B-Rep:• Translating in this direction is non-trivial!• How does one decide if a group of triangles should be

grouped together as part of some larger primitive (e.g. part of a cylinder).

Geometry Appearance Scene Format Faceted Parametric CSG B-Rep Color Material Texture Bump Lights Views Trans. Groups

Animation

3ds √ √ √ √ √ √ √ √ √

igs √ √ √ √ √ √ √

lwo √ √ √ √ √ √

obj √ √ √ √ √ √ √

ply √ √ √ √ √

stp √ √ √ √ √ √

wrl √ √ √ √ √ √ √ √ √ √ √

u3d √ √ √ √ √ √ √ √ √

x3d √ √ √ √ √ √ √ √ √ √ √

Converting Formats (continued…)

• How do we measure information loss?• Which is the best format to use for preservation?

NCSA Polyglot (2009)

• Conversions service based on utilizing any and all available 3rd party software• Imposed Code Reuse: Re-attaching a programmable interface to

compiled software.• Scripted operations within software

• GUI scripting (e.g. AutoHotKey)• Created a simple workflow referred to as an Input/Output Graph• Compared files before/after conversion to measure information

loss• Distributed across multiple machines• Web access

ISDA File Migration Tools

• Conversion Software Registry• Software Servers• Polyglot• Versus

Software that can Convert between Formats

• There is a lot of software available, each with its own unique capabilities

• A lot of it is not free• It would be expensive to buy a package just to check if it truly is

capable of converting between a desired pair of formats• How can someone know what software to get for their

needs?

http://isda.ncsa.illinois.edu/NARA/CSR

The Conversion Software Registry

Adobe 3D Reviewer

The Conversion Software Registry

Input/Output GraphsAdobe 3D Reviewer

Input/Output Graphs

3DS Max

Adobe 3D Reviewer

AutoCAD

Blender

Cinema 4D

K-3D

LightWave 3D

Maya

Wings 3D

Shortest conversion path

Input/Output Graphs

Software Servers

Software Server

• To program against software• i.e. to write new code that can utilize functionality within arbitrary

software, compiled code, where the source code is probably not available.

• Imposed Code Reuse (or Software Reuse): The process of attaching an API like interface to software so that its functionality can be called within new code.

Software Servers

• Shares the functionality of software over the web• In contrast to services which share data: ftpd, nfsd, sambad,

httpd• Similar to services such as: telnetd, sshd, VNC, rdesktop

• The main difference is in the interface:• Uniform across all software

http://host:8182/software/<Application>/<Task>/<Output Format>/<InputFile>

• Simple• Widely accessible• Capable of being programmed against

• Allows any desktop application to become a cloud based web service*

Software Functionality Sharing

Software Functionality Sharing

Software Functionality Sharing

Software Functionality Sharing

Software Functionality Sharing

Software Functionality Sharing

Software Functionality Sharing

Software

• Excel 2010• Word 2010• Power Point 2010• Publisher 2010• One Note• Access 2007• Wordpad• Notepad• Calculator2010• Internet Explorer 8• Winzip• Photoshop CS5• Adobe Acrobat• ABBYY

• 3D Studio Max• Adobe 3D Reviewer• Google Sketchup• Wings 3D• Blender• K3D• Paraview• VTK• Cyberware PLYTool• NIST X3D Tool• Imagemagick• IrfanView• GIMP• Microsfot Paint

Polyglot

• Listens for Software Server broadcasts on the network• Catalogues available input/output operations and

constructs and I/O-graph• Identifies conversion paths between input and output

formats• Carries out CHAINED conversions

Polyglot

Polyglot

Versus

• Java library/framework for comparing file content• Distributed architecture• RESTful Web Interface

• http://<host>/versus/comparisons• dataset1, dataset2• adapter, extractor, measure

Adapters

Name Package Description

Mesh 3D Load 3D files content into a mesh made up of vertices and polygons connecting those vertices.

Audio Audio Encapsulation of audio files.

Bytes Core Simplest possible representation of data.

PDF Doc2Learn Encapsulation of the Doc2Learn PDF document.

Buffered Image Image Standard Java representation of image data.

Image Object Image Encapsulation of the Im2Learn Image Object.

SIFT GPU GPU Encapsulation of image data for SIFT Gpu specific processing.

DescriptorsName Package Description

Double Array Core A single dimensional array containing double values.

MD5 Digest Core A data integrity structure generated from the raw data.

Three Dimensional Double Array

Core A three-dimensional array containing double values.

Vector Core A list of generic elements, allows greater storage flexibility.

Label Histogram Doc2Learn A histogram of labels obtained through Doc2Learn.

Keypoint Image Generic container for invariant feature detectors.Pixel Image Generic type for various image package

descriptors.Color Layout Image A two dimensional grid of sub-images over the

input image.Grayscale Histogram Image A one-dimensional grayscale image histogram.

RGB Histogram Image A three-dimensional RGB color histogram. Pixel Histogram Image A multidimensional histogram for a pixel’s

intensity and position. MOPS Features Fiji Invariant feature type used for image stitching.SIFT Features Fiji Popular invariant feature type used for image

comparison and object matching. SIFT Gpu Gpu Same as SIFT but implemented through Gpu

libraries. Harris Corners OpenCV Well-known corner detector used for image

inference, tracking, and recognition.Hough Circles OpenCV Circles detected in an image with the Hough

Transform.Hough Lines OpenCV Lines detected in an image with the Hough

Transform.SURF Features OpenCV Invariant feature type that can be computed

faster than standard SIFT.

ExtractorsName Package Description

Light Field 3D Surface is represented by silhouettes taken from 3 canonical positions capturing the surface shape minus any concavities (i.e. the convex hull).

Statistics 3D Ignores the surface and focuses on the vertices of a 3D object returning their mean and standard deviation. Simple, but fast to compute.

Surface Area 3D The sum of the area occupied by the polygons making up a surface. Considers surface and is still fast to compute.

Audio Audio Sampling of audio from existing file for histogram usage and comparison.

MD5 Core Creation of the MD5 hash from data.Image Histogram Doc2Learn Generates a non-standard color histogram

Line Graphics Histogram Doc2Learn Generates a histogram to compare vector graphics found in documents.

Text Histogram Doc2Learn Generates a label histogram based on word frequency.

Array Feature Image Generates the three-dimensional double array; a generic image container.

Color Average Vector Feature Image Generates an average RGB color over 9 regions taken from the image.

Grayscale Histogram Image Generates the histogram for grayscale images. Useful for image comparison.

Pixel Histogram Image Generates the multidimensional histogram for feature matching.

RGB Histogram Image Generates the histogram for color images. Useful for image comparison.

Signature Vector Image Feature vector (for an image) containing colorspace information and pixel position.

MOPS Features Fiji Open source implementation for the MOPS detector.

SIFT Features Fiji Open source implementation of Lowe’s method.SIFT Gpu Gpu Gpu implementation for the SIFT detector.

Harris Corners OpenCV Corner detector for images.Hough Circles OpenCV Circle detector for imagesHough Lines OpenCV Line detector for images

SURF Features OpenCV Open source implementation for the SIFT detector.

MeasuresName Package Description

Chessboard Distance CoreAlso known as Chebyshev; the greatest difference along any coordinate dimension (between two vectors)

Dynamic Time Warping Core Similarity metric between two (possibly) varying sequences over time.

Euclidean Distance Core Distance between two n-dimensional points in Euclidean space.

Manhattan Distance CoreAbsolute difference of coordinates of points, distance between two points measured along right angled axes.

MD5 Hash Core Binary measure; either equal or not.

Bhattacharyya Distance Image Measures the overlap between two probability distributions.

Neyman’s χ2 Image Tests the goodness of fit between two distributions. Variant of the standard χ2 test.

Czekanowski Distance ImageSum of the absolute value of the difference of two distributions divided by the sum of the two distributions.

Histogram Euclidean Distance

Doc2Learn / Image

Bin-by-bin comparison using the standard Euclidean distance. Well known and widely used.

Histogram Intersection Doc2Learn / Image

Sum of the absolute value of the difference of two distributions, scaled by one-half. Well known and widely used.

KL Divergence ImageNon-symmetric measure of the difference between two probability distributions. Well known measure of entropy.

Jeffrey Divergence Image Symmetric measure of the difference between two probability distributions.

Motyka Distance Image Sum of the maximum of two distributions divided by the sum of the two distributions.

Normalized Cross Correlation Image Similar to sum of squared differences; invariant

to the magnitude of two points.

Ruzicka Similarity ImageSum of the minimum of two distributions divided by the sum of the maximum of the two distributions.

Sum of Squared Differences Image

Sum of squared differences between two arrays, cheaper to computer than Euclidean distance.

Tanimoto Distance ImageSum of the difference of the max and the min of two distributions divided by the sum of the max.

Wave Hedges Distance ImageSum of the absolute value of the difference between two distributions divided by their maximum.

Invariant Feature Comparison

Fiji / OpenCV /

SIFT Gpu

Compares the invariant features between two images by calculating the pairwise Euclidean distance and voting for a match using a predetermined threshold.

Earth Mover’s Distance OpenCVMeasures the similarity between two probability distributions. This is the minimum cost of transforming one distribution to the other.

Extractor Previews

Measuring Information LossWe would like to assign

a value to each conversion edge …

• With a “universal” converter we could convert files from every format A to every other format B

• Assuming we then had a loader for both format A and format B we could load and compare the 3D content independent of how it is stored.

Measuring 3D Information Loss

good… (e.g. 1.0) not so good… (e.g. 0.1)

• Adobe 3D Reviewer• Blender• Cyberware PlyTool• K-3D• NIST VRML/X3D• VTK

Measuring 3D Information Loss

• Data representation• Meshes

• Loaders• Use 3D similarity as a means of comparing 3D models

• Statistics• Surface Area [Brunnermeier, RTI 1999]• Spin Images [Johnson, PAMI 1999]• Light Fields [Chen, Eurographics 2003]

Statistics

• Use the mean and standard deviation of the vertices to represent the model

• Simple but fast to compute• Sensitive to size and orientation of the model

Surface Area

• Use the sum of face areas to represent the model• Also simple and fast to compute• Sensitive to size, somewhat sensitive to shape. Will

detect loss of faces.

Light Fields [Chen, 2003]

• Compares silhouettes from various viewing angles around a model.

Light Fields

Light Fields

Light Fields

Light Fields

Light Fields

• Fairly fast to compute• Sensitive to shape of convex hull, invariant to rigid

transformations

Spin Images [Johnson, 1999]

• 2D histograms of the in plane and out of plane distances of vertices neighboring a given vertex.

p

N q

a

b

Spin Images

Spin Images

Spin Images

Spin Images

Spin Images

Spin Images

• Expensive to compute• Sensitive to relative vertex position, ignores surface,

invariant to rotations and translations

3D Information Loss and 3D File Loaders

• If we were able to load every file format we really wouldn't need to use software reuse to convert via 3rd party applications.

• Implement loaders for a small number of formats that will make up our test data set• Convert from format A along path to some format B then back to

A again• Estimate path scores by comparing before/after content and

assigning scores to all edges along path

STP to X3D to STP

X3D

WRLSTP

WRL

A3D Reviewer

STP

Vrml97ToX3d

X3dToVrml97 A3D Reviewer

Web Browser …

Application Wrapper A

Application A Application B Application C

Application Wrapper B

Application Wrapper C

Software Server

Software Server Client

Polyglot Client…

Polyglot Panel

…

Application Wrapper X

Application X Application Y Application Z

Application Wrapper Y

Application Wrapper Z

Software Server

Polyglot Server

Software Server Client Software Server Client

Polyglot Web Server

… …

IOGraph Weights Tool

Versus

Software

Software Reuse

Conversion(Polyglot)

Comparison(Versus)

Polyglot StewardComparison (Versus)Conversion (Polyglot)

Software ReuseClosed Source Software

Which conversion preserved the most?

• Using the light fields measure:• Emphasizes shape through silhouettes• Adobe 3D Reviewer between *.pdf and *.stp (61.67)

• Using the spin image measure:• Emphasize shape through relative vertex positions• Adobe 3D Reviewer between *.obj and *.pdf (59.07)

Which is the best format?Within the context of preservation we can define this as the format that retains on average the most information when converted to by other formats.

• Using the light fields measure:• Emphasizes shape through silhouettes• *.stp (40.73)

• Using the spin image measure:• Emphasizes shape through relative vertex positions• *.stl (34.89)• *.stp being a CAD format has more variability in vertex positions

due to tessellation

Word Spotting in Versus

• Include Word Spotting feature code as extractor in Versus• Just another form of content based comparison• Specific to handwriting of course

• Include general purpose efficient indexing into Versus• e.g. hierarchical agglomerative clustering• Not just for Word Spotting

Conclusion

Image Formats by Information Preservation

1. PPM2. PNG3. GIF4. JPG5. …

Image Formats by File Size

1. JPG2. GIF3. PNG4. PPM5. …

Image Software by Information Preservation

1. ImageMagick2. Adobe Photoshop3. GIMP4. Microsoft Paint5. …

3D Formats by File Size

1. MP42. MAX3. PLY4. OBJ5. …

3D Formats by Information Preservation

1. STL2. STP3. OBJ4. MAX5. …

3D Software by Information Preservation

1. Adobe 3D Reviewer2. 3DS Max3. Maya4. Blender5. …

Video Formats by Information Preservation

1. AVI2. MOV3. WMV4. MPG5. …

*Rankings shown demonstrate the type of output that will be obtained from future work and does NOT represent actual results!

1,682 formats2,007 applications

Versus and CBR

• CBIR: Content based image retrieval• Versus as a content based comparison framework can

serve as the back end for general purpose content based retrieval

Acknowledgements

• This research was partially supported by a National Archive and Records Administration (NARA) supplement to NSF PACI cooperative agreement CA #SCI-9619019 and by NCSA Industrial Partners. 

Imaginations unbound

• The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the National Science Foundation, the National Archive and Records Administration, or the U.S. government.

The ISDA Tools (Free and Open Source)

Image, Spatial, and Data Analysis Group

http://isda.ncsa.illinois.edu

Kenton McHenryRob KooperMichal OndrejcekLuigi Marini