Effect of Linearization on Normalized Compression Distance

Jonathan MortensenJulia Wu

DePaul University

July 2009

Introduction Kolmogorov Complexity is an emerging

similarity metric Transformation Distance

Universal Similarity Measure Does not require feature identification and selection

How can it be applied to images? CBIR, Classification

Investigate its effectiveness Discovered some fundamentals have been

overlooked thus far

Outline Background Kolmogorov Complexity and Complearn Research Topics Spatial Transformations Intensity Transformations Image Groupings Conclusion Future Work

Background Li (2004): successful clustering of phylogeny

trees, music, text files 1D to 2D data?

Tran (2007): NCD not a good predictor of visual indistinguishability Only one photograph used, one type of linearization (row-

by-row) Gondra (2008): CBIR using NCD produced

statistically significant measures against H0 of random retrieval and other similarity measures Test set of hundreds of images, inconsistent methods of

compression and concatenation, linearization unclear

Kolmogorov Complexity

max{ ( | *), ( | *)

max{ ( ), ( )

K x y K y xNID

K x K y

K(x) – The length of the shortest program or string x* to produce x

K(x|y) - The shortest binary string to convert output x given input y

E(x,y)=max{K(x|y),K(y|x)} Normalized Information Distance:

Kolmogorov Complexity

Universal, in that it captures all other semi-computable normalized distance measures

Therefore also semi-computable Compression losslessly simplifies strings, and

therefore is used as an approximation, C(x)

( ) min{ ( ), ( )}( , )

max{ ( ), ( )}

C xy C x C yNCD x y

C x C y

“The human brain is incapable of creating anything which is really complex.”--Kolmogorov,  A.N., Statistical Science, 6, p314, 1990

CompLearn Open Source package which implements K-

Complexity Developed by Rudi Cilibrasi, Anna Lissa Cruz,

Steven de Rooij, and Maarten Keijzer Uses basic linux compression tools to develop

the comparison map

Images from “Google Similar Images”

Initial Questions Linearization Methods and Alternatives

How to Preserve a 2D signal Linearization’s affect NCD on spatial

transformations and intensity shifts Do additional feature images lower NCD? CBIR: Can K-Complexity be used with feature

vectors or image semantics

Spatial Transformations Applied 4 types of linearization to 800 images

(original and 7 transformations) Found that each linearization type produced

distinctly different NCDs Certain linearizations result in lower NCDs for

certain transformations

Linearization Methods

Row Major

Column Major

SCPO:

Images transformed to 35% of original size

Hilbert-Peano SPC:

Images transformed to 128x128

Original Image Down Shift Left Shift

90 rotation

180 rotation

270 rotation

Reflection Y Axis Reflection X Axis

Spatial Transformations

Intensity Transformations Additive Constant Three types of noise

Gaussian Speckle Salt and Pepper

Least Significant Bit (LSB) Steganography Contrast Windowing

Additive Constant

P = Intensity + Constant +4, +8, +12… +100

16 bit 255 (+4)-> 259

Truncation 255 (+4)-> 255

Wrap 255 (+4)-> 4

Image 937.jpg+32 and +64 respectively

Additive Constant

Additive Intensity

0.96

0.97

0.98

0.99

1

1.01

1.02

1.03

1.04

1.05

0 20 40 60 80 100 120

Intensity Added

NC

D f

rom

Ori

gin

al

16bit

Truncated

Wrap

Various Noise

Gaussian (Statistical)

Speckle (Multiplicative)

Salt and Pepper (Drop-off)

 

0.32 and 0.64 Variance/Noise Density Respectively

Noise Cont:

Gaussian and Speckle Noise don’t compress well

Gaussian and Salt Pepper experience some posterior decay

Gaussian and Speckle Noise

1

1.01

1.02

1.03

1.04

1.05

1.06

1.07

0 0.2 0.4 0.6 0.8 1 1.2

Variance

NC

D f

rom

Ori

gin

al

Gaussian

Speckle

Salt Pepper

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1 1.2

Noise Density

NC

D f

rom

Ori

gin

al

Salt Pepper

Least Significant Bit Steganography

Hide4PGP “Scrambles” message Changes pixel bit to

most similar color with opposite bit assignment

Spreads secret data over entire file

True Grayscale: Changes two bits per pixel

Image with No Text

Image hiding “Gettysburg Address”

LSB Steganography

LSB Steganography

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0 5000 10000 15000 20000 25000 30000 35000

Bits Hidden

NC

D f

rom

Ori

gin

al

LSB Steganography

Hamming Distance

Hamming: Steganography

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

0 10000 20000 30000 40000

Bits Hidden

Ham

min

g D

ista

nce

Hamming:Steganography

Contrast Windowing Computed Tomography image enhancement

that increases contrast in certain structures Brief Medical Exploration

Contrast Windowing

Soft Tissue Window (50 HU, width 350 HU)

Bone Window (300 HU, width 1500 HU)

Lung Window (-200 HU, width 2000 HU)

Patient 5: Original Image

top left

original bone lung tiss

p1 0 1.028241 1.049258 1.02429

bone 1.028241 0 1.036157 1.011354

lung 1.049258 1.036157 0 1.039524

tiss 1.02429 1.011519 1.039524 0

p3 0 1.02097 1.043942 1.025635

bone 1.020539 0 1.037073 1.014142

lung 1.044137 1.037073 0 1.037244

tiss 1.026016 1.014354 1.037244 0

p5 0 1.020947 1.047888 1.023039

bone 1.020947 0 1.038712 1.019146

lung 1.047888 1.038712 0 1.036131

tiss 1.023039 1.019924 1.036131 0

P1 P3

P5

Cross Dicom Comparison

p1tiss p1lung p1bone p1 p3tiss p3lung p3bone p3 p5tiss p5lung p5bone p5

p1tiss 0.0000 1.0395 1.0115 1.0243 0.9739 1.0390 1.0157 1.0223 0.9813 1.0325 1.0066 1.0234

p1lung 1.0395 0.0000 1.0362 1.0493 1.0362 0.9772 1.0361 1.0485 1.0410 0.9853 1.0412 1.0477

p1bone 1.0114 1.0362 0.0000 1.0282 1.0158 1.0378 0.9642 1.0278 1.0197 1.0365 0.9761 1.0247

p1 1.0243 1.0493 1.0282 0.0000 1.0255 1.0460 1.0258 0.9811 1.0258 1.0455 1.0240 1.0025

p3tiss 0.9741 1.0362 1.0168 1.0255 0.0000 1.0372 1.0144 1.0260 0.9810 1.0328 1.0140 1.0222

p3lung 1.0390 0.9772 1.0378 1.0460 1.0372 0.0000 1.0371 1.0441 1.0434 0.9874 1.0418 1.0513

p3bone 1.0137 1.0361 0.9650 1.0258 1.0141 1.0371 0.0000 1.0205 1.0175 1.0360 0.9728 1.0220

p3 1.0238 1.0485 1.0271 0.9811 1.0256 1.0439 1.0210 0.0000 1.0278 1.0414 1.0218 0.9997

p5tiss 0.9932 1.0410 1.0180 1.0258 0.9821 1.0434 1.0172 1.0278 0.0000 1.0361 1.0199 1.0230

p5lung 1.0325 0.9853 1.0365 1.0455 1.0328 0.9874 1.0360 1.0414 1.0361 0.0000 1.0387 1.0479

p5bone 1.0062 1.0412 0.9757 1.0240 1.0142 1.0418 0.9724 1.0217 1.0191 1.0387 0.0000 1.0209

p5 1.0234 1.0477 1.0247 1.0025 1.0222 1.0513 1.0220 0.9997 1.0230 1.0479 1.0209 0.0000

Conclusion: "How Many" vs "How Little" NCD for Ordinal Comparisons Numerical Redundancy

Entire Picture

Selective

Larger NCD Smaller NCD

SteganographySalt and Pepper Noise

GaussianSpeckleNoise

Additive Constants

Contrast Windowing

Feature Image Comparison and Grouping Feature Image: Pixel based values derived

from the original image 3 Main Types of Linearization Avg NCD inter > Avg NCD intra The greater inter - intra, the better NCD finds

groupings

Feature Image Linearization Image-At-Once – row-order one feature image

at a time Row Concatenation – Appends all images,

then performs row-order linearization Pixel Order – Selects value from same pixel of

each feature image in row-order fashion Gray Row-Major – Grayscales an image and

follows row-order on intensities

Data Set and Methods

Corel Image Database with 10 predefined groupings

Linearized by 5 methods

NCDs were found within a group and then to the left and to the right

Results Nearly every linearization produced

statistically different NCDs Intra Group was always less than Inter Group Gray provided the greatest difference Inter-

Intra Thought this was due to filesize

Triple Concat’ed Gray creating equal filesize: Found an even greater difference

Conclusion NCD is a good model for predefined human

groupings and linearization has little impact on this

Gray-Triple Row-Major may be the best form of linearization

Direction of concatenation does not matter Defined a methodology for any number of

feature images

Conclusion Compressor Errors Numerical Redundancy

Ordinal Variables vs Nominal Variables EX: 195 195 195 195 <=> 198 198 198 198

NCD = 0.100000 199 199 199 199 <=> 202 202 202 202

NCD = 0.128205

NCD needs refinement 2D image as a 1D string?

Future Work Image Scaling and Normalization Additional Feature Images New Forms of Image concatenation Investigate Compressors (Numeric?)

References A. Itani and D. Manohar. Self-Describing Context-Based

pixel ordering. Lecture notes in computer science, pages 124{134, 2002.

M. Li, X. Chen, X. Li, B. Ma, and P. M.B Vitnyi. The similarity metric. IEEE.Transactions on Information Theory, 50:12, 2004.

R. Dafner, D. Cohen-Or, and Y. Matias. Context-based space lling curves. In Computer Graphics Forum, volume 19, pages 209{218. Blackwell Publishers Ltd, 2000.

R. Cilibrasi, Anna L. Cruz, Steven de Rooij, and Maarten Keijzer. CompLearn home. http://www.complearn.org/.

R. Cilibrasi, P. Vitanyi, and R. de Wolf. Algorithmic clustering of music. Arxiv preprint cs.SD/0303025, 2003.

N. Tran. The normalized compression distance and image distinguishability. Proceedings of SPIE, 6492:64921D, 2007.

I. Gondra and D. R. Heisterkamp. Content-based image retrieval with the normalized information distance. Computer Vision and Image Understanding, 111(2):219{228, 2008.

Questions