An Inter-Image Redundancy Measure for Image SetCompression

Xinfeng Zhang, Yabin Zhang, Weisi LinSchool of Computer Engineering

Nanyang Technological University, SingaporeEmail:{xfzhang, wslin}@ntu.edu.sg

Siwei Ma, Wen GaoSchool of Electronics Engineering and Computer Science

Peking University, Beijing, ChinaEmail:{swma, wgao}@pku.edu.cn

Abstract—Image set coding improves the compression effi-ciency by reducing both intra- and inter-image redundancy. Thekey of success is to select representative image(s) to predict set ofsimilar images. This paper proposes an inter-image redundancymeasure for representative image selection in image set com-pression. In the proposed method, the inter-image redundancy ismeasured jointly by the extent of similar content (EOS) and thecorrelation of similar content (COS) shared in two images. Wetake the covered area of matched SIFT points to measure theEOS, and take the distance of the matched SIFT descriptors tomeasure the COS. The image with largest redundancy for theset is selected as the representative one to predict other images.Experimental results show that the proposed method can selectbetter representative image, and achieve bitrate saving up to 9.2%and 20.8% compared with state-of-the-art image set compressionmethod and HEVC inter coding method.

Keywords—Image set compression; set redundancy; SIFT;cloud storage

I. INTRODUCTION

Image and video coding is important for most of thedigital image/video applications, which achieves high com-pact expression for image/video by reducing various typesof redundancy. Traditional image coding methods compresseach image individually by intra prediction [1][2] and entropycoding to reduce spatial redundancy and statistical redundancy,respectively. For lossy compression, quantization technique isutilized to reduce the psychovisual redundancy, e.g., differentquantization steps applied to coefficients in different frequencybands in JPEG compression [3]. In addtion, there are also inter-image redundancy among similar images (also named as setredundancy [4]), which is similar with that in videos [5][6].With the development of cloud storage, a large amount ofsimilar images are uploaded to the cloud but still compressedindividually, and this consumes lots of storage space sincecommon content shared by similar images are stored repeat-edly. Therefore, an image set coding technique could furtherimprove compression efficiency by jointly compressing a setof similar images to reduce the set redundancy among them.

During the last two decades, many image set compressionmethods have been proposed, which can be roughly classifiedinto two categories according to the prediction structures, i.e.,pairwise prediction method and sequential prediction methodillustrated in Fig. 1. Pairwise prediction methods are usuallywith a star topological prediction structure, in which onerepresentative image is selected or generated from one imageset, compressed independently, and then the others (denoted

as predicted images) in image set are compressed by referringto the reconstructed representative image. Karadimitriou et al.proposed some representative image generation methods, in-cluding max-min differential (MMD) method [4] and centroidmethod [7]. The MMD method generates two representativeimages, ”min” image and ”max” image, which are constructedby choosing the minimum and maximum pixel values forevery pixel position, and then compresses difference betweenoriginal pixel values and either the min or the max image(whichever is smaller) for every image in the set. The centroidmethod generates one representative image by averaging thepixel values in the same position among all the images, andthen the average image and the difference images betweenthe representative image and predicted images are compressedindividually. Yeung et al. [8] take a low frequency template asthe representative image by only averaging the low frequencycomponents of all images to capture the common patterns in setof similar images while discarding variation content. However,these methods cannot handle images with large scale geometricdeformations, and also need to compress extra representativeimage(s).

(a) (b)

Fig. 1. Prediction structure for image set compression. (a) star topologicalprediction structure, (b) sequential prediction structure.

Based upon the success of video coding, Zou et al. pro-posed an image set compression and management methodin [9], which arranges the set of similar images as a videosequence, and compresses them with HEVC inter codingdirectly. The coding efficiency for a given image set is mainlydetermined by the image coding order. Zou et al. take thesum of absolute difference (SAD) of motion compensationbetween any two images to measure their correlation, basedon which the coding order is derived by finding a minimumspanning tree (MST). Due to irregular large scale motionexisting among similar images which makes traditional motion

978-1-4799-8391-9/15/$31.00 ©2015 IEEE 1274

estimation inefficient, Shi et al. [10] [11] take advantage ofthe distance of matched SIFT descriptors between any twoimages to measure their correlation. However, these methodsincrease the delay for image access severely, e.g., accessing theimages in leaf node positions need decode all the dependentprecedents. Therefore, the tree depth is usually constraint[11][12][13]. When the depth is one, the prediction structureis the same as the star topological prediction structure.

In this paper, we propose an inter-image redundancymeasure to find the best representative image for image setcompression. For an image, its inter-image redundancy relativeto another image can be considered as how much similarcontent existing in it (denoted as the extent of similar content,EOS) and how correlated the similar content (denoted as thecorrelation of similar content, COS) being. We take the twoaspects jointly to estimate the inter-image redundancy. Forthe EOS of an image, we take the size of the covered areaby matched SIFT points on the other images to measure it.The larger the covered area in other images, the more itsinter-image redundancy exists. For the COS of an image, wetake the distance of matched SIFT descriptors between twoimages to measure it. The smaller distance of the matchedSIFT descriptors is, the more correlated the covered areaby SIFT points between the two images is. Based on theinter-image redundancy measure, we propose an image setcompression method with star topological prediction structureby selecting the representative image with maximum inter-image redundancy, which is compressed with intra codingmethod. The other images are compressed by referring tothe reconstructed representative images which is processed bygeometric and photometric transform just as that in [11].

The remainder of the paper is organized as follows. Insection II, we review the SIFT based image set compressionframework. Section III introduces the proposed inter-imageredundancy measure method and introduces the image set com-pression method in detail. Experimental results are reported inSection IV and Section V concludes the paper.

II. REVIEW OF SIFT-BASED IMAGE SET COMPRESSION

Scale Invariant Feature Transform (SIFT) [14] is an algo-rithm in computer vision to detect and describe local featuresin images, which is invariant to scale, rotation, illuminationand viewpoint. Therefore, the SIFT descriptor is well suitablefor finding the similar content among images by searchingmatched SIFT points. For an image In, there are a set ofSIFT points and the k-th SIFT descriptor is formulated asfn(k) = {pn(k),vn(k)}, where pn(k) = {x1, x2, s, o}represents the coordinate, scale and orientation parameters,while vn(k) is a 128-dimensional local feature descriptorvector.

In [11], Shi et al. proposed an efficient image set compres-sion framework based on SIFT descriptors. Firstly, the clus-tered similar images are organized into a minimum spanningtree (MST) based on the correlation of similar images, wherethe distance of matched SIFT descriptors is used to measuretheir correlation. Then, the set of similar images are arrangedinto a sequence according to the depth-first traversing of theMST. The root image is firstly intra coded without inter-imageprediction. The other images are coded as inter ones predictedfrom their parents by geometric and photometric transform.

III. IMAGE SET COMPRESSION WITH SIFT-BASEDINTER-IMAGE REDUNDANCY MEASURE

A. SIFT-based inter-image redundancy measure method

The prediction order plays an important role in image setcompression for pairwise prediction structure and the sequen-tial prediction structure. For an image set S = {I1, I2, ..., In},the most straight-forward way to measure inter-image re-dundancy is to calculate the residual energy of the motioncompensation between any two images as follows,

e(Ii, Ij) = ‖Ii − T (Ij)‖2 (1)

where the operator T aligns image Ij to Ii based on motionvectors. However, traditional block based motion estimation isdifficult to handle the complexity variations among set of sim-ilar images, e.g., variations in scale, rotation and illuminance.

In this paper, we take SIFT descriptors to measure theextent and correlation of similar content, which are bothused to measure inter-image redundancy. For each images, Ij ,the SIFT points should be detected and described with localhistogram of gradients. Since the coding efficiency does nothave significant difference for smooth area with intra coding orinter coding, we only utilize the SIFT points in the area withlarge local variations to measure the inter-image redundancy.Then, we refine the SIFT points by deleting those ones, whichare in smooth area. To be specific, we discard the SIFT pointswhen the local variance of its neighboring pixels is smallerthan a given threshold, e.g., 30 for image with pixel range [0,255]. In addition, the left SIFT points are further refined bya geometric transform. A homography matrix Hi,j , which isderived through the RANSAC [15] method, is used to modelthe deformation between images Ii and Ij and constrain themapping homogeneity of matched SIFT points. For each pairof matched SIFT points, the deviation of their locations underthe model Hi,j is defined as,

D(ki,j) = ‖xi(ki,j)− xj(ki,j)×Hi,j‖2 (2)

where xi(ki,j) is the coordinate vector, xi(ki,j) = {x1, x2}.The matched SIFT points with deviation D larger than a giventhreshold are also removed. Based on these refinements, mostof the SIFT points left are well related with similar contentbetween two images, e.g., in Fig.2.

In order to measure the COS of inter-image redundancy,we define the distance of matched SIFT descriptors as follows,

d(fi(ki,j), fj(ki,j)) =1

|ui(ki,j)|‖ui(ki,j)− uj(ki,j)‖2 (3)

un(i) =vn(i)

‖vn(i)‖(4)

where |ui(ki,j)| is the number of elements in ui(ki,j). Then,the average distance of all the SIFT points, d̂i,j , is utilized tomeasure the correlation of the similar content between Ii andIj .

di,j =1

Ni,j

Ni,j∑k(i,j)=1

d(fi(ki,j), fj(ki,j)) (5)

where Ni,j is the number of the matched SIFT points betweenIi and Ij .

1275

(a) Matched SIFT points beforesmoothness detection and homogeneity constraint

(b) Matched SIFT points aftersmoothness detection and homogeneity constraint

Fig. 2. The effect of SIFT point refinement.

In order to measure the EOS of inter-image redundancy, wefurther introduce the covered area by SIFT points to estimate it.For image Ii with size of Hi×Wi, its extent of similar contentfor image Ij with size of Hj ×Wj is measured by proportionof area covered by matched SIFT points to the whole imagearea as follows,

pi,j =si,j

Hj ×Wj(6)

Here si,j is the covered area by matched SIFT points in imageIj , which is calculated by adding up the areas of all thetriangles generated from the Delaunay triangulation of thediscrete SIFT point set. An example is illustrated in Fig. 3.There are three different images (the left two images are thesame one actually) in Fig. 3, and the similar content takes upa large proportion to the right image while taking up a smallproportion to the left image. Therefore, the left image has muchmore inter-image redundancy for the other two images thanthat of the right images.

Based on the two parts, the inter-image redundancy mea-sure for image Ii is formulated as follows,

Qi =1

|S|∑j∈Sj 6=i

(pi,j + (1− di,j)

)(7)

where |S| is the number of similar images in set S. Thelarger value of Qi corresponds to more inter-image redundancyrelative to set of other images.

B. Image set compression scheme based on inter-image redun-dancy measure

Considering the low latency of image access, we takethe star topological prediction structure to compress imageset, which only has one image delay for the predicted imageaccess. The framework of our image set compression method

Fig. 3. The prediction area covered by matched SIFT points.

is illustrated in Fig. 4. The SIFT points are first extractedfrom every image and then refined by smoothness detectionand homogeneity constraint. Second, the distance of matchedSIFT descriptors and the covered area are calculated for anytwo images, then the representative image is selected with themaximum inter-image redundancy. Third, the representativeimage is compressed with HEVC intra coding method. Finally,the reconstructed representative image is further processedwith geometric transform and photometric transform just as in[11] to make it approach the corresponding predicted image.Then, the predicted images are compressed with HEVC intercoding method by referring to their corresponding transformedrepresentative images.

Fig. 4. Image set compression framework in our method.

IV. EXPERIMENTAL RESULTS

In this section, we evaluate the performance of our pro-posed inter-image redundancy measure method for image setcompression by comparing with state-of-the-art image setcompression method (denoted as Shi′s method) in [11], andHEVC intra/inter coding methods. We take the two imagesets used in [11], RockBoat with 20 images, WadhamCollegewith 5 images. For each set of images, we first apply ourproposed inter-image redundancy measure method and theSIFT distance based method used in Shi′s method to searchthe representative image. Then, the representative image is

1276

compressed with HEVC intra coding method and the other im-ages compressed with HEVC inter coding method by referringto the reconstructed representative image. For our proposedmethod and Shi′s method, the reconstructed representativeimage is processed by geometric transform and photometrictransform according to the matched SIFT points to gener-ate more approached reference images for the correspondingpredicted image. For HEVC inter coding, the reconstructedrepresentative image is directly used as reference image forpredicted image coding. For HEVC intra coding, each of theimages are compressed individually with HEVC intra codingmethod.

There are different representative images based on ourproposed method and Shi′s method for the two image sets.Fig. 5 shows the rate distortion curves for the two imagesets compression with different methods. Compared with othermethods, the compression performance with the representativeimage determined by our proposed method is improved sig-nificantly. Table I illustrates the bitrate saving compared withHEVC intra coding method. For image set WadhamCollege,our method achieves bitrate saving up to 20.8%. These re-sults show that the representative images determined by ourproposed method have more redundancy content for imageset and provide better prediction for other images than thatin Shi′s method. Therefore, the EOS and COS of similarcontent together play an important role in measuring inter-image redundancy.

30

31

32

33

34

35

36

37

38

39

40

41

0 0.1 0.2 0.3 0.4 0.5

PSNR(dB)

Rate(bpp)

WadhamCollege

HM-intra

HM-inter

Shi's method

Proposed method

35

36

37

38

39

40

41

42

43

44

45

46

0 0.1 0.2 0.3 0.4 0.5 0.6

PSNR(dB)

Rate(bpp)

RockBoat

HM-intraHM-interShi's methodProposed method

(a) (b)Fig. 5. Rate-distortion curves for image set compression.

TABLE I. THE BITRATE SAVING COMPARED WITH HEVC INTRACODING METHOD

Sequence Name HEVC-inter Shi′s method Proposed method

WadhamCollege 0.0% -11.6% -20.8%

RockBoat -1.6% -10.9% -16.6%

V. CONCLUSION

In this paper, we have proposed an inter-image redundancymeasure for image set compression. The proposed methodmeasures the inter-image redundancy by jointly estimating theextent and the correlation of similar content between images.The area covered by matched SIFT points and the distanceof matched SIFT descriptors are used to estimate the EOSand COS, respectively. We apply the proposed inter-imageredundancy measure to determine the representative imagefor image set compression with star topological predictionstructure. Experimental results demonstrate that the determinedrepresentative images by our method can provide better pre-diction for other images and achieve compression performance

improvement. In addition, our proposed method is a generalinter-image redundancy measure, which is not limited to selecta representative image for star topological prediction structurebut also can be used to determine coding order with sequentialprediction structure.

ACKNOWLEDGMENT

This research was carried out at the Rapid-Rich ObjectSearch (ROSE) Lab at the Nanyang Technological University,Singapore. The ROSE Lab is supported by the NationalResearch Foundation, Prime Ministers Office, Singapore, underits IDM Futures Funding Initiative and administered by theInteractive and Digital Media Programme Office.

REFERENCES

[1] G. Sullivan, J. Ohm, W.-J. Han, and T. Wiegand, “Overview of thehigh efficiency video coding (HEVC) standard,” IEEE Transactions onCircuits and Systems for Video Technology, vol. 22, no. 12, pp. 1649–1668, Dec. 2012.

[2] X. Zhang, R. Xiong, S. Ma, and W. Gao, “New image coding schemewith hierarchical representation and adaptive interpolation,” Nov. 2011,pp. 1–4.

[3] G. Wallace, “The JPEG still picture compression standard,” IEEETransactions on Consumer Electronics, vol. 38, no. 1, pp. xviii–xxxiv,Feb. 1992.

[4] K. Karadimitriou and J. M. Tyler, “Min-max compression methods formedical image databases,” SIGMOD Rec., vol. 26, no. 1, pp. 47–52,Mar. 1997.

[5] X. Zhang, R. Xiong, S. Ma, and W. Gao, “Adaptive loop filter withtemporal prediction,” in Picture Coding Symposium (PCS), 2012, May2012, pp. 437–440.

[6] ——, “Adaptive loop filter merge in temporal domain,” JCT-VC F498,JCT-VC Meeting, Torino, Jul. 2011.

[7] K. Karadimitriou and J. M. Tyler, “The centroid method for compressingsets of similar images,” Pattern Recognition Letters, vol. 19, no. 7, pp.585–593, May 1998.

[8] C.-H. Yeung, O. Au, K. Tang, Z. Yu, E. Luo, Y. Wu, and S.-F. Tu,“Compressing similar image sets using low frequency template,” in2011 IEEE International Conference on Multimedia and Expo (ICME),Jul. 2011, pp. 1–6.

[9] R. Zou, O. Au, G. Zhou, S. Li, and L. Sun, “Image set modeling byexploiting temporal-spatial correlations and photo album compression,”in Signal Information Processing Association Annual Summit andConference (APSIPA ASC), 2012 Asia-Pacific, Dec. 2012, pp. 1–4.

[10] Z. Shi, X. Sun, and F. Wu, “Feature-based image set compression,” in2013 IEEE International Conference on Multimedia and Expo (ICME),Jul. 2013, pp. 1–6.

[11] ——, “Photo album compression for cloud storage using local features,”IEEE Journal on Emerging and Selected Topics in Circuits and Systems,vol. 4, no. 1, pp. 17–28, Mar. 2014.

[12] R. Zou, O. Au, G. Zhou, W. Dai, W. Hu, and P. Wan, “Personal photoalbum compression and management,” in 2013 IEEE InternationalSymposium on Circuits and Systems (ISCAS), May 2013, pp. 1428–1431.

[13] Y. Ling, O. Au, R. Zou, J. Pang, H. Yang, and A. Zheng, “Photo albumcompression by leveraging temporal-spatial correlations and HEVC,” in2014 IEEE International Symposium on Circuits and Systems (ISCAS),Jun. 2014.

[14] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,”Int. J. Comput. Vision, vol. 60, no. 2, pp. 91–110, Nov. 2004.

[15] M. A. Fischler and R. C. Bolles, “Random sample consensus: Aparadigm for model fitting with applications to image analysis andautomated cartography,” Commun. ACM, vol. 24, no. 6, pp. 381–395,Jun. 1981.

1277