Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

A HIGH PERFORMANCE ALGORITHM FOR PUZZLE RECONSTRUCTION

PROBLEM

CHUN-WEI TSAI1, SHIH-PANG TSENG2, MING-CHAO CHIANG2, CHU-SING YANG3

1 Applied Geoinformatics, Chia Nan University of Pharmacy & Science, Tainan, Taiwan, R.O.C. 2Computer Science and Engineering, National Sun Yat-sen University, Kaohsiung, Taiwan, R.O.C.

3Electrical Engineering, National Cheng Kung University, Tainan, Taiwan, R.O.C.

E-MAIL: cwtsaiOS07@gmail.com.tsp@maiLtajen.edu.tw.mcchiang@cse.nsysu.edu.tw.csyang@ee.ncku.edu.tw

Abstract: as shown in Fig. l(b). Both the jigsaw puzzle problem and

Since a puzzle solver, for the puzzle reconstruction problem, the edge-matching puzzle problem have been shown to be NP­

can be applied to many other real world problems, various stud- complete [7]. Because they are difficult for humans to solve, a

ies have focused on improving the end result of the puzzle solvers high performance search algorithm is needed to automatically

they proposed for several years. In spite of these efforts, the puzzle find the solution by using a limited computation cost. Appar­

reconstruction problem, however, has never fully solved by nsing endy, the amount of time needed to reconstruct the puzzle de­

a search algorithm with a limited computation time. In this paper, pends on both the size of the puzzle and the shape of the pieces.

an effective search algorithm is presented for the puzzle recon­

struction problem. The proposed algorithm uses ant colony opti­

mization to guide the search directions toward the global optimal

solution, the color information to measure the similarity between

pairs of puzzles, and an effective reconstruction strategy to im­

prove the end result. To evaluate the performance of the proposed

algorithm, we compare it with several state-of-the-art puzzle re­

construction algorithms. The simulation results show that the pro­

posed algorithm outperforms all the state-of-the-art algorithm we

compared in this paper.

Keywords:

Puzzle reconstruction problem; ant colony optimization; ge­

netic algorithm

1. Introduction

The jigsaw puzzle problem [22, 14] is a traditional game that dates back to at least 1760s [13]. The puzzle reconstruc­

tion problem can be divided into two categories: the apicto­rial reconstruction for the jigsaw puzzle and the pictorial re­

construction for the edge-matching puzzle [7, 14]. As shown

in Fig. l(a), the shape information can be used to reconstruct the jigsaw puzzle (i.e., the apictorial puzzle). Unlike the jig­

saw puzzle problem, pieces of the edge-matching puzzle (i.e., the pictorial puzzle) are all in the same shape (e.g., square),

978-1-4673-1487-9/12/$31.00 ©2012 IEEE

(a) Jigsaw puzzle (b) Edge-matching puzzle

Figure 1. Example showing the differences be­tween the jigsaw and edge-matching puzzle problems.

In addition to solving the jigsaw puzzle problem, the solution

to the puzzle reconstruction problem is important in that it can

be applied to many real world problems, such as reconstruct­ing shredded documents [IS], reassembling unknown broken

objects (e.g., ancient objects) [6], and reassembling DNAIRNA

[17]. However, although many studies [14, 1, 24, 20, 3, 25] have focused on solving the puzzle problem described in [10],

the end result is far from optimal, especially when the number of pieces of a puzzle is more than a 100 or so.

Our observation shows that the key to improving the quality

1698

Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

(the accuracy rate) of a puzzle solver lies in two things: (1) find­

ing a "good" similarity measure to ensure that each combina­tion of pieces of a puzzle is correct, and (2) finding an effective

search algorithm to guide the global and local search directions towards the region that has a high potential to find the optimal

solution. For this reason, this paper is focused on three im­

portant research issues: (1) an effective search algorithm (ant colony system, or ACS for short [8]) to guide the search direc­

tions towards the global optimal solution, (2) a "good" similar­ity measure to improve the accuracy rate of the puzzle solver,

and 3) a reconstruction strategy to improve the end result of the

search algorithm used in this paper.

The remainder of the paper is organized as follows. Section 2

first defines the puzzle reconstruction problem and then gives a brief introduction to the ant colony optimization and its vari­

ants. Section 3 describes in detail the proposed algorithm. The experimental results are discussed in Section 4. Also discussed

in this section are the data sets we evaluated and the parameter

settings. We conclude our work in Section 5.

2. Related Works

2.1 The Puzzle Reconstruction Problem

This paper is focused on the edge-matching puzzle problem (the pictorial puzzle problem); i.e, an N by M rectangle puzzle

where N and M denote, respectively, the number of columns and the number of rows. In [24], this problem is defined as

an n x m puzzle piece arrangement, i.e., a "partial piece ar­rangement," with 1 :S n :S Nand 1 :S m :S M. To measure

the performance of the puzzle solver (the method that automat­

ically solves the problem), the accuracy rate (the correct rate) usually used [24, 25] is defined as

Nc (}1 Or = Nt

x 10010, (1)

where Nc denotes the number of correct pieces and Nt = N x

M denotes the total number of pieces of the puzzle in question.

2.2 Solutions to Puzzle Reconstruction Problem

2.2.1 Apictorial Puzzles

To the best of our knowledge, the first algorithm for apic­

torial puzzles was presented by Freeman and Garder in 1964 [10]. Although the algorithm they proposed can only solve a

nine-piece apictorial puzzle reconstruction problem, it eventu­ally inspires many researches on this problem. Because the

shape and color [4] of each piece of a apictorial puzzle are very

different from each other, such as the inner and outer contours of a piece, the shape information of each piece is very impor­

tant to the algorithm of this problem. In [16], Markridis and Pa­pamarkos integrated many technologies (binarization, contour

boundary extraction, comer detection, and color reduction) into

their puzzle solver.

In addition to using the shape and color information, how to

extract the features of each piece of PRP and represent them has become another important research issue because these tech­

nologies can be used to improve the performance of a puzzle solver. For instance, Goldberg et al. [11] attempted to use the

inflection and tangent points of indents and outdents to measure

the fitness of the combinations. In [15], the feature set of each piece is used to compute its distance to the others while the

sampled color information is also used to calculate the fitness between pieces. In [3], Burdea and Wolfson pointed out that the

puzzle reconstruction problem can be regarded as the traveling

salesman problem (TSP) in the sense that solution to the TSP can be used to solve the PRP. Their approach is to first find the

four comers of the frame (image) and then find the other pieces that have a high probability of combining with the four comers.

In the same research, Burdea and Wolfson also pointed out that integrating the global and local search will provide a better so­

lution to PRP.

2.2.2 Pictorial Puzzles

Unlike apictorial puzzles, most studies on solving pictorial puzzles [24, 20, 25, 1, 21] are focused on using high perfor­

mance search algorithm to find the better results. This is be­

cause for this kind of problem, the shape of each piece is the same (e.g., square). Hence, the search algorithm cannot use

shape information to combine pieces in the process of reassem­bling the whole image or frame. Some of the above-mentioned

studies [24, 20] also pointed out that the pictorial puzzle re­

construction problem is much more difficult than the apictorial puzzle reconstruction problem.

Among others, genetic algorithm [24, 20], ant colony op­

timization [25], and Hungarian method [1] are usually used

to solve this kind of puzzle reconstruction problem. In [24], Toyama et al. used only the binarized pixel values on the border

line of each piece to represent the pieces of the image. More­over, an effective crossover operator was presented to exchange

a column of pieces or more (i.e., N pieces or more) at a time.

The same exchange method is applied to exchange pieces on a row or more. In brief, the basic idea of this method is not to

break up the solution structure of the original solutions (chro­mosomes). An extended study [20] used the color information

1699

Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

and modified the evaluation function to improve the end re­

sult of GA for pictorial puzzles. An iterative algorithm using Hungarian method was presented in [1]. The simulation results

show that it is a very effective algorithm for pictorial puzzles in

the sense that it can provide a 100% accuracy rate for most of

the test images with up to 8 by 8 pieces. Recently, Zhao et al.

[25] presented two algorithms based on human heuristics and ant colony system (ACS) [8] to solve the pictorial puzzle prob­

lem. According to our observation, because the methods they proposed use only ACS, but not much technologies designed

specifically for puzzle reconstruction, to solve this problem,

the simulation result is not better than that of [20]. A simple comparison shows that the method based on GA [20] provides

96.0% of accuracy rate for the 8 x 6 puzzles while the method based on ACS [25] can only provide 50% of accuracy rate for

the 4 x 8 puzzles. However, we are not arguing that ACS per­

forms worse than GA. Instead, we are aimed to emphasize that not only the other information available in each piece of the

puzzle but also the solution construction method are very im­portant for the puzzle reconstruction problem because they will

affect the final result of the puzzle solver.

3. The Proposed Algorithm

According to the observation of studies [11, 1], the factors

that affect the final results of the puzzle solvers are not only the

similarity measure but also the global and local search capa­

bility of the employed search algorithm. An efficient algorithm

based on ant colony system is presented in this paper, called ant colony optimization-based puzzle solver (ACOPS). The major

differences between the proposed algorithm ACOPS and the other ACO-based puzzle solvers [25] are as follows:

• We redesign the subsolutions (pieces) construction

method to increase the diversity of solutions (ants) at each iteration.

• The pheromone compute and update method to reinforce

the high performance subsolutions have a better chance to

be retained at later iterations.

• We use color information of each piece of puzzle to com­

pute the similarity between pieces of the puzzle and use it to decide if pieces of the puzzle should be combined.

3.1 The ant colony optimization-based puzzle solver

As shown in Fig. 2, the proposed algorithm is an extension of ant colony optimization. Note that the proposed algorithm

ACOPS is based on ant colony system, but the new operators

we proposed herein can be applied to other ant colony opti­mization algorithms.

1 Initialize solutions.

2 While the tennination criterion is not met

Construct all the solutions using Eqs. (3), (4), and (5).

Update pheromone matrix by using Eqs. (6) and (7).

Perform local search operator.

6 End 7 Output the result.

Figure 2. The ACOPS.

Like traditional ACO algorithms, the very first step of the

proposed algorithm is to initialize the solutions randomly. In addition, the probability p7j for choosing the next sub solution

(pieces) is the same as ACS, which is defined as

(2) otherwise,

where Tij denotes the pheromone value between subsolutions i

and j; TJij the heuristic value, which is also referred to as the heuristic information.

3.1.1 Subsolution Construction Method

In contrast to the traditional ACO (ACS), the proposed algo­

rithm uses a fundamentally different approach to constructing the subsolutions (pieces) of a puzzle. In other words, the tradi­

tional ACO is designed for the traveling salesman problem, and it constructs the subsolutions step by step (i.e., choosing a city

an ant is to go at each iteration). As such, the method described

in [25] is not suitable for the puzzle reconstruction problem. Many studies [11] first find the pieces on the four corners and

then find the best fitting pieces for the borders of the puzzle. The puzzle solver then will tum to find the best fitting pieces

to the rest of the puzzle. However, the puzzle solver (the au­tomatic puzzle reconstruction algorithm) may fall into a local

minimum because the diversity of the solutions is limited. For

this reason, ACOPS uses four different construction methods denoted, respectively, by MI, M2, M3, and M4, each of which

take a different direction, to solve the puzzle. As described in Fig. 3, each ant will randomly choose one of the four construc­

tion methods to construct its solution. That is, ant choosing MI

will construct each column bottom up and all the columns from

left to right; ant choosing M2 will construct each column top

down and all the columns from right to left; ant choosing M3 will construct each row from left to right and all the rows top

1700

Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

down; and ant choosing M4 will construct each row from right

to left and all the rows bottom up.

•... .... .... .... ........ ..... :::: •... :::: .... ... .... ... ... .... . .. ..

. . .. . ... .. -. , . . . .. . . .. .

Figure 3. Example showing the four subsolution construction methods.

3.1.2 Pheromone Compute and Update Method

To simplify the discussion of the pheromone compute and

update method, let 6rs define the similarity between the border columns or rows of pieces r and s of the puzzle as follows: 1

6rs = L Ilvrk - Vsk II, k

(3)

where Vrk and Vsk denote the values of the kth pixel of the columns or rows of pieces r and s that are next to each other.

For any side z of a piece at the border of the puzzle that is not

next to any other piece, 6z. is defined to be 0, and TJz. is defined to be 1 where· denotes a non-existing piece. Also, as shown in

Fig. 4, let us define Ti as

(4)

and TJi as

Till,") � TJl(i,al W

Td(i,b) TJd(i,b)

Figure 4. Example showing how the pheromone and the similarity are computed, which depend on which sides of Pi are next to Pa and Pb•

If Pi and Pa are not associated with the best ant of the current iteration, then

(7)

After the update, if T is larger than Tmax, then T is set to Tmax. If T is smaller than T min, then T is set to T min. The fitness of the

final result of a puzzle F is defined as

F = L6ij. (8)

Vi,j

4. Simulation Results

4.1 Simulation Environment, Parameter Settings, and Data Sets

The empirical analysis was conducted on a Sun X4150 ma­chine with 2.5 GHz Xeon CPU and 4GB of memory using Cen-

tOS 5.4 running Linux 2.6.18. Moreover, all the programs are

(5) written in C++ and compiled using g++ (GNU C++ compiler).

where eia and eib denote, respectively, the sides of pieces i and

a and the sides of pieces i and b that are next to each other,

i.e., the sides for which the pheromone Ti and the similarity

TJi are computed, which depend on the sub solution construc­tion method chosen. For instance, the example given in Fig. 4 shows that the proposed algorithm will consider the similarity

between the left side of Pi and the right side of Pa and the sim­

ilarity between the down side of Pi and the top side of Pb. In other words, in this case, e1 denotes the left side of Pi, and e2 denotes the down side of Pi.

If Pi and Pa are associated with the best ant of the current

iteration, then

Tia = ( 1 - P)Tia + p6T (6)

1 That is, we are assuming that all the pieces of the puzzle in question are

sequentially numbered.

As far as this paper is concerned, the population size is set equal

to 25, and the number of iterations is set equal to 1,000. Also, for the ACO-based algorithm, the initial value of T is set equal

to 0.5; the maximum value of T, denoted Tmax, is set equal to

0.99; the minimum value of T, denoted Tmin, is set equal to

1 - Tmax; 6T is set equal to 1.0; p, 0:, and fJ are set equal to

0.1,1, and 2. As shown in Fig. 5, nine well-known images from

different sources are employed in evaluating the performance of

the proposed algorithm (ACOPS) as well as ACO-based algo­

rithm [25] and GA-based algorithm [24] that we compared in this paper.

4.2 Results

Table 1 shows the results of carrying out the ACO [25], the TFSM [24], and the proposed algorithm for 30 rounds for all

1701

Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

Airplane [23]

512 x 512

Peppers [23]

512 x 512

Painting [2]

2024 x 2365

Baboon [23]

512 x 512

City gate [5]

866 x 603

Mona Lisa [19]

687 x 1024

Lerma [23]

512 x 512

Horses [9]

454 x 801

Impression [12]

1051 x 808

Figure 5. Data sets used in the simulations of this paper.

the data sets with a different number of pieces for the puzzle

reconstruction problem. Note that the accuracy rate is com­puted by using Eq. (1). As shown in Table 1, the proposed

algorithm not only outperforms the other state-of-the-art algo­rithms we compared in this paper but also 100% solves the

puzzle reconstruction problem for most of the simulations we

conducted. Some of the simulation results show that the pro­posed algorithm may degrade the accuracy when the number of

pieces is 12 x 12. But the degradation is small except for ba­boon. According to our observation, this is because the pieces

of the baboon puzzle are very similar to each other. In other words, there are simply not enough texture to differentiate all

the pieces. This explains why the accuracy rate will be down,

especially when the number of pieces is increased.

5. Conclusions

In this paper, we present an effective algorithm for improv­

ing the accuracy rate of puzzle reconstruction algorithm. The

proposed algorithm is based on ant colony system and is called ant colony optimization-based puzzle solver (ACOPS), which

is motivated by the observation that the puzzle reconstruction

algorithm needs to take into account the capability of global

Table 1. Comparison of ACOPS with ACO and TFSM.

ACO [25] TFSM [24] ACOPS

DS # pieces Accuracy Accuracy Accuracy

Airplane 4x4 100.00% 100.00% 100.00% Airplane 8x8 86.94% 64.48% 100.00% Airplane 12 x 12 24.23% 27.07% 100.00%

Baboon 4x4 100.00% 100.00% 100.00% Baboon 8x8 36.24% 77.76% 99.67% Baboon 12 x 12 10.98% 36.69% 78.55%

Peppers 4x4 100.00% 100.00% 100.00% Peppers 8x8 99.90% 69.24% 100.00% Peppers 12 x 12 16.09% 30.92% 100.00%

Lenna 4x4 100.00% 98.69% 100.00% Lenna 8x8 66.72% 54.78% 100.00% Lenna 12 x 12 13.67% 28.19% 100.00%

City gate 4x4 100.00% 98.25% 100.00% City gate 8x8 22.02% 74.61% 100.00% City gate 12 x 12 7.12% 45.45% 97.52%

Horses 4x4 100.00% 99.63% 100.00% Horses 8x8 35.15% 66.95% 100.00% Horses 12 x 12 15.11% 31.76% 99.62%

Painting 4x4 100.00% 98.19% 100.00% Painting 8x8 100.00% 73.83% 100.00% Painting 12 X 12 35.40% 39.72% 100.00%

Mona Lisa 4x4 100.00% 99.38% 100.00% Mona Lisa 8x8 38.83% 55.20% 100.00% Mona Lisa 12 x 12 13.55% 25.09% 98.27%

Impression 4x4 99.75% 99.00% 100.00% Impression 8x8 19.14% 74.37% 100.00% Impression 12 x 12 5.21% 36.21% 95.84%

and local search, and the similarity measure to find a better re­

sult. For this reason, in this research, the ant colony system is employed to search for the global and local solutions. We

also present the sub solution construction methods and the new pheromone compute/update methods to improve the quality of

the end result. The main contributions of this paper are three­

fold: (1) the proposed algorithm can effectively reconstruct most of the puzzles; (2) the proposed algorithm can be easily

combined with other ACO algorithms to improve their perfor­mance; and (3) the proposed algorithm can be easily adapted to

different kinds of combinatorial optimization problems, such

as reassembling shredded documents, reassembling unknown broken objects, and reassembling DNAIRNA. In the future, we

will focus on finding more efficient operators to enhance the performance of ACO for the puzzle reconstruction problem.

1702

Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

Acknowledgments

This work was supported in part by the National Sci­

ence Council, Taiwan, R.O.C., under Contracts NSClOO-2218-E-041-001-MY2, NSC99-2221-E-llO-OS2, NSClOO-2218-E-

006-037, and NSC100-2219-E-006-001.

References

[1] N. Alajlan. Solving square jigsaw puzzles using dynamic pro­

gramming and the hungarian procedure. American Journal of

Applied Sciences, 6(11):1942-1948, 2009.

[2] Art, 2012. Available: http://upload . wikimedia.

org/wikipedia/commons/2/2d/Jan_Vermeer_

van_Delft_Ol1. jpg.

[3] B. Burdea and H. Wolfson. Solving jigsaw puzzles by a robot.

IEEE Transactions on Robotics and Automation, 5(6):752-764,

1989.

[4] M. Chung, M. Fleck, and D. Forsyth. Jigsaw puzzle solver using

shape and color. In Proceedings of the Fourth International Con­

ference on Signal Processing, volume 2, pages 877-880, 1998.

[5] City gate, 2012. Available: http://upload . wikimedia.

org/wikipedia/commons/d/d2/Bianjing_city_

gate. JPG.

[6] H. C. da Gama Leitao and J. Stolfi. Information contents of

fracture lines. In Proceedings of the 8th International Confer­

ence in Central Europe on Computer Graphics, Visualization,

and Interactive Digital Media, volume 2, pages 389-395, 2000.

[7] E. D. Demaine and M. L. Demaine. Jigsaw puzzles, edge

matching, and polyomino packing: Connections and complex­

ity. Graphs and Combinatorics, 23(1):195-208, 2007.

[8] M. Dorigo and L. M. Gambardella. Ant colony system: A co­

operative learning approach to the traveling salesman problem.

IEEE Transactions on Evolutionary Computation, 1(1):53--66,

1997.

[9] Eight horses, 2012. Available: http://upload.

wikimedia.org/wikipedia/commons/7/74/

Giuseppe-Castiglione-Eight-Horses.jpg.

[10] H. Freeman and L. Garder. Apictorial jigsaw puzzles: the com­

puter solution of a problem in pattern recognition. IEEE Trans­

actions on Electronic Computers, EC-13:118-127, 1964.

[11] D. Goldberg, C. Malon, and M. Bern. A global approach to au­

tomatic solution of jigsaw puzzles. In Proceedings of the eigh­

teenth annual symposium on Computational geometry, pages

82-87, 2002.

[12] Impression, 2012. Available: http://upload.

wikimedia.org/wikipedia/commons/5/5c/

Claude_Monet%2C_Impression%2C_soleil_

levant%2C_1872. jpg.

[13] John Spilsbury. Jigsaw puzzle, 1766. Available: http:

//www.bl.uk/learning/artimages/maphist/

minds/jigsawpuzzle/jigsawpuzzle1766.html.

[14] F. Kleber and R. Sablatnig. Scientific puzzle solving: Current

techniques and applications. In P roceedings of the Conference

on Computer Applications and Quantitative Methods in Archae­

ology, pages 1-8,2009.

[15] D. A. Kosiba, P. M. Devaux, S. Balasubramanian, T. L. Gandhi,

and K. Kasturi. An automatic jigsaw puzzle solver. In Pro­

ceedings of the International Conference on Computer Vision & Image P rocessing, volume 1, pages 616--618, 1994.

[16] M. Makridis and N. Papamarkos. A new technique for solving

puzzles. IEEE Transactions on Systems, Man, and Cybernetics,

Part B: Cybernetics, 40(3):789-797,2010.

[17] W. Marande and G. Burger. Mitochondrial DNA as a genomic

jigsaw puzzle. Science, 318(5849):415, 2007.

[18] M. A. O. Marques and C. O. A. Freitas. Reconstructing strip­

shredded documents using color as feature matching. In Pro­

ceedings of the 2009 ACM symposium on Applied Computing,

pages 893-894, 2009.

[19] Mona Lisa, 2012. Available: http://upload.

wikimedia.org/wikipedia/commons/thumb/

e/ec/Mona_Llsa%2C_by_Leonardo_da_Vlnci%2C

from_C2RMF_retouched.jpg/687px-Mona_Lisa%

2C_by_Leonardo_da_Vlnci%2C from_C2RMF

retouched. jpg.

[20] T. Murakami, F. Toyama, K. Shoji, and J. Miyarnichi. Assembly

of puzzles by connecting between blocks. In Proceedings of

the International Conference on Pattern Recognition, pages 1-

4,2008.

[21] T. R. Nielsen, P. Drewsen, and K. Hansen. Solving jigsaw

puzzles using image features. Pattern Recognition Letters,

29(14):1924--1933,2008.

[22] B. Sierrnan. The jigsaw puzzle of digital preservation-an

overview. Liber Quarterly, 19:13-21,2009.

[23] The USC-SIPI Image Database, 2012. Available: http: / /

sipi.usc.edu/database/.

[24] F. Toyama, Y. Fujiki, K. Shoji, and J. Miyarnichi. Assembly of

puzzles using a genetic algorithm. In Proceedings of the Inter­

national Conference on Pattern Recognition, volume 4, pages

389-392, 2002.

[25] Y. X. Zhao, M. C. Su, Z. L. Chou, and J. Lee. A puzzle solver

and its application in speech descrambling. In Proceedings of

the WSEAS International Conference on Computer Engineering

and Applications, pages 171-176,2007.

1703