A Fully-Automated Solver for Multiple Square …...A Fully-Automated Solver for Multiple Square...

A Fully-Automated Solver for Multiple SquareJigsaw Puzzles Using Hierarchical Clustering

October 27, 2016

Zayd Hammoudehhammoudeh@gmail.com

Department of Computer ScienceSan José State University

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

1Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

IntroductionThesis Goals

Primary Goal: Develop a solver that can assemble multiplejigsaw puzzles simultaneously, with performance that exceedsthe state of the art.

Additional Goals:

I Define the first metrics that quantify the quality of outputsfrom a multi-puzzle solver

I Design visualizations for viewing the errors (if any) in asolver output

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

1Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

IntroductionThesis Goals

Primary Goal: Develop a solver that can assemble multiplejigsaw puzzles simultaneously, with performance that exceedsthe state of the art.

Additional Goals:

I Define the first metrics that quantify the quality of outputsfrom a multi-puzzle solver

I Design visualizations for viewing the errors (if any) in asolver output

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

2IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

IntroductionJigsaw Puzzles

I First jigsaw puzzle introduced in the 1760s. Modern jigsawpuzzles introduced in the 1930s.

I First computational jigsaw puzzle solver introduced in1964

I Solving a jigsaw puzzle is NP-complete [1, 2]

I Example Applications: DNA fragment reassembly,shredded document reconstruction, speech descrambling,and image editing

I In most cases, the original, “ground-truth” image isunknown.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

2IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

IntroductionJigsaw Puzzles

I First jigsaw puzzle introduced in the 1760s. Modern jigsawpuzzles introduced in the 1930s.

I First computational jigsaw puzzle solver introduced in1964

I Solving a jigsaw puzzle is NP-complete [1, 2]

I Example Applications: DNA fragment reassembly,shredded document reconstruction, speech descrambling,and image editing

I In most cases, the original, “ground-truth” image isunknown.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

2IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

IntroductionJigsaw Puzzles

I First jigsaw puzzle introduced in the 1760s. Modern jigsawpuzzles introduced in the 1930s.

I First computational jigsaw puzzle solver introduced in1964

I Solving a jigsaw puzzle is NP-complete [1, 2]

I Example Applications: DNA fragment reassembly,shredded document reconstruction, speech descrambling,and image editing

I In most cases, the original, “ground-truth” image isunknown.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

3IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

IntroductionJig Swap Puzzles

Jig Swap Puzzles – Variant of the traditional jigsaw puzzleI All pieces are equal-sized squaresI Substantially more difficult

Ground-Truth Image Randomized Jig Swap Puzzle

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

3IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

IntroductionJig Swap Puzzles

Jig Swap Puzzles – Variant of the traditional jigsaw puzzleI All pieces are equal-sized squaresI Substantially more difficult

Ground-Truth Image

Randomized Jig Swap Puzzle

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

3IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

IntroductionJig Swap Puzzles

Jig Swap Puzzles – Variant of the traditional jigsaw puzzleI All pieces are equal-sized squaresI Substantially more difficult

Ground-Truth Image Randomized Jig Swap Puzzle

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

4IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

IntroductionJig Swap Puzzle Types

There are three primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.

I Type 1: Puzzle dimensions and piece rotation are known.Only piece location is unknown.

I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.

I Mixed-Bag: Pieces come from multiple puzzles.

Mixed-Bag puzzles are the focus of this thesis.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

4IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

IntroductionJig Swap Puzzle Types

There are three primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.

I Type 1: Puzzle dimensions and piece rotation are known.Only piece location is unknown.

I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.

I Mixed-Bag: Pieces come from multiple puzzles.

Mixed-Bag puzzles are the focus of this thesis.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

4IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

IntroductionJig Swap Puzzle Types

There are three primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.

I Type 1: Puzzle dimensions and piece rotation are known.Only piece location is unknown.

I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.

I Mixed-Bag: Pieces come from multiple puzzles.

Mixed-Bag puzzles are the focus of this thesis.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

4IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

IntroductionJig Swap Puzzle Types

There are three primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.

I Type 1: Puzzle dimensions and piece rotation are known.Only piece location is unknown.

I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.

I Mixed-Bag: Pieces come from multiple puzzles.

Mixed-Bag puzzles are the focus of this thesis.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

4IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

IntroductionJig Swap Puzzle Types

There are three primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.

I Type 1: Puzzle dimensions and piece rotation are known.Only piece location is unknown.

I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.

I Mixed-Bag: Pieces come from multiple puzzles.

Mixed-Bag puzzles are the focus of this thesis.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

Introduction5Previous Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Previous WorkPaikin & Tal

Paikin & Tal [4] – Current State of the ArtI Greedy, kernel growing solver

I Supports Type 1, Type 2, and Mixed-Bag puzzles

I Immune to missing pieces

Limitations:I Poor Seed Selection: All decisions are made at runtime

using as few as 13 pieces

I Externally Supplied Information: The solver must betold the number of input puzzles

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

Introduction5Previous Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Previous WorkPaikin & Tal

Paikin & Tal [4] – Current State of the ArtI Greedy, kernel growing solver

I Supports Type 1, Type 2, and Mixed-Bag puzzles

I Immune to missing pieces

Limitations:I Poor Seed Selection: All decisions are made at runtime

using as few as 13 pieces

I Externally Supplied Information: The solver must betold the number of input puzzles

The Mixed-Bag Solver

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

6Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Mixed-Bag SolverBasic Structure

...

Mixed Bag

Final

Assembly

Hierarchical

Segment

Clustering

Segmentation Stitching

Final

Seed Piece

Selection

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

7Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Mixed-Bag SolverFoundations

Paikin & Tal’s AlgorithmI Begin each puzzle with a single piece

I Place all pieces around the expanding kernel

Alternate Jigsaw Puzzle Solving StrategyI Correctly assemble small puzzle regions (i.e., segments)

I Iteratively merge smaller regions to form large ones

I Advantages of this Approach:

I Reduces the size of the problem

I Provides structure to the unordered set of puzzle pieces.

The alternate strategy is the basis of the Mixed-Bag Solver

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

7Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Mixed-Bag SolverFoundations

Paikin & Tal’s AlgorithmI Begin each puzzle with a single piece

I Place all pieces around the expanding kernel

Alternate Jigsaw Puzzle Solving StrategyI Correctly assemble small puzzle regions (i.e., segments)

I Iteratively merge smaller regions to form large ones

I Advantages of this Approach:

I Reduces the size of the problem

I Provides structure to the unordered set of puzzle pieces.

The alternate strategy is the basis of the Mixed-Bag Solver

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

7Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Mixed-Bag SolverFoundations

Paikin & Tal’s AlgorithmI Begin each puzzle with a single piece

I Place all pieces around the expanding kernel

Alternate Jigsaw Puzzle Solving StrategyI Correctly assemble small puzzle regions (i.e., segments)

I Iteratively merge smaller regions to form large ones

I Advantages of this Approach:

I Reduces the size of the problem

I Provides structure to the unordered set of puzzle pieces.

The alternate strategy is the basis of the Mixed-Bag Solver

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

7Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Mixed-Bag SolverFoundations

Paikin & Tal’s AlgorithmI Begin each puzzle with a single piece

I Place all pieces around the expanding kernel

Alternate Jigsaw Puzzle Solving StrategyI Correctly assemble small puzzle regions (i.e., segments)

I Iteratively merge smaller regions to form large ones

I Advantages of this Approach:

I Reduces the size of the problem

I Provides structure to the unordered set of puzzle pieces.

The alternate strategy is the basis of the Mixed-Bag Solver

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

8Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Mixed-Bag SolverOverview

I The Mixed-Bag Solver is fully-automated. It makes noassumptions concerning piece orientation, puzzledimensions, or number of puzzles.

I Input: A bag of puzzle pieces

I Output: One or more disjoint, solved puzzles.

I The Mixed-Bag Solver consists of five distinct stages:

...

Mixed Bag

Final

Assembly

Hierarchical

Segment

Clustering

Segmentation Stitching

Final

Seed Piece

Selection

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag Solver9Assembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

AssemblerMixed-Bag Solver Component

I Role: Place the individual pieces in the solved puzzle.

I Mixed-Bag Solver is independent of the assembler used,giving the solver significant upgradability and flexibility.

I Assembler Used in this Thesis: Paikin & Tal

I Current state of the art

I Allows for more direct comparison of performance

I Natively supports Mixed-Bag puzzles

I Implementation: Assembler re-implemented as part ofthis thesis based off the description in [4]

I Written in Python and fully open source [5]

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag Solver9Assembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

AssemblerMixed-Bag Solver Component

I Role: Place the individual pieces in the solved puzzle.

I Mixed-Bag Solver is independent of the assembler used,giving the solver significant upgradability and flexibility.

I Assembler Used in this Thesis: Paikin & Tal

I Current state of the art

I Allows for more direct comparison of performance

I Natively supports Mixed-Bag puzzles

I Implementation: Assembler re-implemented as part ofthis thesis based off the description in [4]

I Written in Python and fully open source [5]

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

10Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

SegmentationMixed-Bag Solver Stage #1

I Segment: Partial puzzle assembly where this is a highdegree of confidence pieces are placed correctly.

I Role of Segmentation: Provide structure to the set ofpuzzle pieces by partitioning them into disjoint segments

I Input: A bag of puzzle pieces

I Output: Set of saved segments

I Relationship between Puzzle Pieces and Segments:

I Pieces from a single ground-truth input may be separatedinto multiple segments

I A piece can be assigned to at most one segment

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

11Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

SegmentationAlgorithm Overview

I Iterative process consisting of one or more rounds

I In each round, all pieces not yet assigned to a segmentare assembled as if all are from the same input image

I Segments of sufficient size are saved to be used in futureMixed-Bag Solver stages

I Pieces in a saved segment are not placed in future rounds.

I Segmentation terminates if all pieces are assigned to asaved segment or when no segment is larger than theminimum allowed size

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

12Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

SegmentationComposition of a Segment

I Starting a Segment: Each segment is created iterativelystarting with a single seed piece

I Definition of Best Buddies: Any pair of pieces that aremore similar to each other than they are to any other piece.

I Growing the Segment: Add to the segment any piecethat is a neighbor and best buddy of a segment member

I Trimming the SegmentI Articulation Point: Any piece whose removal disconnects

other pieces from the segment seed.I All articulation pieces pieces are removed from the segment.

I After the removal of the articulation points, any pieces nolonger connected to the seed are removed.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

13Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

SegmentationExample – Input Images

Image (a) – 805 Pieces [6]

Image (b) – 540 Pieces [7]

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

14Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

SegmentationExample – First Segmentation Round Output Image

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

15Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

SegmentationExample – Segmented Output Image

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

16Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

StitchingMixed-Bag Solver Stage #2

I Role of Stitching: Quantify the extent that any pair ofsegments is related.

I Input: All puzzle pieces and the set of saved segments

I Output: Segment overlap matrix

I Theoretical Foundation: If two segments are from thesame ground-truth image, they would eventually overlap ifone segment were to expand.

I Segments should be allowed, but not forced, to expand in alldirections.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

16Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

StitchingMixed-Bag Solver Stage #2

I Role of Stitching: Quantify the extent that any pair ofsegments is related.

I Input: All puzzle pieces and the set of saved segments

I Output: Segment overlap matrix

I Theoretical Foundation: If two segments are from thesame ground-truth image, they would eventually overlap ifone segment were to expand.

I Segments should be allowed, but not forced, to expand in alldirections.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

17Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

StitchingStitching Piece Location

I Mini-assembly (MA): Same as a standard assemblyexcept only a fixed number of pieces are placed .

I Stitching Piece (ζx ): A piece near the boundary of asegment that is used as the seed of a mini-assembly

I Segment Overlap: Maximum overlap between anymini-assembly for segment, Φi and another segment Φj .

OverlapΦi ,Φj = arg maxζx∈Φi

|MAζx

⋂Φj |

min(|MAζx |, |Φj |)(1)

I Asymmetry: In most cases:

OverlapΦi ,Φj 6= OverlapΦj ,Φi (2)

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

18Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

StitchingExample – Input Image

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

19Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

StitchingExample – Two Segment Images

Segment #1 Segment #2

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

20Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

StitchingExample – Stitching Piece Locations

Segment #1 Segment #2

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

21Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

StitchingExample – Stitching Piece Locations

Stitching Result from Segment #1

Segment Overlap:

OverlapΦ1,Φ2 = 0.83 (3)

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

22Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Hierarchical Segment ClusteringMixed-Bag Solver Stage #3

I A single ground-truth image may be comprised of multiplesegments.

I Role of Hierarchical Clustering: Merge all segmentsfrom the same input image into a single segment cluster.

I Input: All saved segments and the segment overlap matrix

I Output: A set of segment clusters

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

23Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Hierarchical Segment ClusteringCalculating the Initial Similarity Matrix

I Segment Overlap Matrix: A hollow matrix quantifying therelationship between each pair of segments.

I Hierarchical Clustering Similarity Matrix: A diagonalmatrix quantifying the similarity between segment pairs.

I Quantifying Similarity: Given n segments, the similaritybetween segments Φi and Φj is:

ωi,j =OverlapΦi ,Φj + OverlapΦj ,Φi

2(4)

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

24Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Hierarchical Segment ClusteringMerging Clusters

I After two clusters are combined, the similarity between themerged cluster and all other clusters must be recalculated.

I Single Link Clustering: The similarity between any twoclusters is equal to the maximum similarity between anytwo members in the clusters [8]

I The Mixed-Bag Solver must use single link clustering astwo clusters may only have two member segments that areadjacent.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

25Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Hierarchical Segment ClusteringExample – Single Linking

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

26Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Hierarchical Segment ClusteringExample – Single Linking

Segment 1 Segment 2 Segment 3 Segment 4

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

27Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Hierarchical Segment ClusteringExample – Single Linking

Segment Cluster 1 Segment 3 Segment 4

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

28Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Hierarchical Segment ClusteringExample – Single Linking

Segment Cluster 1 Segment Cluster 2

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

29Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Hierarchical Segment ClusteringTerminating Clustering

I The solver continues merging segment clusters until oneof two criteria is satisfied:

I Only a single segment cluster remains

I Maximum similarity between any segment clusters is belowa predefined threshold

I All remaining segment clusters are passed to the nextsolver stage.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

30Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Final Seed Piece SelectionMixed-Bag Solver Stage #4

Mixed-Bag Solver

I Role of Final Seed Selection: Determine the pieces thatwill be used as the seed for the final output puzzles.

I Input: Set of segment clusters

I Output: Final seed pieces

I A single seed piece is selected from each segment cluster

Paikin & Tal

I All puzzle seeds are selected greedily at run time, whichoften leads to poor decisions.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

30Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Final Seed Piece SelectionMixed-Bag Solver Stage #4

Mixed-Bag Solver

I Role of Final Seed Selection: Determine the pieces thatwill be used as the seed for the final output puzzles.

I Input: Set of segment clusters

I Output: Final seed pieces

I A single seed piece is selected from each segment cluster

Paikin & Tal

I All puzzle seeds are selected greedily at run time, whichoften leads to poor decisions.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

31Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Final Assembly StageMixed-Bag Solver Stage #5

I Role of Final Assembly: Generate the solved puzzlesthat are output by the Mixed-Bag Solver.

I Input: Set of puzzle pieces with the seeds marked

I Output: Final solved puzzles

I All pieces are placed around the seeds selected in theprevious stage.

I Assembly proceeds in this stage normally without anycustom modifications.

Quantifying Solver Quality

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

32Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Quantifying Solver Quality

I Jigsaw puzzle solvers are not able to always correctlyreconstruct the input puzzle(s)

I Metrics compare the quality of solver outputs

I Two Most Common Quality Metrics:

I Direct Accuracy

I Neighbor Accuracy

I Disadvantages of Current Metrics: Neither account for:

I Pieces misplaced in different puzzles

I Extra pieces from other puzzles

I Goal: Define new quality metrics for Mixed-Bag puzzles

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

32Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Quantifying Solver Quality

I Jigsaw puzzle solvers are not able to always correctlyreconstruct the input puzzle(s)

I Metrics compare the quality of solver outputs

I Two Most Common Quality Metrics:

I Direct Accuracy

I Neighbor Accuracy

I Disadvantages of Current Metrics: Neither account for:

I Pieces misplaced in different puzzles

I Extra pieces from other puzzles

I Goal: Define new quality metrics for Mixed-Bag puzzles

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying Quality33Direct Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Quantifying Solver QualityStandard and Enhanced Direct Accuracy

I Standard Direct Accuracy: Fraction of pieces, c placedin the same location in both the ground-truth and solvedimage versus the total number of pieces, n

DA =cn

(5)

I Enhanced Direct Accuracy Score (EDAS): Modifieddirect accuracy that accounts for missing and extra pieces.

EDASPi = arg maxSj∈S

ci,j

ni +∑

k 6=i (mk,j )(6)

I Direct Accuracy Range: 0 to 1

I Perfectly Reconstructed Image: All pieces are placed intheir original location (DA = EDAS = 1)

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying Quality33Direct Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Quantifying Solver QualityStandard and Enhanced Direct Accuracy

I Standard Direct Accuracy: Fraction of pieces, c placedin the same location in both the ground-truth and solvedimage versus the total number of pieces, n

DA =cn

(5)

I Enhanced Direct Accuracy Score (EDAS): Modifieddirect accuracy that accounts for missing and extra pieces.

EDASPi = arg maxSj∈S

ci,j

ni +∑

k 6=i (mk,j )(6)

I Direct Accuracy Range: 0 to 1

I Perfectly Reconstructed Image: All pieces are placed intheir original location (DA = EDAS = 1)

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying Quality33Direct Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Quantifying Solver QualityStandard and Enhanced Direct Accuracy

I Standard Direct Accuracy: Fraction of pieces, c placedin the same location in both the ground-truth and solvedimage versus the total number of pieces, n

DA =cn

(5)

I Enhanced Direct Accuracy Score (EDAS): Modifieddirect accuracy that accounts for missing and extra pieces.

EDASPi = arg maxSj∈S

ci,j

ni +∑

k 6=i (mk,j )(6)

I Direct Accuracy Range: 0 to 1

I Perfectly Reconstructed Image: All pieces are placed intheir original location (DA = EDAS = 1)

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying Quality34Direct Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Direct AccuracyExample – Effect of Shifts

Problem: Direct accuracy is highly vulnerable to shifts, inparticular when puzzle dimensions are not fixed

Ground-Truth Image Solver Output

Conclusion: Direct accuracy can be overly punitive.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying Quality34Direct Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Direct AccuracyExample – Effect of Shifts

Problem: Direct accuracy is highly vulnerable to shifts, inparticular when puzzle dimensions are not fixed

Ground-Truth Image

Solver Output

Conclusion: Direct accuracy can be overly punitive.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying Quality34Direct Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Direct AccuracyExample – Effect of Shifts

Problem: Direct accuracy is highly vulnerable to shifts, inparticular when puzzle dimensions are not fixed

Ground-Truth Image Solver Output

Conclusion: Direct accuracy can be overly punitive.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying Quality34Direct Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Direct AccuracyExample – Effect of Shifts

Problem: Direct accuracy is highly vulnerable to shifts, inparticular when puzzle dimensions are not fixed

Ground-Truth Image Solver Output

Conclusion: Direct accuracy can be overly punitive.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying Quality35Direct Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Direct AccuracyShiftable Enhanced Direct Accuracy Score (SEDAS)

I Solution: Allow the reference point for direct accuracy toshift beyond the upper left corner of the image

I Shiftable Enhanced Direct Accuracy Score (SEDAS):Select the reference point, l , within radius dmin of the upperleft corner of the solved puzzle

I dmin – Manhattan distance between the upper left corner ofthe solved image and the nearest puzzle piece

I Formal Definition of SEDAS:

SEDASPi = arg maxl∈L

(arg max

Sj∈S

ci,j,l

ni +∑

k 6=i (mk,j )

)(7)

I SEDAS Range: 0 to 1

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying Quality36Direct Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Direct AccuracyExample – Shiftable Reference Point

Solver Output

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying Quality36Direct Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Direct AccuracyExample – Shiftable Reference Point

Solver Output Direct Accuracy Reference Point

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying Quality36Direct Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Direct AccuracyExample – Shiftable Reference Point

Solver Output SEDAS Reference Points

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

37Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Quantifying Solver QualityStandard and Enhanced Neighbor Accuracy

I Standard Neighbor Accuracy: Ratio of puzzle piecesides adjacent in both the original and solved images, a,versus the total number of sides, n · q

NA =a

n · q(8)

I Enhanced Neighbor Accuracy Score (ENAS): Modifiedneighbor accuracy that accounts for missing and extrapieces.

ENASPi = arg maxSj∈S

ai,j

q(ni +∑

k 6=i (mk,j )(9)

I Neighbor Accuracy Range: 0 to 1

I Advantage of Neighbor Accuracy: Less vulnerable toshifts than direct accuracy

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

37Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Quantifying Solver QualityStandard and Enhanced Neighbor Accuracy

I Standard Neighbor Accuracy: Ratio of puzzle piecesides adjacent in both the original and solved images, a,versus the total number of sides, n · q

NA =a

n · q(8)

I Enhanced Neighbor Accuracy Score (ENAS): Modifiedneighbor accuracy that accounts for missing and extrapieces.

ENASPi = arg maxSj∈S

ai,j

q(ni +∑

k 6=i (mk,j )(9)

I Neighbor Accuracy Range: 0 to 1

I Advantage of Neighbor Accuracy: Less vulnerable toshifts than direct accuracy

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

37Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Quantifying Solver QualityStandard and Enhanced Neighbor Accuracy

I Standard Neighbor Accuracy: Ratio of puzzle piecesides adjacent in both the original and solved images, a,versus the total number of sides, n · q

NA =a

n · q(8)

I Enhanced Neighbor Accuracy Score (ENAS): Modifiedneighbor accuracy that accounts for missing and extrapieces.

ENASPi = arg maxSj∈S

ai,j

q(ni +∑

k 6=i (mk,j )(9)

I Neighbor Accuracy Range: 0 to 1

I Advantage of Neighbor Accuracy: Less vulnerable toshifts than direct accuracy

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

37Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Quantifying Solver QualityStandard and Enhanced Neighbor Accuracy

I Standard Neighbor Accuracy: Ratio of puzzle piecesides adjacent in both the original and solved images, a,versus the total number of sides, n · q

NA =a

n · q(8)

I Enhanced Neighbor Accuracy Score (ENAS): Modifiedneighbor accuracy that accounts for missing and extrapieces.

ENASPi = arg maxSj∈S

ai,j

q(ni +∑

k 6=i (mk,j )(9)

I Neighbor Accuracy Range: 0 to 1

I Advantage of Neighbor Accuracy: Less vulnerable toshifts than direct accuracy

Experimental Results

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

38Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Experimental Results

I Paikin & Tal’s algorithm is the current state of the art andwas used as the reference for performance comparison

I Standard Test Conditions:I Puzzle Type: 2I Dimensions Fixed: NoI Piece Width: 28 pixelsI Benchmark: Twenty 805 piece images [6]

I Number of Ground-Truth Inputs: 1 to 5

# Puzzles 1 2 3 4 5

# Iterations 20 55 25 8 5

I Test Condition Variation: Only Paikin & Tal’s algorithmwas provided the number of input puzzles.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

38Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Experimental Results

I Paikin & Tal’s algorithm is the current state of the art andwas used as the reference for performance comparison

I Standard Test Conditions:I Puzzle Type: 2I Dimensions Fixed: NoI Piece Width: 28 pixelsI Benchmark: Twenty 805 piece images [6]

I Number of Ground-Truth Inputs: 1 to 5

# Puzzles 1 2 3 4 5

# Iterations 20 55 25 8 5

I Test Condition Variation: Only Paikin & Tal’s algorithmwas provided the number of input puzzles.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

38Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Experimental Results

I Paikin & Tal’s algorithm is the current state of the art andwas used as the reference for performance comparison

I Standard Test Conditions:I Puzzle Type: 2I Dimensions Fixed: NoI Piece Width: 28 pixelsI Benchmark: Twenty 805 piece images [6]

I Number of Ground-Truth Inputs: 1 to 5

# Puzzles 1 2 3 4 5

# Iterations 20 55 25 8 5

I Test Condition Variation: Only Paikin & Tal’s algorithmwas provided the number of input puzzles.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

38Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Experimental Results

I Paikin & Tal’s algorithm is the current state of the art andwas used as the reference for performance comparison

I Standard Test Conditions:I Puzzle Type: 2I Dimensions Fixed: NoI Piece Width: 28 pixelsI Benchmark: Twenty 805 piece images [6]

I Number of Ground-Truth Inputs: 1 to 5

# Puzzles 1 2 3 4 5

# Iterations 20 55 25 8 5

I Test Condition Variation: Only Paikin & Tal’s algorithmwas provided the number of input puzzles.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

38Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Experimental Results

I Paikin & Tal’s algorithm is the current state of the art andwas used as the reference for performance comparison

I Standard Test Conditions:I Puzzle Type: 2I Dimensions Fixed: NoI Piece Width: 28 pixelsI Benchmark: Twenty 805 piece images [6]

I Number of Ground-Truth Inputs: 1 to 5

# Puzzles 1 2 3 4 5

# Iterations 20 55 25 8 5

I Test Condition Variation: Only Paikin & Tal’s algorithmwas provided the number of input puzzles.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

39Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Experimental ResultsDetermining Input Puzzle Count

I Goal: Measure the Mixed-Bag Solver’s accuracydetermining the number of input puzzles

I Importance – The Mixed-Bag Solver must estimate thisaccurately to provide meaningful outputs.

I Single Puzzle Accuracy – Represents the solver’sperformance ceiling

I Multiple Puzzle Accuracy – A more general estimate ofthe solver’s performance

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

39Experimental ResultsInput Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Experimental ResultsDetermining Input Puzzle Count

I Goal: Measure the Mixed-Bag Solver’s accuracydetermining the number of input puzzles

I Importance – The Mixed-Bag Solver must estimate thisaccurately to provide meaningful outputs.

I Single Puzzle Accuracy – Represents the solver’sperformance ceiling

I Multiple Puzzle Accuracy – A more general estimate ofthe solver’s performance

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental Results40Input Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Determining Input Puzzle CountSingle Input Puzzle Results

I Summary: 17 out of the 20 images were correctlyidentified as a single ground-truth input

I Misclassified Images: 3 out of the 20 imagesmisclassified as if they were two images.

I All three images have large areas with little variation (e.g., ablue sky, smooth water)

I The solver’s poor performance on these puzzles is due tothe assembler as noted in [4]

I Note: 85% (17/20) represents the accuracy ceiling whensolving multiple puzzles.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental Results40Input Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Determining Input Puzzle CountSingle Input Puzzle Results

I Summary: 17 out of the 20 images were correctlyidentified as a single ground-truth input

I Misclassified Images: 3 out of the 20 imagesmisclassified as if they were two images.

I All three images have large areas with little variation (e.g., ablue sky, smooth water)

I The solver’s poor performance on these puzzles is due tothe assembler as noted in [4]

I Note: 85% (17/20) represents the accuracy ceiling whensolving multiple puzzles.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental Results41Input Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Determining Input Puzzle CountVisual Comparison of a Misclassified Image

Perfectly ReconstructedImage (a) Misclassified Image (b)

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental Results42Input Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Determining Input Puzzle CountMultiple Input Puzzles

I Goal: Measure the Mixed-Bag Solver’s accuracydetermining the input puzzle count for multiple images

I Procedure: Randomly select a specified number ofimages (between 2 and 5) from the 20 image data set.

I Input Puzzle Count Error: Difference between the actualnumber of input puzzles and the number determined bythe Mixed-Bag Solver.

I Example: If 3 images were supplied to the solver but itdetermined there were 4, the error would be 1.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental Results42Input Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Determining Input Puzzle CountMultiple Input Puzzles

I Goal: Measure the Mixed-Bag Solver’s accuracydetermining the input puzzle count for multiple images

I Procedure: Randomly select a specified number ofimages (between 2 and 5) from the 20 image data set.

I Input Puzzle Count Error: Difference between the actualnumber of input puzzles and the number determined bythe Mixed-Bag Solver.

I Example: If 3 images were supplied to the solver but itdetermined there were 4, the error would be 1.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental Results43Input Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Determining Input Puzzle CountMultiple Input Puzzles – Results

0 1 2 30

20

40

60

80

100

75

167

2

4448

4 4

50 50

0 0

60

20 20

0

Size of Input Puzzle Count Error

Freq

uenc

y(%

)

Mixed-Bag Solver’s Input Puzzle Count Error Frequency

2 Puzzles 3 Puzzles 4 Puzzles 5 Puzzles

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental Results44Input Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Determining Input Puzzle CountMultiple Input Puzzles – Results Summary

I Overall Accuracy: 65%

I Iterations with Error Greater than One: 8%

I Accuracy did not significantly degrade as the number ofinput puzzles increased.

I Over-Rejection of Cluster Mergers: The Mixed-BagSolver never underestimated the number of input puzzles.

I Performance may be improved by reducing the minimumclustering similarity threshold or minimum segment size

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental Results44Input Puzzle Count

Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Determining Input Puzzle CountMultiple Input Puzzles – Results Summary

I Overall Accuracy: 65%

I Iterations with Error Greater than One: 8%

I Accuracy did not significantly degrade as the number ofinput puzzles increased.

I Over-Rejection of Cluster Mergers: The Mixed-BagSolver never underestimated the number of input puzzles.

I Performance may be improved by reducing the minimumclustering similarity threshold or minimum segment size

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

45Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Experimental ResultsPerformance on Multiple Input Puzzles

I Goal: Compare the performance of the Mixed-Bag Solver(MBS) and Paikin & Tal’s algorithm

I Procedure: Randomly select a specified number ofimages and input them into both solvers.

I Quality Metrics Used:I Shiftable Enhanced Direct Accuracy Score (SEDAS)I Enhanced Neighbor Accuracy Score (ENAS)I Perfect Reconstruction Percentage

I Note: The results include the Mixed-Bag Solver’sperformance when it correctly estimated the puzzle count.

I This represents the performance ceiling for optimalhierarchical clustering.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

45Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Experimental ResultsPerformance on Multiple Input Puzzles

I Goal: Compare the performance of the Mixed-Bag Solver(MBS) and Paikin & Tal’s algorithm

I Procedure: Randomly select a specified number ofimages and input them into both solvers.

I Quality Metrics Used:I Shiftable Enhanced Direct Accuracy Score (SEDAS)I Enhanced Neighbor Accuracy Score (ENAS)I Perfect Reconstruction Percentage

I Note: The results include the Mixed-Bag Solver’sperformance when it correctly estimated the puzzle count.

I This represents the performance ceiling for optimalhierarchical clustering.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

46Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Performance on Multiple Input PuzzlesShiftable Enhanced Direct Accuracy Score (SEDAS)

2 3 4 50

0.2

0.4

0.6

0.8

1

Number of Input Puzzles

SE

DA

S

Effect of the Number of Input Puzzles on SEDAS

MBS Correct Puzzle Count MBS All Paikin & Tal

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

47Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Performance on Multiple Input PuzzlesEnhanced Neighbor Accuracy Score (ENAS)

2 3 4 50

0.2

0.4

0.6

0.8

1

Number of Input Puzzles

EN

AS

Effect of the Number of Input Puzzles on ENAS

MBS Correct Puzzle Count MBS All Paikin & Tal

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

48Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Performance on Multiple Input PuzzlesPerfect Reconstruction Percentage

2 3 4 50

5

10

15

20

25

30

Number of Input Puzzles

Per

fect

Rec

onst

ruct

ion

(%)

Effect of the Number of Input Puzzles on Perfect Reconstruction

MBS Correct Puzzle Count MBS All Paikin & Tal

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

49Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Performance on Multiple Input PuzzlesResults Summary

I Summary: The Mixed-Bag Solver significantlyoutperformed Paikin & Tal’s algorithm across all metrics.

I This is notwithstanding that only their algorithm wassupplied with the number of input puzzles.

I Puzzle Input Count: Unlike Paikin & Tal’s algorithm, theMixed-Bag Solver saw no significant decrease inperformance with additional input puzzles

I Effect of Clustering Errors: Performance only decreasedslightly when incorrectly estimated input puzzle count.

I Many of the extra puzzles were relatively insignificant in size

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

49Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Performance on Multiple Input PuzzlesResults Summary

I Summary: The Mixed-Bag Solver significantlyoutperformed Paikin & Tal’s algorithm across all metrics.

I This is notwithstanding that only their algorithm wassupplied with the number of input puzzles.

I Puzzle Input Count: Unlike Paikin & Tal’s algorithm, theMixed-Bag Solver saw no significant decrease inperformance with additional input puzzles

I Effect of Clustering Errors: Performance only decreasedslightly when incorrectly estimated input puzzle count.

I Many of the extra puzzles were relatively insignificant in size

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

49Solver Comparison

Conclusions

Dept. of Computer ScienceSan José State University

Performance on Multiple Input PuzzlesResults Summary

I Summary: The Mixed-Bag Solver significantlyoutperformed Paikin & Tal’s algorithm across all metrics.

I This is notwithstanding that only their algorithm wassupplied with the number of input puzzles.

I Puzzle Input Count: Unlike Paikin & Tal’s algorithm, theMixed-Bag Solver saw no significant decrease inperformance with additional input puzzles

I Effect of Clustering Errors: Performance only decreasedslightly when incorrectly estimated input puzzle count.

I Many of the extra puzzles were relatively insignificant in size

Conclusions

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

50Conclusions

Dept. of Computer ScienceSan José State University

Conclusions

I This thesis presented a fully-automated solver forMixed-Bag puzzles.

I Mixed-Bag Solver significantly outperforms the currentstate of the art while receiving no externally suppliedinformation.

I Introduced the first set of solver quality metrics forMixed-Bag puzzles.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

50Conclusions

Dept. of Computer ScienceSan José State University

Conclusions

I This thesis presented a fully-automated solver forMixed-Bag puzzles.

I Mixed-Bag Solver significantly outperforms the currentstate of the art while receiving no externally suppliedinformation.

I Introduced the first set of solver quality metrics forMixed-Bag puzzles.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Thesis Goals

IntroductionPrevious Work

Mixed-Bag SolverAssembler

Segmentation

Stitching

Hierarchical Clustering

Final Seed Piece Selection

Final Assembly

Quantifying QualityDirect Accuracy

Neighbor Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

51Conclusions

Dept. of Computer ScienceSan José State University

Summary of Topics

Thesis GoalsIntroduction

Previous WorkMixed-Bag Solver

AssemblerSegmentationStitchingHierarchical ClusteringFinal Seed Piece SelectionFinal Assembly

Quantifying QualityDirect AccuracyNeighbor Accuracy

Experimental ResultsInput Puzzle CountSolver Comparison

Conclusions

Appendix

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Conclusions53Future Work

IntroductionPuzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

ConclusionsFuture Work

I Improved AssemblerI Prioritize placement using multiple best buddies

I Address placement performance in regions with low bestbuddy density

I Dynamic determination of the segment clusteringthreshold

I Expanded stitching piece selection

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Conclusions53Future Work

IntroductionPuzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

ConclusionsFuture Work

I Improved AssemblerI Prioritize placement using multiple best buddies

I Address placement performance in regions with low bestbuddy density

I Dynamic determination of the segment clusteringthreshold

I Expanded stitching piece selection

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Conclusions53Future Work

IntroductionPuzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

ConclusionsFuture Work

I Improved AssemblerI Prioritize placement using multiple best buddies

I Address placement performance in regions with low bestbuddy density

I Dynamic determination of the segment clusteringthreshold

I Expanded stitching piece selection

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

Conclusions54Future Work

IntroductionPuzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

List of References I

[1] T. Altman, “Solving the jigsaw puzzle problem in linear time,” Applied Artificial Intelligence, vol. 3, pp. 453–462, Jan. 1990.

[2] E. D. Demaine and M. L. Demaine, “Jigsaw puzzles, edge matching, and polyomino packing: Connections andcomplexity,” Graphs and Combinatorics, vol. 23 (Supplement), pp. 195–208, June 2007.

[3] A. C. Gallagher, “Jigsaw puzzles with pieces of unknown orientation,” in Proceedings of the 2012 IEEE Conference onComputer Vision and Pattern Recognition (CVPR), CVPR ’12, pp. 382–389, IEEE Computer Society, 2012.

[4] G. Paikin and A. Tal, “Solving multiple square jigsaw puzzles with missing pieces,” in Proceedings of the 2015 IEEEConference on Computer Vision and Pattern Recognition (CVPR), CVPR ’15, IEEE Computer Society, 2015.

[5] Z. Hammoudeh, “Github - puzzle solver thesis repository.” https://github.com/ZaydH/Thesis.(Accessed on 10/23/2016).

[6] D. Pomeranz, M. Shemesh, and O. Ben-Shahar, “Computational jigsaw puzzle solving.”https://www.cs.bgu.ac.il/~icvl/icvl_projects/automatic-jigsaw-puzzle-solving/, 2011.(Accessed on 05/01/2016).

[7] A. Olmos and F. A. A. Kingdom, “McGill calibrated colour image database.” http://tabby.vision.mcgill.ca/, 2005.(Accessed on 05/01/2016).

[8] P.-N. Tan, M. Steinbach, and V. Kumar, Introduction to Data Mining, (First Edition).Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc., 2005.

[9] T. S. Cho, S. Avidan, and W. T. Freeman, “A probabilistic image jigsaw puzzle solver,” in Proceedings of the 2010 IEEEConference on Computer Vision and Pattern Recognition (CVPR), CVPR ’10, pp. 183–190, IEEE Computer Society,2010.

[10] D. Pomeranz, M. Shemesh, and O. Ben-Shahar, “A fully automated greedy square jigsaw puzzle solver,” in Proceedingsof the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR ’11, pp. 9–16, IEEEComputer Society, 2011.

[11] H. Braxmeier and S. Steinberger, “Pixabay.” https://pixabay.com/.(Accessed on 05/15/2016).

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

Introduction55Puzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

IntroductionJig Swap Puzzle Types

There are four primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.

I Type 1: Puzzle dimensions and piece rotation are known.May have “anchor” piece(s) fixed in the correct location(s).

I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.

I Type 3: All piece locations are known. Only rotation isunknown.

I Mixed-Bag: Pieces come from multiple puzzles

Mixed-Bag puzzles are the focus of this thesis.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

Introduction55Puzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

IntroductionJig Swap Puzzle Types

There are four primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.

I Type 1: Puzzle dimensions and piece rotation are known.May have “anchor” piece(s) fixed in the correct location(s).

I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.

I Type 3: All piece locations are known. Only rotation isunknown.

I Mixed-Bag: Pieces come from multiple puzzles

Mixed-Bag puzzles are the focus of this thesis.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

Introduction55Puzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

IntroductionJig Swap Puzzle Types

There are four primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.

I Type 1: Puzzle dimensions and piece rotation are known.May have “anchor” piece(s) fixed in the correct location(s).

I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.

I Type 3: All piece locations are known. Only rotation isunknown.

I Mixed-Bag: Pieces come from multiple puzzles

Mixed-Bag puzzles are the focus of this thesis.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

Introduction55Puzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

IntroductionJig Swap Puzzle Types

There are four primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.

I Type 1: Puzzle dimensions and piece rotation are known.May have “anchor” piece(s) fixed in the correct location(s).

I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.

I Type 3: All piece locations are known. Only rotation isunknown.

I Mixed-Bag: Pieces come from multiple puzzles

Mixed-Bag puzzles are the focus of this thesis.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

Introduction55Puzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

IntroductionJig Swap Puzzle Types

There are four primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.

I Type 1: Puzzle dimensions and piece rotation are known.May have “anchor” piece(s) fixed in the correct location(s).

I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.

I Type 3: All piece locations are known. Only rotation isunknown.

I Mixed-Bag: Pieces come from multiple puzzles

Mixed-Bag puzzles are the focus of this thesis.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

Introduction55Puzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

IntroductionJig Swap Puzzle Types

There are four primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.

I Type 1: Puzzle dimensions and piece rotation are known.May have “anchor” piece(s) fixed in the correct location(s).

I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.

I Type 3: All piece locations are known. Only rotation isunknown.

I Mixed-Bag: Pieces come from multiple puzzles

Mixed-Bag puzzles are the focus of this thesis.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

IntroductionPuzzle Types

56Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

Previous Work

I Cho et al. [9] – Introduced the first modern jig swap puzzlesolver

I Graphical model-based Type 1 solver

I Puzzle dimensions are known

I Used one or more anchor pieces

I Defined quality metrics for Type 1 and Type 2 puzzles

I Established the standard comparative test conditions

I Pomeranz et al. [10] – Iterative, greedy Type 1 puzzlesolver

I Eliminated the use of anchor pieces

I Created multiple solver benchmarks of various sizes

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

IntroductionPuzzle Types

56Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

Previous Work

I Cho et al. [9] – Introduced the first modern jig swap puzzlesolver

I Graphical model-based Type 1 solver

I Puzzle dimensions are known

I Used one or more anchor pieces

I Defined quality metrics for Type 1 and Type 2 puzzles

I Established the standard comparative test conditions

I Pomeranz et al. [10] – Iterative, greedy Type 1 puzzlesolver

I Eliminated the use of anchor pieces

I Created multiple solver benchmarks of various sizes

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

IntroductionPuzzle Types

Previous Work

57Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

IntroductionBest Buddies

I Basis of all Modern Jig Swap Solvers: The morecompatible two pieces are, the more likely they are to beadjacent.

I Best Buddies: A pair of puzzles pieces that are morecompatible with each other on their respective sides thanthey are to any other piece [10]

I Note: Not all puzzle pieces will have a best buddy.

∀pk∀sz ,C(pi , sx ,pj , sy ) ≥ C(pi , sx ,pk , sz)

and

∀pk∀sz ,C(pj , sy ,pi , sx ) ≥ C(pj , sy ,pk , sz)

(10)

I Importance of Best Buddies: Key adjacency indicator

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

IntroductionPuzzle Types

Previous Work

Best Buddies58Best Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

Quantifying Solver QualityBest Buddy Density

I Best Buddy Density (BBD): A metric for quantifying thebest buddy profile of an image that is independent ofimage size.

BBD =b

n · q(11)

I A greater BBD means the pieces are more differentiatedmaking the puzzle easier to solve.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

IntroductionPuzzle Types

Previous Work

Best Buddies59Best Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

Best Buddy DensityVisualization

Visualizing Best Buddy DensityI Transform each puzzle piece into a square consisting of

four isosceles triangles.

I Color each triangle according to whether the adjacentpiece is a best buddy. The scheme used in this thesis:

No BestBuddy

Non-AdjacentBest Buddy

AdjacentBest Buddy

No PiecePresent

I Areas with higher best buddy density will have more greentriangles.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

IntroductionPuzzle Types

Previous Work

Best Buddies60Best Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

Best Buddy DensityVisualization Example

Original Image [11]

Best Buddy Visualization

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

IntroductionPuzzle Types

Previous Work

Best Buddies60Best Buddy Density

Experimental ResultsSingle Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

Best Buddy DensityVisualization Example

Original Image [11] Best Buddy Visualization

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

IntroductionPuzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental Results61Single Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

Determining Input Puzzle CountComparison of Best Buddy Density for Misclassified Images

Perfectly ReconstructedImage (a)

Best Buddy Visualization (a)

Misclassified Image (b)

Best Buddy Visualization (b)

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

IntroductionPuzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental Results61Single Input Puzzle

Ten Puzzle Results

Dept. of Computer ScienceSan José State University

Determining Input Puzzle CountComparison of Best Buddy Density for Misclassified Images

Perfectly ReconstructedImage (a) Best Buddy Visualization (a)

Misclassified Image (b) Best Buddy Visualization (b)

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

IntroductionPuzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

62Ten Puzzle Results

Dept. of Computer ScienceSan José State University

Experimental ResultsSolving More than Five Puzzles

I As the number of puzzles increases, the difficulty ofsimultaneously reconstructing them also increases.

I Current State of the Art: Paikin & Tal [4] solved up to fivepuzzles simultaneously.

I Goal: Compare the performance of the Mixed-Bag Solverand Paikin & Tal’s algorithm on 10 puzzles.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

IntroductionPuzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

62Ten Puzzle Results

Dept. of Computer ScienceSan José State University

Experimental ResultsSolving More than Five Puzzles

I As the number of puzzles increases, the difficulty ofsimultaneously reconstructing them also increases.

I Current State of the Art: Paikin & Tal [4] solved up to fivepuzzles simultaneously.

I Goal: Compare the performance of the Mixed-Bag Solverand Paikin & Tal’s algorithm on 10 puzzles.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

IntroductionPuzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

62Ten Puzzle Results

Dept. of Computer ScienceSan José State University

Experimental ResultsSolving More than Five Puzzles

I As the number of puzzles increases, the difficulty ofsimultaneously reconstructing them also increases.

I Current State of the Art: Paikin & Tal [4] solved up to fivepuzzles simultaneously.

I Goal: Compare the performance of the Mixed-Bag Solverand Paikin & Tal’s algorithm on 10 puzzles.

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

IntroductionPuzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

63Ten Puzzle Results

Dept. of Computer ScienceSan José State University

Ten Puzzle ResultsSummary

I Paikin & TalI Seed of nine images came from just three input images

I SEDAS and EDAS greater than 0.9 for only one image

I No perfectly reconstructed images

I Mixed-Bag SolverI SEDAS and EDAS greater than 0.9 for all images

I Four images perfectly reconstructed

I Results comparable to Paikin & Tal’s algorithm solving eachpuzzle individually

51

A Fully-AutomatedSolver for Multiple

Square JigsawPuzzles Using

Hierarchical Clustering

ConclusionsFuture Work

IntroductionPuzzle Types

Previous Work

Best BuddiesBest Buddy Density

Experimental ResultsSingle Input Puzzle

63Ten Puzzle Results

Dept. of Computer ScienceSan José State University

Ten Puzzle ResultsSummary

I Paikin & TalI Seed of nine images came from just three input images

I SEDAS and EDAS greater than 0.9 for only one image

I No perfectly reconstructed images

I Mixed-Bag SolverI SEDAS and EDAS greater than 0.9 for all images

I Four images perfectly reconstructed

I Results comparable to Paikin & Tal’s algorithm solving eachpuzzle individually