Computer Vision Based Chess Playing

Capabilities for the Baxter Humanoid Robot

International Conference on Control, Robotics, and Automation 2016

Andrew Tzer-Yeu Chen, Kevin I-Kai Wang

{andrew.chen, kevin.wang}@auckland.ac.nz

Baxter Humanoid Robot

• Designed for industrial automation

• Compliant, safe to use around humans (no safety cage!)

• Possible human-robot interaction applications

Chess Robots

The Mechanical Turk, c.1783

KUKA Monster and CHESSka robots, 2012https://jeremynicholl.photoshelter.com/image/I0000ylUIaOP6QhE

Chess Robots

• There are three problems to solve:

Perception“What is the current game state?”

Computation“What is the best next move?”

Actuation“How do I play the move?”

Perception

• Most implementations simplify the problem significantly

http://www.eurochess.de/media/catalog/product/1/_/1.png_4.jpg

DGT boards use sensorsembedded in the boardand pieces to detectposition or occupancy

• Boards often have to be in fixed positions

• When CV is used, simplifications include:• Non-standard colours [2]

• Mounting the camera directly above the board [3][4]

Perception

• Baxter has an embedded camera in the arm

Perception

• We process a 1280x800 image in OpenCV

• Camera held at 0.4 radians from the perpendicularSTART

(Input Image)

Image Preparation

Board Detection

Square Detection

Piece Detection

Output Board

Configuration

Morphologically

Closed and

Gaussian Blurred

Dynamic Hough

Line Transform

Dilated

Canny Edge

Detection,

Merged

Piece

Detection

Final output with

detected squares

and pieces

Dynamic Hough Line Transform

• Iterative determination of appropriate vote threshold

• Segments image and uses different thresholds

• Reduces impact of perspective distortion

Vote thresholds of 100, 150, 200, 250, 300, our dynamic hough line transform, final output

Robust Occupancy Detection

• Method A: Canny edge detection + morphological closing

• Method B: Morphological edge detection

• Hybrid approach more resistant to lighting changes/grads

a) greyscale, b) after contrasting and morphological closing, c) occupancy for method A, d) occupancy for method B, e) final occupancy. False positives from method A and B alone highlighted in red

Perception

• Game state perception reasonably resistant to position, orientation, and lighting changes

• Differential image approach• What changed in the game state between frames?

• Square numbers of moved piece(s) identified

• Converted to UCI algebraic format (e.g. g1f3)

• Move is handed over to the computation subsystem

Computation

• Last moved piece fed in from perception subsystem

• Chessnut1 for move validation

• Open-source chess engine Stockfish2 is used

• PyStockfish3 used to wrap it into Python

• User selects difficulty level corresponding to search depth

• Baxter’s move is sent to mechatronics subsystem

1 https://github.com/cgearhart/Chessnut 2 https://stockfishchess.org/ 3 https://github.com/iamjarret/pystockfish

Actuation

• Baxter’s arm has 7 degrees of freedom

• We use IKFast! to do inverse kinematics calculations

• Co-ordinates dynamically calculated based on CV

Overall System Control

• Finite State Machine Model

• Baxter’s screen (face) used to improve HCI• Baxter concentrates when it is his move

• Baxter goes to sleep when the human takes too long

• Baxter celebrates when he wins (he always wins)

Future Work

• Maybe replace mechatronics subsystem with accurate arm that is also much faster

• Fault tolerance to automatically correct situations where pieces are knocked over

• Use of artificial intelligence (AI) to identify piece types

• Further development of facial expressions (HCI)

• Investigating other small scale manipulation tasks• Other board or card games

• Solving pick-and-place style puzzles

• Dispensing medication

Conclusions

• Chess playing robot

• Using CV techniques without unnecessary simplifcations• No DGT board

• Without non-standard board colours

• Without a fixed camera position directly above the board

• With variable lighting conditions

• With small variations in board orientation

• With changes in board position

• High-level open-source tools used to integrate OpenCV, Stockfish, IKFast for perceive-compute-actuate cycle

• Useful case study for HCI research with Baxter

• Improvement upon existing CV chess analysisVideo: https://www.youtube.com/watch?v=6CfNTU7wVT0

Perception

• Tested with varying board orientations and lighting conditions, including extreme (unlikely) scenarios

Test Set BOARD Square Method A

PieceMethod B

PieceHybrid Piece

Full Set (N=270) 99.6 92.2 73.5 60.1 79.9With Correct Square Detection (N=241)

100 100 80.9 66.8 88.8

Normal lighting conditions (N=38)

100 100 92.1 68.4 100

Computer Vision Algorithm Detection Accuracy (%)

Perception

START

(Input Image)

Convert to Greyscale

Apply Gaussian Blurring

Determine Otsu Threshold

Output Board

Configuration

Board Detection

Square Detection Piece Detection

Image Preparation

Canny Edge Detection, Dilate

Find Square Contours

Slice Image, Calculate Angles Apply CLAHE Contrasting

Apply Gaussian Blurring

Morphological Edge Detection

Canny Edge Detection, Dilate

Find Square Contours, Sort

Dynamic Hough Line Transform

Create Masks for each Square

Morphological Closing

Merge Images and Remove Noise

Calculate Intensity Average

Calculate Centroid Intensity

Determine Piece Presence