Computer Vision Based Chess Playing
Capabilities for the Baxter Humanoid Robot
International Conference on Control, Robotics, and Automation 2016
Andrew Tzer-Yeu Chen, Kevin I-Kai Wang
{andrew.chen, kevin.wang}@auckland.ac.nz
Baxter Humanoid Robot
• Designed for industrial automation
• Compliant, safe to use around humans (no safety cage!)
• Possible human-robot interaction applications
Chess Robots
The Mechanical Turk, c.1783
KUKA Monster and CHESSka robots, 2012https://jeremynicholl.photoshelter.com/image/I0000ylUIaOP6QhE
Chess Robots
• There are three problems to solve:
Perception“What is the current game state?”
Computation“What is the best next move?”
Actuation“How do I play the move?”
Perception
• Most implementations simplify the problem significantly
http://www.eurochess.de/media/catalog/product/1/_/1.png_4.jpg
DGT boards use sensorsembedded in the boardand pieces to detectposition or occupancy
• Boards often have to be in fixed positions
• When CV is used, simplifications include:• Non-standard colours [2]
• Mounting the camera directly above the board [3][4]
Perception
• Baxter has an embedded camera in the arm
Perception
• We process a 1280x800 image in OpenCV
• Camera held at 0.4 radians from the perpendicularSTART
(Input Image)
Image Preparation
Board Detection
Square Detection
Piece Detection
Output Board
Configuration
Morphologically
Closed and
Gaussian Blurred
Dynamic Hough
Line Transform
Dilated
Canny Edge
Detection,
Merged
Piece
Detection
Final output with
detected squares
and pieces
Dynamic Hough Line Transform
• Iterative determination of appropriate vote threshold
• Segments image and uses different thresholds
• Reduces impact of perspective distortion
Vote thresholds of 100, 150, 200, 250, 300, our dynamic hough line transform, final output
Robust Occupancy Detection
• Method A: Canny edge detection + morphological closing
• Method B: Morphological edge detection
• Hybrid approach more resistant to lighting changes/grads
a) greyscale, b) after contrasting and morphological closing, c) occupancy for method A, d) occupancy for method B, e) final occupancy. False positives from method A and B alone highlighted in red
Perception
• Game state perception reasonably resistant to position, orientation, and lighting changes
• Differential image approach• What changed in the game state between frames?
• Square numbers of moved piece(s) identified
• Converted to UCI algebraic format (e.g. g1f3)
• Move is handed over to the computation subsystem
Computation
• Last moved piece fed in from perception subsystem
• Chessnut1 for move validation
• Open-source chess engine Stockfish2 is used
• PyStockfish3 used to wrap it into Python
• User selects difficulty level corresponding to search depth
• Baxter’s move is sent to mechatronics subsystem
1 https://github.com/cgearhart/Chessnut 2 https://stockfishchess.org/ 3 https://github.com/iamjarret/pystockfish
Actuation
• Baxter’s arm has 7 degrees of freedom
• We use IKFast! to do inverse kinematics calculations
• Co-ordinates dynamically calculated based on CV
Overall System Control
• Finite State Machine Model
• Baxter’s screen (face) used to improve HCI• Baxter concentrates when it is his move
• Baxter goes to sleep when the human takes too long
• Baxter celebrates when he wins (he always wins)
Future Work
• Maybe replace mechatronics subsystem with accurate arm that is also much faster
• Fault tolerance to automatically correct situations where pieces are knocked over
• Use of artificial intelligence (AI) to identify piece types
• Further development of facial expressions (HCI)
• Investigating other small scale manipulation tasks• Other board or card games
• Solving pick-and-place style puzzles
• Dispensing medication
Conclusions
• Chess playing robot
• Using CV techniques without unnecessary simplifcations• No DGT board
• Without non-standard board colours
• Without a fixed camera position directly above the board
• With variable lighting conditions
• With small variations in board orientation
• With changes in board position
• High-level open-source tools used to integrate OpenCV, Stockfish, IKFast for perceive-compute-actuate cycle
• Useful case study for HCI research with Baxter
• Improvement upon existing CV chess analysisVideo: https://www.youtube.com/watch?v=6CfNTU7wVT0
Perception
• Tested with varying board orientations and lighting conditions, including extreme (unlikely) scenarios
Test Set BOARD Square Method A
PieceMethod B
PieceHybrid Piece
Full Set (N=270) 99.6 92.2 73.5 60.1 79.9With Correct Square Detection (N=241)
100 100 80.9 66.8 88.8
Normal lighting conditions (N=38)
100 100 92.1 68.4 100
Computer Vision Algorithm Detection Accuracy (%)
Perception
START
(Input Image)
Convert to Greyscale
Apply Gaussian Blurring
Determine Otsu Threshold
Output Board
Configuration
Board Detection
Square Detection Piece Detection
Image Preparation
Canny Edge Detection, Dilate
Find Square Contours
Slice Image, Calculate Angles Apply CLAHE Contrasting
Apply Gaussian Blurring
Morphological Edge Detection
Canny Edge Detection, Dilate
Find Square Contours, Sort
Dynamic Hough Line Transform
Create Masks for each Square
Morphological Closing
Merge Images and Remove Noise
Calculate Intensity Average
Calculate Centroid Intensity
Determine Piece Presence