Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

APPLE RECOGNITION BASED ON MACHINE VISION

GUO-QUAN JIANG\ CUI-JUN ZHA02

ISchool of Computer Science and Technology, Henan Polytechnic University, Jiaozuo 454000, China 2School of Resources and Environment Engineering, Henan Polytechnic University, Jiaozuo 454000, China

E-MAI L: jiangguoquan@hpu.edu.cn.zhaocuijun@hpu.edu.cn

Abstract: For an apple harvesting robot, the machine vision system

is a very important part which is used to recognize and locate apples. In this paper, an automatic apple recognition method based on image processing technology is developed for apple

harvesting robot. Firstly, 2R-G-B grey value transform is

performed, and the adaptive threshold Otsu algorithm is used for image segmentation. Then, a series of morphological operations are adopted to eliminate image noise, and edge detection is utilized to obtain the apple edges. Finally, the optimized Hough transform is presented to recognize apples. Experimental results indicate that the proposed method can segment the adjacent and overlapped apples effectively, where the recognition rate can reach 91 %. Meanwhile, the centers and radii of the apples could also be accurately extracted.

Keywords: Apple; Machine vision; Image recognition

1. Introduction

The research of critical technologies on fruit picking robot has demonstrated kinds of practical values. It not only meets some market demands, reduces labor intensity, increases economic benefit, but also tracks new technologies in agricultural world, and promotes the agricultural modernization.

In the vision system of apple harvesting robot, the critical task is to recognize and locate each single apple. Whether it can identify fruit quickly and accurately will directly affect its efficiency and reliability [1, 2]. In [3], some different vision systems to recognize fruits for automated harvesting were summarized. In [4], the Red-Blue (R-B) chromatic aberration information of the images has been used to recognize oranges on the tree. Besides, the fruits with front lighting and back lighting have also been considered. In [5], a new modified Hue-Saturation-Value (HSV) color space was used to segment images and extract the relevant information of the fruits in the segmentation phase before the localization process. In Ref. [6], an apple recognition method based on

978-1-4673-1487-9/12/$31.00 ©2012 IEEE

the color difference ratio and a genetic algorithm was presented. It eliminated the influence of shades, front lighting and backlighting.

Based on the methods mentioned above, this paper will try to describe the development of a robust machine vision recognition method, which guides a harvesting robotic to pick apples in different conditions. In other words, the goal of the presented work is to develop a new method for improving apple recognition accuracy and efficiency.

2. Methods to Identify the Fruits

2.1. Color Feature Analysis

We first select the RGB color system as the basic method for color description. Experimental results show that, the color index 2R -G- B can achieve high contrast values for red apples.

The principle is shown as formula (1):

pixe/(x,y) = {�R -G - B

255

2R�G+B

others (1)

2R?G+B+255

Where, pixe/(x,y) denotes the grey value of

a po int(x, y) in the grey image.

2.2. Image Segmentation

After the color indices have been applied, the image segmentation is used. Among the global thresholding techniques, the Sahoo (1988) study concluded that the Otsu method (Otsu, 1979) was one of the good threshold selection methods for general real world images with respect to uniformity and shape measures. The basic principle of Otsu is looking for an optimal threshold value to divide the gray-level histogram of an image into two parts, which is based on the condition that the between-cluster variance is maximal. Here, the optimal Otsu threshold segmentation [7] is applied to segment the

1148

Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

fruits on the image.

2.3. Image denoising

2.3. 1. Combing dilation and erosion

In order to remove the image random noise after image segmentation, the image morphological filtering is applied. The mathematical morphology theory is developed from geometry. It was introduced by Matheron as a technique for analyzing geometric structures of metallic and geologic samples. Later, it was extended to image analysis by Serra [8]. Image morphological operations include the dilation, erosion, opening and closing operations, and the operations of dilation and erosion are fundamental in morphological image processing. Opening and closing operations are combinations of the dilation and erosion. Opening operations can eliminate a small structure whose size is less than the structure element, which denotes the noise. Closing operations can smooth the edges of an object and close the inner holes.

Let F be a gray scale image, and B be a structural element, where, B can be a circle, a triangle, a square or other simple geometric primitives. Next, set the dilation operator as EB , the erosion operator as ® , the morphological operations of a binarization image can be expressed as:

m

(F ®B)(x,y) = AND[F(x+ i,y+ j) &B(i,j)] (2) i,j�O

m

(FEBB)(x,y) = OR[F(x+i,y+ j)&B(i,j)] (3) i,j�O

Where, (x, y) denotes points that will be processed, and m is the size of the structure element. For the apple image processing, we adopt the opening and closing operations. First n times' erosion is operated, followed by the n times' dilation, and at last the object image G is gained according to the following expression:

G = ( F ®n B)EBn B (4)

Where, n = 3, and the structure element is a 3 X 3 square.

2.3.2. Image holes filling

Usually, the existence of rich textures in apple images causes holes in these object images. In order to remove them, a hole-filled method is applied using the Matlab toolbox, which selects 4-connected background neighbors to fill holes in the binary images.

2.4. Edge detection

Edge detection is an important pre-processing step in image segmentation. An edge is the boundary where distinct intensities change or discontinuities occur. Edge detection is a process that transforms a grey-level image to an edge image, which indicates either the presence or absence of edges[9]. Derivative edge detectors are straightforward methods for edge detection. The differential operators such as Robert and Sobel operators are convolved with images to enhance spatial intensity changes, and a threshold is applied to obtain edge points. Here, the Sobel operator is applied on edge detection.

2.5. Circle detection

Detecting lines and circles in an image is a fundamental issue in applications of image processing. Extracting circles from digital images has received more attention in many industrial applications for decades, since an extracted circle can yield the location of the corresponding circular object t. The Circle Hough transform (CHT) [10] for finding circular shapes is one of the best known algorithms.

In spite of its popularity owing to its simple operation, the CHT has some disadvantages when working on a discrete image. The large amount of storage and computing power required by the CHT are the major disadvantages in real-time applications.

For specific apple images, an optimized method based on CHT is presented. The main measures are given as follows:

(1) Accumulator Threshold Setting In Hough transform, the accumulator threshold affects

the detection results. If the threshold value is set to be smaller, the detection of circular objects is more comprehensive. But it is sensitive to noise. It leads to a result that multiple circles are detected while only one circle exists. At the same time, it will consume too much storage space. Conversely, if the threshold is set to be larger, it will cause the target detection incomplete. After large amounts of image processing experiments, we obtain the following formula:

(Llx,Lly,Llr) = (3n,3n,2n) (5)

Where, Llx,Lly,Llr denote the increments of x­direction circle center, y-direction circle center, and radius increments, respectively. n = L150, where L is the fruit boundary rectangular length (in pixels).

After optimization, the search scope can be controlled effectively, which improves the transform rate. In addition, the threshold selection and parameter setting can be

1149

Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

finished automatically by computers, without human intervention, which achieves the intelligent requirements.

The sizes of accumulator in Hough transform directly affect the speed and accuracy. In order to reduce the amount of calculation, we take the size of image target regions as the selective basis for accumulator unit radius, according to the external rectangular size to set intervals.

(2) Search Range Setting In Hough transform, all possible pixels are to be

'voted', which will consume a lot of time and space. To shorten the processing time, we set a search

radius'i, such as formula (6) shown below:

'i E[min{aphJ/4,max{aphJ*1/2], i=I,2,.··,m (6)

Where, ai' hi denote the length and width of a rectangular boundary (in pixel), respectively; m is the number of image regions. After optimization, the search scope has been effectively controlled, and the conversion speed is improved. Besides, interval selection and parameter setting can be finished automatically by

(a) Apple color image

(c) Binarization image

computers.

3. Results and discussion

There are in total of 86 images in different conditions that are used to test the algorithm in our experiment. The image resolution is 640 X 480 pixels. It can accurately identity 78 pictures. The recognition rate reaches 91%. Experimental results are shown in Figure 1 and Table 1. However, parts of the apple images can not be correctly identified, which may be resulted by:

(1) Insufficient maturity, the reflective and other factors that lead to the identification of more mature fruit deformities, and the algorithm cannot recover the original shapes.

(2) Serious shielding and overlapping (overlapping beyond overall 112 of a fruit) lead to a failure feature extraction.

(b) Grey level image

(d) Denoised image

1150

Proceedings of the 2012 International Conference on Machine Learning and Cybernetics, Xian, 15-17 July, 2012

(e) Edge detection image (f) Apple recognition result

Figure 1. Apple automatic recognition process based on machine vision

TABLE 1. RIPE APPLE RECOGNITION RESULTS Image recognition Integrated circumstance Partial occlusion Without shelter

Number oftotal images Number of correct identification

Recognition rate (percent)

4. Conclusions

86 78

91.0

An apple detection method based on machine vision is developed. The 2R - G - B color values can effectively transform a color image to a grey image even under different natural conditions. The optimized Hough transform can accurately extract the center coordinates and radii of the spherical fruits. The correct recognition rate reaches 91 %. However, when the fruit roundness is poor that produces a false target, there may be some shortcomings for the proposed algorithm. Future works will be performed on this area to minimize unrecognization of apples.

Acknowledgements

This research was supported by the Henan Polytechnic University Doctorial Fund (Grant No. B2010-27).

References

[1] Muscato G., Prestifilippo M., Abbate Nunzio., Rizzuto Ivan, "A prototype of an orange picking robot: past history, the new robot and experimental results", Industrial Robot, Vol. 32, pp. 128-38,2005.

[2] Wang Jinjing, Zhao Dean, Ji Wei, Zhang Chao, "Apple fruit recognition based on support vector machine using in harvesting robot", Transactions of the Chinese Society for Agricultural Machinery, Vol. 40, No. 1, pp. 148-151. 2009.

52 46

88.5

34 32

94.1

[3] Jimenez AR, Jain AK, Ceres R, Pons J L., "Automatic fruit recognition: a survey and new results using range-attenuation images", Pattern Recognition. Vol. 32,pp. 1719-1736,pp. 1999.

[4] Xu Huirong, Ye Zunzhong, Ying Yibin, "Identification of citrus fruit in a tree canopy using color information", Transactions of the Chinese Society of Agricultural Engineering, Vol. 21, No.5, pp. 98-10 1,2005.

[5] Plebe A, Grasso G., " Localization of Spherical Fruits for Robotic Harvesting", Machine Vision Appl. Vol. 13, pp. 70-79,2001.

[6] Si Yongsheng, Qiao Jun ,Liu Gang, "Recognition and shape features extraction of apples based on machine vision", Transactions of the Chinese Society for Agricultural Machinery, Vol. 40, No. 8, pp. 161-165, 2009.

[7] Jianjun, Y., Hanping M., Suyu Z., "Segmentation methods of fruit image based on color difference, Journal of Communication and Computer, Vol. 6, pp. 40-45,2009.

[8] Serra J., Image analysis and mathematical morphology, Academic Press, New York, 1982.

[9] Chanda B., Kundu M.K., Padmaja Y.V., "A multi-scale morphologic edge detector", Pattern Recognition, Vol. 31, No 10, pp. 1469-1478, 1998.

[10] Duda, R., Hart, P., "Use of the Hough transform to detect lines and curves in pictures", Communications of the ACM. Vol. 15,pp. 11-15, 1975.

1151