Image Processing Based Detection & Size …...accurate segmentation and detection of fruit in images of tree canopies. Fruit measure in image processing can be used to estimate the

Image Processing Based Detection & Size Estimation of Fruit on Mango

Tree Canopies

Santi Kumari Behera

Assistant Professor, Department of Computer Science & Engineering,

VSSUT, Burla, Odisha, India

Shrabani Sangita

Research Scholar, Department of Computer Science & Engineering,

VSSUT, Burla, Odisha India.

Prabira Kumar Sethy

Assistant Professor, Department of Electronics,

Sambalpur University, Odisha India.

Amiya Kumar Rath

Professor, Department of Computer Science & Engineering,

VSSUT, Burla, Odisha India.

Abstract National fruit mango contributes a major part in national

growth. Due to the flavour, nutrition & taste mango is one of

the popular fruit. For increment in profitability of agricultural

industry the proper detection, grading and packaging of fruit

are very important. Image processing techniques hold the

promise of enabling rapid and accurate fruit crop yield

predictions in the field. The key to fulfilling this promise is

accurate segmentation and detection of fruit in images of tree

canopies. Fruit measure in image processing can be used to

estimate the size of fruit which used to inform management

decisions at an individual tree level. Here an image processing

based technique is developed to detect and measure the size of

each mango successfully, with the help of Randomizes Hough

Transform (RHT) the fruit us detected properly. The on-tree

fruit is detected with 98% accuracy and it measures each fruit

with 3.07% average error.

Keywords:Image Processing,Colour Thresholding, Detection,

Measurement, Randomized Hough Transform (RHT)

Introduction The king of fruit mango is a tropical fruit which consumed

worldwide. As mango is a seasonal fruit, it collected from the

tree on that season and transported to different location. But

there has been a decrease in production of good quality fruits,

due to improper cultivation, manual inspection, and lack of

maintenance, postharvest losses in handling and processing,

lack knowledge in quick quality evaluation techniques. Also,

rising labour costs, shortage of skilled laborers and the need to

enhance generation forms have all put pressure on producers

and processors for the demand of a rapid, economic,

consistent and nod-destructive inspection method. In such a

scenario, automation can reduce the costs by promoting

production efficiency. The on tree fruit detection, counting,

grading and packaging of mangoes, all are the important part

of good mango production. It is a tedious job and difficult for

the farmer to always maintain constant attentiveness.

On-tree estimation of fruit measure is valuable for the

expectation of estimation of harvest yield and can advise

pressing material (plate embed) buying and showcasing

courses of action. For physiological investigations, estimation

of the span of the individual fruit after some time permits

estimation of fruit development rate and its reaction to the

ailment and agronomic conditions. Blunder is happen at the

season of estimating the extent of each fruit on-tree, due to

natural product covering way.

In past many research work has been carried for detecting,

localizing & counting of on tree fruit [1-3] by implementing

KNN, SVM, ANN & texture analysis. It is easily detected

fruit but sometimes the problem is occurs due to overlapping,

shape or colour. An image processing technique is developed

International Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Number 4 (2018) Spl.© Research India Publications. http://www.ripublication.com

6

to estimate the volume and mass of the post-harvest fruit [4]

& weight is the estimate with the help of water displacement

method (WDM). For identifying specific curves like ellipse

the RHT (Round Hough Transform) is used [5-6]. In image

processing technique the colour analysis is very important, it

is helpful for segmentation [7-9]. Many types of colour model

are present like RGB, L*a*b, CMYK, HSV, YCbCr.

Detecting shape and estimating volume is one of the important

aspects of research [10-13], which help to grade fruit. The

shape of fruit help to analyze, whether it is circular, elliptical

or any other type of shape. After the shape analysis, the

volume measurement is an easy task. The favorable feature of

our paper is measuring the size of mango from open-field

images rather the post-harvest grading as previous research.

Materials & Methods The primal recognition technique, that human visual system

adapts to distinguish between green and magenta color mango

fruits and to identify fruits amidst background leaves, is the

color recognition technique. Therefore, a novel approach is

proposed using image processing for performing three

functions: 1) color space segmentation for background

removal 2) Detection of Mango 3) size measurement of

mango.

Figure 1: System Overview for Measurement of Mango Size.

L*a*b color space segmentation

The colour threshold app which helps to threshold colour

images by manipulating the colour components of the

images based on different colour spaces. This is chosen

to removing the background.

Figure 2: Colour thresholding for L*a*b* colour space.

Figure 3: Colour thresholding for L*a*b colour space.

To separate the mango from background, a pixel-based

division was led. The arranged RGB picture was then changed

over into CIE L*a*b* shading space to isolates the organic

product shape in the L* channel. This approach isolate the

picture into two classes of pixels, so that the intra-class

change is insignificant, making a double picture. Due to their

Measure

Size of

Fruit


7

convexity and smoothness, the reflectance of mango organic

product is superior to anything the encompassing leaves or

branches and were expelled utilizing shading limit in light of

a* and b* channels, in first case Fig.2 especially for the green

mango the value of L*a*b* should be 5<= L<=85,-

40<=a<=20,-40<=b<=35 and in second case Fig.3 specially

for other colour mango the value of L*a*b* should be

0<L<100,10<a<60,-30<=b<=70.Then the morphological

activity of territory opening was connected to evacuate little

protests, trailed by shutting to fill gaps in leafy foods pixels

were dealt with as zero.

Detection of Mango Fruit

After the color thresholding process the image is converted to

binary image, then we use the strel function for structuring the

objects. Then detect the edge and use Randomized Hough

Transform (RHT) for detect the elliptical part because mango

is elliptical by shape. So we can detect the mango fruit.

Elliptical Fitting for Reorganization of Complete Fruit

As mango natural product has an inexact circle form, a basic

technique for elliptical fitting in light of picture minute used to

recognize if the imaged organic product has an entire shape.

This strategy for estimating ellipticity has been accounted for

to yield preferred exactness over different techniques [14-15]

and to have a decent mistake resistance (i.e., resilience of

flawed circle shapes, concerning mango natural product).

Three criteria were utilized to judge whether the organic

product form was finished:

1 Ellipse region: an entire natural product at a camera

separation of around 15cm-50cm ought to have a zone of

1000 to 1800 pixels. Littler patches could be foundation or

deficient organic product, while bigger patches could be

bunched natural product.

2 Area (A): the proportion of genuine associated segment (a

division aftereffect of a natural product) measure (A) to

Ascertain circle region in light of the fitted ellipse Major Axis

a and Minor Axis b, defined by:

Area (A )= π×a×b (1)

3 Bounding box length versus Ellipse real length: the length

of a bounding box simply epitomizing the natural product was

utilized as a part of estimation of the organic product length.

The real pivot of the fitted ellipse is typically bigger than the

length of the bounding box, in any case, thick stalk closes that

were not evacuated by the line filter could bring about an

overestimation of natural product length. Accordingly, if the

length of bounding box was 4 pixels bigger than the ellipse

major axis a, the protest was rejected from measure

estimation.

(a) (b) (c)

Figure 4: Ellipse fitting and find Major Axis and Minor Axis

(a) Elliptical area detected (b) Total Height and Width of the

fruit (c) red part is the major axis and green part is the minor

axis.

Size Estimation of Each Mango

Once an associated segment was perceived as a totally natural

product, a bounding box was made, being the littlest rectangle

containing the segment, from that we found the mango height

and width in pixel, the pixel is converted to ‘cm’, which will

be helpful for the measurement. As the shape is an elliptical

area so here we have to calculate the major axis (a) and the

minor axis (b), as we found the total height and width so after

dividing by 2 we found the major axis (a) and the minor axis

(b) Fig.4, so we can find the area of the fruit.

a=Width/2 (Major Axis) (2)

b=Height/2 (Minor Axis) (3)

Area (A )= π×a×b (4)

Mango Fruit Size Estimation

Figure 5: Block diagram for Mango fruit size estimation.


8

Step 1: First we take the image which is taken by the

conventional RGB camera in a healthy weather.

Step 2: The colour threshold app which helps to threshold

colour images by manipulating the colour components of the

images based on different colour spaces. This is chosen to

removing the background.

Step 3: For removing the small objects from a particular

image, we use morphological operation. Here we use strel

function for creating structuring element.

Step 4: Basically for analyzing lines and curves in an image

the Hough Transform is used. But in case of detecting specific

curves like ellipse, it always faces the problem. So for

detecting ellipse, we use Randomized Hough Transform.

Step 5: The label block help to identify each object which is

connected to the background. The background pixels are

denoted as 0 in colour black and the object is in colour white.

The first object pixel is denoted as 1 then the next pixel is 2

and it continues as its number of the object. It helps to label

each object separately.

Step 6: After separate each object, we count each pixel of the

object and create a box for every object. We analyze its area

and count the pixel as width and height. Then we can find

each object’s height and width. (Major Axis & Minor

Axis).Here the pixel value is converted to ‘cm’ unit.

Result & Discussion We validate the proposed method in two ways (a) the sample

image having only one mango fruit (b) the sample images

having more than one mango fruit (Bunch). Here we take 10

number of the sample images having single mango fruit and

5 number of the sample images which contain more than one

mango fruit. The RGB image of on-tree mango as input,

then the colour thresholding process is occurs where we use

L*a*b colour space for remove background, as it is an on-

tree operation it cannot remove all the background.Then the

morphological binary operation occurs it help to remove the

small points. Then use RHT to identifying fruits then we

labeled each fruit by a bounding box, then we find major axis

and minor axis value then we calculate the area. The fruit

sample is collected from a mango garden for analysis. Prior

to validation by algorithm we measure the fruit size by

measurement tape, shown in Fig. 6. The 10 number of

samples having single mango fruit is validated by the

proposed algorithm and implied 3.07% error. Fig.10 contains

the graphical representation of Estimation of Actual and

Estimated Size of Single Mango Fruit The another way to

evaluate the performance of proposed algorithm is by taking

sample images having more than one mango fruit. Fig.8 (a)

cotain 2 number of mago fruit which manual measurement

area is 107.186 & 193.956 and algorithmic measurement

area is 106.06438 &189.50 which implied 1.0468%

&2.297% error. Fig.8(b) contain 4 number of mango fruit

which manual measurement is 049.25, 87.948, 164.9025,

106.794 and algorithmic measurement area is

39.29,79.18,142.32,96.147 which implied 20.22%,9.9695%,

13.6945%,15.2653% error. Fig 8(c) contain 2 number of

mango fruit which manual measurement area is 74.598 &

70.6725 and algorithmic measurement area is 69.91 & 66.94

which implied 6.284% & 5.2814% error. Fig.8(d) contain 4

number of mango fruit which manual measurement area is

84.808,84.82,118.72,115.9029 and algorithmic measurement

area is 70.769,69.90,110.8285,99.25 which implied

16.5657%,17.5902%,6.6494%,14.368% error. Fig.8(e)

contain 5 number of mango fruit which manual measurement

area is 108.364,90.69,70.6725,103.653,126.546 which

implied 5.2826%, 2.9331%, 11.2441%, 7.6409%,8.7909%

error. Fig.12 contains the graphical representation of

Comparison of Manual and Estimated size Measurement of

Each Mango of a Bunch. In Table 1 and Table 2 all the

information about fruit size is present.

Figure 6: Sample of measurement (a) measurement of Major

Axis (b) Measurement of Minor Axis.

Sample image of single mango fruit

(a) (b)

(c) (d)

(e) (f)


9

(g) (h)

(i) (j)

Figure 7: Sample images for single mango fruit

Sample images of bunch mango fruit

(a) (b)

(c) (d)

(e)

Figure 8: Sample images of bunch mango fruit

Size Measurement of Fruit

Workflow of proposed method

(Single Mango Size Estimation)

(c)

(c) (d)

(e) (f)

(g)

Figure 9: Process of size estimation (a) Original Image (b)

L*a*b Image (c) Binarized Image (d) Spur removed

Image (e) RHT applied Image (f) Bounding Box for each

Mango fruit (g) Measure the Major and Minor Axis.

(a) (b)


10

Table 1: Actual and Estimated Size of Single Mango Fruit.

0

20

40

60

80

100

120

140

160

Actual Estimated Error

Figure 10: Graphical Representation of Estimation of

Actual and Estimated Size of Single Mango Fruit

Workflow of proposed method

(Size Estimation Of Bunch Mango)

(a) (b)

(c) (d)

(e) (f) (g) (h)

[a= 3.1584 , b= 3.96] [a = 3.49, b = 7.35]

(i) (j) (k) (l)

[a=5.808, b=7.8016] [a=3.8026, b=8.052]

Figure 11(a-l): Process for size estimation of Mango Fruit in

Bunch (a) Original Image (b) Back ground Removed Image

(c) Label Each Mango of the Bunch (d) RHT applied Image

(e) Bounding Box of 1st Mango (f) Measurement of size of 1st

Mango (g) Bounding Box of 2nd Mango (h) Measurement of

size of 2nd Mango (i) Bounding Box of 3rd Mango (j)

Measurement of size of 3rd Mango (k)Bounding Box of 4th

Mango (l) Measurement of size of 4th Mango.

Input

Image

Actual Size Estimated Size % of Error in

Area cm2

Major

Axis(a

) cm

Minor

Axis(b

) cm

Are

a

cm2

Major

Axis(a)

cm

Minor

Axis(b

) cm

Are

a

cm2

(Estimated Area~

Actual Area)

/Actual Area × 100%

1 4.25 5 66.

74

3.8016 5.1744 61.7

86

7.4228%

2 4.75 4.5 67.

138

4.85 4.479 68.2

7

1.6861%

3 4.75 4.5 67.

138

4.68 4.548 66.8

85

0.3768%

4 7 5.5 120

.92

8.5 4.5 116.

81

3.52%

5 1.375 3.35 14.46

1.848 2.49 14.505

0.31%

6 1.375 3.35 14.

46

1.716 2.584 13.9

2

0.03%

7 4.5 5.5 77.

739

7

4.487 5.356 75.4

85

2.9003%

8 4.5 5.5 77.

7397

4.458 5.356 74.9

97

3.5281%

9 2.35 2.5 18.

453

2.14 2.487 16.7

2

9.4%

10 7.5 6 141.34

5

7.452 6.135 143.6

1.5307%

Average Error 3.07%


11

Table 2: Comparison of Manual and Estimated size

Measurement of Each Mango of a Bunch

(a)

(b)

(c)

(d)

(e)

Figure 12: Graphical representation of Comparison of

Manual and Estimated size Measurement of Each Mango of a

Bunch

I

n

pu

t

I

m

ag

e

Fr

uit

Numb

er

Estimated Actual % of Error

in

Measurement of Area

M

ajor

Ax

is(a)(

cm

)

Mi

nor

Ax

is(b)

(c

m)

Ar

ea (c

m2

)

Majo

r Axis

(a)

(cm)

Mino

r Axis

(b)

(cm)

Area

(cm2

)

(Estimated

Area ~ Actual Area) /

Actual Area

× 100%

Aver

age Error

1 1 5.016

6.732

106.0

64

5.25 6.5 107.186

1.0468% 1.67

21%

2 5.87

10.27

8

189.5

0

6.5 9.5 193.956

2.2974%

2 1 3.1

589

3.9

6

39.

29

4 3.9 49.2

5

20.2274%

2 3.4

3

7.3

5

79.

18

4 7 87.9

48

9.9695% 14.7

89

3 5.808

7.801

6

142.3

2

6 8.75 164.9025

13.6945%

4 3.802

8.052

96.14

7

4.25 8.5 113.4686

25

15.2653%

3 1 4.75

4.686

69.91

4.75 5 74.5987

6.2852%

2 4.8

48

4.1

12

66.

94

5 4.5 70.6

725

5.2814 5.78

3

4 1 3.168

7.112

70.76

9

3.75 7.2 84.82

16.5657%

2 3.1

28

7.1

15

69.

90

3.75 7.2 84.8

2

17.5902% 13.7

9

3 4.3

56

8.1 11

0.8

258

4.5 8.4 118.

72

6.6494%

4 3.9

6

7.9

8

99.

25

4.5 8.2 115.

9029

14.368%

5 1 5.52

5.924

102.6

4

5.75 6 108.3645

5.2826%

2 5.1

96

5.3

94

88.

03

5.25 5.5 90.6

9

2.933

3 4.3

46

4

4.6 62.

72

6

4.5 5 70.6

725

11.2441% 7.11

7

4 5.264

5.79

195.7

33

5.5 6 103.653

7.6409%

5 5.396

6.848

115.8

01

1

5.75 7 126.546

8.4909%


12

Conclusions & Future Work Here we proposed the Image Processing technique for on tree

fruit detection and measure each fruit size. In this paper, we

use colour thresholding, RHT & region detecting. The main

advantage of this paper is we detect every colour of mango

fruit and don’t face any problem regarding background.

The future work may be extended to improve in size

measurement section and try to measure the weight of each

fruit and analyse mature immature, which will be helpful for

the farmers.

References

[1] Gongal, A., et al. "Sensors and systems for fruit

detection and localization: A review." Computers and

Electronics in Agriculture 116 (2015): 8-19

[2] Payne, Alison B., et al. "Estimation of mango crop

yield using image analysis–segmentation method."

Computers and electronics in agriculture 91 (2013): 57-

64.

[3] Qureshi, W. S., et al. "Machine vision for counting fruit

on mango tree canopies." Precision Agriculture 18.2

(2017): 224-244.

[4] Omid, M., M. Khojastehnazhand, and A.

Tabatabaeefar. "Estimating volume and mass of citrus

fruits by image processing technique." Journal of food

Engineering 100.2 (2010): 315-321.

[5] Sethy, Prabira Kumar, Shwetapadma Panda, Santi

Kumari Behera, and Amiya Kumar Rath. "On Tree

Detection, Counting & Post-Harvest grading of fruits

Based on Image Processing and Machine Learning

Approach-A Review."

[6] Basca, Cosmin A., Mihai Talos, and Remus Brad.

"Randomized hough transform for ellipse detection

with result clustering." Computer as a Tool, 2005.

EUROCON 2005. The International Conference on.

Vol. 2. IEEE, 2005.

[7] Mustafa, Nur Badariah Ahmad, et al. "Classification of

fruits using Probabilistic Neural Networks-

Improvement using color features." TENCON 2011-

2011 IEEE Region 10 Conference. IEEE, 2011.

[8] Mendoza, Fernando, Petr Dejmek, and José M.

Aguilera. "Calibrated color measurements of

agricultural foods using image analysis." Postharvest

Biology and Technology 41.3 (2006): 285-295.

[9] Yam, Kit L., and Spyridon E. Papadakis. "A simple

digital imaging method for measuring and analyzing

color of food surfaces." Journal of food engineering

61.1 (2004): 137-142.

[10] Spreer, Wolfram, and Joachim Müller. "Estimating the

mass of mango fruit (Mangifera indica, cv. Chok Anan)

from its geometric dimensions by optical

measurement." Computers and electronics in

agriculture 75.1 (2011): 125-131.

[11] Moreda, G. P., Ortiz-Cañavate, J., García-Ramos, F. J.,

& Ruiz-Altisent, M. (2009). Non-destructive

technologies for fruit and vegetable size determination–

a review. Journal of Food Engineering, 92(2), 119-136.

[12] Wang, Ta Yuan, and Sing Kiong Nguang. "Low cost

sensor for volume and surface area computation of axi-

symmetric agricultural products." Journal of Food

Engineering 79.3 (2007): 870-877.

[13] Wang, Zhenglin, Kerry B. Walsh, and Brijesh Verma.

"On-Tree Mango Fruit Size Estimation Using RGB-D

Images." Sensors17.12 (2017): 2738.

[14] Rosin, Paul L. "Measuring shape: ellipticity,

rectangularity, and triangularity." Machine Vision and

Applications 14.3 (2003): 172-184.

[15] Stojmenovic, Milos, and Amiya Nayak. "Direct ellipse

fitting and measuring based on shape boundaries."

Pacific-Rim Symposium on Image and Video

Technology. Springer, Berlin, Heidelberg, 2007.


13

Documents

Image Processing Based Detection & Size …...accurate segmentation and detection of fruit in images of tree canopies. Fruit measure in image processing can be used to estimate the