Rapid Recognition and Picking Points Automatic Positioning Method for Table Grape in Natural Environment

doi:10.12133/j.smartag.SA202304001

Abstract

Abstract:

[Objective] Rapid recognition and automatic positioning of table grapes in the natural environment is the prerequisite for the automatic picking of table grapes by the picking robot. [Methods] An rapid recognition and automatic picking points positioning method based on improved K-means clustering algorithm and contour analysis was proposed. First, euclidean distance was replaced by a weighted gray threshold as the judgment basis of K-means similarity. Then the images of table grapes were rasterized according to the K value, and the initial clustering center was obtained. Next, the average gray value of each cluster and the percentage of pixel points of each cluster in the total pixel points were calculated. And the weighted gray threshold was obtained by the average gray value and percentage of adjacent clusters. Then, the clustering was considered as have ended until the weighted gray threshold remained unchanged. Therefore, the cluster image of table grape was obtained. The improved clustering algorithm not only saved the clustering time, but also ensured that the K value could change adaptively. Moreover, the adaptive Otsu algorithm was used to extract grape cluster information, so that the initial binary image of the table grape was obtained. In order to reduce the interference of redundant noise on recognition accuracy, the morphological algorithms (open operation, close operation, images filling and the maximum connected domain) were used to remove noise, so the accurate binary image of table grapes was obtained. And then, the contours of table grapes were obtained by the Sobel operator. Furthermore, table grape clusters grew perpendicular to the ground due to gravity in the natural environment. Therefore, the extreme point and center of gravity point of the grape cluster were obtained based on contour analysis. In addition, the linear bundle where the extreme point and the center of gravity point located was taken as the carrier, and the similarity of pixel points on both sides of the linear bundle were taken as the judgment basis. The line corresponding to the lowest similarity value was taken as the grape stem, so the stem axis of the grape was located. Moreover, according to the agronomic picking requirements of table grapes, and combined with contour analysis, the region of interest (ROI) in picking points could be obtained. Among them, the intersection of the grapes stem and the contour was regarded as the middle point of the bottom edge of the ROI. And the 0.8 times distance between the left and right extreme points was regarded as the length of the ROI, the 0.25 times distance between the gravity point and the intersection of the grape stem and the contour was regarded as the height of the ROI. After that, the central point of the ROI was captured. Then, the nearest point between the center point of the ROI and the grape stem was determined, and this point on the grape stem was taken as the picking point of the table grapes. Finally, 917 grape images (including Summer Black, Moldova, and Youyong) taken by the rear camera of MI8 mobile phone at Jinniu Mountain Base of Shandong Fruit and Vegetable Research Institute were verified experimentally. Results and Discussions] The results showed that the success rate was 90.51% when the error between the table grape picking points and the optimal points were less than 12 pixels, and the average positioning time was 0.87 s. The method realized the fast and accurate localization of table grape picking points. On top of that, according to the two cultivation modes (hedgerow planting and trellis planting) of table grapes, a simulation test platform based on the Dense mechanical arm and the single-chip computer was set up in the study. 50 simulation tests were carried out for the four conditions respectively, among which the success rate of localization for purple grape picking point of hedgerow planting was 86.00%, and the average localization time was 0.89 s; the success rate of localization for purple grape identification and localization of trellis planting was 92.00%, and the average localization time was 0.67 s; the success rate of localization for green grape picking point of hedgerow planting was 78.00%, and the average localization time was 0.72 s; and the success rate of localization for green grape identification and localization of trellis planting was 80.00%, and the average localization time was 0.71 s. [Conclusions] The experimental results showed that the method proposed in the study can meet the requirements of table grape picking, and can provide technical supports for the development of grape picking robot.

Key words: table grape, K-means, contour analysis method, fruit stem axis, picking point, picking robot

CLC Number:

S126

ZHU Yanjun, DU Wensheng, WANG Chunying, LIU Ping, LI Xiang. Rapid Recognition and Picking Points Automatic Positioning Method for Table Grape in Natural Environment[J]. Smart Agriculture, 2023, 5(2): 23-34.

Figures/Tables 17

Fig.1

Fig. 2

Fig. 3

Fig. 4

Fig.5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Table 1

Table 2

Localization information of table grapes

样本号	点D坐标	点M坐标	最优采摘点范围			点E坐标			与最优采摘点的误差			运行时间/s
样本号	$x D, y D$	$x M, y M$	X	Y		x	y		e_x	e_y	e	运行时间/s
1	（157，145）	（155，122）	155		133	146		128	9	5	10.3	0.68
2	（155，133）	（153，116）	153		124	148		119	5	5	7.1	0.78
3	（160，143）	（159，116）	159		129	155		133	4	4	5.7	0.66
4	（142，52）	（142，19）	142		35	149		40	7	5	8.6	0.59
5	（152，115）	（151，91）	151		102	136		134	15	32	35.3	1.42
6	（172，149）	（169，1230）	170		135	176		139	6	4	7.2	0.68
7	（163，38）	（161，7）	161		22	165		28	4	6	7.2	0.65
8	（155，26）	（156，-14）	155		6	148		13	7	7	9.9	0.67
9	（204，187）	（204，172）	204		179	199		170	5	9	10.3	0.64
10	（209，55）	（208，20）	208		37	201		43	7	6	9.2	1.28

Table 2

Fig.11

Fig.12

Fig.13

Fig. 14

Table 3

References 29

1	皆彦吉. 果蔬采摘机器人的研究现状、问题及对策[J]. 时代农机, 2018, 45(9): ID 42.
	JIE Y J. Research status, problems and countermeasures of fruit and vegetable picking robot[J]. Times agricultural machinery, 2018, 45(9): ID 42.
2	袁志英, 陈进, 郭鹏, 等. 果蔬采摘机器人的发展现状、问题及对策分析[C]// 中国农业机械学会国际学术年会. 北京, 中国: 中国农业机械学会, 2012.
	YUAN Z Y, CHEN J, GUO P, et al. Analysis on the development status, problems and countermeasures of fruit and vegetable picking robot[C]// International Academic Annual Conference of Chinese Society of Agricultural Machinery. Beijing, China: Chinese Society of Agricultural Machinery, 2012.
3	张洁, 李艳文. 果蔬采摘机器人的研究现状、问题及对策[J]. 机械设计, 2010, 27(6): 1-5.
	ZHANG J, LI Y W. Research situation, problems and solutions of fruit-vegetable picking robots[J]. Journal of machine design, 2010, 27(6): 1-5.
4	KONDO N, SHIBANO Y, MOHRI K, et al. Basic studies on robot to work in vineyard (Part 1) manipulator and harvesting hand[J]. Journal of the Japanese society of agricultural machinery, 1993, 55(6): 85-94.
5	KONDO N, SHIBANO Y, MOHRI K, et al. Basic studies on robot to work in vineyard (Part 2) discriminating, position detecting and harvesting experiments by using visual sensor[J]. Journal of the Japanese society of agricultural machinery, 1994, 56(1): 45-53.
6	MONTA M, KONDO N, SHIBANO Y, et al. Basic studies on robot to work in vineyard (Part 3) measurement of physical properties for robotization and manufacture of berry thinning hand[J]. Journal of the Japanese society of agricultural machinery, 1994, 56(2): 93-100.
7	LIU S, WHITTY M. Automatic grape bunch detection in vineyards with an SVM classifier[J]. Journal of applied logic, 2015, 13(4): 643-653.
8	REIS M J C S, MORAIS R, PERES E, et al. Automatic detection of bunches of grapes in natural environment from color images[J]. Journal of applied logic, 2012, 10(4): 285-290.
9	雷旺雄, 卢军. 葡萄采摘机器人采摘点的视觉定位[J]. 江苏农业学报, 2020, 36(4): 1015-1021.
	LEI W X, LU J. Visual positioning method for picking point of grape picking robot[J]. Jiangsu Journal of Agricultural Sciences, 2020, 36(4): 1015-1021.
10	PEREZ-ZAVALA R, MIGUEL T T, AUAT C F, et al. A pattern recognition strategy for visual grape bunch detection in vineyards[J]. Computers and Electronics in Agriculture, 2018, 151: 136-149.
11	BERENSTEIN R, SHAHAR OBEN, SHAPIRO A, et al. Grape clusters and foliage detection algorithms for autonomous selective vineyard sprayer[J]. Intelligent service robotics, 2010, 3(4): 233-243.
12	刘平, 朱衍俊, 张同勋, 等. 自然环境下贴叠葡萄串的识别与图像分割算法[J]. 农业工程学报, 2020, 36(6): 161-169.
	LIU P, ZHU Y J, ZHANG T X, et al. Algorithm for recognition and image segmentation of overlapping grape cluster in natural environment[J]. Transactions of the Chinese society of agricultural engineering, 2020, 36(6): 161-169.
13	MIAO Y B, HUANG L L, ZHANG S. A two-step phenotypic parameter measurement strategy for overlapped grapes under different light conditions[J]. Sensors, 2021, 21(13), ID 4532.
14	LUO L F, TANG Y C, LU Q H, et al. A vision methodology for harvesting robot to detect cutting points on peduncles of double overlapping grape clusters in a vineyard[J]. Computers in industry, 2018, 99: 130-139.
15	罗陆锋, 邹湘军, 熊俊涛, 等. 自然环境下葡萄采摘机器人采摘点的自动定位[J]. 农业工程学报, 2015, 31(2): 14-21.
	LUO L F, ZOU X J, XIONG J T, et al. Automatic positioning for picking point of grape picking robot in natural environment[J]. Transactions of the Chinese society of agricultural engineering, 2015, 31(2): 14-21.
16	XIONG J T, LIU Z, LIN R, et al. Green grape detection and picking-point calculation in a night-time natural environment using a charge-coupled device (CCD) vision sensor with artificial illumination[J]. Sensors, 2018, 18(4): ID 969.
17	熊俊涛, 何志良, 汤林越, 等. 非结构环境中扰动葡萄采摘点的视觉定位技术[J]. 农业机械学报, 2017, 48(4): 29-33, 81.
	XIONG J T, HE Z L, TANG L Y, et al. Visual localization of disturbed grape picking point in non-structural environment[J]. Transactions of the Chinese society for agricultural machinery, 2017, 48(4): 29-33, 81.
18	LUO L F, TANG Y C, ZOU X J, et al. Vision-based extraction of spatial information in grape clusters for harvesting robots[J]. Biosystems Engineering, 2016, 151(11): 90-104.
19	袁妍. 基于深度视觉的棚架葡萄采摘机器人手眼系统设计与果梗近景识别研究[D]. 镇江: 江苏大学, 2019.
	YUAN Y. Hand-eye system design and close-shot stem recognition for robotic harvesting of trellis grape clusters based on depth sensing[D]. Zhenjiang: Jiangsu University, 2019.
20	罗庆, 饶元, 金秀, 等. 基于改进YOLOv5s和多模态图像的树上毛桃检测(英文)[J]. 智慧农业(中英文), 2022, 4(4): 84-104.
	LUO Q, RAO Y, JIN X, et al. Multi-class on-tree peach detection using improved YOLOv5s and multi-modal images[J]. Smart Agriculture, 2022, 4(4): 84-104.
21	商枫楠, 周学成, 梁英凯, 等. 基于改进YOLOX的自然环境中火龙果检测方法[J]. 智慧农业(中英文), 2022, 4(3): 120-131.
	SHANG F N, ZHOU X C, LIANG Y K, et al. Detection method for dragon fruit in natural environment based on improved YOLOX[J]. Smart agriculture, 2022, 4(3): 120-131.
22	周文静, 查志华, 吴杰. 改进圆形Hough变换的田间红提葡萄果穗成熟度判别[J]. 农业工程学报, 2020, 36(9): 205-213.
	ZHOU W J, ZHA Z H, WU J. Maturity discrimination of "Red Globe" grape cluster in grapery by improved circle Hough transform[J]. Transactions of the Chinese society of agricultural engineering, 2020, 36(9): 205-213.
23	宁政通, 罗陆锋, 廖嘉欣, 等. 基于深度学习的葡萄果梗识别与最优采摘定位[J]. 农业工程学报, 2021, 37(9): 222-229.
	NING Z T, LUO L F, LIAO J X, et al. Recognition and the optimal picking point location of grape stems based on deep learning[J]. Transactions of the Chinese society of agricultural engineering, 2021, 37(9): 222-229.
24	孙碧玉. 基于深度学习的番茄果实目标检测和番茄串采摘点定位技术研究[D]. 天津: 天津理工大学, 2021.
	SUN B Y. Research on tomato fruit target detection and tomato string picking point location technology based on deep learning[D]. Tianjin: Tianjin University of Technology, 2021.
25	江梅, 孙飒爽, 何东健, 等. 融合K-means聚类分割算法与凸壳原理的遮挡苹果目标识别与定位方法[J]. 智慧农业, 2019, 1(2): 45-54.
	JIANG M, SUN S S, HE D J, et al. Recognition and localization method of occluded apples based on K-means clustering segmentation algorithm and convex hull theory[J]. Smart agriculture, 2019, 1(2): 45-54.
26	陈宏, 马峻, 陈寿宏, 等. 改进K-means聚类与Otsu算法的光栅投影轮廓有效点自动提取方法[J]. 激光杂志, 2020, 41(9): 40-46.
	CHEN H, MA J, CHEN S H, et al. Automatic extraction of effective points from fringe projection profilometry based on improved K-means clustering and Otsu algorithms[J]. Laser journal, 2020, 41(9): 40-46.
27	夏亚飞. 基于K均值聚类和二维Otsu的棉花HSV图像分割[J]. 软件, 2020, 41(7): 170-173.
	XIA Y F. Cotton HSV image segmentation based on K-means clustering and two-dimensional otsu[J]. Computer engineering & software, 2020, 41(7): 170-173.
28	曹帅帅, 陈雪鑫, 苗圃, 等. 基于PSO与K-均值聚类算法优化结合的图像分割方法[J]. 计算机与现代化, 2020(1): 22-27.
	CAO S S, CHEN X X, MIAO P, et al. Image segmentation method based on optimization of PSO algorithm and K-means clustering algorithm[J]. Computer and modernization, 2020(1): 22-27.
29	YU Y, ZHANG K L, YANG L, et al. Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN[J]. Computers and electronics in agriculture, 2019, 163: ID 104846.

样本号	初始K值	最终K值		改进后聚类时间/s	改进前目标簇的误差平方和	改进后目标簇的误差平方和
1	18	7	2.65	0.66	0.0101	0.0098
2	20	8	3.23	0.77	0.0092	0.0110
3	19	7	2.35	0.65	0.0083	0.0083
4	20	7	2.21	0.58	0.0053	0.0066
5	21	4	3.68	1.40	0.0122	0.0115
6	21	6	2.68	0.67	0.0152	0.0147
7	18	3	2.64	0.64	0.0110	0.0113
8	18	8	2.69	0.65	0.0115	0.0101
9	20	9	3.01	0.63	0.0093	0.0100
10	20	8	3.77	1.26	0.0082	0.0090

[1]	CHEN Dong, Tian'en CHEN, JIANG Shuwen, ZHANG Chi, WANG Cong, LU Mengyao. Optimal Model of Chicken Distribution Vehicle Scheduling Based on Order Clustering [J]. Smart Agriculture, 2020, 2(4): 137-148.
[2]	Jiang Mei, Sun Sashuang, He Dongjian, Song Huaibo. Recognition and localization method of occluded apples based on K-means clustering segmentation algorithm and convex hull theory [J]. Smart Agriculture, 2019, 1(2): 45-54.