基于轻量级无锚点深度卷积神经网络的树上苹果检测模型

doi:10.12133/j.smartag.2020.2.1.202001-SA004

摘要/Abstract

摘要：

为提高现有苹果目标检测模型在硬件资源受限制条件下的性能和适应性，实现在保持较高检测精度的同时，减轻模型计算量，降低检测耗时，减少模型计算和存储资源占用的目的，本研究通过改进轻量级的MobileNetV3网络，结合关键点预测的目标检测网络（CenterNet），构建了用于苹果检测的轻量级无锚点深度学习网络模型（M-CenterNet），并通过与CenterNet和单次多重检测器（Single Shot Multibox Detector，SSD）网络比较了模型的检测精度、模型容量和运行速度等方面的综合性能。对模型的测试结果表明，本研究模型的平均精度、误检率和漏检率分别为88.9%、10.9%和5.8%；模型体积和帧率分别为14.2MB和8.1fps；在不同光照方向、不同远近距离、不同受遮挡程度和不同果实数量等条件下有较好的果实检测效果和适应能力。在检测精度相当的情况下，所提网络模型体积仅为CenterNet网络的1/4；相比于SSD网络，所提网络模型的AP提升了3.9%，模型体积降低了84.3%；本网络模型在CPU环境中的运行速度比CenterNet和SSD网络提高了近1倍。研究结果可为非结构环境下果园作业平台的轻量化果实目标检测模型研究提供新的思路。

关键词: 机器视觉, 深度学习, 轻量级网络, 无锚点, 苹果检测

Abstract:

Intelligent production and robotic oporation are the efficient and sustainable agronomic route to cut down economic and environmental costs and boosting orchard productivity. In the actual scene of the orchard, high performance visual perception system is the premise and key for accurate and reliable operation of the automatic cultivation platform. Most of the existing apple detection models, however, are difficult to be used on the platforms with limited hardware resources in terms of computing power and storage capacity due to too many parameters and large model volume. In order to improve the performance and adaptability of the existing apple detection model under the condition of limited hardware resources, while maintaining detection accuracy, reducing the calculation of the model and the model computing and storage footprint, shorten detection time, this method improved the lightweight MobileNetV3 and combined the object detection network which was based on keypoint prediction (CenterNet) to build a lightweight anchor-free model (M-CenterNet) for apple detection. The proposed model used heatmap to search the center point (keypotint) of the object, and predict whether each pixel was the center point of the apple, and the local offset of the keypoint and object size of the apple were estimated based on the extracted center point without the need for grouping or Non-Maximum Suppression (NMS). In view of its advantages in model volume and speed, improved MobileNetV3 which was equipped with transposed convolutional layers for the better semantic information and location information was used as the backbone of the network. Compared with CenterNet and SSD (Single Shot Multibox Detector), the comprehensive performance, detection accuracy, model capacity and running speed of the model were compared. The results showed that the average precision, error rate and miss rate of the proposed model were 88.9%, 10.9% and 5.8%, respectively, and its model volume and frame rate were 14.2MB and 8.1fps. The proposed model is of strong environmental adaptability and has a good detection effect under the circumstance of various light, different occlusion, different fruits’ distance and number. By comparing the performance of the accuracy with the CenterNet and the SSD models, the results showed that the proposed model was only 1/4 of the size of CenterNet model while has comparable detection accuracy. Compared with the SSD model, the average precision of the proposed model increased by 3.9%, and the model volume decreased by 84.3%. The proposed model runs almost twice as fast using CPU than the CenterNet and SSD models. This study provided a new approach for the research of lightweight model in fruit detection with orchard mobile platform under unstructured environment.

Key words: machine vision, deep learning, lightweight network, anchor-free, apple detection

中图分类号:

TP183

夏雪, 孙琦鑫, 侍啸, 柴秀娟. 基于轻量级无锚点深度卷积神经网络的树上苹果检测模型[J]. 智慧农业（中英文）, 2020, 2(1): 99-110.

Xia Xue, Sun Qixin, Shi Xiao, Chai Xiujuan. Apple detection model based on lightweight anchor-free deep convolutional neural network[J]. Smart Agriculture, 2020, 2(1): 99-110.

参考文献

1	Kang H, Chen C. Fruit detection and segmentation for apple harvesting using visual sensor in orchards[J]. Sensors, 2019, 19(20): 4599-4614.
2	王丹丹, 何东健. 基于R-FCN深度卷积神经网络的机器人疏果前苹果目标的识别[J]. 农业工程学报, 2019, 35(3): 156-163.
2	Wang D, He D. Recognition of apple targets before fruits thinning by robot based on R-FCN deep convolution neural network[J]. Transactions of the CSAE, 2019, 35(3): 156-163.
3	赵德安，吴任迪，刘晓洋，等. 基于YOLO深度卷积神经网络的复杂背景下机器人采摘苹果定位[J]. 农业工程学报, 2019, 35(3): 164-173.
3	Zhao D, Wu R, Liu X, et al. Apple positioning based on YOLO deep convolutional neural network for picking robot in complex background[J]. Transactions of the CSAE, 2019, 35(3): 164-173.
4	Gené-Mola J, Vilaplana V, Rosell-Polo J R, et al. Multi-modal deep learning for Fuji apple detection using RGB-D cameras and their radiometric capabilities[J]. Computers and Electronics in Agriculture, 2019, 162: 689-698.
5	Underwood J P, Hung C, Whelan B, et al. Mapping almond orchard canopy volume, flowers, fruit and yield using LiDAR and vision sensors[J]. Computers and Electronics in Agriculture, 2016, 130: 83-96.
6	Bargoti S, Underwood J P. Image segmentation for fruit detection and yield estimation in apple orchards[J]. Journal of Field Robotics, 2017, 34(6): 1039-1060.
7	Silwal A, Gongal A, Karkee M. Apple identification in field environment with over the row machine vision system[J]. Agricultural Engineering International: CIGR Journal, 2014, 16(4): 66-75.
8	Wachs J P, Stern H I, Burks T, et al. Low and high-level visual feature-based apple detection from multi-modal images[J]. Precision Agriculture, 2010, 11(6): 717-735.
9	Qureshi W S, Payne A, Walsh K B, et al. Machine vision for counting fruit on mango tree canopies[J]. Precision Agriculture, 2017, 18(2): 224-244.
10	Zhou R, Damerow L, Sun Y, et al. Using colour features of cv.‘Gala’ apple fruits in an orchard in image processing to predict yield[J]. Precision Agriculture, 2012, 13(5): 568-580.
11	Wang Q, Nuske S, Bergerman M, et al. Automated crop yield estimation for apple orchards[C]// Experimental Robotics. Heidelberg: Springer, 2013: 745-758.
12	Xiong J, Liu Z, Lin R, et al. Green grape detection and picking-point calculation in a night-time natural environment using a charge-coupled device (ccd) vision sensor with artificial illumination[J]. Sensors, 2018, 18(4): 969-986.
13	Wang D, He D, Song H, et al. Combining SUN-based visual attention model and saliency contour detection algorithm for apple image segmentation[J]. Multimedia Tools and Applications, 2019, (78): 17391-17411.
14	Gongal A, Amatya S, Karkee M, et al. Sensors and systems for fruit detection and localization: A review[J]. Computers and Electronics in Agriculture, 2015, 116: 8-19.
15	Song Y, Glasbey C A, Horgan G W, et al. Automatic fruit recognition and counting from multiple images[J]. Biosystems Engineering, 2014, (118): 203-215.
16	Luo L, Tang Y, Zou X, et al. Robust grape cluster detection in a vineyard by combining the AdaBoost framework and multiple color components[J]. Sensors, 2016, 16(12): 2098-2118.
17	Wang C, Lee W S, Zou X, et al. Detection and counting of immature green citrus fruit based on the local binary patterns (lbp) feature using illumination-normalized images[J]. Precision Agriculture, 2018, 19(6): 1062-1083.
18	Guo Q, Chen Y, Tang Y, et al. Lychee fruit detection based on monocular machine vision in orchard environment[J]. Sensors, 2019, 19: no.4091.
19	Kestur R, Meduri A, Narasipura O. MangoNet: A deep semantic segmentation architecture for a method to detect and count mangoes in an open orchard[J]. Engineering Applications of Artificial Intelligence, 2019, 77: 59-69.
20	Szegedy C, Ioffe S, Vanhoucke V, et al. Inception-v4, inception-resnet and the impact of residual connections on learning[C]// Thirty-First AAAI Conference on Artificial Intelligence, 2017: 1-12.
21	Liu W, Wang Z, Liu X, et al. A survey of deep neural network architectures and their applications[J]. Neurocomputing, 2017, 234: 11-26.
22	Sa I, Ge Z, Dayoub F, et al. Deepfruits: a fruit detection system using deep neural networks[J]. Sensors, 2016, 16(8): 1222-1245.
23	Yu Y, Zhang K, Yang L, et al. Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN[J]. Computers and Electronics in Agriculture, 2019, (163): 104846-104855.
24	陈桂芬, 赵姗, 曹丽英, 等. 基于迁移学习与卷积神经网络的玉米植株病害识别[J]. 智慧农业, 2019, 1(2): 34-44.
24	Chen G, Zhao S, Cao L, et al. Corn plant disease recognition based on migration learning and convolutional neural network[J]. Smart Agriculture, 2019, 1(2): 34-44.
25	Tian Y, Yang G, Wang Z, et al. Apple detection during different growth stages in orchards using the improved YOLO-V3 model[J]. Computers and Electronics in Agriculture, 2019, (157): 417-426.
26	Koirala A, Walsh K B, Wang Z, et al. Deep learning for real-time fruit detection and orchard fruit load estimation: Benchmarking of ‘MangoYOLO’[J]. Precision Agriculture, 2019, (20): 1107-1135.
27	Williams H A M, Jones M H, Nejati M, et al. Robotic kiwifruit harvesting using machine vision, convolutional neural networks, and robotic arms[J]. Biosystems Engineering, 2019, (181): 140-156.
28	Wang D, Zhang N, Sun X, et al. AFP-Net: Realtime Anchor-Free Polyp Detection in Colonoscopy[C]// 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, 2019.
29	Law H, Deng J. Cornernet: Detecting objects as paired keypoints[C]// Proceedings of the European Conference on Computer Vision (ECCV), 2018: 734-750.
30	Duan K, Bai S, Xie L, et al. Centernet: Keypoint triplets for object detection[C]// Proceedings of the IEEE International Conference on Computer Vision. 2019: 6569-6578.
31	Zhou X, Zhuo J, Krahenbuhl P. Bottom-up object detection by grouping extreme and center points[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019: 850-859.
32	Zhou X, Wang D, Kr?henbühl P. Objects as Points[J]. Cornell University, 2019, arXiv: 1904. 07850.
33	Howard A, Sandler M, Chu G, et al. Searching for mobilenetv3[J]. Cornell University, 2019, arXiv: 1905. 02244.
34	郑冬, 李向群, 许新征. 基于轻量化 SSD 的车辆及行人检测网络[J]. 南京师大学报 (自然科学版), 2019, 42(1): 73-81.
34	Zheng D, Li X, Xu X. Vehicle and pedestrian detection model based on lightweight SSD[J]. Journal of Nanjing Normal University (Natural Science Edition). 2019, 42(1): 73-81.
35	白傑, 郝培涵, 陈思汉. 用轻量化卷积神经网络图像语义分割的交通场景理解[J]. 汽车安全与节能学报, 2018, 9(4): 433-440.
35	Bai J, Hao P, Chen S. Traffic scene understanding using image semantic segmentation with an improved lightweight convolutional-neural-network[J]. Journal of Automotive Safety and Energy. 2018, 9(4): 433-440.
36	毕鹏程, 罗健欣, 陈卫卫. 轻量化卷积神经网络技术研究[J]. 计算机工程与应用, 2019, 55(16): 25-35.
36	Bi P, Luo J, Chen W. Research on lightweight convolutional neural network technology[J]. Computer Engineering and Applications. 2019, 55(16): 25-35.
37	Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 7132-7141.
38	Lin T Y, Maire M, Belongie S, et al. Microsoft coco: common objects in context[C]// European Conference on Computer Vision. Springer, Cham, 2014: 740-755.

[1]	毛克彪, 张晨阳, 施建成, 王旭明, 郭中华, 李春树, 董立新, 吴门新, 孙瑞静, 武胜利, 姬大彬, 蒋玲梅, 赵天杰, 邱玉宝, 杜永明, 徐同仁. 基于人工智能的地球物理参数反演范式理论及判定条件[J]. 智慧农业(中英文), 2023, 5(2): 161-171.
[2]	夏雪, 柴秀娟, 张凝, 周硕, 孙琦鑫, 孙坦. 用于边缘计算设备的果树挂果量轻量化估测模型[J]. 智慧农业(中英文), 2023, 5(2): 1-12.
[3]	郭阳阳, 杜书增, 乔永亮, 梁栋. 深度学习在家畜智慧养殖中研究应用进展[J]. 智慧农业(中英文), 2023, 5(1): 52-65.
[4]	左敏, 胡天宇, 董微, 张可心, 张青川. 基于Informer神经网络的农产品物流需求预测分析——以华中地区为例[J]. 智慧农业(中英文), 2023, 5(1): 34-43.
[5]	计洁, 金洲, 王儒敬, 刘海燕, 李志远. 基于递进式卷积网络的农业命名实体识别方法[J]. 智慧农业(中英文), 2023, 5(1): 122-131.
[6]	胡松涛, 翟瑞芳, 王应华, 刘志, 朱剑忠, 任荷, 杨万能, 宋鹏. 基于多源数据的马铃薯植株表型参数提取[J]. 智慧农业(中英文), 2023, 5(1): 132-145.
[7]	刘晓航, 张昭, 刘嘉滢, 张漫, 李寒, FLORES Paulo, 韩雄哲. 基于多种深度学习算法的田间玉米籽粒检测与计数[J]. 智慧农业(中英文), 2022, 4(4): 49-60.
[8]	许钰林, 康孟珍, 王秀娟, 华净, 王浩宇, 沈震. 基于深度学习的玉米和大豆期货价格智能预测[J]. 智慧农业(中英文), 2022, 4(4): 156-163.
[9]	罗庆, 饶元, 金秀, 江朝晖, 王坦, 王丰仪, 张武. 基于改进YOLOv5s和多模态图像的树上毛桃检测[J]. 智慧农业(中英文), 2022, 4(4): 84-104.
[10]	张志博, 赵西宁, 高晓东, 张利, 杨孟豪. 基于改进Linknet网络的黄土高原苹果园精准提取[J]. 智慧农业(中英文), 2022, 4(3): 95-107.
[11]	商枫楠, 周学成, 梁英凯, 肖明玮, 陈桥, 罗陈迪. 基于改进YOLOX的自然环境中火龙果检测方法[J]. 智慧农业(中英文), 2022, 4(3): 120-131.
[12]	何锐敏, 郑可锋, 尉钦洋, 张小斌, 张俊, 朱怡航, 赵懿滢, 顾清. 基于改进Mask R-CNN模型的工厂化养蚕蚕体识别与计数[J]. 智慧农业(中英文), 2022, 4(2): 163-173.
[13]	庄家煜, 许世卫, 李杨, 熊露, 刘克宝, 钟志平. 基于深度学习的多种农产品供需预测模型[J]. 智慧农业(中英文), 2022, 4(2): 174-182.
[14]	邵明月, 张建华, 冯全, 柴秀娟, 张凝, 张文蓉. 深度学习在植物叶部病害检测与识别的研究进展[J]. 智慧农业(中英文), 2022, 4(1): 29-46.
[15]	郭秀明, 诸叶平, 李世娟, 张杰, 吕纯阳, 刘升平. 农业复杂环境下尺度自适应小目标识别算法——以蜜蜂为研究对象[J]. 智慧农业(中英文), 2022, 4(1): 140-149.