Infield Corn Kernel Detection and Counting Based on Multiple Deep Learning Networks

doi:10.12133/j.smartag.SA202207004

Abstract

Abstract:

Machine vision has been increasingly used for agricultural sensing tasks. The detection method based on deep learning for infield corn kernels can improve the detection accuracy. In order to obtain the number of lost corn kernels quickly and accurately after the corn harvest, and evaluate the corn harvest combine performance on grain loss, the method of directly using deep learning technology to count corn kernels in the field was developed and evaluated. Firstly, an RGB camera was used to collect image with different backgrounds and illuminations, and the datasets were generated. Secondly, different target detection networks for kernel recognition were constructed, including Mask R-CNN, EfficientDet-D5, YOLOv5-L and YOLOX-L, and the collected 420 effective images were used to train, verify and test each model. The number of images in train, verify and test datasets were 200, 40 and 180, respectively. Finally, the counting performances of different models were evaluated and compared according to the recognition results of test set images. The experimental results showed that among the four models, YOLOv5-L had overall the best performance, and could reliably identify corn kernels under different scenes and light conditions. The average precision (AP) value of the model for the image detection of the test set was 78.3%, and the size of the model was 89.3 MB. The correct rate of kernel count detection in four scenes of non-occlusion, surface mid-level-occlusion, surface severe-occlusion and aggregation were 98.2%, 95.5%, 76.1% and 83.3%, respectively, and F₁ values were 94.7%, 93.8%, 82.8% and 87%, respectively. The overall detection correct rate and F₁ value of the test set were 90.7% and 91.1%, respectively. The frame rate was 55.55 f/s, and the detection and counting performance were better than Mask R-CNN, EfficientDet-D5 and YOLOX-L networks. The detection accuracy was improved by about 5% compared with the second best performance of Mask R-CNN. With good precision, high throughput, and proven generalization, YOLOv5-L can realize real-time monitoring of corn harvest loss in practical operation.

Key words: harvest loss, infield corn kernel, deep learning, kernel count, YOLOv5-L, YOLOX-L, Mask R-CNN, EfficientDet-D5

CLC Number:

S126

LIU Xiaohang, ZHANG Zhao, LIU Jiaying, ZHANG Man, LI Han, FLORES Paulo, HAN Xiongzhe. Infield Corn Kernel Detection and Counting Based on Multiple Deep Learning Networks[J]. Smart Agriculture, 2022, 4(4): 49-60.

References

1	张伟锋, 刘新娇, 李秋枫, 等. 世界玉米生产与贸易概况分析[J]. 世界农业, 2014(3): 111-114.
1	ZHANG W, LIU X, LI Q, et al. General situation analysis of world corn production and trade[J]. World Agriculture, 2014(3): 111-114.
2	WU Y, LI X, MAO E, et al. Design and development of monitoring device for corn grain cleaning loss based on piezoelectric effect[J]. Computers and Electronics in Agriculture, 2020, 179(12): ID 105793.
3	XU L, WEI C, LIANG Z, et al. Development of rapeseed cleaning loss monitoring system and experiments in a combine harvester[J]. Biosystems Engineering, 2019, 178: 118-130.
4	VALIENTE-GONZáLEZJM, ANDREU-GARCíAG, POTTER P, et al. Automatic corn (Zea mays) kernel inspection system using novelty detection based on principal component analysis[J]. Biosystems Engineering, 2014, 117: 94-103.
5	ORLANDI G, CALVINI R, FOCA G, et al. Automated quantification of defective maize kernels by means of multivariate image analysis[J]. Food Control, 2018, 85: 259-268.
6	LI X, DAI B, HONG S, et al. Corn classification system based on computer vision[J]. Symmetry, 2019, 11(4): 591-591.
7	张恬, 赵德安, 周童. 图像处理在联合收割机夹带损失检测中的应用[J]. 农机化研究, 2009, 31(4): 70-72.
7	ZHANG T, ZHAO D, ZHOU T. Application of image processing on combine harvester attachment loss[J]. Journal of Agricultural Mechanization Research, 2009, 31(4): 70-72.
8	辛博, 吴涛, 陈春林, 等. 一种基于图像处理的谷物收割清选损失实时在线检测方法: CN107123115A[P]. 2017-09-01.
8	XIN B, WU T, CHEN C, et al. A real-time online detection method for grain harvesting and cleaning loss based on image processing: CN107123115A[P]. 2017-09-01.
9	WELLINGTON C K, BRUNS A J, SIERRA V S, et al. Grain quality monitoring: US10664726B2[P]. 2017-10-02.
10	孙红, 李松, 李民赞, 等. 农业信息成像感知与深度学习应用研究进展[J]. 农业机械学报, 2020, 51(5): 1-17.
10	SUN H, LI S, LI M, et al. Research progress of image sensing and deep learning in agriculture[J]. Transactions of the CSAM, 2020, 51(5): 1-17.
11	MONHOLLEN N S, SHINNERS K J, FRIEDE J C, et al. In-field machine vision system for identifying corn kernel losses[J]. Computers and Electronics in Agriculture, 2020, 174: ID 105496.
12	吕璐, 程虎, 朱鸿泰, 等. 基于深度学习的目标检测研究与应用综述[J]. 电子与封装, 2022, 22(1): 72-80.
12	LYU L, CHENG H, ZHU H, et al. Progress of research and application of object detection based on deep learning[J]. Electronics & Packaging, 2022, 22(1): 72-80.
13	董丽君, 曾志高, 易胜秋, 等. 基于YOLOv5的遥感图像目标检测[J]. 湖南工业大学学报, 2022, 36(3): 44-50.
13	DONG L, ZENG Z, YI S, et al. Research on a YOLOv5-Based remote sensing image target detection[J]. Journal of Hunan University of Technology, 2022, 36(3): 44-50.
14	谢富, 朱定局. 深度学习目标检测方法综述[J]. 计算机系统应用, 2022, 31(2): 1-12.
14	XIE F, ZHU D. Survey on deep learning object detection[J]. Computer Systems & Applications, 2022, 31(2): 1-12.
15	寇大磊, 权冀川, 张仲伟. 基于深度学习的目标检测框架进展研究[J]. 计算机工程与应用, 2019, 55(11): 25-34.
15	KOU D, QUAN J, ZHANG Z. Research on progress of object detection framework based on deep learning[J]. Computer Engineering and Applications, 2019, 55(11): 25-34.
16	包晓敏, 王思琪. 基于深度学习的目标检测算法综述[J]. 传感器与微系统, 2022, 41(4): 5-9.
16	BAO X, WANG S. Survey of object detection algorithm based on deep learning[J]. Transducer and Microsystem Technologies, 2022, 41(4): 5-9.
17	HE K, GKIOXARI G, DOLLáR P, et al. Mask r-cnn[C]// The IEEE International Conference on Computer Vision. Piscataway, New York, USA: IEEE, 2017: 2961-2969.
18	TAN M, PANG R, LE Q. Efficientdet: Scalable and efficient object detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, New York, USA: IEEE, 2020: 10781-10790.
19	BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLOv4: Optimal speed and accuracy of object detection[J/OL]. arXiv:2004.10934[cs.CV], 2020.
20	GE Z, LIU S, WANG F, et al. YOLOx: Exceeding yolo series in 2021[J/OL]. arXiv:, 2021.

[1]	LI Hao, DU Yuqiu, XIAO Xingzhu, CHEN Yanxi. Remote Sensing Identification Method of Cultivated Land at Hill County of Sichuan Basin Based on Deep Learning [J]. Smart Agriculture, 2024, 6(3): 34-45.
[2]	NIE Ganggang, RAO Honghui, LI Zefeng, LIU Muhua. Severity Grading Model for Camellia Oleifera Anthracnose Infection Based on Improved YOLACT [J]. Smart Agriculture, 2024, 6(3): 138-147.
[3]	ZHANG Jing, ZHAO Zexuan, ZHAO Yanru, BU Hongchao, WU Xingyu. Oilseed Rape Sclerotinia in Hyperspectral Images Segmentation Method Based on Bi-GRU and Spatial-Spectral Information Fusion [J]. Smart Agriculture, 2024, 6(2): 40-48.
[4]	FAN Jiangchuan, WANG Yuanqiao, GOU Wenbo, CAI Shuangze, GUO Xinyu, ZHAO Chunjiang. Fast Extracting Method for Strawberry Leaf Age and Canopy Width Based on Instance Segmentation Technology [J]. Smart Agriculture, 2024, 6(2): 95-106.
[5]	PANG Chunhui, CHEN Peng, XIA Yi, ZHANG Jun, WANG Bing, ZOU Yan, CHEN Tianjiao, KANG Chenrui, LIANG Dong. HI-FPN: A Hierarchical Interactive Feature Pyramid Network for Accurate Wheat Lodging Localization Across Multiple Growth Periods [J]. Smart Agriculture, 2024, 6(2): 128-139.
[6]	ZHANG Yuyu, BING Shuying, JI Yuanhao, YAN Beibei, XU Jinpu. Grading Method of Fresh Cut Rose Flowers Based on Improved YOLOv8s [J]. Smart Agriculture, 2024, 6(2): 118-127.
[7]	ZHANG Jianhua, YAO Qiong, ZHOU Guomin, WU Wendi, XIU Xiaojie, WANG Jian. Intelligent Identification of Crop Agronomic Traits and Morphological Structure Phenotypes: A Review [J]. Smart Agriculture, 2024, 6(2): 14-27.
[8]	GUO Wang, YANG Yusen, WU Huarui, ZHU Huaji, MIAO Yisheng, GU Jingqiu. Big Models in Agriculture: Key Technologies, Application and Future Directions [J]. Smart Agriculture, 2024, 6(2): 1-13.
[9]	ZHOU Huamao, WANG Jing, YIN Hua, CHEN Qi. Phenotype Analysis of Pleurotus Geesteranus Based on Improved Mask R-CNN [J]. Smart Agriculture, 2023, 5(4): 117-126.
[10]	WANG Herong, CHEN Yingyi, CHAI Yingqian, XU Ling, YU Huihui. Image Segmentation Method Combined with VoVNetv2 and Shuffle Attention Mechanism for Fish Feeding in Aquaculture [J]. Smart Agriculture, 2023, 5(4): 137-149.
[11]	LI Zhengkai, YU Jiahui, PAN Shijia, JIA Zefeng, NIU Zijie. Individual Tree Skeleton Extraction and Crown Prediction Method of Winter Kiwifruit Trees [J]. Smart Agriculture, 2023, 5(4): 92-104.
[12]	TANG Hui, WANG Ming, YU Qiushi, ZHANG Jiaxi, LIU Liantao, WANG Nan. Root Image Segmentation Method Based on Improved UNet and Transfer Learning [J]. Smart Agriculture, 2023, 5(3): 96-109.
[13]	PAN Weiting, SUN Mengli, YUN Yan, LIU Ping. Identification Method of Wheat Grain Phenotype Based on Deep Learning of ImCascade R-CNN [J]. Smart Agriculture, 2023, 5(3): 110-120.
[14]	GUAN Bolun, ZHANG Liping, ZHU Jingbo, LI Runmei, KONG Juanjuan, WANG Yan, DONG Wei. The Key Issues and Evaluation Methods for Constructing Agricultural Pest and Disease Image Datasets: A Review [J]. Smart Agriculture, 2023, 5(3): 17-34.
[15]	LONG Jianing, ZHANG Zhao, LIU Xiaohang, LI Yunxia, RUI Zhaoyu, YU Jiangfan, ZHANG Man, FLORES Paulo, HAN Zhexiong, HU Can, WANG Xufeng. Wheat Lodging Types Detection Based on UAV Image Using Improved EfficientNetV2 [J]. Smart Agriculture, 2023, 5(3): 62-74.