Infield Corn Kernel Detection and Counting Based on Multiple Deep Learning Networks

doi:10.12133/j.smartag.SA202207004

Abstract

Abstract:

Machine vision has been increasingly used for agricultural sensing tasks. The detection method based on deep learning for infield corn kernels can improve the detection accuracy. In order to obtain the number of lost corn kernels quickly and accurately after the corn harvest, and evaluate the corn harvest combine performance on grain loss, the method of directly using deep learning technology to count corn kernels in the field was developed and evaluated. Firstly, an RGB camera was used to collect image with different backgrounds and illuminations, and the datasets were generated. Secondly, different target detection networks for kernel recognition were constructed, including Mask R-CNN, EfficientDet-D5, YOLOv5-L and YOLOX-L, and the collected 420 effective images were used to train, verify and test each model. The number of images in train, verify and test datasets were 200, 40 and 180, respectively. Finally, the counting performances of different models were evaluated and compared according to the recognition results of test set images. The experimental results showed that among the four models, YOLOv5-L had overall the best performance, and could reliably identify corn kernels under different scenes and light conditions. The average precision (AP) value of the model for the image detection of the test set was 78.3%, and the size of the model was 89.3 MB. The correct rate of kernel count detection in four scenes of non-occlusion, surface mid-level-occlusion, surface severe-occlusion and aggregation were 98.2%, 95.5%, 76.1% and 83.3%, respectively, and F₁ values were 94.7%, 93.8%, 82.8% and 87%, respectively. The overall detection correct rate and F₁ value of the test set were 90.7% and 91.1%, respectively. The frame rate was 55.55 f/s, and the detection and counting performance were better than Mask R-CNN, EfficientDet-D5 and YOLOX-L networks. The detection accuracy was improved by about 5% compared with the second best performance of Mask R-CNN. With good precision, high throughput, and proven generalization, YOLOv5-L can realize real-time monitoring of corn harvest loss in practical operation.

Key words: harvest loss, infield corn kernel, deep learning, kernel count, YOLOv5-L, YOLOX-L, Mask R-CNN, EfficientDet-D5

CLC Number:

S126

LIU Xiaohang, ZHANG Zhao, LIU Jiaying, ZHANG Man, LI Han, FLORES Paulo, HAN Xiongzhe. Infield Corn Kernel Detection and Counting Based on Multiple Deep Learning Networks[J]. Smart Agriculture, 2022, 4(4): 49-60.

References

1	张伟锋, 刘新娇, 李秋枫, 等. 世界玉米生产与贸易概况分析[J]. 世界农业, 2014(3): 111-114.
1	ZHANG W, LIU X, LI Q, et al. General situation analysis of world corn production and trade[J]. World Agriculture, 2014(3): 111-114.
2	WU Y, LI X, MAO E, et al. Design and development of monitoring device for corn grain cleaning loss based on piezoelectric effect[J]. Computers and Electronics in Agriculture, 2020, 179(12): ID 105793.
3	XU L, WEI C, LIANG Z, et al. Development of rapeseed cleaning loss monitoring system and experiments in a combine harvester[J]. Biosystems Engineering, 2019, 178: 118-130.
4	VALIENTE-GONZáLEZJM, ANDREU-GARCíAG, POTTER P, et al. Automatic corn (Zea mays) kernel inspection system using novelty detection based on principal component analysis[J]. Biosystems Engineering, 2014, 117: 94-103.
5	ORLANDI G, CALVINI R, FOCA G, et al. Automated quantification of defective maize kernels by means of multivariate image analysis[J]. Food Control, 2018, 85: 259-268.
6	LI X, DAI B, HONG S, et al. Corn classification system based on computer vision[J]. Symmetry, 2019, 11(4): 591-591.
7	张恬, 赵德安, 周童. 图像处理在联合收割机夹带损失检测中的应用[J]. 农机化研究, 2009, 31(4): 70-72.
7	ZHANG T, ZHAO D, ZHOU T. Application of image processing on combine harvester attachment loss[J]. Journal of Agricultural Mechanization Research, 2009, 31(4): 70-72.
8	辛博, 吴涛, 陈春林, 等. 一种基于图像处理的谷物收割清选损失实时在线检测方法: CN107123115A[P]. 2017-09-01.
8	XIN B, WU T, CHEN C, et al. A real-time online detection method for grain harvesting and cleaning loss based on image processing: CN107123115A[P]. 2017-09-01.
9	WELLINGTON C K, BRUNS A J, SIERRA V S, et al. Grain quality monitoring: US10664726B2[P]. 2017-10-02.
10	孙红, 李松, 李民赞, 等. 农业信息成像感知与深度学习应用研究进展[J]. 农业机械学报, 2020, 51(5): 1-17.
10	SUN H, LI S, LI M, et al. Research progress of image sensing and deep learning in agriculture[J]. Transactions of the CSAM, 2020, 51(5): 1-17.
11	MONHOLLEN N S, SHINNERS K J, FRIEDE J C, et al. In-field machine vision system for identifying corn kernel losses[J]. Computers and Electronics in Agriculture, 2020, 174: ID 105496.
12	吕璐, 程虎, 朱鸿泰, 等. 基于深度学习的目标检测研究与应用综述[J]. 电子与封装, 2022, 22(1): 72-80.
12	LYU L, CHENG H, ZHU H, et al. Progress of research and application of object detection based on deep learning[J]. Electronics & Packaging, 2022, 22(1): 72-80.
13	董丽君, 曾志高, 易胜秋, 等. 基于YOLOv5的遥感图像目标检测[J]. 湖南工业大学学报, 2022, 36(3): 44-50.
13	DONG L, ZENG Z, YI S, et al. Research on a YOLOv5-Based remote sensing image target detection[J]. Journal of Hunan University of Technology, 2022, 36(3): 44-50.
14	谢富, 朱定局. 深度学习目标检测方法综述[J]. 计算机系统应用, 2022, 31(2): 1-12.
14	XIE F, ZHU D. Survey on deep learning object detection[J]. Computer Systems & Applications, 2022, 31(2): 1-12.
15	寇大磊, 权冀川, 张仲伟. 基于深度学习的目标检测框架进展研究[J]. 计算机工程与应用, 2019, 55(11): 25-34.
15	KOU D, QUAN J, ZHANG Z. Research on progress of object detection framework based on deep learning[J]. Computer Engineering and Applications, 2019, 55(11): 25-34.
16	包晓敏, 王思琪. 基于深度学习的目标检测算法综述[J]. 传感器与微系统, 2022, 41(4): 5-9.
16	BAO X, WANG S. Survey of object detection algorithm based on deep learning[J]. Transducer and Microsystem Technologies, 2022, 41(4): 5-9.
17	HE K, GKIOXARI G, DOLLáR P, et al. Mask r-cnn[C]// The IEEE International Conference on Computer Vision. Piscataway, New York, USA: IEEE, 2017: 2961-2969.
18	TAN M, PANG R, LE Q. Efficientdet: Scalable and efficient object detection[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, New York, USA: IEEE, 2020: 10781-10790.
19	BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLOv4: Optimal speed and accuracy of object detection[J/OL]. arXiv:2004.10934[cs.CV], 2020.
20	GE Z, LIU S, WANG F, et al. YOLOx: Exceeding yolo series in 2021[J/OL]. arXiv:, 2021.

[1]	XU Yulin, KANG Mengzhen, WANG Xiujuan, HUA Jing, WANG Haoyu, SHEN Zhen. Corn and Soybean Futures Price Intelligent Forecasting Based on Deep Learning [J]. Smart Agriculture, 2022, 4(4): 156-163.
[2]	LUO Qing, RAO Yuan, JIN Xiu, JIANG Zhaohui, WANG Tan, WANG Fengyi, ZHANG Wu. Multi-Class on-Tree Peach Detection Using Improved YOLOv5s and Multi-Modal Images [J]. Smart Agriculture, 2022, 4(4): 84-104.
[3]	SHANG Fengnan, ZHOU Xuecheng, LIANG Yingkai, XIAO Mingwei, CHEN Qiao, LUO Chendi. Detection Method for Dragon Fruit in Natural Environment Based on Improved YOLOX [J]. Smart Agriculture, 2022, 4(3): 120-131.
[4]	ZHANG Zhibo, ZHAO Xining, GAO Xiaodong, ZHANG Li, YANG Menghao. Accurate Extraction of Apple Orchard on the Loess Plateau Based on Improved Linknet Network [J]. Smart Agriculture, 2022, 4(3): 95-107.
[5]	HE Ruimin, ZHENG Kefeng, WEI Qinyang, ZHANG Xiaobin, ZHANG Jun, ZHU Yihang, ZHAO Yiying, GU Qing. Identification and Counting of Silkworms in Factory Farm Using Improved Mask R-CNN Model [J]. Smart Agriculture, 2022, 4(2): 163-173.
[6]	ZHUANG Jiayu, XU Shiwei, LI Yang, XIONG Lu, LIU Kebao, ZHONG Zhiping. Supply and Demand Forecasting Model of Multi-Agricultural Products Based on Deep Learning [J]. Smart Agriculture, 2022, 4(2): 174-182.
[7]	YANG Guofeng, HE Yong, FENG Xuping, LI Xiyao, ZHANG Jinnuo, YU Zeyu. Methods and New Research Progress of Remote Sensing Monitoring of Crop Disease and Pest Stress Using Unmanned Aerial Vehicle [J]. Smart Agriculture, 2022, 4(1): 1-16.
[8]	SHAO Mingyue, ZHANG Jianhua, FENG Quan, CHAI Xiujuan, ZHANG Ning, ZHANG Wenrong. Research Progress of Deep Learning in Detection and Recognition of Plant Leaf Diseases [J]. Smart Agriculture, 2022, 4(1): 29-46.
[9]	LI Zhijun, YANG Shenghui, SHI Deshuai, LIU Xingxing, ZHENG Yongjun. Yield Estimation Method of Apple Tree Based on Improved Lightweight YOLOv5 [J]. Smart Agriculture, 2021, 3(2): 100-114.
[10]	FLORES Paulo, ZHANG Zhao. Wheat Lodging Ratio Detection Based on UAS Imagery Coupled with Different Machine Learning and Deep Learning Algorithms [J]. Smart Agriculture, 2021, 3(2): 23-34.
[11]	QIU Wenjie, YE Jin, HU Liangqing, YANG Juan, LI Qili, MO Jianyou, YI Wanmao. Distilled-MobileNet Model of Convolutional Neural Network Simplified Structure for Plant Disease Recognition [J]. Smart Agriculture, 2021, 3(1): 109-117.
[12]	LI Liangde, WANG Xiujuan, KANG Mengzhen, HUA Jing, FAN Menghan. Agricultural Named Entity Recognition Based on Semantic Aggregation and Model Distillation [J]. Smart Agriculture, 2021, 3(1): 118-128.
[13]	GUO Wei, WU Huarui, ZHU Huaji. Design and Application of Facility Greenhouse Image Collecting and Environmental Data Monitoring Robot System [J]. Smart Agriculture, 2020, 2(3): 48-60.
[14]	LI Hualong, LI Miao, ZHAN Kai, LIU Xianwang, YANG Xuanjiang, HU Zelin, GUO Panpan. Construction Method and Performance Test of Prediction Model for Laying Hen Breeding Environmental Quality Evaluation [J]. Smart Agriculture, 2020, 2(3): 37-47.
[15]	WEI Jing, WANG Yuting, YUAN Huizhu, ZHANG Menglei, WANG Zhenying. Identification and Morphological Analysis of Adult Spodoptera Frugiperda and Its Close Related Species Using Deep Learning [J]. Smart Agriculture, 2020, 2(3): 75-85.