A fast extraction method of broccoli phenotype based on machine vision and deep learning

doi:10.12133/j.smartag.2020.2.1.201912-SA003

Abstract

Abstract:

How to accurately obtain the area and freshness of broccoli head in the field condition is the key step to determine broccoli growth. However, the rapid segmentation and grading of broccoli ball remains difficult due to the low equipment development level. In this research, we combined an advanced computer vision technique with a deep learning architecture to allow the acquisition of real-time and accurate information about broccoli ball. By constructing a private image dataset with 442 of broccoli-ball images (acquired using a self-developed imaging system) under controlled conditions, a deep convolutional neural network named “Improved ResNet” was trained to extract the broccoli pixels from the background. The technical process of our method includes: (1) take the orthophoto images of broccoli head based on a near-ground image acquisition platform and establish the original data set; (2) preprocess the training images and input the model for segmentation; (3) use the PSOA and Otsu algorithm for fine segment based on color characteristics to obtain the freshness information. The experimental results demonstrated that the precision of the segmentation model is about 0.9 which is robust to the interference of soil reflectance fluctuation, canopy shadow, leaf occlusion and so on. Our experiments showed that a combination of improved ResNet and PSOA method got higher broccoli balls segmenting and grading precision. One major advantage of this approach is that dealing with only a few images, reducing the data volume and memory requirements for the image processing. All of the methods were evaluated using ground-truth data from three different varieties, which we also make available to the research community for subsequent algorithm development and result comparison. Compared with other 4 approaches, the evaluation results shows better performance regarding the segmentation and grading accuracy. The results of SSIM, precision, recall and F-measure by using Improved ResNet were about 0.911, 0.897, 0.908 and 0.907 respectively, which were 10%~15% higher than the traditional approaches. In addition, on the basis of the segmentation results, PSO-Otsu method was proved that it can be used to achieve a quickly analysis to the freshness of the ball, with the mean accuracy of 0.82. Overall, the proposed method is a high-throughput method to acquire multi-phenotype parameters of broccoli in field condition, which can support the research of broccoli field monitoring and traits tracking.

Key words: deep learning, broccoli phenotype, computer vision, automatic grading, field platform

CLC Number:

Zhou Chengquan, Ye Hongbao, Yu Guohong, Hu Jun, Xu Zhifu. A fast extraction method of broccoli phenotype based on machine vision and deep learning[J]. Smart Agriculture, 2020, 2(1): 121-132.

Figures/Tables 12

Fig.1

Fig.2

Fig.3

Fig.4

Table 1

Fig.5

Table 2

Fig.6

Table 3

Fig. 7

Table 4

Fig. 8

References 28

1	郭香凤, 向进乐, 李秀珍, 等. 贮藏温度对西兰花净菜品质的影响[J]. 农业机械学报, 2008, 39(2): 201-204.
2	夏永泉, 李耀斌, 曾莎. 基于HSI颜色空间的植物叶片病斑提取方法[J]. 江苏农业科学, 2015(08): 420-422.
3	高理文, 林小桦. 基于Lab*彩色空间和局域动态阈值的药用植物叶片图像分割[J]. 计算机应用与软件, 2014， (1): 232-235.
	Gao L, Lin X. Segmentation of images of medicinal plant leaves based on Lab* colour space and local dynamic threshold[J]. Computer Applications and Software, 2014， (1): 232-235.
4	董晓辉. 基于多彩色空间的麦田监控图像分割技术研究[D]. 开封：河南大学， 2015.
	Dong X. Wheat mornitoring image segmatation technology using variety of color spaces[D]. Kaifeng: Henan Agricultural University, 2015.
5	魏丽冉, 岳峻, 李振波, 等. 基于核函数支持向量机的植物叶部病害多分类检测方法[J]. 农业机械学报, 2017, (S0): 166-171.
	Wei L, Yue J, Li Z, et al. Multi-classification detection method of plant leaf disease based on kernel function SVM[J]. Transactions of the CSAM, 2017, (S0): 166-171.
6	周俊, 刘丽川, 杨继平. 基于K-均值聚类与小波分析的声发射信号去噪[J]. 石油化工高等学校学报, 2013, 26(3): 69-73.
	Zhou J, Liu L, Yang J. Acoustic emission signal denoising based on K-means clustering and wavelet analysis[J]. Journal of Petrochemical Universities, 2013, 26(3): 69-73.
7	李先锋, 朱伟兴, 纪滨, 等. 基于图像处理和蚁群优化的形状特征选择与杂草识别[J]. 农业工程学报, 2010, 26(10): 178-182.
	Li X, Zhu W, Ji B, et al. Shape feature selection and weed recognition based on image processing and ant colony optimization[J]. Transactions of the CSAE, 2010, 26(10): 178-182.
8	袁定帅, 陈洁, 赖晓芳, 等. 不同贮藏条件对西兰花感官品质及抗氧化物质的影响[J]. 食品科技, 2017， (4): 40-45.
	Yuan D, Chen J, Lai X, et al. Effects of diifferent storage conditions on sensory quality and antioxidants of broccoli[J]. Food Science and Techonology, 2017， (4): 40-45.
9	Lee S H, Chan C S, Wilkin P, et al. Deep-plant: Plant identification with convolutional neural networks[C]// IEEE International Conference on Image Processing (ICIP), 2015: 452-456.
10	Dyrmann M, Karstoft H, Midtiby H S. Plant species classification using deep convolutional neural network[J]. Biosystems Engineering, 2016, 151: 72-80.
11	Too E C, Li Y, Njuki S, et al. A comparative study of fine-tuning deep learning models for plant disease identification[J]. Computers and Electronics in Agriculture, 2018, 161: 272-279.
12	Fuentes A, Yoon S, Kim S C, et al. A robust deep-learning-based detector for real-time tomato plant diseases and pests recognition[J]. Sensors, 2017, 17: 2022-2042.
13	李淼, 王敬贤, 李华龙, 等. 基于 CNN 和迁移学习的农作物病害识别方法研究[J]. 智慧农业, 2019, 1(3): 46-55.
	Li M, Wang J, Li H, et al. Method for identifying crop disease based on CNN and transfer learning[J]. Smart Agriculture, 2019, 1(3): 46-55.
14	陈桂芬, 赵姗, 曹丽英, 等. 基于迁移学习与卷积神经网络的玉米植株病害识别[J]. 智慧农业, 2019, 1(2): 34-44.
	Chen G, Zhao S, Cao L, et al. Corn plant disease recognition based on migration learning and convolutional neural network[J]. Smart Agriculture, 2019, 1(2): 34-44.
15	Fu Z, Robles-Kelly A. A quadratic programming approach to image labelling[J]. IET Computer Vision, 2009, 2(4): 193-207.
16	刘万军, 梁雪剑, 曲海成. 不同池化模型的卷积神经网络学习性能研究[J]. 中国图象图形学报, 2016, 21(9): 1178-1190.
	Liu W, Liang X, Qu H. Learning performance of convolutional neural networks with different pooling models[J]. Journal of Image and Graphics, 2016, 21(9): 1178-1190.
17	李林, 顾进锋, 宋安捷, 等. 基于GPU的生态环境遥感评价模型并行化研究[J]. 农业机械学报, 2017(5): 140-146.
	Li L, Gu J, Song A, et al. Parallelization on model of ecological environment remote sensing evaluation based GPU[J]. Transactions of the CSAM, 2017(5): 140-146.
18	He K, Zhang X, Ren S, et al. Identity mappings in deep residual networks[J]. Computer Science, 2016: 630-645.
19	Wang Y, Yang A, Chen X, et al. A deep learning approach for blind drift calibration of sensor networks[J]. IEEE Sensors Journal, 2017, 17(13): 4158-4171.
20	Gong L, Jiang S, Yang Z, et al. Automated pulmonary nodule detection in CT images using 3D deep squeeze-and-excitation networks[J]. International Journal of Computer Assisted Radiology and Surgery, 2019, 14(11): 1969-1979.
21	Zhou X, Jin K, Shang Y, et al. Visually interpretable representation learning for depression recognition from facial images[J]. IEEE Transactions on Affective Computing, 2018, DOI: 10.1109/TAFFC.2018.2828819. doi: 10.1109/TAFFC.2018.2828819
22	Wurfl T, Hoffmann M, Christlein V, et al. Deep learning computed tomography: Learning projection-domain weights from image Domain in limited angle problems[J]. IEEE Transactions on Medical Imaging, 2018, 37(6): 1454-1463.
23	Fan Y C, Chen Y C, Chou S Y. Vivid-DIBR based 2D-3D image conversion system for 3D display[J]. Journal of Display Technology, 2017, 10(10): 887-898.
24	Philipp P, Felix V. Optimal approximation of piecewise smooth functions using deep ReLU neural networks[J]. Neural Networks, 2018, 108: 296-330.
25	Cai H, Yang Z, Cao X, et al. A new iterative triclass thresholding technique in image segmentation[J]. IEEE Transactions on Image Processing, 2014, 23(3): 1038-1046.
26	Hao G, Xu W. Image segmentation using Quantum-behaved partical swarm optimization algorithm[J]. Computer Engineering and Applications, 2007, 43(33): 831-835.
27	李长缨, 简元才, 杜广岑, 等. 青花菜耐贮性鉴定方法和标准[J]. 华北农学报, 1999, 14(4):134-136.
	Li C, Jian Y, Du G, et al. Appraising method and standard for storage durability in broccoli[J]. Acta Agriculturae Boreali-Sinica, 1999, 14(4): 134-136.
28	侯格贤, 吴成柯. 图像分割质量评价方法研究[J]. 中国图象图形学报, 2000, 5A(1): 39-43.
	Hou G, Wu C. Researches on evaluation methods for images segmentation[J]. Journal of Image and Graphics, 2000, 5A(1): 39-43.

分级	黄色区域占比（%）
Level 0	0
Level 1	<1
Level 2	<5
Level 3	5-10
Level 4	>10

指标	处理方法
指标	改进ResNet	ResNet	GoogleNet	CIVE	ExG
SSIM	0.911	0.778	0.762	0.771	0.802
Precision	0.897	0.797	0.659	0.789	0.801
Recall	0.908	0.801	0.712	0.821	0.793
F-measure	0.907	0.788	0.701	0.808	0.797

指标	西兰花品种
指标	浙青452	台绿1号	台绿2号
SSIM	0.865	0.856	0.824
Precision	0.825	0.894	0.899
Recall	0.838	0.841	0.873
F-measure	0.851	0.865	0.895

指标	晴天顺光（83张）	晴天逆光（86张）	阴天顺光（81张）	阴天逆光（94张）
SSIM	0.877	0.856	0.918	0.874
Precision	0.853	0.848	0.901	0.864
Recall	0.876	0.837	0.913	0.857
F-measure	0.895	0.859	0.915	0.873

[1]	LI Zhijun, YANG Shenghui, SHI Deshuai, LIU Xingxing, ZHENG Yongjun. Yield Estimation Method of Apple Tree Based on Improved Lightweight YOLOv5 [J]. Smart Agriculture, 2021, 3(2): 100-114.
[2]	FLORES Paulo, ZHANG Zhao. Wheat Lodging Ratio Detection Based on UAS Imagery Coupled with Different Machine Learning and Deep Learning Algorithms [J]. Smart Agriculture, 2021, 3(2): 23-34.
[3]	QIU Wenjie, YE Jin, HU Liangqing, YANG Juan, LI Qili, MO Jianyou, YI Wanmao. Distilled-MobileNet Model of Convolutional Neural Network Simplified Structure for Plant Disease Recognition [J]. Smart Agriculture, 2021, 3(1): 109-117.
[4]	LI Liangde, WANG Xiujuan, KANG Mengzhen, HUA Jing, FAN Menghan. Agricultural Named Entity Recognition Based on Semantic Aggregation and Model Distillation [J]. Smart Agriculture, 2021, 3(1): 118-128.
[5]	WEI Jing, WANG Yuting, YUAN Huizhu, ZHANG Menglei, WANG Zhenying. Identification and Morphological Analysis of Adult Spodoptera Frugiperda and Its Close Related Species Using Deep Learning [J]. Smart Agriculture, 2020, 2(3): 75-85.
[6]	FLORES Paulo, ZHANG Zhao, MATHEW Jithin, JAHAN Nusrat, STENGER John. Distinguishing Volunteer Corn from Soybean at Seedling Stage Using Images and Machine Learning [J]. Smart Agriculture, 2020, 2(3): 61-74.
[7]	GUO Wei, WU Huarui, ZHU Huaji. Design and Application of Facility Greenhouse Image Collecting and Environmental Data Monitoring Robot System [J]. Smart Agriculture, 2020, 2(3): 48-60.
[8]	LI Hualong, LI Miao, ZHAN Kai, LIU Xianwang, YANG Xuanjiang, HU Zelin, GUO Panpan. Construction Method and Performance Test of Prediction Model for Laying Hen Breeding Environmental Quality Evaluation [J]. Smart Agriculture, 2020, 2(3): 37-47.
[9]	Xia Xue, Sun Qixin, Shi Xiao, Chai Xiujuan. Apple detection model based on lightweight anchor-free deep convolutional neural network [J]. Smart Agriculture, 2020, 2(1): 99-110.
[10]	Wu Huarui. Method of tomato leaf diseases recognition method based on deep residual network [J]. Smart Agriculture, 2019, 1(4): 42-49.
[11]	Chen Guifen, Zhao Shan, Cao Liying, Fu Siwei, Zhou Jiaxin. Corn plant disease recognition based on migration learning and convolutional neural network [J]. Smart Agriculture, 2019, 1(2): 34-44.