基于机器视觉与深度学习的西兰花表型快速提取方法研究

doi:10.12133/j.smartag.2020.2.1.201912-SA003

摘要/Abstract

摘要：

准确获取西兰花花球面积和新鲜度是确定其长势的关键步骤，本研究通过对深度残差网络ResNet进行改进得到一种新型的西兰花花球分割模型，并通过花球部位黄绿颜色占比判断其新鲜度，实现低成本高效准确地西兰花表型信息提取。主要技术流程包括：（1）基于地面自动影像获取平台拍摄西兰花花球正射影像并建立原始数据集；（2）对训练图像进行预处理并输入模型进行分割；（3）基于颜色信息用粒子群结构PSO和大津法Otsu对分割结果进一步进行阈值分割，获取其新鲜度指标。试验结果表明：本研究建立的分割模型精度优于传统深度学习模型和基于颜色空间变换和阈值分割模型，4个评价指标结构相似性指数(SSIM)、平均精度(Precision)、平均召回率(Recall)、F-度量(F-measure)结果分别为0.911、0.897、0.908和0.907，相比于传统方法提升了10%-15%，且对土壤反射率波动、冠层阴影、辐射强度变化等干扰具有一定的鲁棒性。同时，在分割结果的基础上采用PSO-Otsu法可以实现花球新鲜度快速分析，其精度超过了0.8。本研究结果实现了西兰花田间多表型参数的高通量获取，可以为作物田间长势监测研究提供重要参考。

关键词: 深度学习, 西兰花表型, 机器视觉, 自动分级, 田间平台

Abstract:

How to accurately obtain the area and freshness of broccoli head in the field condition is the key step to determine broccoli growth. However, the rapid segmentation and grading of broccoli ball remains difficult due to the low equipment development level. In this research, we combined an advanced computer vision technique with a deep learning architecture to allow the acquisition of real-time and accurate information about broccoli ball. By constructing a private image dataset with 442 of broccoli-ball images (acquired using a self-developed imaging system) under controlled conditions, a deep convolutional neural network named “Improved ResNet” was trained to extract the broccoli pixels from the background. The technical process of our method includes: (1) take the orthophoto images of broccoli head based on a near-ground image acquisition platform and establish the original data set; (2) preprocess the training images and input the model for segmentation; (3) use the PSOA and Otsu algorithm for fine segment based on color characteristics to obtain the freshness information. The experimental results demonstrated that the precision of the segmentation model is about 0.9 which is robust to the interference of soil reflectance fluctuation, canopy shadow, leaf occlusion and so on. Our experiments showed that a combination of improved ResNet and PSOA method got higher broccoli balls segmenting and grading precision. One major advantage of this approach is that dealing with only a few images, reducing the data volume and memory requirements for the image processing. All of the methods were evaluated using ground-truth data from three different varieties, which we also make available to the research community for subsequent algorithm development and result comparison. Compared with other 4 approaches, the evaluation results shows better performance regarding the segmentation and grading accuracy. The results of SSIM, precision, recall and F-measure by using Improved ResNet were about 0.911, 0.897, 0.908 and 0.907 respectively, which were 10%~15% higher than the traditional approaches. In addition, on the basis of the segmentation results, PSO-Otsu method was proved that it can be used to achieve a quickly analysis to the freshness of the ball, with the mean accuracy of 0.82. Overall, the proposed method is a high-throughput method to acquire multi-phenotype parameters of broccoli in field condition, which can support the research of broccoli field monitoring and traits tracking.

Key words: deep learning, broccoli phenotype, computer vision, automatic grading, field platform

中图分类号:

周成全, 叶宏宝, 俞国红, 胡俊, 徐志福. 基于机器视觉与深度学习的西兰花表型快速提取方法研究[J]. 智慧农业(中英文), 2020, 2(1): 121-132.

Zhou Chengquan, Ye Hongbao, Yu Guohong, Hu Jun, Xu Zhifu. A fast extraction method of broccoli phenotype based on machine vision and deep learning[J]. Smart Agriculture, 2020, 2(1): 121-132.

图/表 12

图1

图2

图3

图4

表1

图5

表2

图6

表3

图7

表4

图8

参考文献 28

1	郭香凤, 向进乐, 李秀珍, 等. 贮藏温度对西兰花净菜品质的影响[J]. 农业机械学报, 2008, 39(2): 201-204.
2	夏永泉, 李耀斌, 曾莎. 基于HSI颜色空间的植物叶片病斑提取方法[J]. 江苏农业科学, 2015(08): 420-422.
3	高理文, 林小桦. 基于Lab*彩色空间和局域动态阈值的药用植物叶片图像分割[J]. 计算机应用与软件, 2014， (1): 232-235.
	Gao L, Lin X. Segmentation of images of medicinal plant leaves based on Lab* colour space and local dynamic threshold[J]. Computer Applications and Software, 2014， (1): 232-235.
4	董晓辉. 基于多彩色空间的麦田监控图像分割技术研究[D]. 开封：河南大学， 2015.
	Dong X. Wheat mornitoring image segmatation technology using variety of color spaces[D]. Kaifeng: Henan Agricultural University, 2015.
5	魏丽冉, 岳峻, 李振波, 等. 基于核函数支持向量机的植物叶部病害多分类检测方法[J]. 农业机械学报, 2017, (S0): 166-171.
	Wei L, Yue J, Li Z, et al. Multi-classification detection method of plant leaf disease based on kernel function SVM[J]. Transactions of the CSAM, 2017, (S0): 166-171.
6	周俊, 刘丽川, 杨继平. 基于K-均值聚类与小波分析的声发射信号去噪[J]. 石油化工高等学校学报, 2013, 26(3): 69-73.
	Zhou J, Liu L, Yang J. Acoustic emission signal denoising based on K-means clustering and wavelet analysis[J]. Journal of Petrochemical Universities, 2013, 26(3): 69-73.
7	李先锋, 朱伟兴, 纪滨, 等. 基于图像处理和蚁群优化的形状特征选择与杂草识别[J]. 农业工程学报, 2010, 26(10): 178-182.
	Li X, Zhu W, Ji B, et al. Shape feature selection and weed recognition based on image processing and ant colony optimization[J]. Transactions of the CSAE, 2010, 26(10): 178-182.
8	袁定帅, 陈洁, 赖晓芳, 等. 不同贮藏条件对西兰花感官品质及抗氧化物质的影响[J]. 食品科技, 2017， (4): 40-45.
	Yuan D, Chen J, Lai X, et al. Effects of diifferent storage conditions on sensory quality and antioxidants of broccoli[J]. Food Science and Techonology, 2017， (4): 40-45.
9	Lee S H, Chan C S, Wilkin P, et al. Deep-plant: Plant identification with convolutional neural networks[C]// IEEE International Conference on Image Processing (ICIP), 2015: 452-456.
10	Dyrmann M, Karstoft H, Midtiby H S. Plant species classification using deep convolutional neural network[J]. Biosystems Engineering, 2016, 151: 72-80.
11	Too E C, Li Y, Njuki S, et al. A comparative study of fine-tuning deep learning models for plant disease identification[J]. Computers and Electronics in Agriculture, 2018, 161: 272-279.
12	Fuentes A, Yoon S, Kim S C, et al. A robust deep-learning-based detector for real-time tomato plant diseases and pests recognition[J]. Sensors, 2017, 17: 2022-2042.
13	李淼, 王敬贤, 李华龙, 等. 基于 CNN 和迁移学习的农作物病害识别方法研究[J]. 智慧农业, 2019, 1(3): 46-55.
	Li M, Wang J, Li H, et al. Method for identifying crop disease based on CNN and transfer learning[J]. Smart Agriculture, 2019, 1(3): 46-55.
14	陈桂芬, 赵姗, 曹丽英, 等. 基于迁移学习与卷积神经网络的玉米植株病害识别[J]. 智慧农业, 2019, 1(2): 34-44.
	Chen G, Zhao S, Cao L, et al. Corn plant disease recognition based on migration learning and convolutional neural network[J]. Smart Agriculture, 2019, 1(2): 34-44.
15	Fu Z, Robles-Kelly A. A quadratic programming approach to image labelling[J]. IET Computer Vision, 2009, 2(4): 193-207.
16	刘万军, 梁雪剑, 曲海成. 不同池化模型的卷积神经网络学习性能研究[J]. 中国图象图形学报, 2016, 21(9): 1178-1190.
	Liu W, Liang X, Qu H. Learning performance of convolutional neural networks with different pooling models[J]. Journal of Image and Graphics, 2016, 21(9): 1178-1190.
17	李林, 顾进锋, 宋安捷, 等. 基于GPU的生态环境遥感评价模型并行化研究[J]. 农业机械学报, 2017(5): 140-146.
	Li L, Gu J, Song A, et al. Parallelization on model of ecological environment remote sensing evaluation based GPU[J]. Transactions of the CSAM, 2017(5): 140-146.
18	He K, Zhang X, Ren S, et al. Identity mappings in deep residual networks[J]. Computer Science, 2016: 630-645.
19	Wang Y, Yang A, Chen X, et al. A deep learning approach for blind drift calibration of sensor networks[J]. IEEE Sensors Journal, 2017, 17(13): 4158-4171.
20	Gong L, Jiang S, Yang Z, et al. Automated pulmonary nodule detection in CT images using 3D deep squeeze-and-excitation networks[J]. International Journal of Computer Assisted Radiology and Surgery, 2019, 14(11): 1969-1979.
21	Zhou X, Jin K, Shang Y, et al. Visually interpretable representation learning for depression recognition from facial images[J]. IEEE Transactions on Affective Computing, 2018, DOI: 10.1109/TAFFC.2018.2828819. doi: 10.1109/TAFFC.2018.2828819
22	Wurfl T, Hoffmann M, Christlein V, et al. Deep learning computed tomography: Learning projection-domain weights from image Domain in limited angle problems[J]. IEEE Transactions on Medical Imaging, 2018, 37(6): 1454-1463.
23	Fan Y C, Chen Y C, Chou S Y. Vivid-DIBR based 2D-3D image conversion system for 3D display[J]. Journal of Display Technology, 2017, 10(10): 887-898.
24	Philipp P, Felix V. Optimal approximation of piecewise smooth functions using deep ReLU neural networks[J]. Neural Networks, 2018, 108: 296-330.
25	Cai H, Yang Z, Cao X, et al. A new iterative triclass thresholding technique in image segmentation[J]. IEEE Transactions on Image Processing, 2014, 23(3): 1038-1046.
26	Hao G, Xu W. Image segmentation using Quantum-behaved partical swarm optimization algorithm[J]. Computer Engineering and Applications, 2007, 43(33): 831-835.
27	李长缨, 简元才, 杜广岑, 等. 青花菜耐贮性鉴定方法和标准[J]. 华北农学报, 1999, 14(4):134-136.
	Li C, Jian Y, Du G, et al. Appraising method and standard for storage durability in broccoli[J]. Acta Agriculturae Boreali-Sinica, 1999, 14(4): 134-136.
28	侯格贤, 吴成柯. 图像分割质量评价方法研究[J]. 中国图象图形学报, 2000, 5A(1): 39-43.
	Hou G, Wu C. Researches on evaluation methods for images segmentation[J]. Journal of Image and Graphics, 2000, 5A(1): 39-43.

分级	黄色区域占比（%）
Level 0	0
Level 1	<1
Level 2	<5
Level 3	5-10
Level 4	>10

指标	处理方法
指标	改进ResNet	ResNet	GoogleNet	CIVE	ExG
SSIM	0.911	0.778	0.762	0.771	0.802
Precision	0.897	0.797	0.659	0.789	0.801
Recall	0.908	0.801	0.712	0.821	0.793
F-measure	0.907	0.788	0.701	0.808	0.797

指标	西兰花品种
指标	浙青452	台绿1号	台绿2号
SSIM	0.865	0.856	0.824
Precision	0.825	0.894	0.899
Recall	0.838	0.841	0.873
F-measure	0.851	0.865	0.895

指标	晴天顺光（83张）	晴天逆光（86张）	阴天顺光（81张）	阴天逆光（94张）
SSIM	0.877	0.856	0.918	0.874
Precision	0.853	0.848	0.901	0.864
Recall	0.876	0.837	0.913	0.857
F-measure	0.895	0.859	0.915	0.873

[1]	李志军, 杨圣慧, 史德帅, 刘星星, 郑永军. 基于轻量化改进YOLOv5的苹果树产量测定方法[J]. 智慧农业（中英文）, 2021, 3(2): 100-114.
[2]	. 基于无人机图像以及不同机器学习和深度学习模型的小麦倒伏率检测[J]. 智慧农业（中英文）, 2021, 3(2): 23-34.
[3]	邱文杰, 叶进, 胡亮青, 杨娟, 李其利, 莫贱友, 易万茂. 面向植物病害识别的卷积神经网络精简结构Distilled-MobileNet模型[J]. 智慧农业（中英文）, 2021, 3(1): 109-117.
[4]	李亮德, 王秀娟, 康孟珍, 华净, 樊梦涵. 基于语义融合与模型蒸馏的农业实体识别[J]. 智慧农业（中英文）, 2021, 3(1): 118-128.
[5]	韩书庆, 张晶, 程国栋, 彭英琦, 张建华, 吴建寨. 奶牛跛行自动识别技术研究现状与挑战[J]. 智慧农业（中英文）, 2020, 2(3): 21-36.
[6]	郭威, 吴华瑞, 朱华吉. 设施温室影像采集与环境监测机器人系统设计及应用[J]. 智慧农业（中英文）, 2020, 2(3): 48-60.
[7]	李华龙, 李淼, 詹凯, 刘先旺, 杨选将, 胡泽林, 郭盼盼. 蛋鸡设施养殖环境质量评价预测模型构建方法及性能测试[J]. 智慧农业（中英文）, 2020, 2(3): 37-47.
[8]	魏靖, 王玉亭, 袁会珠, 张梦蕾, 王振营. 基于深度学习与特征可视化方法的草地贪夜蛾及其近缘种成虫识别[J]. 智慧农业（中英文）, 2020, 2(3): 75-85.
[9]	FLORESPaulo, MATHEWJithin, JAHANNusrat, STENGERJohn. 利用图像和机器学习检测大豆作物幼苗期玉米杂苗[J]. 智慧农业（中英文）, 2020, 2(3): 61-74.
[10]	夏雪, 孙琦鑫, 侍啸, 柴秀娟. 基于轻量级无锚点深度卷积神经网络的树上苹果检测模型[J]. 智慧农业（中英文）, 2020, 2(1): 99-110.
[11]	吴华瑞. 基于深度残差网络的番茄叶片病害识别方法[J]. 智慧农业, 2019, 1(4): 42-49.
[12]	陈桂芬, 赵姗, 曹丽英, 傅思维, 周佳鑫. 基于迁移学习与卷积神经网络的玉米植株病害识别[J]. 智慧农业, 2019, 1(2): 34-44.
[13]	周超, 徐大明, 吝凯, 陈澜, 张松, 孙传恒, 杨信廷. 基于近红外机器视觉的鱼类摄食强度评估方法研究[J]. 智慧农业, 2019, 1(1): 76-84.