Distilled-MobileNet Model of Convolutional Neural Network Simplified Structure for Plant Disease Recognition

doi:10.12133/j.smartag.2021.3.1.202009-SA004

Abstract

Abstract:

The development of convolutional neural networks(CNN) has brought a large number of network parameters and huge model volumes, which greatly limites the application on devices with small computing resources, such as single-chip microcomputers and mobile devices. In order to solve the problem, a structured model compression method was studied in this research. Its core idea was using knowledge distillation to transfer the knowledge from the complex integrated model to a lightweight small-scale neural network. Firstly, VGG16 was used to train a teacher model with a higher recognition rate, whose volume was much larger than the student model. Then the knowledge in the model was transfered to MobileNet by using distillation. The parameters number of the VGG16 model was greatly reduced. The knowledge-distilled model was named Distilled-MobileNet, and was applied to the classification task of 38 common diseases (powdery mildew, Huanglong disease, etc.) of 14 crops (soybean, cucumber, tomato, etc.). The performance test of knowledge distillation on four different network structures of VGG16, AlexNet, GoogleNet, and ResNet showed that when VGG16 was used as a teacher model, the accuracy of the model was improved to 97.54%. Using single disease recognition rate, average accuracy rate, model memory and average recognition time as 4 indicators to evaluate the accuracy of the trained Distilled-MobileNet model in a real environment, the results showed that, the average accuracy of the model reached 97.62%, and the average recognition time was shortened to 0.218 s, only accounts for 13.20% of the VGG16 model, and the model size was reduced to only 19.83 MB, which was 93.60% smaller than VGG16. Compared with traditional neural networks, distilled-mobile model has a significant improvement in reducing size and shorting recognition time, and can provide a new idea for disease recognition on devices with limited memory and computing resources.

Key words: disease identification, deep learning, model compression, knowledge distillation, convolutional neural network

CLC Number:

TP3-0

QIU Wenjie, YE Jin, HU Liangqing, YANG Juan, LI Qili, MO Jianyou, YI Wanmao. Distilled-MobileNet Model of Convolutional Neural Network Simplified Structure for Plant Disease Recognition[J]. Smart Agriculture, 2021, 3(1): 109-117.

Figures/Tables 14

Fig. 1

Fig. 2

Fig. 3

Table 1

Fig. 4

Fig. 5

Fig. 6

Table 2

Influence of the parameters α?and T?on the accuracy of the model validation set

				单位：%
$T$	$α$ =0.2	$α$ =0.4	$α$ =0.6	$α$ =0.8
$1$	83.59	84.93	89.06	79.86
2	84.37	82.81	87.50	77.34
3	81.25	85.15	90.62	80.46
4	75.40	82.13	85.49	83.21

Table 2

Table 3

CNN training hyperparameters using three optimizers

超参数	SGD	Adam	Nadam
Batch Size	32	32	32
Learning Rate	0.001	0.001	0.001
Momentum	0.9	—	—
Decay	None	None	None
$β 1$	—	0.9	0.9
$β 2$	—	0.999	0.999
$∈$	—	1e^-7	1e^-7

Table 3

Fig. 7

Table 4

Table 5

Fig. 8

Table 6

References 24

1	KAMILARIS A, PRENAFETA-BOLDÚ F X. A review of the use of convolutional neural networks in agriculture[J]. The Journal of Agricultural Science, 2018, 156(3): 312-322.
2	KARLEKAR A, SEAL A. SoyNet: Soybean leaf diseases classification[J]. Computers and Electronics in Agriculture, 2020, 172: ID 105342.
3	TETILA E C, MACHADO B B, MENEZES G K, et al. Automatic recognition of soybean leaf diseases Using UAV images and deep convolutional neural networks[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 17(5): 903-907.
4	ADEEL A, KHAN M A, AKRAM T, et al. Entropy controlled deep features selection framework for grape leaf diseases recognition[J]. Expert Systems, 2020(1): DOI: 10.1111/exsy.12569.
5	ZHANG X, QIAO Y, MENG F, et al. Identification of maize leaf diseases using improved deep convolutional neural networks[J]. IEEE Access, 2018(6): 30370-30377.
6	陈桂芬, 赵姗, 曹丽英, 等.基于迁移学习与卷积神经网络的玉米植株病害识别[J]. 智慧农业, 2019, 1(2): 34-44.
	CHEN G, ZHAO S, CAO L, et al. Corn plant disease recognition based on migration learning and convolutional neural network[J]. Smart Agriculture, 2019, 1(2): 34-44.
7	吴华瑞. 基于深度残差网络的番茄叶片病害识别方法[J]. 智慧农业, 2019, 1(4): 42-49.
	WU H. Method of tomato leaf diseases recognition method based on deep residual network[J]. Smart Agriculture, 2019, 1(4): 42-49.
8	雷小康, 尹志刚, 赵瑞莲. 基于FPGA的卷积神经网络定点加速[J]. 计算机应用, 2020, 40(10): 2811-2816.
	LEI X, YIN Z, ZHAO R. FPGA-based convolutional neural network fixed-point acceleration[J]. Computer Applications, 2020, 40(10): 2811-2816.
9	HAN R, LIU C, LI S, et al. Accelerating deep learning systems via critical set identification and model compression[J]. IEEE Transactions on Computers, 2020, 69(7): 1059-1070.
10	RAHMAN C R, ARKO P S, ALI M E, et al. Identification and recognition of rice diseases and pests using convolutional neural networks[EB/OL]. arXiv:1812.01
	043v3 [cs.CV] 4 Mar2020.
11	SHIH K H, CHIU C, LIN J A, et al. Real-time object detection with reduced region proposal network via multi-feature concatenation[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31: 2164-2173.
12	HC A, YL A, KCY B, et al. Block change learning for knowledge distillation[J]. Information Sciences, 2020, 513: 360-371.
13	WU M, CHIU C, WU K. Multi-teacher knowledge distillation for compressed video action recognition based on deep learning[C]// ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Piscataway, New York, USA: IEEE, 2019.
14	HUGHES D P, SALATHE M. An open access repository of images on plant health to enable the development of mobile disease diagnostics[EB/OL]. 2015. arXiv:1511.
	08060 [cs.CY].
15	HINTON G, VINYALS O, DEAN J. Distilling the knowledge in a neural network[EB/OL]. Computer Science, 2015. arXiv:1503.
	02531 [stat.ML].
16	SHORTEN C, KHOSHGOFTAAR T M. A survey on image data augmentation for deep learning[J]. Journal of Big Data, 2019, 6(1): ID 60.
17	RADFORD A, METZ L, CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks[EB/OL]. 2015. arXiv:1511.06434v
18	cs.LG] 19 Nov 2015.
19	WENG L, PRENEEL B. A secure perceptual hash algorithm for image content authentication[C]// 12th IFIP TC 6/TC 11 International Conference on Communications and Multimedia Security. Berlin Heidelberg, German: Springer Verlag, 2011: 108-121.
20	PAN S J, YANG Q. A survey on transfer learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2009, 22(10): 1345-1359.
21	HOWARD A G, ZHU M, CHEN B, et al. MobileNets: Efficient convolutional neural networks for mobile vision applications[EB/OL]. 2017. arXiv:1704.
	04861 [cs.CV].
22	KAILATH T. The divergence and Bhattacharyya distance measures in signal selection[J]. IEEE Transactions on Communication Technology, 1967, 15(1): 52-60.
23	ASHOK A, RHINEHART N, BEAINY F, et al. N2N learning: Network to network compression via policy Gradient Reinforcement Learning[C]// 6th International Conference on Learning Representations, ICLR 2018. Vancouver, BC, Canada: International Conference on Learning Representations, ICLR, 2018.
24	CHEBOTAR Y, WATERS A. Distilling knowledge from ensembles of neural networks for speech recognition[C]// Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. San Francisco, CA, United States: International Speech Communication Association, 2016: 3439-3433.

试验样本	图片数量/张
PlantVillage数据集	54,305
DCGAN样本扩充	5861
数据增强	76,639
自建数据集	9645

网络模型	验证集准确率/%	内存占用/MB
ResNet	94.44	181.3
AlexNet	89.97	443.0
VGG16	96.52	1022.0
MobileNet	92.05	25.1
GoogleNet	76.58	58.1

学生模型	教师模型	准确率/%		模型内存/MB		体积降低/%	识别时间/ ms
学生模型	教师模型	教师	学生	教师	学生	体积降低/%	教师	学生
MobileNet	VGG16	95.50	97.54	304.0	20.8	93.16	1635	261
	AlexNet	93.56	94.89	356.6	21.0	94.11	490	254
	GoogleNet	92.19	94.70	36.6	20.9	42.90	487	260
	ResNet	95.36	95.16	179.8	21.0	88.32	928	249
	平均	——	——	——	——	79.62	——	——

识别模型	单个病害识别率/%		平均准确率/%	内存占用/MB	平均识别时间/s	总识别时间/s
识别模型	芒果白粉病	芒果炭疽病	平均准确率/%	内存占用/MB	平均识别时间/s	总识别时间/s
VGG16	95.68	94.55	95.15	309.98	1.652	15,636
AlexNet	94.51	94.77	94.64	359.43	0.494	4675
GoogleNet	94.03	95.14	94.58	37.49	0.487	4609
ResNet	94.17	93.20	93.68	183.47	0.943	8925
Distilled-MobileNet	97.59	97.65	97.62	19.83	0.218	2063

[1]	SHAO Mingyue, ZHANG Jianhua, FENG Quan, CHAI Xiujuan, ZHANG Ning, ZHANG Wenrong. Research Progress of Deep Learning in Detection and Recognition of Plant Leaf Diseases [J]. Smart Agriculture, 2022, 4(1): 29-46.
[2]	YANG Guofeng, HE Yong, FENG Xuping, LI Xiyao, ZHANG Jinnuo, YU Zeyu. Methods and New Research Progress of Remote Sensing Monitoring of Crop Disease and Pest Stress Using Unmanned Aerial Vehicle [J]. Smart Agriculture, 2022, 4(1): 1-16.
[3]	ZHOU Qiaoli, MA Li, CAO Liying, YU Helong. Identification of Tomato Leaf Diseases Based on Improved Lightweight Convolutional Neural Networks MobileNetV3 [J]. Smart Agriculture, 2022, 4(1): 47-56.
[4]	WANG Ying, LI Yue, WU Tingting, SUN Shi, WANG Minjuan. Fast Counting Method of Soybean Seeds Based on Density Estimation and VGG-Two [J]. Smart Agriculture, 2021, 3(4): 111-122.
[5]	LI Zhijun, YANG Shenghui, SHI Deshuai, LIU Xingxing, ZHENG Yongjun. Yield Estimation Method of Apple Tree Based on Improved Lightweight YOLOv5 [J]. Smart Agriculture, 2021, 3(2): 100-114.
[6]	FLORES Paulo, ZHANG Zhao. Wheat Lodging Ratio Detection Based on UAS Imagery Coupled with Different Machine Learning and Deep Learning Algorithms [J]. Smart Agriculture, 2021, 3(2): 23-34.
[7]	LI Liangde, WANG Xiujuan, KANG Mengzhen, HUA Jing, FAN Menghan. Agricultural Named Entity Recognition Based on Semantic Aggregation and Model Distillation [J]. Smart Agriculture, 2021, 3(1): 118-128.
[8]	WEI Jing, WANG Yuting, YUAN Huizhu, ZHANG Menglei, WANG Zhenying. Identification and Morphological Analysis of Adult Spodoptera Frugiperda and Its Close Related Species Using Deep Learning [J]. Smart Agriculture, 2020, 2(3): 75-85.
[9]	FLORES Paulo, ZHANG Zhao, MATHEW Jithin, JAHAN Nusrat, STENGER John. Distinguishing Volunteer Corn from Soybean at Seedling Stage Using Images and Machine Learning [J]. Smart Agriculture, 2020, 2(3): 61-74.
[10]	GUO Wei, WU Huarui, ZHU Huaji. Design and Application of Facility Greenhouse Image Collecting and Environmental Data Monitoring Robot System [J]. Smart Agriculture, 2020, 2(3): 48-60.
[11]	LI Hualong, LI Miao, ZHAN Kai, LIU Xianwang, YANG Xuanjiang, HU Zelin, GUO Panpan. Construction Method and Performance Test of Prediction Model for Laying Hen Breeding Environmental Quality Evaluation [J]. Smart Agriculture, 2020, 2(3): 37-47.
[12]	Zhou Chengquan, Ye Hongbao, Yu Guohong, Hu Jun, Xu Zhifu. A fast extraction method of broccoli phenotype based on machine vision and deep learning [J]. Smart Agriculture, 2020, 2(1): 121-132.
[13]	Xia Xue, Sun Qixin, Shi Xiao, Chai Xiujuan. Apple detection model based on lightweight anchor-free deep convolutional neural network [J]. Smart Agriculture, 2020, 2(1): 99-110.
[14]	Wu Huarui. Method of tomato leaf diseases recognition method based on deep residual network [J]. Smart Agriculture, 2019, 1(4): 42-49.
[15]	Chen Guifen, Zhao Shan, Cao Liying, Fu Siwei, Zhou Jiaxin. Corn plant disease recognition based on migration learning and convolutional neural network [J]. Smart Agriculture, 2019, 1(2): 34-44.