HI-FPN: A Hierarchical Interactive Feature Pyramid Network for Accurate Wheat Lodging Localization Across Multiple Growth Periods

doi:10.12133/j.smartag.SA202310002

Smart Agriculture ›› 2024, Vol. 6 ›› Issue (2): 128-139.doi: 10.12133/j.smartag.SA202310002

• Special Issue--Agricultural Information Perception and Models • Previous Articles Next Articles

HI-FPN: A Hierarchical Interactive Feature Pyramid Network for Accurate Wheat Lodging Localization Across Multiple Growth Periods

PANG Chunhui¹^,⁶^,⁷, CHEN Peng¹^,⁶^,⁷(), XIA Yi¹, ZHANG Jun¹, WANG Bing², ZOU Yan³^,⁴, CHEN Tianjiao³^,⁴, KANG Chenrui³^,⁵, LIANG Dong¹()

^1. National Engineering Research Center for Agro-Ecological Big Data Analysis & Application/ Information Materials and Intelligent Sensing Laboratory of Anhui Province/ Institutes of Physical Science and Information Technology & School of Internet, Anhui University, Hefei 230601, China
^2. School of Management Science and Engineering, Anhui University of Finance & Economics, Bengbu 233030, China
^3. Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Science, Hefei 230031, China
^4. University of Science and Technology of China, Hefei 230031, China
^5. Southwest University of Science and Technology, Mianyang 621010, China
^6. Agricultural Sensors and Intelligent Perception Technology Innovation Center of Anhui Province, Zhongke Hefei Institutes of Collaborative Research and Innovation for Intelligent Agriculture, Hefei 231131, China
^7. Anhui Rocvision Intelligent Technology Co. , Ltd, Hefei 230000, China

Received:2023-10-03 Online:2024-03-30
Foundation items:
National Natural Science Foundation of China Projects(62072002;62273001); Anhui Provincial Major Science and Technology Special Project(202003a06020016); Supported by the Special Fund for Anhui Agriculture Research System (2021-2025); Excellent Scientific Research Innovation Team of Anhui Province Universities(2022AH010005)
About author:
PANG Chunhui, research interests is computer vision. E-mail: huihui_xiaozi@163.com
corresponding author:
1. CHEN Peng, Ph.D., Professor, research interests are computer vision and data analysis. E-mail: pengchen@ustc.edu;
2. LIANG Dong, Ph.D., Professor, research interests are computer vision and smart agriculture. E-mail: dliang@ahu.edu.cn

Abstract

Abstract:

[Objective] Wheat lodging is one of the key isuess threatening stable and high yields. Lodging detection technology based on deep learning generally limited to identifying lodging at a single growth stage of wheat, while lodging may occur at various stages of the growth cycle. Moreover, the morphological characteristics of lodging vary significantly as the growth period progresses, posing a challenge to the feature capturing ability of deep learning models. The aim is exploring a deep learning-based method for detecting wheat lodging boundaries across multiple growth stages to achieve automatic and accurate monitoring of wheat lodging. [Methods] A model called Lodging2Former was proposed, which integrates the innovative hierarchical interactive feature pyramid network (HI-FPN ) on top of the advanced segmentation model Mask2Former. The key focus of this network design lies in enhancing the fusion and interaction between feature maps at adjacent hierarchical levels, enabling the model to effectively integrate feature information at different scales. Building upon this, even in complex field backgrounds, the Lodging2Former model significantly enhances the recognition and capturing capabilities of wheat lodging features at multiple growth stages. [Results and Discussions] The Lodging2Former model demonstrated superiority in mean average precision (mAP) compared to several mainstream algorithms such as mask region-based convolutional neural network (Mask R-CNN), segmenting objects by locations (SOLOv2), and Mask2Former. When applied to the scenario of detecting lodging in mixed growth stage wheat, the model achieved mAP values of 79.5%, 40.2%, and 43.4% at thresholds of 0.5, 0.75, and 0.5 to 0.95, respectively. Compared to Mask2Former, the performance of the improved model was enhanced by 1.3% to 4.3%. Compared to SOLOv2, a growth of 9.9% to 30.7% in mAP was achieved; and compared to the classic Mask R-CNN, a significant improvement of 24.2% to 26.4% was obtained. Furthermore, regardless of the IoU threshold standard, the Lodging2Former exhibited the best detection performance, demonstrating good robustness and adaptability in the face of potential influencing factors such as field environment changes. [Conclusions] The experimental results indicated that the proposed HI-FPN network could effectively utilize contextual semantics and detailed information in images. By extracting rich multi-scale features, it enabled the Lodging2Former model to more accurately detect lodging areas of wheat across different growth stages, confirming the potential and value of HI-FPN in detecting lodging in multi-growth-stage wheat.

Key words: drone, deep learning, wheat lodging detection, feature pyramid network, Mask2Former

CLC Number:

TP274

PANG Chunhui, CHEN Peng, XIA Yi, ZHANG Jun, WANG Bing, ZOU Yan, CHEN Tianjiao, KANG Chenrui, LIANG Dong. HI-FPN: A Hierarchical Interactive Feature Pyramid Network for Accurate Wheat Lodging Localization Across Multiple Growth Periods[J]. Smart Agriculture, 2024, 6(2): 128-139.

Figures/Tables 11

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Table 1

Fig. 9

Fig. 10

References 27

1	WEGREN S K. Challenges to global food security: A policy approach to the 2021–2022 food crisis[J]. World food policy, 2023, 9( 1): 127- 148.
2	KISZONAS A M, MORRIS C F. Wheat breeding for quality: A historical review[J]. Cereal chemistry, 2018, 95( 1): 17- 34.
3	WU W, MA B L. A new method for assessing plant lodging and the impact of management options on lodging in canola crop production[J]. Scientific reports, 2016, 6: ID 31890.
4	BERRY P M, SPINK J. Predicting yield losses caused by lodging in wheat[J]. Field crops research, 2012, 137: 19- 26.
5	GULATI A, TERWAY P, HUSSAIN S. Crop insurance in India: Key issues and way forward[R/OL]. Working paper, 2018. [ 2023-09-20].
6	SHAH L, YAHYA M, SHAH S M A, et al. Improving lodging resistance: using wheat and rice as classical examples[J]. International journal of molecular sciences, 2019, 20( 17): ID 4211.
7	ZHANG H S, LIN H, LI Y, et al. Mapping urban impervious surface with dual-polarimetric SAR data: An improved method[J]. Landscape and urban planning, 2016, 151: 55- 63.
8	ZHANG Z, FLORES P, IGATHINATHANE C, et al. Wheat lodging detection from UAS imagery using machine learning algorithms[J]. Remote sensing, 2020, 12( 11): ID 1838.
9	LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International journal of computer vision, 2004, 60( 2): 91- 110.
10	KITANO B T, MENDES C C T, GEUS A R, et al. Corn plant counting using deep learning and UAV images[J]. IEEE geoscience and remote sensing letters, 2024: 1- 5.
11	REN S, HE K, GIRSHICK R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE trans pattern anal Mach intell, 2017, 39( 6): 1137- 1149.
12	JIANG P Y, ERGU D J, LIU F Y, et al. A review of yolo algorithm developments[J]. Procedia computer science, 2022, 199: 1066- 1073.
13	SAMET N, HICSONMEZ S, AKBAS E. Reducing label noise in anchor-free object detection[EB/OL]. arXiv: 2008.01167, 2020.
14	YANG B H, ZHU Y, ZHOU S J. Accurate wheat lodging extraction from multi-channel UAV images using a lightweight network model[J]. Sensors, 2021, 21( 20): ID 6826.
15	ZHAO X, YUAN Y, SONG M, et al. Use of unmanned aerial vehicle imagery and deep learning unet to extract rice lodging[J]. Sensors, 2019, 19( 18): ID 3859.
16	SU Z B, WANG Y, XU Q, et al. LodgeNet: Improved rice lodging recognition using semantic segmentation of UAV high-resolution remote sensing images[J]. Computers and electronics in agriculture, 2022, 196: ID 106873.
17	ZHANG D Y, DING Y, CHEN P F, et al. Automatic extraction of wheat lodging area based on transfer learning method and deeplabv3+ network[J]. Computers and electronics in agriculture, 2020, 179: ID 105845.
18	LIN T Y, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, New Jersey, USA: IEEE, 2017: 2117- 2125.
19	RONNEBERGER O, FISCHER P, BROX T. U-net: Convolutional networks for biomedical image segmentation[C]// In: Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference. Munich, Germany: Springer 2015: 234- 241.
20	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// 31st Conference on Neural Information Processing Systems (NIPS 2017). Long Beach, USA: NIPS, 2017.
21	DAI Q F, GUO Y H, LI Z, et al. Citrus disease image generation and classification based on improved FastGAN and EfficientNet-B5[J]. Agronomy, 2023, 13( 4): ID 988.
22	ZHANG L, DU J M, DONG S F, et al. AM-ResNet: Low-energy-consumption addition-multiplication hybrid ResNet for pest recognition[J]. Computers and electronics in agriculture, 2022, 202: ID 107357.
23	HE K M, GKIOXARI G, DOLLAR P, et al. Mask R-CNN[C]// 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway, New Jersey, USA: IEEE, 2017: 2961- 2969.
24	WANG X, ZHANG R, KONG T, et al. Solov2: Dynamic and fast instance segmentation[C]// 34th Conference on Neural Information Processing Systems (NeurIPS 2020). Vancouver, Canada: NeurIPS, 2020.
25	CHENG B W, MISRA I, SCHWING A G, et al. Masked-attention mask transformer for universal image segmentation[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, New Jersey, USA: IEEE, 2022: 1290- 1299.
26	SINGH D, WANG X, KUMAR U, et al. High-throughput phenotyping enabled genetic dissection of crop lodging in wheat[J]. Frontiers in plant science, 2019, 10: ID 412524.
27	YU J, CHENG T, CAI N, et al. Wheat lodging segmentation based on Lstm_PSPNet deep learning network[J]. Drones, 2023, 7( 2): ID 143.

Growth Stages	Method	mIoU			mAP
Growth Stages	Method	0.5	0.2	0.01	0.5	0.75	0.5：0.95
Multiple stages	Mask-RCNN	54.2	57.1	57.0	54.4	13.8	19.2
	SOLOV2	34.9	54.4	58.4	69.6	9.5	25.0
	Mask2Former	63.7	63.7	63.7	78.2	36.2	39.1
	Lodging2Former	66.0	66.0	68.7	79.5	40.2	43.4
Grain-filling stage	Mask-RCNN	64.9	65.1	66.8	60.4	6.9	16.0
	SOLOV2	61.2	61.2	62.6	80.5	5.2	25.4
	Mask2Former	72.4	72.4	72.4	95.6	60.4	48.4
	Lodging2Former	73.0	73.0	73.0	97.4	60.8	50.2
Early maturity stage	Mask-RCNN	78.4	79.8	79.8	80.5	31.0	35.7
	SOLOV2	48.7	67.1	71.1	89.0	26.6	43.7
	Mask2Former	76.5	76.5	76.5	94.7	60.1	59.0
	Lodging2Former	81.4	81.4	81.4	96.6	60.2	59.1
Late maturity stage	Mask-RCNN	43.8	45.1	45.6	37.9	2.6	9.8
	SOLOV2	36.4	36.4	37.8	41.7	3.8	15.6
	Mask2Former	45.8	45.8	45.8	53.5	11.8	20.0
	Lodging2Former	47.8	47.8	47.8	53.8	11.8	20.6

[1]	LUO Youlu, PAN Yonghao, XIA Shunxing, TAO Youzhi. Lightweight Apple Leaf Disease Detection Algorithm Based on Improved YOLOv8 [J]. Smart Agriculture, 2024, 6(5): 128-138.
[2]	LIU Yi, ZHANG Yanjun. ReluformerN: Lightweight High-Low Frequency Enhanced for Hyperspectral Agricultural Lancover Classification [J]. Smart Agriculture, 2024, 6(5): 74-87.
[3]	NIAN Yue, ZHAO Kaixuan, JI Jiangtao. Cow Hoof Slippage Detecting Method Based on Enhanced DeepLabCut Model [J]. Smart Agriculture, 2024, 6(5): 153-163.
[4]	ZHANG Yanqi, ZHOU Shuo, ZHANG Ning, CHAI Xiujuan, SUN Tan. A Regional Farming Pig Counting System Based on Improved Instance Segmentation Algorithm [J]. Smart Agriculture, 2024, 6(4): 53-63.
[5]	WENG Zhi, FAN Qi, ZHENG Zhiqiang. Automatic Measurement Method of Beef Cattle Body Size Based on Multimodal Image Information and Improved Instance Segmentation Network [J]. Smart Agriculture, 2024, 6(4): 64-75.
[6]	HOU Yiting, RAO Yuan, SONG He, NIE Zhenjun, WANG Tan, HE Haoxu. A Rapid Detection Method for Wheat Seedling Leaf Number in Complex Field Scenarios Based on Improved YOLOv8 [J]. Smart Agriculture, 2024, 6(4): 128-137.
[7]	LI Hao, DU Yuqiu, XIAO Xingzhu, CHEN Yanxi. Remote Sensing Identification Method of Cultivated Land at Hill County of Sichuan Basin Based on Deep Learning [J]. Smart Agriculture, 2024, 6(3): 34-45.
[8]	NIE Ganggang, RAO Honghui, LI Zefeng, LIU Muhua. Severity Grading Model for Camellia Oleifera Anthracnose Infection Based on Improved YOLACT [J]. Smart Agriculture, 2024, 6(3): 138-147.
[9]	ZHANG Jing, ZHAO Zexuan, ZHAO Yanru, BU Hongchao, WU Xingyu. Oilseed Rape Sclerotinia in Hyperspectral Images Segmentation Method Based on Bi-GRU and Spatial-Spectral Information Fusion [J]. Smart Agriculture, 2024, 6(2): 40-48.
[10]	ZHANG Yuyu, BING Shuying, JI Yuanhao, YAN Beibei, XU Jinpu. Grading Method of Fresh Cut Rose Flowers Based on Improved YOLOv8s [J]. Smart Agriculture, 2024, 6(2): 118-127.
[11]	ZHANG Jianhua, YAO Qiong, ZHOU Guomin, WU Wendi, XIU Xiaojie, WANG Jian. Intelligent Identification of Crop Agronomic Traits and Morphological Structure Phenotypes: A Review [J]. Smart Agriculture, 2024, 6(2): 14-27.
[12]	GUO Wang, YANG Yusen, WU Huarui, ZHU Huaji, MIAO Yisheng, GU Jingqiu. Big Models in Agriculture: Key Technologies, Application and Future Directions [J]. Smart Agriculture, 2024, 6(2): 1-13.
[13]	ZHOU Huamao, WANG Jing, YIN Hua, CHEN Qi. Phenotype Analysis of Pleurotus Geesteranus Based on Improved Mask R-CNN [J]. Smart Agriculture, 2023, 5(4): 117-126.
[14]	WANG Herong, CHEN Yingyi, CHAI Yingqian, XU Ling, YU Huihui. Image Segmentation Method Combined with VoVNetv2 and Shuffle Attention Mechanism for Fish Feeding in Aquaculture [J]. Smart Agriculture, 2023, 5(4): 137-149.
[15]	LI Zhengkai, YU Jiahui, PAN Shijia, JIA Zefeng, NIU Zijie. Individual Tree Skeleton Extraction and Crown Prediction Method of Winter Kiwifruit Trees [J]. Smart Agriculture, 2023, 5(4): 92-104.

HI-FPN: A Hierarchical Interactive Feature Pyramid Network for Accurate Wheat Lodging Localization Across Multiple Growth Periods

RichHTML

CNKI Download
(Recommend)

PDF (PC)