A Multi-Focal Green Plant Image Fusion Method Based on Stationary Wavelet Transform and Parameter-Adaptation Dual Channel Pulse-Coupled Neural Network

doi:10.12133/j.smartag.SA202308005

Abstract

Abstract:

[Objective] To construct the 3D point cloud model of green plants a large number of clear images are needed. Due to the limitation of the depth of field of the lens, part of the image would be out of focus when the green plant image with a large depth of field is collected, resulting in problems such as edge blurring and texture detail loss, which greatly affects the accuracy of the 3D point cloud model. However, the existing processing algorithms are difficult to take into account both processing quality and processing speed, and the actual effect is not ideal. The purpose of this research is to improve the quality of the fused image while taking into account the processing speed. [Methods] A plant image fusion method based on non-subsampled shearlet transform (NSST) based parameter-adaptive dual channel pulse-coupled neural network (PADC-PCNN) and stationary wavelet transform (SWT) was proposed. Firstly, the RGB image of the plant was separated into three color channels, and the G channel with many features such as texture details was decomposed by NSST in four decomposition layers and 16 directions, which was divided into one group of low frequency subbands and 64 groups of high frequency subbands. The low frequency subband used the gradient energy fusion rule, and the high frequency subband used the PADC-PCNN fusion rule. In addition, the weighting of the eight-neighborhood modified Laplacian operator was used as the link strength of the high-frequency fusion part, which enhanced the fusion effect of the detailed features. At the same time, for the R and B channels with more contour information and background information, a SWT with fast speed and translation invariance was used to suppress the pseudo-Gibbs effect. Through the high-precision and high-stability multi-focal length plant image acquisition system, 480 images of 8 experimental groups were collected. The 8 groups of data were divided into an indoor light group, natural light group, strong light group, distant view group, close view group, overlooking group, red group, and yellow group. Meanwhile, to study the application range of the algorithm, the focus length of the collected clear plant image was used as the reference (18 mm), and the image acquisition was adjusted four times before and after the step of 1.5 mm, forming the multi-focus experimental group. Subjective evaluation and objective evaluation were carried out for each experimental group to verify the performance of the algorithm. Subjective evaluation was analyzed through human eye observation, detail comparison, and other forms, mainly based on the human visual effect. The image fusion effect of the algorithm was evaluated using four commonly used objective indicators, including average gradient (AG), spatial frequency (SF), entropy (EN), and standard deviation (SD). [Results and Discussions] The proposed PADC-PCNN-SWT algorithm and other five algorithms of common fast guided filtering algorithm (FGF), random walk algorithm (RW), non-subsampled shearlet transform based PCNN (NSST-PCNN) algorithm, SWT algorithm and non-subsampled shearlet transform based parameter-adaptive dual-channel pulse-coupled neural network (NSST-PADC) and were compared. In the objective evaluation data except for the red group and the yellow group, each index of the PADC-PCNN-SWT algorithm was second only to the NSST-PADC algorithm, but the processing speed was 200.0% higher than that of the NSST-PADC algorithm on average. At the same time, compared with the FDF, RW, NSST-PCNN, and SWT algorithms, the PADC-PCN -SWT algorithm improved the clarity index by 5.6%, 8.1%, 6.1%, and 17.6%, respectively, and improved the spatial frequency index by 2.9%, 4.8%, 7.1%, and 15.9%, respectively. However, the difference between the two indicators of information entropy and standard deviation was less than 1%, and the influence was ignored. In the yellow group and the red group, the fusion quality of the non-green part of the algorithm based on PADC-PCNN-SWT was seriously degraded. Compared with other algorithms, the sharpness index of the algorithm based on PADC-PCNN-SWT decreased by an average of 1.1%, and the spatial frequency decreased by an average of 5.1%. However, the indicators of the green part of the fused image were basically consistent with the previous several groups of experiments, and the fusion effect was good. Therefore, the algorithm based on PADC-PCNN-SWT only had a good fusion effect on green plants. Finally, by comparing the quality of four groups of fused images with different focal length ranges, the results showed that the algorithm based on PADC-PCNN-SWT had a better contour and color restoration effect for out-of-focus images in the range of 15-21 mm, and the focusing range based on PADC-PCNN-SWT was about 6 mm. [Conclusions] The multi-focal length image fusion algorithm based on PADC-PCNN-SWT achieved better detail fusion performance and higher image fusion efficiency while ensuring fusion quality, providing high-quality data, and saving a lot of time for building 3D point cloud model of green plants.

Key words: multi-focal length, image fusion, stationary wavelet transform, parameter-adaptive, pulse-coupled, neural network

LI Jiahao, QU Hongjun, GAO Mingzhe, TONG Dezhi, GUO Ya. A Multi-Focal Green Plant Image Fusion Method Based on Stationary Wavelet Transform and Parameter-Adaptation Dual Channel Pulse-Coupled Neural Network[J]. Smart Agriculture, 2023, 5(3): 121-131.

Figures/Tables 12

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Table 1

Fig. 9

Table 2

Table 3

References 29

1	刘刚, 司永胜, 冯娟. 农林作物三维重建方法研究进展[J]. 农业机械学报, 2014, 45(6): 38-46, 19.
	LIU G, SI Y S, FENG J. 3D reconstruction of agriculture and forestry crops[J]. Transactions of the Chinese society for agricultural machinery, 2014, 45(6): 38-46, 19.
2	陆健强, 兰玉彬, 毋志云, 等. 植物三维建模ICP点云配准优化[J]. 农业工程学报, 2022, 38(2): 183-191.
	LU J Q, LAN Y B, WU Z Y, et al. Optimization of ICP point cloud registration in plants 3D modeling[J]. Transactions of the Chinese society of agricultural engineering, 2022, 38(2): 183-191.
3	李百明, 吴茜, 吴劼, 等. 基于多视角自动成像系统的作物三维点云重建策略优化[J]. 农业工程学报, 2023, 39(9): 161-171.
	LI B M, WU Q, WU J, et al. Optimization of crop 3D point cloud reconstruction strategy based on the multi-view automatic imaging system[J]. Transactions of the Chinese society of agricultural engineering, 2023, 39(9): 161-171.
4	杨洪飞, 夏晖, 陈忻, 等. 图像融合在空间目标三维重建中的应用[J]. 红外与激光工程, 2018, 47(9): 374-381.
	YANG H F, XIA H, CHEN X, et al. Application of image fusion in 3D reconstruction of space target[J]. Infrared and laser engineering, 2018, 47(9): 374-381.
5	LIU A N, NIE W Z, GAO Y, et al. Multi-modal clique-graph matching for view-based 3D model retrieval[J]. IEEE transactions on image processing, 2016, 25(5): 2103-2116.
6	CHO S, KIM T, JUNG D H, et al. Plant growth information measurement based on object detection and image fusion using a smart farm robot[J]. Computers and electronics in agriculture, 2023, 207: ID 107703.
7	LIU J C, CHIANG C Y, CHEN S. Image-based plant recognition by fusion of multimodal information[C]// 2016 10th International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS). Piscataway, New Jersey, USA: IEEE, 2016: 5-11.
8	CHOUDHURY SDAS, MATURU S, SAMAL A, et al. Leveraging image analysis to compute 3D plant phenotypes based on voxel-grid plant reconstruction[J]. Frontiers in plant science, 2020, 11: ID 521431.
9	张丽霞, 曾广平, 宣兆成. 多源图像融合方法的研究综述[J]. 计算机工程与科学, 2022, 44(2): 321-334.
	ZHANG L X, ZENG G P, XUAN Z C. A survey of fusion methods for multi-source image[J]. Computer engineering & science, 2022, 44(2): 321-334.
10	李娇, 杨艳春, 党建武, 等. NSST与引导滤波相结合的多聚焦图像融合算法[J]. 哈尔滨工业大学学报, 2018, 50(11): 145-152.
	LI J, YANG Y C, DANG J W, et al. NSST and guided filtering for multi-focus image fusion algorithm[J]. Journal of Harbin institute of technology, 2018, 50(11): 145-152.
11	TAN W, TIWARI P, PANDEY H M, et al. Multimodal medical image fusion algorithm in the era of big data[J]. Neural computing and applications, 2020.
12	曾志勇, 刘光宇, 曹禹, 等. 结合PCNN和局部能量的NSST域声呐图像融合[J]. 西北大学学报(自然科学版), 2023, 53(2): 248-255.
	ZENG Z Y, LIU G Y, CAO Y, et al. NSST domain sonar image fusion combining PCNN and local energy[J]. Journal of Northwest university (natural science edition), 2023, 53(2): 248-255.
13	TAN W, ZHANG J J, XIANG P, et al. Infrared and visible image fusion via NSST and PCNN in multiscale morphological gradient domain[C]// Optics, Photonics and Digital Technologies for Imaging Applications VI. Burlingame, California, USA: SPIE, 2020, 11353: 297-303.
14	龚循强, 侯昭阳, 吕开云, 等.结合改进Laplacian能量和参数自适应双通道ULPCNN的遥感影像融合方法[J/OL].测绘学报:1-15[2023-11-10].
	GONG X Q, HOU Z Y, LYU K Y, et al. Remote sensing image fusion method combining improved Laplacian energy and parameter adaptive dual-channel unit-linking pulse coupled neural network[J/OL]. Acta geodaetica et cartographica Sinica, 2023: 1-15[2023-11-10].
15	邸敬, 尹世杰, 廉敬. 基于NSST的改进双通道PCNN多聚焦RGB图像融合[J]. 计算机应用研究, 2022, 39(1): 308-311, 315.
	DI J, YIN S J, LIAN J. Improved dual-channel PCNN multi-focus RGB image fusion based on NSST[J]. Application research of computers, 2022, 39(1): 308-311, 315.
16	ULLAH H, ZHAO Y Q, ABDALLA F Y O, et al. Fast local Laplacian filtering based enhanced medical image fusion using parameter-adaptive PCNN and local features-based fuzzy weighted matrices[J]. Applied intelligence, 2022, 52(7): 7965-7984.
17	LIU L Q, ZHOU Y B, HUO J Y, et al. Heterogenous image fusion model with SR-dual-channel PCNN significance region for NSST in an apple orchard[J]. Applied intelligence, 2023, 53(18): 21325-21346.
18	HUANG W, JING Z L. Multi-focus image fusion using pulse coupled neural network[J]. Pattern recognition letters, 2007, 28(9): 1123-1132.
19	王昶, 郭东升. 基于离散平稳小波的影像恢复去噪方法[J]. 北京测绘, 2022, 36(11): 1557-1563.
	WANG C, GUO D S. Image recovery denoising method based on stationary wavelet transform[J]. Beijing surveying and mapping, 2022, 36(11): 1557-1563.
20	孔玲君, 张志华, 曾茜, 等. 基于NSST和SWT的红外与可见光图像融合算法研究[J]. 包装工程, 2018, 39(19): 216-222.
	KONG L J, ZHANG Z H, ZENG X, et al. Infrared and visible image fusion algorithm based on NSST and SWT[J]. Packaging engineering, 2018, 39(19): 216-222.
21	LI S T, YANG B, HU J W. Performance comparison of different multi-resolution transforms for image fusion[J]. Information fusion, 2011, 12(2): 74-84.
22	BASHIR R, JUNEJO R, QADRI N N, et al. SWT and PCA image fusion methods for multi-modal imagery[J]. Multimedia tools and applications, 2019, 78(2): 1235-1263.
23	DWIVEDY P, POTNIS A, SOOFI S, et al. Comparative study of MSVD, PCA, DCT, DTCWT, SWT and Laplacian Pyramid based image fusion[C]// 2017 International Conference on Recent Innovations in Signal processing and Embedded Systems (RISE). Piscataway, New Jersey, USA: IEEE, 2018: 269-273.
24	AYMAZ S, KÖSE C. Multi-focus image fusion using Stationary Wavelet Transform (SWT) with Principal Component Analysis (PCA)[C]// 2017 10th International Conference on Electrical and Electronics Engineering (ELECO). Piscataway, New Jersey, USA: IEEE, 2018: 1176-1180.
25	ZHOU F Q, LI X S, LI J, et al. Multifocus image fusion based on fast guided filter and focus pixels detection[J]. IEEE access, 2019, 7: 50780-50796.
26	MA J L, ZHOU Z Q, WANG B, et al. Multi-focus image fusion using boosted random walks-based algorithm with two-scale focus maps[J]. Neurocomputing, 2019, 335: 9-20.
27	JIN X, NIE R C, ZHOU D M, et al. Multifocus color image fusion based on NSST and PCNN[J]. Journal of sensors, 2016, 2016: 1-12.
28	XIANG T Z, YAN L, GAO R R. A fusion algorithm for infrared and visible images based on adaptive dual-channel unit-linking PCNN in NSCT domain[J]. Infrared physics & technology, 2015, 69: 53-61.
29	杨艳春, 李娇, 王阳萍. 图像融合质量评价方法研究综述[J]. 计算机科学与探索, 2018, 12(7): 1021-1035.
	YANG Y C, LI J, WANG Y P. Review of image fusion quality evaluation methods[J]. Journal of frontiers of computer science and technology, 2018, 12(7): 1021-1035.

源图像	算法	AG/（GL·Px^-1）	SF/（LP·Px^-1）	EN/（bit·Px^-1）	SD/GL
第一组	FGF	2.78	10.25	6.52	81.57
	RW	2.72	9.88	6.51	81.50
	NSST-PCNN	2.87	9.95	6.59	80.87
	NSST-PADC	3.25	11.43	6.63	81.60
	SWT	2.52	8.52	6.50	81.08
	PADC-PCNN-SWT	2.98	10.32	6.57	81.52
第二组	FGF	3.29	10.85	6.80	67.14
	RW	3.21	10.62	6.80	67.00
	NSST-PCNN	3.19	9.89	6.83	66.14
	NSST-PADC	3.67	11.83	6.89	67.12
	SWT	2.92	9.22	6.79	66.42
	PADC-PCNN-SWT	3.41	11.11	6.84	66.96
第三组	FGF	3.21	10.59	6.64	53.33
	RW	3.14	10.57	6.64	53.20
	NSST-PCNN	3.17	10.43	6.64	52.32
	NSST-PADC	3.69	12.16	6.72	53.49
	SWT	2.86	9.21	6.63	52.64
	PADC-PCNN-SWT	3.38	10.97	6.68	53.22
第四组	FGF	2.85	10.77	6.71	72.43
	RW	2.80	9.37	6.71	72.38
	NSST-PCNN	2.81	9.02	6.75	71.33
	NSST-PADC	3.28	10.77	6.79	72.44
	SWT	2.58	8.33	6.70	71.98
	PADC-PCNN-SWT	3.05	10.06	6.74	72.43
第五组	FGF	2.93	11.62	6.57	70.36
	RW	2.87	11.63	6.57	70.30
	NSST-PCNN	2.97	11.76	6.62	69.86
	NSST-PADC	3.35	13.21	6.66	70.47
	SWT	2.66	10.45	6.56	69.94
	PADC-PCNN-SWT	3.09	12.04	6.61	70.41
第六组	FGF	2.00	9.31	6.09	66.35
	RW	1.94	9.07	6.08	66.29
	NSST-PCNN	1.98	8.86	6.06	65.90
	NSST-PADC	2.27	10.11	6.17	66.34
	SWT	1.78	8.12	6.06	66.01
	PADC-PCNN-SWT	2.11	9.29	6.13	66.33
第七组	FGF	4.07	14.88	7.19	76.94
	RW	4.06	15.09	7.19	76.86
	NSST-PCNN	3.90	13.09	7.13	75.16
	NSST-PADC	4.48	16.11	7.21	77.03
	SWT	3.76	12.58	7.16	74.79
	PADC-PCNN-SWT	3.97	13.06	7.17	75.65
第八组	FGF	4.17	15.87	7.69	74.21
	RW	4.23	15.74	7.69	74.03
	NSST-PCNN	4.35	15.58	7.66	74.53
	NSST-PADC	4.43	16.11	7.69	74.94
	SWT	3.69	14.37	7.68	72.89
	PADC-PCNN-SWT	3.98	14.63	7.69	73.12

焦距/mm	AG/（GL·Px^-1）	SF/（LP·Px^-1）	EN/（bit·Px^-1）	SD/GL
12-15	2.70	9.79	6.42	80.20
15-18	3.02	11.10	6.48	81.13
18-21	2.92	11.27	6.45	81.80
21-24	2.73	10.74	6.32	80.56

[1]	YAO Jianen, LIU Haiqiu, YANG Man, FENG Jinying, CHEN Xiu, ZHANG Peipei. Reconstruction of U.S. Regional-Scale Soybean SIF Based on MODIS Data and BP Neural Network [J]. Smart Agriculture, 2024, 6(5): 40-50.
[2]	LIU Liqi, WEI Guangyuan, ZHOU Ping. Prediction and Mapping of Soil Total Nitrogen Using GF-5 Image Based on Machine Learning Optimization Modeling [J]. Smart Agriculture, 2024, 6(5): 61-73.
[3]	LI Minghuang, SU Lide, ZHANG Yong, ZONG Zheying, ZHANG Shun. Automatic Measurement of Mongolian Horse Body Based on Improved YOLOv8n-pose and 3D Point Cloud Analysis [J]. Smart Agriculture, 2024, 6(4): 91-102.
[4]	ZHANG Jun, CHEN Yuyan, QIN Zhenyu, ZHANG Mengyao, ZHANG Jun. Remote Sensing Extraction Method of Terraced Fields Based on Improved DeepLab v3+ [J]. Smart Agriculture, 2024, 6(3): 46-57.
[5]	JIA Wenshen, LYU Haolin, ZHANG Shang, QIN Yingdong, ZHOU Wei. Using a Portable Visible-near Infrared Spectrometer and Machine Learning to Distinguish and Quantify Mold Contamination in Wheat [J]. Smart Agriculture, 2024, 6(1): 89-100.
[6]	ZUO Min, HU Tianyu, DONG Wei, ZHANG Kexin, ZHANG Qingchuan. Forecast and Analysis of Agricultural Products Logistics Demand Based on Informer Neural Network: Take the Central China Aera as An Example [J]. Smart Agriculture, 2023, 5(1): 34-43.
[7]	ZHANG Zhibo, ZHAO Xining, GAO Xiaodong, ZHANG Li, YANG Menghao. Accurate Extraction of Apple Orchard on the Loess Plateau Based on Improved Linknet Network [J]. Smart Agriculture, 2022, 4(3): 95-107.
[8]	ZHOU Qiaoli, MA Li, CAO Liying, YU Helong. Identification of Tomato Leaf Diseases Based on Improved Lightweight Convolutional Neural Networks MobileNetV3 [J]. Smart Agriculture, 2022, 4(1): 47-56.
[9]	SHAO Mingyue, ZHANG Jianhua, FENG Quan, CHAI Xiujuan, ZHANG Ning, ZHANG Wenrong. Research Progress of Deep Learning in Detection and Recognition of Plant Leaf Diseases [J]. Smart Agriculture, 2022, 4(1): 29-46.
[10]	WANG Ying, LI Yue, WU Tingting, SUN Shi, WANG Minjuan. Fast Counting Method of Soybean Seeds Based on Density Estimation and VGG-Two [J]. Smart Agriculture, 2021, 3(4): 111-122.
[11]	LI Dongbo, HUANG Lyuwen, ZHAO Xubo. Detection Method of Apple Mould Core Based on Dielectric Characteristics [J]. Smart Agriculture, 2021, 3(4): 66-76.
[12]	LI Zhijun, YANG Shenghui, SHI Deshuai, LIU Xingxing, ZHENG Yongjun. Yield Estimation Method of Apple Tree Based on Improved Lightweight YOLOv5 [J]. Smart Agriculture, 2021, 3(2): 100-114.
[13]	YIN Hang, LI Xiangtong, XU Longqin, LI Jingbin, LIU Shuangyin, CAO Liang, FENG Dachun, GUO Jianjun, LI Liqiao. EMD-RF-LSTM: Combination Prediction Model of Dissolved Oxygen Concentration in Prawn Culture [J]. Smart Agriculture, 2021, 3(2): 115-125.
[14]	LI Yan, SHEN Jie, XIE Hang, GAO Guangyin, LIU Jianxiong, LIU Jie. Detection and Grading Method of Pomelo Shape Based on Contour Coordinate Transformation and Fitting [J]. Smart Agriculture, 2021, 3(1): 86-95.
[15]	QIU Wenjie, YE Jin, HU Liangqing, YANG Juan, LI Qili, MO Jianyou, YI Wanmao. Distilled-MobileNet Model of Convolutional Neural Network Simplified Structure for Plant Disease Recognition [J]. Smart Agriculture, 2021, 3(1): 109-117.