High-Precision Fish Pose Estimation Method Based on Inproved HRNet

doi:10.12133/j.smartag.SA202502001

Abstract

Abstract:

[Objective] Fish pose estimation (FPE) provides fish physiological information, facilitating health monitoring in aquaculture. It aids decision-making in areas such as fish behavior recognition. When fish are injured or deficient, they often display abnormal behaviors and noticeable changes in the positioning of their body parts. Moreover, the unpredictable posture and orientation of fish during swimming, combined with the rapid swimming speed of fish, restrict the current scope of research in FPE. In this research, a FPE model named HPFPE is presented, designed to capture the swimming posture of fish and accurately detect their key points. [Methods] On the one hand, this model incorporated the CBAM module into the HRNet framework. The attention module enhanced accuracy without adding computational complexity, while effectively capturing a broader range of contextual information. On the other hand, the model incorporated dilated convolution to increase the receptive field, allowing it to capture more spatial context. [Results and Discussions] Experiments showed that compared with the baseline method, the average precision (AP) of HPFPE based on different backbones and input sizes on the oplegnathus punctatus datasets had increased by 0.62, 1.35, 1.76, and 1.28, respectively, while the average recall (AR) had also increased by 0.85, 1.5, 1.4, and 1, respectively. Additionally, HPFPE outperformed other mainstream methods, including DeepPose, CPM, SCNet, and Lite-HRNet. Furthermore, when compared to other methods using the ornamental fish data, HPFPE achieved the highest AP and AR values of 52.96, and 59.50, respectively. Conclusions The proposed HPFPE can accurately estimate fish posture and assess their swimming patterns, serving as a valuable reference for applications such as fish behavior recognition.

Key words: aquaculture, computer vision, fish pose estimation, key point, attention mechanism

CLC Number:

S951.2

PENG Qiujun, LI Weiran, LIU Yeqiang, LI Zhenbo. High-Precision Fish Pose Estimation Method Based on Inproved HRNet[J]. Smart Agriculture, doi: 10.12133/j.smartag.SA202502001.

Figures/Tables 21

Fig. 1

Table 1

Fig. 2

Table 2

Fig. 4

Table 3

Fig. 5

Fig. 6

Table 4

Table 5

Table 6

Fig. 7

Table 7

Table 8

Fig. 8

Table 9

Table 10

Table 11

Table 12

Fig. 9

References 30

1	PENG Q , LI W , LI Z . Review of fish behavior recognition methods based on artificial intelligence[J]. Transactions of the Chinese society for agricultural machinery, 2023, 54(S1): 283-295.
2	ZHENG C , WU W H , CHEN C , et al . Deep learning-based human pose estimation: A survey[J]. ACM computing surveys, 2024, 56(1): 1-37.
3	SUN K , XIAO B , LIU D , et al . Deep high-resolution representation learning for human pose estimation[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, New Jersey, USA: IEEE, 2019: 5686-5696.
4	LIU J J , HOU Q B , CHENG M M , et al . Improving convolutional networks with self-calibrated convolutions[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, New Jersey, USA: IEEE, 2020: 10093-10102.
5	YU C Q , XIAO B , GAO C X , et al . Lite-HRNet: A lightweight high-resolution network[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, New Jersey, USA: IEEE, 2021: 10435-10445.
6	YAN S J , XIONG Y J , LIN D H . Spatial temporal graph convolutional networks for skeleton-based action recognition[C]// Proceedings of the AAAI conference on artificial intelligence. Palo Alto, California USA: AAAI Press. 2018, 32(1): 7444-7452.
7	MARKOVITZ A , SHARIR G , FRIEDMAN I , et al . Graph embedded pose clustering for anomaly detection[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, New Jersey, USA: IEEE, 2020: 10536-10544.
8	DAS S, SHARMA S , DAI R , et al . VPN: Learning video-pose embedding for activities of daily living[C]// Computer Vision – ECCV 2020. Cham, Germany: Springer International Publishing, 2020: 72-90.
9	YANG X T , ZHANG S , LIU J T , et al . Deep learning for smart fish farming: Applications, opportunities and challenges[J]. Reviews in aquaculture, 2021, 13(1): 66-90.
10	YUE J , YANG H H , JIA S X , et al . A multi-scale features-based method to detect Oplegnathus [J]. Information processing in agriculture, 2021, 8(3): 437-445.
11	WANG H R , CHEN Y Y , CHAI Y Q , et al . Image segmentation method combined with VoVNetv2 and shuffle attention mechanism for fish feeding in aquaculture[J]. Smart agriculture, 2023, 5(4): 137-149.
12	LI W R , LI F , LI Z B . CMFTNet: Multiple fish tracking based on counterpoised JointNet[J]. Computers and electronics in agriculture, 2022, 198: ID 107018.
13	ZHAO Y Y , LI W R , LI Y M , et al . LFCNet: A lightweight fish counting model based on density map regression[J]. Computers and electronics in agriculture, 2022, 203: ID 107496.
14	BANAN A , NASIRI A , TAHERI-GARAVAND A . Deep learning-based appearance features extraction for automated carp species identification[J]. Aquacultural engineering, 2020, 89: ID 102053.
15	DU Z Z , XU X B , BAI Z Z , et al . Feature fusion strategy and improved GhostNet for accurate recognition of fish feeding behavior[J]. Computers and electronics in agriculture, 2023, 214: ID 108310.
16	LI B B , YUE J , JIA S X , et al . Recognition of abnormal body surface characteristics of Oplegnathus punctatus [J]. Information processing in agriculture, 2022, 9(4): 575-585.
17	HU W C , YANG X T , MA P C , et al . DCA-MVIT: Fused DSGated convolution and CA attention for fish feeding behavior recognition in recirculating aquaculture systems[J]. Aquaculture, 2025, 598: ID 742008.
18	MATHIS A , MAMIDANNA P , CURY K M , et al . DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning[J]. Nature neuroscience, 2018, 21(9): 1281-1289.
19	WU X , HUANG J P , WANG L M . Pose estimation-based visual perception system for analyzing fish swimming[J]. IEEE sensors journal, 2024, 24(8): 13293-13303.
20	WANG L , ZOU H Q , CHEN S H . Fish identification and tracking based on pose estimation[C]// 2024 IEEE 19th Conference on Industrial Electronics and Applications (ICIEA). Piscataway, New Jersey, USA: IEEE, 2024.
21	HAIPENG C , CHAOXU Q , ZHIYU M . Fish key feature point detection and sign identification based on deep learning[J]. Journal of Chinese agricultural mechanization, 2024, 45(6): 201-207.
22	YU F , KOLTUN V . Multi-scale context aggregation by dilated convolutions[EB/OL]. arXiv: 1511.07122, 2015.
23	WOO S, PARK J , LEE J Y , et al . CBAM: Convolutional Block attention module[C]// Computer Vision – ECCV 2018. Cham, Germany: Springer International Publishing, 2018: 3-19.
24	HU J , SHEN L , SUN G . Squeeze-and-excitation networks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, New Jersey, USA: IEEE, 2018: 7132-7141.
25	WANG Q L , WU B G , ZHU P F , et al . ECA-net: Efficient channel attention for deep convolutional neural networks[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, New Jersey, USA: IEEE, 2020: 11531-11539.
26	HOU Q B , ZHOU D Q , FENG J S . Coordinate attention for efficient mobile network design[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, New Jersey, USA: IEEE, 2021: 13708-13717.
27	LI Y X , HOU Q B , ZHENG Z H , et al . Large selective kernel network for remote sensing object detection[C]// 2023 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway, New Jersey, USA: IEEE, 2023: 16748-16759.
28	KAN Z H , CHEN S S , ZHANG C , et al . Self-correctable and adaptable inference for generalizable human pose estimation[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 17-24, 2023. Vancouver, BC, Canada. IEEE, 2023: 5537-5546.
29	HUANG L Z , LI Y L , TIAN H B , et al . Semi-supervised 2D human pose estimation driven by position inconsistency pseudo label correction module[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, New Jersey, USA: IEEE, 2023: 693-703.
30	WANG H X , ZHOU L , CHEN Y Y , et al . Regularizing vector embedding inBottom-up human pose estimation[C]//Computer Vision – ECCV 2022. Cham, Germany: Springer, 2022: 107-122.

Number	Name	Location
1	belly	Anterior end of the pelvic fin
2	eye_left	Left eye
3	eye_right	Right eye
4	back	Anterior end of dorsal fin
5	tail_top	Anterior end of tail
6	tail_up	Top of tail
7	tail_low	Bottom of tail

Dataset	Training set images	Test set images	Image total	Fish total
Oplegnathus punctatus images before augmentation	400	100	500	800
Oplegnathus punctatus images after augmentation	800	100	900	1 800
Ornamental fish images	240	60	300	3 000
Final dataset total	1 040	160	1 200	4 800

Layer	Multi-scale feature map	Number of branches	Number of blocks in each branch	Number of exchange units
Stage 1	1/4	1	4	0
Stage 2	1/4， 1/8	2	4， 4	1
Stage 3	1/4， 1/8， 1/16	3	4， 4， 4	4
Stage 4	1/4， 1/8， 1/16， 1/32	4	4， 4， 4， 4	3

Configuration	Parameter
CPU	Intel（R） Xeon（R） Silver 4210R CPU
GPU	NVIDIA Tesla V100
Operating system	Ubuntu 18.04.6 LTS
Deep learning framework	PyTorch 1.8.1
Programming language	Python 3.8

Number of Stage	HRNet-32	HRNet-48
Stage 2	64	96
Stage 3	128	192
Stage 4	256	384