Automatic Measurement Method of Beef Cattle Body Size Based on Multimodal Image Information and Improved Instance Segmentation Network

doi:10.12133/j.smartag.SA202310007

Abstract

Abstract:

[Objective] The body size parameter of cattle is a key indicator reflecting the physical development of cattle, and is also a key factor in the cattle selection and breeding process. In order to solve the demand of measuring body size of beef cattle in the complex environment of large-scale beef cattle ranch, an image acquisition device and an automatic measurement algorithm of body size were designed. [Methods] Firstly, the walking channel of the beef cattle was established, and when the beef cattle entered the restraining device through the channel, the RGB and depth maps of the image on the right side of the beef cattle were acquired using the Inter RealSense D455 camera. Secondly, in order to avoid the influence of the complex environmental background, an improved instance segmentation network based on Mask2former was proposed, adding CBAM module and CA module, respectively, to improve the model's ability to extract key features from different perspectives, extracting the foreground contour from the 2D image of the cattle, partitioning the contour, and comparing it with other segmentation algorithms, and using curvature calculation and other mathematical methods to find the required body size measurement points. Thirdly, in the processing of 3D data, in order to solve the problem that the pixel point to be measured in the 2D RGB image was null when it was projected to the corresponding pixel coordinates in the depth-valued image, resulting in the inability to calculate the 3D coordinates of the point, a series of processing was performed on the point cloud data, and a suitable point cloud filtering and point cloud segmentation algorithm was selected to effectively retain the point cloud data of the region of the cattle's body to be measured, and then the depth map was 16. Then the depth map was filled with nulls in the field to retain the integrity of the point cloud in the cattle body region, so that the required measurement points could be found and the 2D data could be returned. Finally, an extraction algorithm was designed to combine 2D and 3D data to project the extracted 2D pixel points into a 3D point cloud, and the camera parameters were used to calculate the world coordinates of the projected points, thus automatically calculating the body measurements of the beef cattle. [Results and Discussions] Firstly, in the part of instance segmentation, compared with the classical Mask R-CNN and the recent instance segmentation networks PointRend and Queryinst, the improved network could extract higher precision and smoother foreground images of cattles in terms of segmentation accuracy and segmentation effect, no matter it was for the case of occlusion or for the case of multiple cattles. Secondly, in three-dimensional data processing, the method proposed in the study could effectively extract the three-dimensional data of the target area. Thirdly, the measurement error of body size was analysed, among the four body size measurement parameters, the smallest average relative error was the height of the cross section, which was due to the more prominent position of the cross section, and the different standing positions of the cattle have less influence on the position of the cross section, and the largest average relative error was the pipe circumference, which was due to the influence of the greater overlap of the two front legs, and the higher requirements for the standing position. Finally, automatic body measurements were carried out on 137 beef cattle in the ranch, and the automatic measurements of the four body measurements parameters were compared with the manual measurements, and the results showed that the average relative errors of body height, cross section height, body slant length, and tube girth were 4.32%, 3.71%, 5.58% and 6.25%, respectively, which met the needs of the ranch. The shortcomings were that fewer body-size parameters were measured, and the error of measuring circumference-type body-size parameters was relatively large. Later studies could use a multi-view approach to increase the number of body rule parameters to be measured and improve the accuracy of the parameters in the circumference category. [Conclusions] The article designed an automatic measurement method based on two-dimensional and three-dimensional contactless body measurements of beef cattle. Moreover, the innovatively proposed method of measuring tube girth has higher accuracy and better implementation compared with the current research on body measurements in beef cattle. The relative average errors of the four body tape parameters meet the needs of pasture measurements and provide theoretical and practical guidance for the automatic measurement of body tape in beef cattle.

Key words: cattle body size measurement, deep learning, point cloud segmentation, instance segmentation, attention mechanism, Mask2former

WENG Zhi, FAN Qi, ZHENG Zhiqiang. Automatic Measurement Method of Beef Cattle Body Size Based on Multimodal Image Information and Improved Instance Segmentation Network[J]. Smart Agriculture, 2024, 6(4): 64-75.

Figures/Tables 23

Table 1

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Table 2

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Table 3

Fig. 12

Table 4

Table 5

Fig. 13

Fig. 14

Fig. 15

Fig. 16

Fig. 17

Fig. 18

References 22

1	张智慧, 李伟, 韩永胜. 牛体尺影响因素及其应用[J]. 中国畜牧杂志, 2018, 54(1): 9-13.
	ZHANG Z H, LI W, HAN Y S. Influence factors and applications for body size of cattle[J]. Chinese journal of animal science, 2018, 54(1): 9-13.
2	ZHANG A L, WU B P, TANA WUYUN C, et al. Algorithm of sheep body dimension measurement and its applications based on image analysis[J]. Computers and electronics in agriculture, 2018, 153: 33-45.
3	WANG W, ZHANG Y, HE J, et al. Research on yak body ruler and weight measurement method based on deep learning and binocular vision[EB/OL]. Preprints, 2021, 2021120349. [2023-09-28].
4	GUO H, MA X D, MA Q, et al. LSSA_CAU: An interactive 3d point clouds analysis software for body measurement of livestock with similar forms of cows or pigs[J]. Computers and electronics in agriculture, 2017, 138: 60-68.
5	HUANG L W, GUO H, RAO Q Q, et al. Body dimension measurements of Qinchuan cattle with transfer learning from LiDAR sensing[J]. Sensors, 2019, 19(22): ID 5046.
6	王可, 郭浩, 刘威林, 等. 基于点云旋转归一化的猪体体尺测点提取方法[J]. 农业工程学报, 2017(S1): 253-259.
	WANG K, GUO H, LIU W L, et al. Extraction of porcine body size measurement points based on point cloud rotational normalization[J]. Transactions of the Chinese society of agricultural engineering, 2017(S1): 253-259.
7	张婷, 李琦. 基于三维点云的牛体尺测量研究[J].黑龙江畜牧兽医, 2023, (3): 58-61.
	ZHANG T, LI Q, et al. A study of bovine body measurements based on a three-dimensional point cloud[J]. Heilongjiang animal science and veterinary medicine, 2023, (3): 58-61.
8	LI G X, LIU X L, MA Y F, et al. Body size measurement and live body weight estimation for pigs based on back surface point clouds[J]. Biosystems engineering, 2022, 218: 10-22.
9	YIN L, ZHU J M, LIU C X, et al. Point cloud-based pig body size measurement featured by standard and non-standard postures[J]. Computers and electronics in agriculture, 2022, 199: ID 107135.
10	YANG G Y, XU X S, SONG L, et al. Automated measurement of dairy cows body size via 3D point cloud data analysis[J]. Computers and electronics in agriculture, 2022, 200: ID 107218.
11	石炜, 陈禹州, 张万锴, 等. 基于无人机平台的牛体点云重构及体尺测量系统研究[J]. 畜牧与饲料科学, 2022, 43(4): 93-103.
	SHI W, CHEN Y Z, ZHANG W K, et al. Cattle point cloud reconstruction and body size measurement system development based on unmanned air vehicle(UAV) platform[J]. Animal husbandry and feed science, 2022, 43(4): 93-103.
12	CHENG B W, MISRA I, SCHWING A G, et al. Masked-attention mask transformer for universal image segmentation[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, New Jersey, USA: IEEE, 2022: 1290-1299.
13	王永智. 牛的外貌鉴定、体尺测量及体重估测[J]. 养殖技术顾问, 2013(12): ID 20.
	WANG Y Z. Appearance identification, body measurement and weight estimation of cattle[J]. Technical advisor for animal husbandry, 2013(12): ID 20.
14	CHENG B W, SCHWING A G, KIRILLOV A. Per-pixel classification is not all you need for semantic segmentation[EB/OL]. arXiv: 2107.06278, 2021.
15	ZHU X Z, SU W J, LU L W, et al. Deformable DETR: Deformable transformers for end-to-end object detection[EB/OL]. arXiv: 2010.04159, 2020.
16	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[M]// Computer vision-ECCV 2018. Cham: Springer International Publishing, 2018: 3-19.
17	HOU Q B, ZHOU D Q, FENG J S. Coordinate attention for efficient mobile network design[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, New Jersey, USA: IEEE, 2021: 13713-13722.
18	Abeywickrama T, Aamir Cheema M, Taniar D. k-Nearest Neighbors on Road Networks: A Journey in Experimentation and In-Memory Implementation[EB/OL]. arXiv: 1601.01549, 2016.
19	Bäcklund H, Hedblom A, Neijman N. A density-based spatial clustering of application with noise[J]. Data Mining TNM033, 2011, 33: 11-30.
20	HE K M, GKIOXARI G, DOLLAR P, et al. Mask R-CNN[C]// 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway, New Jersey, USA: IEEE, 2017: 2961-2969.
21	KIRILLOV A, WU Y X, HE K M, et al. PointRend: Image segmentation as rendering[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, New Jersey, USA: IEEE, 2020: 9799-9808.
22	FANG Y X, YANG S S, WANG X G, et al. Instances as queries[C]// 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway, New Jersey, USA: IEEE, 2021: 6910-6919.

模型	bbox_mAP _0.5/%	segm_mAP _0.5/%
Mask2former	83.0	84.3
CB-Mask2former	83.5	84.9
CA-Mask2former	84.5	85.7
CC-Mask2former	86.8	88.1

模型	bbox_mAP _0.5/%	segm_mAP _0.5/%
Mask R-CNN^［20］	84.2	84.2
PointRend^［21］	70.4	69.2
Queryinst^［22］	86.0	84.2
CC-Mask2former	86.8	88.1

[1]	HU Chengxi, TAN Lixin, WANG Wenyin, SONG Min. Lightweight Tea Shoot Picking Point Recognition Model Based on Improved DeepLabV3+ [J]. Smart Agriculture, 2024, 6(5): 119-127.
[2]	LUO Youlu, PAN Yonghao, XIA Shunxing, TAO Youzhi. Lightweight Apple Leaf Disease Detection Algorithm Based on Improved YOLOv8 [J]. Smart Agriculture, 2024, 6(5): 128-138.
[3]	LIU Yi, ZHANG Yanjun. ReluformerN: Lightweight High-Low Frequency Enhanced for Hyperspectral Agricultural Lancover Classification [J]. Smart Agriculture, 2024, 6(5): 74-87.
[4]	NIAN Yue, ZHAO Kaixuan, JI Jiangtao. Cow Hoof Slippage Detecting Method Based on Enhanced DeepLabCut Model [J]. Smart Agriculture, 2024, 6(5): 153-163.
[5]	ZHANG Yanqi, ZHOU Shuo, ZHANG Ning, CHAI Xiujuan, SUN Tan. A Regional Farming Pig Counting System Based on Improved Instance Segmentation Algorithm [J]. Smart Agriculture, 2024, 6(4): 53-63.
[6]	LI Minghuang, SU Lide, ZHANG Yong, ZONG Zheying, ZHANG Shun. Automatic Measurement of Mongolian Horse Body Based on Improved YOLOv8n-pose and 3D Point Cloud Analysis [J]. Smart Agriculture, 2024, 6(4): 91-102.
[7]	HOU Yiting, RAO Yuan, SONG He, NIE Zhenjun, WANG Tan, HE Haoxu. A Rapid Detection Method for Wheat Seedling Leaf Number in Complex Field Scenarios Based on Improved YOLOv8 [J]. Smart Agriculture, 2024, 6(4): 128-137.
[8]	WENG Zhi, LIU Haixin, ZHENG Zhiqiang. CSD-YOLOv8s: Dense Sheep Small Target Detection Model Based on UAV Images [J]. Smart Agriculture, 2024, 6(4): 42-52.
[9]	WANG Yuxiao, SHI Yuanyuan, CHEN Zhaoda, WU Zhenfang, CAI Gengyuan, ZHANG Sumin, YIN Ling. Pig Back Transformer: Automatic 3D Pig Body Measurement Model [J]. Smart Agriculture, 2024, 6(4): 76-90.
[10]	DAI Xin, WANG Junhao, ZHANG Yi, WANG Xinjie, LI Yanxing, DAI Baisheng, SHEN Weizheng. Automatic Detection Method of Dairy Cow Lameness from Top-view Based on the Fusion of Spatiotemporal Stream Features [J]. Smart Agriculture, 2024, 6(4): 18-28.
[11]	LI Hao, DU Yuqiu, XIAO Xingzhu, CHEN Yanxi. Remote Sensing Identification Method of Cultivated Land at Hill County of Sichuan Basin Based on Deep Learning [J]. Smart Agriculture, 2024, 6(3): 34-45.
[12]	ZHU Yiping, WU Huarui, GUO Wang, WU Xiaoyan. Identification Method of Kale Leaf Ball Based on Improved UperNet [J]. Smart Agriculture, 2024, 6(3): 128-137.
[13]	NIE Ganggang, RAO Honghui, LI Zefeng, LIU Muhua. Severity Grading Model for Camellia Oleifera Anthracnose Infection Based on Improved YOLACT [J]. Smart Agriculture, 2024, 6(3): 138-147.
[14]	ZHANG Ronghua, BAI Xue, FAN Jiangchuan. Crop Pest Target Detection Algorithm in Complex Scenes:YOLOv8-Extend [J]. Smart Agriculture, 2024, 6(2): 49-61.
[15]	ZHANG Jing, ZHAO Zexuan, ZHAO Yanru, BU Hongchao, WU Xingyu. Oilseed Rape Sclerotinia in Hyperspectral Images Segmentation Method Based on Bi-GRU and Spatial-Spectral Information Fusion [J]. Smart Agriculture, 2024, 6(2): 40-48.