Abstract:

[Objective] To address the current issues of intelligent pruning robots, such as insufficient recognition accuracy of fruit tree branches and inaccurate localization of pruning points in complex field environments, a deep learning method based on the fusion of images and point clouds was proposed in this research. The method enables non-contact segmentation of dormant high-spindle apple tree branches and measurement of phenotypic parameter, which also achieving automatic identification and precise localization of pruning points. [Methods] Localized RGB-D data were collected from apple trees using a Realsense D435i camera, a device capable of accurate depth measurements within the range of 0.3~3.0 m. Data acquisition took place between early and mid-January 2024, from 9:00 AM to 4:00 PM daily. To maintain uniformity, the camera was mounted on a stand at a distance of 0.4~0.5 m from the main stems of the apple trees. Following data collection, trunks and branches were manually annotated using Labelme software. The OpenCV library was also employed for data augmentation, which helped prevent overfitting during model training. To improve segmentation accuracy of tree trunks and branches in RGB images, an enhanced U-Net model was introduced. This model utilized VGG16 (Visual Geometry Group 16) as its backbone feature extraction network and incorporated the convolutional block attention module (CBAM) at the up-sampling stage. Based on the segmentation results, a multimodal data processing pipeline was established. First, segmented branch mask maps were derived from skeleton lines extracted using OpenCV's algorithm. The first-level branch connection points were identified based on their positions relative to the trunk. Subsequently, potential pruning points were then searched within local neighborhoods through coordinate translation. An edge detection algorithm was applied to locate the nearest edge pixels relative to these potential pruning points. By extending the diameter line of branch pixel points in the images and integrating with depth information, the actual diameter of the branches could be estimated. Additionally, branch spacing was calculated using vertical coordinates differences of potential pruning points in the pixel coordinate system, alongside depth information. Meanwhile, trunk point cloud data were acquired by merging the trunk mask maps with the depth maps. Preprocessing of the point cloud enabled the estimation of the average trunk diameter in the local view through cylindrical fitting using the randomized sampling consistency (RANSAC) algorithm. Finally, an intelligent pruning decision-making algorithm was developed by investigating of orchardists' pruning experience, analyzing relevant literature, and integrating phenotypic parameter acquisition methods, thus achieving accurate prediction of apple tree pruning points. [Results and Discussions] The improved U-Net model proposed achieved a mean pixel accuracy (mPA) of 95.52% for branch segmentation, representing a 2.74 percent point improvement over the original architecture. Corresponding increases were observed in mean intersection over union (mIoU) and precision metrics. Comparative evaluations against DeepLabV3+, PSPNet, and the baseline U-Net were conducted under both backlight and front-light illumination conditions. The improved model demonstrated superior segmentation performance and robustness across all tested scenarios. Ablation experiments indicated that replacing the original feature extractor with VGG16 yielded a 1.52 percent point mPA improvement, accompanied by simultaneous gains in mIoU and precision. The integration of the CBAM at the up sampling stage further augmented the model's capacity to resolve fine branch structures. Phenotypic parameter estimation using segmented branch masks combined with depth maps showed strong correlations with manual measurements. Specifically, the coefficient of determination (R²) values for primary branch diameter, branch spacing, and trunk diameter were 0.96, 0.95, and 0.91, respectively. The mean absolute errors (MAE) were recorded as 1.33, 13.96, and 5.11 mm, surpassing the accuracy of visual assessments by human pruning operators. The intelligent pruning decision system achieved an 87.88% correct identification rate for pruning points, with an average processing time of 4.2 s per viewpoint. These results confirm the practical feasibility and operational efficiency of the proposed method in real-world agricultural settings. [Conclusions] An efficient and accurate method for identifying pruning points on apple trees was proposed, which integrates image and point cloud data through deep learning. The results indicate that this method could provide significant support for the application of intelligent pruning robots in modern agriculture. It not only offers high feasibility but also exhibits outstanding efficiency and accuracy in practical applications, thus laying a solid foundation for the advancement of agricultural automation.

Key words: pruning point identification, RGB-D, U-Net, diameter estimation, 3D point clouds, VGG16, pruning robot