Abstract:

Objective China is one of the world's largest apple producers. However, labor costs account for a high proportion of total harvesting expenses, and obstacle avoidance path planning of robotic arms remains a core bottleneck restricting the replacement of manual labor with robots. In orchard environments, rigid obstacles such as traction wires and thick branches are densely intertwined with flexible obstacles such as thin twigs and leaves. Traditional path planning algorithms rely on static geometric modeling, making them difficult to adapt to dynamic geometric boundaries and leading to long planning times and poor adaptability. Although deep reinforcement learning methods have been applied, they still suffer from limited generalization and low training efficiency when rigid and flexible obstacles are handled differently. To address the path planning challenge caused by rigid-flexible hybrid obstacles in unstructured orchards, an obstacle avoidance method based on expert-demonstration-guided deep reinforcement learning is proposed, providing theoretical and technical support for the intelligent harvesting robot operation. Methods First, a high-fidelity simulation environment was constructed based on the MuJoCo physics engine, which modeled a 4-degree-of-freedom robotic arm, a three-fingered gripper, and rigid-flexible hybrid obstacles. The combination of rigid tree trunks and flexible branches modeled using rope-based models, together with the precise configuration of elasticity and damping parameters, enabled realistic physical deformation to be reproduced. The designed 425-dimensional state vector covered key data such as the end-effector position of the robotic arm, fruit position, and attributes of nearly 50 obstacle segments. The action space was discretized into 8 actions, including gripper opening/closing and six-direction displacement of the end-effector with a step size of 0.01 m. The harvesting task was decomposed into three phases: approaching, grasping, and placing. The simulation environment could automatically complete phase switching and task judgment and distinguish rigid and flexible obstacles using labels—brown for rigid branches and green for flexible branches. Collision detection was incorporated to monitor interference between the gripper and obstacles, providing realistic scenario support for algorithm training. In terms of algorithm design, an expert-demonstration-guided soft actor-critic (EG-SAC) algorithm was proposed. A hierarchical reward function was designed that assigned rewards according to the distance variation between the end-effector and the target point in each phase, and imposed penalties for collisions, timeouts, and other abnormal situations. A total of 600 expert trajectories were generated by analytical calculation. For flexible occlusion scenarios, branch-poking guide points were designed to guide the robotic arm to approach the target accurately. The policy network adopted a three-layer fully connected neural network. After pre-training via behavioral cloning, it was optimized using the hindsight experience replay (HER)-SAC algorithm. Kullback-Leibler (KL) divergence regularization was introduced to constrain policy deviation, thereby balancing the exploitation of expert experience and the optimization potential of reinforcement learning. Orchard field tests were conducted using the self-developed multi-arm harvesting robot developed by the research team. The test site was a standard high-density dwarf orchard. The vision system acquired image and depth information via an RGB-D sensor, detected fruits using YOLOv8 (You Only Look Once), and generated 3D point clouds. Three typical scenarios were selected, with 20 fruit samples in each scenario. Scenario 1: simple flexible occlusion, Scenario 2: rigid-flexible hybrid occlusion, and Scenario 3: rigid occlusion. The initial poses of the robotic arm were kept consistent within the same round of tests and randomized across different rounds to verify generalization ability. Results and Discussions In simulation experiments, a system comprising a robot mobile platform, a multi-degree-of-freedom robotic arm, and a rigid-flexible hybrid fruit tree model was built based on MuJoCo. This system could randomly generate branch poses and fruit positions and synchronously randomize the initial poses of the robotic arm to ensure training-sample diversity. Comparative experiments were set up with an EG-SAC experimental group and a standard SAC baseline group. The results showed that EG-SAC converged rapidly andoutperfored the standard SAC, with a 75% improvement in convergence speed, and could drive the robotic arm to implement differentiated obstacle avoidance behaviors. The field test results showed that: the success rates of the EG-SAC algorithm in the three scenarios were 80%, 60%, and 75%, respectively, with an average success rate of 71.7%, approximately twice that of the RRT algorithm; the average collision rate was only 13.3%, much lower than the 81.7% of the RRT algorithm; and the operation time ranged from 12.11 to 12.62 s, comparable to that of the RRT algorithm. Failure mode analysis indicated that the main cause of RRT algorithm failure was obstacle misgrasping, accounting for 42.0%, while the failures of the EG-SAC algorithm were mostly attributed to fine-grained control issues. Conclusions In conclusion, this study addresses the obstacle avoidance challenge in unstructured orchards through a differentiated interaction mechanism for rigid-flexible obstacles, an expert demonstration-guided reinforcement learning framework, and high-fidelity simulation and field verification. The research findings provide a promising reference for the practical application of harvesting robots. Future work could further improve the response speed through model optimization and hardware acceleration. In addition, the expert-guided framework could be extended to other complex manipulation tasks, which may help promote the practical application of agricultural robots.

Key words: apple harvesting, rigid-flexible mixture, MuJoCo, expert demonstration, reinforcement learning, obstacle