Autonomous Excavation Using Reinforcement Learning with Proximal Policy Optimization

Abstract

This thesis presents a reinforcement learning based approach for grading, applied in the Volvo excavator EC550E. The work was based around a simulation model of the excavator which facilitated easy training of the algorithm. A hydraulic controller was trained using proximal policy optimization. Together with the hydraulic controller, a PID was implemented as a positional controller for a complete system capable of performing grading tasks. Training was conducted by testing different reward functions and parameter choices to improve policy performance. The results showcases hyperparameter evaluation, velocity tracking accuracy for the hydraulic controller as well as grading accuracy of the complete system. The implemented solution had an accuracy of ± 4 cm during grading. However, the hydraulic controller was not able to consistently follow the target velocities in the cylinders, particularly for the bucket. In future works the hydraulic controller needs to be retrained for better precision before being deployed in a real machine. This thesis shows the potential and possibility of replacing traditional control policies with an machine-learning driven approach.