Training end-to-end planners in sensor-level simulation

Abstract

End-to-end autonomous driving planners are commonly trained with imitation learning on offline expert demonstrations, an approach that can achieve strong openloop performance. However, planners following this paradigm usually suffer during closed-loop deployment, since they have not been exposed to the consequences of their own actions during training. Addressing this requires closed-loop training, which has typically involved high-level representations, forgoing the benefits of sensor-level end-to-end planning. This work investigates whether recent advances in efficient 3D Gaussian Splatting can make closed-loop training of sensor-level end-to-end planners feasible. A closedloop simulator is developed by reconstructing driving sequences and integrating them with a trajectory tracker and a kinematic vehicle model capable of executing the planner’s predicted trajectories. This makes it possible to render new views that deviate from the original logged trajectory. The resulting framework is used to fine-tune a pretrained Latent TransFuser planner through reinforcement learning and closed-loop imitation learning strategies. The simulator shows that closed-loop execution reveals failure modes that are not visible from open-loop evaluation, highlighting the importance of closed-loop training and evaluation. The explored training strategies produced mixed but some modest improvements over the baseline, supporting the feasibility of Gaussian Splattingbased sensor-level simulation as a training platform and laying the groundwork for future work on scalable closed-loop learning for end-to-end autonomous driving.