The Multimodal IMR-TrajNet - End-to-end Deep Ego Trajectory Prediction using Front-facing Camera Images and Standard Definition Maps

Abstract

Abstract Ego trajectory prediction is crucial for the development of autonomous vehicles (AV), enabling them to navigate complex environments safely and efficiently. While Highdefinition (HD) maps are commonly used due to their detailed information, they come with high computational costs and scalability issues. Standard Definition (SD) maps, on the other hand, offer a more scalable and cost-effective solution, providing sufficient conceptual context to enhance prediction models. Moreover, these maps may also contain navigation routes that further indicate the future behavior of the AV. By integrating SD maps with front-facing camera images, we can capture the real-world scenario more accurately, addressing potential inaccuracies in the maps and improving the robustness of the prediction. Our method explores different ways to extract and utilize features of SD maps for trajectory prediction, employing both Graph Neural Network (GNN)-based and transformer-based map methods for conducting this extraction. We tested various approaches for fusing the features from these modalities, including normal concatenation and cross-attention mechanisms, and for decoding the trajectories. Additionally, we introduced an auxiliary task to guide the model in learning more accurate trajectory shapes and utilized multimodal predictions to capture the inherent variability in possible driving paths. The results demonstrate a significant improvement in trajectory prediction accuracy, with our approach achieving up to four times better performance compared to the baseline model that relies solely on visual data. The inclusion of SD maps and route information not only reduces errors but also enhances the model’s robustness across various driving scenarios. These findings highlight the potential of using SD maps to advance autonomous vehicle technology.