Captioning Engine for AD/ADAS Data using Multi-Modal Large Language Models

Abstract

Autonomous Driving (AD) and Advanced Driver Assistance Systems (ADAS) are dependent on perception models that can understand and interpret their environment and surroundings. These models require a large amount of annotated data to achieve this goal. However, it is often difficult to curate large datasets as it is time-consuming, most footage is often redundant, and it is hard to find the rare edge cases that matter the most. Vision-language models (VLMs) offer a scalable alternative to automate the process of generating captions for driving scenarios. This thesis investigates how such captions can be used through representation learning for downstream tasks. First, a captioning engine combines rule-based detection of driving maneuvers (lane changes, cut-ins and cut-outs) with a VLM to generate detailed captions describing the environment and interactions in the scene. The rule-based component compensates for the VLM’s limited ability to reason about temporal progression across frames, while the VLM contributes rich descriptions of the surrounding scene. Secondly, the resulting video and caption pairs are used to fine-tune two contrastive vision-language embedding models, CLIP (ViT-L/14) and the Perception Encoder (L14-336), with the goal to align the captions and videos in a shared representation space. To our knowledge, this is the first use of the Perception Encoder in the AD/ADAS domain. Our results show that a rule-based classifier on dash-cam sequences could be effectively used as context enrichment of the caption generating VLM. Fine-tuning on these detailed and rich captions increases text-to-video Recall@1 from 25% to 83%. CLIP outperforms the Perception Encoder as the choice of backbone across all metrics and all maneuver types after fine-tuning. Further testing is needed to evaluate the generalization of this method beyond the maneuvers and dataset considered here. Future work includes extending the rule based system to additional maneuvers, such as lead-vehicle braking and vulnerable road-user interactions, and evaluating whether the learned representations transfer to other datasets.