Eye in the Sky - Vision System for Automatic Detection and Tracking of Flying Objects

Abstract

The widespread deployment of Unmanned Aerial Vehicles (UAVs) in both civilian airspace and military operations has made onboard perception an increasingly critical capability. Detecting other airborne objects from a moving platform is particularly challenging: targets are often small, the camera itself is in motion, and IR imagery introduces additional complexity due to scarce training data. This thesis presents a modular system designed to handle these constraints in real-time, using both RGB and IR cameras mounted on a UAV. Object detection is handled by RF-DETR, a transformer-based detector suitable for a real-time application. For tracking, we extend ByteTrack with optical-flow based ego-motion compensation, which proves important when the camera platform itself is moving. Classification relies on a CLIP-based model that we adapt to IR through cross-modal knowledge distillation combined with Low-Rank Adaptation (LoRA) fine-tuning, allowing us to transfer visual representations from RGB to IR without requiring pretraining on IR imagery. Experiments on simulated UAV imagery show that the RGB detector achieves an F1 score of 0.924 and the IR detector 0.824, despite the latter having access to substantially less training data. The classifier adaptation has a large effect on drone recall, which increases from 0.252 to 0.654 in RGB mode and from 0.049 to 0.617 in IR mode. Tracking performance, measured with the HOTA metric, reaches around 20 in RGB mode and 14 in IR mode, with most errors caused by long occlusions and very distant targets. The complete pipeline runs at approximately 21 fps on a standard laptop GPU, suggesting that it is capable of real-world deployment. The initial results are promising and indicate that the proposed system can handle several real-world constraints. However, two key limitations remain: performance drops on far-range targets, and all evaluation is performed in simulation with pseudo-IR imagery, leaving a gap to real-world deployment.