Enhancing Emergency Operations with Sensor Fusion and AI

Abstract

In recent years, society has experienced rapid technological advancements across all fields, pushing the boundaries of what was once thought possible. Emergency services, such as fire departments, have largely been left behind. Operating in hazardous, smoke-filled environments places firefighters at extreme risk, where no compromises can be made on the tools and technologies that support them. This project addresses that gap by exploring the development of a Multi-Environment Camera (MEV-Cam) system designed to aid firefighters during rescue operations in visually impaired environments. The study focuses on the fusion of LiDAR and thermal imaging, investigating how the sensors can leverage their strengths and complement each others weaknesses to provide more reliable scene understanding. To address these challenges, four tailored datasets were created to simulate the intended application scenario, each containing synchronized data from the sensors. Fusion based models were developed for enhanced 3D visualization. For scene under standing, deep learning models were investigated to both LiDAR and thermal data, using architectures based on YOLO and PointNet. Additionally, pose estimation was explored using monocular visual odometry. A major focus of the project was on enhancing the raw collected data to ensure the sensors could be effectively used in this system. In parallel, a hardware prototype was developed to enable efficient data collection in real world environments. Results show that airborne particles significantly degrade LiDAR performance, while thermal sensors remain relatively unaffected. However, sensor fusion can compen sate for these limitations. Deep learning models demonstrated the ability to ac curately interpret scene structure under degraded conditions. After effective data enhancement, the MEV-Cam system showed improved performance across its vari ous modules. While these results highlight the promise of the technology, they also reveal current limitations and suggest several innovative directions for future work. This project marks the beginning of a new project to support life saving operations, by utilizing the latest sensor technology and AI.