Machine Learning for In-Vehicle Occupant Classification Using Radar Heatmaps and 3D Camera-Based Keypoints

Abstract

This thesis investigates how machine learning (ML) models can classify vehicle occupants as adults, forward-facing children, or rear-facing children based on interior sensor data. The goal is to improve safety systems, such as airbag deployment, by identifying the occupant type using radar and camera keypoint data. We evaluate radar-based heatmaps using a deep learning model and 3D keypoints derived from a camera using ML models, comparing their performance and data requirements. Our primary method combines a Convolutional Neural Network (CNN) with a Long Short-Term Memory (LSTM) network to analyze sequences of radar heatmaps. The architecture is trained and validated using data from 61 passengers captured in a stationary vehicle with synchronized radar and camera sensors. Additional experiments use classical ML models on body keypoints extracted from the camera. In addition. we apply incremental training and extrapolate learning curves. The best radar-only model, based on ResNet-50 + LSTM, achieved a macro F1-score of 0.867 and classification accuracy of 89.2%, with strong per-class performance (F1 = 0.831 for rear-facing children, 0.934 for forward-facing children, and 0.836 for adults). A smaller TinyCNN + LSTM model maintained competitive results while reducing memory demands. Keypoint-only classifiers reliably separated adults from forward-facing children with F1-scores above 0.96, though rear-facing cases were excluded. Overall, the results show that deep learning models can accurately classify occupant types using either radar or camera alone. We also show that more data will significantly improve performance, especially for underrepresented classes such as rear-facing children. By extrapolating learning curves, we provide practical estimates of data requirements, enabling informed decisions about dataset expansion.