Statistical Evaluation of Radar Simulation Models Towards Real Data

Abstract

This thesis aimed to create a method that could be used by Volvo Car Corporation (VCC) to statistically evaluate the truthfulness and accuracy of the built-in radar models used by VCC. Using pre-collected data from European New Car Assessment Programme (Euro NCAP) scenarios—specifically Car-to-Car Rear moving (CCRm) and Car-to-Bicyclist Nearside Adult Obstructed (CBNAO) — a method was developed to extract real world data, enabling the recreation of actual scenarios within a simulated environment. Furthermore, a comparison between the simulated and real data was conducted. This comparison was conducted using the statistical metric Double Validation Metric (DVM), which is a combination of the Area Validation Metric (AVM), Corrected Area Validation Metric (CAVM), and the model bias within the simulated data. The developed method includes parsing of Hierarchical Data Format version 5 (HDF5) files, enabling reading and manipulating these files. It also features a Graphical User Interface (GUI) that reads one real file and one simulated file, visualizes the desired detection columns within a given timestamp interval, and a statistical evaluator that performs all the calculations, plots the Empirical Cumulative Distribution Function (eCDF), and analyzes their statistical characteristics. The results demonstrated tendencies pointing towards both reliability and unreliability. Different parameters showed varying degrees of correlation for different scenarios, runs, and speeds. However, one clear trend was the high tendency for detection differences between the simulated model and the real model, with the simulated model outnumbering the real model. Based on this, the developed method can be used to some extent to relay information about whether the radar model is trustworthy. However, its effectiveness heavily depends on the specific area of usage. In conclusion, the method effectively visualizes the detections, indicating where they occur and how they differ in number. While the model may not be suitable for final certification, it can be a valuable tool for statistical approximation of the radar model’s performance.