Synthetic Data Generation for Industrial Perception Using 3DGS

Abstract

Autonomous mobile robots operating in industrial environments require reliable camera-based perception to detect pedestrians, forklifts, and other vehicles in dynamic shared workspaces. However, collecting and manually annotating sufficiently large and diverse real-world datasets in factory environments is costly and timeconsuming. This thesis investigates the use of 3D Gaussian Splatting (3DGS) as the basis of a synthetic-data-generation pipeline for training industrial object-detection models and examines factors affecting simulation-to-reality transfer. A 3DGS reconstruction of a real factory environment was integrated into Unreal Engine as a digital-twin background and combined with dynamic simulated actors representing pedestrians, forklifts, and other vehicles. Synthetic images and corresponding object-detection annotations were generated from multiple virtual camera viewpoints and compiled into datasets for training YOLOv8-based detection models. A model trained on a 21-camera synthetic dataset achieved an mAP50 of 0.971 on synthetic test data but only 0.227 on real factory images, demonstrating a substantial simulation-to-reality domain gap. Experiments showed that both camera-viewpoint diversity and environmental representation affected real-world generalisation. Models trained using the 3DGS factory representation transferred better to real images than those trained using a conventional Unreal Engine environment. However, because the appearance encoded by 3DGS is tied to the lighting conditions under which the scene was captured, lighting differences could not be varied independently, limiting the extent to which this improvement could be attributed solely to the background representation. Diffusionbased style transfer substantially improved zero-shot real-domain performance, increasing mAP50 from 0.125 to 0.440, while subsequent pseudo-labelling increased it further to 0.504 without requiring manually annotated real training images. Synthetic pre-training followed by fine-tuning on real data increased mean mAP50 from 0.876 to 0.891 compared with real-only fine-tuning, although this difference fell within the observed variability across data splits. In contrast, joint training with style-adapted synthetic and real data produced more substantial gains when only limited annotated real data was available. Overall, the results indicate that 3DGSbased synthetic data is most valuable as a complement to real-world data rather than as a replacement for it, particularly when viewpoint diversity, visual domain alignment, and training strategy are considered together.