Multi-View Pedestrian Detection Without Scene-Specific Fine-Tuning via 3D Foundation Models

Abstract

Multi-view pedestrian detection in occluded scenes often relies on scene-specific 3D reconstructions, which are accurate but require lengthy optimization. This thesis investigates whether an off-the-shelf 3D vision foundation model (MapAnything) can provide metric 3D point clouds for this task without scene-specific fine-tuning. We first characterize the foundation model’s geometric error, finding that calibrated inputs yield manageable scale drift correctable via ground-plane scaling, whereas uncalibrated inputs cause severe frame-to-frame distortion (averaging 66.0 cm spatial drift), degrading detection accuracy from 59.9 MODA to 29.1 MODA. To integrate this foundation-model geometry into a detector, we build a differentiable bird’s-eye view (BEV) pipeline based on Lift-Splat-Shoot and PointPillars. To handle the 30–80 cm point cloud dispersion typical of foundation models, we use dilated convolutions to enlarge the BEV backbone’s receptive field. This architecture achieves 68.7 MODA on Wildtrack and 64.0 MODA on MultiviewX. Compared to a per-scene optimized baseline (DCHM), which reaches 83.0 MODA in our singleiteration supervised reproduction and approximately 95 MODA in the best supervised results reported by its authors, the gap reflects the limitations of off-the-shelf foundation geometry. Finally, we show that auxiliary mechanisms like soft-depth sampling can cause oversmoothing, and cross-subset experiments reveal that BEV convolutional networks severely overfit to the training camera rig layout (dropping up to 28.35 MODA when testing on unseen camera subsets).