Defect-aware 3D Geometry Reconstruction from 2D Annotations and 3D Scan Data

Abstract

Components produced through additive manufacturing require particular methods of inspection to locate and characterize surface defects on those components. Although 2D image-based defect detection is an accepted method, linking that 2D defect information to the 3D geometry of the part by reconstructing the location and severity of the defect-induced deformation of the 3D surface is challenging with limited 3D scan data and the absence of certified 3D reference geometry. This gap between 2D defect annotation and 3D defect geometry motivates this thesis. This master’s thesis develops and analyzes a pipeline that projects 2D defect annotations onto a nominal 3D CAD model to reconstruct defect-aware 3D geometry with the help of structured-light scan data from the 3D-ADAM dataset. The pipeline projects 2D defect masks to 3D nominal surface points through the sensor’s organized point cloud, registers each PLY scan to the nominal CAD, computes the signed deviation between defective and nominal scans, and finally reconstructs the defect on the CAD model in the form of a point cloud, a mesh, and a deviation map. The thesis also explores the possibility of predicting defect deviation directly on the nominal CAD using a PointNet++ model. A geometry-only formulation shows significant limitations in predicting deviation magnitude and sign as this information is unavailable in the nominal geometry. After additionally providing a coarse 2D heatmap of the deviation as input, the magnitude prediction error was reduced by around 28% relative to the geometry-only model and 36% relative to the class-average baseline, and the direction of deviation was also recovered. The accuracy of defect placement onto the 3D surface is bounded by single-view registration and partially unresolved mirror ambiguity. This thesis validates the scan-based defect mapping and reconstruction pipeline, and describes the information required to predict defect deviation. Furthermore, this thesis identifies RGB-based defect prediction to reduce reliance on the 3D scan at inference as the key direction for future research.