Gauss-Newton Optimizer for 3D Gaussian Splatting Reconstruction

Abstract

Recent advancements in neural rendering have enabled highly realistic and efficient representations of 3D scenes from multi-view images. Among these, 3D Gaussian Splatting (3DGS) has emerged as a promising technique that represents scenes using a collection of 3D Gaussian primitives. Since it is a new and active field of research, progress has been made in terms of reducing reconstruction time. We present a new method of optimizing 3DGS by replacing ADAM with a Gauss-Newton (GN) optimizer integrated with the differentiable rasterizer. To save on memory, we introduce custom CUDA kernels that cache radiance and transmittance instead of explicitly storing all gradients. In addition, we introduce a sparsity-aware memory scheme that allows us to store more relevant data while minimizing waste. In each GN iteration, we compute update directions from multiple image subsets using several kernels and aggregate them through a weighted mean. Finally, a line search algorithm is used to determine the optimal update step length based on the sum of squared residuals objective function. Our optimizer converges significantly faster than ADAM per iteration, but comes with a high memory footprint. Due to GN’s computational complexity, it requires more execution time to reach a similar quality level as the original ADAM optimizer. Our GN optimizer demonstrates encouraging results during training, and with further improvements and research detailed in this paper, we believe it could become a strong competitor to ADAM.