TransFURmer: Hair and Fur rendering using SwinIR - Transformer-based Neural Rendering of Hair and Fur using SwinIR

Abstract

While ray tracing algorithms offer photorealistic rendering of hair and fur, their high computational cost makes them unfeasible for real-time applications. Conversely, fast rendering techniques such as rasterization fail to capture accurate lighting effects and high-frequency details. This thesis investigates the feasibility of a Transformer-based pipeline tailored for rendering hair and fur. A SwinIR-based architecture is adapted within a Conditional Generative Adversarial Network (CGAN) framework to translate input buffers into high-quality renders. Trained on the SyntheticFur dataset, the model utilizes a combination of losses, notably Frequency Domain (FFT) loss, Learned Perceptual Image Patch Similarity (LPIPS) loss, and Wasserstein GAN with gradient penalty (WGAN-GP) loss, to accurately reconstruct the fine details and improve perceptual realism. The final model successfully generalizes to unseen geometry and the implemented FFT and LPIPS loss proved highly effective at preserving the texture of individual hair strands. However, the current model yields an average throughput of 0.13 fps, failing to satisfy real-time latency constraints. Additionally, hardware limits restricted the training crop size, and the training dataset was limited to simple primitives lacking complex self-shadowing scenarios. This likely contributed to shading inconsistencies and visual artifacts observed in the final renders. Despite these limitations, the final model is a faster alternative to path tracing for offline rendering tasks, and demonstrates the potential viability of Transformer-based neural rendering for complex hair and fur.