NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis CVPR 2021
- Pratul P. Srinivasan Google Research
- Boyang Deng Google Research
- Xiuming Zhang MIT
- Matthew Tancik UC Berkeley
- Ben Mildenhall UC Berkeley
- Jonathan T. Barron Google Research
Example Results: Relighting and View Synthesis
Example Results: Material Editing
Abstract
We present a method that takes as input a set of images of a scene illuminated by unconstrained known lighting, and produces as output a 3D representation that can be rendered from novel viewpoints under arbitrary lighting conditions. Our method represents the scene as a continuous volumetric function parameterized as MLPs whose inputs are a 3D location and whose outputs are the following scene properties at that input location: volume density, surface normal, material parameters, distance to the first surface intersection in any direction, and visibility of the external environment in any direction. Together, these allow us to render novel views of the object under arbitrary lighting, including indirect illumination effects. The predicted visibility and surface intersection fields are critical to our model's ability to simulate direct and indirect illumination during training, because the brute-force techniques used by prior work are intractable for lighting conditions outside of controlled setups with a single light. Our method outperforms alternative approaches for recovering relightable 3D scene representations, and performs well in complex lighting settings that have posed a significant challenge to prior work.
Technical Video
BibTeX
Acknowledgements
Our implementation is based on Boyang Deng's awesome JAX implementation of NeRF.The website template was borrowed from Michaël Gharbi.