Existing UV mapping algorithms are designed to operate on well-behaved meshes, instead of the geometry representations produced by state-of-the-art 3D reconstruction and generation techniques. As such, applying these methods to the volume densities recovered by neural radiance fields and related techniques (or meshes triangulated from such fields) results in texture atlases that are too fragmented to be useful for tasks such as view synthesis or appearance editing. We present a UV mapping method designed to operate on geometry produced by 3D reconstruction and generation techniques. Instead of computing a mapping defined on a mesh's vertices, our method Nuvo uses a neural field to represent a continuous UV mapping, and optimizes it to be a valid and well-behaved mapping for just the set of visible points, i.e. only points that affect the scene's appearance. We show that our model is robust to the challenges posed by ill-behaved geometry, and that it produces editable UV mappings that can represent detailed appearance.
Nuvo uses a neural field to represent a given scene's UV mapping. A "chart assignment" MLP c outputs probabilities for a categorical distribution over charts for any surface point x, "texture coordinate" MLPs t map from 3D points x to 2D UV coordinates u, and ``surface coordinate'' MLPs s map from 2D UV coordinates to 3D points on the surface. Here we visualize Nuvo's learned mappings for charts 1 and 3 in an atlas consisting of 4 charts.
Nuvo parameterizes a UV mapping as a neural field that is optimized to partition the scene into a set of charts, each representing a low-distortion mapping from a region of the scene onto a 2D square. Here we're visualizing Nuvo's optimization convergence for computing UV mappings for the "Bust of Queen Nefertiti" mesh using 4, 8, or 16 charts.
Nuvo's UV mappings enable us to use off-the-shelf 2D image editing tools to edit the appearance of generated and 3D reconstructed content. Here, we're modifying mesh albedos in UV space using Adobe Firefly's generative inpainting.
Thanks to Rick Szeliski and Peter Hedman for feedback and comments.
The "Bust of Queen Nefertiti" mesh shown above was downloaded from Alec Jacobson's collection of 3D models, and was created from a 3D scan of Thutmose's statue, located in the Neues Museum in Berlin. The "Hydria Apothecary Vase" mesh seen in our video (downloaded from Sketchfab) is from the Pharmacy Museum in the Jagiellonian University Medical College of Kraków, Poland, digitized by the Regional Digitalisation Lab of the Malopolska Institute of Culture.
@article{srinivasan2023nuvo,
author = {Pratul P. Srinivasan and Stephan J. Garbin and Dor Verbin and Jonathan T. Barron and Ben Mildenhall},
title = {Nuvo: Neural UV Mapping for Unruly 3D Representations},
journal = {arXiv},
year = {2023},
}