pith. sign in

arxiv: 2606.26715 · v2 · pith:WIZBFPJZnew · submitted 2026-06-25 · 💻 cs.CV · cs.GR

Extracting Neural Materials from Multi-view Images

Kim Youwang , Jon Hasselgren , Peter Kocsis , Andrea Weidlich , Tae-Hyun Oh , Jacob Munkberg This is my paper

Pith reviewed 2026-06-29 05:07 UTC · model grok-4.3

classification 💻 cs.CV cs.GR
keywords neural materialsinverse renderingmaterial extractionmulti-view imagesdifferentiable renderingpath tracingmaterial decompositionuncertainty estimation
0
0 comments X

The pith

NeuMatEx extracts spatially varying neural materials from multi-view images by combining a learned reconstruction model with inverse path tracing.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents NeuMatEx as a method to acquire neural materials, which represent complex specular and scattering effects in a compact form. Direct optimization fails due to the nonlinear latent space, so the approach first trains a Large Material Reconstruction Model to output base color, material latents, and per-pixel uncertainty from input images. These outputs initialize and guide a subsequent differentiable inverse rendering step that uses path tracing. The uncertainty term anchors confident regions to the prior prediction, reducing the risk that lighting or reflections get baked into the extracted material. Experiments indicate the resulting materials show improved visual fidelity and decomposition compared with standard physically based rendering pipelines on both synthetic and captured data.

Core claim

NeuMatEx extracts neural materials by training a Large Material Reconstruction Model to predict base color, neural latents, and aleatoric uncertainty directly from images, then refining the prediction through inverse path tracing while using the uncertainty map to constrain high-confidence areas and prevent lighting effects from being absorbed into the material parameters.

What carries the argument

Large Material Reconstruction Model (LMRM) that supplies initialization of base color and neural latents plus uncertainty guidance for the inverse path tracing stage.

If this is right

  • Materials extracted by NeuMatEx render with higher visual quality than those produced by PBR-based inverse rendering.
  • The method yields cleaner separation of material properties from lighting and view-dependent effects.
  • The pipeline operates on both synthetic renderings and real captured multi-view photographs.
  • Uncertainty weighting reduces unwanted transfer of specular highlights and shadows into the recovered material maps.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same prior-plus-optimization structure could be tested on single-image or sparse-view capture settings where multi-view consistency is weaker.
  • Replacing the LMRM with a lighter network might trade some accuracy for faster per-asset processing in production pipelines.
  • The uncertainty output could serve as a diagnostic for regions where additional views or different lighting would most improve the reconstruction.

Load-bearing premise

The nonlinear geometry of neural material latent spaces renders direct optimization from images ineffective without an external learned prior for both starting values and uncertainty weighting.

What would settle it

Perform the full inverse path tracing optimization on the paper's test scenes while disabling the LMRM initialization and uncertainty term, then compare final render error and material separation quality against the reported NeuMatEx results.

Figures

Figures reproduced from arXiv: 2606.26715 by Andrea Weidlich, Jacob Munkberg, Jon Hasselgren, Kim Youwang, Peter Kocsis, Tae-Hyun Oh.

Figure 1
Figure 1. Figure 1: NeuMatEx extracts Neural Materials from multi-view images, a richer material representation that goes beyond PBR. Our differentiable inverse rendering method decomposes Lambertian diffuse lobe and “neural” specular lobes for complex real world material￾light interactions such as haze, dust, clearcoat, fuzz, scatter, and even their mixtures, while being able to be path-traced in real-time rates. Abstract Ne… view at source ↗
Figure 2
Figure 2. Figure 2: NeuMatEx consists of two stages. (a) Neural Material Initialization: Given input images and a 3D mesh geometry, a Large Material Reconstruction Model (LMRM) predicts a feature triplane in a single forward pass. The triplane is decoded by two MLPs to jointly predict an initial neural material and per-material uncertainty that reflect ambiguous surface regions. (b) Test-Time Optimization (TTO): We further op… view at source ↗
Figure 3
Figure 3. Figure 3: Why do we need test-time optimization? While our feed-forward prediction gives reasonable initial materials (a), test￾time optimization (b) recovers finer details (top) and corrects color shifts and material decomposition in challenging cases (bottom). where β = 0.5 and ⌊·⌋ denotes the stop-gradient operation, applied per material channel. We apply the stop-gradient to the material mean Gmat inside Lunc so… view at source ↗
Figure 4
Figure 4. Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative results. Our method, NeuMatEx, faithfully reconstructs a wide variety of neural materials from images. Reconstructed neural materials demonstrate complex multi-lobe effects including haze, clearcoat, dust, fuzz, scattering and even their combinations. uncertainty-guided material regularizer (Eq. 14), arg min T Lphoto + λreg Lreg, (15) where λreg weights uncertainty-guided material regulariza￾ti… view at source ↗
Figure 6
Figure 6. Figure 6: Rendering performance. For a set of reference neural materials, and the runtime performance for path tracing in 1080p resolution with 1 spp and 10 bounces is ∼4 ms on an RTX 5090. real-time performance within a path-tracing context. Please refer to Zeltner et al. [60] for a thorough analysis of neural material runtime performance and implementation details. 6.2. Comparison with PBR-based Methods As represe… view at source ↗
Figure 7
Figure 7. Figure 7: PBR vs. Neural Material (128 spp, relit). All methods use known fixed geometry. For the optimization methods, we use known poses and lighting. Hunyuan3D-2.1 [48] and TRELLIS.2 [54] are monocular feed-forward PBR estimation models included to highlight the limitations of PBR, and do not represent a fair comparison to optimization methods. NVDiffRecMC++ is our own extension of NVDiffRecMC [18] with a feed-fo… view at source ↗
Figure 8
Figure 8. Figure 8: Effects of uncertainty-guided regularization. Uncon￾strained test-time optimization (column 2) suffers from lighting baked into base color and neural latents. Our uncertainty-guided regularization (column 3) yields well-behaved material intrinsics with improved details over the LMRM initialization (see crops) [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 11
Figure 11. Figure 11: Limitations. We build on top of a pre-trained neural material basis [58], inheriting its limitations. This is apparent in the above real-world examples from the DTC [9] dataset, where the pre￾dicted materials lie outside the valid domain of the basis, resulting in specular artifacts. Our test-time-optimization mostly mitigates this, but artifacts may still be observed, e.g., in the crevices. teapot. In co… view at source ↗
Figure 9
Figure 9. Figure 9: Real-world captures. We apply NeuMatEx to real-world captures from the DTC [9] dataset. Despite the DTC dataset being composed mostly of simple materials and not designed with neural material reconstruction in mind, our extracted materials capture effects beyond the standard PBR, e.g., clearcoat on the teapot [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Relighting real-world neural materials. Relit ren￾derings of our extracted neural materials show that the recovered reflectance generalizes across illumination conditions, faithfully reproducing complex specular behavior without baked-in effects. estimate the initial material. During test-time optimization, we optimize against all real photographs to avoid baking any novel-view-synthesis artifacts. We com… view at source ↗
read the original abstract

Neural materials can represent complex specular reflections and scattering effects in a compact, universal basis. However, acquiring and authoring such materials remains challenging. We present NeuMatEx, a differentiable inverse rendering method for extracting spatially varying neural materials from images. The nonlinear structure of neural material latent spaces makes optimization with naive inverse rendering infeasible. To address this, we train a Large Material Reconstruction Model (LMRM) that directly predicts initialbase color, neural material latents, and aleatoric uncertainty guides from images. This material prior provides a good initialization and better constrains our subsequent optimization using inverse path tracing. The predicted uncertainty further helps by anchoring high-confidence regions more tightly to the LMRM prediction, preventing lighting and complex specular effects from being baked into materials. Experiments on synthetic and real assets show that NeuMatEx extracts complex materials with better visual quality and material decomposition than PBR-based methods.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper introduces NeuMatEx, a differentiable inverse-rendering pipeline for recovering spatially varying neural materials from multi-view images. It trains a Large Material Reconstruction Model (LMRM) that predicts base color, neural-material latents, and aleatoric uncertainty from images; these predictions initialize and regularize a subsequent inverse-path-tracing optimization. The central premise is that the nonlinearity of neural-material latent spaces renders direct gradient-based optimization infeasible, so the LMRM prior is required. Experiments on synthetic and real assets are claimed to yield higher visual quality and better material decomposition than PBR baselines.

Significance. A validated method that reliably extracts complex neural materials would be useful for graphics pipelines that already employ neural representations. The two-stage design (learned prior plus physics-based refinement) is conceptually coherent, but the manuscript supplies neither quantitative metrics nor the ablation evidence needed to establish that the LMRM stage is necessary or that the reported gains exceed what the neural representation alone would achieve.

major comments (2)
  1. [Abstract] Abstract: the claim that 'the nonlinear structure of neural material latent spaces makes optimization with naive inverse rendering infeasible' is presented without any supporting experiment (failed GD runs on latents alone, loss curves comparing random vs. LMRM initialization, or quantification of lighting bake-in). This premise is load-bearing for the entire two-stage architecture.
  2. [Abstract] Abstract: the statement that NeuMatEx 'extracts complex materials with better visual quality and material decomposition than PBR-based methods' is unsupported by any numerical metrics, error bars, tables, or protocol details on data exclusion or scene selection. The central empirical claim therefore cannot be evaluated.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the abstract claims. We address each major comment below and will revise the manuscript accordingly to provide the requested supporting evidence.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that 'the nonlinear structure of neural material latent spaces makes optimization with naive inverse rendering infeasible' is presented without any supporting experiment (failed GD runs on latents alone, loss curves comparing random vs. LMRM initialization, or quantification of lighting bake-in). This premise is load-bearing for the entire two-stage architecture.

    Authors: We agree that the abstract presents this premise without direct experimental support. In the revised manuscript we will add an ablation subsection (new Figure and accompanying text) that reports failed direct gradient-descent runs on the latent codes, loss curves contrasting random versus LMRM initialization, and qualitative examples of lighting bake-in when the uncertainty term is ablated. These additions will be placed in Section 4 to justify the two-stage design. revision: yes

  2. Referee: [Abstract] Abstract: the statement that NeuMatEx 'extracts complex materials with better visual quality and material decomposition than PBR-based methods' is unsupported by any numerical metrics, error bars, tables, or protocol details on data exclusion or scene selection. The central empirical claim therefore cannot be evaluated.

    Authors: The current version relies on qualitative visual comparisons. We acknowledge that quantitative metrics and protocol details are required. In the revision we will insert a new results table reporting PSNR, SSIM, and per-material decomposition errors with error bars over multiple runs, together with a clear description of the evaluation protocol (scene selection criteria, data exclusion rules, and rendering settings). These changes will appear in Section 5. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation self-contained

full rationale

The paper's core method (NeuMatEx) relies on training an LMRM prior to initialize and constrain inverse path-tracing optimization over neural material latents. No equations, fitted parameters renamed as predictions, or self-citation chains are presented that reduce the claimed extraction results to inputs by construction. The premise that naive optimization is infeasible is stated as motivation but is not derived from prior results within the paper; experimental comparisons to PBR baselines supply the supporting evidence. This is the common case of an independent learned component plus empirical validation.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The central claim rests on the existence and effectiveness of a pre-trained LMRM that is not further detailed; no explicit free parameters, axioms, or invented entities are enumerated in the abstract.

pith-pipeline@v0.9.1-grok · 5693 in / 1032 out tokens · 20463 ms · 2026-06-29T05:07:14.212004+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

67 extracted references · 8 canonical work pages · 2 internal anchors

  1. [1]

    Inverse path tracing for joint material and lighting estimation

    Dejan Azinovic, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner. Inverse path tracing for joint material and lighting estimation. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2

    2019

  2. [2]

    Neural reflectance fields for appearance acquisition.arXiv preprint, 2008.03824,

    Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Milo ˇs Ha ˇsan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition.arXiv preprint, 2008.03824,

    work page arXiv 2008

  3. [3]

    Taming optimization vari- ance in compact neural shading networks

    Benedikt Bitterli, Petrik Clarberg, Chris Cummings, Aaron Lefohn, Steve Marschner, Jan Nov´ak, Fabrice Rousselle, An- drea Weidlich, and Tizian Zeltner. Taming optimization vari- ance in compact neural shading networks. InACM SIG- GRAPH Conference Proceedings, 2026. 2

    2026

  4. [4]

    Two-shot spatially-varying BRDF and shape estimation

    Mark Boss, Varun Jampani, Kihwan Kim, Hendrik Lensch, and Jan Kautz. Two-shot spatially-varying BRDF and shape estimation. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2

    2020

  5. [5]

    Neural-pil: Neural pre- integrated lighting for reflectance decomposition

    Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, and Hendrik Lensch. Neural-pil: Neural pre- integrated lighting for reflectance decomposition. InAdvances in Neural Information Processing Systems (NeurIPS), 2021. 2

    2021

  6. [6]

    Physically-based shading at Disney

    Brent Burley. Physically-based shading at Disney. InACM SIGGRAPH Conference Proceedings, pages 1–7, 2012. 3

    2012

  7. [7]

    DIB-R++: Learning to predict lighting and material with a hybrid differentiable renderer

    Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khalis, Or Litany, and Sanja Fidler. DIB-R++: Learning to predict lighting and material with a hybrid differentiable renderer. InAdvances in Neural Information Processing Systems (NeurIPS), 2021. 2

    2021

  8. [8]

    Objaverse: A Uni- verse of Annotated 3D Objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A Uni- verse of Annotated 3D Objects. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 4

    2023

  9. [9]

    Digital Twin Catalog: A Large-Scale Photoreal- istic 3D Object Digital Twin Dataset

    Zhao Dong, Ka Chen, Zhaoyang Lv, Hong-Xing Yu, Yunzhi Zhang, Cheng Zhang, Yufeng Zhu, Stephen Tian, Zhengqin Li, Geordie Moffatt, Sean Christofferson, James Fort, Xiaqing Pan, Mingfei Yan, Jiajun Wu, Carl Yuheng Ren, and Richard Newcombe. Digital Twin Catalog: A Large-Scale Photoreal- istic 3D Object Digital Twin Dataset. InIEEE Conference on Computer Vis...

    2025

  10. [10]

    Neural Layered BRDFs

    Jiahui Fan, Beibei Wang, Miloˇs Haˇsan, Jian Yang, and Ling- Qi Yan. Neural Layered BRDFs. InACM SIGGRAPH Con- ference Proceedings, 2022. 1, 2, 3

    2022

  11. [11]

    RNG: Relightable Neural Gaussians

    Jiahui Fan, Fujun Luan, Jian Yang, Milos Hasan, and Beibei Wang. RNG: Relightable Neural Gaussians. InIEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR),

  12. [12]

    Deep Inverse Rendering for High-Resolution SVBRDF Estimation from an Arbitrary Number of Images.ACM Trans- actions on Graphics, 38(4), 2019

    Duan Gao, Xiao Li, Yue Dong, Pieter Peers, Kun Xu, and Xin Tong. Deep Inverse Rendering for High-Resolution SVBRDF Estimation from an Arbitrary Number of Images.ACM Trans- actions on Graphics, 38(4), 2019. 2

    2019

  13. [13]

    Relightable 3D Gaussian: Real- time Point Cloud Relighting with BRDF Decomposition and Ray Tracing

    Jian Gao, Chun Gu, Youtian Lin, Hao Zhu, Xun Cao, Li Zhang, and Yao Yao. Relightable 3D Gaussian: Real- time Point Cloud Relighting with BRDF Decomposition and Ray Tracing. InEuropean Conference on Computer Vision (ECCV), 2024. 2

    2024

  14. [14]

    Linear Light Source Reflectometry.ACM Transactions on Graphics, 22(3):749–758, 2003

    Andrew Gardner, Chris Tchou, Tim Hawkins, and Paul De- bevec. Linear Light Source Reflectometry.ACM Transactions on Graphics, 22(3):749–758, 2003. 2

    2003

  15. [15]

    Wil- son, and Paul Debevec

    Abhijeet Ghosh, Tongbo Chen, Pieter Peers, Cyrus A. Wil- son, and Paul Debevec. Estimating Specular Roughness and Anisotropy from Second Order Spherical Gradient Illumina- tion.Computer Graphics Forum, 28(4):1161–1170, 2009

    2009

  16. [16]

    Guarnera, G

    D. Guarnera, G. C. Guarnera, A. Ghosh, C. Denk, and M. Glencross. BRDF Representation and Acquisition. InPro- ceedings of the 37th Annual Conference of the European Association for Computer Graphics: State of the Art Reports, pages 625–650, 2016. 2

    2016

  17. [17]

    MaterialGAN: Reflectance Capture Us- ing a Generative SVBRDF Model.ACM Transactions on Graphics, 39(6), 2020

    Yu Guo, Cameron Smith, Miloˇs Haˇsan, Kalyan Sunkavalli, and Shuang Zhao. MaterialGAN: Reflectance Capture Us- ing a Generative SVBRDF Model.ACM Transactions on Graphics, 39(6), 2020. 2

    2020

  18. [18]

    Shape, Light, and Material Decomposition from Images us- ing Monte Carlo Rendering and Denoising

    Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, Light, and Material Decomposition from Images us- ing Monte Carlo Rendering and Denoising. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. 2, 4, 5, 6, 7

    2022

  19. [19]

    VideoMatGen: PBR Materials through Joint Gen- erative Modeling

    Jon Hasselgren, Zheng Zeng, Milo ˇs Ha ˇsan, and Jacob Munkberg. VideoMatGen: PBR Materials through Joint Gen- erative Modeling. InIEEE Conference on Computer Vision and Pattern Recognition Findings (CVPRF), 2026. 2

    2026

  20. [20]

    2D Gaussian Splatting for Geometrically Accurate Radiance Fields

    Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2D Gaussian Splatting for Geometrically Accurate Radiance Fields. InACM SIGGRAPH Conference Proceedings. Association for Computing Machinery, 2024. 8

    2024

  21. [21]

    GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflec- tive Surfaces

    Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaox- iao Long, Wenping Wang, and Yuexin Ma. GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflec- tive Surfaces. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

    2024

  22. [22]

    TensoIR: Tensorial Inverse Rendering

    Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Song- fang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, and Hao Su. TensoIR: Tensorial Inverse Rendering. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2

    2023

  23. [23]

    James T. Kajiya. The rendering equation. InACM SIGGRAPH Conference Proceedings. ACM Press, 1986. 4

    1986

  24. [24]

    Real shading in Unreal Engine 4.ACM SIG- GRAPH Course on Physically Based Shading Theory and Practice, 4(3):1, 2013

    Brian Karis. Real shading in Unreal Engine 4.ACM SIG- GRAPH Course on Physically Based Shading Theory and Practice, 4(3):1, 2013. 3

    2013

  25. [25]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. InInternational Conference on Learning Representations (ICLR), 2015. 5

    2015

  26. [26]

    Intrin- sic image diffusion for indoor single-view material estimation

    Peter Kocsis, Vincent Sitzmann, and Matthias Nießner. Intrin- sic image diffusion for indoor single-view material estimation. 10 InIEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2024. 2

    2024

  27. [27]

    Intrinsic image fusion for multi-view 3d material reconstruction

    Peter Kocsis, Lukas H¨ollein, and Matthias Nießner. Intrinsic image fusion for multi-view 3d material reconstruction. In IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2026. 2

    2026

  28. [28]

    NeuMIP: Multi-Resolution Neural Materials.ACM Transactions on Graphics, 40(4),

    Alexandr Kuznetsov, Krishna Mullia, Zexiang Xu, Milo ˇs Haˇsan, and Ravi Ramamoorthi. NeuMIP: Multi-Resolution Neural Materials.ACM Transactions on Graphics, 40(4),

  29. [29]

    Lensch, Jochen Lang, Asla M

    Hendrik P.A. Lensch, Jochen Lang, Asla M. Sa, and Hans- Peter Seidel. Planned sampling of spatially varying BRDFs. Computer Graphics Forum, 2003. 2

    2003

  30. [30]

    Learning to recon- struct shape and spatially-varying reflectance from a single image

    Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Learning to recon- struct shape and spatially-varying reflectance from a single image. InACM SIGGRAPH Asia Conference Proceedings, page 269. ACM, 2018. 2

    2018

  31. [31]

    Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image

    Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2

    2020

  32. [32]

    Marshall, Yufeng Ren, Richard Newcombe, and Zhao Dong

    Zhengqin Li, Dilin Wang, Ka Chen, Zhaoyang Lv, Thu Nguyen-Phuoc, Milim Lee, Jia bin Huang, Lei Xiao, Cheng Zhang, Yufeng Zhu, Carl S. Marshall, Yufeng Ren, Richard Newcombe, and Zhao Dong. LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fields. InIEEE Conference on Computer Vision and Patt...

    2025

  33. [33]

    Gs-ir: 3d gaussian splatting for inverse rendering

    Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. Gs-ir: 3d gaussian splatting for inverse rendering. In IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2024. 2

    2024

  34. [34]

    IRIS: inverse ren- dering of indoor scenes from low dynamic range images

    Chih-Hao Lin, Jia-Bin Huang, Zhengqin Li, Zhao Dong, Christian Richardt, Tuotuo Li, Michael Zollh¨ofer, Johannes Kopf, Shenlong Wang, and Changil Kim. IRIS: inverse ren- dering of indoor scenes from low dynamic range images. In IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2025. 2

    2025

  35. [35]

    MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors

    Yehonathan Litman, Or Patashnik, Kangle Deng, Aviral Agrawal, Rushikesh Zawar, Fernando De la Torre, and Shub- ham Tulsiani. MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors. InInternational Conference on 3D Vision (3DV), 2025. 2, 4

    2025

  36. [36]

    Unified shape and svbrdf recovery using differentiable monte carlo rendering.arXiv preprint, 2103.15208, 2021

    Fujun Luan, Shuang Zhao, Kavita Bala, and Zhao Dong. Unified shape and svbrdf recovery using differentiable monte carlo rendering.arXiv preprint, 2103.15208, 2021. 2

    work page arXiv 2021

  37. [37]

    RNA: Relightable Neural Assets.ACM Transactions on Graphics, 44(1), 2024

    Krishna Mullia, Fujun Luan, Xin Sun, and Miloˇs Haˇsan. RNA: Relightable Neural Assets.ACM Transactions on Graphics, 44(1), 2024. 2, 3

    2024

  38. [38]

    Extracting Triangular 3D Models, Materials, and Lighting From Images

    Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas M ¨uller, and Sanja Fidler. Extracting Triangular 3D Models, Materials, and Lighting From Images. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2

    2022

  39. [39]

    VideoMat: Extracting PBR Materials from Video Diffusion Models

    Jacob Munkberg, Zian Wang, Ruofan Liang, Tianchang Shen, and Jon Hasselgren. VideoMat: Extracting PBR Materials from Video Diffusion Models. InEurographics Symposium on Rendering (EGSR), 2025. 2

    2025

  40. [40]

    Material and Lighting Reconstruction for Complex Indoor Scenes with Texture-space Differentiable Rendering

    Merlin Nimier-David, Zhao Dong, Wenzel Jakob, and An- ton Kaplanyan. Material and Lighting Reconstruction for Complex Indoor Scenes with Texture-space Differentiable Rendering. InEurographics Symposium on Rendering - DL- only Track. The Eurographics Association, 2021. 2

    2021

  41. [41]

    Scalable Diffusion Models with Transformers

    William Peebles and Saining Xie. Scalable Diffusion Models with Transformers. InIEEE International Conference on Computer Vision (ICCV), 2023. 3

    2023

  42. [42]

    Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 4rd edition, 2023

    Matt Pharr, Wenzel Jakob, and Greg Humphreys.Physically Based Rendering: From Theory to Implementation. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 4rd edition, 2023. 5

    2023

  43. [43]

    Generative Neural Materials.ACM Transactions on Graphics, 43:11,

    Nithin Raghavan, Krishna Mullia, Alexander Trevithick, Fu- jun Luan, Miloˇs Haˇsan, and Ravi Ramamoorthi. Generative Neural Materials.ACM Transactions on Graphics, 43:11,

  44. [44]

    Neural Precomputed Radiance Transfer.Computer Graphics Forum, 41(2), 2022

    Gilles Rainer, Adrien Bousseau, Tobias Ritschel, and George Drettakis. Neural Precomputed Radiance Transfer.Computer Graphics Forum, 41(2), 2022. 2

    2022

  45. [45]

    On joint estimation of pose, ge- ometry and svBRDF from a handheld scanner

    Carolin Schmitt, Simon Donne, Gernot Riegler, Vladlen Koltun, and Andreas Geiger. On joint estimation of pose, ge- ometry and svBRDF from a handheld scanner. InIEEE Con- ference on Computer Vision and Pattern Recognition (CVPR),

  46. [46]

    On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks

    Maximilian Seitzer, Arash Tavakoli, Dimitrije Antic, and Georg Martius. On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks. InInterna- tional Conference on Learning Representations (ICLR), 2022. 4

    2022

  47. [47]

    Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials

    Yawar Siddiqui, Tom Monnier, Filippos Kokkinos, Mahendra Kariya, Yanir Kleiman, Emilien Garreau, Oran Gafni, Natalia Neverova, Andrea Vedaldi, Roman Shapovalov, and David Novotny. Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials. InAd- vances in Neural Information Processing Systems (NeurIPS),

  48. [48]

    Hunyuan3D 2.1: From Images to High-Fidelity 3D Assets with Production-Ready PBR Material

    Tencent Hunyuan3D Team. Hunyuan3D 2.1: From Images to High-Fidelity 3D Assets with Production-Ready PBR Mate- rial.arXiv preprint, 2506.15442, 2025. 2, 6, 7

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  49. [49]

    Eric Veach and Leonidas J. Guibas. Optimally Combining Sampling Techniques for Monte Carlo Rendering. InACM SIGGRAPH Conference Proceedings, pages 419–428, 1995. 5

    1995

  50. [50]

    Wan: Open and Advanced Large-Scale Video Generative Models

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint, 2503.20314, 2025. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  51. [51]

    CRM: Single Image to 3D Textured Mesh with Convo- lutional Reconstruction Model

    Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. CRM: Single Image to 3D Textured Mesh with Convo- lutional Reconstruction Model. InEuropean Conference on Computer Vision (ECCV), 2024. 4 11

    2024

  52. [52]

    MeshLRM: Large Reconstruction Model for High- Quality Meshes.arXiv preprint, 2404.12385, 2024

    Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexi- ang Xu. MeshLRM: Large Reconstruction Model for High- Quality Meshes.arXiv preprint, 2404.12385, 2024. 3

    work page arXiv 2024

  53. [53]

    Advances in Ge- ometry and Reflectance Acquisition (Course Notes)

    Michael Weinmann and Reinhard Klein. Advances in Ge- ometry and Reflectance Acquisition (Course Notes). InSIG- GRAPH Asia 2015 Courses, 2015. 2

    2015

  54. [54]

    Native and Compact Structured Latents for 3D Generation

    Jianfeng Xiang, Xiaoxue Chen, Sicheng Xu, Ruicheng Wang, Zelong Lv, Yu Deng, Hongyuan Zhu, Yue Dong, Hao Zhao, Nicholas Jing Yuan, and Jiaolong Yang. Native and Compact Structured Latents for 3D Generation. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2026. 2, 6, 7

    2026

  55. [55]

    ReNeRF: Re- lightable Neural Radiance Fields with Nearfield Lighting

    Yingyan Xu, Gaspard Zoss, Prashanth Chandran, Markus Gross, Derek Bradley, and Paulo Gotardo. ReNeRF: Re- lightable Neural Radiance Fields with Nearfield Lighting . In IEEE International Conference on Computer Vision (ICCV),

  56. [56]

    VideoNeuMat: Neural Material Extraction from Generative Video Models

    Bowen Xue, Saeed Hadadan, Zheng Zeng, Fabrice Rousselle, Zahra Montazeri, and Milos Hasan. VideoNeuMat: Neural Material Extraction from Generative Video Models. InACM SIGGRAPH Conference Proceedings, 2026. 2, 3

    2026

  57. [57]

    Paint- it: Text-to-Texture Synthesis via Deep Convolutional Tex- ture Map Optimization and Physically-Based Rendering

    Kim Youwang, Tae-Hyun Oh, and Gerard Pons-Moll. Paint- it: Text-to-Texture Synthesis via Deep Convolutional Tex- ture Map Optimization and Physically-Based Rendering. In IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2024. 2

    2024

  58. [58]

    Toward Richer Material Generation via Procedural Data Enhancement

    Yunchen (Blaire) Yu, Jacob Munkberg, Jon Hasselgren, Chris Cummings, Steve Marschner, and Andrea Weidlich. Toward Richer Material Generation via Procedural Data Enhancement. arXiv preprint, 2606.14988, 2026. 1, 2, 3, 4, 7, 8, 9

    work page arXiv 2026

  59. [59]

    Monte Carlo Estimators for Differential Light Transport.ACM Transactions on Graphics, 40(4):78:1–78:16,

    Tizian Zeltner, S´ebastien Speierer, Iliyan Georgiev, and Wen- zel Jakob. Monte Carlo Estimators for Differential Light Transport.ACM Transactions on Graphics, 40(4):78:1–78:16,

  60. [60]

    Real- time Neural Appearance Models.ACM Transactions on Graphics, 43(3), 2024

    Tizian Zeltner*, Fabrice Rousselle*, Andrea Weidlich*, Petrik Clarberg*, Jan Nov´ak*, Benedikt Bitterli*, Alex Evans, Tom´aˇs Davidoviˇc, Simon Kallweit, and Aaron Lefohn. Real- time Neural Appearance Models.ACM Transactions on Graphics, 43(3), 2024. 1, 2, 3, 5, 6

    2024

  61. [61]

    Relighting Neural Radiance Fields with Shadow and Highlight Hints

    Chong Zeng, Guojun Chen, Yue Dong, Pieter Peers, Hongzhi Wu, and Xin Tong. Relighting Neural Radiance Fields with Shadow and Highlight Hints. InACM SIGGRAPH Conference Proceedings, 2023. 2

    2023

  62. [62]

    RGB ↔X: image decomposition and synthesis using material-and lighting-aware diffusion models

    Zheng Zeng, Valentin Deschaintre, Iliyan Georgiev, Yannick Hold-Geoffroy, Yiwei Hu, Fujun Luan, Ling-Qi Yan, and Miloˇs Haˇsan. RGB ↔X: image decomposition and synthesis using material-and lighting-aware diffusion models. InACM SIGGRAPH Conference Proceedings, 2024. 2

    2024

  63. [63]

    Freeman, Kai Zhang, and Fujun Luan

    Tianyuan Zhang, Zhengfei Kuang, Haian Jin, Zexiang Xu, Sai Bi, Hao Tan, He Zhang, Yiwei Hu, Milos Hasan, William T. Freeman, Kai Zhang, and Fujun Luan. RelitLRM: Generative Relightable Radiance for Large Reconstruction Models.arXiv preprint, 2410.06231, 2024. 3

    work page arXiv 2024

  64. [64]

    NeR- Factor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination.ACM Transactions on Graphics,

    Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. NeR- Factor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination.ACM Transactions on Graphics,

  65. [65]

    Image GANs meet Differentiable Rendering for Inverse Graphics and Inter- pretable 3D Neural Rendering

    Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image GANs meet Differentiable Rendering for Inverse Graphics and Inter- pretable 3D Neural Rendering. InInternational Conference on Learning Representations (ICLR), 2021. 2

    2021

  66. [66]

    Modeling indirect illumination for inverse rendering

    Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indirect illumination for inverse rendering. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2

    2022

  67. [67]

    Texverse: A uni- verse of 3d objects with high-resolution textures,

    Yibo Zhang, Li Zhang, Rui Ma, and Nan Cao. TexVerse: A Universe of 3D Objects with High-Resolution Textures.arXiv preprint, 2508.10868, 2025. 4 12

    work page arXiv 2025