Yi, Shinyoung, Kim, Donggun, Choi, Kiseok et al. · arXiv (Cornell University) · 2022
Recent differentiable rendering techniques have become key tools to tackle many inverse problems in graphics and vision. Existing models, however, assume steady-state light transport, i.e., infinite speed of light. While this is a safe assumption for many applications, recent advances in ultrafast imaging leverage the wealth of information that can be extracted from the exact time of flight of light. In this context, physically-based transient rendering allows to efficiently simulate and analyze light transport considering that the speed of light is indeed finite. In this paper, we introduce a novel differentiable transient rendering framework, to help bring the potential of differentiable approaches into the transient regime. To differentiate the transient path integral we need to take into account that scattering events at path vertices are no longer independent; instead, tracking the time of flight of light requires treating such scattering events at path vertices jointly as a multidimensional, evolving manifold. We thus turn to the generalized transport theorem, and introduce a novel correlated importance term, which links the time-integrated contribution of a path to its light throughput, and allows us to handle discontinuities in the light and sensor functions. Last, we present results in several challenging scenarios where the time of flight of light plays an important role such as optimizing indices of refraction, non-line-of-sight tracking with nonplanar relay walls, and non-line-of-sight tracking around two corners.
Cited by 0Open access
Hiroharu Kato, Deniz Beker, Mihai Adrian Morariu et al. · arXiv (Cornell University) · 2020
Deep neural networks (DNNs) have shown remarkable performance improvements on vision-related tasks such as object detection or image segmentation. Despite their success, they generally lack the understanding of 3D objects which form the image, as it is not always possible to collect 3D information about the scene or to easily annotate it. Differentiable rendering is a novel field which allows the gradients of 3D objects to be calculated and propagated through images. It also reduces the requirement of 3D data collection and annotation, while enabling higher success rate in various applications. This paper reviews existing literature and discusses the current state of differentiable rendering, its applications and open research problems.
Cited by 120Open access
Sebastian Weiß, Rüdiger Westermann · IEEE Transactions on Visualization and Computer Graphics · 2021
We present a differentiable volume rendering solution that provides differentiability of all continuous parameters of the volume rendering process. This differentiable renderer is used to steer the parameters towards a setting with an optimal solution of a problem-specific objective function. We have tailored the approach to volume rendering by enforcing a constant memory footprint via analytic inversion of the blending functions. This makes it independent of the number of sampling steps through the volume and facilitates the consideration of small-scale changes. The approach forms the basis for automatic optimizations regarding external parameters of the rendering process and the volumetric density field itself. We demonstrate its use for automatic viewpoint selection using differentiable entropy as objective, and for optimizing a transfer function from rendered images of a given volume. Optimization of per-voxel densities is addressed in two different ways: First, we mimic inverse tomography and optimize a 3D density field from images using an absorption model. This simplification enables comparisons with algebraic reconstruction techniques and state-of-the-art differentiable path tracers. Second, we introduce a novel approach for tomographic reconstruction from images using an emission-absorption model with post-shading via an arbitrary transfer function.
Cited by 0Open access
Yan, Kai · eScholarship (California Digital Library) · 2025
Differentiable rendering focusing on estimating derivative of forward-rendering result, which capable of differentiating with respect to a wide range of scene parameters including material properties and object geometries. In this thesis, we explore different types of variance in differentiable rendering: variance in gradient estimates in differentiable rendering and inverse rendering, and differentiating rendering variance.First, we introduce an efficient technique for estimating boundary integrals in path-space differentiable rendering, which significantly reduces estimation variance, enabling robust inverse rendering with complex geometry and lighting.Second, we introduce an image-space adaptive sampling technique that accelerates inverse rendering by dynamically adjusting pixel sampling probabilities based on gradient variance during inverse rendering.In the end, we establish a framework for differentiating rendering variance with respect to both scene parameters and sampling probabilities. While traditional differentiable rendering differentiates pixel values, we differentiate their variance. This introduces the concept of Variance-aware inverse rendering, which alters a virtual scene and/or an estimator in an optimal way to offer balance between rendering bias and variance.
Cited by 0Open access
Cheng Zhang, Bailey Miller, Kai Yan et al. · ACM Transactions on Graphics · 2020
Physics-based differentiable rendering, the estimation of derivatives of radiometric measures with respect to arbitrary scene parameters, has a diverse array of applications from solving analysis-by-synthesis problems to training machine learning pipelines incorporating forward rendering processes. Unfortunately, general-purpose differentiable rendering remains challenging due to the lack of efficient estimators as well as the need to identify and handle complex discontinuities such as visibility boundaries. In this paper, we show how path integrals can be differentiated with respect to arbitrary differentiable changes of a scene. We provide a detailed theoretical analysis of this process and establish new differentiable rendering formulations based on the resulting differential path integrals. Our path-space differentiable rendering formulation allows the design of new Monte Carlo estimators that offer significantly better efficiency than state-of-the-art methods in handling complex geometric discontinuities and light transport phenomena such as caustics. We validate our method by comparing our derivative estimates to those generated using the finite-difference method. To demonstrate the effectiveness of our technique, we compare inverse-rendering performance with a few state-of-the-art differentiable rendering methods.
Cited by 144
Quentin Le Lidec, Ivan Laptev, Schmid, Cordelia et al. · HAL (Le Centre pour la Communication Scientifique Directe) · 2021
International audience
Cited by 0Open access
Frans Zdyb, Albert Alonso, Julius B. Kirkegaard · Lecture notes in computer science · 2025
Cited by 0Open access
Christoph Neuhauser · ArXiv.org · 2025
Differentiable rendering is a technique that aims to invert the rendering process to enable optimizing rendering parameters from a set of images. In this article, we present a differentiable volume rendering solution called DiffTetVR for tetrahedral meshes. Unlike previous works based on regular grids, this enables the optimization of vertex positions and the local subdivision of the mesh without relying on multigrid methods. We present an efficient implementation of the forward rendering process, deduce the derivatives for the backwards pass and regularization terms for avoiding degenerate tetrahedra, and finally show how the tetrahedral mesh can be subdivided locally to enable a coarse-to-fine optimization process. The source code is made publicly available on GitHub at https://github.com/chrismile/DiffTetVR.
Cited by 0Open access
Brandon Y. Feng, Rodrigo Ferrer-Chávez, Aviad Levis et al. · IEEE Transactions on Computational Imaging · 2025
Direct imaging of exoplanets is crucial for advancing our understanding of planetary systems beyond our solar system, but it faces significant challenges due to the high contrast between host stars and their planets. Wavefront aberrations introduce speckles in the telescope science images, which are patterns of diffracted starlight that can mimic the appearance of planets, complicating the detection of faint exoplanet signals. Traditional post-processing methods, operating primarily in the image intensity domain, do not integrate wavefront sensing data. These data, measured mainly for adaptive optics corrections, have been overlooked as a potential resource for post-processing, partly due to the challenge of the evolving nature of wavefront aberrations. In this paper, we present a differentiable rendering approach that leverages these wavefront sensing data to improve exoplanet detection. Our differentiable renderer models wave-based light propagation through a coronagraphic telescope system, allowing gradient-based optimization to significantly improve starlight subtraction and increase sensitivity to faint exoplanets. Simulation experiments based on the James Webb Space Telescope configuration demonstrate the effectiveness of our approach, achieving substantial improvements in contrast and planet detection limits. Our results showcase how the computational advancements enabled by differentiable rendering can revitalize previously underexploited wavefront data, opening new avenues for enhancing exoplanet imaging and characterization.
Cited by 1Open access
Hoon-Gyu Chung, Seokjun Choi, Seung‐Hwan Baek · 2024
We present differentiable point-based inverse rendering, DPIR, an analysis-by-synthesis method that processes images captured under diverse illuminations to estimate shape and spatially-varying BRDF. To this end, we adopt point-based rendering, eliminating the need for multiple samplings per ray, typical of volumetric rendering, thus significantly enhancing the speed of inverse rendering. To realize this idea, we devise a hybrid point-volumetric representation for geometry and a regularized basis-BRDF representation for reflectance. The hybrid geometric representation enables fast rendering through point-based splatting while retaining the geometric details and stability inherent to SDF-based representations. The regularized basis-BRDF mitigates the ill-posedness of inverse rendering stemming from limited light-view angular samples. We also propose an efficient shadow detection method using point-based shadow map rendering. Our extensive evaluations demonstrate that DPIR outperforms prior works in terms of reconstruction accuracy, computational efficiency, and memory footprint. Furthermore, our explicit point-based representation and rendering enables intuitive geometry and reflectance editing.
Cited by 2
Christoph Neuhauser · arXiv (Cornell University) · 2025
Differentiable rendering is a technique that aims to invert the rendering process to enable optimizing rendering parameters from a set of images. In this article, we present a differentiable volume rendering solution called DiffTetVR for tetrahedral meshes. Unlike previous works based on regular grids, this enables the optimization of vertex positions and the local subdivision of the mesh without relying on multigrid methods. We present an efficient implementation of the forward rendering process, deduce the derivatives for the backwards pass and regularization terms for avoiding degenerate tetrahedra, and finally show how the tetrahedral mesh can be subdivided locally to enable a coarse-to-fine optimization process. The source code is made publicly available on GitHub at https://github.com/chrismile/DiffTetVR.
Cited by 0Open access
Delio Vicini, Sébastien Speierer, Wenzel Jakob · ACM Transactions on Graphics · 2022
Physically-based differentiable rendering has recently emerged as an attractive new technique for solving inverse problems that recover complete 3D scene representations from images. The inversion of shape parameters is of particular interest but also poses severe challenges: shapes are intertwined with visibility, whose discontinuous nature introduces severe bias in computed derivatives unless costly precautions are taken. Shape representations like triangle meshes suffer from additional difficulties, since the continuous optimization of mesh parameters cannot introduce topological changes. One common solution to these difficulties entails representing shapes using signed distance functions (SDFs) and gradually adapting their zero level set during optimization. Previous differentiable rendering of SDFs did not fully account for visibility gradients and required the use of mask or silhouette supervision, or discretization into a triangle mesh. In this article, we show how to extend the commonly used sphere tracing algorithm so that it additionally outputs a reparameterization that provides the means to compute accurate shape parameter derivatives. At a high level, this resembles techniques for differentiable mesh rendering, though we show that the SDF representation admits a particularly efficient reparameterization that outperforms prior work. Our experiments demonstrate the reconstruction of (synthetic) objects without complex regularization or priors, using only a per-pixel RGB loss.
Cited by 105Open access
Sankeerth Durvasula, Adrian Zhao, Fan Chen et al. · arXiv (Cornell University) · 2023
Differentiable rendering is a technique used in an important emerging class of visual computing applications that involves representing a 3D scene as a model that is trained from 2D images using gradient descent. Recent works (e.g. 3D Gaussian Splatting) use a rasterization pipeline to enable rendering high quality photo-realistic imagery at high speeds from these learned 3D models. These methods have been demonstrated to be very promising, providing state-of-art quality for many important tasks. However, training a model to represent a scene is still a time-consuming task even when using powerful GPUs. In this work, we observe that the gradient computation phase during training is a significant bottleneck on GPUs due to the large number of atomic operations that need to be processed. These atomic operations overwhelm atomic units in the L2 partitions causing stalls. To address this challenge, we leverage the observations that during the gradient computation: (1) for most warps, all threads atomically update the same memory locations; and (2) warps generate varying amounts of atomic traffic (since some threads may be inactive). We propose DISTWAR, a software-approach to accelerate atomic operations based on two key ideas: First, we enable warp-level reduction of threads at the SM sub-cores using registers to leverage the locality in intra-warp atomic updates. Second, we distribute the atomic computation between the warp-level reduction at the SM and the L2 atomic units to increase the throughput of atomic computation. Warps with many threads performing atomic updates to the same memory locations are scheduled at the SM, and the rest using L2 atomic units. We implement DISTWAR using existing warp-level primitives. We evaluate DISTWAR on widely used raster-based differentiable rendering workloads. We demonstrate significant speedups of 2.44x on average (up to 5.7x).
Cited by 0Open access
Held, Jan, Son, Sanghyun, Vandeghen, Renaud et al. · arXiv (Cornell University) · 2025
Primitive-based splatting methods like 3D Gaussian Splatting have revolutionized novel view synthesis with real-time rendering. However, their point-based representations remain incompatible with mesh-based pipelines that power AR/VR and game engines. We present MeshSplatting, a mesh-based reconstruction approach that jointly optimizes geometry and appearance through differentiable rendering. By enforcing connectivity via restricted Delaunay triangulation and refining surface consistency, MeshSplatting creates end-to-end smooth, visually high-quality meshes that render efficiently in real-time 3D engines. On Mip-NeRF360, it boosts PSNR by +0.69 dB over the current state-of-the-art MiLo for mesh-based novel view synthesis, while training 2x faster and using 2x less memory, bridging neural rendering and interactive 3D graphics for seamless real-time scene interaction. The project page is available at https://meshsplatting.github.io/.
Cited by 0Open access
Held, Jan, Son, Sanghyun, Vandeghen, Renaud et al. · ArXiv.org · 2025
Primitive-based splatting methods like 3D Gaussian Splatting have revolutionized novel view synthesis with real-time rendering. However, their point-based representations remain incompatible with mesh-based pipelines that power AR/VR and game engines. We present MeshSplatting, a mesh-based reconstruction approach that jointly optimizes geometry and appearance through differentiable rendering. By enforcing connectivity via restricted Delaunay triangulation and refining surface consistency, MeshSplatting creates end-to-end smooth, visually high-quality meshes that render efficiently in real-time 3D engines. On Mip-NeRF360, it boosts PSNR by +0.69 dB over the current state-of-the-art MiLo for mesh-based novel view synthesis, while training 2x faster and using 2x less memory, bridging neural rendering and interactive 3D graphics for seamless real-time scene interaction. The project page is available at https://meshsplatting.github.io/.
Cited by 0Open access
Hidenobu Matsuki · Imperial College Research Computing Service Data Repository · 2024
This thesis investigates differentiable rendering for dense visual Simultaneous Localization and Mapping (SLAM), specifically examining how novel rendering algorithms and 3D scene representations enhance reconstruction and spatial understanding. The choice of scene representation is fundamental to the efficiency of 3D modeling. Traditional approaches, which rely on direct 3D measurements from sensors or predictions from pre-trained networks, often struggle with limited operational ranges or training data biases. This necessitates 3D representations that can be flexibly optimized using real-time 2D observations. With advancements in computing hardware, 3D representations optimized through differentiable rendering have gained prominence. These methods achieve high-fidelity inverse rendering and novel view synthesis. While early techniques required lengthy optimization, the introduction of explicit data structures has shifted performance toward real-time, enabling integration into SLAM. Unlike offline reconstruction, online SLAM requires continuous estimation and updating of unknown environments from sequential data, posing unique challenges for optimization stability and memory management. This thesis examines the use of 3D scene representations in combination with differentiable rendering pipelines to enable real-time SLAM systems. The work begins by developing classical depth map based SLAM methods to establish a foundation in existing standard pipelines. Building on this baseline, the thesis explores advanced scene representations — specifically NeRF and the latest 3D Gaussian Splatting techniques — as novel approaches to SLAM. Through detailed experiments and analysis, the thesis demonstrates how to effectively address the open-world challenges inherent in SLAM tasks using these modern representations. It also highlights the unique advantages that differentiable rendering offers, such as improved accuracy and flexibility, positioning it as a promising direction for the future of visual SLAM.
Cited by 0Open access
Jakub Pelc · Digital Repository (National Repository of Grey Literature) · 2026
Mitsuba 3 is a research-oriented renderer capable of auto-differentiation and inverse rendering. It currently runs on a single CPU or GPU, hence unable to take advantage of distributed computing or multi-gpu systems. Our work provides a library that overcomes this constraint, allowing both the forward rendering and differentiable rendering to occur on a cluster of machines or multiple GPUs on a single machine. This is exposed to the user using an API similar to that of base Mitsuba. We have also implemented adaptive image-space sampling, allowing the renderer to preferrably spend computational resources on demanding regions of the image, which leads to less noisy images at a fixed computation time compared to traditional rendering.
Cited by 0
Igor Kinev, Anatoly Lysykh, Dmitry Zhdanov et al. · 2023
Currently, for most VR/AR applications it is necessary to reconstruct physically-correctly the real world, and the optical properties of objects. In this paper, to solve this problem, we propose to use a parametric model of optical properties, which is a mix of several known models, which has a small number of parameters and correctly simulates the propagation of light in the scene. In addition, based on the parameters of the model, differential rendering of the image is performed, as a result of which the obtained differentials can be further used in the optimizer. The optimizer, in turn, can modify the model parameters to improve the rendering quality and thus bring the proposed model of optical properties closer to the real optical properties of objects.
Cited by 0
Johannes Schüle, Peter Somers, AR Salehah et al. · SSRN Electronic Journal · 2022
Cited by 2Open access
Yichen Sheng, Zixun Yu, Lu Ling et al. · 2024
Bokeh is widely used in photography to draw attention to the subject while effectively isolating distractions in the background. Computational methods can simulate bokeh effects without relying on a physical camera lens, but the inaccurate lens modeling in existing filtering-based meth-ods leads to artifacts that need post-processing or learning-based methods to fix. We propose Dr.Bokeh, a novel ren-dering method that addresses the issue by directly correcting the defect that violates physics in the current filtering-based bokeh rendering equation. Dr.Bokeh first preprocesses the input RGBD to obtain a layered scene representation. Dr.Bokeh then takes the layered representation and user-defined lens parameters to render photo-realistic lens blur based on the novel occlusion-aware bokeh rendering method. Experiments show that the non-learning based renderer Dr.Bokeh outperforms state-of-the-art bokeh ren-dering algorithms in terms of photo-realism. In addition, extensive quantitative and qualitative evaluations show that the more accurate lens model pushes the limit of depth-from-defocus.
Cited by 7