preprint · arXiv (Cornell University) · 2019

Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

Wenzheng Chen, Jun Gao, Huan Ling, Smith, Edward J., Jaakko Lehtinen, Alec Jacobson, Sanja Fidler

Cited by 252Open access

Abstract

Many machine learning models operate on images, but ignore the fact that images are 2D projections formed by 3D geometry interacting with light, in a process called rendering. Enabling ML models to understand image formation might be key for generalization. However, due to an essential rasterization step involving discrete assignment operations, rendering pipelines are non-differentiable and thus largely inaccessible to gradient-based ML techniques. In this paper, we present {\emph DIB-R}, a differentiable rendering framework which allows gradients to be analytically computed for all pixels in an image. Key to our approach is to view foreground rasterization as a weighted interpolation of local properties and background rasterization as a distance-based aggregation of global geometry. Our approach allows for accurate optimization over vertex positions, colors, normals, light directions and texture coordinates through a variety of lighting models. We showcase our approach in two ML applications: single-image 3D object prediction, and 3D textured object generation, both trained using exclusively using 2D supervision. Our project website is: https://nv-tlabs.github.io/DIB-R/

Cite this paper

APA 7

Chen, W., Gao, J., Ling, H., Smith, E. J., Lehtinen, J., Jacobson, A., & Fidler, S. (2019). Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer. arXiv (Cornell University), 32, 9605–9616. https://doi.org/10.48550/arxiv.1908.01210

MLA 9

Chen, Wenzheng, et al. “Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer.” arXiv (Cornell University), vol. 32, 2019, pp. 9605–16, https://doi.org/10.48550/arxiv.1908.01210.

Chicago

Chen, Wenzheng, Jun Gao, Huan Ling, Edward J. Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. 2019. “Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer.” arXiv (Cornell University) 32: 9605–16. https://doi.org/10.48550/arxiv.1908.01210.

Harvard

Chen, W. et al. (2019) ‘Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer’, arXiv (Cornell University), 32, pp. 9605–9616. Available at: https://doi.org/10.48550/arxiv.1908.01210.

BibTeX

@misc{chen2019learning,
  title = {{Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer}},
  author = {Chen, Wenzheng and Gao, Jun and Ling, Huan and Smith, Edward J. and Lehtinen, Jaakko and Jacobson, Alec and Fidler, Sanja},
  journal = {arXiv (Cornell University)},
  volume = {32},
  pages = {9605--9616},
  year = {2019},
  doi = {10.48550/arxiv.1908.01210}
}

Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.