Header logo is

Learning Priors for Semantic 3D Reconstruction


Conference Paper


We present a novel semantic 3D reconstruction framework which embeds variational regularization into a neural network. Our network performs a fixed number of unrolled multi-scale optimization iterations with shared interaction weights. In contrast to existing variational methods for semantic 3D reconstruction, our model is end-to-end trainable and captures more complex dependencies between the semantic labels and the 3D geometry. Compared to previous learning-based approaches to 3D reconstruction, we integrate powerful long-range dependencies using variational coarse-to-fine optimization. As a result, our network architecture requires only a moderate number of parameters while keeping a high level of expressiveness which enables learning from very little data. Experiments on real and synthetic datasets demonstrate that our network achieves higher accuracy compared to a purely variational approach while at the same time requiring two orders of magnitude less iterations to converge. Moreover, our approach handles ten times more semantic class labels using the same computational resources.

Author(s): Ian Cherabier and Johannes Schönberger and Martin Oswald and Marc Pollefeys and Andreas Geiger
Book Title: Computer Vision – ECCV 2018
Year: 2018
Month: September
Publisher: Springer International Publishing

Department(s): Autonomous Vision
Research Project(s): Deep, Probabilistic and Semantic 3D Reconstruction
Bibtex Type: Conference Paper (inproceedings)

DOI: https://doi.org/10.1007/978-3-030-01258-8_20
Event Place: Munich, Germany

Address: Cham

Links: pdf
Project Page


  title = {Learning Priors for Semantic 3D Reconstruction },
  author = {Cherabier, Ian and Sch{\"o}nberger, Johannes and Oswald, Martin and Pollefeys, Marc and Geiger, Andreas},
  booktitle = {Computer Vision -- ECCV 2018},
  publisher = {Springer International Publishing},
  address = {Cham},
  month = sep,
  year = {2018},
  month_numeric = {9}