Header logo is

Label Efficient Visual Abstractions for Autonomous Driving


Conference Paper


It is well known that semantic segmentation can be used as an effective intermediate representation for learning driving policies. However, the task of street scene semantic segmentation requires expensive annotations. Furthermore, segmentation algorithms are often trained irrespective of the actual driving task, using auxiliary image-space loss functions which are not guaranteed to maximize driving metrics such as safety or distance traveled per intervention. In this work, we seek to quantify the impact of reducing segmentation annotation costs on learned behavior cloning agents. We analyze several segmentation-based intermediate representations. We use these visual abstractions to systematically study the trade-off between annotation efficiency and driving performance, ie, the types of classes labeled, the number of image samples used to learn the visual abstraction model, and their granularity (eg, object masks vs. 2D bounding boxes). Our analysis uncovers several practical insights into how segmentation-based visual abstractions can be exploited in a more label efficient manner. Surprisingly, we find that state-of-the-art driving performance can be achieved with orders of magnitude reduction in annotation cost. Beyond label efficiency, we find several additional training benefits when leveraging visual abstractions, such as a significant reduction in the variance of the learned policy when compared to state-of-the-art end-to-end driving models.

Author(s): Aseem Behl and Kashyap Chitta and Aditya Prakash and Eshed Ohn-Bar and Andreas Geiger
Book Title: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Year: 2020
Month: October
Publisher: IEEE

Department(s): Autonomous Vision
Bibtex Type: Conference Paper (conference)
Paper Type: Conference

Event Place: Las Vegas

Links: pdf
Project Page


  title = {Label Efficient Visual Abstractions for Autonomous Driving},
  author = {Behl, Aseem and Chitta, Kashyap and Prakash, Aditya and Ohn-Bar, Eshed and Geiger, Andreas},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  publisher = {IEEE},
  month = oct,
  year = {2020},
  month_numeric = {10}