Header logo is

Optical Flow with Semantic Segmentation and Localized Layers


Conference Paper


Existing optical flow methods make generic, spatially homogeneous, assumptions about the spatial structure of the flow. In reality, optical flow varies across an image depending on object class. Simply put, different objects move differently. Here we exploit recent advances in static semantic scene segmentation to segment the image into objects of different types. We define different models of image motion in these regions depending on the type of object. For example, we model the motion on roads with homographies, vegetation with spatially smooth flow, and independently moving objects like cars and planes with affine motion plus deviations. We then pose the flow estimation problem using a novel formulation of localized layers, which addresses limitations of traditional layered models for dealing with complex scene motion. Our semantic flow method achieves the lowest error of any published monocular method in the KITTI-2015 flow benchmark and produces qualitatively better flow and segmentation than recent top methods on a wide range of natural videos.

Author(s): Laura Sevilla-Lara and Deqing Sun and Varun Jampani and Michael J. Black
Book Title: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)
Pages: 3889--3898
Year: 2016
Month: June

Department(s): Perceiving Systems
Research Project(s): Layered Optical Flow
Semantic Optical Flow
Bibtex Type: Conference Paper (inproceedings)
Paper Type: Conference

Event Name: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) 2016

Links: video
Kitti Precomputed Data (1.6GB)
Attachments: pdf
YouTube Sequences


  title = {Optical Flow with Semantic Segmentation and Localized Layers},
  author = {Sevilla-Lara, Laura and Sun, Deqing and Jampani, Varun and Black, Michael J.},
  booktitle = { IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
  pages = {3889--3898},
  month = jun,
  year = {2016},
  month_numeric = {6}