Skip to Main content Skip to Navigation
Journal articles

Convolutional Nets and Watershed Cuts for Real-Time Semantic Labeling of RGBD Videos

Abstract : This work addresses multi-class segmentation of indoor scenes with RGB-D inputs. While this area of research has gained much attention recently, most works still rely on handcrafted features. In contrast, we apply a multiscale convolutional network to learn features directly from the images and the depth information. Using a frame by frame labeling, we obtain nearly state-of-the-art performance on the NYU-v2 depth dataset with an accuracy of 64.5%. We then show that the labeling can be further improved by exploiting the temporal consistency in the video sequence of the scene. To that goal, we present a method producing temporally consistent superpixels from a streaming video. Among the di erent methods producing superpixel segmentations of an image, the graph-based approach of Felzenszwalb and Huttenlocher is broadly employed. One of its interesting properties is that the regions are computed in a greedy manner in quasi-linear time by using a minimum spanning tree. In a framework exploiting minimum spanning trees all along, we propose an efficient video segmentation approach that computes temporally consistent pixels in a causal manner, filling the need for causal and real-time applications. We illustrate the labeling of indoor scenes in video sequences that could be processed in real-time using appropriate hardware such as an FPGA.
Complete list of metadata

Cited literature [48 references]  Display  Hide  Download
Contributor : Laurent Najman Connect in order to contact the contributor
Submitted on : Tuesday, September 23, 2014 - 5:13:34 PM
Last modification on : Saturday, January 15, 2022 - 3:58:33 AM
Long-term archiving on: : Friday, April 14, 2017 - 3:32:11 PM


Files produced by the author(s)


  • HAL Id : hal-01066586, version 1


Camille Couprie, Clément Farabet, Laurent Najman, Yann Lecun. Convolutional Nets and Watershed Cuts for Real-Time Semantic Labeling of RGBD Videos. Journal of Machine Learning Research, Microtome Publishing, 2014, 15, pp.3489−3511. ⟨hal-01066586⟩



Record views


Files downloads