HAL will be down for maintenance from Friday, June 10 at 4pm through Monday, June 13 at 9am. More information
Skip to Main content Skip to Navigation
Conference papers

Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Space

Abstract : Learning causal relationships in high-dimensional data (images, videos) is a hard task, as they are often defined on low-dimensional manifolds and must be extracted from complex signals dominated by appearance, lighting, textures and also spurious correlations in the data. We present a method for learning counterfactual reasoning of physical processes in pixel space, which requires the prediction of the impact of interventions on initial conditions. Going beyond the identification of structural relationships, we deal with the challenging problem of forecasting raw video over long horizons. Our method does not require the knowledge or supervision of any ground truth positions or other object or scene properties. Our model learns and acts on a suitable hybrid latent representation based on a combination of dense features, sets of 2D keypoints and an additional latent vector per keypoint. We show that this better captures the dynamics of physical processes than purely dense or sparse representations. We introduce a new challenging and carefully designed counterfactual benchmark for predictions in pixel space and outperform strong baselines in physics-inspired ML and video prediction.
Document type :
Conference papers
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03630595
Contributor : Steeven Janny Connect in order to contact the contributor
Submitted on : Tuesday, April 5, 2022 - 10:21:28 AM
Last modification on : Friday, May 13, 2022 - 4:01:00 PM

File

CoPhy_ICLR.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03630595, version 1

Citation

Steeven Janny, Fabien Baradel, Natalia Neverova, Madiha Nadri, Greg Mori, et al.. Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Space. International Conference on Learning Representations, Apr 2022, Virtuel, United States. ⟨hal-03630595⟩

Share

Metrics

Record views

13

Files downloads

0