Dense Pose Transfer

Abstract : In this work we integrate ideas from surface-based modelling with neural synthesis: we propose a combination of surface-based pose estimation and deep generative models that allows us to perform accurate pose transfer, i.e. synthesize a new image of a person based on a single image of that person and the image of a pose donor. We use a dense pose estimation system that maps pixels from both images to a common surface-based coordinate system, allowing the two images to be brought in correspondence with each other. We inpaint and refine the source image intensities in the surface coordinate system, prior to warping them onto the target pose. These predictions are fused with those of a convolutional predictive module through a neural synthesis module allowing for training the whole pipeline jointly end-to-end, optimizing a combination of adversarial and perceptual losses. We show that dense pose estimation is a substantially more powerful conditioning input than landmark- or mask-based alternatives, and report systematic improvements over state of the art generators on DeepFashion and MVC datasets. Given an input image and a target pose we use DensePose [1] to drive the generation process. This is achieved through the complementary streams of a data-driven predictive module and a surface-based module that warps the texture to UV-coordinates, interpolates on the surface and warps back to the target image. A blending module combines the complementary merits of these two streams in a single end-to-end trainable framework.
Type de document :
Communication dans un congrès
European Conference on Computer Vision (ECCV) 2018, Sep 2018, Munich, Germany
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-01951850
Contributeur : Riza Alp Guler <>
Soumis le : mardi 11 décembre 2018 - 16:43:32
Dernière modification le : jeudi 7 février 2019 - 15:37:47

Fichier

SurfaceCoordination (6).pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01951850, version 1

Citation

Natalia Neverova, Riza Alp Guler, Iasonas Kokkinos. Dense Pose Transfer. European Conference on Computer Vision (ECCV) 2018, Sep 2018, Munich, Germany. 〈hal-01951850〉

Partager

Métriques

Consultations de la notice

46

Téléchargements de fichiers

21