ReenactGAN: Learning to Reenact Faces via Boundary Transfer

Wayne Wu; Yunxuan Zhang; Cheng Li; Chen Qian; Chen Change Loy

doi:10.1007/978-3-030-01246-5_37

ReenactGAN: Learning to Reenact Faces via Boundary Transfer

Wayne Wu^*, Yunxuan Zhang, Cheng Li, Chen Qian, Chen Change Loy

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

37 Citations (Scopus)

Abstract

We present a novel learning-based framework for face reenactment. The proposed method, known as ReenactGAN, is capable of transferring facial movements and expressions from an arbitrary person’s monocular video input to a target person’s video. Instead of performing a direct transfer in the pixel space, which could result in structural artifacts, we first map the source face onto a boundary latent space. A transformer is subsequently used to adapt the source face’s boundary to the target’s boundary. Finally, a target-specific decoder is used to generate the reenacted target face. Thanks to the effective and reliable boundary-based transfer, our method can perform photo-realistic face reenactment. In addition, ReenactGAN is appealing in that the whole reenactment process is purely feed-forward, and thus the reenactment process can run in real-time (30 FPS on one GTX 1080 GPU). Dataset and model are publicly available on our project page (Project Page: https://wywu.github.io/projects/ReenactGAN/ReenactGAN.html).

Original language	English
Title of host publication	Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings
Editors	Martial Hebert, Vittorio Ferrari, Cristian Sminchisescu, Yair Weiss
Publisher	Springer Verlag
Pages	622-638
Number of pages	17
ISBN (Print)	9783030012458
DOIs	https://doi.org/10.1007/978-3-030-01246-5_37
Publication status	Published - 2018
Externally published	Yes
Event	15th European Conference on Computer Vision, ECCV 2018 - Munich, Germany Duration: Sept 8 2018 → Sept 14 2018

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11205 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	15th European Conference on Computer Vision, ECCV 2018
Country/Territory	Germany
City	Munich
Period	9/8/18 → 9/14/18

Bibliographical note

Publisher Copyright:
© 2018, Springer Nature Switzerland AG.

ASJC Scopus Subject Areas

Theoretical Computer Science
General Computer Science

Keywords

Face alignment
Face generation
Face reenactment
GAN

Access to Document

10.1007/978-3-030-01246-5_37

Cite this

Wu, W., Zhang, Y., Li, C., Qian, C., & Loy, C. C. (2018). ReenactGAN: Learning to Reenact Faces via Boundary Transfer. In M. Hebert, V. Ferrari, C. Sminchisescu, & Y. Weiss (Eds.), Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings (pp. 622-638). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11205 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-01246-5_37

Wu, Wayne ; Zhang, Yunxuan ; Li, Cheng et al. / ReenactGAN : Learning to Reenact Faces via Boundary Transfer. Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. editor / Martial Hebert ; Vittorio Ferrari ; Cristian Sminchisescu ; Yair Weiss. Springer Verlag, 2018. pp. 622-638 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{3fbaf7dffc814289a6a79343f270d40a,

title = "ReenactGAN: Learning to Reenact Faces via Boundary Transfer",

abstract = "We present a novel learning-based framework for face reenactment. The proposed method, known as ReenactGAN, is capable of transferring facial movements and expressions from an arbitrary person{\textquoteright}s monocular video input to a target person{\textquoteright}s video. Instead of performing a direct transfer in the pixel space, which could result in structural artifacts, we first map the source face onto a boundary latent space. A transformer is subsequently used to adapt the source face{\textquoteright}s boundary to the target{\textquoteright}s boundary. Finally, a target-specific decoder is used to generate the reenacted target face. Thanks to the effective and reliable boundary-based transfer, our method can perform photo-realistic face reenactment. In addition, ReenactGAN is appealing in that the whole reenactment process is purely feed-forward, and thus the reenactment process can run in real-time (30 FPS on one GTX 1080 GPU). Dataset and model are publicly available on our project page (Project Page: https://wywu.github.io/projects/ReenactGAN/ReenactGAN.html).",

keywords = "Face alignment, Face generation, Face reenactment, GAN",

author = "Wayne Wu and Yunxuan Zhang and Cheng Li and Chen Qian and Loy, \{Chen Change\}",

note = "Publisher Copyright: {\textcopyright} 2018, Springer Nature Switzerland AG.; 15th European Conference on Computer Vision, ECCV 2018 ; Conference date: 08-09-2018 Through 14-09-2018",

year = "2018",

doi = "10.1007/978-3-030-01246-5\_37",

language = "English",

isbn = "9783030012458",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "622--638",

editor = "Martial Hebert and Vittorio Ferrari and Cristian Sminchisescu and Yair Weiss",

booktitle = "Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings",

address = "Germany",

}

Wu, W, Zhang, Y, Li, C, Qian, C & Loy, CC 2018, ReenactGAN: Learning to Reenact Faces via Boundary Transfer. in M Hebert, V Ferrari, C Sminchisescu & Y Weiss (eds), Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11205 LNCS, Springer Verlag, pp. 622-638, 15th European Conference on Computer Vision, ECCV 2018, Munich, Germany, 9/8/18. https://doi.org/10.1007/978-3-030-01246-5_37

ReenactGAN: Learning to Reenact Faces via Boundary Transfer. / Wu, Wayne; Zhang, Yunxuan; Li, Cheng et al.
Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. ed. / Martial Hebert; Vittorio Ferrari; Cristian Sminchisescu; Yair Weiss. Springer Verlag, 2018. p. 622-638 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11205 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - ReenactGAN

T2 - 15th European Conference on Computer Vision, ECCV 2018

AU - Wu, Wayne

AU - Zhang, Yunxuan

AU - Li, Cheng

AU - Qian, Chen

AU - Loy, Chen Change

PY - 2018

Y1 - 2018

N2 - We present a novel learning-based framework for face reenactment. The proposed method, known as ReenactGAN, is capable of transferring facial movements and expressions from an arbitrary person’s monocular video input to a target person’s video. Instead of performing a direct transfer in the pixel space, which could result in structural artifacts, we first map the source face onto a boundary latent space. A transformer is subsequently used to adapt the source face’s boundary to the target’s boundary. Finally, a target-specific decoder is used to generate the reenacted target face. Thanks to the effective and reliable boundary-based transfer, our method can perform photo-realistic face reenactment. In addition, ReenactGAN is appealing in that the whole reenactment process is purely feed-forward, and thus the reenactment process can run in real-time (30 FPS on one GTX 1080 GPU). Dataset and model are publicly available on our project page (Project Page: https://wywu.github.io/projects/ReenactGAN/ReenactGAN.html).

AB - We present a novel learning-based framework for face reenactment. The proposed method, known as ReenactGAN, is capable of transferring facial movements and expressions from an arbitrary person’s monocular video input to a target person’s video. Instead of performing a direct transfer in the pixel space, which could result in structural artifacts, we first map the source face onto a boundary latent space. A transformer is subsequently used to adapt the source face’s boundary to the target’s boundary. Finally, a target-specific decoder is used to generate the reenacted target face. Thanks to the effective and reliable boundary-based transfer, our method can perform photo-realistic face reenactment. In addition, ReenactGAN is appealing in that the whole reenactment process is purely feed-forward, and thus the reenactment process can run in real-time (30 FPS on one GTX 1080 GPU). Dataset and model are publicly available on our project page (Project Page: https://wywu.github.io/projects/ReenactGAN/ReenactGAN.html).

KW - Face alignment

KW - Face generation

KW - Face reenactment

KW - GAN

UR - http://www.scopus.com/inward/record.url?scp=85055134894&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85055134894&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-01246-5_37

DO - 10.1007/978-3-030-01246-5_37

M3 - Conference contribution

AN - SCOPUS:85055134894

SN - 9783030012458

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 622

EP - 638

BT - Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings

A2 - Hebert, Martial

A2 - Ferrari, Vittorio

A2 - Sminchisescu, Cristian

A2 - Weiss, Yair

PB - Springer Verlag

Y2 - 8 September 2018 through 14 September 2018

ER -

Wu W, Zhang Y, Li C, Qian C, Loy CC. ReenactGAN: Learning to Reenact Faces via Boundary Transfer. In Hebert M, Ferrari V, Sminchisescu C, Weiss Y, editors, Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. Springer Verlag. 2018. p. 622-638. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-01246-5_37

ReenactGAN: Learning to Reenact Faces via Boundary Transfer

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus Subject Areas

Keywords

Access to Document

Other files and links

Cite this