A Temporally-Aware Interpolation Network for Video Frame Inpainting

Ximeng Sun, Ryan Szeto, Jason J. Corso

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

We propose the first deep learning solution to video frame inpainting, a more challenging but less ambiguous task than related problems such as general video inpainting, frame interpolation, and video prediction. We devise a pipeline composed of two modules: a bidirectional video prediction module and a temporally-aware frame interpolation module. The prediction module makes two intermediate predictions of the missing frames, each conditioned on the preceding and following frames respectively, using a shared convolutional LSTM-based encoder-decoder. The interpolation module blends the intermediate predictions, using time information and hidden activations from the video prediction module to resolve disagreements between the predictions. Our experiments demonstrate that our approach produces more accurate and qualitatively satisfying results than a state-of-the-art video prediction method and many strong frame inpainting baselines. Our code is available at https://github.com/sunxm2357/TAI_video_frame_inpainting.

Original languageEnglish
Title of host publicationComputer Vision - ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers
EditorsKonrad Schindler, C.V. Jawahar, Greg Mori, Hongdong Li
Pages249-264
Number of pages16
DOIs
StatePublished - 2019
Event14th Asian Conference on Computer Vision, ACCV 2018 - Perth, Australia
Duration: 2 Dec 20186 Dec 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11363 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference14th Asian Conference on Computer Vision, ACCV 2018
Country/TerritoryAustralia
CityPerth
Period2/12/186/12/18

Keywords

  • Frame interpolation
  • Video inpainting
  • Video prediction

Fingerprint

Dive into the research topics of 'A Temporally-Aware Interpolation Network for Video Frame Inpainting'. Together they form a unique fingerprint.

Cite this