Skip to main navigation Skip to search Skip to main content

A Temporally-Aware Interpolation Network for Video Frame Inpainting

  • Ryan Szeto
  • , Ximeng Sun
  • , Kunyi Lu
  • , Jason J. Corso
    • University of Michigan, Ann Arbor
    • Boston University

    Research output: Contribution to journal › Article › peer-review

    23 Scopus citations

    Abstract

    In this work, we explore video frame inpainting, a task that lies at the intersection of general video inpainting, frame interpolation, and video prediction. Although our problem can be addressed by applying methods from other video interpolation or extrapolation tasks, doing so fails to leverage the additional context information that our problem provides. To this end, we devise a method specifically designed for video frame inpainting that is composed of two modules: A bidirectional video prediction module and a temporally-Aware frame interpolation module. The prediction module makes two intermediate predictions of the missing frames, each conditioned on the preceding and following frames respectively, using a shared convolutional LSTM-based encoder-decoder. The interpolation module blends the intermediate predictions by using time information and hidden activations from the video prediction module to resolve disagreements between the predictions. Our experiments demonstrate that our approach produces smoother and more accurate results than state-of-The-Art methods for general video inpainting, frame interpolation, and video prediction.

    Original languageEnglish
    Article number8892406
    Pages (from-to)1053-1068
    Number of pages16
    JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
    Volume42
    Issue number5
    DOIs
    StatePublished - 1 May 2020

    Keywords

    • Video inpainting
    • frame interpolation
    • temporal upsampling
    • video prediction

    Fingerprint

    Dive into the research topics of 'A Temporally-Aware Interpolation Network for Video Frame Inpainting'. Together they form a unique fingerprint.

    Cite this