TY - GEN
T1 - A Temporally-Aware Interpolation Network for Video Frame Inpainting
AU - Sun, Ximeng
AU - Szeto, Ryan
AU - Corso, Jason J.
N1 - Publisher Copyright:
© 2019, Springer Nature Switzerland AG.
PY - 2019
Y1 - 2019
N2 - We propose the first deep learning solution to video frame inpainting, a more challenging but less ambiguous task than related problems such as general video inpainting, frame interpolation, and video prediction. We devise a pipeline composed of two modules: a bidirectional video prediction module and a temporally-aware frame interpolation module. The prediction module makes two intermediate predictions of the missing frames, each conditioned on the preceding and following frames respectively, using a shared convolutional LSTM-based encoder-decoder. The interpolation module blends the intermediate predictions, using time information and hidden activations from the video prediction module to resolve disagreements between the predictions. Our experiments demonstrate that our approach produces more accurate and qualitatively satisfying results than a state-of-the-art video prediction method and many strong frame inpainting baselines. Our code is available at https://github.com/sunxm2357/TAI_video_frame_inpainting.
AB - We propose the first deep learning solution to video frame inpainting, a more challenging but less ambiguous task than related problems such as general video inpainting, frame interpolation, and video prediction. We devise a pipeline composed of two modules: a bidirectional video prediction module and a temporally-aware frame interpolation module. The prediction module makes two intermediate predictions of the missing frames, each conditioned on the preceding and following frames respectively, using a shared convolutional LSTM-based encoder-decoder. The interpolation module blends the intermediate predictions, using time information and hidden activations from the video prediction module to resolve disagreements between the predictions. Our experiments demonstrate that our approach produces more accurate and qualitatively satisfying results than a state-of-the-art video prediction method and many strong frame inpainting baselines. Our code is available at https://github.com/sunxm2357/TAI_video_frame_inpainting.
KW - Frame interpolation
KW - Video inpainting
KW - Video prediction
UR - https://www.scopus.com/pages/publications/85067237848
UR - https://www.scopus.com/pages/publications/85067237848#tab=citedBy
U2 - 10.1007/978-3-030-20893-6_16
DO - 10.1007/978-3-030-20893-6_16
M3 - Conference contribution
AN - SCOPUS:85067237848
SN - 9783030208929
T3 - Lecture Notes in Computer Science
SP - 249
EP - 264
BT - Computer Vision – ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers
A2 - Jawahar, C.V.
A2 - Li, Hongdong
A2 - Mori, Greg
A2 - Schindler, Konrad
T2 - 2018 Scene Understanding and Modelling Challenge, SUMO 2018, 2018Learning and Inference Methods for High-Performance Imaging, LIMHPI 2018, 2018 Attention/Intention Understanding, AIU 2018, 2018 Museum Exhibit Identification Challenge for Domain Adaptation and Few-Shot Learning, 2018 RGB-D—Sensing and Understanding via Combined Color and Depth, 2018 Dense 3D Reconstruction for Dynamic Scenes, 2018 AI Aesthetics in Art and Media, AIAM 2018, 3rd International Workshop on Robust Reading, IWRR 2018, 2018 Artificial Intelligence for Retinal Image Analysis, AIRIA 2018, 2018 Combining Vision and Language, 1st International Workshop on Advanced Machine Vision for Real-Life and Industrially Relevant Applications, AMV 2018
Y2 - 2 December 2018 through 6 December 2018
ER -