TY - GEN
T1 - A Temporally-Aware Interpolation Network for Video Frame Inpainting
AU - Sun, Ximeng
AU - Szeto, Ryan
AU - Corso, Jason J.
N1 - Publisher Copyright:
© 2019, Springer Nature Switzerland AG.
PY - 2019
Y1 - 2019
N2 - We propose the first deep learning solution to video frame inpainting, a more challenging but less ambiguous task than related problems such as general video inpainting, frame interpolation, and video prediction. We devise a pipeline composed of two modules: a bidirectional video prediction module and a temporally-aware frame interpolation module. The prediction module makes two intermediate predictions of the missing frames, each conditioned on the preceding and following frames respectively, using a shared convolutional LSTM-based encoder-decoder. The interpolation module blends the intermediate predictions, using time information and hidden activations from the video prediction module to resolve disagreements between the predictions. Our experiments demonstrate that our approach produces more accurate and qualitatively satisfying results than a state-of-the-art video prediction method and many strong frame inpainting baselines. Our code is available at https://github.com/sunxm2357/TAI_video_frame_inpainting.
AB - We propose the first deep learning solution to video frame inpainting, a more challenging but less ambiguous task than related problems such as general video inpainting, frame interpolation, and video prediction. We devise a pipeline composed of two modules: a bidirectional video prediction module and a temporally-aware frame interpolation module. The prediction module makes two intermediate predictions of the missing frames, each conditioned on the preceding and following frames respectively, using a shared convolutional LSTM-based encoder-decoder. The interpolation module blends the intermediate predictions, using time information and hidden activations from the video prediction module to resolve disagreements between the predictions. Our experiments demonstrate that our approach produces more accurate and qualitatively satisfying results than a state-of-the-art video prediction method and many strong frame inpainting baselines. Our code is available at https://github.com/sunxm2357/TAI_video_frame_inpainting.
KW - Frame interpolation
KW - Video inpainting
KW - Video prediction
UR - http://www.scopus.com/inward/record.url?scp=85067237848&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85067237848&partnerID=8YFLogxK
U2 - 10.1007/978-3-030-20893-6_16
DO - 10.1007/978-3-030-20893-6_16
M3 - Conference contribution
AN - SCOPUS:85067237848
SN - 9783030208929
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 249
EP - 264
BT - Computer Vision - ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers
A2 - Schindler, Konrad
A2 - Jawahar, C.V.
A2 - Mori, Greg
A2 - Li, Hongdong
T2 - 14th Asian Conference on Computer Vision, ACCV 2018
Y2 - 2 December 2018 through 6 December 2018
ER -