Skip to main navigation Skip to search Skip to main content

The DEVIL is in the Details: A Diagnostic Evaluation Benchmark for Video Inpainting

  • Ryan Szeto
  • , Jason J. Corso
    • University of Michigan, Ann Arbor

    Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

    7 Scopus citations

    Abstract

    Quantitative evaluation has increased dramatically among recent video inpainting work, but the video and mask content used to gauge performance has received relatively little attention. Although attributes such as camera and background scene motion inherently change the difficulty of the task and affect methods differently, existing evaluation schemes fail to control for them, thereby providing minimal insight into inpainting failure modes. To address this gap, we propose the Diagnostic Evaluation of Video Inpainting on Landscapes (DEVIL) benchmark, which consists of two contributions: (i) a novel dataset of videos and masks labeled according to several key inpainting failure modes, and (ii) an evaluation scheme that samples slices of the dataset characterized by a fixed content attribute, and scores performance on each slice according to reconstruction, realism, and temporal consistency quality. By revealing systematic changes in performance induced by particular characteristics of the input content, our challenging benchmark enables more insightful analysis into video inpainting methods and serves as an invaluable diagnostic tool for the field. Our code and data are available at github.com/MichiganCOG/devil.

    Original languageEnglish
    Title of host publicationProceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
    Pages21022-21031
    Number of pages10
    ISBN (Electronic)9781665469463
    DOIs
    StatePublished - 2022
    Event2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022 - New Orleans, United States
    Duration: 19 Jun 2022 → 24 Jun 2022

    Publication series

    NameProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
    Volume2022-June
    ISSN (Print)1063-6919

    Conference

    Conference2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
    Country/TerritoryUnited States
    CityNew Orleans
    Period19/06/22 → 24/06/22

    Keywords

    • Datasets and evaluation
    • Image and video synthesis and generation
    • Video analysis and understanding

    Fingerprint

    Dive into the research topics of 'The DEVIL is in the Details: A Diagnostic Evaluation Benchmark for Video Inpainting'. Together they form a unique fingerprint.

    Cite this