Human-Centered Deferred Inference: Measuring User Interactions and Setting Deferral Criteria for Human-AI Teams

Stephan J. Lemmer, Anhong Guo, Jason J. Corso

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

2 Scopus citations

Abstract

Although deep learning holds the promise of novel and impactful interfaces, realizing such promise in practice remains a challenge: since dataset-driven deep-learned models assume a one-time human input, there is no recourse when they do not understand the input provided by the user. Works that address this via deferred inference - soliciting additional human input when uncertain - show meaningful improvement, but ignore key aspects of how users and models interact. In this work, we focus on the role of users in deferred inference and argue that the deferral criteria should be a function of the user and model as a team, not simply the model itself. In support of this, we introduce a novel mathematical formulation, validate it via an experiment analyzing the interactions of 25 individuals with a deep learning-based visiolinguistic model, and identify user-specific dependencies that are under-explored in prior work. We conclude by demonstrating two human-centered procedures for setting deferral criteria that are simple to implement, applicable to a wide variety of tasks, and perform equal to or better than equivalent procedures that use much larger datasets.

Original languageEnglish
Title of host publicationIUI 2023 - Proceedings of the 28th International Conference on Intelligent User Interfaces
Pages681-694
Number of pages14
ISBN (Electronic)9798400701061
DOIs
StatePublished - 27 Mar 2023
Event28th International Conference on Intelligent User Interfaces, IUI 2023 - Sydney, Australia
Duration: 27 Mar 202331 Mar 2023

Publication series

NameInternational Conference on Intelligent User Interfaces, Proceedings IUI

Conference

Conference28th International Conference on Intelligent User Interfaces, IUI 2023
Country/TerritoryAustralia
CitySydney
Period27/03/2331/03/23

Keywords

  • deferred inference
  • neural networks
  • referring expression comprehension

Fingerprint

Dive into the research topics of 'Human-Centered Deferred Inference: Measuring User Interactions and Setting Deferral Criteria for Human-AI Teams'. Together they form a unique fingerprint.

Cite this