TY - GEN
T1 - Human-Centered Deferred Inference
T2 - 28th International Conference on Intelligent User Interfaces, IUI 2023
AU - Lemmer, Stephan J.
AU - Guo, Anhong
AU - Corso, Jason J.
N1 - Publisher Copyright:
© 2023 Owner/Author.
PY - 2023/3/27
Y1 - 2023/3/27
N2 - Although deep learning holds the promise of novel and impactful interfaces, realizing such promise in practice remains a challenge: since dataset-driven deep-learned models assume a one-time human input, there is no recourse when they do not understand the input provided by the user. Works that address this via deferred inference - soliciting additional human input when uncertain - show meaningful improvement, but ignore key aspects of how users and models interact. In this work, we focus on the role of users in deferred inference and argue that the deferral criteria should be a function of the user and model as a team, not simply the model itself. In support of this, we introduce a novel mathematical formulation, validate it via an experiment analyzing the interactions of 25 individuals with a deep learning-based visiolinguistic model, and identify user-specific dependencies that are under-explored in prior work. We conclude by demonstrating two human-centered procedures for setting deferral criteria that are simple to implement, applicable to a wide variety of tasks, and perform equal to or better than equivalent procedures that use much larger datasets.
AB - Although deep learning holds the promise of novel and impactful interfaces, realizing such promise in practice remains a challenge: since dataset-driven deep-learned models assume a one-time human input, there is no recourse when they do not understand the input provided by the user. Works that address this via deferred inference - soliciting additional human input when uncertain - show meaningful improvement, but ignore key aspects of how users and models interact. In this work, we focus on the role of users in deferred inference and argue that the deferral criteria should be a function of the user and model as a team, not simply the model itself. In support of this, we introduce a novel mathematical formulation, validate it via an experiment analyzing the interactions of 25 individuals with a deep learning-based visiolinguistic model, and identify user-specific dependencies that are under-explored in prior work. We conclude by demonstrating two human-centered procedures for setting deferral criteria that are simple to implement, applicable to a wide variety of tasks, and perform equal to or better than equivalent procedures that use much larger datasets.
KW - deferred inference
KW - neural networks
KW - referring expression comprehension
UR - http://www.scopus.com/inward/record.url?scp=85152146522&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85152146522&partnerID=8YFLogxK
U2 - 10.1145/3581641.3584092
DO - 10.1145/3581641.3584092
M3 - Conference contribution
AN - SCOPUS:85152146522
T3 - International Conference on Intelligent User Interfaces, Proceedings IUI
SP - 681
EP - 694
BT - IUI 2023 - Proceedings of the 28th International Conference on Intelligent User Interfaces
Y2 - 27 March 2023 through 31 March 2023
ER -