TY - JOUR
T1 - VORI
T2 - A framework for testing voice user interface interactability
AU - Alrumayh, Abrar S.
AU - Tan, Chiu C.
N1 - Publisher Copyright:
© 2022
PY - 2022/9
Y1 - 2022/9
N2 - The ability of Voice User Interface (VUI) to understand how users will express their commands naturally and intuitively is an essential component of user experience, especially when the user is interacting with the VUI for the first time. Designing an automated method for testing the usability of VUI is a challenge for two reasons. First, there are many different ways for a user to express the same intention, e.g. “play some music”, ””put some music on”, etc., that is difficult to determine in advance. Second, many VUI apps today typically rely on the platform service provider (e.g. Amazon, Google, etc.) to perform many of the speech recognition and natural language processing tasks, and these services are provided as a blackbox. Consequently, it is difficult for the app developer to obtain information about errors and user feedback. In this paper, we propose a framework, VORI, to systematically evaluate the interactability of VUI, as well as a new metric for quantifying the interactability of a VUI. We use VORI to analyze 127 applications on Alexa by sending over 82,931 commands. Our analysis results highlight that 41.7% of apps only accept strict input that has to exactly match the developer's predefined sample commands with an interactability score of 20% or less. This suggests developers should consider a better interactability strategy in the design of VUIs, and more research is needed to further explore the design space to improve the interactability.
AB - The ability of Voice User Interface (VUI) to understand how users will express their commands naturally and intuitively is an essential component of user experience, especially when the user is interacting with the VUI for the first time. Designing an automated method for testing the usability of VUI is a challenge for two reasons. First, there are many different ways for a user to express the same intention, e.g. “play some music”, ””put some music on”, etc., that is difficult to determine in advance. Second, many VUI apps today typically rely on the platform service provider (e.g. Amazon, Google, etc.) to perform many of the speech recognition and natural language processing tasks, and these services are provided as a blackbox. Consequently, it is difficult for the app developer to obtain information about errors and user feedback. In this paper, we propose a framework, VORI, to systematically evaluate the interactability of VUI, as well as a new metric for quantifying the interactability of a VUI. We use VORI to analyze 127 applications on Alexa by sending over 82,931 commands. Our analysis results highlight that 41.7% of apps only accept strict input that has to exactly match the developer's predefined sample commands with an interactability score of 20% or less. This suggests developers should consider a better interactability strategy in the design of VUIs, and more research is needed to further explore the design space to improve the interactability.
KW - Home voice assistant
KW - Smart speaker
KW - Usability
KW - Voice user interface
UR - http://www.scopus.com/inward/record.url?scp=85133767759&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85133767759&partnerID=8YFLogxK
U2 - 10.1016/j.hcc.2022.100069
DO - 10.1016/j.hcc.2022.100069
M3 - Article
AN - SCOPUS:85133767759
VL - 2
JO - High-Confidence Computing
JF - High-Confidence Computing
IS - 3
M1 - 100069
ER -