TY - GEN
T1 - Crowdsourcing the Perception of Machine Teaching
AU - Hong, Jonggi
AU - Lee, Kyungjun
AU - Xu, June
AU - Kacorri, Hernisa
N1 - Publisher Copyright:
© 2020 ACM.
PY - 2020/4/21
Y1 - 2020/4/21
N2 - Teachable interfaces can empower end-users to attune machine learning systems to their idiosyncratic characteristics and environment by explicitly providing pertinent training examples. While facilitating control, their effectiveness can be hindered by the lack of expertise or misconceptions. We investigate how users may conceptualize, experience, and reflect on their engagement in machine teaching by deploying a mobile teachable testbed in Amazon Mechanical Turk. Using a performance-based payment scheme, Mechanical Turkers (N=100) are called to train, test, and re-train a robust recognition model in real-time with a few snapshots taken in their environment. We find that participants incorporate diversity in their examples drawing from parallels to how humans recognize objects independent of size, viewpoint, location, and illumination. Many of their misconceptions relate to consistency and model capabilities for reasoning. With limited variation and edge cases in testing, the majority of them do not change strategies on a second training attempt.
AB - Teachable interfaces can empower end-users to attune machine learning systems to their idiosyncratic characteristics and environment by explicitly providing pertinent training examples. While facilitating control, their effectiveness can be hindered by the lack of expertise or misconceptions. We investigate how users may conceptualize, experience, and reflect on their engagement in machine teaching by deploying a mobile teachable testbed in Amazon Mechanical Turk. Using a performance-based payment scheme, Mechanical Turkers (N=100) are called to train, test, and re-train a robust recognition model in real-time with a few snapshots taken in their environment. We find that participants incorporate diversity in their examples drawing from parallels to how humans recognize objects independent of size, viewpoint, location, and illumination. Many of their misconceptions relate to consistency and model capabilities for reasoning. With limited variation and edge cases in testing, the majority of them do not change strategies on a second training attempt.
KW - crowdsourcing
KW - interactive machine learning
KW - object recognition
KW - personalization
KW - teachable interfaces
UR - http://www.scopus.com/inward/record.url?scp=85091270562&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85091270562&partnerID=8YFLogxK
U2 - 10.1145/3313831.3376428
DO - 10.1145/3313831.3376428
M3 - Conference contribution
AN - SCOPUS:85091270562
T3 - Conference on Human Factors in Computing Systems - Proceedings
BT - CHI 2020 - Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
T2 - 2020 ACM CHI Conference on Human Factors in Computing Systems, CHI 2020
Y2 - 25 April 2020 through 30 April 2020
ER -