Revisiting blind photography in the context of teachable object recognizers

Kyungjun Lee, Jonggi Hong, Simone Pimento, Ebrima Jarjue, Hernisa Kacorri

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

27 Scopus citations

Abstract

For people with visual impairments, photography is essential in identifying objects through remote sighted help and image recognition apps. This is especially the case for teachable object recognizers, where recognition models are trained on user's photos. Here, we propose real-time feedback for communicating the location of an object of interest in the camera frame. Our audio-haptic feedback is powered by a deep learning model that estimates the object center location based on its proximity to the user's hand. To evaluate our approach, we conducted a user study in the lab, where participants with visual impairments (N = 9) used our feedback to train and test their object recognizer in vanilla and cluttered environments. We found that very few photos did not include the object (2% in the vanilla and 8% in the cluttered) and the recognition performance was promising even for participants with no prior camera experience. Participants tended to trust the feedback even though they know it can be wrong. Our cluster analysis indicates that better feedback is associated with photos that include the entire object. Our results provide insights into factors that can degrade feedback and recognition performance in teachable interfaces.

Original languageEnglish
Title of host publicationASSETS 2019 - 21st International ACM SIGACCESS Conference on Computers and Accessibility
Pages83-95
Number of pages13
ISBN (Electronic)9781450366762
DOIs
StatePublished - 24 Oct 2019
Event21st International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS 2019 - Pittsburgh, United States
Duration: 28 Oct 201930 Oct 2019

Publication series

NameASSETS 2019 - 21st International ACM SIGACCESS Conference on Computers and Accessibility

Conference

Conference21st International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS 2019
Country/TerritoryUnited States
CityPittsburgh
Period28/10/1930/10/19

Keywords

  • Hand
  • Object recognition
  • Sonification
  • Visual impairments

Fingerprint

Dive into the research topics of 'Revisiting blind photography in the context of teachable object recognizers'. Together they form a unique fingerprint.

Cite this