TY - GEN
T1 - CustomSight
T2 - 38th Annual ACM Symposium on User Interface Software and Technology, UIST 2025
AU - Rahman, Adil
AU - Khan, Rifat Rahman
AU - Hong, Jonggi
AU - Valencia, Stephanie
AU - Heo, Seongkook
N1 - Publisher Copyright:
© 2025 Copyright held by the owner/author(s).
PY - 2025/9/27
Y1 - 2025/9/27
N2 - LLM-powered assistive technologies (ATs) have enabled blind and visually impaired (BVI) users to query personalized, goal-oriented information about their visual environment. However, the accuracy of system responses depends heavily on well-framed, query-relevant images, which can be difficult for BVI users to capture. We present CustomSight, an LLM-powered AT that helps BVI users effectively query visual information by providing task-aware, real-time guidance to frame the camera and automatically capture images when relevant content is in view. When a user issues a query, CustomSight generates a Dynamic Filter-a custom pipeline that encodes logic tied to the user’s intent, monitors the live feed, and triggers context-aware feedback and image capture. The captured image is sent to the LLM to fetch accurate visual information.
AB - LLM-powered assistive technologies (ATs) have enabled blind and visually impaired (BVI) users to query personalized, goal-oriented information about their visual environment. However, the accuracy of system responses depends heavily on well-framed, query-relevant images, which can be difficult for BVI users to capture. We present CustomSight, an LLM-powered AT that helps BVI users effectively query visual information by providing task-aware, real-time guidance to frame the camera and automatically capture images when relevant content is in view. When a user issues a query, CustomSight generates a Dynamic Filter-a custom pipeline that encodes logic tied to the user’s intent, monitors the live feed, and triggers context-aware feedback and image capture. The captured image is sent to the LLM to fetch accurate visual information.
KW - Accessibility
KW - Assistive Technology
KW - Blind and Low Vision
KW - LLMs
UR - https://www.scopus.com/pages/publications/105020850823
UR - https://www.scopus.com/pages/publications/105020850823#tab=citedBy
U2 - 10.1145/3746058.3758401
DO - 10.1145/3746058.3758401
M3 - Conference contribution
AN - SCOPUS:105020850823
T3 - UIST Adjunct 2025 - Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology
BT - UIST Adjunct 2025 - Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology
A2 - Bianchi, Andrea
A2 - Glassman, Elena
A2 - Zhao, Shengdong
A2 - Kim, Jeeeun
A2 - Oakley, Ian
A2 - Mackay, Wendy E.
Y2 - 28 September 2025 through 1 October 2025
ER -