Egocentric object recognition leveraging the 3D shape of the grasping hand

Yizhou Lin, Gang Hua, Philippos Mordohai

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

We present a systematic study on the relationship between the 3D shape of a hand that is about to grasp an object and recognition of the object to be grasped. In this paper, we investigate the direction from the shape of the hand to object recognition for unimpaired users. Our work shows that the 3D shape of a grasping hand from an egocentric point of view can help improve recognition of the objects being grasped. Previous work has attempted to exploit hand interactions or gaze information in the egocentric setting to guide object segmentation. However, all such analyses are conducted in 2D. We hypothesize that the 3D shape of a grasping hand is highly correlated to the physical attributes of the object being grasped. Hence, it can provide very beneficial visual information for object recognition. We validate this hypothesis by first building a 3D, egocentric vision pipeline to segment and reconstruct dense 3D point clouds of the grasping hands. Then, visual descriptors are extracted from the point cloud and subsequently fed into an object recognition system to recognize the object being grasped. Our experiments demonstrate that the 3D hand shape can indeed greatly help improve the visual recognition accuracy, when compared with the baseline where only 2D image features are utilized.

Original languageEnglish
Title of host publicationComputer Vision - ECCV 2014 Workshops, Proceedings
EditorsCarsten Rother, Lourdes Agapito, Michael M. Bronstein
Pages746-762
Number of pages17
ISBN (Electronic)9783319161983
DOIs
StatePublished - 2015
Event13th European Conference on Computer Vision, ECCV 2014 - Zurich, Switzerland
Duration: 6 Sep 201412 Sep 2014

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume8927
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference13th European Conference on Computer Vision, ECCV 2014
Country/TerritorySwitzerland
CityZurich
Period6/09/1412/09/14

Keywords

  • Activity monitoring systems
  • Egocentric and first-person vision
  • Mobile and wearable systems
  • Rehabilitation aids

Fingerprint

Dive into the research topics of 'Egocentric object recognition leveraging the 3D shape of the grasping hand'. Together they form a unique fingerprint.

Cite this