Egocentric object recognition leveraging the 3D shape of the grasping hand

Yizhou Lin; Gang Hua; Philippos Mordohai

doi:10.1007/978-3-319-16199-0_52

Egocentric object recognition leveraging the 3D shape of the grasping hand

Yizhou Lin
, Gang Hua
, Philippos Mordohai

Department of Computer Science

Stevens Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

We present a systematic study on the relationship between the 3D shape of a hand that is about to grasp an object and recognition of the object to be grasped. In this paper, we investigate the direction from the shape of the hand to object recognition for unimpaired users. Our work shows that the 3D shape of a grasping hand from an egocentric point of view can help improve recognition of the objects being grasped. Previous work has attempted to exploit hand interactions or gaze information in the egocentric setting to guide object segmentation. However, all such analyses are conducted in 2D. We hypothesize that the 3D shape of a grasping hand is highly correlated to the physical attributes of the object being grasped. Hence, it can provide very beneficial visual information for object recognition. We validate this hypothesis by first building a 3D, egocentric vision pipeline to segment and reconstruct dense 3D point clouds of the grasping hands. Then, visual descriptors are extracted from the point cloud and subsequently fed into an object recognition system to recognize the object being grasped. Our experiments demonstrate that the 3D hand shape can indeed greatly help improve the visual recognition accuracy, when compared with the baseline where only 2D image features are utilized.

Original language	English
Title of host publication	Computer Vision - ECCV 2014 Workshops, Proceedings
Editors	Carsten Rother, Lourdes Agapito, Michael M. Bronstein
Pages	746-762
Number of pages	17
ISBN (Electronic)	9783319161983
DOIs	https://doi.org/10.1007/978-3-319-16199-0_52
State	Published - 2015
Event	13th European Conference on Computer Vision, ECCV 2014 - Zurich, Switzerland Duration: 6 Sep 2014 → 12 Sep 2014

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	8927
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	13th European Conference on Computer Vision, ECCV 2014
Country/Territory	Switzerland
City	Zurich
Period	6/09/14 → 12/09/14

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being

Keywords

Activity monitoring systems
Egocentric and first-person vision
Mobile and wearable systems
Rehabilitation aids

Access to Document

10.1007/978-3-319-16199-0_52

Cite this

Lin, Y., Hua, G., & Mordohai, P. (2015). Egocentric object recognition leveraging the 3D shape of the grasping hand. In C. Rother, L. Agapito, & M. M. Bronstein (Eds.), Computer Vision - ECCV 2014 Workshops, Proceedings (pp. 746-762). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8927). https://doi.org/10.1007/978-3-319-16199-0_52

Lin, Yizhou ; Hua, Gang ; Mordohai, Philippos. / Egocentric object recognition leveraging the 3D shape of the grasping hand. Computer Vision - ECCV 2014 Workshops, Proceedings. editor / Carsten Rother ; Lourdes Agapito ; Michael M. Bronstein. 2015. pp. 746-762 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{5a73e2727b9247e8b9b2d2776b3636eb,

title = "Egocentric object recognition leveraging the 3D shape of the grasping hand",

abstract = "We present a systematic study on the relationship between the 3D shape of a hand that is about to grasp an object and recognition of the object to be grasped. In this paper, we investigate the direction from the shape of the hand to object recognition for unimpaired users. Our work shows that the 3D shape of a grasping hand from an egocentric point of view can help improve recognition of the objects being grasped. Previous work has attempted to exploit hand interactions or gaze information in the egocentric setting to guide object segmentation. However, all such analyses are conducted in 2D. We hypothesize that the 3D shape of a grasping hand is highly correlated to the physical attributes of the object being grasped. Hence, it can provide very beneficial visual information for object recognition. We validate this hypothesis by first building a 3D, egocentric vision pipeline to segment and reconstruct dense 3D point clouds of the grasping hands. Then, visual descriptors are extracted from the point cloud and subsequently fed into an object recognition system to recognize the object being grasped. Our experiments demonstrate that the 3D hand shape can indeed greatly help improve the visual recognition accuracy, when compared with the baseline where only 2D image features are utilized.",

keywords = "Activity monitoring systems, Egocentric and first-person vision, Mobile and wearable systems, Rehabilitation aids",

author = "Yizhou Lin and Gang Hua and Philippos Mordohai",

note = "Publisher Copyright: {\textcopyright} Springer International Publishing Switzerland 2015.; 13th European Conference on Computer Vision, ECCV 2014 ; Conference date: 06-09-2014 Through 12-09-2014",

year = "2015",

doi = "10.1007/978-3-319-16199-0\_52",

language = "English",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

pages = "746--762",

editor = "Carsten Rother and Lourdes Agapito and Bronstein, \{Michael M.\}",

booktitle = "Computer Vision - ECCV 2014 Workshops, Proceedings",

}

Lin, Y, Hua, G & Mordohai, P 2015, Egocentric object recognition leveraging the 3D shape of the grasping hand. in C Rother, L Agapito & MM Bronstein (eds), Computer Vision - ECCV 2014 Workshops, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8927, pp. 746-762, 13th European Conference on Computer Vision, ECCV 2014, Zurich, Switzerland, 6/09/14. https://doi.org/10.1007/978-3-319-16199-0_52

Egocentric object recognition leveraging the 3D shape of the grasping hand. / Lin, Yizhou; Hua, Gang; Mordohai, Philippos.
Computer Vision - ECCV 2014 Workshops, Proceedings. ed. / Carsten Rother; Lourdes Agapito; Michael M. Bronstein. 2015. p. 746-762 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8927).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Egocentric object recognition leveraging the 3D shape of the grasping hand

AU - Lin, Yizhou

AU - Hua, Gang

AU - Mordohai, Philippos

N1 - Publisher Copyright: © Springer International Publishing Switzerland 2015.

PY - 2015

Y1 - 2015

N2 - We present a systematic study on the relationship between the 3D shape of a hand that is about to grasp an object and recognition of the object to be grasped. In this paper, we investigate the direction from the shape of the hand to object recognition for unimpaired users. Our work shows that the 3D shape of a grasping hand from an egocentric point of view can help improve recognition of the objects being grasped. Previous work has attempted to exploit hand interactions or gaze information in the egocentric setting to guide object segmentation. However, all such analyses are conducted in 2D. We hypothesize that the 3D shape of a grasping hand is highly correlated to the physical attributes of the object being grasped. Hence, it can provide very beneficial visual information for object recognition. We validate this hypothesis by first building a 3D, egocentric vision pipeline to segment and reconstruct dense 3D point clouds of the grasping hands. Then, visual descriptors are extracted from the point cloud and subsequently fed into an object recognition system to recognize the object being grasped. Our experiments demonstrate that the 3D hand shape can indeed greatly help improve the visual recognition accuracy, when compared with the baseline where only 2D image features are utilized.

AB - We present a systematic study on the relationship between the 3D shape of a hand that is about to grasp an object and recognition of the object to be grasped. In this paper, we investigate the direction from the shape of the hand to object recognition for unimpaired users. Our work shows that the 3D shape of a grasping hand from an egocentric point of view can help improve recognition of the objects being grasped. Previous work has attempted to exploit hand interactions or gaze information in the egocentric setting to guide object segmentation. However, all such analyses are conducted in 2D. We hypothesize that the 3D shape of a grasping hand is highly correlated to the physical attributes of the object being grasped. Hence, it can provide very beneficial visual information for object recognition. We validate this hypothesis by first building a 3D, egocentric vision pipeline to segment and reconstruct dense 3D point clouds of the grasping hands. Then, visual descriptors are extracted from the point cloud and subsequently fed into an object recognition system to recognize the object being grasped. Our experiments demonstrate that the 3D hand shape can indeed greatly help improve the visual recognition accuracy, when compared with the baseline where only 2D image features are utilized.

KW - Activity monitoring systems

KW - Egocentric and first-person vision

KW - Mobile and wearable systems

KW - Rehabilitation aids

UR - https://www.scopus.com/pages/publications/84928787973

UR - https://www.scopus.com/pages/publications/84928787973#tab=citedBy

U2 - 10.1007/978-3-319-16199-0_52

DO - 10.1007/978-3-319-16199-0_52

M3 - Conference contribution

AN - SCOPUS:84928787973

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 746

EP - 762

BT - Computer Vision - ECCV 2014 Workshops, Proceedings

A2 - Rother, Carsten

A2 - Agapito, Lourdes

A2 - Bronstein, Michael M.

T2 - 13th European Conference on Computer Vision, ECCV 2014

Y2 - 6 September 2014 through 12 September 2014

ER -

Lin Y, Hua G, Mordohai P. Egocentric object recognition leveraging the 3D shape of the grasping hand. In Rother C, Agapito L, Bronstein MM, editors, Computer Vision - ECCV 2014 Workshops, Proceedings. 2015. p. 746-762. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-16199-0_52

Egocentric object recognition leveraging the 3D shape of the grasping hand

Abstract

Publication series

Conference

UN SDGs

Keywords

Access to Document

Other files and links

Fingerprint

Cite this