TY - JOUR
T1 - Single-camera 3D head fitting for mixed reality clinical applications
AU - Mane, Tejas
AU - Bayramova, Aylar
AU - Daniilidis, Kostas
AU - Mordohai, Philippos
AU - Bernardis, Elena
N1 - Publisher Copyright:
© 2022 Elsevier Inc.
PY - 2022/4
Y1 - 2022/4
N2 - We address the problem of estimating the shape of a person's head, defined as the geometry of the complete head surface, from a video taken with a single moving camera, and determining the alignment of the fitted 3D head for all video frames, irrespective of the person's pose. 3D head reconstructions commonly tend to focus on perfecting the face reconstruction, leaving the scalp to a statistical approximation. Our goal is to reconstruct the head model of each person to enable future mixed reality applications. To do this, we recover a dense 3D reconstruction and camera information via structure-from-motion and multi-view stereo. These are then used in a new two-stage fitting process to recover the 3D head shape by iteratively fitting a 3D morphable model of the head with the dense reconstruction in canonical space and fitting it to each person's head, using both traditional facial landmarks and scalp features extracted from the head's segmentation mask. Our approach recovers consistent geometry for varying head shapes, from videos taken by different people, with different smartphones, and in a variety of environments from living rooms to outdoor spaces.
AB - We address the problem of estimating the shape of a person's head, defined as the geometry of the complete head surface, from a video taken with a single moving camera, and determining the alignment of the fitted 3D head for all video frames, irrespective of the person's pose. 3D head reconstructions commonly tend to focus on perfecting the face reconstruction, leaving the scalp to a statistical approximation. Our goal is to reconstruct the head model of each person to enable future mixed reality applications. To do this, we recover a dense 3D reconstruction and camera information via structure-from-motion and multi-view stereo. These are then used in a new two-stage fitting process to recover the 3D head shape by iteratively fitting a 3D morphable model of the head with the dense reconstruction in canonical space and fitting it to each person's head, using both traditional facial landmarks and scalp features extracted from the head's segmentation mask. Our approach recovers consistent geometry for varying head shapes, from videos taken by different people, with different smartphones, and in a variety of environments from living rooms to outdoor spaces.
KW - 3D head reconstruction
KW - 3DMM fitting
KW - Head features
UR - http://www.scopus.com/inward/record.url?scp=85125584166&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85125584166&partnerID=8YFLogxK
U2 - 10.1016/j.cviu.2022.103384
DO - 10.1016/j.cviu.2022.103384
M3 - Article
AN - SCOPUS:85125584166
SN - 1077-3142
VL - 218
JO - Computer Vision and Image Understanding
JF - Computer Vision and Image Understanding
M1 - 103384
ER -