HuMMan: Multi-modal 4D Human Dataset for Versatile Sensing and Modeling

Zhongang Cai; Daxuan Ren; Ailing Zeng; Zhengyu Lin; Tao Yu; Wenjia Wang; Xiangyu Fan; Yang Gao; Yifan Yu; Liang Pan; Fangzhou Hong; Mingyuan Zhang; Chen Change Loy; Lei Yang; Ziwei Liu

doi:10.1007/978-3-031-20071-7_33

HuMMan: Multi-modal 4D Human Dataset for Versatile Sensing and Modeling

Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, Fangzhou Hong, Mingyuan Zhang, Chen Change Loy, Lei Yang^*, Ziwei Liu

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

58 Citations (Scopus)

Abstract

4D human sensing and modeling are fundamental tasks in vision and graphics with numerous applications. With the advances of new sensors and algorithms, there is an increasing demand for more versatile datasets. In this work, we contribute HuMMan, a large-scale multi-modal 4D human dataset with 1000 human subjects, 400k sequences and 60M frames. HuMMan has several appealing properties: 1) multi-modal data and annotations including color images, point clouds, keypoints, SMPL parameters, and textured meshes; 2) popular mobile device is included in the sensor suite; 3) a set of 500 actions, designed to cover fundamental movements; 4) multiple tasks such as action recognition, pose estimation, parametric human recovery, and textured mesh reconstruction are supported and evaluated. Extensive experiments on HuMMan voice the need for further study on challenges such as fine-grained action recognition, dynamic human mesh reconstruction, point cloud-based parametric human recovery, and cross-device domain gaps (Homepage: https://caizhongang.github.io/projects/HuMMan/ ).

Original language	English
Title of host publication	Computer Vision – ECCV 2022 - 17th European Conference, Proceedings
Editors	Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	557-577
Number of pages	21
ISBN (Print)	9783031200700
DOIs	https://doi.org/10.1007/978-3-031-20071-7_33
Publication status	Published - 2022
Externally published	Yes
Event	17th European Conference on Computer Vision, ECCV 2022 - Tel Aviv, Israel Duration: Oct 23 2022 → Oct 27 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13667 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	17th European Conference on Computer Vision, ECCV 2022
Country/Territory	Israel
City	Tel Aviv
Period	10/23/22 → 10/27/22

Bibliographical note

Publisher Copyright:
© 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

ASJC Scopus Subject Areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-031-20071-7_33

Cite this

Cai, Z., Ren, D., Zeng, A., Lin, Z., Yu, T., Wang, W., Fan, X., Gao, Y., Yu, Y., Pan, L., Hong, F., Zhang, M., Loy, C. C., Yang, L., & Liu, Z. (2022). HuMMan: Multi-modal 4D Human Dataset for Versatile Sensing and Modeling. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, & T. Hassner (Eds.), Computer Vision – ECCV 2022 - 17th European Conference, Proceedings (pp. 557-577). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13667 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-20071-7_33

Cai, Zhongang ; Ren, Daxuan ; Zeng, Ailing et al. / HuMMan : Multi-modal 4D Human Dataset for Versatile Sensing and Modeling. Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. editor / Shai Avidan ; Gabriel Brostow ; Moustapha Cissé ; Giovanni Maria Farinella ; Tal Hassner. Springer Science and Business Media Deutschland GmbH, 2022. pp. 557-577 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{0657d4dd4f4f43bdaaa17e833643ae3d,

title = "HuMMan: Multi-modal 4D Human Dataset for Versatile Sensing and Modeling",

abstract = "4D human sensing and modeling are fundamental tasks in vision and graphics with numerous applications. With the advances of new sensors and algorithms, there is an increasing demand for more versatile datasets. In this work, we contribute HuMMan, a large-scale multi-modal 4D human dataset with 1000 human subjects, 400k sequences and 60M frames. HuMMan has several appealing properties: 1) multi-modal data and annotations including color images, point clouds, keypoints, SMPL parameters, and textured meshes; 2) popular mobile device is included in the sensor suite; 3) a set of 500 actions, designed to cover fundamental movements; 4) multiple tasks such as action recognition, pose estimation, parametric human recovery, and textured mesh reconstruction are supported and evaluated. Extensive experiments on HuMMan voice the need for further study on challenges such as fine-grained action recognition, dynamic human mesh reconstruction, point cloud-based parametric human recovery, and cross-device domain gaps (Homepage: https://caizhongang.github.io/projects/HuMMan/ ).",

author = "Zhongang Cai and Daxuan Ren and Ailing Zeng and Zhengyu Lin and Tao Yu and Wenjia Wang and Xiangyu Fan and Yang Gao and Yifan Yu and Liang Pan and Fangzhou Hong and Mingyuan Zhang and Loy, \{Chen Change\} and Lei Yang and Ziwei Liu",

note = "Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 17th European Conference on Computer Vision, ECCV 2022 ; Conference date: 23-10-2022 Through 27-10-2022",

year = "2022",

doi = "10.1007/978-3-031-20071-7\_33",

language = "English",

isbn = "9783031200700",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "557--577",

editor = "Shai Avidan and Gabriel Brostow and Moustapha Ciss{\'e} and Farinella, \{Giovanni Maria\} and Tal Hassner",

booktitle = "Computer Vision – ECCV 2022 - 17th European Conference, Proceedings",

address = "Germany",

}

Cai, Z, Ren, D, Zeng, A, Lin, Z, Yu, T, Wang, W, Fan, X, Gao, Y, Yu, Y, Pan, L, Hong, F, Zhang, M, Loy, CC, Yang, L & Liu, Z 2022, HuMMan: Multi-modal 4D Human Dataset for Versatile Sensing and Modeling. in S Avidan, G Brostow, M Cissé, GM Farinella & T Hassner (eds), Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13667 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 557-577, 17th European Conference on Computer Vision, ECCV 2022, Tel Aviv, Israel, 10/23/22. https://doi.org/10.1007/978-3-031-20071-7_33

HuMMan: Multi-modal 4D Human Dataset for Versatile Sensing and Modeling. / Cai, Zhongang; Ren, Daxuan; Zeng, Ailing et al.
Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. ed. / Shai Avidan; Gabriel Brostow; Moustapha Cissé; Giovanni Maria Farinella; Tal Hassner. Springer Science and Business Media Deutschland GmbH, 2022. p. 557-577 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13667 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - HuMMan

T2 - 17th European Conference on Computer Vision, ECCV 2022

AU - Cai, Zhongang

AU - Ren, Daxuan

AU - Zeng, Ailing

AU - Lin, Zhengyu

AU - Yu, Tao

AU - Wang, Wenjia

AU - Fan, Xiangyu

AU - Gao, Yang

AU - Yu, Yifan

AU - Pan, Liang

AU - Hong, Fangzhou

AU - Zhang, Mingyuan

AU - Loy, Chen Change

AU - Yang, Lei

AU - Liu, Ziwei

PY - 2022

Y1 - 2022

N2 - 4D human sensing and modeling are fundamental tasks in vision and graphics with numerous applications. With the advances of new sensors and algorithms, there is an increasing demand for more versatile datasets. In this work, we contribute HuMMan, a large-scale multi-modal 4D human dataset with 1000 human subjects, 400k sequences and 60M frames. HuMMan has several appealing properties: 1) multi-modal data and annotations including color images, point clouds, keypoints, SMPL parameters, and textured meshes; 2) popular mobile device is included in the sensor suite; 3) a set of 500 actions, designed to cover fundamental movements; 4) multiple tasks such as action recognition, pose estimation, parametric human recovery, and textured mesh reconstruction are supported and evaluated. Extensive experiments on HuMMan voice the need for further study on challenges such as fine-grained action recognition, dynamic human mesh reconstruction, point cloud-based parametric human recovery, and cross-device domain gaps (Homepage: https://caizhongang.github.io/projects/HuMMan/ ).

AB - 4D human sensing and modeling are fundamental tasks in vision and graphics with numerous applications. With the advances of new sensors and algorithms, there is an increasing demand for more versatile datasets. In this work, we contribute HuMMan, a large-scale multi-modal 4D human dataset with 1000 human subjects, 400k sequences and 60M frames. HuMMan has several appealing properties: 1) multi-modal data and annotations including color images, point clouds, keypoints, SMPL parameters, and textured meshes; 2) popular mobile device is included in the sensor suite; 3) a set of 500 actions, designed to cover fundamental movements; 4) multiple tasks such as action recognition, pose estimation, parametric human recovery, and textured mesh reconstruction are supported and evaluated. Extensive experiments on HuMMan voice the need for further study on challenges such as fine-grained action recognition, dynamic human mesh reconstruction, point cloud-based parametric human recovery, and cross-device domain gaps (Homepage: https://caizhongang.github.io/projects/HuMMan/ ).

UR - http://www.scopus.com/inward/record.url?scp=85142668253&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85142668253&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-20071-7_33

DO - 10.1007/978-3-031-20071-7_33

M3 - Conference contribution

AN - SCOPUS:85142668253

SN - 9783031200700

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 557

EP - 577

BT - Computer Vision – ECCV 2022 - 17th European Conference, Proceedings

A2 - Avidan, Shai

A2 - Brostow, Gabriel

A2 - Cissé, Moustapha

A2 - Farinella, Giovanni Maria

A2 - Hassner, Tal

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 23 October 2022 through 27 October 2022

ER -

Cai Z, Ren D, Zeng A, Lin Z, Yu T, Wang W et al. HuMMan: Multi-modal 4D Human Dataset for Versatile Sensing and Modeling. In Avidan S, Brostow G, Cissé M, Farinella GM, Hassner T, editors, Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. Springer Science and Business Media Deutschland GmbH. 2022. p. 557-577. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-20071-7_33