3D Human Pose and Shape Estimation Through Collaborative Learning and Multi-View Model-Fitting

Research output: Chapter in Book/Report/Conference proceedingPaper in conference proceedingpeer-review


3D human pose and shape estimation plays a vital role in many computer vision applications. There are many deep learning based methods attempting to solve the problem only relying on single-view RGB images for training the network. However, since some public datasets are captured from multi-view cameras system, we propose a novel method to tackle the problem by putting optimization-based multi-view model-fitting into a regression-based learning loop from multi-view images. Firstly, a convolutional neural network (CNN) regresses the pose and shape of a parametric human body model (SMPL) from multi-view images. Then, utilizing the regressed pose and shape as initialization, we propose an improved multi-view optimization method based on the SMPLify method (MV-SMPLify) to fit the SMPL model to the multi-view images simultaneously. Subsequently, the optimized parameters can be adopted to supervise the training of the CNN model. This whole process forms a self-supervising framework which can combine the advantages of the CNN approach and the optimization-based approach through a collaborative process. In addition, the multi-view images can provide more comprehensive supervision for the training. Experiments on public datasets qualitatively and quantitatively demonstrate that our method outperforms previous approaches in a number of ways.
Original languageEnglish
Title of host publicationWACV - IEEE Winter Conference on Applications of Computer Vision
PublisherIEEE Computer Society
Number of pages10
ISBN (Electronic)978-1-6654-0477-8
ISBN (Print)978-1-6654-4640-2
Publication statusPublished - 2021
Event2021 IEEE Winter Conference on Applications of Computer Vision (WACV) - Waikoloa, United States
Duration: 2021 Jan 32021 Jan 8

Publication series

NameIEEE Winter Conference on Applications of Computer Vision (WACV)
ISSN (Electronic)2642-9381


Conference2021 IEEE Winter Conference on Applications of Computer Vision (WACV)
Abbreviated titleWACV 2021
Country/TerritoryUnited States

Subject classification (UKÄ)

  • Computer Vision and Robotics (Autonomous Systems)

Cite this