research-article

Real-time Physics-based Motion Capture with Sparse Sensors

Authors:

Sheldon Andrews,

Kenny MitchellAuthors Info & Claims

CVMP '16: Proceedings of the 13th European Conference on Visual Media Production (CVMP 2016)

Article No.: 5, Pages 1 - 10

https://doi.org/10.1145/2998559.2998564

Published: 12 December 2016 Publication History

Abstract

We propose a framework for real-time tracking of humans using sparse multi-modal sensor sets, including data obtained from optical markers and inertial measurement units. A small number of sensors leaves the performer unencumbered by not requiring dense coverage of the body. An inverse dynamics solver and physics-based body model are used, ensuring physical plausibility by computing joint torques and contact forces. A prior model is also used to give an improved estimate of motion of internal joints. The behaviour of our tracker is evaluated using several black box motion priors. We show that our system can track and simulate a wide range of dynamic movements including bipedal gait, ballistic movements such as jumping, and interaction with the environment. The reconstructed motion has low error and appears natural. As both the internal forces and contacts are obtained with high credibility, it is also useful for human movement analysis.

References

[1]

Open Dynamics Engine. http://www.ode.org/, 2015.

[2]

OptiTrack Motive. http://www.optitrack.com/, 2016.

[3]

Perception Neuron. http://www.neuronmocap.com/, 2016.

[4]

Vicon Blade. http://www.vicon.com/, 2016.

[5]

Xsens MVN. http://www.xsens.com/, 2016.

[6]

D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, and J. Davis. Scape: shape completion and animation of people. ACM Trans. on Graphics, 24(3):408--416, 2005.

Digital Library

[7]

M. A. Brubaker, L. Sigal, and D. Fleet. Physics-based human motion modeling for people tracking: A short tutorial. In Image, pages 1--48, 2009.

[8]

M. A. Brubaker, L. Sigal, and D. J. Fleet. Estimating contact dynamics. In IEEE 12th Intl. Conf. on Computer Vision, pages 2389--2396, 2009.

[9]

S. R. Buss. Introduction to inverse kinematics with jacobian transpose, pseudoinverse and damped least squares methods. IEEE Journal of Robotics and Automation, 17(1-19):16, 2004.

[10]

E. Catto. Soft constraints. In Game Developers Conference, 2011.

[11]

J. Chai and J. K. Hodgins. Performance animation from low-dimensional control signals. ACM Trans. on Graphics, 24(3):686--696, 2005.

Digital Library

[12]

X. Chen and D. Cai. Large scale spectral clustering with landmark-based representation. In AAAI, 2011.

Digital Library

[13]

M. Dou, S. Khamis, Y. Degtyarev, P. Davidson, S. R. Fanello, A. Kowdle, S. O. Escolano, C. Rhemann, D. Kim, J. Taylor, et al. Fusion 4D: real-time performance capture of challenging scenes. ACM Trans. on Graphics, 35(4):114, 2016.

Digital Library

[14]

K. Endo, D. Paluska, and H. Herr. A quasi-passive model of human leg function in level-ground walking. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems, pages 4935--4939, 2006.

[15]

S. Ha, Y. Bai, and C. K. Liu. Human motion reconstruction from force sensors. In Proc. of the 2011 ACM SIGGRAPH/Eurographics Symp. on Computer Animation, pages 129--138, 2011.

Digital Library

[16]

L. Hoyet, K. Ryall, R. McDonnell, and C. O'Sullivan. Sleight of hand: perception of finger motion from reduced marker sets. In Proc. of the ACM SIGGRAPH Symp. on Interactive 3D Graphics and Games, I3D '12, pages 79--86, New York, NY, USA, 2012. ACM.

Digital Library

[17]

C. Kang, N. Wheatland, M. Neff, and V. Zordan. Automatic hand-over animation for free-hand motions from low resolution input. In Intl. Conf. on Motion in Games, pages 244--253, 2012.

[18]

B. Koniaris, I. Huerta, M. Kosek, K. Darragh, C. Malleson, J. Jamrozy, N. Swafford, J. Guitian, B. Moon, A. Israr, S. Andrews, and K. Mitchell. Iridium: immersive rendered interactive deep media. In ACM SIGGRAPH 2016 VR Village. ACM, 2016.

Digital Library

[19]

B. Krüger, J. Tautges, A. Weber, and A. Zinke. Fast local and global similarity searches in large motion capture databases. In Proc. of the 2010 ACM SIGGRAPH/Eurographics Symp. on Computer Animation, pages 1--10, 2010.

Digital Library

[20]

Y. Lee, S. Kim, and J. Lee. Data-driven biped control. ACM Trans. on Graphics, 29(4):129, 2010.

Digital Library

[21]

S. Levine and J. Popović. Physically plausible simulation for character animation. In Proc. of the ACM SIGGRAPH/Eurographics Symp. on Computer Animation, 2012.

Digital Library

[22]

H. Liu, X. Wei, J. Chai, I. Ha, and T. Rhee. Realtime human motion control with a small number of inertial sensors. In Symp. on Interactive 3D Graphics and Games, pages 133--140. ACM, 2011.

Digital Library

[23]

L. Liu, K. Yin, M. van de Panne, T. Shao, and W. Xu. Sampling-based contact-rich motion control. ACM Transctions on Graphics, 29(4):Article 128, 2010.

Digital Library

[24]

E. C. Martinez-Villalpando and H. Herr. Agonist-antagonist active knee prosthesis: a preliminary study in level-ground walking. J. Rehabil. Res. Dev, 46(3):361--374, 2009.

[25]

L. A. Schwarz, D. Mateus, and N. Navab. Multiple-activity human body tracking in unconstrained environments. In Intl. Conf. on Articulated Motion and Deformable Objects, pages 192--202, 2010.

Digital Library

[26]

A. Shapiro, A. Feng, R. Wang, H. Li, M. Bolas, G. Medioni, and E. Suma. Rapid avatar capture and simulation using commodity depth sensors. Computer Animation and Virtual Worlds, 25(3-4):201--211, 2014.

Digital Library

[27]

R. Slyper and J. K. Hodgins. Action capture with accelerometers. In Proc. of the 2008 ACM SIGGRAPH/Eurographics Symp. on Computer Animation, pages 193--199, 2008.

Digital Library

[28]

J. Tautges, A. Zinke, B. Krüger, J. Baumann, A. Weber, T. Helten, M. Müller, H.-P. Seidel, and B. Eberhardt. Motion reconstruction using sparse accelerometer data. ACM Trans. on Graphics, 30(3):18, 2011.

Digital Library

[29]

D. Vlasic, R. Adelsberger, G. Vannucci, J. Barnwell, M. Gross, W. Matusik, and J. Popović. Practical motion capture in everyday surroundings. In ACM Trans. on Graphics, volume 26, page 35, 2007.

Digital Library

[30]

T. von Marcard, G. Pons-Moll, and B. Rosenhahn. Human pose estimation from video and imus. IEEE Trans. on Pattern Analysis and Machine Intelligence, 38(8):1533--1547, 2016.

Digital Library

[31]

M. Vondrak, L. Sigal, and O. C. Jenkins. Physical simulation for probabilistic motion tracking. In In Computer Vision and Pattern Recognition, 2008.

[32]

P. Zhang, K. Siu, J. Zhang, C. K. Liu, and J. Chai. Leveraging depth cameras and wearable pressure sensors for full-body kinematics and dynamics capture. ACM Trans. on Graphics, 33:221, 2014.

Digital Library

[33]

Y. Zheng and K. Yamane. Human motion tracking control with strict contact force constraints for floating-base humanoid robots. In 2013 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pages 34--41, 2013.

Cited By

Koniaris BSinclair DMitchell K(2024)DanceMark: An Open Telemetry Framework for Latency-Sensitive Real-Time Networked Immersive Experiences2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00091(462-463)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00091
Van Wouwe TLee SFalisse ADelp SLiu C(2024)DiffusionPoser: Real-Time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00243(2513-2523)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00243
Hu BGuo HYang YTao XHe J(2024)Lightweight Physics-Based Character for Generating Sensible Postures in Dynamic EnvironmentsIEEE Access10.1109/ACCESS.2024.341722012(89660-89678)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3417220
Show More Cited By

Index Terms

Real-time Physics-based Motion Capture with Sparse Sensors
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
    1. Animation
      1. Motion capture
      2. Motion processing

Index terms have been assigned to the content through auto-classification.

Recommendations

Data-driven inverse dynamics for human motion

Inverse dynamics is an important and challenging problem in human motion modeling, synthesis and simulation, as well as in robotics and biomechanics. Previous solutions to inverse dynamics are often noisy and ambiguous particularly when double stances ...
Real time automatic skeleton and motion estimation for character animation
International Workshop Motion in Games (MIG08)

Motion capture is prevalent in the pipeline of realistic articulated character animation. To define accurate joint positions and joint orientations for the movement of a hierarchical human-like character without using a pre-defined skeleton remains a ...
From motion capture to action capture: a review of imitation learning techniques and their application to VR-based character animation
VRST '06: Proceedings of the ACM symposium on Virtual reality software and technology

We present a novel method for virtual character animation that we call action capture. In this approach, virtual characters learn to imitate the actions of Virtual Reality (VR) users by tracking not only the users' movements but also their interactions ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

CVMP '16: Proceedings of the 13th European Conference on Visual Media Production (CVMP 2016)

December 2016

90 pages

ISBN:9781450347440

DOI:10.1145/2998559

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

In-Cooperation

The Foundry: The Foundry Visionmongers Ltd.
University of Bath: University of Bath

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 December 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

InnovateUK

Conference

CVMP 2016

CVMP 2016: 13th European Conference on Visual Media Production

December 12 - 13, 2016

London, United Kingdom

Acceptance Rates

Overall Acceptance Rate 40 of 67 submissions, 60%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

39
Total Citations
View Citations
521
Total Downloads

Downloads (Last 12 months)75
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Koniaris BSinclair DMitchell K(2024)DanceMark: An Open Telemetry Framework for Latency-Sensitive Real-Time Networked Immersive Experiences2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00091(462-463)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00091
Van Wouwe TLee SFalisse ADelp SLiu C(2024)DiffusionPoser: Real-Time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00243(2513-2523)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00243
Hu BGuo HYang YTao XHe J(2024)Lightweight Physics-Based Character for Generating Sensible Postures in Dynamic EnvironmentsIEEE Access10.1109/ACCESS.2024.341722012(89660-89678)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3417220
Xiao XWang JFeng PGong AZhang XZhang J(2024)Fast Human Motion reconstruction from sparse inertial measurement units considering the human shapeNature Communications10.1038/s41467-024-46662-515:1Online publication date: 18-Mar-2024
https://doi.org/10.1038/s41467-024-46662-5
Liang HZhang WLi WYu JXu L(2024)InterGen: Diffusion-Based Multi-human Motion Generation Under Complex InteractionsInternational Journal of Computer Vision10.1007/s11263-024-02042-6132:9(3463-3483)Online publication date: 26-Mar-2024
https://doi.org/10.1007/s11263-024-02042-6
Mitchell K(2023)Editorial: Games May Host the First Rightful AI CitizensGames: Research and Practice10.1145/36068341:2(1-7)Online publication date: 18-Jul-2023
https://dl.acm.org/doi/10.1145/3606834
Lee SStarke SYe YWon JWinkler A(2023)QuestEnvSim: Environment-Aware Simulated Motion Tracking from Sparse SensorsACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591504(1-9)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588432.3591504
Sharma ASalchow-Hömmen CMollyn VNittala AHedderich MKoelle MSeel TSteimle J(2023)SparseIMU: Computational Design of Sparse IMU Layouts for Sensing Fine-grained Finger MicrogesturesACM Transactions on Computer-Human Interaction10.1145/356989430:3(1-40)Online publication date: 10-Jun-2023
https://dl.acm.org/doi/10.1145/3569894
Chen XJiang XZhan LGuo SRuan QLuo GLiao MQin Y(2023)Full-body Human Motion Reconstruction with Sparse Joint Tracking Using Flexible SensorsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/356470020:2(1-19)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3564700
Ren YZhao CHe YCong PLiang HYu JXu LMa Y(2023)LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR SensorsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324708829:5(2337-2347)Online publication date: 22-Feb-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3247088
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents