Abstract
Video-based performance driven facial animation is appealing as it offers the lowest cost, a simplified setup, and the potential use of legacy sources and uncontrolled videos. It is also difficult as it is ill-posed due to the loss of depth. This chapter introduces techniques in video-based facial reconstruction in three levels. Given the input video, the first level is to reconstruct 3D head poses and large-scale facial deformation at each frame. Representations of the facial deformation as well as the related 2D feature detection/tracking and 3D shape parameters optimization methods are introduced. Next, we discuss methods on recovering the fine-scale surface details such as emerging and disappearing wrinkles and folds. Finally, we briefly introduce the advanced applications based on the reconstructed facial performance, such as video editing and facial component enhancement.
Similar content being viewed by others
References
Baltrušaitis T, Robinson P, Morency LP (2012) 3D constrained local model for rigid and non-rigid facial tracking. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2610–2617
Basri R, Jacobs DW (2003) Lambertian reflectance and linear subspaces. IEEE Trans Pattern Anal Mach Intell 25(2):218–233
Beeler T, Bickel B, Beardsley P, Sumner B, Gross M (2010) High-quality single-shot capture of facial geometry. ACM Trans Graph 29(4):40:1–40:9
Beeler T, Hahn F, Bradley D, Bickel B, Beardsley P, Gotsman C, Sumner RW, Gross M (2011) High-quality passive facial performance capture using anchor frames. ACM Trans Graph 30(4):75:1–75:10
Bickel B, Botsch M, Angst R, Matusik W, Otaduy M, Pfister H, Gross M (2007) Multi-scale capture of facial geometry and motion. ACM Trans Graph 26(3):33:1–33:10
Blanz V, Vetter T (1999) A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing, New York, pp 187–194
Bouaziz S, Wang Y, Pauly M (2013) Online modeling for real-time facial animation. ACM Trans Graph 32(4):40:1–40:10. doi:10.1145/2461912.2461976
Bradley D, Heidrich W, Popa T, Sheffer A (2010) High resolution passive facial performance capture. ACM Trans Graph 29(4):41:1–41:10
Cao X, Wei Y, Wen F, Sun J (2012) Face alignment by explicit shape regression. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2887–2894
Cao C, Weng Y, Lin S, Zhou K (2013) 3D shape regression for real-time facial animation. ACM Trans Graph 32(4):41:1–41:10. doi:10.1145/2461912.2462012
Cao C, Hou Q, Zhou K (2014a) Displaced dynamic expression regression for real-time facial tracking and animation. ACM Transactions on Graphics (TOG) 33(4):43
Cao C, Weng Y, Zhou S, Tong Y, Zhou K (2014b) Facewarehouse: a 3D facial expression database for visual computing. IEEE Trans Vis Comput Graph 20(3):413–425
Cao C, Bradley D, Zhou K, Beeler T (2015) Real-time high-fidelity facial performance capture. ACM Transactions on Graphics (TOG) 34(4):46
Chai J, Xiao J, Hodgins J (2003) Vision-based control of 3D facial animation. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on computer animation, pp 193–206
Chen YL, Wu Ht, Shi F, Tong X, Chai J (2013) Accurate and robust 3D facial capture using a single rgbd camera. In: IEEE international conference on computer vision (ICCV), pp 3615–3622
Garrido P, Valgaerts L, Wu C, Theobalt C (2013) Reconstructing detailed dynamic face geometry from monocular video. ACM Trans Graph 32(6):158
Garrido P, Zollhöfer M, Casas D, Valgaerts L, Varanasi K, Pérez P, Theobalt C (2016) Reconstruction of personalized 3D face rigs from monocular video. ACM Trans Graph (TOG) 35(3):28
Horn BK, Brooks MJ (1989) Shape from shading. MIT Press, Cambridge, MA
Hsieh PL, Ma C, Yu J, Li H (2015) Unconstrained real-time facial performance capture. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1675–1683
Huang H, Chai J, Tong X, Wu HT (2011) Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition. ACM Trans Graph 30(4):74:1–74:10
Ichim AE, Bouaziz S, Pauly M (2015) Dynamic 3D avatar creation from handheld video input. ACM Trans Graph (TOG) 34(4):45
Kemelmacher-Shlizerman I, Basri R (2011) 3D face reconstruction from a single image using a single reference face shape. IEEE Trans Pattern Anal Mach Intell 33(2):394–405
Li H, Yu J, Ye Y, Bregler C (2013) Real-time facial animation with on-the-fly correctives. ACM Trans Graph 32(4):42:1–42:10. doi:10.1145/2461912.2462019
Li H, Trutoiu L, Olszewski K, Wei L, Trutna T, Hsieh PL, Nicholls A, Ma C (2015) Facial performance sensing head-mounted display. ACM Trans Graph (TOG) 34(4):47
Liu Y, Xu F, Chai J, Tong X, Wang L, Huo Q (2015) Videoaudio driven real-time facial animation. ACM Trans Graph 34(6):182:1–182:10. doi:10.1145/2816795.2818122
Ma WC, Jones A, Chiang JY, Hawkins T, Frederiksen S, Peers P, Vukovic M, Ouhyoung M, Debevec P (2008) Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans Graph 27(5):121:1–121:10
Matthews I, Baker S (2004) Active appearance models revisited. Int J Comp Vision 60(2):135–164
Ren S, Cao X, Wei Y, Sun J (2014) Face alignment at 3000 fps via regressing local binary features. In: IEEE conference on computer vision and pattern recognition (CVPR), IEEE, pp 1685–1692
Saragih JM, Lucey S, Cohn JF (2011) Real-time avatar animation from a single image. In: IEEE international conference on automatic face & gesture recognition and workshops (FG 2011), IEEE, pp 117–124
Shi F, Wu HT, Tong X, Chai J (2014) Automatic acquisition of high-fidelity facial performances using monocular videos. ACM Transactions on Graphics (TOG) 33(6):222
Sumner RW, Popović J (2004) Deformation transfer for triangle meshes. ACM Transactions on Graphics (TOG) 23(3):399–405
Suwajanakorn S, Kemelmacher-Shlizerman I, Seitz SM (2014) Total moving face reconstruction. In: European conference on computer vision, Springer, pp 796–812
Suwajanakorn S, Seitz SM, Kemelmacher-Shlizerman I (2015) What makes tom hanks look like tom hanks. In: Proceedings of the IEEE international conference on computer vision, pp 3952–3960
Thies J, Zollhöfer M, Stamminger M, Theobalt C, Nießner M (2016) Face2face: real-time face capture and reenactment of RGB videos. In: Proceedings of computer vision and pattern recognition (CVPR), IEEE 1
Valgaerts L, Wu C, Bruhn A, Seidel HP, Theobalt C (2012) Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans Graph 31(6):187:1–187:11. doi:10.1145/2366145.2366206
Vlasic D, Brand M, Pfister H, Popović J (2005) Face transfer with multilinear models. ACM Trans Graph (TOG) 24:426–433
Wang C, Shi F, Xia S, Chai J (2016) Real-time 3D eye gaze animation using a single RGB camera. ACM Trans Graph (TOG) 35:1
Weise T, Li H, Van Gool L, Pauly M (2009) Face/off: live facial puppetry. In: Symposium on computer animation, pp 7–16. doi:10.1145/1599470.1599472
Weise T, Bouaziz S, Li H, Pauly M (2011) Real-time performance-based facial animation. ACM Trans Graph 30(4):77:1–77:10
Xiong X, De la Torre F (2013) Supervised descent method and its applications to face alignment. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 532–539
Zhang L, Snavely N, Curless B, Seitz S (2004) Spacetime faces: high resolution capture for modeling and animation. ACM Transactions on Graphics 23(3):548–558
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing AG
About this entry
Cite this entry
Shi, F. (2016). Video-Based Performance Driven Facial Animation. In: Müller, B., et al. Handbook of Human Motion. Springer, Cham. https://doi.org/10.1007/978-3-319-30808-1_189-1
Download citation
DOI: https://doi.org/10.1007/978-3-319-30808-1_189-1
Received:
Accepted:
Published:
Publisher Name: Springer, Cham
Online ISBN: 978-3-319-30808-1
eBook Packages: Springer Reference EngineeringReference Module Computer Science and Engineering
Publish with us
Chapter history
-
Latest
Video-Based Performance Driven Facial Animation- Published:
- 15 February 2017
DOI: https://doi.org/10.1007/978-3-319-30808-1_189-2
-
Original
Video-Based Performance Driven Facial Animation- Published:
- 10 November 2016
DOI: https://doi.org/10.1007/978-3-319-30808-1_189-1