Abstract
Fine-grained action recognition is a challengeable task due to its background independence and complex semantics over time and space. Multidimensional attention is essential for this task to capture discriminative spatial details, temporal and channel features. However, multidimensional attention has challenge in keeping the balance between adaptive feature perception and computational overhead. To address this issue, this paper proposes a Lightweight Multidimensional Self-Attention Network (LMSA-Net) which can adaptively capture the discriminative features over multiple dimensions in an efficient manner. It is worth remarking that the contextual relationship between time and channel is established in temporal stream, which is complementary for spatial attention in spatial stream. Compared with the RGB based models, LMSA-Net achieves state-of-the-art performance in two fine-grained action recognition datasets, i.e. FSD-10 and Diving48-V2. In addition, it can be found that the streams of LMSA-Net are end-to-end trainable to reduce the overhead of computation and storage, and the recognition accuracy can reach the level of two-stage two-stream models.
This study was funded by National Natural Science Foundation of Peoples Republic of China (61672130, 61972064), the Fundamental Research Fund for Dalian Youth Star of Science and Technology (No. 2019RQ035), LiaoNing Revitalization Talents Program (XLYC1806006) and CCF-Baidu Open Fund (No. 2021PP15002000).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Shao, D., Zhao, Y., Dai, B., Lin, D.: Finegym: a hierarchical video dataset for fine-grained action understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2616–2625 (2020)
Soomro, K., Zamir, A.R., Shah, M.: Ucf101: a dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)
Wang, Y., Wang, S., Tang, J., O’Hare, N., Chang, Y., Li, B.: Hierarchical attention network for action recognition in videos. arXiv preprint arXiv:1607.06416 (2016)
Zhang, C., Zou, Y., Chen, G., Gan, L.: Pan: towards fast action recognition via learning persistence of appearance. arXiv preprint arXiv:2008.03462 (2020)
Sharma, S., Kiros, R., Salakhutdinov, R.: Action recognition using visual attention. arXiv preprint arXiv:1511.04119 (2015)
Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. arXiv preprint arXiv:1406.2199 (2014)
Fan, L., Huang, W., Gan, C., Ermon, S., Gong, B., Huang, J.: End-to-end learning of motion representation for video understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6016–6025 (2018)
Huang, G., Bors, A.G.: Video classification with finecoarse networks. arXiv preprint arXiv:2103.15584 (2021)
Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)
Perez-Rua, J.-M., Martinez, B., Zhu, X., Toisoul, A., Escorcia, V., Xiang, T.: Knowing what, where and when to look: efficient video action modeling with attention. arXiv preprint arXiv:2004.01278 (2020)
Shi, L., Zhang, Y., Cheng, J., Lu, H.: Skeleton-based action recognition with multi-stream adaptive graph convolutional networks. IEEE Trans. Image Process. 29, 9532–9545 (2020)
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., Liu, W.: Ccnet: criss-cross attention for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 603–612 (2019)
Liu, S., et al.: FSD-10: a fine-grained classification dataset for figure skating. Neurocomputing 413, 360–367 (2020)
Li, Y., Li, Y., Vasconcelos, N.: Resound: towards action recognition without representation bias. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 513–528 (2018)
Zhang, H., Xin, M., Wang, S., Yang, Y., Zhang, L., Wang, H.: End-to-end temporal attention extraction and human action recognition. Mach. Vis. Appl. 29(7), 1127–1142 (2018). https://doi.org/10.1007/s00138-018-0956-5
Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need for video understanding? arXiv preprint arXiv:2102.05095 (2021)
Cao, Z., Simon, T., Wei, S.-E., Sheikh, Y.: Realtime multi-person 2d pose estimation using part affinity fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7291–7299 (2017)
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Lin, J., Gan, C., Han, S.: TSM: temporal shift module for efficient video understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7083–7093 (2019)
Hu, L., Feng, L., Liu, S.: Hfnet: a novel model for human focused sports action recognition. In: Proceedings of the 3rd International Workshop on Multimedia Content Analysis in Sports, pp. 35–43 (2020)
Carreira, J., Zisserman, J.: Quo vadis, action recognition? A new model and the kinetics dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308 (2017)
Yu, B., Yin, H., Zhu, Z.: Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting. arXiv preprint arXiv:1709.04875 (2017)
Woo, S., Park, J., Lee, J.-Y., Kweon, I.S. : Cbam: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)
Feichtenhofer, C., Fan, H., Malik, H., He, K.: Slowfast networks for video recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6202–6211 (2019)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, H., Liu, S., Feng, L., Hu, L., Li, X., Fu, H. (2021). A Lightweight Multidimensional Self-attention Network for Fine-Grained Action Recognition. In: Mantoro, T., Lee, M., Ayu, M.A., Wong, K.W., Hidayanto, A.N. (eds) Neural Information Processing. ICONIP 2021. Lecture Notes in Computer Science(), vol 13109. Springer, Cham. https://doi.org/10.1007/978-3-030-92270-2_49
Download citation
DOI: https://doi.org/10.1007/978-3-030-92270-2_49
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-92269-6
Online ISBN: 978-3-030-92270-2
eBook Packages: Computer ScienceComputer Science (R0)