research-article

KnuckleTouch: Enabling Knuckle Gestures on Capacitive Touchscreens using Deep Learning

Authors:

Robin Schweigert,

Simon Hagenmayer,

Maximilian Weiß,

Andreas BullingAuthors Info & Claims

MuC '19: Proceedings of Mensch und Computer 2019

Pages 387 - 397

https://doi.org/10.1145/3340764.3340767

Published: 08 September 2019 Publication History

Abstract

While mobile devices have become essential for social communication and have paved the way for work on the go, their interactive capabilities are still limited to simple touch input. A promising enhancement for touch interaction is knuckle input but recognizing knuckle gestures robustly and accurately remains challenging. We present a method to differentiate between 17 finger and knuckle gestures based on a long short-term memory (LSTM) machine learning model. Furthermore, we introduce an open source approach that is ready-to-deploy on commodity touch-based devices. The model was trained on a new dataset that we collected in a mobile interaction study with 18 participants. We show that our method can achieve an accuracy of 86.8% on recognizing one of the 17 gestures and an accuracy of 94.6% to differentiate between finger and knuckle. In our evaluation study, we validate our models and found that the LSTM gesture recognizing archived an accuracy of 88.6%. We show that KnuckleTouch can be used to improve the input expressiveness and to provide shortcuts to frequently used functions.

References

[1]

Lisa Anthony and Jacob O. Wobbrock. 2010. A Lightweight Multistroke Recognizer for User Interface Prototypes. In Proceedings of Graphics Interface 2010 (GI '10). Canadian Information Processing Society, Toronto, Ont., Canada, Canada, 245--252. http://dl.acm.org/citation.cfm?id=1839214.1839258

Digital Library

[2]

Ann Blandford, Dominic Furniss, and Stephann Makri. 2016. Qualitative Hci Research: Going Behind the Scenes. Morgan & Claypool Publishers. 1--115 pages.

[3]

John Brooke. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4--7.

[4]

Daniel Buschek and Florian Alt. 2017. ProbUI: Generalising Touch Target Representations to Enable Declarative Gesture Definition for Probabilistic GUIs. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 4640--4653.

Digital Library

[5]

Xiang 'Anthony' Chen, Tovi Grossman, Daniel J. Wigdor, and George Fitzmaurice. 2014. Duet: Exploring Joint Interactions on a Smart Phone and a Smart Watch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 159--168.

Digital Library

[6]

Paulin Coulibaly, François Anctil, and Bernard Bobée. 2000. Daily reservoir inflow forecasting using artificial neural networks with stopped training approach. Journal of Hydrology 230, 3 (2000), 244--257.

[7]

Nicholas Gillian and Joseph A. Paradiso. 2014. The Gesture Recognition Toolkit. J. Mach. Learn. Res. 15, 1 (Jan. 2014), 3483--3487. http://dl.acm.org/citation.cfm?id=2627435.2697076

Digital Library

[8]

Aakar Gupta, Muhammed Anwar, and Ravin Balakrishnan. 2016. Porous Interfaces for Small Screen Multitasking Using Finger Identification. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 145--156.

Digital Library

[9]

Gunnar Harboe and Elaine M. Huang. 2015. Real-World Affinity Diagramming Practices: Bridging the Paper-Digital Gap. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 95--104.

Digital Library

[10]

Chris Harrison, Julia Schwarz, and Scott E. Hudson. 2011. TapSense: Enhancing Finger Interaction on Touch Surfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 627--636.

Digital Library

[11]

Sandra G. Hart. 2006. NASA-Task Load Index (NASA-TLX); 20 years later. In Proceedings of the Human Factors and Ergonomic Society annual meeting, Vol. 50. SAGE Publications, SAGE Publications, Los Angeles, CA, USA, 904--908.

[12]

Christian Holz and Patrick Baudisch. 2010. The Generalized Perceived Input Point Model and How to Double Touch Accuracy by Extracting Fingerprints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 581--590.

Digital Library

[13]

Christian Holz, Senaka Buthpitiya, and Marius Knaust. 2015. Bodyprint: Biometric User Identification on Mobile Devices Using the Capacitive Touchscreen to Scan Body Parts. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 3011--3014.

Digital Library

[14]

Abinaya Kumar, Aishwarya Radjesh, Sven Mayer, and Huy Viet Le. 2019. Improving the Input Accuracy of Touchscreens using Deep Learning. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (2019-05-04) (CHI'19 EA). ACM, New York, NY, USA.

Digital Library

[15]

Huy Viet Le, Thomas Kosch, Patrick Bader, Sven Mayer, and Niels Henze. 2018. PalmTouch: Using the Palm As an Additional Input Modality on Commodity Smartphones. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 360, 13 pages.

Digital Library

[16]

Huy Viet Le, Sven Mayer, Patrick Bader, and Niels Henze. 2017. A Smartphone Prototype for Touch Interaction on the Whole Device Surface. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17). ACM, New York, NY, USA, Article 100, 8 pages.

Digital Library

[17]

Huy Viet Le, Sven Mayer, and Niels Henze. 2018. InfiniTouch: Finger-Aware Interaction on Fully Touch Sensitive Smartphones. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). ACM, New York, NY, USA, 779--792.

Digital Library

[18]

Huy Viet Le, Sven Mayer, and Niels Henze. 2019. Investigating the Feasibility of Finger Identification on Capacitive Touchscreens Using Deep Learning. In Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI '19). ACM, New York, NY, USA, 637--649.

Digital Library

[19]

Yang Li. 2010. Protractor: A Fast and Accurate Gesture Recognizer. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 2169--2172.

Digital Library

[20]

Allan Christian Long, Jr., James A. Landay, and Lawrence A. Rowe. 1999. Implications for a Gesture Design Tool. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '99). ACM, New York, NY, USA, 40--47.

Digital Library

[21]

Pedro Lopes, Ricardo Jota, and Joaquim A. Jorge. 2011. Augmenting Touch Interaction Through Acoustic Sensing. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '11). ACM, New York, NY, USA, 53--56.

Digital Library

[22]

Sven Mayer, Huy Viet Le, and Niels Henze. 2017. Estimating the Finger Orientation on Capacitive Touchscreens Using Convolutional Neural Networks. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS '17). ACM, New York, NY, USA, 220--229.

Digital Library

[23]

Dean Rubine. 1991. Specifying Gestures by Example. In Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '91). ACM, New York, NY, USA, 329--337.

Digital Library

[24]

Dean Harris Rubine. 1992. The Automatic Recognition of Gestures. Ph.D. Dissertation. Pittsburgh, PA, USA. UMI Order No. GAX92-16029.

[25]

Gineke A. ten Holt, Marcel J. T. Reinders, and Emile A. Hendriks. 2007. Multi-dimensional dynamic time warping for gesture recognition. In Thirteenth Annual Conference of the Advanced School for Computing and Imaging, Vol. 300.

[26]

Paulo Trigueiros, António Fernando Ribeiro, and L. P. Reis. 2012. A comparison of machine learning algorithms applied to hand gesture recognition. In 7th Iberian Conference on Information Systems and Technologies (CISTI'12). 1--6.

[27]

Radu-Daniel Vatavu, Lisa Anthony, and Jacob O. Wobbrock. 2012. Gestures As Point Clouds: A $P Recognizer for User Interface Prototypes. In Proceedings of the 14th ACM International Conference on Multimodal Interaction (ICMI '12). ACM, New York, NY, USA, 273--280.

Digital Library

[28]

Jacob O. Wobbrock, Andrew D. Wilson, and Yang Li. 2007. Gestures Without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST '07). ACM, New York, NY, USA, 159--168.

Digital Library

[29]

Robert Xiao, Julia Schwarz, and Chris Harrison. 2015. Estimating 3D Finger Angle on Commodity Touchscreens. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces (ITS '15). ACM, New York, NY, USA, 47--50.

Digital Library

[30]

Shumin Zhai and Per-Ola Kristensson. 2003. Shorthand Writing on Stylus Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '03). ACM, New York, NY, USA, 97--104.

Digital Library

Cited By

Li CYu JHe KFeng JZhou J(2024)SwivelTouch: Boosting Touchscreen Input with 3D Finger Rotation GestureProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595848:2(1-30)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659584
Huang ZGao CWang HDeng XLai YMa CQin SLiu YWang H(2024)SpeciFingersProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435598:1(1-28)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643559
He KLi CDuan YFeng JZhou J(2024)TrackPoseProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314597:4(1-22)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631459
Show More Cited By

Index Terms

KnuckleTouch: Enabling Knuckle Gestures on Capacitive Touchscreens using Deep Learning
1. Hardware
  1. Communication hardware, interfaces and storage
    1. Tactile and hand-based interfaces
      1. Touch screens
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI
    2. Interaction devices
      1. Touch screens

Recommendations

Palm Vein Recognition Based on Convolutional Neural Network

Convolutional neural networks (CNNs) were popular in ImageNet large scale visual recognition competition (ILSVRC 2012) because of their identification ability and computational efficiency. This paper proposes a palm vein recognition method based on CNN. ...
In-Datacenter Performance Analysis of a Tensor Processing Unit
ISCA'17

Many architects believe that major improvements in cost-energy-performance must now come from domain-specific hardware. This paper evaluates a custom ASIC---called a Tensor Processing Unit (TPU) --- deployed in datacenters since 2015 that accelerates ...
Forecasting multistep daily stock prices for long-term investment decisions: A study of deep learning models on global indices
Abstract
Deep machine learning algorithms play an important role in facilitating the development of predictive models for the stock market. However, most studies focus on predicting next-day stock prices or movements, limiting the usability of the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

MuC '19: Proceedings of Mensch und Computer 2019

September 2019

863 pages

ISBN:9781450371988

DOI:10.1145/3340764

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 September 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Conference

MuC'19

MuC'19: Mensch-und-Computer

September 8 - 11, 2019

Hamburg, Germany

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
215
Total Downloads

Downloads (Last 12 months)30
Downloads (Last 6 weeks)2

Reflects downloads up to 13 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li CYu JHe KFeng JZhou J(2024)SwivelTouch: Boosting Touchscreen Input with 3D Finger Rotation GestureProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595848:2(1-30)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659584
Huang ZGao CWang HDeng XLai YMa CQin SLiu YWang H(2024)SpeciFingersProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435598:1(1-28)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643559
He KLi CDuan YFeng JZhou J(2024)TrackPoseProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314597:4(1-22)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631459
Shen ZLi SZhao XZou J(2024)CT-Auth: Capacitive Touchscreen-Based Continuous Authentication on SmartphonesIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.327787936:1(90-106)Online publication date: Jan-2024
https://doi.org/10.1109/TKDE.2023.3277879
Shende STembhurne JAnsari N(2024)Deep learning based authentication schemes for smart devices in different modalities: progress, challenges, performance, datasets and future directionsMultimedia Tools and Applications10.1007/s11042-024-18350-583:28(71451-71493)Online publication date: 8-Feb-2024
https://doi.org/10.1007/s11042-024-18350-5
Villarreal-Narvaez SSluÿters AVanderdonckt JVatavu R(2023)Brave New GES World: A Systematic Literature Review of Gestures and Referents in Gesture Elicitation StudiesACM Computing Surveys10.1145/363645856:5(1-55)Online publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1145/3636458
Rakhmetulla GRen YArif A(2023)GeShort: One-Handed Mobile Text Editing and Formatting with Gestural Shortcuts and a Floating ClipboardProceedings of the ACM on Human-Computer Interaction10.1145/36042597:MHCI(1-23)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604259
Suzuki TManabe H(2023)Estimation of Brush Type Passive Stylus Angles Using Capacitive ImageProceedings of the Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3569009.3573112(1-6)Online publication date: 26-Feb-2023
https://dl.acm.org/doi/10.1145/3569009.3573112
Wolf F(2023)Software-basierte FunktionenSoftware im Automobil10.1007/978-3-662-67156-6_2(51-140)Online publication date: 13-Jun-2023
https://doi.org/10.1007/978-3-662-67156-6_2
Vatavu R(2023)Gesture-Based InteractionHandbook of Human Computer Interaction10.1007/978-3-319-27648-9_20-1(1-47)Online publication date: 9-Feb-2023
https://doi.org/10.1007/978-3-319-27648-9_20-1
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents