research-article

Finger-Aware Shortcuts

Authors:

Daniel VogelAuthors Info & Claims

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Pages 4274 - 4285

https://doi.org/10.1145/2858036.2858355

Published: 07 May 2016 Publication History

Abstract

We evaluate and demonstrate finger, hand, and posture identification as keyboard shortcuts. By detecting the hand and finger used to press a key, and open or closed hand postures, a key press can have multiple command mappings. A formative study reveals performance and preference patterns when using different fingers and postures to press a key. The results are used to develop a computer vision algorithm to identify fingers and hands on a keyboard captured by a built-in lap top camera and reflector. This algorithm is built into a background service to enable system-wide finger-aware shortcut keys in any application. A controlled experiment uses the service to compare the performance of Finger-Aware Shortcuts with existing methods. The results show Finger-Aware Shortcuts are comparable with a common class of shortcuts using multiple modifier keys. Finally, application demonstrations illustrate different use cases and mappings for Finger-Aware Shortcuts and extend the idea to two-handed key presses, continuous parameter control, and menu selection.

Supplementary Material

suppl.mov (pn1584-file3.mp4)

Supplemental video

Download
101.52 MB

MP4 File (p4274-zheng.mp4)

Download
167.20 MB

References

[1]

Gilles Bailly, Thomas Pietrzak, Jonathan Deber, and Daniel J. Wigdor. 2013. Métamorphe: Augmenting Hotkey Usage with Actuated Keys. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 563-572.

Digital Library

[2]

Ruth Ben'ary. 1989. Touch Typing in Ten Lessons (revised edition ed.). Perigee Books, NY, NY, USA.

[3]

Florian Block, Hans Gellersen, and Nicolas Villar. 2010. Touch-Display Keyboards: Transforming Keyboards into Interactive Surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). 1145-1154.

Digital Library

[4]

Géry Casiez, Nicolas Roussel, and Daniel Vogel. 2012. 1 C Filter: A Simple Speed-based Low-pass Filter for Noisy Input in Interactive Systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). 2527-2530.

Digital Library

[5]

Paul H. Dietz, Benjamin Eidelson, Jonathan Westhues, and Steven Bathiche. 2009. A Practical Pressure Sensitive Computer Keyboard. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST '09). 55-58.

Digital Library

[6]

Wilbert O. Galitz. 2007. The Essential Guide to User Interface Design: An Introduction to GUI Design Principles and Techniques (3, illustrated ed.). John Wiley & Sons, NY, NY, USA.

Digital Library

[7]

Alix Goguey, Géry Casiez, Thomas Pietrzak, Daniel Vogel, and Nicolas Roussel. 2014. Adoiraccourcix: Multi-touch Command Selection Using Finger Identification. In Proceedings of the 26th Conference on L'Interaction Homme-Machine (IHM '14). 28-37.

Digital Library

[8]

Alix Goguey, Mathieu Nancel, Géry Casiez, and Daniel Vogel. 2016. The Performance and Preference of Different Fingers and Chords for Pointing, Dragging, and Object Transformation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '16). 12p.

Digital Library

[9]

Tovi Grossman, Pierre Dragicevic, and Ravin Balakrishnan. 2007. Strategies for Accelerating On-line Learning of Hotkeys. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). 1591-1600.

Digital Library

[10]

Robert J. K. Jacob, Audrey Girouard, Leanne M. Hirshfield, Michael S. Horn, Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. 2008. Reality-Based Interaction: A Framework for Post-WIMP Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). 201-210.

Digital Library

[11]

John Karat, James E. McDonald, and Matt Anderson. 1986. A Comparison of Menu Selection Techniques: Touch Panel, Mouse and Keyboard. International Journal of Man-Machine Studies 25, 1 (July 1986), 73-88.

Digital Library

[12]

David Kim, Shahram Izadi, Jakub Dostal, Christoph Rhemann, Cem Keskin, Christopher Zach, Jamie Shotton, Timothy Large, Steven Bathiche, Matthias Niessner, D. Alex Butler, Sean Fanello, and Vivek Pradeep. 2014. RetroDepth: 3D Silhouette Sensing for High-precision Input on and Above Physical Surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). 1377-1386.

Digital Library

[13]

Donald E. Knuth and Andrew Binstock. 2008. Interview with Donald Knuth. (25 Apr 2008). Retrieved Sep 20, 2015 from http://www.informit.com/articles/article.aspx?p=1193856.

[14]

Brian Krisler and Richard Alterman. 2008. Training Towards Mastery: Overcoming the Active User Paradox. In Proceedings of the 5th Nordic Conference on Human-computer Interaction: Building Bridges (NordiCHI '08). 239-248.

Digital Library

[15]

Gordon Kurtenbach and William Buxton. 1994. User Learning and Performance with Marking Menus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '94). 258-264.

Digital Library

[16]

David M. Lane, H. Albert Napier, S. Camille Peres, and Anikó Sándor. 2005. Hidden Costs of Graphical User Interfaces: Failure to Make the Transition from Menus and Icon Toolbars to Keyboard Shortcuts. International Journal of Human-Computer Interaction 18, 2 (May 2005), 133-144.

[17]

Jinha Lee, Alex Olwal, Hiroshi Ishii, and Cati Boulanger. 2013. SpaceTop: Integrating 2D and Spatial 3D Interactions in a See-through Desktop Environment. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 189-192.

Digital Library

[18]

Sylvain Malacria, Gilles Bailly, Joel Harrison, Andy Cockburn, and Carl Gutwin. 2013a. Promoting Hotkey Use through Rehearsal with ExposeHK. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 573-582.

Digital Library

[19]

Sylvain Malacria, Joey Scarr, Andy Cockburn, Carl Gutwin, and Tovi Grossman. 2013b. Skillometers: Reflective Widgets That Motivate and Help Users to Improve Performance. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). 321-330.

Digital Library

[20]

Shahzad Malik and Joe Laszlo. 2004. Visual Touchpad: A Two-handed Gestural Input Device. In Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI '04). 289-296.

Digital Library

[21]

Hugh McLoone, Ken Hinckley, and Edward Cutrell. 2003. Bimanual Interaction on the Microsoft Office Keyboard. In Proceedings of IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT '03). 49-56.

[22]

Craig S. Miller, Svetlin Denkov, and Richard C. Omanson. 2011. Categorization Costs for Hierarchical Keyboard Commands. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). 2765-2768.

Digital Library

[23]

Thomas A. Mysliwiec. 1994. FingerMouse: A Freehand Computer Pointing Interface. In Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (FG '95). 372-377.

[24]

Michael Ortega and Laurence Nigay. 2009. AirMouse: Finger Gesture for 2D and 3D Interaction. In Proceedings of the 12th IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT '09). 214-227.

Digital Library

[25]

S. Camille Peres, Michael D. Fleetwood, Minmin Yang, Franklin P. Tamborello, and Danielle Paige Smith. 2005. Pros, Cons, and Changing Behavior: An Application in the Use of the Keyboard to Issue Commands. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (HFES '05), Vol. 49. 637-641.

[26]

Thomas Pietrzak, Sylvain Malacria, and Gilles Bailly. 2014. CtrlMouse et TouchCtrl: Duplicating Mode Delimiters on the Mouse. In Proceedings of the 26th Conference on L'Interaction Homme-Machine (IHM '14). 38-47.

Digital Library

[27]

Jun Rekimoto, Takaaki Ishizawa, Carsten Schwesig, and Haruo Oba. 2003. PreSense: Interaction Techniques for Finger Sensing Input Devices. In Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology (UIST '03). 203-212.

Digital Library

[28]

Atsushi Sugiura and Yoshiyuki Koseki. 1998. A User Interface Using Fingerprint Recognition: Holding Commands and Data Objects on Fingers. In Proceedings of the 11th Annual ACM Symposium on User Interface Software and Technology (UIST '98). 71-79.

Digital Library

[29]

Susanne Tak, Piet Westendorp, and Iris van Rooij. 2013. Satisficing and the Use of Keyboard Shortcuts: Being Good Enough Is Enough? Interacting with Computers 25, 5 (Sept. 2013), 404-416.

[30]

Stuart Taylor, Cem Keskin, Otmar Hilliges, Shahram Izadi, and John Helmes. 2014. Type-Hover-Swipe in 96 Bytes: A Motion Sensing Mechanical Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). 1695-1704.

Digital Library

[31]

Jingtao Wang and John Canny. 2004. FingerSense: Augmenting Expressiveness to Physical Pushing Button by Fingertip Identification. In CHI '04 Extended Abstracts on Human Factors in Computing Systems (CHI EA '04). 1267-1270.

Digital Library

[32]

Malte Weiss, Julie Wagner, Yvonne Jansen, Roger Jennings, Ramsin Khoshabeh, James D. Hollan, and Jan Borchers. 2009. SLAP Widgets: Bridging the Gap Between Virtual and Physical Controls on Tabletops. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). 481-490.

Digital Library

[33]

Andrew D. Wilson. 2006. Robust Computer Vision-based Detection of Pinching for One and Two-handed Gesture Input. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (UIST '06). 255-258.

Digital Library

[34]

Andrew D. Wilson and Edward Cutrell. 2005. FlowMouse: A Computer Vision-Based Pointing and Gesture Input Device. In Proceedings of IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT '05). 565-578.

Digital Library

[35]

Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). 143-146.

Digital Library

[36]

Jacob O. Wobbrock, Andrew D. Wilson, and Yang Li. 2007. Gestures without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST '07). 159-168.

Digital Library

[37]

Erdem Yoruk, Ender Konukoglu, Bulent Sankur, and Jérôme Darbon. 2006. Shape-based Hand Recognition. Image Processing, IEEE Transactions on 15, 7 (July 2006), 1803-1815.

Digital Library

[38]

Haimo Zhang and Yang Li. 2014. GestKeyboard: Enabling Gesture-based Interaction on Ordinary Physical Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). 1675-1684.

Digital Library

Cited By

Huang ZGao CWang HDeng XLai YMa CQin SLiu YWang H(2024)SpeciFingersProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435598:1(1-28)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643559
Joshi NAbtahi PSodhi RBartov NRushing JCollins CVogel DGlueck M(2023)Transferable Microgestures Across Hand Posture and Location Constraints: Leveraging the Middle, Ring, and Pinky FingersProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606713(1-17)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606713
Luo YBarhudarian ESeyed T(2023)Project Mihr: Enabling Gestural Interactions on a Keyboard using a Graphene-based FabricExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585873(1-7)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585873
Show More Cited By

Index Terms

Finger-Aware Shortcuts
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

FingerArc and FingerChord: Supporting Novice to Expert Transitions with Guided Finger-Aware Shortcuts
UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology

Keyboard shortcuts can be more efficient than graphical input, but they are underused by most users. To alleviate this, we present "Guided Finger-Aware Shortcuts" to reduce the gulf between graphical input and shortcut activation. The interaction ...
Finger identification and hand gesture recognition techniques for natural user interface
APCHI '13: Proceedings of the 11th Asia Pacific Conference on Computer Human Interaction

The natural user interface using hand gesture have been popular field in Human-Computer-Interaction(HCI). Many research papers have been proposed in this field. They proposed vision-based, glove-based and depth-based approach for hand gesture ...
Finger controller: natural user interaction using finger gestures
HCI'13: Proceedings of the 15th international conference on Human-Computer Interaction: interaction modalities and techniques - Volume Part IV

We present a new natural user interaction technique using finger gesture recognition and finger identification with Kinect depth data. We developed a gesture version drawing, multi-touch and mapping on 3d space interactions. We implemented three type ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

May 2016

6108 pages

ISBN:9781450333627

DOI:10.1145/2858036

General Chairs:
Jofish Kaye
Yahoo
,
Allison Druin
University of Maryland / National Park Service
,
Program Chairs:
Cliff Lampe
University of Michigan
,
Dan Morris
Microsoft
,
Juan Pablo Hourcade
University of Iowa

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI'16

Sponsor:

SIGCHI

CHI'16: CHI Conference on Human Factors in Computing Systems

May 7 - 12, 2016

California, San Jose, USA

Acceptance Rates

CHI '16 Paper Acceptance Rate 565 of 2,435 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
1,063
Total Downloads

Downloads (Last 12 months)66
Downloads (Last 6 weeks)4

Reflects downloads up to 13 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Huang ZGao CWang HDeng XLai YMa CQin SLiu YWang H(2024)SpeciFingersProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435598:1(1-28)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643559
Joshi NAbtahi PSodhi RBartov NRushing JCollins CVogel DGlueck M(2023)Transferable Microgestures Across Hand Posture and Location Constraints: Leveraging the Middle, Ring, and Pinky FingersProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606713(1-17)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606713
Luo YBarhudarian ESeyed T(2023)Project Mihr: Enabling Gestural Interactions on a Keyboard using a Graphene-based FabricExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585873(1-7)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585873
Yeo HWu EKim DLee JKim HOh STakagi LWoo WKoike HQuigley A(2023)OmniSense: Exploring Novel Input Sensing and Interaction Techniques on Mobile Device with an Omni-Directional CameraProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580747(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580747
Bailly GKhamassi MGirard B(2023)Computational Model of the Transition from Novice to Expert Interaction TechniquesACM Transactions on Computer-Human Interaction10.1145/350555730:5(1-33)Online publication date: 23-Sep-2023
https://dl.acm.org/doi/10.1145/3505557
Grubert J(2023)Mixed Reality Interaction TechniquesSpringer Handbook of Augmented Reality10.1007/978-3-030-67822-7_5(109-129)Online publication date: 1-Jan-2023
https://doi.org/10.1007/978-3-030-67822-7_5
Kim JOakley I(2022)SonarID: Using Sonar to Identify Fingers on a SmartwatchProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501935(1-10)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501935
McGill MBrewster SDe Sa Medeiros DBovet SGutierrez MKehoe A(2022)Creating and Augmenting Keyboards for Extended Reality with the Keyboard Augmentation ToolkitACM Transactions on Computer-Human Interaction10.1145/349049529:2(1-39)Online publication date: 16-Jan-2022
https://dl.acm.org/doi/10.1145/3490495
Magrofuoco NRoselli PVanderdonckt J(2021)Two-dimensional Stroke Gesture RecognitionACM Computing Surveys10.1145/346540054:7(1-36)Online publication date: 18-Jul-2021
https://dl.acm.org/doi/10.1145/3465400
Keddisseh ESerrano MDubois E(2021)SmartCom: Exploiter un Smartphone pour Interagir avec les Barres d'OutilsProceedings of the 32nd Conference on l'Interaction Homme-Machine10.1145/3450522.3451244(1-10)Online publication date: 13-Apr-2021
https://dl.acm.org/doi/10.1145/3450522.3451244
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents