research-article

Public Access

SignFi: Sign Language Recognition Using WiFi

Authors:

Shuangquan Wang,

Woosub JungAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 2, Issue 1

Article No.: 23, Pages 1 - 21

https://doi.org/10.1145/3191755

Published: 26 March 2018 Publication History

Abstract

We propose SignFi to recognize sign language gestures using WiFi. SignFi uses Channel State Information (CSI) measured by WiFi packets as the input and a Convolutional Neural Network (CNN) as the classification algorithm. Existing WiFi-based sign gesture recognition technologies are tested on no more than 25 gestures that only involve hand and/or finger gestures. SignFi is able to recognize 276 sign gestures, which involve the head, arm, hand, and finger gestures, with high accuracy. SignFi collects CSI measurements to capture wireless signal characteristics of sign gestures. Raw CSI measurements are pre-processed to remove noises and recover CSI changes over sub-carriers and sampling time. Pre-processed CSI measurements are fed to a 9-layer CNN for sign gesture classification. We collect CSI traces and evaluate SignFi in the lab and home environments. There are 8,280 gesture instances, 5,520 from the lab and 2,760 from the home, for 276 sign gestures in total. For 5-fold cross validation using CSI traces of one user, the average recognition accuracy of SignFi is 98.01%, 98.91%, and 94.81% for the lab, home, and lab+home environment, respectively. We also run tests using CSI traces from 5 different users in the lab environment. The average recognition accuracy of SignFi is 86.66% for 7,500 instances of 150 sign gestures performed by 5 different users.

Supplemental Material

ZIP File - ma

Supplemental movie, appendix, image and software files for, SignFi: Sign Language Recognition Using WiFi

Download
1.28 MB

References

[1]

H. Abdelnasser, M. Youssef, and K. A. Harras. 2015. WiGest: A ubiquitous WiFi-based gesture recognition system. In 2015 IEEE Conference on Computer Communications (INFOCOM). 1472--1480.

[2]

Naomi K. Caselli, Zed Sevcikova Sehyr, Ariel M. Cohen-Goldberg, and Karen Emmorey. 2017. ASL-LEX: A lexical database of American Sign Language. Behavior Research Methods 49, 2 (April 2017), 784--801.

[3]

Liwei Chan, Rong-Hao Liang, Ming-Chang Tsai, Kai-Yin Cheng, Chao-Huai Su, Mike Y. Chen, Wen-Huang Cheng, and Bing-Yu Chen. 2013. FingerPad: Private and Subtle Interaction Using Fingertips. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). 255--260.

Digital Library

[4]

Ke-Yu Chen, Kent Lyons, Sean White, and Shwetak Patel. 2013. uTrack: 3D Input Using Two Magnetic Sensors. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). 237--244.

Digital Library

[5]

Ke-Yu Chen, Shwetak N. Patel, and Sean Keller. 2016. Finexus: Tracking Precise Motions of Multiple Fingertips Using Magnetic Sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). 1504--1514.

Digital Library

[6]

Ching-Hua Chuan, Eric Regina, and Caroline Guardino. 2014. American Sign Language Recognition Using Leap Motion Sensor. In Proceedings of the 2014 13th International Conference on Machine Learning and Applications (ICMLA '14). 541--544.

Digital Library

[7]

Biyi Fang, Jillian Co, and Mi Zhang. 2017. DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation. In Proceedings of the 15th ACM Conference on Embedded Networked Sensor Systems (SenSys '17).

Digital Library

[8]

Makiko Funasaka, Yu Ishikawa, Masami Takata, and Kazuki Joe. 2015. Sign Language Recognition using Leap Motion Controller. In Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA).

[9]

David Goldberg, Dennis Looney, and Natalia Lusin. 2015. Enrollments in Languages Other Than English in United States Institutions of Higher Education, Fall 2013. In Modern Language Association.

[10]

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org.

Digital Library

[11]

Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. SoundWave: Using the Doppler Effect to Sense Gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). 1911--1914.

Digital Library

[12]

Daniel Halperin, Wenjun Hu, Anmol Sheth, and David Wetherall. 2011. Tool Release: Gathering 802.11n Traces with Channel State Information. SIGCOMM Comput. Commun. Rev. 41, 1 (Jan. 2011), 53--53.

Digital Library

[13]

Jie Huang, Wengang Zhou, Houqiang Li, and Weiping Li. 2015. Sign Language Recognition using 3D Convolutional Neural Networks. In 2015 IEEE International Conference on Multimedia andExpo (ICME). 1--6.

[14]

Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37 (ICML'15). JMLR.org, 448--456. http://dl.acm.org/citation.cfm?id=3045118.3045167

Digital Library

[15]

Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing Gesture Recognition to All Devices. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation (NSDI'14). 303--316. http://dl.acm.org/citation.cfm?id=2616448.2616477

Digital Library

[16]

Swarun Kumar, Diego Cifuentes, Shyamnath Gollakota, and Dina Katabi. 2013. Bringing Cross-layer MIMO to Today's Wireless LANs. In Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM (SIGCOMM '13). 387--398.

Digital Library

[17]

H.L. Lane, R. Hoffmeister, and B.J. Bahan. 1996. A Journey Into the Deaf-world. DawnSignPress.

[18]

Hong Li, Wei Yang, Jianxin Wang, Yang Xu, and Liusheng Huang. 2016. WiFinger: Talk to Your Smart Devices with Finger-grained Gesture. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). 250--261.

Digital Library

[19]

Jaime Lien, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous Gesture Sensing with Millimeter Wave Radar. ACM Trans. Graph. 35, 4, Article 142 (July 2016), 19 pages.

Digital Library

[20]

Rajesh B. Mapari and Govind Kharat. 2016. American Static Signs Recognition Using Leap Motion Sensor. In Proceedings of the Second International Conference on Information and Communication Technology for Competitive Strategies (ICTCS '16). Article 67, 5 pages.

Digital Library

[21]

Pedro Melgarejo, Xinyu Zhang, Parameswaran Ramanathan, and David Chu. 2014. Leveraging Directional Antenna Capabilities for Fine-grained Gesture Recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '14). 541--551.

Digital Library

[22]

P. Molchanov, S. Gupta, K. Kim, and K. Pulli. 2015. Multi-sensor system for driver's hand-gesture recognition. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Vol. 1. 1--8.

[23]

MotionSavvy. 2017. MotionSavvy UNI. (2017). Retrieved July 23, 2017 from http://www.motionsavvy.com/uni.html.

[24]

D. Naglot and M. Kulkarni. 2016. Real time sign language recognition using the leap motion controller. In 2016 International Conference on Inventive Computation Technologies (ICICT), Vol. 3. 1--5.

[25]

Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. FingerIO: Using Active Sonar for Fine-Grained Finger Tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). 1515--1525.

Digital Library

[26]

University of Washington. 2016. SignAloud Demo. (2016). Retrieved July 23, 2017 from https://youtu.be/4uY-MyoRq4c.

[27]

Taiwoo Park, Jinwon Lee, Inseok Hwang, Chungkuk Yoo, Lama Nachman, and Junehwa Song. 2011. E-Gesture: A Collaborative Architecture for Energy-efficient Gesture Recognition with Hand-worn Sensor and Mobile Devices. In Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems (SenSys '11). 260--273.

Digital Library

[28]

Lionel Pigou, Sander Dieleman, Pieter-Jan Kindermans, and Benjamin Schrauwen. 2015. Sign Language Recognition Using Convolutional Neural Networks. Springer International Publishing, Cham, 572--578.

[29]

Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home Gesture Recognition Using Wireless Signals. In Proceedings of the 19th Annual International Conference on Mobile Computing and Networking (MobiCom '13). 27--38.

Digital Library

[30]

Luis Quesada, Gustavo López, and Luis A. Guerrero. 2015. Sign Language Recognition Using Leap Motion. Springer International Publishing, Cham, 277--288.

[31]

Wenjie Ruan, Quan Z. Sheng, Lei Yang, Tao Gu, Peipei Xu, and Longfei Shangguan. 2016. AudioGest: Enabling Fine-grained Hand Gesture Detection by Decoding Echo Signal. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). 474--485.

Digital Library

[32]

C. Savur and F. Sahin. 2015. Real-Time American Sign Language Recognition System Using Surface EMG Signal. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA). 497--502.

[33]

Jiacheng Shang and Jie Wu. 2017. A Robust Sign Language Recognition System with Multiple Wi-Fi Devices. In Proceedings of the Workshop on Mobility in the Evolving Internet Architecture (MobiArch '17). 19--24.

Digital Library

[34]

SignAll. 2017. SignAll Prototype. (2017). Retrieved July 23, 2017 from http://www.signall.us.

[35]

StartASL. 2017. Basic Words in Sign Language. (2017). Retrieved July 14, 2017 from https://www.startasl.com/basic-words-in-sign-language.html.

[36]

Chao Sun, Tianzhu Zhang, and Changsheng Xu. 2015. Latent Support Vector Machine Modeling for Sign Language Recognition with Kinect. ACM Trans. Intell. Syst. Technol. 6, 2, Article 20 (March 2015), 20 pages.

Digital Library

[37]

Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu-Han Kim. 2015. WiDraw: Enabling Hands-free Drawing in the Air on Commodity WiFi Devices. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom '15). 77--89.

Digital Library

[38]

Sheng Tan and Jie Yang. 2016. WiFinger: Leveraging Commodity WiFi for Fine-grained Finger Gesture Recognition. In Proceedings of the 17th ACM International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc '16). 201--210.

Digital Library

[39]

David Tse and Pramod Viswanath. 2005. Fundamentals of Wireless Communication. Cambridge University Press.

Digital Library

[40]

Aditya Virmani and Muhammad Shahzad. 2017. Position and Orientation Agnostic Gesture Recognition Using WiFi. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '17). 252--264.

Digital Library

[41]

Jue Wang, Deepak Vasisht, and Dina Katabi. 2014. RF-IDraw: Virtual Touch Screen in the Air Using RF Signals. In Proceedings of the 2014 ACM Conference on SIGCOMM (SIGCOMM '14). 235--246.

Digital Library

[42]

Wei Wang, Alex X. Liu, and Ke Sun. 2016. Device-free Gesture Tracking Using Acoustic Signals. In Proceedings of the 22Nd Annual International Conference on Mobile Computing and Networking (MobiCom '16). 82--94.

Digital Library

[43]

Hikaru Watanabe, Masahiro Mochizuki, Kazuya Murao, and Nobuhiko Nishio. 2016. A Recognition Method for Continuous Gestures with an Accelerometer. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (UbiComp '16). 813--822.

Digital Library

[44]

Hongyi Wen, Julian Ramos Rojas, and Anind K. Dey. 2016. Serendipity: Finger Gesture Recognition Using an Off-the-Shelf Smartwatch. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). 3847--3851.

Digital Library

[45]

Chao Xu, Parth H. Pathak, and Prasant Mohapatra. 2015. Finger-writing with Smartwatch: A Case for Finger and Hand Gesture Recognition Using Smartwatch. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications (HotMobile '15). 9--14.

Digital Library

[46]

Sangki Yun, Yi-Chao Chen, Huihuang Zheng, Lili Qiu, and Wenguang Mao. 2017. Strata: Fine-Grained Acoustic-based Device-Free Tracking. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '17). 15--28.

Digital Library

[47]

Zahoor Zafrulla, Helene Brashear, Thad Starner, Harley Hamilton, and Peter Presti. 2011. American Sign Language Recognition with the Kinect. In Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI '11). 279--286.

Digital Library

[48]

Ouyang Zhang and Kannan Srinivasan. 2016. Mudra: User-friendly Fine-grained Gesture Recognition Using WiFi Signals. In Proceedings of the 12th International on Conference on Emerging Networking Experiments and Technologies (CoNEXT '16). 83--96.

Digital Library

[49]

Chen Zhao, Ke-Yu Chen, Md Tanvir Islam Aumi, Shwetak Patel, and Matthew S. Reynolds. 2014. SideSwipe: Detecting In-air Gestures Around Mobile Devices Using Actual GSM Signal. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). 527--534.

Digital Library

Cited By

Xie JLi ZFeng CLin JMeng X(2024)Wi-AM: Enabling Cross-Domain Gesture Recognition with Commodity Wi-FiSensors10.3390/s2405135424:5(1354)Online publication date: 20-Feb-2024
https://doi.org/10.3390/s24051354
Hassan MKelsey TRahman F(2024)Adversarial AI applied to cross-user inter-domain and intra-domain adaptation in human activity recognition using wireless signalsPLOS ONE10.1371/journal.pone.029888819:4(e0298888)Online publication date: 18-Apr-2024
https://doi.org/10.1371/journal.pone.0298888
Zhang HGu L(2024)Environment-independent textile fiber identification using Wi-Fi channel state informationTextile Research Journal10.1177/0040517524122793494:11-12(1316-1333)Online publication date: 19-Feb-2024
https://doi.org/10.1177/00405175241227934
Show More Cited By

Index Terms

SignFi: Sign Language Recognition Using WiFi
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction techniques
      1. Gestural input
  2. Interaction design

Recommendations

Recent methods and databases in vision-based hand gesture recognition

The paper surveys RGB and RGB-D sensors based hand gesture recognition methods.Dynamic as well as static gesture (posture/pose) recognition methods are reviewed.Qualitative as well as quantitative comparison of algorithms is provided.Twenty-six publicly ...
Signing Exact English (SEE): Modeling and recognition

We present effective and robust algorithms to recognize isolated signs in Signing Exact English (SEE). The sign-level recognition scheme comprises classifiers for handshape, hand movement and hand location. The SEE gesture data are acquired using ...
Isolated sign language recognition using Convolutional Neural Network hand modelling and Hand Energy Image

This paper presents an isolated sign language recognition system that comprises of two main phases: hand tracking and hand representation. In the hand tracking phase, an annotated hand dataset is used to extract the hand patches to pre-train ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 2, Issue 1

March 2018

1370 pages

EISSN:2474-9567

DOI:10.1145/3200905

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 March 2018

Accepted: 01 January 2018

Revised: 01 November 2017

Received: 01 August 2017

Published in IMWUT Volume 2, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

NSF

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

241
Total Citations
View Citations
5,593
Total Downloads

Downloads (Last 12 months)1,226
Downloads (Last 6 weeks)139

Reflects downloads up to 15 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xie JLi ZFeng CLin JMeng X(2024)Wi-AM: Enabling Cross-Domain Gesture Recognition with Commodity Wi-FiSensors10.3390/s2405135424:5(1354)Online publication date: 20-Feb-2024
https://doi.org/10.3390/s24051354
Hassan MKelsey TRahman F(2024)Adversarial AI applied to cross-user inter-domain and intra-domain adaptation in human activity recognition using wireless signalsPLOS ONE10.1371/journal.pone.029888819:4(e0298888)Online publication date: 18-Apr-2024
https://doi.org/10.1371/journal.pone.0298888
Zhang HGu L(2024)Environment-independent textile fiber identification using Wi-Fi channel state informationTextile Research Journal10.1177/0040517524122793494:11-12(1316-1333)Online publication date: 19-Feb-2024
https://doi.org/10.1177/00405175241227934
Wang ZChen ZZhang YGeng CSong WSun ZGuo BYu ZChen L(2024)GrainSense: A Wireless Grain Moisture Sensing System Based on Wi-Fi SignalsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785898:3(1-25)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678589
Hou WWu C(2024)RFBoost: Understanding and Boosting Deep WiFi Sensing via Physical Data AugmentationProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596208:2(1-26)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659620
Sheng BHan RXiao FGuo ZGui L(2024)MetaFormerProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435508:1(1-27)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643550
Liu YYu AWang LGuo BLi YYi EZhang D(2024)UniFiProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314297:4(1-29)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631429
Liao PWang XAn LMao SZhao TYang C(2024)TFSemantic: A Time–Frequency Semantic GAN Framework for Imbalanced Classification Using Radio SignalsACM Transactions on Sensor Networks10.1145/361409620:4(1-22)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3614096
Zhang XXu HLiu JHan J(2024)TomFi: Small Object Tracking Using Commodity WiFiACM Transactions on Sensor Networks10.1145/358877220:4(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3588772
Kaur HRani VKumar M(2024)Human activity recognition: A comprehensive reviewExpert Systems10.1111/exsy.13680Online publication date: 27-Jul-2024
https://doi.org/10.1111/exsy.13680
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents