Cited By
View all- Warchoł DKapuściński T(2023)A Four-Stage Mahalanobis-Distance-Based Method for Hand Posture RecognitionApplied Sciences10.3390/app13221234713:22(12347)Online publication date: 15-Nov-2023
Sign-to-911: Emergency Call Service for Sign Language Users with Assistive AR Glasses
Authors: 
Yunqi Guo, Jinghao Zhao, Boyan Ding, Congkai Tan, Weichong Ling, Zhaowei Tan, Jennifer Miyaki, Hongzhe Du, Songwu LuAuthors Info & Claims
Yunqi Guo, Jinghao Zhao, Boyan Ding, Congkai Tan, Weichong Ling, Zhaowei Tan, Jennifer Miyaki, Hongzhe Du, Songwu LuAuthors Info & ClaimsArticle No.: 46, Pages 1 - 15
Abstract
Sign-to-911 offers a compact mobile system solution to fast and runtime American Sign Language (ASL) and English translations. It is designated as 911 call services for ASL users with hearing disabilities upon emergencies. It enables bidirectional translations of ASL-to-English and English-to-ASL. The signer wears the AR glasses, runs Sign-to-911 on his/her smartphone and glasses, and interacts with a 911 operator. The design of Sign-to-911 departs from the popular deep learning based solution paradigm, and adopts simpler traditional AI/machine learning (ML) models. The key is to exploit ASL linguistic features to simplify the model structures and improve accuracy and speed. It further leverages recent component solutions from graphics, vision, natural language processing, and AI/ML. Our evaluation with six ASL signers and 911 call records has confirmed its viability.
References
[1]
Accessibility service. https://developer.android.com/reference/android/accessibilityservice/AccessibilityService. Accessed: 2022-11-29.
[2]
Ahmed, M. A., Zaidan, B. B., Zaidan, A. A., Salih, M. M., and Lakulu, M. M. b. A review on systems-based sensory gloves for sign language recognition state of the art between 2007 and 2017. Sensors 18, 7 (2018), 2208.
[3]
Andersson, C., Nilsson, I., Olsson, P., Slot, G., Sörensen, J., Liechty, D., Hunter, R., Kambhatla, S., Adermann, S., M., C., Godia-Caner, P., Mettälä, R., and Box, M. Rfcomm with ts 07.10, 2001.
[4]
Android camera api. https://developer.android.com/guide/topics/media/camera, Mar. 2023.
[5]
Android (go edition). https://www.android.com/versions/go-edition/, Mar. 2023.
[6]
Android mediacodec. https://developer.android.com/reference/android/media/MediaCodec, Mar. 2023.
[7]
Android motion sensors. https://developer.android.com/guide/topics/sensors/sensors_motion, Mar. 2023.
[8]
Android texttospeech. https://developer.android.com/reference/android/speech/tts/TextToSpeech, Mar. 2023.
[9]
Andronix app. https://andronix.app/. Accessed: 2023-03-09.
[10]
Asl translator. https://apps.apple.com/us/app/asl-translator/id421784745?correlationId=fc9f5193-5430-4cf1-8249-0b7052ee005c, Mar. 2023.
[11]
Bandukda, M., and Holloway, C. Audio ar to support nature connectedness in people with visual disabilities. In Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers (2020), pp. 204--207.
[12]
Bazarevsky, V., Grishchenko, I., Raveendran, K., Zhu, T., Zhang, F., and Grundmann, M. Blazepose: On-device real-time body pose tracking. arXiv preprint arXiv:2006.10204 (2020).
[13]
Bird, S., Klein, E., and Loper, E. Natural language processing with Python: analyzing text with the natural language toolkit. " O'Reilly Media, Inc.", 2009.
[14]
Cephei, A. Vosk offline speech recognition api. https://alphacephei.com/vosk/. Accessed: 2022-11-29.
[15]
CHIRICO, M. Emergency - 911 calls. https://www.kaggle.com/datasets/mchirico/montcoalert. Accessed: 2022-11-29.
[16]
Commission, F. C. Real-time text (rtt). https://www.fcc.gov/real-time-text, 2023. Accessed: 2023-07-30.
[17]
Commission, F. C. Telecommunications relay service (trs). https://www.fcc.gov/consumers/guides/telecommunications-relay-service-trs, 2023. Accessed: 2023-07-30.
[18]
Commission, F. C. Text to 911: What you need to know. https://www.fcc.gov/consumers/guides/what-you-need-know-about-text-911, 2023. Accessed: 2023-07-30.
[19]
Coulter, G. R. Current Issues in ASL Phonology: Phonetics and Phonology, Vol. 3, vol. 3. Academic Press, 2014.
[20]
Department, E. P. 9-1-1 call scripts. https://www.eugene-or.gov/2892/9-1-1-Call-Scripts. Accessed: 2022-11-20.
[21]
Dick, E. Current and potential uses of ar/vr for equity and inclusion. Tech. rep., Information Technology and Innovation Foundation, 2021.
[22]
ElKoura, G., and Singh, K. Handrix: animating the human hand. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation (2003), pp. 110--119.
[23]
Fang, B., Co, J., and Zhang, M. Deepasl: Enabling ubiquitous and non-intrusive word and sentence-level sign language translation. In Proceedings of the 15th ACM conference on embedded network sensor systems (2017), pp. 1--13.
[24]
Friedman, L. A. Phonology of a soundless language: phonological structure of the American sign language. University of California, Berkeley, 1976.
[25]
Google. Filament. https://github.com/google/filament, 11 2022.
[26]
Hand talk: Your website accessible in asl. https://www.handtalk.me/en/, Jan 2023.
[27]
Hanke, T. Hamnosys-representing sign language data in language resources and language processing contexts. In LREC (2004), vol. 4, pp. 1--6.
[28]
Hastie, T., Tibshirani, R., Friedman, J. H., and Friedman, J. H. The elements of statistical learning: data mining, inference, and prediction, vol. 2. Springer, 2009.
[29]
Homonyms in sign language. https://www.handspeak.com/learn/209, Mar. 2023.
[30]
Hou, J., Li, X.-Y., Zhu, P., Wang, Z., Wang, Y., Qian, J., and Yang, P. Signspeaker: A real-time, high-precision smartwatch-based sign language translator. In The 25th Annual International Conference on Mobile Computing and Networking (2019), pp. 1--15.
[31]
Inmo air. https://vr-compare.com/headset/inmoair, Mar. 2023.
[32]
Insta 360 go 2. https://www.insta360.com/cn/product/insta360-go2/, Mar. 2023.
[33]
Jiang, S., Sun, B., Wang, L., Bai, Y., Li, K., and Fu, Y. Skeleton aware multi-modal sign language recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), pp. 3413--3423.
[34]
Jin, Y., Gao, Y., Zhu, Y., Wang, W., Li, J., Choi, S., Li, Z., Chauhan, J., Dey, A. K., and Jin, Z. Sonicasl: An acoustic-based sign language gesture recognizer using earphones. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 2 (2021), 1--30.
[35]
Koehn, P. Statistical machine translation. Cambridge University Press, 2009.
[36]
Li, D., Rodriguez, C., Yu, X., and Li, H. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In Proceedings of the IEEE/CVF winter conference on applications of computer vision (2020), pp. 1459--1469.
[37]
Liddell, S. K. Grammar, gesture, and meaning in American Sign Language. Cambridge University Press, 2003.
[38]
Liddell, S. K., and Johnson, R. E. American sign language: The phonological base. Sign language studies 64, 1 (1989), 195--277.
[39]
Macháček, D., Žilinec, M., and Bojar, O. Lost in interpreting: Speech translation from source or interpreter? arXiv preprint arXiv:2106.09343 (2021).
[40]
MacKenzie, C. L., and Iberall, T. The grasping hand. Elsevier, 1994.
[41]
Manifest.permission. https://developer.android.com/reference/android/Manifest.permission. Accessed: 2022-11-29.
[42]
Mediapipe framework on android. https://developers.google.com/mediapipe/framework/getting_started/android, Mar. 2023.
[43]
Mitchell, R. E., Young, T. A., Bachelda, B., and Karchmer, M. A. How many people use asl in the united states? why estimates need updating. Sign Language Studies 6, 3 (2006), 306--335.
[44]
News, C. Captioned smart glasses let deaf people see, rewind conversations. https://www.cbsnews.com/miami/news/captioned-smart-glasses-let-deaf-people-see-rewind-conversations/, 2022. Accessed: 2023-07-30.
[45]
on Deafness, N. I., and Disorders, O. C. Quick statistics about hearing. https://www.nidcd.nih.gov/health/statistics/quick-statistics-hearing, Mar. 2023.
[46]
OpenAI. 71 pct police involved shooting 911 transcript 4/4/2018. https://www.nyc.gov/assets/nypd/downloads/pdf/public_information/911-transcripts-police-involved-shooting-040418.pdf, 2020. Accessed: 2023-02-25.
[47]
OpenAI. Transcript of 911 call placed by jason ravnsborg on saturday, september 12, 2020. https://dps.sd.gov/application/files/7216/0260/1522/911-call-transcribed.pdf, 2020. Accessed: 2023-02-25.
[48]
OpenAI. chatgpt. https://chat.openai.com/chat, 2022. Accessed: 2023-02-29.
[49]
Pedregosa, F., Varoqaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12 (2011), 2825--2830.
[50]
Pennec, X. Computing the mean of geometric features application to the mean rotation. PhD thesis, INRIA, 1998.
[51]
PlasticsToday. Ar glasses transcribe and display spoken language for hearing impaired. https://www.plasticstoday.com/medical/ar-glasses-transcribe-and-display-spoken-language-hearing-impaired. Accessed: 2023-06-20.
[52]
Salvador, S., and Chan, P. Toward accurate dynamic time warping in linear time and space. Intelligent Data Analysis 11, 5 (2007), 561--580.
[53]
Saunders, B., Camgoz, N. C., and Bowden, R. Progressive transformers for end-to-end sign language production. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XI 16 (2020), Springer, pp. 687--705.
[54]
Sehyr, Z. S., Caselli, N., Cohen-Goldberg, A. M., and Emmorey, K. The asl-lex 2.0 project: A database of lexical and phonological properties for 2,723 signs in american sign language. The Journal of Deaf Studies and Deaf Education 26, 2 (2021), 263--277.
[55]
Shao, Q., Sniffen, A., Blanchet, J., Hillis, M. E., Shi, X., Haris, T. K., Liu, J., Lamberton, J., Malzkuhn, M., Quandt, L. C., et al. Teaching american sign language in mixed reality. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 4 (2020), 1--27.
[56]
Shi, B., Del Rio, A. M., Keane, J., Michaux, J., Brentari, D., Shakhnarovich, G., and Livescu, K. American sign language fingerspelling recognition in the wild. In 2018 IEEE Spoken Language Technology Workshop (SLT) (2018), IEEE, pp. 145--152.
[57]
Sign language translator. https://apps.apple.com/us/app/sign-language-translator/id1458992650, Mar. 2023.
[58]
Signing savvy. https://www.signingsavvy.com/, Mar. 2023.
[59]
Sondej, F. Autocorrect. https://github.com/filyp/autocorrect, Mar. 2023. Version: 2.0.4, Accessed: 2023-03-01.
[60]
Specia, L., Raj, D., and Turchi, M. Machine translation evaluation versus quality estimation. Machine translation 24 (2010), 39--50.
[61]
Stoll, S., Camgoz, N. C., Hadfield, S., and Bowden, R. Text2sign: towards sign language production using neural machine translation and generative adversarial networks. International Journal of Computer Vision 128, 4 (2020), 891--908.
[62]
Stoll, S., Hadfield, S., and Bowden, R. Signsynth: Data-driven sign language video generation. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23--28, 2020, Proceedings, Part IV 16 (2020), Springer, pp. 353--370.
[63]
Tennant, R. A., Gluszak, M., and Brown, M. G. The American sign language handshape dictionary. Gallaudet University Press, 1998.
[64]
The asl interpreter shortage and its impact on accessibility in college settings. https://nationaldeafcenter.org/news-items/the-asl-interpreter-shortage-and-its-impact-on-accessibility-in-college-settings/, Dec. 2022.
[65]
Transmission rate vs. bandwidth in bluetooth technology. https://resources.pcb.cadence.com/blog/2022-transmission-rate-vs-bandwidth-in-bluetooth-technology, Mar. 2023.
[66]
Unmarked and marked handshapes in sign language. https://www.handspeak.com/learn/index.php?id=439. Accessed: 2023-02-29.
[67]
Valli, C., and Lucas, C. Linguistics of American sign language: An introduction. Gallaudet University Press, 2000.
[68]
Vicars, B. Classifiers. https://www.lifeprint.com/asl101/pages-signs/classifiers/classifiers-main.htm, 2023. Accessed: 2023-07-15.
[69]
Vivo sign language translator. https://assist.vivo.com/detail/translate, Mar. 2023.
[70]
Voiss. https://www.projectvoiss.org/, Mar. 2023.
[71]
Vuzix blade 2. https://vr-compare.com/headset/vuzixblade2, Mar. 2023.
[72]
Wikipedia contributors. Mixamo --- Wikipedia, the free encyclopedia. https://en.wikipedia.org/w/index.php?title=Mixamo&oldid=1106970206, 2022. Accessed: 7-December-2022.
[73]
Wikipedia contributors. Signing space --- Wikipedia, the free encyclopedia. https://en.wikipedia.org/wiki/Signing_space, 2023. Accessed: 2023-07-30.
[74]
Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C.-L., and Grundmann, M. Mediapipe hands: On-device real-time hand tracking. arXiv preprint arXiv:2006.10214 (2020).
[75]
Zhang, Q., Jing, J., Wang, D., and Zhao, R. Wearsign: Pushing the limit of sign language translation using inertial and emg wearables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 1 (2022), 1--27.
[76]
Zhao, Y., Bennett, C. L., Benko, H., Cutrell, E., Holz, C., Morris, M. R., and Sinclair, M. Enabling people with visual impairments to navigate virtual reality with a haptic and auditory cane simulation. In Proceedings of the 2018 CHI conference on human factors in computing systems (2018), pp. 1--14.
[77]
Zhou, Z., Chen, K., Li, X., Zhang, S., Wu, Y., Zhou, Y., Meng, K., Sun, C., He, Q., Fan, W., et al. Sign-to-speech translation using machine-learning-assisted stretchable sensor arrays. Nature Electronics 3, 9 (2020), 571--578.
Index Terms
- Sign-to-911: Emergency Call Service for Sign Language Users with Assistive AR Glasses
Recommendations
American and Russian Sign Language Dactyl Recognition and Text2Sign Translation
Analysis of Images, Social Networks and TextsAbstractSign language is the main way to communicate for people from deaf community. However, common people mostly do not know sign language. In this paper, we overview several real-time sign language dactyl recognition systems using deep convolutional ...
A machine translation system from Arabic sign language to Arabic
AbstractArabic sign language (ArSL) is one of the sign languages that is used in Arab countries. This language has structure and grammar that differ from spoken Arabic. Available ArSL recognition systems perform direct mapping between the recognized sign ...
Cross-modal Neural Sign Language Translation
MM '19: Proceedings of the 27th ACM International Conference on MultimediaSign Language is the primary means of communication for the majority of the Deaf and hard-of-hearing communities. Current computational approaches in this general research area have focused specifically on sign language recognition and the translation ...
Comments
Information & Contributors
Information
Published In
October 2023
1605 pages
ISBN:9781450399906
DOI:10.1145/3570361
- Chairs:
- Xavier Costa,
- Joerg Widmer,
- Co-chairs:
- Diego Perino,
- Domenico Giustiniano,
- Program Chair:
- Haitham Al Hassanieh,
- Program Co-chairs:
- Arash Asadi,
- Landon Cox
Copyright © 2023 Owner/Author(s).
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 02 October 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ACM MobiCom '23
Sponsor:
ACM MobiCom '23: 29th Annual International Conference on Mobile Computing and Networking
October 2 - 6, 2023
Madrid, Spain
Acceptance Rates
Overall Acceptance Rate 440 of 2,972 submissions, 15%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 814Total Downloads
- Downloads (Last 12 months)814
- Downloads (Last 6 weeks)88
Reflects downloads up to 14 Sep 2024
Other Metrics
Citations
Cited By
View all- Warchoł DKapuściński T(2023)A Four-Stage Mahalanobis-Distance-Based Method for Hand Posture RecognitionApplied Sciences10.3390/app13221234713:22(12347)Online publication date: 15-Nov-2023
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in