Article

Using eye movements to determine referents in a spoken dialogue system

Authors:

Jason Baldridge,

Beth Ann Hockey,

Roger W. Remington,

Leland S. StoneAuthors Info & Claims

PUI '01: Proceedings of the 2001 workshop on Perceptive user interfaces

Pages 1 - 5

https://doi.org/10.1145/971478.971489

Published: 15 November 2001 Publication History

Abstract

Most computational spoken dialogue systems take a "literary" approach to reference resolution. With this type of approach, entities that are mentioned by a human interactor are unified with elements in the world state based on the same principles that guide the process during text interpretation. In human-to-human interaction, however, referring is a much more collaborative process. Participants often under-specify their referents, relying on their discourse partners for feedback if more information is needed to uniquely identify a particular referent. By monitoring eye-movements during this interaction, it is possible to improve the performance of a spoken dialogue system on referring expressions that are underspecified according to the literary model. This paper describes a system currently under development that employs such a strategy.

References

[1]

Allopenna, P., Magnuson, J., and Tanenhaus, M., 1998. Tracking the time course of spoken word recognition. Journal of Memory and Language, vol. 38, pages 419--439.

[2]

Baldwin, B., 1995. CogNiac: A discourse processing engine. Ph.D. Thesis, University of Pennsylvania, Department of Computer and Information Sciences.

Digital Library

[3]

Clark, H., and Schaefer, E., 1989. Collaborating on contributions to converstations. In Dietrich, R., and Graumann, C. (eds.) Language processing in Social Contexts. Elsevier Press.

[4]

Clark, H., and Wilkes-Gibbs, D., 1986. Referring as a collaborative process. Cognition, vol. 22, pages 1--39.

[5]

Crain, S. and Steedman, M., 1985. On not being led up the garden path: the use of context by the Psychological Parser. In Dowty, D., Kartunnen, L., and Zwicky, A. (eds.) Natural Language Parsing: Psychological, Computational and Theoretical Perspectives. Cambridge University Press.

[6]

Dowding, J., Gawron, M., Appelt, D., Cherny, L., Moore, R. and Moran, D. 1993. Gemini: A natural language system for spoken language understanding. In Proceedings for the Thirty-First Annual Meeting of the Association for Computational Linguistics.

Digital Library

[7]

Dowding, J., Hockey, B. A., Gawron, M. J., and Culy, C. 2001. Practical issues in compiling typed unification grammars for speech recognition. Proceedings for the 39th Annual Meeting of the Association for Computational Linguistics. Toulouse, France.

Digital Library

[8]

Griffin, Z., and Bock, K. 2000. What the eyes say about speaking. Psychological Science, vol. 11(4), pages 274--279.

[9]

Martin, D., Cheyer, A. and Moran, D. 1998. Building distributed software systems with the open agent architecture. In Proceedings of the Third International Conference on the Practical Application of Intelligent Agents and Multi-Agent Technology.

[10]

Moore, R., Dowding, J., Bratt, H., Gawron, J., Gorfu, Y., and Cheyer, A. 1997. CommandTalk: A spoken-language interface for battlefield simulations. In Proceedings of the Fifth Conference on Applied Natural Language Processing, pages 1--7.

Digital Library

[11]

PSA, 2001. Personal Satellite Assistant (PSA) Project. http://ic.arc.nasa.gov/ic/psa. As of 2 July 2001.

[12]

Nuance, 2001. Nuance Communications, Inc. http://www.nuance.com. As of July 2, 2001.

[13]

SMI, 2001. Sensomotoric Instruments, Inc. http://www.smi.de/el/index.html. As of 2 July 2001.

[14]

Stent, A., Dowding, J., Gawron, J., Bratt, E., and Moore, R. 1999. The CommandTalk spoken dialog system. In Proceedings of the Thirty-Seventh Annual Meeting of the Association for Computational Linguistics, pages 183--190.

Digital Library

[15]

Tanenhaus, M., Spivey-Knowlton, M., Eberhard, K., and Sedivy, J., 1995. Integration of visual and linguistic information in spoken language comprehension. Science, vol. 268, pages 632--634.

[16]

Van Eijck, J., and Moore, R. 1992. Semantic rules for English. In Alshawi, H. (editor) The Core Language Engine. MIT Press.

Cited By

Andrist SGleicher MMutlu BMark GFussell SLampe Cschraefel mHourcade JAppert CWigdor D(2017)Looking CoordinatedProceedings of the 2017 CHI Conference on Human Factors in Computing Systems10.1145/3025453.3026033(2571-2582)Online publication date: 2-May-2017
https://dl.acm.org/doi/10.1145/3025453.3026033
Andrist SCollier WGleicher MMutlu BShaffer D(2015)Look together: analyzing gaze coordination with epistemic network analysisFrontiers in Psychology10.3389/fpsyg.2015.010166Online publication date: 21-Jul-2015
https://doi.org/10.3389/fpsyg.2015.01016
Ratzka A(2013)User Interface Patterns for Multimodal InteractionTransactions on Pattern Languages of Programming III10.1007/978-3-642-38676-3_4(111-167)Online publication date: 2013
https://doi.org/10.1007/978-3-642-38676-3_4
Show More Cited By

Index Terms

Using eye movements to determine referents in a spoken dialogue system
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Touch screens

Recommendations

Eye Gaze for Spoken Language Understanding in Multi-modal Conversational Interactions
ICMI '14: Proceedings of the 16th International Conference on Multimodal Interaction

When humans converse with each other, they naturally amalgamate information from multiple modalities (i.e., speech, gestures, speech prosody, facial expressions, and eye gaze). This paper focuses on eye gaze and its combination with speech. We develop a ...
From vocal to multimodal dialogue management
ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces

Multimodal, speech-enabled systems pose different research problems when compared to unimodal, voice-only dialogue systems. One of the important issues is the question of how a multimodal interface should look like in order to make the multimodal ...
Human-robot collaborative tutoring using multiparty multimodal spoken dialogue
HRI '14: Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction

In this paper, we describe a project that explores a novel experimental setup towards building a spoken, multi-modally rich, and human-like multiparty tutoring robot. A human-robot interaction setup is designed, and a human-human dialogue corpus is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

PUI '01: Proceedings of the 2001 workshop on Perceptive user interfaces

November 2001

241 pages

ISBN:9781450374736

DOI:10.1145/971478

Copyright © 2001 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 November 2001

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

PUI01

PUI01: Workshop on Perceptive User Interfaces

November 15 - 16, 2001

Florida, Orlando, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
251
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Andrist SGleicher MMutlu BMark GFussell SLampe Cschraefel mHourcade JAppert CWigdor D(2017)Looking CoordinatedProceedings of the 2017 CHI Conference on Human Factors in Computing Systems10.1145/3025453.3026033(2571-2582)Online publication date: 2-May-2017
https://dl.acm.org/doi/10.1145/3025453.3026033
Andrist SCollier WGleicher MMutlu BShaffer D(2015)Look together: analyzing gaze coordination with epistemic network analysisFrontiers in Psychology10.3389/fpsyg.2015.010166Online publication date: 21-Jul-2015
https://doi.org/10.3389/fpsyg.2015.01016
Ratzka A(2013)User Interface Patterns for Multimodal InteractionTransactions on Pattern Languages of Programming III10.1007/978-3-642-38676-3_4(111-167)Online publication date: 2013
https://doi.org/10.1007/978-3-642-38676-3_4
Prasov ZChai JStaab S(2008)What's in a gaze?Proceedings of the 13th international conference on Intelligent user interfaces10.1145/1378773.1378777(20-29)Online publication date: 13-Jan-2008
https://dl.acm.org/doi/10.1145/1378773.1378777
Ou JOh LFussell SBlum TYang J(2008)Predicting Visual Focus of Attention From Intention in Remote Collaborative TasksIEEE Transactions on Multimedia10.1109/TMM.2008.200136310:6(1034-1045)Online publication date: 1-Oct-2008
https://dl.acm.org/doi/10.1109/TMM.2008.2001363
Wang DLi JZhang JDai G(2008)A pen and speech-based storytelling system for Chinese childrenComputers in Human Behavior10.1016/j.chb.2008.03.01424:6(2507-2519)Online publication date: 17-Sep-2008
https://dl.acm.org/doi/10.1016/j.chb.2008.03.014
Wang DZhang JLi JDai GLin Q(2007)A multimodal 3D storytelling system for Chinese childrenProceedings of the 2nd international conference on Technologies for e-learning and digital entertainment10.5555/1772177.1772231(511-526)Online publication date: 11-Jun-2007
https://dl.acm.org/doi/10.5555/1772177.1772231
Wang DZhang JLi JDai GLin Q(2007)A Multimodal 3D Storytelling System for Chinese ChildrenTechnologies for E-Learning and Digital Entertainment10.1007/978-3-540-73011-8_50(511-526)Online publication date: 2007
https://doi.org/10.1007/978-3-540-73011-8_50
Wang DZhang JDai G(2006)A multimodal fusion framework for children's storytelling systemsProceedings of the First international conference on Technologies for E-Learning and Digital Entertainment10.1007/11736639_71(585-588)Online publication date: 16-Apr-2006
https://dl.acm.org/doi/10.1007/11736639_71
Ou JOh LFussell SBlum TYang JLazzari GPianesi FCrowley JMase KOviatt S(2005)Analyzing and predicting focus of attention in remote collaborative tasksProceedings of the 7th international conference on Multimodal interfaces10.1145/1088463.1088485(116-123)Online publication date: 4-Oct-2005
https://dl.acm.org/doi/10.1145/1088463.1088485
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents