Cited By
View all- Schmitt M(2024)Strategic Integration of Artificial Intelligence in the C-Suite: The Role of the Chief AI OfficerSSRN Electronic Journal10.2139/ssrn.4812654Online publication date: 2024
Who Determines What Is Relevant? Humans or AI? Why Not Both?
Abstract
A spectrum of human-artificial intelligence collaboration in assessing relevance.
References
[1]
Bender, E.M. et al. On the dangers of stochastic parrots: Can language models be too big? In FAccT '21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event. (Toronto, Canada, Mar. 3-10, 2021). M.C. Elish, W. Isaac, and R.S. Zemel, Eds. ACM, 2021
[2]
Daniel, F. et al. Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions. ACM Comput. Surv., 51, 1 (2018).
[3]
Faggioli, G. et al. Perspectives on large language models for relevance judgment. In Proceedings of the 2023 SIGIR Conference on the Theory of Information Retreival (ICTIR), 2023
[4]
Fok, R. and Weld, D.S. In Search of Verifiability: Explanations rarely enable complementary performance in AI-advised decision making, 2023; arXiv Preprint arXiv:2305.07722.
[5]
Hamalainen, P. et al. Evaluating large language models in generating synthetic HCI research data: A case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023
[6]
Hancock, P.A. Task partitioning effects in semi-automated human-machine system performance. Ergonomics 56, 9 (2013)
[7]
Kashefi, A. and Mukerji, T. ChatGPT for programming numerical methods, 2023; arXiv Preprint arXiv:2303.12093.
[8]
Lyu, Q. et al. Faithful chain-of-thought reasoning. CoRR, 2023
[9]
MacAvaney, S. and Soldaini, L. One-shot labeling for automatic relevance estimation. In SIGIR '23: The 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2023
[10]
Madani, A. et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology, 2023; https://www.nature.com/articles/s41587-022-01618-2.
[11]
Ouyang, L. et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, 5 (2022).
[12]
Sander, D. and Dietz, L. EXAM: How to evaluate retrieve-and-generate systems for users who do not (yet) know what they want. In Proceedings of the 2nd International Conference on Design of Experimental Search & Information REtrieval Systems (DESIRES), 2021; https://www.cs.unh.edu/~dietz/papers/sander2021exam.pdf
[13]
Seo, S. et al. Active learning on pre-trained language model with task-independent triplet loss. In Proceedings of the 34th Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The 12th Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, (Feb. 22-Mar. 1, 2022). AAAI Press, 2022; https://ojs.aaai.org/index.php/AAAI/article/view/21378
[14]
Sorensen, T. et al. An information-theoretic approach to prompt engineering without ground truth labels. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. S. Muresan, P. Nakov, and A. Villavicencio, Eds. Association for Computational Linguistics, 2022
[15]
Veselovsky, V. et al. Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks. CoRR, 2023
[16]
Windsor, J. et al. Preference testing: A comparison of two presentation methods. Research in Developmental Disabilities 15, 6 (1994)
[17]
Xu, J. et al. On the role of human and machine metadata in relevance judgment tasks. Information Processing and Management 60, 2 (2023)
Index Terms
- Who Determines What Is Relevant? Humans or AI? Why Not Both?
Recommendations
Humans as teammates: The signal of human–AI teaming enhances consumer acceptance of chatbots
AbstractHuman–artificial intelligence (AI) teaming is a service system in which AI agents work interdependently toward a common goal alongside human agents. Although many consumer services rely on chatbots working with humans, little is known about the ...
Highlights- The signal of Human–AI teaming can enhance consumer acceptance of chatbots.
- Perceived effectiveness mediates the signaling effect of human–AI teaming.
- Perceived authenticity mediates the signaling effect of human–AI teaming.
- AI ...
Bringing Humans at the Epicenter of Artificial Intelligence: A Confluence of AI, HCI and Human Centered Computing
AbstractArtificial Intelligence (AI) and Human Computer Interaction (HCI) often intersect under the purview of different technologies in a plethora of application areas, wherein it becomes pivotal to identify and interpret the exact roles of both the ...
Artificial argumentation for humans
IJCAI'18: Proceedings of the 27th International Joint Conference on Artificial IntelligenceThe latest years have seen an increasing interest in the topic of Artificial Intelligence (AI), the challenges it is facing, and the recent advances it has achieved, e.g., intelligent personal assistants. Differently from the past, where research on AI ...
Comments
Information & Contributors
Information
Published In
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 25 March 2024
Online First: 15 March 2024
Published in CACM Volume 67, Issue 4
Check for updates
Qualifiers
- Opinion
Funding Sources
- The Swiss National Science Foundation (SNSF)
- Australian Research Council
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 2,491Total Downloads
- Downloads (Last 12 months)2,491
- Downloads (Last 6 weeks)185
Reflects downloads up to 15 Sep 2024
Other Metrics
Citations
Cited By
View all- Schmitt M(2024)Strategic Integration of Artificial Intelligence in the C-Suite: The Role of the Chief AI OfficerSSRN Electronic Journal10.2139/ssrn.4812654Online publication date: 2024
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderDigital Edition
View this article in digital edition.
Digital EditionHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in