CoChat
Sign Up
Cited by 19Multimodal Machine Learning ApplicationsArtificial Intelligence ApplicationsPsychiatry, Mental Health, Neuroscience

Your Prompt is My Command: On Assessing the Human-Centred Generality of Multimodal Models

Open access
Schellaert, Wout, Martínez-Plumed, Fernando, Vold, Karina, Burden, John, A. M. Casares, Pablo, Sheng Loe, Bao, Reichart, Roi, Ó hÉigeartaigh, Sean +2 more·2023·Journal of Artificial Intelligence Research
DOI 10.1613/jair.1.14157
Open this paper in CoChat — free accountOpen PDF

Abstract

Even with obvious deficiencies, large prompt-commanded multimodal models are proving to be flexible cognitive tools representing an unprecedented generality. But the directness, diversity, and degree of user interaction create a distinctive “human-centred generality” (HCG), rather than a fully autonomous one. HCG implies that —for a specific user— a system is only as general as it is effective for the user’s relevant tasks and their prevalent ways of prompting. A human-centred evaluation of general-purpose AI systems therefore needs to reflect the personal nature of interaction, tasks and cognition. We argue that the best way to understand these systems is as highly-coupled cognitive extenders, and to analyse the bidirectional cognitive adaptations between them and humans. In this paper, we give a formulation of HCG, as well as a high-level overview of the elements and trade-offs involved in the prompting process. We end the paper by outlining some essential research questions and suggestions for improving evaluation practices, which we envision as characteristic for the evaluation of general artificial intelligence in the future. This paper appears in the AI & Society track.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1613/jair.1.14157

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • These demands are exacerbated by GenAI's non-determinism and model flexibility in terms of (a) the wide range of explicit and implicit parameters that users can adjust, and (b) systems' ability to work with prompts at a wide range of abstraction.

    “We posit that these demands are exacerbated by GenAI's non-determinism and model flexibility (not to be confused with metacognitive flexibility) in terms of (a) the wide range of explicit and implicit parameters that users can adjust, and (b) systems' ability to work with prompts at a wide range of abstraction [156,157].”

    From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [157]

    Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • The generative nature of GenAI means that many user workflows will or have shifted from users generating content to evaluating it.

    “The generative nature of GenAI-its 'originality' [157]-means that many user workflows will or have shifted from users generating content to evaluating it [149,154], as already documented in programming [156] and manufacturing design [70].”

    From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [157]

    Section 4, “Elements of the Prompting Process” (page unavailable), describes generated images and code as potential outputs that save users from producing them themselves. Its “System” table also identifies user validation of an answer as a step. These passages support a generate-and-evaluate workflow, but do not establish that many user workflows have shifted, or will shift, from content creation to evaluation. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • GenAI can produce (often extensive) content, such as entire emails, presentations, or software.

    “GenAI can produce (often extensive) content, such as entire emails, presentations, or software.”

    From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [157]

    The Introduction describes generative models producing flexible outputs, and §2, “A New Kind of Cognitive Tool,” says they can generate novel content. The modalities table includes code generation. Our copy does not establish the more specific claim that they produce entire emails, presentations, or software; §4 instead gives an auto-completed piece of code as an example. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Evaluating these outputs for quality therefore becomes far more important and effortful (in terms of cognitive load) compared to 'auto-complete' phrase suggestions or intelligent code completion.

    “Evaluating these outputs for quality therefore becomes far more important and effortful (in terms of cognitive load) compared to 'auto-complete' phrase suggestions or intelligent code completion [156,157].”

    From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [157]

    The paper treats validating whether an output is fit for purpose as a cost (Table 2, System) and says users’ cognitive effort matters when evaluating multimodal systems (§2, “A New Kind of Cognitive Tool”). It does not establish that assessing their outputs is more important or more cognitively effortful than assessing auto-complete suggestions or intelligent code completion. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • GenAI's potential stems from a unique combination of model flexibility, generality, and originality.

    “Their potential stems from a unique combination of model flexibility (in their input/output space), generality (in their applicability to a wide range of tasks), and originality (in their ability to generate novel content) [157].”

    From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [157]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 84

  1. Abid, A., Farooqi, M., & Zou, J. (2021). Large language models associate Muslims with violence. Nature Machine Intelligence, 3 (6), 461-463.
  2. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., & Simonyan, K. (2022). Flamingo: A Visual Language Model for Few-Shot Learning. In Advances in Neural Information Processing Systems.
  3. Applin, S. A., & Fischer, M. D. (2015). New technologies and mixed-use convergence: How humans and algorithms are adapting to each other. In 2015 IEEE international symposium on technology and society (ISTAS), pp. 1-6. IEEE.
  4. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., & Kaplan, J. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv:2112.00861 [cs]. · arXiv
  5. Ben-David, E., Oved, N., & Reichart, R. (2022). Pada: Example-based prompt learning for on-the-fly adaptation to unseen domains. Transactions of the Association for Computational Linguistics, 10, 414-433.
  6. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, p. 610-623, New York, NY, USA. Association for Computing Machinery.
  7. Bhatt, U., Antorán, J., Zhang, Y., Liao, Q. V., Sattigeri, P., Fogliato, R., Melançon, G., Kr- ishnan, R., Stanley, J., Tickoo, O., Nachman, L., Chunara, R., Srikumar, M., Weller, A., & Xiang, A. (2021). Uncertainty as a Form of Transparency: Measuring, Commu- nicating, and Using Uncertainty. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES '21, pp. 401-413, New York, NY, USA. Association for Computing Machinery.
  8. BigScience, et al. (2023). BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. arXiv:2211.05100. · arXiv
  9. Bommasani, R., et al. (2021). On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. · arXiv
  10. Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driessche, G. B., Lespiau, J.-B., Damoc, B., Clark, A., et al. (2022). Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, pp. 2206-2240. PMLR.
  11. Bourguet, M.-L. (2003). Designing and prototyping multimodal commands. In Interact, Vol. 3, pp. 717-720.
  12. Brown, T., et al. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33, pp. 1877-1901.
  13. Casares, P. A. M., Loe, B. S., Burden, J., O'hEigeartaigh, S., & Hernandez-Orallo, J. (2022). How general-purpose is a language model? Usefulness and safety with human prompters in the wild. In AAAI.
  14. Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., & Tsaneva-Atanasova, K. (2019). Artificial intelligence, bias and clinical safety. BMJ Quality & Safety, 28 (3), 231-237.
  15. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. · arXiv
  16. Cheng, L., Varshney, K. R., & Liu, H. (2021). Socially Responsible AI Algorithms: Issues, Purposes, and Challenges. Journal of Artificial Intelligence Research, 71, 1137-1181.
  17. Chowdhery, A., & et al. (2022). PaLM: Scaling Language Modeling with Pathways. arXiv:2204.02311 [cs]. · arXiv
  18. Clark, A. (2004). Natural-born cyborgs: Minds, technologies, and the future of human intel- ligence. Oxford University Press.
  19. Clark, A. (2008). Supersizing the Mind: Embodiment, Action, and Cognitive Extension. Philosophy of Mind Series. Oxford University Press, New York.
  20. Clark, A., & Chalmers, D. (1998). The extended mind. Analysis, 58 (1), 7-19.
  21. Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly, pp. 319-340.
  22. DeLone, W. H., & McLean, E. R. (1992). Information systems success: The quest for the dependent variable. Information systems research, 3 (1), 60-95.
  23. Dinga, R., Penninx, B. W., Veltman, D. J., Schmaal, L., & Marquand, A. F. (2019). Beyond accuracy: measures for assessing machine learning models, pitfalls and guidelines. bioRxiv, p. 743138.
  24. Dror, I. E., & Harnad, S. (2008). Cognition distributed: How cognitive technology extends our minds, Vol. 16. John Benjamins Publishing.
  25. Eichenberg, C., Black, S., Weinbach, S., Parcalabescu, L., & Frank, A. (2022). MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2416- 2428, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  26. Fallis, D. (2021). The Epistemic Threat of Deepfakes. Philosophy & Technology, 34 (4), 623-643.
  27. Gilbert, S. J., Bird, A., Carpenter, J. M., Fleming, S. M., Sachdeva, C., & Tsai, P.-C. (2020). Optimal use of reminders: Metacognition, effort, and cognitive offloading.. Journal of Experimental Psychology: General, 149 (3), 501.
  28. Heersmink, R. (2021). Varieties of artifacts: Embodied, perceptual, cognitive, and affective. Topics in Cognitive Science, 13 (4), 573-596.
  29. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020). Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations.
  30. Hernández-Orallo, J., & Vold, K. (2019). AI extenders: the ethical and societal implica- tions of humans cognitively extended by ai. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 507-513.
  31. Hutchinson, B., Prabhakaran, V., Denton, E., Webster, K., Zhong, Y., & Denuyl, S. (2020). Social Biases in NLP Models as Barriers for Persons with Disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5491-5501, Online. Association for Computational Linguistics.
  32. Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020). How can we know what language models know?. Transactions of the Association for Computational Linguistics, 8, 423-438.
  33. Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Dodds, Z. H., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J., Kernion, J., Kravec, S., Lovitt, L., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., & Kaplan, J. (2022). Language Models (Mostly) Know What They Know. arXiv preprint arXiv:2207.05221. · arXiv
  34. Kharitonov, E., Lee, A., Polyak, A., Adi, Y., Copet, J., Lakhotia, K., Nguyen, T. A., Riviere, M., Mohamed, A., Dupoux, E., & Hsu, W.-N. (2022). Text-free prosody-aware generative spoken language modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8666- 8681, Dublin, Ireland. Association for Computational Linguistics.
  35. Kocielnik, R., Amershi, S., & Bennett, P. N. (2019). Will you accept an imperfect AI? exploring designs for adjusting end-user expectations of ai systems. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1-14.
  36. Kumar, S. (2022). Answer-level Calibration for Free-form Multiple Choice Question Answer- ing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 665-679, Dublin, Ireland. Association for Computational Linguistics.
  37. Laurel, B. (2013). Computers as theatre. Addison-Wesley.
  38. Lazar, J., Feng, J. H., & Hochheiser, H. (2017). Research methods in human-computer interaction. Morgan Kaufmann.
  39. Lee, M., Liang, P., & Yang, Q. (2022). Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In CHI Conference on Human Factors in Computing Systems, pp. 1-19.
  40. Morgan Kaufmann.
  41. Lieberman, H., & Maulsby, D. (1996). Instructible agents: Software that just keeps getting better. IBM systems journal, 35 (3.4), 539-556.
  42. Lin, S., Hilton, J., & Evans, O. (2022). Teaching Models to Express Their Uncertainty in Words. arXiv preprint arXiv:2205.14334. · arXiv
  43. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586. · arXiv
  44. Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022). Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitiv- ity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086-8098, Dublin, Ireland. Association for Computational Linguistics.
  45. Marsh, E. J., & Rajaram, S. (2019). The digital expansion of the mind: Implications of internet usage for memory and cognition. Journal of Applied Research in Memory and Cognition, 8 (1), 1-14.
  46. Martinez-Plumed, F., Barredo, P., Heigeartaigh, S. O., & Hernandez-Orallo, J. (2021). Re- search community dynamics behind popular ai benchmarks. Nature Machine Intelli- gence, 3 (7), 581-589.
  47. Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2021). A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54 (6), 1-35.
  48. Menary, R. (2010). The extended mind. Mit Press.
  49. Mielke, S. J., Szlam, A., Dinan, E., & Boureau, Y.-L. (2022). Reducing conversational agents' overconfidence through linguistic calibration. arXiv preprint arXiv:2012.14983. · arXiv
  50. Mishra, S., Khashabi, D., Baral, C., Choi, Y., & Hajishirzi, H. (2021). Reframing instruc- tional prompts to gptk's language. arXiv preprint arXiv:2109.07830. · arXiv
  51. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021). Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. · arXiv
  52. Nelson, R. R., & Cheney, P. H. (1987). Training end users: An exploratory study. MIS quarterly, pp. 547-559.
  53. Ngo, H., Araújo, J. G., Hui, J., & Frosst, N. (2021). No news is good news: A critique of the one billion word benchmark. arXiv preprint arXiv:2110.12609. · arXiv
  54. Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021). GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741. · arXiv
  55. Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., Sutton, C., & Odena, A. (2021). Show Your Work: Scratchpads for Intermediate Computation with Language Models. arXiv preprint arXiv:2112.00114. · arXiv
  56. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agar- wal, S., Slama, K., Gray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.
  57. Patel, A., Bhattamishra, S., & Goyal, N. (2021). Are NLP Models really able to Solve Simple Math Word Problems?. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080-2094, Online. Association for Computational Linguistics.
  58. Perez, E., Lewis, P., Yih, W.-t., Cho, K., & Kiela, D. (2020). Unsupervised question decom- position for question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8864-8880, Online. Associa- tion for Computational Linguistics.
  59. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022). Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. · arXiv
  60. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021). Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR.
  61. Rapp, A., Curti, L., & Boldi, A. (2021). The human side of human-chatbot interaction: A systematic literature review of ten years of research on text-based chatbots. Interna- tional Journal of Human-Computer Studies, p. 102630.
  62. Reynolds, L., & McDonell, K. (2021). Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA '21, New York, NY, USA. Association for Computing Machinery.
  63. Rini, R. (2020). Deepfakes and the Epistemic Backstop. Philosophers' Imprint, 20 (24), 1-16.
  64. Risko, E. F., & Gilbert, S. J. (2016). Cognitive offloading. Trends in cognitive sciences, 20 (9), 676-688.
  65. Rossi, F., Venable, K., & Walsh, T. (2011). A Short Introduction to Preferences: Between Artificial Intelligence and Social Choice, Vol. 5. Morgan & Claypool Publishers.
  66. Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022). Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487. · arXiv
  67. Scao, T. L., & Rush, A. M. (2021). How many data points is a prompt worth?. arXiv preprint arXiv:2103.08493. · arXiv
  68. Schmidhuber, J., Schlögl, S., & Ploder, C. (2021). Cognitive load and productivity implica- tions in human-chatbot interaction. In 2021 IEEE 2nd International Conference on Human-Machine Systems (ICHMS), pp. 1-6. IEEE.
  69. Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020). Eliciting knowledge from language models using automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222-4235.
  70. Shneiderman, B. (2022). Human-Centered AI. Oxford University Press.
  71. Shoham, Y. (2017). Towards the AI index. AI Magazine, 38 (4), 71-77.
  72. Sparrow, B., Liu, J., & Wegner, D. M. (2011). Google effects on memory: Cognitive conse- quences of having information at our fingertips. Science, 333 (6043), 776-778.
  73. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615. · arXiv
  74. Venkatesh, V., & Goyal, S. (2010). Expectation disconfirmation and technology adoption: polynomial modeling and response surface analysis. MIS quarterly, pp. 281-303.
  75. Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., & Shieber, S. (2020). Investigating Gender Bias in Language Models Using Causal Mediation Analysis. In NeurIPS.
  76. Wang, X., Ruder, S., & Neubig, G. (2022). Expanding pretrained models to thousands more languages via lexicon-based adaptation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 863-877, Dublin, Ireland. Association for Computational Linguistics.
  77. Ward, A. F. (2013). Supernormal: How the internet is changing our memories and our minds. Psychological Inquiry, 24 (4), 341-348.
  78. Ward, A. F., Duke, K., Gneezy, A., & Bos, M. W. (2017). Brain drain: The mere presence of one's own smartphone reduces available cognitive capacity. Journal of the Association for Consumer Research, 2 (2), 140-154.
  79. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., & Zhou, D. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems.
  80. Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., Kenton, Z., Brown, S., Hawkins, W., Stepleton, T., Biles, C., Birhane, A., Haas, J., Rimell, L., Hendricks, L. A., Isaac, W., Legassick, S., Irving, G., & Gabriel, I. (2021). Ethical and social risks of harm from Language Models. arXiv:2112.04359 [cs]. · arXiv
  81. Woods, D. D., & Hollnagel, E. (2006). Joint cognitive systems: Patterns in cognitive systems engineering. CRC Press.
  82. Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. (2022). Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789. · arXiv
  83. Zeng, W., Ren, X., Su, T., Wang, H., Liao, Y., Wang, Z., Jiang, X., Yang, Z., Wang, K., Zhang, X., et al. (2021). Pangu-α: Large-scale autoregressive pretrained chinese language models with auto-parallel computation. arXiv preprint arXiv:2104.12369. · arXiv
  84. Zhang, D., Mishra, S., Brynjolfsson, E., Etchemendy, J., Ganguli, D., Grosz, B., Lyons, T., Manyika, J., Niebles, J. C., Sellitto, M., et al. (2021). The AI index 2021 annual report. arXiv preprint arXiv:2103.06312. · arXiv

Cited by

Sorted by citations

Authors
Schellaert, W., Martínez-Plumed, F., Vold, K., Burden, J., A. M. Casares, P., Sheng Loe, B., Reichart, R., Ó hÉigeartaigh, S., Korhonen, A., & Hernández-Orallo, J.
Affiliations
  • University of Toronto (CA)
  • Technion – Israel Institute of Technology (IL)
ORCID
Venue
Journal of Artificial Intelligence Research
Publisher
AI Access Foundation
Published
2023-06-12
Volume
77
Pages
377-394
ISSN
1076-9757 (ISSN-L)
DOI
10.1613/jair.1.14157
OpenAlex
W4380355772
Cited by
19
References
0
Citations by year
2026: 3 · 2025: 8 · 2024: 8
FWCI
1.73
Type
journal article
Language
en
Open access
Licence
other-oa
Locations
  • Journal of Artificial Intelligence Research (publishedVersion, other-oa)
  • PhilPapers (PhilPapers Foundation) (submittedVersion)
Funders
  • Future of Life Institute (RFP2-152)
  • European Commission (2014‐2020, MCIN /AEI/ 10.13039/501100011033, 10.13039/501100011033, 13039/501100011033, 501100011033, HORIZON2020, /AEI/10.13039/501100011033, 952215, RTI2018-094403-B-C32)
  • Generalitat Valenciana (501100011033, 952215, 2014‐2020, PROMETEO/2019/098, 13039/501100011033, AEI/10.13039/501100011033, 10.13039/501100011033, INNEST/2021/317, PID2021-122830OB-C42, MCIN/AEI/10.13039/501100011033/)
  • Defense Advanced Research Projects Agency (HR00112120007)
  • Social Sciences and Humanities Research Council of Canada (13039)
  • Directorate-General for Communications Networks, Content and Technology
  • Universitat Politècnica de València (MCIN/AEI/10.13039/501100011033/, 13039/501100011033, MCIN/AEI/10, DOCEMPR22)
  • European Regional Development Fund (MCIN/AEI/10, 2014-2020’, RTI2018-094403-B-C32, PID2021-122830OB-C42, 10.13039/501100011033, 13039/501100011033, AEI/10.13039/501100011033, 501100011033, RTI2018, MCIN/ AEI/ 10.13039/501100011033)
  • Agencia Estatal de Investigación (AEI//10.13039/501100011033/, 501100011033, 10.13039/501100011033, MCIN/AEI/10.13039/501 100 011 033, 13039, .13039/501100011033, MCIN / AEI/10, PID2021-122830OB-C42, 10.13039, AEI/10, PID2021)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
Keywords
multimodal large language models, prompting, human-AI interaction, human-AI coupling, user interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review