CoChat
Sign Up
PreprintCited by 3Ethics and Social Impacts of AIArtificial Intelligence ApplicationsTopic Modeling

Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge

Open access
Wallach, Hanna, Desai, Meera, Cooper, A. Feder, Wang, Angelina, Atalla, Chad, Barocas, Solon, Blodgett, Su Lin, Chouldechova, Alexandra +12 more·2025·arXiv
DOI 10.48550/arxiv.2502.00561arXiv 2502.00561
Open this paper in CoChat — free accountOpen PDF

Abstract

The measurement tasks involved in evaluating generative AI (GenAI) systems lack sufficient scientific rigor, leading to what has been described as "a tangle of sloppy tests [and] apples-to-oranges comparisons" (Roose, 2024). In this position paper, we argue that the ML community would benefit from learning from and drawing on the social sciences when developing and using measurement instruments for evaluating GenAI systems. Specifically, our position is that evaluating GenAI systems is a social science measurement challenge. We present a four-level framework, grounded in measurement theory from the social sciences, for measuring concepts related to the capabilities, behaviors, and impacts of GenAI systems. This framework has two important implications: First, it can broaden the expertise involved in evaluating GenAI systems by enabling stakeholders with different perspectives to participate in conceptual debates. Second, it brings rigor to both conceptual and operational debates by offering a set of lenses for interrogating validity.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.00561

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent calls in generative AI evaluation frameworks to separate the systematization of the concept to be measured and the operationalization of the measure technically further emphasizes the need to articulate what is evaluated before determining how to evaluate it.

    “Recent calls -in generative AI evaluation frameworks [115] -to separate the systematization of the concept to be measured, and the operationalization of the measure technically,further emphasizes the need to articulate what is evaluated before determining how to evaluate it.”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [115]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 66

  1. Abebe, R., Barocas, S., Kleinberg, J., Levy, K., Raghavan, M., and Robinson, D. G. Roles for computing in so- cial change. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 252-260, 2020.
  2. Adcock, R. and Collier, D. Measurement validity: A shared standard for qualitative and quantitative research. Ameri- can Political Science Review, 95(3):529-546, 2001.
  3. Aerni, M., Rando, J., Debenedetti, E., Carlini, N., Ippolito, D., and Tramèr, F. Measuring non-adversarial repro- duction of training data in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=590yfqz1LE.
  4. Alaa, A., Hartvigsen, T., Golchini, N., Dutta, S., Dean, F., Raji, I. D., and Zack, T. Position: Medical large language model benchmarks should prioritize construct validity. In Proceedings of the 42nd International Conference on Machine Learning, 2025. Forthcoming.
  5. Amdahl, G. M., Blaauw, G. A., and Brooks, F. P. Architec- ture of the IBM System/360. IBM Journal of Research and Development, 8(2), 1964.
  6. Berelson, B. Content Analysis in Communication Research. Free Press, 1952. ISBN 1953-07730-000.
  7. Blili-Hamelin, B. and Hancox-Li, L. Making intelligence: Ethical values in IQ and ML benchmarks. In Proceed- ings of the 2023 ACM Conference on Fairness, Ac- countability, and Transparency, pp. 271-284. Associ- ation for Computing Machinery, 2023. URL https: //doi.org/10.1145/3593013.3593996. · DOI
  8. Blodgett, S. L., Barocas, S., Daumé III, H., and Wallach, H. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguis- tics, pp. 5454-5476. Association for Computational Lin- guistics, 2020. doi: 10.18653/v1/2020.acl-main.485. URL https://aclanthology.org/2020.acl- main.485/. · DOI · Paper page
  9. Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., and Wal- lach, H. Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pp. 1004-1015. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.acl-long.81. URL https://aclanthology.org/2021.acl- long.81/. · DOI · Paper page
  10. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learn- ers. In Proceedings of the 33rd Conference on Neural Information Processing Systems, pp. 1877-1901, 2020.
  11. Cardwell, M. Dictionary of Psychology. Routledge, 1996.
  12. Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Er- lingsson, Ú., Oprea, A., and Raffel, C. Extracting training data from large language models. In 30th USENIX Secu- rity Symposium (USENIX Security 21), pp. 2633-2650. USENIX Association, 2021.
  13. Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., Balle, B., Ippolito, D., and Wallace, E. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5270, 2023a.
  14. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramèr, F., and Zhang, C. Quantifying memorization across neural language models. In International Conference on Learn- ing Representations, 2023b.
  15. Cooper, A. F. and Grimmelmann, J. The files are in the computer: Copyright, memorization, and generative AI. Chicago Kent Law Review, 2025. Forthcoming.
  16. Cooper, A. F., Abrams, E., and NA, N. Emergent unfairness in algorithmic fairness-accuracy trade-off research. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 46-54. Association for Comput- ing Machinery, 2021. doi: 10.1145/3461702.3462519. · DOI
  17. Cooper, A. F., Lee, K., Grimmelmann, J., Ippolito, D., Callison-Burch, C., Choquette-Choo, C. A., Mireshghal- lah, N., Brundage, M., Mimno, D., Choksi, M. Z., Balkin, J. M., Carlini, N., Sa, C. D., Frankle, J., Ganguli, D., Gip- son, B., Guadamuz, A., Harris, S. L., Jacobs, A. Z., Joh, E., Kamath, G., Lemley, M., Matthews, C., McLeavey, C., McSherry, C., Nasr, M., Ohm, P., Roberts, A., Rubin, T., Samuelson, P., Schubert, L., Vaccaro, K., Villa, L., Wu, F., and Zeide, E. Report of the 1st Workshop on Gen- erative AI and Law. arXiv preprint arXiv:2311.06477, 2023. · arXiv
  18. Cooper, A. F., Choquette-Choo, C. A., Bogen, M., Jagielski, M., Filippova, K., Liu, K. Z., Chouldechova, A., Hayes, J., Huang, Y., Mireshghallah, N., Shumailov, I., Triantafillou, E., Kairouz, P., Mitchell, N., Liang, P., Ho, D. E., Choi, Y., Koyejo, S., Delgado, F., Grimmelmann, J., Shmatikov, V., Sa, C. D., Barocas, S., Cyphert, A., Lemley, M., danah boyd, Vaughan, J. W., Brundage, M., Bau, D., Neel, S., Jacobs, A. Z., Terzis, A., Wallach, H., Papernot, N., and Lee, K. Machine unlearning doesn't do what you think: Lessons for generative AI policy, research, and practice. arXiv preprint arXiv:2412.06966, 2024. · arXiv
  19. Corvi, E., Washington, H., Reed, S., Atalla, C., Choulde- chova, A., Dow, P. A., Garcia-Gathright, J., Pangakis, N., Sheng, E., Vann, D., et al. Taxonomizing represen- tational harms using speech act theory. arXiv preprint arXiv:2504.00928, 2025. · arXiv
  20. Cronbach, L. J. and Meehl, P. E. Construct validity in psychological tests. Psychological bulletin, 52(4):281, 1955. doi: 10.1037/h0040957. · DOI · Paper page
  21. Eriksson, M., Purificato, E., Noroozian, A., Vinagre, J., Chaslot, G., Gomez, E., and Fernandez-Llorca, D. Can we trust AI benchmarks? An interdisciplinary review of current issues in AI evaluation. arXiv:2502.06559, 2025. · arXiv
  22. Goldfarb-Tarrant, S., Marchant, R., Muñoz Sánchez, R., Pandya, M., and Lopez, A. Intrinsic bias metrics do not correlate with application bias. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pp. 1926-1940. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.acl-long.150. URL https://aclanthology.org/2021.acl- long.150/. · DOI · Paper page
  23. Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. A survey on LLM-as-a- judge. arXiv preprint arXiv:2411.15594, 2024. · arXiv
  24. Halterman, A. and Keith, K. A. Codebook LLMs: Adapting political science codebooks for LLM use and adapting LLM to follow codebooks. arXiv preprint arXiv:2407.10747, 2024. · arXiv
  25. Hand, D. J. Meaurement Theory and Practice. Wiley, 2004.
  26. Hayes, J., Swanberg, M., Chaudhari, H., Yona, I., Shu- mailov, I., Nasr, M., Choquette-Choo, C. A., Lee, K., and Cooper, A. F. Measuring memorization in language models via probabilistic extraction. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 9266-9291. Association for Computational Linguistics, 2025. URL https://aclanthology.org/2025. naacl-long.469/.
  27. He, C., Luo, R., Bai, Y., Hu, S., Thai, Z., Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., Liu, J., Qi, L., Liu, Z., and Sun, M. OlympiadBench: A challenging bench- mark for promoting AGI with olympiad-level bilingual multimodal scientific problems, 2024. URL https: //aclanthology.org/2024.acl-long.211/.
  28. Huang, Y., Sun, L., Wang, H., Wu, S., Zhang, Q., Li, Y., Gao, C., Huang, Y., Lyu, W., Zhang, Y., Li, X., Sun, H., Liu, Z., Liu, Y., Wang, Y., Zhang, Z., Vid- gen, B., Kailkhura, B., Xiong, C., ..., and Zhao, Y. Po- sition: TrustLLM: Trustworthiness in large language models. In Proceedings of the 41st International Con- ference on Machine Learning, volume 235 of Proceed- ings of Machine Learning Research, pp. 20166-20270, 2024. URL https://proceedings.mlr.press/ v235/huang24x.html.
  29. Hutchinson, B., Rostamzadeh, N., Greer, C., Heller, K., and Prabhakaran, V. Evaluation gaps in machine learning practice. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 1859- 1876, 2022.
  30. Jacobs, A. Z. Measurement as governance in and for respon- sible AI. arXiv preprint arXiv:2109.05658, 2021. · arXiv
  31. Jacobs, A. Z. and Wallach, H. Measurement and fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 375-385, 2021.
  32. Katzman, J., Wang, A., Scheuerman, M., Blodgett, S. L., Laird, K., Wallach, H., and Barocas, S. Taxonomizing and measuring representational harms: A look at image tagging. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 37, pp. 14277-14285, 2023.
  33. Kowalski, R. Algorithm = Logic + Control. Communica- tions of the ACM, 22(7):424-436, 1979.
  34. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating train- ing data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics, volume 1, pp. 8424-8445, 2022.
  35. Lee, K., Cooper, A. F., and Grimmelmann, J. Talkin' 'bout AI generation: Copyright and the generative-AI supply chain. Journal of the Copyright Society, 2025. Forthcom- ing.
  36. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C. A., Manning, C. D., Re, C., Acosta-Navas, D., Hud- son, D. A., Zelikman, E., Durmus, E., Ladhak, F., Rong, F., Ren, H., Yao, H., WANG, J., Santhanam, K., Orr, L., Zheng, L., Yuksekgonul, M., Suzgun, M., Kim, N., Guha, N., Chatterji, N. S., Khattab, O., Henderson, P., Huang, Q., Chi, R. A., Xie, S. M., Santurkar, S., Gan- guli, S., Hashimoto, T., Icard, T., Zhang, T., Chaudhary, V., Wang, W., Li, X., Mai, Y., Zhang, Y., and Koreeda, Y. Holistic evaluation of language models. Transac- tions on Machine Learning Research, 2023. ISSN 2835- 8856. URL https://openreview.net/forum? id=iO4LZibEqW.
  37. Liu, Y. L., Blodgett, S. L., Cheung, J., Liao, Q. V., Olteanu, A., and Xiao, Z. ECBD: Evidence-centered benchmark design for NLP. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16349-16365. Association for Computational Linguistics, 2024. URL https:// aclanthology.org/2024.acl-long.861.
  38. Messick, S. Validity. ETS Research Report Series, 1987. doi: 10.1002/j.2330-8516.1987.tb00244.x. · DOI
  39. Messick, S. Validity and washback in language testing. Language Testing, 13(3):241-256, 1996.
  40. Mikhaylov, S., Laver, M., and Benoit, K. R. Coder relia- bility and misclassification in the human coding of party manifestos. Political analysis, 20(1):78-91, 2012.
  41. Morehouse, K. N., Swaroop, S., and Pan, W. Position: Rethinking LLM bias probing using lessons from the social sciences. Proceedings of the 42nd International Conference on Machine Learning, 2025. Forthcoming.
  42. Mulligan, D. K., Koopman, C., and Doty, N. Privacy is an essentially contested concept: A multi-dimensional analytic for mapping privacy. Philosophical Transac- tions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2083):20160118, 2016.
  43. Mulligan, D. K., Kroll, J. A., Kohli, N., and Wong, R. Y. This thing called fairness: Disciplinary confusion realiz- ing a value in technology. In Proceedings of the ACM on Human-Computer Interaction, volume 3, pp. 1-36. Association for Computing Machinery, 2019.
  44. Nadeem, M., Bethke, A., and Reddy, S. StereoSet: Mea- suring stereotypical bias in pretrained language mod- els. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pp. 5356- 5371. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.acl-long.416. URL https: //aclanthology.org/2021.acl-long.416/. · DOI · Paper page
  45. Nangia, N., Vania, C., Bhalerao, R., and Bowman, S. R. CrowS-Pairs: A challenge dataset for mea- suring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1953-1967. Association for Computational Lin- guistics, 2020. doi: 10.18653/v1/2020.emnlp-main. 154. URL https://aclanthology.org/2020. emnlp-main.154/. · DOI · Paper page
  46. Nasr, M., Rando, J., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., Choquette-Choo, C. A., Tramèr, F., and Lee, K. Scalable extraction of train- ing data from aligned, production language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=vjel3nWP2a. National Institute for Standards and Technology. Artificial intelligence Risk Management Framework: Generative Artificial Intelligence Profile, 2024. URL https:// doi.org/10.6028/NIST.AI.600-1. NIST Trust- worthy and Responsible AI NIST AI 600-1. · DOI · Paper page
  47. Nolte, H., Finck, M., and Meding, K. Machine learn- ers should acknowledge the legal implications of large language models as personal data. arXiv preprint arXiv:2503.01630, 2025. · arXiv
  48. Pangakis, N. and Wolken, S. Keeping humans in the loop: Human-centered automated annotation with generative AI. Proceedings of the International AAAI Conference on Web and Social Media, 2025. Forthcoming.
  49. Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3419-3448. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022. emnlp-main.225. URL https://aclanthology. org/2022.emnlp-main.225/. · DOI · Paper page
  50. Prashanth, U. S., Deng, A., O'Brien, K., V, J. S., Khan, M. A., Borkar, J., Choquette-Choo, C. A., Fuehne, J. R., Biderman, S., Ke, T., Lee, K., and Saphra, N. Recite, reconstruct, recollect: Memorization in LMs as a multifaceted phenomenon. In The Thirteenth In- ternational Conference on Learning Representations, 2025. URL https://openreview.net/forum? id=3E8YNv1HjU.
  51. Raji, D., Denton, E., Bender, E. M., Hanna, A., and Paullada, A. AI and the everything in the whole wide world benchmark. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL https://datasets- benchmarks-proceedings.neurips. cc/paper_files/paper/2021/file/ 084b6fbb10729ed4da8c3d3f5a3ae7c9- Paper-round2.pdf.
  52. Rauh, M., Marchal, N., Manzini, A., Hendricks, L. A., Co- manescu, R., Akbulut, C., Stepleton, T., Mateos-Garcia, J., Bergman, S., Kay, J., Griffin, C., Bariach, B., Gabriel, I., Rieser, V., Isaac, W., and Weidinger, L. Gaps in the safety evaluation of generative AI. AAAI/ACM Confer- ence on AI, Ethics, and Society (AIES), 2024.
  53. Salaudeen, O., Reuel, A., Ahmed, A., Bedi, S., Robertson, Z., Sundar, S., Domingue, B., Wang, A., and Koyejo, S. Measurement to meaning: A validity-centered frame- work for AI evaluation. arXiv preprint arXiv:2505.10573, 2025. · arXiv
  54. Saltzer, J. H., Reed, D. P., and Clark, D. D. End-to- end arguments in system design. ACM Transactions on Computer Systems (TOCS), 2(4):277-288, 1984. ISSN 0734-2071. URL https://doi.org/10.1145/ 357401.357402. · DOI
  55. Savoldi, B., Gaido, M., Bentivogli, L., Negri, M., and Turchi, M. Gender bias in machine transla- tion. Transactions of the Association for Computa- tional Linguistics, 9:845-874, 2021. doi: 10.1162/tacl a 00401. URL https://aclanthology.org/ 2021.tacl-1.51/. · DOI
  56. Somepalli, G., Singla, V., Goldblum, M., Geiping, J., and Goldstein, T. Diffusion art or digital forgery? Investigat- ing data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6048-6058, 2023.
  57. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agar- wal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., ..., and Wu, Z. Beyond the imita- tion game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learn- ing Research, 2023. ISSN 2835-8856. URL https: //openreview.net/forum?id=uyTL5Bvosj.
  58. van der Wal, O., Bachmann, D., Leidinger, A., van Maanen, L., Zuidema, W., and Schulz, K. Undesirable biases in NLP: Addressing challenges of measurement. Journal of Artificial Intelligence Research, 79:1-40, 2024. URL https://doi.org/10.1613/jair.1.15195. · DOI
  59. Wang, A., Barocas, S., Laird, K., and Wallach, H. Mea- suring representational harms in image captioning. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 324-335, 2022.
  60. Wang, B., Chen, W., Pei, H., Xie, C., Kang, M., Zhang, C., Xu, C., Xiong, Z., Dutta, R., Schaeffer, R., Truong, S., Arora, S., Mazeika, M., Hendrycks, D., Lin, Z., Cheng, Y., Koyejo, S., Song, D., and Li, B. DecodingTrust: A comprehensive assessment of trustworthiness in GPT models. In Proceed- ings of the 36th Conference on Neural Information Processing Systems, volume 36, pp. 31232-31339, 2023a. URL https://proceedings.neurips. cc/paper_files/paper/2023/file/ 63cb9921eecf51bfad27a99b2c53dd6d- Paper-Datasets_and_Benchmarks.pdf.
  61. Wang, X., Jiang, L., Hernandez-Orallo, J., Stillwell, D., Sun, L., Luo, F., and Xie, X. Evaluating general-purpose AI with psychometrics. arXiv preprint arXiv:2310.16379v2, 2023b. · arXiv
  62. Weidinger, L., Rauh, M., Marchal, N., Manzini, A., Hen- dricks, L. A., Mateos-Garcia, J., Bergman, S., Kay, J., Griffin, C., Bariach, B., Gabriel, I., Rieser, V., and Isaac, W. Sociotechnical safety evaluation of generative AI systems. arXiv preprint arXiv:2310.11986, 2023. · arXiv
  63. Yu, J., Lin, X., Yu, Z., and Xing, X. GPTFuzzer: Red team- ing large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253, 2023. · arXiv
  64. Zaller, J. The nature and origins of mass opinion. Cam- bridge University Press, 1992.
  65. Zhao, D., Andrews, J., Papakyriakopoulos, O., and Xiang, A. Position: Measure dataset diversity, don't just claim it. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 60644-60673, 2024. URL https://proceedings.mlr.press/ v235/zhao24a.html.
  66. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging LLM-as-a-judge with MT-Bench and ChatBot Arena. In Proceedings of the 36th Conference on Neural Informa- tion Processing Systems, pp. 46595-46623, 2023.

Cited by

Sorted by citations

Authors
Wallach, H., Desai, M., Cooper, A. F., Wang, A., Atalla, C., Barocas, S., Blodgett, S. L., Chouldechova, A., Corvi, E., Dow, P. A., Garcia-Gathright, J., Olteanu, A., Pangakis, N., Reed, S., Sheng, E., Vann, D., Vaughan, J. W., Vogel, M., Washington, H., & Jacobs, A. Z.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2502.00561
arXiv
2502.00561
OpenAlex
W4407123721
Cited by
1
References
0
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Microsoft Research
  • Instituto de Ciencias del Mar y Limnología, Universidad Nacional Autónoma de México
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
generative AI evaluation, measurement theory, social science measurement, measurement instruments, validity, conceptual framework, AI capabilities, AI impact, stakeholder engagement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review