The DOI matches its Crossref record; 69 of the 69 references it cites were found in a registry or an index.
MetaWriter: Exploring the Potential and Perils of AI Writing Support in Scientific Peer Review
Abstract
Recent advances in Large Language Models (LLMs) show the potential to significantly augment or even replace complex human writing activities. However, for complex tasks where people need to make decisions as well as write a justification, the trade offs between making work efficient and hindering decisions remain unclear. In this paper, we explore this question in the context of designing intelligent scaffolding for writing meta-reviews for an academic peer review process. We prototyped a system called "MetaWriter'' trained on five years of open peer review data to support meta-reviewing. The system highlights common topics in the original peer reviews, extracts key points by each reviewer, and on request, provides a preliminary draft of a meta-review that can be further edited. To understand how novice and experienced meta-reviewers use MetaWriter, we conducted a within-subject study with 32 participants. Each participant wrote meta-reviews for two papers: one with and one without MetaWriter. We found that MetaWriter significantly expedited the authoring process and improved the coverage of meta-reviews, as rated by experts, compared to the baseline. While participants recognized the efficiency benefits, they raised concerns around trust, over-reliance, and agency. We also interviewed six paper authors to understand their opinions of using machine intelligence to support the peer review process and reported critical reflections. We discuss implications for future interactive AI writing tools to support complex synthesis work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“AI systems pose ethical issues including generating inaccuracies [165], violating academic integrity [107], amplifying dominant views [148], and fostering human biases like confirmation bias and over-reliance [107].”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [165]
In the citing paper
“Much of the current literature focuses on isolated AI-generated reviews or single-reviewer assistance (Chen et al. 2025;Sun et al. 2024;Farber 2025).”
From Who Reviews the Reviewer? A Multi‐Agent LLM Architecture With Meta‐Review Synthesis for Editorial Peer Review (2026) · cites this paper as Sun et al. 2024
In the citing paper
“Similarly, MetaWriter demonstrated that AI support can accelerate meta-review synthesis and improve the coverage of reviewer feedback, particularly for less experienced reviewers (Sun et al. 2024).”
From Who Reviews the Reviewer? A Multi‐Agent LLM Architecture With Meta‐Review Synthesis for Editorial Peer Review (2026) · cites this paper as Sun et al. 2024
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“Prior work has raised concerns about the potential loss of agency or control when working with AI on complex tasks [6, 33, 75].”
BackgroundThe use of MMR, diversity-based reranking for reordering documents and producing summaries
Carbonell, J., & Goldstein, J. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.291025 · Cited by 2,329
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
The Corsini encyclopedia of psychology
(2010) · Choice Reviews Online · doi:10.5860/choice.47-6008 · Cited by 1,082
Scaffolding Complex Learning: The Mechanisms of Structuring and Problematizing Student Work
Reiser, B. J. (2004) · Journal of the Learning Sciences · doi:10.1207/s15327809jls1303_2 · Cited by 1,054
Chapter 3: Rethinking Transfer: A Simple Proposal With Multiple Implications
BRANSFORD, J. D., & SCHWARTZ, D. L. (1999) · Review of Research in Education · doi:10.3102/0091732x024001061 · Cited by 881 · cited 1×
“…the system not only provides an example from some other context (which requires users to conceptually “transfer” insights into the current task [13, 25]), MetaWriter also generates a highly contextual example that uses language from the current context, potentially reducing the burden of…”
Peer Review: A Flawed Process at the Heart of Science and Journals
Smith, R. (2006) · Journal of the Royal Society of Medicine · doi:10.1177/014107680609900414 · Cited by 880
Learning and transfer: A general role for analogical encoding.
Gentner, D., Loewenstein, J., & Thompson, L. (2003) · Journal of Educational Psychology · doi:10.1037/0022-0663.95.2.393 · Cited by 865
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 1×
“This concern around over-reliance also appears in other domains of AI-assisted decision-making [8, 15].”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Lin, C.-Y., & Och, F. J. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1219032 · Cited by 737
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 1×
“Previous studies also raise concerns that LLMs could exhibit biases in human-AI collaboration tasks [15, 32, 44, 65, 78, 85].”
BackgroundWhat can natural language processing do for clinical decision support?
Demner-Fushman, D., Chapman, W. W., & McDonald, C. J. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.08.007 · Cited by 688
Wrangler: interactive visual specification of data transformation scripts
Kandel, S., Paepcke, A., Hellerstein, J., & Heer, J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979444 · Cited by 581
Voyager: Exploratory Analysis via Faceted Browsing of Visualization Recommendations
Wongsuphasawat, K., Moritz, D., Anand, A., Mackinlay, J., Howe, B., & Heer, J. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2015.2467191 · Cited by 507
Wu, T., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517582 · Cited by 483
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477 · cited 1×
“Researchers pointed out automation of creative work may result in “cannibalizing” the creativity of artists engaged in script writing tasks[59, 86].”
BackgroundThree-Point Likert Scales Are Good Enough
Jacoby, J., & Matell, M. S. (1971) · Journal of Marketing Research · doi:10.1177/002224377100800414 · Cited by 455
Texygen: A Benchmarking Platform for Text Generation Models
Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., & Yu, Y. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210080 · Cited by 408
Effects of Editorial Peer Review: A Systematic Review
Jefferson, T., Alderson, P., Wager, E., & Davidoff, F. (2002) · JAMA · doi:10.1001/jama.287.21.2784 · Cited by 347
Lee, M., Liang, P., & Yang, Q. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502030 · Cited by 345 · cited 2×
“CoAuthor further explored the capabilities of LLM working as a writing collaborator and reasoned about its ability to generate fluent text, generate new ideas and work jointly with writers [50].”
BackgroundCollins, A. (2005) · Cambridge University Press eBooks · doi:10.1017/cbo9780511816833.005 · Cited by 343
Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342 · cited 2×
“Understanding the tension between human agency and machine automation can help researchers design systems that successfully weave AI scaffolding into the task without sacrificing a user’s agency [31, 83].”
BackgroundPlan-and-Write: Towards Better Automatic Storytelling
Yao, L., Peng, N., Weischedel, R., Knight, K., Zhao, D., & Yan, R. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017378 · Cited by 308
Agency plus automation: Designing artificial intelligence into interactive systems
Heer, J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1807184115 · Cited by 279 · cited 3×
“For systems where humans collaborate with an AI system, researchers have noted that the efficiency gains from automation might also limit the agency and creativity of human users [31].”
BackgroundCo-Writing with Opinionated Language Models Affects Users’ Views
Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581196 · Cited by 265 · cited 2×
“6.2 Understanding how AI systems impact trust for individuals versus communities of use Previous research showed that writing with an opinionated language model can affect participants’ attitudes on social topics [39].”
BackgroundInfluential"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations
Lakkaraju, H., & Bastani, O. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375833 · Cited by 241
Value-Sensitive Algorithm Design: Method, Case Study, and Lessons
Zhu, H., Yu, B., Halfaker, A., & Terveen, L. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274463 · Cited by 220
Identifying Appropriate Support for Propositions in Online User Comments
Park, J., & Cardie, C. (2014) · Workshop on Argument Mining (ArgMining) · doi:10.3115/v1/w14-2105 · Cited by 200
Habernal, I., & Gurevych, I. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1150 · Cited by 190
A Conceptual Framework for Human–AI Hybrid Adaptivity in Education
Holstein, K., Aleven, V., & Rummel, N. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-52237-7_20 · Cited by 168 · cited 1×
“Previous studies also raise concerns that LLMs could exhibit biases in human-AI collaboration tasks [15, 32, 44, 65, 78, 85].”
BackgroundModeling Argument Strength in Student Essays
Persing, I., & Ng, V. (2015) · doi:10.3115/v1/p15-1053 · Cited by 161
Computational support for academic peer review
Price, S., & Flach, P. A. (2017) · Communications of the ACM · doi:10.1145/2979672 · Cited by 149
Metaphoria: An Algorithmic Companion for Metaphor Creation
Gero, K. I., & Chilton, L. B. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300526 · Cited by 146
Yuan, A., Luther, K., Krause, M., Vennix, S. I., Dow, S. P., & Hartmann, B. (2016) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2818048.2819953 · Cited by 132
Self-citationDrozdal, J., Weisz, J., Wang, D., Dass, G., Yao, B., Zhao, C., Muller, M., Ju, L., & Su, H. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377501 · Cited by 119
ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis
Wang, Q., Zeng, Q., Huang, L., Knight, K., Ji, H., & Rajani, N. F. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.44 · Cited by 113
The affordance of anchored discussion for the collaborative processing of academic texts
van der Pol, J., Admiraal, W., & Simons, P. R. J. (2006) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-006-9657-6 · Cited by 110
Stapleton, L., Lee, M. H., Qing, D., Wright, M., Chouldechova, A., Holstein, K., Wu, Z. S., & Zhu, H. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533177 · Cited by 110
Can We Automate Scientific Reviewing?
Yuan, W., Liu, P., & Neubig, G. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12862 · Cited by 96
Argument Mining for Understanding Peer Reviews
Hua, X., Nikolov, M., Badugu, N., & Wang, L. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1219 · Cited by 86 · cited 2×
“Previous research has used computational methods to provide support to streamline several parts of the peer review process, such as matching submissions with appropriate peer reviewers, authoring more comprehensive and decisive reviews, as well as review quality assessment [7, 9, 35, 73, 84, 96].”
BackgroundMethodIntroAssist: A Tool to Support Writing Introductory Help Requests
Hui, J. S., Gergle, D., & Gerber, E. M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173596 · Cited by 76
Weidele, D. K. I., Weisz, J. D., Oduor, E., Muller, M. J., Andrés, J., Gray, A., & Wang, D. (2020) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3377325.3377538 · Cited by 74
Kawakami, A., Sivaraman, V., Stapleton, L., Cheng, H.-F., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533556 · Cited by 74
Task-Aware Representation of Sentences for Generic Text Classification
Halder, K., Akbik, A., Krapac, J., & Vollgraf, R. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.285 · Cited by 65
Argumentation Quality Assessment: Theory vs. Practice
Wachsmuth, H., Naderi, N., Habernal, I., Hou, Y., Hirst, G., Gurevych, I., & Stein, B. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-2039 · Cited by 63
Kim, J., Suh, S., Chilton, L. B., & Xia, H. (2023) · Proceedings of the 2023 ACM Designing Interactive Systems Conference · doi:10.1145/3563657.3595996 · Cited by 61 · cited 2×
“…AI have the potential to enhance scaffolding for complex tasks like writing; as shown in recent research, text editors can now generate new ideas [27], create metaphors [26, 46] or story lines [92–94], summarize long text and rewrite sentences [3] or extend a seed of story into longer writing [4].”
BackgroundAPE: Argument Pair Extraction from Peer Review and Rebuttal via Multi-task Learning
Cheng, L., Bing, L., Yu, Q., Lu, W., & Si, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.569 · Cited by 58
ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing
Sun, L., Chan, A., Chang, Y. S., & Dow, S. P. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645159 · Cited by 50
Self-citationThe arbitrariness of reviews, and advice for school administrators
Langford, J., & Guzdial, M. (2015) · Communications of the ACM · doi:10.1145/2732417 · Cited by 49
Bhat, A., Agashe, S., Oberoi, P., Mohile, N., Jangir, R., & Joshi, A. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584060 · Cited by 47
Boys’ gaming identities and opportunities for learning
Scholes, L., Mills, K. A., & Wallace, E. (2021) · Learning Media and Technology · doi:10.1080/17439884.2021.1936017 · Cited by 35
Norkutė, M., Herger, N., Michalak, L., Mulder, A., & Gao, S. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3443441 · Cited by 33
Bartoli, A., Lorenzo, A. D., Medvet, E., & Tarlao, F. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-45507-5_2 · Cited by 31
Lettersmith: Scaffolding Written Professional Communication Among College Students
Hui, J. S., & Sprouse, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581029 · Cited by 27
MReD: A Meta-Review Dataset for Structure-Controllable Text Generation
Shen, C., Cheng, L., Zhou, R., Bing, L., You, Y., & Si, L. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.198 · Cited by 23
Arous, I., Yang, J., Khayati, M., & Cudré-Mauroux, P. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3450088 · Cited by 18
Teaching Workplace Genre Ecologies and Pedagogical Goals Through Résumés and Cover Letters
Doan, S. C. (2021) · Business and Professional Communication Quarterly · doi:10.1177/23294906211031810 · Cited by 17
DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions
Kennard, N., O’Gorman, T., Das, R., Sharma, A., Bagchi, C., Clinton, M., Yelugam, P. K., Zamani, H., & McCallum, A. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.89 · Cited by 17
DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions
Kennard, N., O'Gorman, T., Das, R., Sharma, A., Bagchi, C., Clinton, M., Yelugam, P. K., Zamani, H., & McCallum, A. (2021) · arXiv · doi:10.48550/arxiv.2110.08520 · Cited by 17
A Deep Neural Architecture for Decision-Aware Meta-Review Generation
Kumar, A., Ghosal, T., & Ekbal, A. (2021) · 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL) · doi:10.1109/jcdl52503.2021.00064 · Cited by 15
MetaGen: An academic Meta-review Generation system
Bhatia, C., Pradhan, T., & Pal, S. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401190 · Cited by 13
Rahman, S., Siangliulue, P., & Marcus, A. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415240 · Cited by 13
Did AI get more negative recently?
Beese, D., Altunbaş, B., Güzeler, G., & Eger, S. (2023) · Royal Society Open Science · doi:10.1098/rsos.221159 · Cited by 10
Mapping the Design Space of Human-AI Interaction in Text Summarization
Cheng, R., Smith-Renner, A., Zhang, K., Tetreault, J. R., & Jaimes, A. (2022) · arXiv · doi:10.48550/arxiv.2206.14863 · Cited by 4
Research works printed but not linked: 27
- Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77--101.
- Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao. 2019. Does my rebuttal matter? insights from a major nlp conference. arXiv preprint arXiv:1903.11367 (2019). · arXiv
- Katy Ilonka Gero, Vivian Liu, and Lydia B. Chilton. 2021. Sparks: Inspiration for Science Writing using Language Models. arXiv. http://arxiv.org/abs/2110.07640 arXiv:2110.07640 [cs]. · arXiv
- Navita Goyal, Eleftheria Briakou, Amanda Liu, Connor Baumler, Claire Bonial, Jeffrey Micher, Clare R Voss, Marine Carpuat, and Hal Daumé III. 2023. What Else Do I Need to Know? The Effect of Background Information on Users' Reliance on AI Systems. arXiv preprint arXiv:2305.14331 (2023). · arXiv
- Chao-Chun Hsu and Chenhao Tan. 2021. Decision-Focused Summarization. arXiv preprint arXiv:2109.06896 (2021). · arXiv
- Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022. Survey of hallucination in natural language generation. Comput. Surveys (2022).
- Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine Van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (peerread): Collection, insights and nlp applications. arXiv preprint arXiv:1804.09635 (2018). · arXiv
- Miao Li Jianzhong Qi and Jey Han Lau. 2022. PeerSum: A Peer Review Dataset for Abstractive Multi-document Summarization. http://arxiv.org/abs/2203.01769 arXiv:2203.01769 [cs]. · arXiv
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74--81.
- Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 (2021). · arXiv
- Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345 (2019). · arXiv
- Alison McCook. 2006. Is peer review broken? Submissions are up reviewers are overtaxed and authors are lodging complaint after complaint about the process at top-tier journals. What's wrong with peer review? The scientist 20 2 (2006) 26--35.
- Piotr Mirowski Kory W. Mathewson Jaylen Pittman and Richard Evans. 2022. Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals. arXiv. http://arxiv.org/abs/2209.14958 arXiv:2209.14958 [cs]. · arXiv
- Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038 (2019). · arXiv
- Andrea Papenmeier, Gwenn Englebienne, and Christin Seifert. 2019. How model accuracy and explanation fidelity influence user trust. arXiv preprint arXiv:1907.12652 (2019). · arXiv
- Napol Rachatasumrit, Gonzalo Ramos, Jina Suh, Rachel Ng, and Christopher Meek. 2021. ForSense: Accelerating Online Research Through Sensemaking Integration and Machine Research Support. In 26th International Conference on Intelligent User Interfaces. 608--618.
- Nihar B Shah. 2022. An overview of challenges, experiments, and computational solutions in peer review (extended version). Commun. ACM (2022).
- Nihar B Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike Von Luxburg. 2018. Design and analysis of the NIPS 2016 review process. Journal of machine learning research (2018).
- Ben Shneiderman and Pattie Maes. 1997. Direct manipulation vs. interface agents. interactions 4, 6 (1997), 42--61.
- Raimel Sobrino-Duque, Juan Manuel Carrillo-de Gea, Juan José López-Jiménez, Joaquín Nicolás Ros, and José Luis Fernández-Alemán. 2022. Usevalia: Managing Inspection-Based Usability Audits. International Journal of Human-- Computer Interaction (2022), 1--25.
- Ivan Stelmakh Nihar B Shah and Aarti Singh. 2019. PeerReview4All: Fair and accurate reviewer assignment in peer review. In Algorithmic Learning Theory. PMLR 828--856.
- T Elizabeth Workman, Marcelo Fiszman, and John F Hurdle. 2012. Text summarization as a decision support aid. BMC medical informatics and decision making 12, 1 (2012), 1--12.
- Wenting Xiong and Diane Litman. 2011. Automatically Predicting Peer-Review Helpfulness. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Portland, Oregon, USA, 502--507. https://aclanthology.org/P11--2088
- Jiacheng Xu and Greg Durrett. 2019. Neural extractive text summarization with syntactic compression. arXiv preprint arXiv:1902.00863 (2019). · arXiv
- Kevin Yang, Dan Klein, Nanyun Peng, and Yuandong Tian. 2022. Doc: Improving long story coherence with detailed outline control. arXiv preprint arXiv:2212.10077 (2022). · arXiv
- Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: Story Writing With Large Language Models. In 27th International Conference on Intelligent User Interfaces. 841--852.
- Zheng Zhang, Jie Gao, Ranjodh Singh Dhaliwal, and Toby Jia-Jun Li. 2023. VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping. arXiv preprint arXiv:2304.07810 (2023). · arXiv
Cited by
'I'm Categorizing LLM as a Productivity Tool': Examining Ethics of LLM Use in HCI Research Practices
Kapania, S., Wang, R., Li, T. J.-J., Li, T., & Shen, H. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711000 · Cited by 79
ImaginateAR: AI-Assisted In-Situ Authoring in Augmented Reality
Lee, J., Aleotti, F., Mazala, D., Garcia-Hernando, G., Vicente, S., Johnston, O., Kraus-Liang, I., Powierza, J., Shin, D.-H., Froehlich, J. E., et al. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747635 · Cited by 51
ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing
Sun, L., Chan, A., Chang, Y. S., & Dow, S. P. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645159 · Cited by 50
Self-citationBlockchain and Its Application in the Peer Review of Scientific Works: A Systematic Review
Morales-Alarcón, C. H., Poveda, E. M. B., Yánez, H. M. V., & Guisñan, P. A. B. (2024) · Publications · doi:10.3390/publications12040040 · Cited by 42
Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process
Reza, M., Thomas-Mitchell, J., Dushniku, P., Laundry, N., Williams, J. J., & Kuzminykh, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757566 · Cited by 37
“…processes and contexts–for example, strong representation of Generating and Translating activities in Creative and Academic settings, and limited focus on Monitoring across all contexts. process across them; we also report how many papers 1 were coded into each writing context, as shown in Table 1.”
Metadata
- Authors
- Sun, L., Tao, S., Hu, J., & Dow, S. P.
- Affiliations
- University of California San Diego (US)
- University of Wisconsin–Madison (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2024-04-17
- Volume
- 8
- Issue
- CSCW1
- Pages
- 1-32
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4396232287
- Semantic Scholar
- 1e13c7a04a77a65ef1bae0748360c582a83bed91
- Cited by
- 37
- References
- 60
- Citations by year
- 2026: 12 · 2025: 19 · 2024: 6
- FWCI
- 55.04
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Topics
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- large language models, open peer review, scientific peer review, trust in AI, overreliance on AI, writing efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.