Cited by 49Topic ModelingNatural Language Processing TechniquesInformation Retrieval and Search Behavior

Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models

Open access
Shao, Yijia, Jiang, Yucheng, Kanell, Theodore, Xu, Peter, Khattab, Omar, Lam, Monica·2024·Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)

Abstract

Yijia Shao, Yucheng Jiang, Theodore Kanell, Peter Xu, Omar Khattab, Monica Lam. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.naacl-long.347

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing approaches using fixed checklists or static rubrics struggle to adapt to diverse tasks, specialized domains, and nuanced quality aspects of deep research tasks.

    “Existing approaches using fixed checklists [23] or static rubrics [25, 1] struggle to adapt to diverse tasks, specialized domains, and nuanced quality aspects of deep research tasks.”

    From DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents (2025) · cites this paper as [25]

    Section 4.2 (page unavailable) describes a five-point rubric for evaluating Wikipedia-like articles, so the paper supports the reference to a rubric-based approach. Neither that section nor Appendix C.2 says such rubrics struggle to adapt to diverse tasks, specialized domains, or nuanced quality aspects of deep research tasks. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 35

  1. Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W Bruce Croft. 2019. Asking clari- fying questions in open-domain information-seeking conversations. In Proceedings of the 42nd interna- tional acm sigir conference on research and develop- ment in information retrieval, pages 475-484.
  2. Nishant Balepur, Jie Huang, and Kevin Chang. 2023. Expository text generation: Imitate, retrieve, para- phrase. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 11896-11919, Singapore. Association for Computational Linguistics.
  3. Wayne C Booth, Gregory G Colomb, and Joseph M Williams. 2003. The craft of research. University of Chicago press.
  4. Laura Dietz and John Foley. 2019. Trec car y3: Com- plex answer retrieval overview. In Proceedings of Text REtrieval Conference (TREC).
  5. Ann-Marie Eriksson and Åsa Mäkitalo. 2015. Supervi- sion at the outline stage: Introducing and encounter- ing issues of sustainable development through aca- demic writing assignments. Text & Talk, 35(2):123- 153.
  6. Xiaocheng Feng, Ming Liu, Jiahao Liu, Bing Qin, Yibo Sun, and Ting Liu. 2018. Topic-to-essay generation with neural networks. In IJCAI, pages 4078-4084.
  7. Tira Nur Fitria. 2023. Artificial intelligence (ai) tech- nology in openai chatgpt application: A review of chatgpt in writing english essay. In ELT Forum: Jour- nal of English Language Teaching, volume 12, pages 44-58.
  8. Pasi Fränti and Radu Mariescu-Istodor. 2023. Soft preci- sion and recall. Pattern Recognition Letters, 167:115- 121.
  9. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 6465-6488, Singapore. Associa- tion for Computational Linguistics.
  10. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1-43.
  11. Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. 2023a. Mistral 7b. arXiv preprint arXiv:2310.06825. · arXiv
  12. Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023b. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pages 7969-7992, Singapore. As- sociation for Computational Linguistics.
  13. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In In- ternational Conference on Machine Learning, pages 15696-15707.
  14. PMLR.
  15. Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-search- predict: Composing retrieval and language mod- els for knowledge-intensive NLP. arXiv preprint arXiv:2212.14024. · arXiv
  16. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Za- haria, and Christopher Potts. 2023. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714. · arXiv
  17. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. Prometheus: Inducing fine-grained evalua- tion capability in language models. arXiv preprint arXiv:2310.08491. · arXiv
  18. Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
  19. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summariz- ing long sequences. In International Conference on Learning Representations.
  20. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettle- moyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076-12100, Singa- pore. Association for Computational Linguistics.
  21. Julià Minguillón, Maura Lerga, Eduard Aibar, Josep Lladós-Masllorens, and Antoni Meseguer-Artola. 2017. Semi-automatic generation of a corpus of wikipedia articles on science and technology. Profe- sional de la Información, 26(5):995-1005.
  22. Rosa Munoz-Luna. 2015. Main ingredients for suc- cess in l2 academic writing: Outlining, drafting and proofreading. PloS one, 10(6):e0128309.
  23. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser- assisted question-answering with human feedback.
  24. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
  25. Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gul- wani. 2022. Synchromesh: Reliable code generation from pre-trained language models. In International Conference on Learning Representations.
  26. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language mod- els. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687-5711, Singa- pore. Association for Computational Linguistics.
  27. Ashwin Ram. 1991. A theory of questions and question asking. Journal of the Learning Sciences, 1(3-4):273- 318.
  28. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented lan- guage models. Transactions of the Association for Computational Linguistics.
  29. D Gordon Rohman. 1965. Pre-writing the stage of dis- covery in the writing process. College composition and communication, 16(2):106-112.
  30. Christina Sauper and Regina Barzilay. 2009. Auto- matically generating Wikipedia articles: A structure- aware approach. In Proceedings of the Joint Con- ference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 208-216, Suntec, Singapore. Association for Computational Linguistics.
  31. Sina Semnani, Violet Yao, Heidi Zhang, and Monica Lam. 2023. WikiChat: Stopping the hallucination of large language model chatbots by few-shot ground- ing on Wikipedia. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2387-2413, Singapore. Association for Computa- tional Linguistics.
  32. Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. 2022. Nearest neighbor zero-shot inference. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3254-3265, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. · DOI
  33. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.
  34. Cyril Zakka, Akash Chaurasia, Rohan Shad, Alex R Dalal, Jennifer L Kim, Michael Moor, Kevin Alexan- der, Euan Ashley, Jack Boyd, Kathleen Boyd, et al. 2023. Almanac: Retrieval-augmented language mod- els for clinical medicine. Research Square.
  35. Shuyan Zhou, Uri Alon, Frank F. Xu, Zhengbao Jiang, and Graham Neubig. 2023. Docprompting: Gener- ating code by retrieving the docs. In The Eleventh International Conference on Learning Representa- tions.

Cited by

Sorted by citations

Authors
Shao, Y., Jiang, Y., Kanell, T., Xu, P., Khattab, O., & Lam, M.
Affiliations
Stanford University (US)
Venue
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Published
2024
Pages
6252-6278
OpenAlex
W4401042999
Cited by
41
References
0
Citations by year
2026: 15 · 2025: 22 · 2024: 4
FWCI
12.98
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
large language models, knowledge distillation, text summarization, information retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review