CoChat
Sign Up
PreprintCited by 343Artificial Intelligence in Healthcare and EducationTopic ModelingMultimodal Machine Learning Applications

Can large language models reason about medical questions?

Open access
Liévin, Valentin, Hother, Christoffer Egeberg, Motzfeldt, Andreas Geert, Winther, Ole·2022·arXiv
DOI 10.48550/arxiv.2207.08143arXiv 2207.08143PMC PMC10935498
Open this paper in CoChat — free accountOpen PDF

Abstract

Although large language models (LLMs) often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge. We set out to investigate whether close- and open-source models (GPT-3.5, LLama-2, etc.) can be applied to answer and reason about difficult real-world-based questions. We focus on three popular medical benchmarks (MedQA-USMLE, MedMCQA, and PubMedQA) and multiple prompting scenarios: Chain-of-Thought (CoT, think step-by-step), few-shot and retrieval augmentation. Based on an expert annotation of the generated CoTs, we found that InstructGPT can often read, reason and recall expert knowledge. Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions, but also reaches the passing score on three datasets: MedQA-USMLE 60.2%, MedMCQA 62.7% and PubMedQA 78.2%. Open-source models are closing the gap: Llama-2 70B also passed the MedQA-USMLE with 62.5% accuracy.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2207.08143

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • CoT improvements are significantly limited in medical question answering.

    “For example, the results from Liévin et al. (2022) demonstrate that CoT improvements are significantly limited.”

    From MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2024) · cites this paper as Liévin et al., 2022

    The Discussion (p. 9) says single-sample CoT prompting was not competitive with direct prompting, supporting a limitation for that setting. But the Conclusion (p. 10) calls CoT a powerful tool that improved performance, and the Discussion (p. 10) reports gains from combining many CoTs. The unqualified claim that CoT improvements are significantly limited overstates the paper's finding. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The demand for extensive domain knowledge and advanced reasoning skills makes eliciting medical expertise via simple prompting challenging, preventing LLMs from effectively handling medical tasks.

    “(ii) The demand for extensive domain knowledge (Schmidt and Rikers, 2007) and advanced reasoning skills (Liévin et al., 2022) makes eliciting medical expertise via simple prompting challenging (Kung et al., 2023;Singhal et al., 2023a).”

    From MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2024) · cites this paper as Liévin et al., 2022

  • LLMs have shown the ability to perform accurate single-turn medical question-answering.

    “Recently, LLMs have shown the ability to encode clinical knowledge7,8, retrieve relevant medical texts9, and perform accurate single-turn medical question-answering3,10–12.”

    From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [10]

  • Zero-shot InstructGPT and Codex are capable of complex medical reasoning tasks but still need further improvement.

    “In professional domain reasoning tasks, zero-shot InstructGPT and Codex are capable of complex medical reasoning tasks, but still need to be further improved [117].”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [117]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 60

  1. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Madeleine Clare Elish, William Isaac, and Richard S. Zemel (eds.), FAccT '21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021 , pp. 610623. ACM, 2021. doi: 10.1145/3442188.3445922. URL https://doi.org/10.1145/ 3442188.3445922 . · DOI · Paper page
  2. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An Open-source Autoregressive Language Model. CoRR , abs/2204.06745, 2022. doi: 10.48550/ ARXIV.2204.06745. URL https://doi.org/10.48550/arXiv.2204.06745 . · DOI · Paper page
  3. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. CoRR , abs/2112.04426, 2021. URL https://arxiv.org/abs/2112.04426 . · arXiv
  4. T Brown, B Mann, N Ryder, and others. Language models are few-shot learners. Advances in neural information processing systems , 2020. ISSN 1049-5258.
  5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating Large Language Models Trained on Code. CoRR , abs/2107.03374, 2021. URL https://arxiv.org/abs/2107.03374 . · arXiv
  6. Zeming Chen, Alejandro Hernández-Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. CoRR , abs/2311.16079, 2023. doi: 10. 48550/ARXIV.2311.16079. URL https://doi.org/10.48550/arXiv.2311.16079 . · DOI · Paper page
  7. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling Language Modeling with Pathways. CoRR , abs/2204.02311, 2022. doi: 10.48550/ARXIV.2204.02311. URL https://doi.org/10.48550/arXiv.2204.02311 . · DOI · Paper page
  8. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling Instructionfinetuned Language Models. CoRR , abs/2210.11416, 2022. doi: 10.48550/ARXIV.2210.11416. URL https://doi.org/10.48550/arXiv.2210.11416 . · DOI · Paper page
  9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. CoRR , abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168 . · arXiv
  10. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. CoRR , abs/2305.14314, 2023. doi: 10.48550/ARXIV.2305.14314. URL https://doi.org/10.48550/arXiv.2305.14314 . · DOI · Paper page
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR , abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805 . · arXiv
  12. Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. Compositional Semantic Parsing with Large Language Models. CoRR , abs/2209.15003, 2022. doi: 10.48550/ARXIV.2209.15003. URL https: //doi.org/10.48550/arXiv.2209.15003 . · DOI
  13. William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. CoRR , abs/2101.03961, 2021. URL https://arxiv.org/abs/2101.03961 . · arXiv
  14. Aidan Gilson, Conrad W Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, and David Chartash. How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med Educ , 9:e45312, Feb 2023. ISSN 2369-3762. doi: 10.2196/45312. URL https://mededu.jmir.org/2023/1/e45312 . · DOI · Paper page
  15. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific Language Model Pretraining for Biomedical Natural Language Processing. CoRR , abs/2007.15779, 2020. URL https://arxiv.org/ abs/2007.15779 . · arXiv
  16. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. CoRR , abs/2009.03300, 2020. URL https://arxiv.org/abs/2009.03300 . · arXiv
  17. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training Compute-optimal Large Language Models. CoRR , abs/2203.15556, 2022. doi: 10. 48550/ARXIV.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556 . · DOI · Paper page
  18. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams. CoRR , abs/2009.13081, 2020. URL https://arxiv.org/abs/2009. 13081 . · arXiv
  19. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. PubMedQA: A Dataset for Biomedical Research Question Answering. CoRR , abs/1909.06146, 2019. URL http://arxiv.org/abs/1909.06146 . · arXiv
  20. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language Models (Mostly) Know What They Know. CoRR , abs/2207.05221, 2022. doi: 10.48550/ARXIV.2207.05221. URL https://doi.org/10.48550/arXiv.2207. 05221 . · DOI · Paper page
  21. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. CoRR , abs/2001.08361, 2020. URL https://arxiv.org/abs/2001.08361 . · arXiv
  22. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wentau Yih. Dense Passage Retrieval for Open-domain Question Answering. CoRR , abs/2004.04906, 2020. URL https://arxiv.org/abs/2004.04906 . · arXiv
  23. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-shot Reasoners. CoRR , abs/2205.11916, 2022. doi: 10.48550/ARXIV. 2205.11916. URL https://doi.org/10.48550/arXiv.2205.11916 . · DOI
  24. Diane M Korngiebel and Sean D Mooney. Considering the possibilities and pitfalls of Generative Pre-trained Transformer 3 (GPT-3) in healthcare delivery. NPJ digital medicine , 4(1):93, June 2021. ISSN 2398-6352. doi: 10.1038/s41746-021-00464-x. · DOI · Paper page
  25. Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, and Victor Tseng. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health , 2(2):1-12, 02 2023. doi: 10.1371/journal.pdig. 0000198. URL https://doi.org/10.1371/journal.pdig.0000198 . · DOI
  26. Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. Internetaugmented language models through few-shot prompting for open-domain question answering. CoRR , abs/2203.05115, 2022. doi: 10.48550/ARXIV.2203.05115. URL https://doi.org/ 10.48550/arXiv.2203.05115 . · DOI · Paper page
  27. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. On the Advance of Making Language Models Better Reasoners. CoRR , abs/2206.02336, 2022. doi: 10. 48550/ARXIV.2206.02336. URL https://doi.org/10.48550/arXiv.2206.02336 . · DOI
  28. Valentin Liévin, Andreas Geert Motzfeldt, Ida Riis Jensen, and Ole Winther. Variational Opendomain Question Answering. CoRR , abs/2210.06345, 2022. doi: 10.48550/ARXIV.2210.06345. URL https://doi.org/10.48550/arXiv.2210.06345 . · DOI
  29. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pretrain, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. CoRR , abs/2107.13586, 2021. URL https://arxiv.org/abs/2107.13586 . · arXiv
  30. Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining. CoRR , abs/2210.10341, 2022. doi: 10.48550/ARXIV.2210.10341. URL https://doi.org/10. 48550/arXiv.2210.10341 . · DOI
  31. Tomás Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. CoRR , abs/1310.4546, 2013. URL http://arxiv.org/abs/1310.4546 . · arXiv
  32. Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of GPT-4 on Medical Challenge Problems. CoRR , abs/2303.13375, 2023. doi: 10.48550/ARXIV. 2303.13375. URL https://doi.org/10.48550/arXiv.2303.13375 . · DOI
  33. Maxwell I. Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show Your Work: Scratchpads for Intermediate Computation with Language Models. CoRR , abs/2112.00114, 2021. URL https://arxiv.org/abs/2112.00114 . · arXiv
  34. Simon Ott, Konstantin Hebenstreit, Valentin Liévin, Christoffer Egeberg Hother, Milad Moradi, Maximilian Mayrhauser, Robert Praas, Ole Winther, and Matthias Samwald. ThoughtSource: A central hub for large language model reasoning data. CoRR , abs/2301.11596, 2023. doi: 10. 48550/ARXIV.2301.11596. URL https://doi.org/10.48550/arXiv.2301.11596 . · DOI · Paper page
  35. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. CoRR , abs/2203.02155, 2022. doi: 10.48550/ARXIV.2203.02155. URL https://doi.org/10.48550/arXiv.2203.02155 . · DOI · Paper page
  36. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. MedMCQA : A Largescale Multi-subject Multi-choice Dataset for Medical domain Question Answering. CoRR , abs/2203.14371, 2022. doi: 10.48550/ARXIV.2203.14371. URL https://doi.org/10. 48550/arXiv.2203.14371 . · DOI · Paper page
  37. Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-1162. · DOI · Paper page
  38. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. CoRR , abs/1802.05365, 2018. URL http://arxiv.org/abs/1802.05365 . · arXiv
  39. Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling Language Models: Methods, Analysis & Insights from Training Gopher. CoRR , abs/2112.11446, 2021. URL https://arxiv.org/abs/2112.11446 . · arXiv
  40. Joshua Robinson, Christopher Michael Rytting, and David Wingate. Leveraging Large Language Models for Multiple Choice Question Answering. CoRR , abs/2210.12353, 2022. doi: 10.48550/ ARXIV.2210.12353. URL https://doi.org/10.48550/arXiv.2210.12353 . · DOI
  41. Emre Sezgin, Joseph Sirrianni, and Simon L Linwood. Operationalizing and Implementing Pretrained, Large Artificial Intelligence Linguistic Models in the US Health Care System: Outlook of Generative Pretrained Transformer 3 (GPT-3) as a Service Model. JMIR medical informatics , 10(2):e32875, February 2022. ISSN 2291-9694. doi: 10.2196/32875. · DOI
  42. Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Schärli, Aakanksha Chowdhery, Philip Andrew Mansfield, Blaise Agüera y Arcas, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle K. Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large Language Models Encode Clinical Knowledge. CoRR , abs/2212.13138, 2022. doi: 10.48550/ARXIV.2212.13138. URL https://doi.org/10.48550/arXiv.2212.13138 . · DOI · Paper page
  43. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Andrew Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Agüera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle K. Barral, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan. Towards Expert-level Medical Question Answering with Large Language Models. CoRR , abs/2305.09617, 2023a. doi: 10.48550/ARXIV.2305. 09617. URL https://doi.org/10.48550/arXiv.2305.09617 . · DOI
  44. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Andrew Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Agüera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle K. Barral, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan. Towards Expert-level Medical Question Answering with Large Language Models. CoRR , abs/2305.09617, 2023b. doi: 10.48550/ARXIV.2305. 09617. URL https://doi.org/10.48550/arXiv.2305.09617 . · DOI
  45. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zheng, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using DeepSpeed and Megatron to Train Megatron-turing NLG 530B, A Large-scale Generative Language Model. CoRR , abs/2201.11990, 2022. URL https://arxiv.org/abs/2201.11990 . · arXiv
  46. Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, and et al. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. CoRR , abs/2206.04615, 2022. doi: 10.48550/ARXIV.2206.04615. URL https:// doi.org/10.48550/arXiv.2206.04615 . · DOI · Paper page
  47. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging BIGbench Tasks and Whether Chain-of-thought Can Solve Them. CoRR , abs/2210.09261, 2022. doi: 10.48550/ARXIV.2210.09261. URL https://doi.org/10.48550/arXiv.2210. 09261 . · DOI · Paper page
  48. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A Large Language Model for Science. CoRR , abs/2211.09085, 2022. doi: 10.48550/ARXIV.2211.09085. URL https:// doi.org/10.48550/arXiv.2211.09085 . · DOI · Paper page
  49. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. MeierHellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Agüera y Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoc Le. LaMDA: Language Models for Dialog Applications. CoRR , abs/2201.08239, 2022. URL https://arxiv.org/abs/2201.08239 . · arXiv
  50. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Finetuned Chat Models. CoRR , abs/2307.09288, 2023. doi: 10.48550/ARXIV.2307.09288. URL https://doi.org/10.48550/arXiv.2307.09288 . · DOI · Paper page
  51. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. CoRR , abs/1706.03762, 2017. URL http://arxiv.org/abs/1706.03762 . · arXiv
  52. Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial GLUE: A Multi-task Benchmark for Robustness Evaluation of Language Models. CoRR , abs/2111.02840, 2021. URL https://arxiv.org/abs/2111. 02840 . · arXiv
  53. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, and Denny Zhou. Selfconsistency Improves Chain of Thought Reasoning in Language Models. CoRR , abs/2203.11171, 2022. doi: 10.48550/ARXIV.2203.11171. URL https://doi.org/10.48550/arXiv. 2203.11171 . · DOI · Paper page
  54. Yubo Wang, Xueguang Ma, and Wenhu Chen. Augmenting Black-box LLMs with Medical Textbooks for Clinical Question Answering. CoRR , abs/2309.02233, 2023. doi: 10.48550/ARXIV. 2309.02233. URL https://doi.org/10.48550/arXiv.2309.02233 . · DOI
  55. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. Chain of Thought Prompting Elicits Reasoning in Large Language Models. CoRR , abs/2201.11903, 2022. URL https://arxiv.org/abs/2201.11903 . · arXiv
  56. Michihiro Yasunaga, Jure Leskovec, and Percy Liang. LinkBERT: Pretraining Language Models with Document Links. CoRR , abs/2203.15827, 2022. doi: 10.48550/ARXIV.2203.15827. URL https://doi.org/10.48550/arXiv.2203.15827 . · DOI
  57. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open Pre-trained Transformer Language Models. CoRR , abs/2205.01068, 2022. doi: 10.48550/ARXIV.2205.01068. URL https://doi.org/10.48550/arXiv.2205. 01068 . · DOI · Paper page
  58. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. CoRR , abs/2306.05685, 2023. doi: 10.48550/ARXIV.2306.05685. URL https://doi.org/10.48550/arXiv.2306. 05685 . · DOI · Paper page
  59. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less Is More for Alignment. CoRR , abs/2305.11206, 2023. doi: 10.48550/ARXIV.2305. 11206. URL https://doi.org/10.48550/arXiv.2305.11206 . · DOI
  60. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed H. Chi. Least-to-most Prompting Enables Complex Reasoning in Large Language Models. CoRR , abs/2205.10625, 2022. doi: 10.48550/ARXIV. 2205.10625. URL https://doi.org/10.48550/arXiv.2205.10625 . · DOI

Cited by

Sorted by citations

Related

Authors
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O.
Affiliations
  • Technical University of Denmark (DK)
  • Copenhagen University Hospital (DK)
  • University of Copenhagen (DK)
Corresponding author
Liévin, Valentin
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Volume
5
Issue
3
Pages
100943-100943
ISSN
2666-3899 (ISSN-L)
DOI
10.48550/arxiv.2207.08143
arXiv
2207.08143
PMC
PMC10935498
OpenAlex
W4392359953
Cited by
343
References
0
Citations by year
2026: 95 · 2025: 108 · 2024: 79 · 2023: 60 · 2022: 1
FWCI
26.36
Type
journal article
Language
en
Open access
Licence
Locations
  • Patterns (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
  • European High Performance Computing Joint Undertaking
  • DeepMind
  • Danmarks Grundforskningsfond
  • Novo Nordisk
  • Novo Nordisk Fonden (NNF20OC0062606)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, medical question answering, medical reasoning, chain-of-thought prompting, retrieval augmentation, few-shot prompting, prompt engineering, open-source language models, Llama 2, GPT-3.5, MedMCQA, PubMedQA

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review