Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 192

The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.

Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques, and Tools

Mayer, Ruben, Jacobsen, Hans-Arno · 2021 · ACM Computing Surveys

Abstract

Deep Learning (DL) has had an immense success in the recent past, leading to state-of-the-art results in various domains, such as image recognition and natural language processing. One of the reasons for this success is the increasing size of DL models and the proliferation of vast amounts of training data being available. To keep on improving the performance of DL, increasing the scalability of DL systems is necessary. In this survey, we perform a broad and thorough investigation on challenges, techniques and tools for scalable DL on distributed infrastructures. This incorporates infrastructures for DL, methods for parallel DL training, multi-tenant resource scheduling, and the management of training and model data. Further, we analyze and compare 11 current open-source DL frameworks and tools and investigate which of the techniques are commonly implemented in practice. Finally, we highlight future research trends in DL systems that deserve further research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3363554

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 162

  1. Abadi Martín, 2016, Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI’16)
  2. 10.1145/2976749.2978318 · DOI · Paper page
  3. 10.5555/2627435.2638571 · doi:10.5555/2627435.2638571
  4. Akiba Takuya, 2017, Proceedings of the Workshop on ML Systems in the 31st Annual Conference on Neural Information Processing Systems (NIPS’17)
  5. Alistarh Dan, 2017, QSGD: Communication-efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems 30. Curran Associates
  6. 10.1145/1721654.1721672 · DOI · Paper page
  7. 10.1109/TPDS.2017.2752706 · DOI
  8. Baker Bowen, 2017, Proceedings of the International Conference on Learning Representations.
  9. Barroso Luiz Andre, The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines
  10. Tal Ben-Nun and Torsten Hoefler. 2018. Demystifying parallel and distributed deep learning: An in-depth concurrency analysis. Retrieved from http://arxiv.org/abs/1802.09941. Tal Ben-Nun and Torsten Hoefler. 2018. Demystifying parallel and distributed deep learning: An in-depth concurrency analysis. Retrieved from http://arxiv.org/abs/1802.09941. · arXiv
  11. Bergstra James, 2011, Proceedings of the BigLearning Workshop (NIPS’11), 3
  12. Bergstra James, 2012, Random search for hyper-parameter optimization, J. Mach. Learn. Res., 13, 1
  13. 10.1147/JRD.2017.2716578 · DOI
  14. Boag Scott, 2017, Proceedings of the Workshop on ML Systems (NIPS’17)
  15. Bonawitz Keith, 2019, Proceedings of the Conference on Systems and Machine Learning (SysML’19)
  16. 10.1145/3219819.3219861 · DOI · Paper page
  17. 10.1145/1961189.1961199 · DOI · Paper page
  18. Chen K., 2016, Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’16)
  19. Chen Tianqi, 2015, Proceedings of Workshop on Machine Learning Systems (LearningSys’15)
  20. 10.1109/ACCESS.2014.2325029 · DOI · Paper page
  21. Sharan Chetlur Cliff Woolley Philippe Vandermersch Jonathan Cohen John Tran Bryan Catanzaro and Evan Shelhamer. 2014. cuDNN: Efficient primitives for deep learning. Retrieved from http://arxiv.org/abs/1410.0759. Sharan Chetlur Cliff Woolley Philippe Vandermersch Jonathan Cohen John Tran Bryan Catanzaro and Evan Shelhamer. 2014. cuDNN: Efficient primitives for deep learning. Retrieved from http://arxiv.org/abs/1410.0759. · arXiv
  22. Chilimbi Trishul, 2014, Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14)
  23. Cipar James, 2013, Proceedings of the 14th Workshop on Hot Topics in Operating Systems (HotOS’13)
  24. 10.1162/NECO_a_00052 · DOI · Paper page
  25. Coates Adam, 2013, Proceedings of the 30th International Conference on International Conference on Machine Learning, 28
  26. Crankshaw Daniel, 2017, Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI’17)
  27. Cui Henggang, Proceedings of the USENIX Conference on USENIX Annual Technical Conference (USENIX ATC’14)
  28. Cui Henggang, Proceedings of the 11th European Conference on Computer Systems (EuroSys’16)
  29. Dai Wei, Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI’15)
  30. Jeffrey Dean Greg Corrado Rajat Monga Kai Chen Matthieu Devin Mark Mao Andrew Senior Paul Tucker Ke Yang Quoc V. Le etal 2012. Large scale distributed deep networks. In Advances in Neural Information Processing Systems. MIT Press 1223--1231. Jeffrey Dean Greg Corrado Rajat Monga Kai Chen Matthieu Devin Mark Mao Andrew Senior Paul Tucker Ke Yang Quoc V. Le et al. 2012. Large scale distributed deep networks. In Advances in Neural Information Processing Systems. MIT Press 1223--1231.
  31. 10.1145/1327452.1327492 · DOI · Paper page
  32. 10.1017/atsip.2013.9 · DOI · Paper page
  33. 10.1515/pralin-2017-0036 · DOI
  34. Duchi John, 2011, Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12 (July
  35. 10.1007/s10278-017-9965-6 · DOI · Paper page
  36. Farabet C., 2011, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11)
  37. Fischer Volker, Advances in Neural Information Processing Systems 31
  38. Frank Eibe, Weka-A Machine Learning Workbench for Data Mining, 10.1007/978-0-387-09823-4_66 · DOI · Paper page
  39. Ghodsi Ali, 2011, Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation (NSDI’11)
  40. Goodfellow Ian, Advances in Neural Information Processing Systems
  41. Graves Alex, 2014, International Conference on Machine Learning. MIT Press, 1764--1772
  42. Gujarati Arpan, Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference (Middleware’17)
  43. Gupta Suyog, 2015, Proceedings of the 32nd International Conference on International Conference on Machine Learning, 37
  44. 10.1109/MIS.2009.36 · DOI · Paper page
  45. Harlap Aaron, 2018, Gibbons
  46. Harlap Aaron, Proceedings of the Conference on Systems and Machine Learning (SysML’18)
  47. Harlap Aaron, Proceedings of the 12th European Conference on Computer Systems (EuroSys’17)
  48. Hashemi Sayed Hadi, Proceedings of the Conference on Systems and Machine Learning (SysML’19)
  49. Hauswald J., Proceedings of the ACM/IEEE 42nd Annual International Symposium on Computer Architecture (ISCA’15)
  50. Hazelwood K., 2018, Proceedings of the IEEE International Symposium on High Performance Computer Architecture (HPCA’18)
  51. Hestness Joel, 2017, Yang Yang, and Yanqi Zhou.
  52. Hindman Benjamin, 2011, Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation (NSDI’11)
  53. G. E. Hinton and R. R. Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313 5786 (2006) 504--507. DOI:https://doi.org/10.1126/science.1127647 arXiv:http://science.sciencemag.org/content/313/5786/504.full.pdf G. E. Hinton and R. R. Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313 5786 (2006) 504--507. DOI:https://doi.org/10.1126/science.1127647 arXiv:http://science.sciencemag.org/content/313/5786/504.full.pdf · DOI · Paper page
  54. 10.1145/2500887 · DOI
  55. Yanping Huang Yonglong Cheng Dehao Chen HyoukJoong Lee Jiquan Ngiam Quoc V. Le and Zhifeng Chen. 2018. GPipe: Efficient training of giant neural networks using pipeline parallelism. Retrieved from http://arxiv.org/abs/1811.06965. Yanping Huang Yonglong Cheng Dehao Chen HyoukJoong Lee Jiquan Ngiam Quoc V. Le and Zhifeng Chen. 2018. GPipe: Efficient training of giant neural networks using pipeline parallelism. Retrieved from http://arxiv.org/abs/1811.06965. · arXiv
  56. 10.1145/3187009.3177734 · doi:10.1145/3187009.3177734
  57. 10.1113/jphysiol.1962.sp006837 · DOI · Paper page
  58. Hutter Frank, 2014, Proceedings of the 31st International Conference on International Conference on Machine Learning, 32
  59. 10.1109/CVPR.2016.284 · DOI · Paper page
  60. 10.1145/2897824.2925974 · DOI · Paper page
  61. Ishakian V., 2018, Proceedings of the IEEE International Conference on Cloud Engineering (IC2E’18)
  62. 10.1145/3286490.3286561 · DOI
  63. Myeongjae Jeon Shivaram Venkataraman Amar Phanishayee Junjie Qian Wencong Xiao and Fan Yang. 2019. Analysis of large-scale multi-tenant GPU clusters for DNN training workloads. arXiv preprint arXiv:1901.05758. Myeongjae Jeon Shivaram Venkataraman Amar Phanishayee Junjie Qian Wencong Xiao and Fan Yang. 2019. Analysis of large-scale multi-tenant GPU clusters for DNN training workloads. arXiv preprint arXiv:1901.05758. · arXiv
  64. 10.1145/3190508.3190530 · DOI
  65. 10.1145/2647868.2654889 · DOI · Paper page
  66. Jia Zhihao, 2018, Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research), Jennifer Dy and Andreas Krause (Eds.), 80
  67. Jia Zhihao, 2018, Beyond data and model parallelism for deep neural networks. CoRR abs/1807.05358 · arXiv
  68. 10.1145/3035918.3035933 · DOI · Paper page
  69. 10.1162/tacl_a_00065 · DOI · Paper page
  70. Johnson Rie, 2013, Proceedings of the 26th International Conference on Neural Information Processing Systems, 1
  71. Jouppi N. P., Proceedings of the ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA’17)
  72. Karkus Peter, 2017, QMDP-net: Deep learning for planning under partial observability, Advances in Neural Information Processing Systems, 30, I
  73. Kim Jin Kyu, Proceedings of the 11th European Conference on Computer Systems (EuroSys’16)
  74. Koliousis Alexandros, 2019, Pietzuch
  75. Konec̆ný Jakub, 2016, Proceedings of the NIPS Workshop on Private Multi-Party Machine Learning.
  76. Alex Krizhevsky. 2014. One weird trick for parallelizing convolutional neural networks. Retrieved from http://arxiv.org/abs/1404.5997. Alex Krizhevsky. 2014. One weird trick for parallelizing convolutional neural networks. Retrieved from http://arxiv.org/abs/1404.5997. · arXiv
  77. Krizhevsky Alex, Proceedings of the 25th International Conference on Neural Information Processing Systems, 1
  78. 10.1145/3035918.3054775 · DOI
  79. 10.1145/1815961.1816021 · DOI · Paper page
  80. Lee Lee Yun Seong, 2016, Proceedings of International Conference on Machine Learning (ICML’16)
  81. 10.1177/0278364914549607 · DOI · Paper page
  82. Li Ang, 2019, Jieyang Chen, Jiajia Li, Xu Liu, Nathan R. Tallent, and Kevin J. Barker.
  83. 10.1145/2741948.2741965 · DOI
  84. 10.5555/2685048.2685095 · doi:10.5555/2685048.2685095
  85. 10.1016/j.future.2017.02.006 · DOI · Paper page
  86. 10.1145/3187009.3177737 · DOI
  87. Lian Xiangru, Advances in Neural Information Processing Systems
  88. Lin Yujun, 2018, Proceedings of the International Conference on Learning Representations.
  89. Liu Jiayi, 2018, Proceedings of the Deep Learning Day at SIGKDD Conference on Knowledge Discovery and Data Mining (KDD’18)
  90. Mai Luo, 2015, Proceedings of the 7th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud’15)
  91. 10.1145/1807167.1807184 · DOI · Paper page
  92. Dominic Masters and Carlo Luschi. 2018. Revisiting small batch training for deep neural networks. Retrieved from arxiv:1804.07612 http://arxiv.org/abs/1804.07612. Dominic Masters and Carlo Luschi. 2018. Revisiting small batch training for deep neural networks. Retrieved from arxiv:1804.07612 http://arxiv.org/abs/1804.07612. · arXiv
  93. 10.1145/3154842.3154843 · doi:10.1145/3154842.3154843
  94. McMahan H. Brendan, 2017, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS’17)
  95. Meng Xiangrui, 2016, MLlib: Machine learning in apache spark, J. Mach. Learn. Res., 17, 1
  96. Miao H., 2017, Proceedings of the IEEE 33rd International Conference on Data Engineering (ICDE’17)
  97. Mirhoseini Azalia, 2018, Proceedings of the International Conference on Learning Representations.
  98. Mirhoseini Azalia, 2017, Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research), Doina Precup and Yee Whye Teh (Eds.), 70
  99. Mnih Volodymyr, Proceedings of the 29th International Conference on Machine Learning (ICML’12)
  100. Moritz Philipp, Proceedings of International Conference on Learning Representations. arXiv preprint arXiv:1511
  101. Moritz Philipp, 2018, Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI’18)
  102. Narayanan Deepak, 2018, Proceedings of the NIPS Workshop on Systems for Machine Learning.
  103. Nilsson Nils, The Quest for Artificial Intelligence: A History of Ideas and Achievements, 10.1017/CBO9780511819346 · DOI · Paper page
  104. Noel Cyprien, 2014, Proceedings of the NIPS Workshop on Distributed Machine Learning and Matrix Computations.
  105. 10.1145/2733373.2807410 · DOI
  106. 10.1145/3286490.3286562 · doi:10.1145/3286490.3286562
  107. Park Jay H., 2019, Noh
  108. Paszke Adam, 2017, Proceedings of the Conference and Workshop on Neural Information Processing Systems (NIPS-W’17)
  109. Pedregosa Fabian, 2011, Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 12 (Oct.
  110. 10.1145/3190508.3190517 · DOI · Paper page
  111. Pinto Christian, 2018, Hoard: A distributed data caching system to accelerate deep-learning training on the cloud.
  112. 10.1145/3234150 · DOI · Paper page
  113. Qiao Aurick, Proceedings of the USENIX Annual Technical Conference (USENIX ATC 18)
  114. 10.1145/3135974.3135994 · DOI
  115. Recht Benjamin, 2011, Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems
  116. 10.1145/3199605 · DOI
  117. Sebastian Ruder. 2016. An overview of gradient descent optimization algorithms. Retrieved from http://arxiv.org/abs/1609.04747. Sebastian Ruder. 2016. An overview of gradient descent optimization algorithms. Retrieved from http://arxiv.org/abs/1609.04747. · arXiv
  118. Theo Ryffel Andrew Trask Morten Dahl Bobby Wagner Jason Mancuso Daniel Rueckert and Jonathan Passerat-Palmbach. 2018. A generic framework for privacy preserving deep learning. Retrieved from http://arxiv.org/abs/1811.04017. Theo Ryffel Andrew Trask Morten Dahl Bobby Wagner Jason Mancuso Daniel Rueckert and Jonathan Passerat-Palmbach. 2018. A generic framework for privacy preserving deep learning. Retrieved from http://arxiv.org/abs/1811.04017. · arXiv
  119. Christopher De Sa Megan Leszczynski Jian Zhang Alana Marzoev Christopher R. Aberger Kunle Olukotun and Christopher Ré. 2018. High-accuracy low-precision training. Retrieved from http://arxiv.org/abs/1803.03383. Christopher De Sa Megan Leszczynski Jian Zhang Alana Marzoev Christopher R. Aberger Kunle Olukotun and Christopher Ré. 2018. High-accuracy low-precision training. Retrieved from http://arxiv.org/abs/1803.03383. · arXiv
  120. 10.1016/j.neunet.2014.09.003 · DOI · Paper page
  121. 10.21437/Interspeech.2014-274 · DOI · Paper page
  122. Sergeev Alexander, 2018, Horovod: Fast and easy distributed deep learning in TensorFlow.
  123. Shazeer Noam, 2018, Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems
  124. 10.14778/1920841.1920931 · DOI · Paper page
  125. Socher Richard, Advances in Neural Information Processing Systems 26
  126. 10.1145/2806777.2806945 · DOI
  127. 10.21437/Interspeech.2015-354 · DOI · Paper page
  128. Sainbayar Sukhbaatar and Rob Fergus. 2014. Learning from noisy labels with deep neural networks. Retrieved from http://arxiv.org/abs/1406.2080. Sainbayar Sukhbaatar and Rob Fergus. 2014. Learning from noisy labels with deep neural networks. Retrieved from http://arxiv.org/abs/1406.2080. · arXiv
  129. 10.1109/CVPR.2016.308 · DOI · Paper page
  130. Tao Zeyi, 2018, Proceedings of the USENIX Workshop on Hot Topics in Edge Computing (HotEdge'18)
  131. Tokui Seiya, 2015, Proceedings of Workshop on Machine Learning Systems (LearningSys) in The 29th Annual Conference on Neural Information Processing Systems (NIPS’15)
  132. 10.1145/79173.79181 · DOI
  133. Vanhoucke Vincent, Proceedings of the Deep Learning and Unsupervised Feature Learning Workshop (NIPS’11)
  134. 10.1145/2939502.2939516 · DOI
  135. Abhinav Vishnu Charles Siegel and Jeffrey Daily. 2016. Distributed TensorFlow with MPI. Retrieved from http://arxiv.org/abs/1603.02339. Abhinav Vishnu Charles Siegel and Jeffrey Daily. 2016. Distributed TensorFlow with MPI. Retrieved from http://arxiv.org/abs/1603.02339. · arXiv
  136. Wang C., 2017, DLAU: A scalable deep-learning accelerator unit on FPGA, IEEE Trans. Comput.-Aid. Design Integr. Circ. Syst., 36, 3
  137. Wang Minjie, 2014, Proceedings of the NIPS Workshop on Distributed Machine Learning and Matrix Computations.
  138. 10.1145/2637166.2637231 · DOI
  139. 10.1145/3274808.3274828 · doi:10.1145/3274808.3274828
  140. 10.1145/3003665.3003669 · DOI · Paper page
  141. 10.1145/2987550.2987586 · DOI
  142. Wei Jinliang, Proceedings of the 6th ACM Symposium on Cloud Computing (SoCC’15)
  143. 10.1145/2723372.2742793 · DOI
  144. Wen Wei, 2017, Terngrad: Ternary gradients to reduce communication in distributed deep learning. In Advances in Neural Information Processing Systems
  145. Yonghui Wu Mike Schuster Zhifeng Chen Quoc V. Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey Jeff Klingner Apurva Shah Melvin Johnson Xiaobing Liu Lukasz Kaiser Stephan Gouws Yoshikiyo Kato Taku Kudo Hideto Kazawa Keith Stevens George Kurian Nishant Patil Wei Wang Cliff Young Jason Smith Jason Riesa Alex Rudnick Oriol Vinyals Greg Corrado Macduff Hughes and Jeffrey Dean. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. Retrieved from http://arxiv.org/abs/1609.08144. Yonghui Wu Mike Schuster Zhifeng Chen Quoc V. Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey Jeff Klingner Apurva Shah Melvin Johnson Xiaobing Liu Lukasz Kaiser Stephan Gouws Yoshikiyo Kato Taku Kudo Hideto Kazawa Keith Stevens George Kurian Nishant Patil Wei Wang Cliff Young Jason Smith Jason Riesa Alex Rudnick Oriol Vinyals Greg Corrado Macduff Hughes and Jeffrey Dean. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. Retrieved from http://arxiv.org/abs/1609.08144. · arXiv
  146. Wu Zonghan, 2019, Yu
  147. Xiao Tong, 2015, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15)
  148. Xiao Wencong, 2018, Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI’18)
  149. 10.1109/TBDATA.2015.2472014 · DOI · Paper page
  150. 10.1145/2783258.2783270 · DOI
  151. 10.1145/3190508.3190551 · DOI
  152. Zaharia Matei, Presented as part of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI’12). USENIX
  153. 10.1145/2966986.2967011 · DOI · Paper page
  154. 10.1145/2684746.2689060 · DOI · Paper page
  155. Zhang C., 2018, Proceedings of the IEEE 38th International Conference on Distributed Computing Systems (ICDCS’18)
  156. Zhang H., 2016, Proceedings of the IEEE 16th International Conference on Data Mining (ICDM’16)
  157. Zhang Haoyu, Proceedings of the Symposium on Cloud Computing (SoCC’17)
  158. Zhang Hao, Proceedings of the USENIX Conference on Usenix Annual Technical Conference (USENIX ATC’17)
  159. Zhao Huasha, 1972, Proceedings of the SIAM International Conference on Data Mining. 785--793
  160. 10.1109/CVPR.2016.485 · DOI · Paper page
  161. Shuchang Zhou Zekun Ni Xinyu Zhou He Wen Yuxin Wu and Yuheng Zou. 2016. DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients. Retrieved from http://arxiv.org/abs/1606.06160. Shuchang Zhou Zekun Ni Xinyu Zhou He Wen Yuxin Wu and Yuheng Zou. 2016. DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients. Retrieved from http://arxiv.org/abs/1606.06160. · arXiv
  162. Zoph Barret, 2017, Proceedings of the International Conference on Learning Representations.

Cited by

See all 192

Metadata

Authors
Mayer, R., & Jacobsen, H.-A.
Venue
ACM Computing Surveys
Published
2020-02-06
Volume
53
Issue
1
Pages
1-37
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
OpenAlex
W3004495293
Cited by
192
References
28
FWCI
25.80
Type
journal article
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.