Cited by 1,473Machine Learning in BioinformaticsBioinformatics and Genomic NetworksGenomics and Chromatin Dynamics

Deep learning for computational biology

Open access
Angermueller, Christof, Pärnamaa, Tanel, Parts, Leopold, Stegle, Oliver·2016·Molecular Systems Biology

Abstract

Technological advances in genomics and imaging have led to an explosion of molecular and cellular profiling data from large numbers of samples. This rapid increase in biological data dimension and acquisition rate is challenging conventional analysis strategies. Modern machine learning methods, such as deep learning, promise to leverage very large data sets for finding hidden structure within them, and for making accurate predictions. In this review, we discuss applications of this new breed of analysis approaches in regulatory genomics and cellular imaging. We provide background of what deep learning is, and the settings in which it can be successfully applied to derive biological insights. In addition to presenting specific applications and providing tips for practical use, we also highlight possible pitfalls and limitations to guide computational biologists when and how to make the most use of this new technology.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.15252/msb.20156651

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Massively parallel reporter assays coupled with machine learning have proven useful in addressing similar voids by producing quantitative biological insight that would be difficult to achieve through traditional approaches.

    “Massively parallel reporter assays (MPRAs) – methods that assess thousands to millions of sequence variants in a single experiment – coupled with machine learning have proven useful in addressing similar voids by producing quantitative biological insight that would be difficult to achieve through traditional approaches3–9.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 97

  1. Agathocleous M, Christodoulou G, Promponas V, Christodoulou C, Vassiliades V, Antoniou A (2010) Protein secondary structure prediction with bidirectional recurrent neural nets: can weight updating for each residue enhance performance? In Artificial Intelligence Applications and Innovations, Papadopoulos H, Andreou AS, Bramer M (eds), Vol. 339, pp 128– 137. Berlin Heidelberg: Springer
  2. Alain G, Bengio Y, Rifai S (2012) Regularized auto‐encoders estimate local statistics. In Proc. CoRR, pp 1– 17
  3. Alipanahi B, Delong A, Weirauch MT, Frey BJ (2015) Predicting the sequence specificities of DNA‐ and RNA‐binding proteins by deep learning. Nat Biotechnol 33: 831– 838 26213851
  4. Angermueller C, Lee H, Reik W, Stegle O (2016) Accurate prediction of single‐cell DNA methylation states using deep learning. bioRxiv. doi: 10.1101/055715 PMC5387360 28395661 · DOI
  5. Asgari E, Mofrad MRK (2015) ProtVec: a continuous distributed representation of biological sequences. PLoS ONE 10: e0141287 26555596 PMC4640716
  6. Bach S, Binder A, Montavon G, Klauschen F, Muller KR, Samek W (2015) On pixel‐wise explanations for non‐linear classifier decisions by layer‐wise relevance propagation. PLoS ONE 10: e0130140 26161953 PMC4498753
  7. Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. arXiv:1409.0473 · arXiv
  8. Bastien F, Lamblin P, Pascanu R, Bergstra J, Goodfellow I, Bergeron A, Bouchard N, Warde‐Farley D, Bengio Y (2012) Theano: new features and speed improvements. arXiv:1211.5590 · arXiv
  9. Bell JT, Pai AA, Pickrell JK, Gaffney DJ, Pique‐Regi R, Degner JF, Gilad Y, Pritchard JK (2011) DNA methylation patterns associate with genetic and gene expression variation in HapMap cell lines. Genome Biol 12: R10 21251332 PMC3091299
  10. Bengio Y, Courville A, Vincent P (2013) Representation learning: a review and new perspectives. Pattern Anal Mach Intell IEEE Trans 35: 1798– 1828 23787338
  11. Bergstra J, Cox DD (2013) Hyperparameter optimization and boosting for classifying facial expressions: how good can a “Null” model be? arXiv:1306.3476 · arXiv
  12. Che Z, Purushotham S, Khemani R, Liu Y (2015) Distilling knowledge from deep networks with applications to healthcare domain. arXiv:1512.03542 · arXiv
  13. Cheng C, Yan KK, Yip KY, Rozowsky J, Alexander R, Shou C, Gerstein M (2011) A statistical framework for modeling gene expression using chromatin features and application to modENCODE datasets. Genome Biol 12: R15 21324173 PMC3188797
  14. Ciresan D, Giusti A, Gambardella LM, Schmidhuber J (2012) Deep neural networks segment neuronal membranes in electron microscopy images. In Advances in neural information processing systems, pp 2843– 2851. Cambridge: MIT Press
  15. Ciresan DC, Giusti A, Gambardella LM, Schmidhuber J (2013) Mitosis detection in breast cancer histology images with deep neural networks. In Medical Image Computing and Computer‐Assisted Intervention–MICCAI 2013, pp 411– 418. Berlin Heidelberg: Springer 24579167
  16. Collobert R, Kavukcuoglu K, Farabet C (2011) Torch7: a matlab‐like environment for machine learning. In BigLearn, NIPS Workshop
  17. Dahl GE, Jaitly N, Salakhutdinov R (2014) Multi‐task neural networks for QSAR predictions. arXiv:1406.1231 · arXiv
  18. Dauphin YN, Pascanu R, Gulcehre C, Cho K, Ganguli S, Bengio Y (2014) Identifying and attacking the saddle point problem in high‐dimensional non‐convex optimization. In Advances in neural information processing systems, pp 2933– 2941. Cambridge: MIT Press
  19. Deng L, Togneri R (2015) Deep dynamic models for learning hidden representations of speech features In Speech and audio processing for coding, enhancement and recognition, Ogunfunmi T, Togneri R, Narasimha M (eds), pp 153– 195. New York: Springer
  20. Eduati F, Mangravite LM, Wang T, Tang H, Bare JC, Huang R, Norman T, Kellen M, Menden MP, Yang J, Zhan X, Zhong R, Xiao G, Xia M, Abdo N, Kosyk O, Collaboration N‐N‐UDT, Friend S, Dearry A, Simeonov A et al (2015) Prediction of human population responses to toxic compounds by a collaborative competition. Nat Biotechnol 33: 933– 940 26258538 PMC4568441
  21. Eickholt J, Cheng J (2012) Predicting protein residue‐residue contacts using deep networks and boosting. Bioinformatics 28: 3066– 3072 23047561 PMC3509494
  22. Eickholt J, Cheng J (2013) DNdisorder: predicting protein disorder using boosting and deep networks. BMC Bioinformatics 14: 88 23497251 PMC3599628
  23. Fakoor R, Ladhak F, Nazi A, Huber M (2013) Using deep learning to enhance cancer diagnosis and classification In Proceedings of the ICML Workshop on the Role of Machine Learning in Transforming Healthcare, Atlanta, GA: JMLR, W&CP
  24. Ferrari A, Lombardi S, Signoroni A (2015) Bacterial colony counting by Convolutional Neural Networks In Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual International Conference of the IEEE, pp 7458– 7461. New York: IEEE 26738016
  25. Gawehn E, Hiss JA, Schneider G (2016) Deep learning in drug discovery. Mol Informatics 35: 3– 14 27491648
  26. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 580– 587. New York: IEEE
  27. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, pp 249– 256. JMLR Conference Proceedings
  28. Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks. In International Conference on Artificial Intelligence and Statistics, pp 315– 323. JMLR Conference Proceedings
  29. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. Cambridge: MIT Press in preparation
  30. Graves A, Mohamed A‐R, Hinton G (2013) Speech recognition with deep recurrent neural networks. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 6645– 6649. New York: IEEE
  31. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313: 504– 507 16873662
  32. Hinton GE, Osindero S, Teh Y‐W (2006) A fast learning algorithm for deep belief nets. Neural Comput 18: 1527– 1554 16764513
  33. Hinton GE (2012) A practical guide to training restricted Boltzmann machines In Neural networks: tricks of the trade, Montavon G, Orr G, Müller K‐R (eds), pp 599– 619. Heidelberg Berlin: Springer
  34. Hutter F, Hoos HH, Leyton‐Brown K (2011) Sequential model‐based optimization for general algorithm configuration In Learning and intelligent optimization, Coello Coello CA (ed.), Vol. 6683, pp 507– 523. Heidelberg Berlin: Springer
  35. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167 · arXiv
  36. Jain V, Murray JF, Roth F, Turaga S, Zhigulin V, Briggman KL, Helmstaedter MN, Denk W, Seung HS (2007) Supervised learning of image restoration with convolutional networks. In 2007 IEEE 11th International Conference on Computer Vision, pp 1– 8. New York: IEEE
  37. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia, pp 675– 678. New York: ACM
  38. Kang HM, Ye C, Eskin E (2008) Accurate discovery of expression quantitative trait loci under confounding from spurious and genuine regulatory hotspots. Genetics 180: 1909– 1925 18791227 PMC2600931
  39. Karlic R, Chung HR, Lasserre J, Vlahovicek K, Vingron M (2010) Histone modification levels are predictive for gene expression. Proc Natl Acad Sci USA 107: 2926– 2931 20133639 PMC2814872
  40. Kell DB (2005) Metabolomics, machine learning and modelling: towards an understanding of the language of cells. Biochem Soc Trans 33: 520– 524 15916555
  41. Kelley DR, Snoek J, Rinn J (2016) Basset: learning the regulatory code of the accessible genome with deep convolutional neural networks. Genome Res doi: 10.1101/gr.200535.115 PMC4937568 27197224 · DOI · Paper page
  42. Kingma DP, Welling M (2013) Auto‐encoding variational bayes. arXiv:1312.6114 · arXiv
  43. Kingma D, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980 · arXiv
  44. Koh PW, Pierson E, Kundaje A (2016) Denoising genome‐wide histone ChIP‐seq with convolutional neural networks. bioRxiv doi: 10.1101/052118 PMC5870713 28881977 · DOI
  45. Kraus OZ, Ba LJ, Frey B (2015) Classifying and segmenting microscopy images using convolutional multiple instance learning. arXiv:1511.05286v1 PMC4908336 27307644 · arXiv
  46. Lee B, Lee T, Na B, Yoon S (2015) DNA‐level splice junction prediction using deep recurrent neural networks. arXiv:1512.05135 · arXiv
  47. Leung MKK, Xiong HY, Lee LJ, Frey BJ (2014) Deep learning of the tissue‐regulated splicing code. Bioinformatics 30: i121– i129 24931975 PMC4058935
  48. Leung MKK, Delong A, Alipanahi B, Frey BJ (2016) Machine learning in genomic medicine: a review of computational problems and data sets. In Proceedings of the IEEE, Vol. 104, pp 176– 197. New York: IEEE
  49. Li SZ (2009) Markov random field modeling in image analysis. Berlin Heidelberg: Springer Science & Business Media
  50. Li J, Ching T, Huang S, Garmire LX (2015) Using epigenomics data to predict gene expression in lung cancer. BMC Bioinformatics 16(Suppl. 5): S10 PMC4402699 25861082
  51. Libbrecht MW, Noble WS (2015) Machine learning applications in genetics and genomics. Nat Rev Genet 16: 321– 332 25948244 PMC5204302
  52. Lipton ZC (2015) A critical review of recurrent neural networks for sequence learning. arXiv:1506.00019 · arXiv
  53. Lipton ZC, Kale DC, Elkan C, Wetzell R (2015) Learning to diagnose with LSTM recurrent neural networks. arXiv:1511.03677 · arXiv
  54. Lyons J, Dehzangi A, Heffernan R, Sharma A, Paliwal K, Sattar A, Zhou Y, Yang Y (2014) Predicting backbone Cα angles and dihedrals from protein sequences by stacked sparse auto‐encoder deep neural network. J Comput Chem 35: 2040– 2046 25212657
  55. Mamoshina P, Vieira A, Putin E, Zhavoronkov A (2016) Applications of deep learning in biomedicine. Mol Pharm 13: 1445– 1454 27007977
  56. Märtens K, Hallin J, Warringer J, Liti G, Parts L (2016) Predicting quantitative traits from genome and phenome with near perfect accuracy. Nat Commun 7: 11512 27160605 PMC4866306
  57. McCulloch WS, Pitts W (1943) A logical calculus of the ideas immanent in nervous activity. Bull Math Biophys 5: 115– 133 2185863
  58. Menden MP, Iorio F, Garnett M, McDermott U, Benes CH, Ballester PJ, Saez‐Rodriguez J (2013) Machine learning prediction of cancer cell sensitivity to drugs based on genomic and chemical properties. PLoS ONE 8: e61318 23646105 PMC3640019
  59. Michalski RS, Carbonell JG, Mitchell TM (2013) Machine learning: an artificial intelligence approach. Berlin Heidelberg: Springer Science & Business Media
  60. Montgomery SB, Sammeth M, Gutierrez‐Arcelus M, Lach RP, Ingle C, Nisbett J, Guigo R, Dermitzakis ET (2010) Transcriptome genetics using second generation sequencing in a Caucasian population. Nature 464: 773– 777 20220756 PMC3836232
  61. Murphy KP (2012) Machine learning: a probabilistic perspective. Cambridge: MIT Press
  62. Nesterov Y (2013) Introductory lectures on convex optimization: a basic course, Vol. 87, Berlin Heidelberg: Springer Science & Business Media
  63. Ning F, Delhomme D, LeCun Y, Piano F, Bottou L, Barbano PE (2005) Toward automatic phenotyping of developing embryos from videos. Image Process IEEE Trans 14: 1360– 1371 16190471
  64. Park Y, Kellis M (2015) Deep learning for regulatory genomics. Nat Biotechnol 33: 825– 826 26252139
  65. Pärnamaa T, Parts L (2016) Accurate classification of protein subcellular localization from high throughput microscopy images using deep learning. bioRxiv doi: 10.1101/050757 PMC5427497 28391243 · DOI
  66. Parts L, Stegle O, Winn J, Durbin R (2011) Joint genetic analysis of gene expression data with inferred cellular phenotypes. PLoS Genet 7: e1001276 21283789 PMC3024309
  67. Parts L, Liu YC, Tekkedil MM, Steinmetz LM, Caudy AA, Fraser AG, Boone C, Andrews BJ, Rosebrock AP (2014) Heritability and genetic basis of protein level variation in an outbred population. Genome Res 24: 1363– 1370 24823668 PMC4120089
  68. Pickrell JK, Marioni JC, Pai AA, Degner JF, Engelhardt BE, Nkadori E, Veyrieras JB, Stephens M, Gilad Y, Pritchard JK (2010) Understanding mechanisms underlying human gene expression variation with RNA sequencing. Nature 464: 768– 772 20220758 PMC3089435
  69. Rakitsch B, Stegle O (2016) Modelling local gene networks increases power to detect trans‐acting genetic effects on gene expression. Genome Biol 17: 33 26911988 PMC4765046
  70. Rampasek L, Goldenberg A (2016) TensorFlow: biology's gateway to deep learning? Cell Syst 2: 12– 14 27136685
  71. Razavian A, Azizpour H, Sullivan J, Carlsson S (2014) CNN features off‐the‐shelf: an astounding baseline for recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 806– 813. New York: IEEE
  72. Salakhutdinov R, Larochelle H (2010) Efficient learning of deep Boltzmann machines In International Conference on Artificial Intelligence and Statistics, pp 693– 700. JMLR Conference Proceedings
  73. Salakhutdinov R, Hinton G (2012) An efficient learning procedure for deep Boltzmann machines. Neural Comput 24: 1967– 2006 22509963
  74. Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61: 85– 117 25462637
  75. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large‐scale image recognition. arXiv:1409.1556 · arXiv
  76. Sønderby SK, Winther O (2014) Protein secondary structure prediction with long short term memory networks. arXiv:1412.7828 · arXiv
  77. Spencer M, Eickholt J, Cheng J (2015) A deep learning network approach to ab initio protein secondary structure prediction. IEEE/ACM Trans Comput Biol Bioinformatics 12: 103– 112 PMC4348072 25750595
  78. Springenberg JT, Dosovitskiy A, Brox T, Riedmiller M (2014) Striving for simplicity: the all convolutional net. arXiv:1412.6806 · arXiv
  79. Stegle O, Parts L, Durbin R, Winn J (2010) A Bayesian framework to account for complex non‐genetic factors in gene expression levels greatly increases power in eQTL studies. PLoS Comput Biol 6: e1000770 20463871 PMC2865505
  80. Stormo GD, Schneider TD, Gold L, Ehrenfeucht A (1982) Use of the ‘Perceptron’ algorithm to distinguish translational initiation sites in E. coli. Nucleic Acids Res 10: 2997– 3011 7048259 PMC320670
  81. Sutskever I (2013) Training recurrent neural networks. PhD Thesis, Graduate Department of Computer Science, University of Toronto, Toronto, Canada
  82. Sutskever I, Martens J, Dahl G, Hinton G (2013) On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning (ICML‐13), pp 1139– 1147. JMLR Conference Proceedings
  83. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp 3104– 3112. Cambridge: MIT Press
  84. Swan AL, Mobasheri A, Allaway D, Liddell S, Bacardit J (2013) Application of machine learning to proteomics data: classification and biomarker identification in postgenomics biology. OMICS 17: 595– 610 24116388 PMC3837439
  85. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015a) Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1– 9. New York: IEEE
  86. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2015b) Rethinking the inception architecture for computer vision. arXiv:1512.00567 · arXiv
  87. Team TTD, Al‐Rfou R, Alain G, Almahairi A, Angermueller C, Bahdanau D, Ballas N, Bastien F, Bayer J, Belikov A (2016) Theano: a python framework for fast computation of mathematical expressions. arXiv:1605.02688 · arXiv
  88. Wang K, Cao K, Hannenhalli S (2015) Chromatin and genomic determinants of alternative splicing. In Proceedings of the 6th ACM Conference on Bioinformatics, Computational Biology and Health Informatics, pp 345– 354. New York: ACM PMC5558438 28825057
  89. Xie Y, Xing F, Kong X, Su H, Yang L (2015) Beyond classification: structured regression for robust cell detection using convolutional neural network. In Medical Image Computing and Computer‐Assisted Intervention–MICCAI 2015, pp 358– 365. Heidelberg Berlin: Springer PMC5226438 28090601
  90. Xiong HY, Alipanahi B, Lee LJ, Bretschneider H, Merico D, Yuen RKC, Hua Y, Gueroussov S, Najafabadi HS, Hughes TR, Morris Q, Barash Y, Krainer AR, Jojic N, Scherer SW, Blencowe BJ, Frey BJ (2015) The human splicing code reveals new insights into the genetic determinants of disease. Science 347: 1254806 25525159 PMC4362528
  91. Xiong C, Merity S, Socher R (2016) Dynamic memory networks for visual and textual question answering. arXiv:1603.01417 · arXiv
  92. Xu R, Wunsch D II, Frank R (2007) Inference of genetic regulatory networks with recurrent neural network models using particle swarm optimization. IEEE/ACM Trans Comput Biol Bioinformatics 4: 681– 692 17975278
  93. Xu Y, Mo T, Feng Q, Zhong P, Lai M, Chang EI (2014) Deep learning of feature representation with multiple instance learning for medical image analysis. In 2014 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp 1626– 1630. New York: IEEE
  94. Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks? In Advances in neural information processing systems 27, Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ (eds), pp 3320– 3328. Cambridge: MIT Press
  95. Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In Computer vision–ECCV 2014, pp 818– 833. Heidelberg Berlin: Springer
  96. Zhang W, Li R, Zeng T, Sun Q, Kumar S, Ye J, Ji S (2015) Deep model based transfer and multi‐task learning for biological image analysis. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 1475– 1484. New York: ACM
  97. Zhou J, Troyanskaya OG (2015) Predicting effects of noncoding variants with deep learning‐based sequence model. Nat Methods 12: 931– 934 26301843 PMC4768299

Cited by

Sorted by citations

Authors
Angermueller, C., Pärnamaa, T., Parts, L., & Stegle, O.
Affiliations
  • European Bioinformatics Institute (GB)
  • Wellcome Sanger Institute (GB)
Corresponding author
  • Parts, Leopold
  • Stegle, Oliver
Venue
Molecular Systems Biology
Published
2016-07-01
Volume
12
Issue
7
Pages
878-878
ISSN
1744-4292 (ISSN-L)
OpenAlex
W2502949459
Cited by
1,473
References
27
Citations by year
2026: 94 · 2025: 159 · 2024: 140 · 2023: 133 · 2022: 133 · 2021: 170 · 2020: 215 · 2019: 197 · 2018: 154 · 2017: 67 · 2016: 9 · 2013: 1
FWCI
12.90
Type
journal article
Language
en
Open access
Locations
  • Molecular Systems Biology (publishedVersion, other-oa)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • European Molecular Biology Organization
  • European Commission
  • Estonian Research Council (IUT34‐4)
  • Wellcome Trust
  • European Research Council (N635290)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Chromatin Dynamics
Keywords
deep learning, computational biology, regulatory genomics, cell imaging, high-dimensional biological data, machine learning, deep learning limitations
MeSH terms
Machine Learning, Humans, Models, Genetic, Computational Biology, Genomics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review