A Multi-center Study on the Reproducibility of Drug-Response Assays in Mammalian Cell Lines
Open accessAbstract
Evidence that some high-impact biomedical results cannot be repeated has stimulated interest in practices that generate findable, accessible, interoperable, and reusable (FAIR) data. Multiple papers have identified specific examples of irreproducibility, but practical ways to make data more reproducible have not been widely studied. Here, five research centers in the NIH LINCS Program Consortium investigate the reproducibility of a prototypical perturbational assay: quantifying the responsiveness of cultured cells to anti-cancer drugs. Such assays are important for drug development, studying cellular networks, and patient stratification. While many experimental and computational factors impact intra- and inter-center reproducibility, the factors most difficult to identify and control are those with a strong dependency on biological context. These factors often vary in magnitude with the drug being analyzed and with growth conditions. We provide ways to identify such context-sensitive factors, thereby improving both the theory and practice of reproducible cell-based assays.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Although five research centers applied cells and chemicals of the same stock, center-to-center variations up to 200-fold were observed in growth inhibition rates.
“Although the five research centers applied cells and chemicals of the same stock, astonishing center-to-center variations up to 200-fold were observed in growth inhibition rates.”
From In Vitro research Reproducibility: Keeping up High Standards (2019) · cites this paper as Niepel et al., 2019
The Introduction and Results (‘Measuring Drug Responses in Collaboration’) say that five centers received identical cell aliquots and drug stocks, and that initial inter-center variability reached 200-fold. The 200-fold figure refers specifically to GR50 values, a measure of drug potency, not to growth inhibition rates themselves. (LLM, conf=0.86)
Experimental — accuracy not yet measuredATP levels do not necessarily directly correlate to the number of viable cells, resulting in identical EC50 values for some drugs but differing greatly for others.
“ATP levels do not necessarily directly correlate to the number of viable cells, resulting in identical EC50 values for some drugs, but differing greatly for others.”
From In Vitro research Reproducibility: Keeping up High Standards (2019) · cites this paper as Niepel et al., 2019
The Results (Technical Drivers of Variability; page not provided) say ATP-based measurements can diverge from viable-cell counts, with the extent depending on the drug and cell line. The paper reports small differences for some drugs and large differences in GRmax for others, but it does not establish identical or greatly differing EC50 values; for two drugs, GR50 could not be determined from the ATP-assay data. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe finding of significant interlaboratory variability is consistent with previous attempts to replicate identical protocols across multiple sites for animal and cell line studies.
“The finding of significant interlaboratory variability is also consistent with previous attempts to replicate identical protocols across multiple sites for animal (Crabbe et al., 1999) and cell line studies (Elliott et al., 2017;Niepel et al., 2019).”
From Estimating the replicability of Brazilian biomedical science (2025) · cites this paper as Niepel et al., 2019
Cell seeding, i.e., slight variations in initial cell numbers, was identified as one key source of these variations.
“Cell seeding, i.e. slight variations in initial cell numbers, was identified as one key source of these variations (for more details see Recommendation 5) (Cell density and medium change).”
From In Vitro research Reproducibility: Keeping up High Standards (2019) · cites this paper as Niepel et al., 2019
The subtle interplay between experimental methods and a vast array of poorly defined sources of biological variability was found to be the main cause of the observed irreproducibility.
“Overall, the subtle interplay between experimental methods and a vast array of poorly defined sources of biological variability was found to be the main cause of the observed irreproducibility.”
From In Vitro research Reproducibility: Keeping up High Standards (2019) · cites this paper as Niepel et al., 2019
Niepel and colleagues concluded that most examples of irreproducibility are themselves irreproducible.
“As a consequence of the huge number of individual biological factors involved, Niepel and colleagues came to the rather discouraging conclusion that “most examples of irreproducibility are themselves irreproducible” (Niepel et al., 2019).”
From In Vitro research Reproducibility: Keeping up High Standards (2019) · cites this paper as Niepel et al., 2019
Niepel et al. 2019 systematically addressed inter- and intra-study center variations and identified factors contributing to insufficient reproducibility.
“A recent evocative study of the mammary epithelial cell line MCF10A and growth rate inhibition by anti-cancer drugs systematically addressed inter- and intra-study center variations and identified factors contributing to insufficient reproducibility (Niepel et al., 2019).”
From In Vitro research Reproducibility: Keeping up High Standards (2019) · cites this paper as Niepel et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423 · see citing works
The Cancer Cell Line Encyclopedia enables predictive modelling of anticancer drug sensitivity
Barretina, J., Caponigro, G., Stransky, N., Venkatesan, K., Margolin, A. A., Kim, S., Wilson, C. J., Lehár, J., Kryukov, G. V., Sonkin, D., et al. (2012) · Nature · doi:10.1038/nature11003 · Cited by 7,950 · see citing works
Self-citationRaise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 2× · see citing works
“Adoption of FAIR approaches also responds to concerns from industrial and academic groups about the reproducibility and utility of biomedical research (Arrowsmith, 2011; Baker, 2016; Begley and Ellis, 2012; Prinz et al., 2011) and the adequacy of data-reporting standards (Errington et al.”
BackgroundSystematic identification of genomic markers of drug sensitivity in cancer cells
Garnett, M. J., Edelman, E. J., Heidorn, S. J., Greenman, C. D., Dastur, A., Lau, K. W., Greninger, P., Thompson, I. R., Luo, X., Soares, J., et al. (2012) · Nature · doi:10.1038/nature11005 · Cited by 2,456 · see citing works
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · see citing works
Widespread potential for growth-factor-driven resistance to anticancer kinase inhibitors
Wilson, T. R., Fridlyand, J., Yan, Y., Penuel, E., Burton, L., Chan, E., Peng, J., Lin, E., Wang, Y., Sosman, J., et al. (2012) · Nature · doi:10.1038/nature11249 · Cited by 1,134 · see citing works
Arrowsmith, J. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439 · Cited by 622 · see citing works
Inconsistency in large pharmacogenomic studies
Haibe-Kains, B., El-Hachem, N., Birkbak, N. J., Jin, A. C., Beck, A. H., Aerts, H. J. W. L., & Quackenbush, J. (2013) · Nature · doi:10.1038/nature12831 · Cited by 499 · see citing works
Pharmacogenomic agreement between two cancer cell line data sets
The Cancer Cell Line Encyclopedia Consortium, & The Genomics of Drug Sensitivity in Cancer Consortium (2015) · Nature · doi:10.1038/nature15736 · Cited by 418 · see citing works
Flagged first, then by citations
Research works printed but not linked: 51
- AlQuraishi M, and Sorger PK (2016). Reproducibility will only come with data liberation. Sci. Transl. Med. 8, 339ed7, Seventh Edition PMC5084089 27194726
- Ashley EA (2016). Towards precision medicine. Nat. Rev. Genet 17, 507– 522. 27528417
- Baker M (2016). Biotech giant publishes failures to confirm high-profile science. Nature 530, 141. 26863961
- Bao R, Huang L, Andrade J, Tan W, Kibbe WA, Jiang H, and Feng G (2014). Review of current methods, applications, and data management for the bioinformatics analysis of whole exome sequencing. Cancer Inform. 13, 67– 82. PMC4179624 25288881
- Ben-David U, Siranosian B, Ha G, Tang H, Oren Y, Hinohara K, Strathdee CA, Dempster J, Lyons NJ, Burns R, (2018). Genetic and transcriptional evolution alters cancer cell line drug response. Nature 560, 325– 330. 30089904 PMC6522222
- Bouhaddou M, DiStefano MS, Riesel EA, Carrasco E, Holzapfel HY, Jones DC, Smith GR, Stern AD, Somani SS, Thompson TV, (2016). Drug response consistency in CCLE and CGP. Nature 540, E9– E10. 27905419 PMC5554885
- Boutros M, Brás LP, and Huber W (2006). Analysis of cell-based RNAi screens. Genome Biol. 7, R66. 16869968 PMC1779553
- Bushway PJ, Azimi B, Heynen-Genel S, Price JH, and Mercola M (2010). Hybrid median filter background estimator for correcting distortions in microtiter plate data. Assay Drug Dev. Technol. 8, 238– 250. 20230301 PMC3096555
- Carpenter AE, Jones TR, Lamprecht MR, Clarke C, Kang IH, Friman O, Guertin DA, Chang JH, Lindquist RA, Moffat J, (2006). CellProfiler: image analysis software for identifying and quantifying cell phenotypes. Genome Biol. 7, R100. 17076895 PMC1794559
- Cowell JK, LaDuca J, Rossi MR, Burkhardt T, Nowak NJ, and Matsui S (2005). Molecular characterization of the t(3;9) associated with immortalization in the MCF10A cell line. Cancer Genet. Cytogenet. 163, 23– 29. 16271952
- Coyle MP Jr., Green DP, and Monsanto EH (1989). Advances in carpal bone injury and disease. Hand Clin. 5, 471– 486. 2670972
- Cravatt BF, and Gottesfeld JM (2010). Chemical biology meets biological chemistry minireview series. J. Biol. Chem. 285, 11031– 11032. 20147295 PMC2856974
- Debnath J, Muthuswamy SK, and Brugge JS (2003). Morphogenesis and oncogenesis of MCF-10A mammary epithelial acini grown in three-dimensional basement membrane cultures. Methods 30, 256– 268. 12798140
- ELife editorial. (2017). The challenges of replication. eLife 6, e23693. 28182866 PMC5302882
- Errington TM, Iorns E, Gunn W, Tan FE, Lomax J, and Nosek BA (2014). An open investigation of the reproducibility of cancer biology research. ELife 3. PMC4270077 25490932
- Fallahi-Sichani M, Becker V, Izar B, Baker GJ, Lin JR, Boswell SA, Shah P, Rotem A, Garraway LA, and Sorger PK (2017). Adaptive resistance of melanoma cells to RAF inhibition via reversible induction of a slowly dividing de-differentiated state. Mol. Syst. Biol. 13, 905. 28069687 PMC5248573
- Fletcher JI, Haber M, Henderson MJ, and Norris MD (2010). ABC transporters in cancer: more than just drug efflux pumps. Nat. Rev. Cancer 10, 147– 156. 20075923
- Goecks J, Nekrutenko A, and Taylor J; Galaxy Team (2010). Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences. Genome Biol. 11, R86. 20738864 PMC2945788
- Goodspeed A, Heiser LM, Gray JW, and Costello JC (2016). Tumorderived cell lines as molecular models of cancer pharmacogenomics. Mol. Cancer Res. 14, 3– 13. 26248648 PMC4828339
- Hafner M, Niepel M, Chung M, and Sorger PK (2016). Growth rate inhibition metrics correct for confounders in measuring sensitivity to cancer drugs. Nat. Methods 13, 521– 527. 27135972 PMC4887336
- Hafner M, Niepel M, and Sorger PK (2017a). Alternative drug sensitivity metrics improve preclinical cancer pharmacogenomics. Nat. Biotechnol. 35, 500– 502. 28591115 PMC5668135
- Hafner M, Niepel M, Subramanian K, and Sorger PK (2017b). Designing drug-response experiments and quantifying their results. Curr. Protoc. Chem. Biol. 9, 96– 116. 28628201 PMC5729909
- Harris EA, Koh EJ, Moffat J, and McMillen DR (2016a). Automated inference procedure for the determination of cell growth parameters. Phys. Rev. E 93, 012402. 26871096
- Harris LA, Frick PL, Garbett SP, Hardeman KN, Paudel BB, Lopez CF, Quaranta V, and Tyson DR (2016b). An unbiased metric of antiproliferative drug effect in vitro. Nat. Methods 13, 497– 500. 27135974 PMC4887341
- Haverty PM, Lin E, Tan J, Yu Y, Lam B, Lianoglou S, Neve RM, Martin S, Settleman J, Yauch RL, (2016). Reproducible pharmacogenomic profiling of cancer cell line panels. Nature 533, 333– 337. 27193678
- Heiser LM, Sadanandam A, Kuo WL, Benz SC, Goldstein TC, Ng S, Gibb WJ, Wang NJ, Ziyad S, Tong F, (2012). Subtype and pathway specific responses to anticancer compounds in breast cancer. Proc. Natl. Acad. Sci. USA 109, 2724– 2729. 22003129 PMC3286973
- Ioannidis JP (2017). Acknowledging and overcoming nonreproducibility in basic and preclinical research. JAMA 317, 1019– 1020. 28192565
- Kim YM, Yang S, Xu W, Li S, and Yang X (2008). Continuous in vitro exposure to low-dose genistein induces genomic instability in breast epithelial cells. Cancer Genet. Cytogenet. 186, 78– 84. 18940470 PMC2590788
- Lachmann A, Giorgi FM, Alvarez MJ, and Califano A (2016). Detection and removal of spatial bias in multiwell assays. Bioinformatics 32, 1959– 1965. 27153732
- Lam HY, Clark MJ, Chen R, Chen R, Natsoulis G, O’Huallachain M, Dewey FE, Habegger L, Ashley EA, Gerstein MB, (2011). Performance comparison of whole-genome sequencing platforms. Nat. Biotechnol. 30, 78– 82. 22178993 PMC4076012
- List M (2017). Using docker compose for the simple deployment of an integrated drug target screening platform. J. Integr. Bioinform. 14. PMC6042832 28600904
- Marella NV, Malyavantham KS, Wang J, Matsui S, Liang P, and Berezney R (2009). Cytogenetic and cDNA microarray expression analysis of MCF10 human breast cancer progression cell lines. Cancer Res. 69, 5946– 5953. 19584277 PMC2826242
- Mazoure B, Nadon R, and Makarenkov V (2017). Identification and correction of spatial bias are essential for obtaining quality data in high-throughput screening technologies. Sci. Rep. 7, 11921. 28931934 PMC5607347
- Morrison SJ (2014). Time to do something about reproducibility. eLife 3 PMC4260475 25493617
- Muranen T, Selfors LM, Worster DT, Iwanicki MP, Song L, Morales FC, Gao S, Mills GB, and Brugge JS (2012). Inhibition of PI3K/mTOR leads to adaptive resistance in matrix-attached cancer cells. Cancer Cell 21, 227– 239. 22340595 PMC3297962
- Nature editorial. (2017). Replication studies offer much more than technical details. Nature 541, 259– 260. 28102281
- Niepel M, Hafner M, Chung M, and Sorger PK (2017). Measuring cancer drug sensitivity and resistance in cultured cells. Curr. Protoc. Chem. Biol. 9, 55– 74. 28628199 PMC5538315
- Nosek BA, and Errington TM (2017). Making sense of replications. ELife 6. PMC5245957 28100398
- Omberg L, Ellrott K, Yuan Y, Kandoth C, Wong C, Kellen MR, Friend SH, Stuart J, Liang H, and Margolin AA (2013). Enabling transparent and collaborative computational analysis of 12 tumor types within the Cancer Genome Atlas. Nat. Genet. 45, 1121– 1126. 24071850 PMC3950337
- Orth JD, Kohler RH, Foijer F, Sorger PK, Weissleder R, and Mitchison TJ (2011). Analysis of mitosis and antimitotic drug responses in tumors by in vivo microscopy and single-cell pharmacodynamics. Cancer Res. 71, 4608– 4616. 21712408 PMC3129392
- Pelz O, Gilsdorf M, and Boutros M (2010). Web cellHTS2: a web-application for the analysis of high-throughput screening data. BMC Bioinformatics 11, 185. 20385013 PMC3098057
- Ramirez M, Rajaram S, Steininger RJ, Osipchuk D, Roth MA, Morinishi LS, Evans L, Ji W, Hsu CH, Thurley K, (2016). Diverse drug-resistance mechanisms can emerge from drug-tolerant cancer persister cells. Nat. Commun. 7, 10690. 26891683 PMC4762880
- Röyttäa M, Laine KM, and Häarkönen P (1987). Morphological studies on the effect of Taxol on cultured human prostatic cancer cells. Prostate 11, 95– 106. 2889203
- Salani B, Marini C, Rio AD, Ravera S, Massollo M, Orengo AM, Amaro A, Passalacqua M, Maffioli S, Pfeffer U, (2013). Metformin impairs glucose consumption and survival in Calu-1 cells by direct inhibition of hexokinase-II. Sci. Rep. 3, 2070. 23797762 PMC3691576
- Sandve GK, Nekrutenko A, Taylor J, and Hovig E (2013). Ten simple rules for reproducible computational research. PLoS Comput. Biol. 9, e1003285. 24204232 PMC3812051
- Schenone M, Dančík V, Wagner BK, and Clemons PA (2013). Target identification and mechanism of action in chemical biology and drug discovery. Nat. Chem. Biol. 9, 232– 240. 23508189 PMC5543995
- Seashore-Ludlow B, Rees MG, Cheah JH, Cokol M, Price EV, Coletti ME, Jones V, Bodycombe NE, Soule CK, Gould J, (2015). Harnessing connectivity in a large-scale small-molecule sensitivity dataset. Cancer Discov. 5, 1210– 1223. 26482930 PMC4631646
- Soliman GA, Steenson S, and Etekpo A (2016). Effects of metformin and a mammalian target of rapamycin (mTOR) ATPCompetitive inhibitor on targeted metabolomics in pancreatic cancer cell line. Mol Biol (Los Angel). PMC5315031 28217402
- Soule HD, Maloney TM, Wolman SR, Peterson WD Jr., Brenz R, McGrath CM, Russo J, Pauley RJ, Jones RF, and Brooks SC (1990). Isolation and characterization of a spontaneously immortalized human breast epithelial cell line, MCF-10. Cancer Res. 50, 6075– 6086. 1975513
- Tolliday N (2010). High-throughput assessment of Mammalian cell viability by determination of adenosine triphosphate levels. Curr. Protoc. Chem. Biol. 2, 153– 161. 23839967
- Yonesaka K, Zejnullahu K, Lindeman N, Homes AJ, Jackman DM, Zhao F, Rogers AM, Johnson BE, and J€anne PA (2008). Autocrine production of amphiregulin predicts sensitivity to both gefitinib and cetuximab in EGFR wild-type cancers. Clin. Cancer Res. 14, 6963– 6973. 18980991 PMC3227691
Cited by
Sorted by citations
In Vitro research Reproducibility: Keeping up High Standards
Hirsch, C., & Schildknecht, S. (2019) · Frontiers in Pharmacology · doi:10.3389/fphar.2019.01484 · Cited by 161 · see citing works
Tristan, C. A., Ormanoglu, P., Slamecka, J., Malley, C., Chu, P.-H., Jovanovic, V. M., Gedik, Y., Jethmalani, Y., Bonney, C., Barnaeva, E., et al. (2021) · Stem Cell Reports · doi:10.1016/j.stemcr.2021.11.004 · Cited by 80 · see citing works
Estimating the replicability of Brazilian biomedical science
Initiative, T. B. R., Amaral, O. B., Carneiro, C. F. D., Neves, K., Sampaio, A. P. W., Gomes, B. V., Abreu, M. B. D., Tan, P. B., Mota, G. P. S., Goulart, R. N., et al. (2025) · bioRxiv · doi:10.1101/2025.04.02.645026 · Cited by 13 · see citing works
“The finding of significant interlaboratory variability is also consistent with previous attempts to replicate identical protocols across multiple sites for animal (Crabbe et al., 1999) and cell line studies (Elliott et al., 2017; Niepel et al., 2019).”
Autophagy dark genes: Can we find them with machine learning?
Ranjbar, M., Yang, J. J., Kumar, P., Byrd, D. R., Bearer, E. L., & Oprea, T. I. (2023) · Natural Sciences · doi:10.1002/ntls.20220067 · Cited by 7 · see citing works
- Authors
- Niepel, M., Hafner, M., Mills, C. E., Subramanian, K., Williams, E. H., Chung, M., Gaudio, B., Barrette, A. M., Stern, A. D., Hu, B., Korkola, J. E., Gray, J. W., Birtwistle, M. R., Heiser, L. M., Sorger, P. K., Shamu, C. E., Jayaraman, G., Azeloglu, E. U., Iyengar, R., Sobie, E. A., Mills, G. B., Liby, T., Jaffe, J. D., Alimova, M., Davison, D., Lu, X., Golub, T. R., Subramanian, A., Shelley, B., Svendsen, C. N., Ma’ayan, A., Medvedovic, M., Feiler, H. S., Smith, R., & Devlin, K.
- Affiliations
- Harvard University (US)
- Icahn School of Medicine at Mount Sinai (US)
- Oregon Health & Science University (US)
- Corresponding author
- Birtwistle, Marc R.
- Heiser, Laura M.
- Sorger, Peter K.
- ORCID
- Niepel, Mario
- Hafner, Marc
- Mills, Caitlin E.
- Subramanian, Kartik
- Williams, Elizabeth H.
- Chung, Mirra
- Gaudio, Benjamin
- Barrette, Anne Marie
- Stern, Alan D.
- Hu, Bin
- Korkola, James E.
- Gray, Joe W.
- Birtwistle, Marc R.
- Heiser, Laura M.
- Sorger, Peter K.
- Shamu, Caroline E.
- Azeloglu, Evren U.
- Iyengar, Ravi
- Sobie, Eric A.
- Mills, Gordon B.
- Liby, Tiera
- Jaffe, Jacob D.
- Golub, Todd R.
- Subramanian, Aravind
- Svendsen, Clive N.
- Ma’ayan, Avi
- Medvedovic, Mario
- Feiler, Heidi S.
- Smith, Rebecca
- Devlin, Kaylyn
- Venue
- Cell Systems
- Publisher
- Elsevier BV
- Published
- 2019-07
- Volume
- 9
- Issue
- 1
- Pages
- 35-48.e5
- ISSN
- 2405-4712 (ISSN-L)
- PMC
- PMC6700527
- OpenAlex
- W2959521790
- Cited by
- 133
- References
- 9
- Citations by year
- 2026: 16 · 2025: 19 · 2024: 8 · 2023: 19 · 2022: 32 · 2021: 18 · 2020: 18 · 2019: 3
- FWCI
- 21.01
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Cell Systems (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, cc-by)
- Funders
- Thermo Fisher
- Micron,
- PDX Pharmaceuticals
- Susan G. Komen Foundation
- Prospect Creek Foundation
- NCI
- NIH
- AztraZeneca
- Karus Therapeutics
- Nanostring
- Pfizer
- Tesaro
- Adelson Medical Research Foundation
- Breast Cancer Research Foundation
- Komen Research Foundation
- Ovarian Cancer Research Foundation
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
- Keywords
- reproducibility, mammalian cell lines, anticancer agents, FAIR data, cell-based drug screening, patient stratification
- MeSH terms
- Drug Development, Animals, Antineoplastic Agents, Humans, Mammals, Neoplasms, Reproducibility of Results, Observer Variation, Cell Culture Techniques, Computational Biology, Cell Line, Tumor, High-Throughput Screening Assays
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
