Statistical evaluation of the Predictive Toxicology Challenge 2000–2001
Abstract
Motivation: The development of in silico models to predict chemical carcinogenesis from molecular structure would help greatly to prevent environmentally caused cancers. The Predictive Toxicology Challenge (PTC) competition was organized to test the state-of-the-art in applying machine learning to form such predictive models. Results: Fourteen machine learning groups generated 111 models. The use of Receiver Operating Characteristic (ROC) space allowed the models to be uniformly compared regardless of the error cost function. We developed a statistical method to test if a model performs significantly better than random in ROC space. Using this test as criteria five models performed better than random guessing at a significance level p of 0.05 (not corrected for multiple testing). Statistically the best predictor was the Viniti model for female mice, with p value below 0.002. The toxicologically most interesting models were Leuven2 for male mice, and Kwansei for female rats. These models performed well in the statistical analysis and they are in the middle of ROC space, i.e. distant from extreme cost assumptions. These predictive models were also independently judged by domain experts to be among the three most interesting, and are believed to include a small but significant amount of empirically learned toxicological knowledge. Availability: PTC details and data can be found at: http://www.predictive-toxicology.org/ptc/ Contact: hannu.toivonen@cs.helsinki.fi * To whom correspondence should be addressed.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Detection, Estimation, And Modulation Theory
Trees, H. L. V. (1968) · Cited by 5,964 · see citing works
The foundations of cost-sensitive learning
Elkan, C. (2001) · Cited by 1,812 · see citing works
Provost, F. J., & Fawcett, T. E. (1997) · Cited by 723 · see citing works
Molecular feature mining in HIV data
Krämer, S., Raedt, L. D., & Helma, C. (2001) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/502512.502533 · Cited by 228 · see citing works
Self-citationZeiger, E. (1998) · Regulatory Toxicology and Pharmacology · doi:10.1006/rtph.1998.1234 · Cited by 196 · see citing works
Data Quality in Predictive Toxicology: Reproducibility of Rodent Carcinogenicity Experiments
Gottmann, E., Kramer, S., Pfahringer, B., & Helma, C. (2001) · Environmental Health Perspectives · doi:10.1289/ehp.01109509 · Cited by 127 · see citing works
Self-citationLiu, M., Sussman, N., Klopman, G., & Rosenkranz, H. S. (1996) · Mutation research. Fundamental and molecular mechanisms of mutagenesis · doi:10.1016/0027-5107(96)00111-x · Cited by 50 · see citing works
Knowledge discovery and data mining in toxicology
Helma, C., Gottmann, E., & Krämer, S. (2000) · Statistical Methods in Medical Research · doi:10.1177/096228020000900403 · Cited by 50 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
A Comprehensive Survey on Graph Neural Networks
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. S. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2978386 · Cited by 11,000 · see citing works
Graph neural networks: A review of methods and applications
Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., Wang, L., Li, C., & Sun, M. (2020) · AI Open · doi:10.1016/j.aiopen.2021.01.001 · Cited by 6,880 · see citing works
- Authors
- Toivonen, H., Srinivasan, A., King, R. D., Kramer, S., & Helma, C.
- Affiliations
- University of Helsinki (FI)
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2003-07-01
- Volume
- 19
- Issue
- 10
- Pages
- 1183-1193
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- OpenAlex
- W188630073
- Cited by
- 196
- References
- 8
- Citations by year
- 2026: 7 · 2025: 15 · 2024: 19 · 2023: 16 · 2022: 16 · 2021: 16 · 2020: 17 · 2019: 6 · 2018: 18 · 2017: 6 · 2016: 2 · 2015: 5 · 2014: 5 · 2013: 6 · 2012: 4
- FWCI
- 4.40
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Bioinformatics (publishedVersion)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Cancer Research › Carcinogens and Genotoxicity Assessment
- Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
- Keywords
- in silico models, machine learning, receiver operating characteristic, significance testing, model comparison, molecular structure
- MeSH terms
- Algorithms, Animals, Artificial Intelligence, Carcinogens, Data Collection, Environmental Exposure, Female, Government Programs, Male, Models, Biological, Neoplasms, Sensitivity and Specificity, Sex Factors, Species Specificity, Structure-Activity Relationship, Toxicology, United States, Carcinogenicity Tests, Reproducibility of Results, Models, Statistical, Databases, Factual, Risk Assessment, Mice, Rats
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
