Comparison of the Predictive Performance and Interpretability of Random Forest and Linear Models on Benchmark Data Sets
Abstract
The ability to interpret the predictions made by quantitative structure-activity relationships (QSARs) offers a number of advantages. While QSARs built using nonlinear modeling approaches, such as the popular Random Forest algorithm, might sometimes be more predictive than those built using linear modeling approaches, their predictions have been perceived as difficult to interpret. However, a growing number of approaches have been proposed for interpreting nonlinear QSAR models in general and Random Forest in particular. In the current work, we compare the performance of Random Forest to those of two widely used linear modeling approaches: linear Support Vector Machines (SVMs) (or Support Vector Regression (SVR)) and partial least-squares (PLS). We compare their performance in terms of their predictivity as well as the chemical interpretability of the predictions using novel scoring schemes for assessing heat map images of substructural contributions. We critically assess different approaches for interpreting Random Forest models as well as for obtaining predictions from the forest. We assess the models on a large number of widely employed public-domain benchmark data sets corresponding to regression and binary classification problems of relevance to hit identification and toxicology. We conclude that Random Forest typically yields comparable or possibly better predictive performance than the linear modeling approaches and that its predictions may also be interpreted in a chemically and biologically meaningful way. In contrast to earlier work looking at interpretation of nonlinear QSAR models, we directly compare two methodologically distinct approaches for interpreting Random Forest models. The approaches for interpreting Random Forest assessed in our article were implemented using open-source programs that we have made available to the community. These programs are the rfFC package ( https://r-forge.r-project.org/R/?group_id=1725 ) for the R statistical programming language and the Python program HeatMapWrapper [ https://doi.org/10.5281/zenodo.495163 ] for heat map generation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A relatively wide array of machine learning approaches have been applied to prediction of different biomedical properties of nanoparticles such as predicting cellular uptake, cytotoxicity, molecular loading, molecular release, nanoparticle adherence, nanoparticle size and polydispersity.
“A relatively wide array of machine learning approaches34 have been applied to prediction of different biomedical properties of nanoparticles such as predicting cellular uptake, cytotoxicity, molecular loading, molecular release, nanoparticle adherence, nanoparticle size and polydispersity65.”
From Exploiting machine learning for end-to-end drug discovery and development (2019) · cites this paper as [65]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Part of this paper’s reference list could not be split into single entries.
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196 · see citing works
A tutorial on support vector regression
Smola, A. J., & Schölkopf, B. (2004) · Statistics and Computing · doi:10.1023/b:stco.0000035301.49549.88 · Cited by 13,482 · see citing works
Open Babel: An open chemical toolbox
O'Boyle, N. M., Banck, M. S., James, C. A. J., Morley, C., Vandermeersch, T., & Hutchison, G. R. (2011) · Journal of Cheminformatics · doi:10.1186/1758-2946-3-33 · Cited by 10,705 · see citing works
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490 · see citing works
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · see citing works
Extended-Connectivity Fingerprints
Rogers, D. O. W., & Hahn, M. (2010) · Journal of Chemical Information and Modeling · doi:10.1021/ci100050t · Cited by 7,597 · see citing works
An Introduction to Categorical Data Analysis
Agresti, A. (2006) · Wiley series in probability and statistics · doi:10.1002/0470114754 · Cited by 6,697 · see citing works
Introduction to Computational Chemistry
Jensen, F. (1998) · Cited by 4,673 · see citing works
Bias in random forest variable importance measures: Illustrations, sources and a solution
Strobl, C., Boulesteix, A., Zeileis, A., & Hothorn, T. (2007) · BMC Bioinformatics · doi:10.1186/1471-2105-8-25 · Cited by 3,755 · see citing works
Random Forest: A Classification and Regression Tool for Compound Classification and QSAR Modeling
Svetnik, V., Liaw, A. I., Tong, C., Culberson, J. C., Sheridan, R. P., & Feuston, B. P. (2003) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci034160g · Cited by 3,613 · see citing works
An introduction to kernel-based learning algorithms
Müller, K. R., Mika, S., Rätsch, G., Tsuda, K., & Schölkopf, B. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.914517 · Cited by 3,485 · see citing works
ROCR: visualizing classifier performance in R
Sing, T., Sander, O., Beerenwinkel, N., & Lengauer, T. (2005) · Computer applications in the biosciences · doi:10.1093/bioinformatics/bti623 · Cited by 3,172 · see citing works
<i>p</i>-σ-π Analysis. A Method for the Correlation of Biological Activity and Chemical Structure
Hansch, C., & Fujita, T. (1964) · Journal of the American Chemical Society · doi:10.1021/ja01062a035 · Cited by 2,313 · see citing works
QSAR Modeling: Where have you been? Where are you going to?
Cherkasov, A., Muratov, E. N., Fourches, D., Varnek, A., Baskin, I. I., Cronin, M., Dearden, J., Gramatica, P., Martin, Y. C., Todeschini, R., et al. (2014) · Journal of Medicinal Chemistry · doi:10.1021/jm4004285 · Cited by 2,164 · see citing works
Assessing the accuracy of prediction algorithms for classification: an overview
Baldi, P., Brunak, S., Chauvin, Y., Andersen, C. A. F., & Nielsen, H. (2000) · Bioinformatics · doi:10.1093/bioinformatics/16.5.412 · Cited by 2,111 · see citing works
The pls Package: Principal Component and Partial Least Squares Regression in R
Mevik, B., & Wehrens, R. (2007) · Journal of Statistical Software · doi:10.18637/jss.v018.i02 · Cited by 1,826 · see citing works
A Medicinal Chemist’s Guide to Molecular Interactions
Bissantz, C., Kuhn, B., & Ståhl, M. (2010) · Journal of Medicinal Chemistry · doi:10.1021/jm100112j · Cited by 1,741 · see citing works
SMILES. 2. Algorithm for generation of unique SMILES notation
Weininger, D., Weininger, A., & Weininger, J. L. (1989) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci00062a008 · Cited by 1,286 · see citing works
Inference for the Generalization Error
Nadeau, C., & Bengio, Y. (2003) · Machine Learning · doi:10.1023/a:1024068626366 · Cited by 1,092 · see citing works
No Unbiased Estimator of the Variance of K-Fold Cross-ValidationPreprint
Bengio, Y., & Grandvalet, Y. (2003) · RePEc: Research Papers in Economics · Cited by 886 · see citing works
Beware of R 2 : Simple, Unambiguous Assessment of the Prediction Accuracy of QSAR and QSPR Models
Alexander, D. L., Tropsha, A., & Winkler, D. A. (2015) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5b00206 · Cited by 754 · see citing works
Derivation and Validation of Toxicophores for Mutagenicity Prediction
Kazius, J., McGuire, R., & Bursi, R. (2004) · Journal of Medicinal Chemistry · doi:10.1021/jm040835a · Cited by 668 · see citing works
Is cross-validation valid for small-sample microarray classification?
Braga-Neto, U., & Dougherty, E. R. (2004) · Bioinformatics · doi:10.1093/bioinformatics/btg419 · Cited by 599 · see citing works
Comparing two K-category assignments by a K-category correlation coefficient
Gorodkin, J. (2004) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2004.09.006 · Cited by 545 · see citing works
Halide Effects in Transition Metal Catalysis
Fagnou, K., & Lautens, M. (2002) · Angewandte Chemie International Edition · doi:10.1002/1521-3773(20020104)41:1 · Cited by 520 · see citing works
Machine learning methods in chemoinformatics
Mitchell, J. B. O. (2014) · Wiley Interdisciplinary Reviews Computational Molecular Science · doi:10.1002/wcms.1183 · Cited by 470 · see citing works
How not to develop a quantitative structure–activity or structure–property relationship (QSAR/QSPR)
Dearden, J. C., Cronin, M. T. D., & Kaiser, K. L. E. (2009) · SAR and QSAR in Environmental Research · doi:10.1080/10629360902949567 · Cited by 468 · see citing works
Böhm, M., Stürzebecher, J., & Klebe, G. (1999) · Journal of Medicinal Chemistry · doi:10.1021/jm981062r · Cited by 440 · see citing works
Pybel: a Python wrapper for the OpenBabel cheminformatics toolkit
O’Boyle, N. M., Morley, C., & Hutchison, G. (2008) · Chemistry Central Journal · doi:10.1186/1752-153x-2-5 · Cited by 412 · see citing works
Maximum Unbiased Validation (MUV) Data Sets for Virtual Screening Based on PubChem Bioactivity Data
Rohrer, S. G., & Baumann, K. (2009) · Journal of Chemical Information and Modeling · doi:10.1021/ci8002649 · Cited by 403 · see citing works
Drawing the PDB: Protein−Ligand Complexes in Two Dimensions
Stierand, K., & Rarey, M. (2010) · ACS Medicinal Chemistry Letters · doi:10.1021/ml100164p · Cited by 313 · see citing works
A Comparison of Methods for Modeling Quantitative Structure−Activity Relationships
Sutherland, J. J., O’Brien, L. A., & Weaver, D. F. (2004) · Journal of Medicinal Chemistry · doi:10.1021/jm0497141 · Cited by 230 · see citing works
Predicting Drug-Induced Hepatotoxicity Using QSAR and Toxicogenomics Approaches
Low, Y., Uehara, T., Minowa, Y., Yamada, H., Ohno, Y., Urushidani, T., Sedykh, A., Muratov, E. N., Кузьмин, В. Е., Fourches, D., et al. (2011) · Chemical Research in Toxicology · doi:10.1021/tx200148a · Cited by 224 · see citing works
lazar: a modular predictive toxicology framework
Maunz, A., Gütlein, M., Rautenberg, M., et al. (2013) · Frontiers in Pharmacology · doi:10.3389/fphar.2013.00038 · Cited by 175 · see citing works
DePriest, S. A., Mayer, D., Naylor, C. B., & Marshall, G. R. (1993) · Journal of the American Chemical Society · doi:10.1021/ja00066a004 · Cited by 167 · see citing works
Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them)
Berrar, D., & Flach, P. (2012) · Briefings in Bioinformatics · doi:10.1093/bib/bbr008 · Cited by 136 · see citing works
Interpreting Random Forest Classification Models Using a Feature Contribution Method
Palczewska, A., Palczewski, J., Robinson, R. M., & Neagu, D. C. (2014) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-04717-1_9 · Cited by 132 · see citing works
Self-citationA study in machine learning from imbalanced data for sentence boundary detection in speech
Liu, Y., Chawla, N. V., Harper, M. P., Shriberg, E. E., & Stolcke, A. (2005) · Computer Speech & Language · doi:10.1016/j.csl.2005.06.002 · Cited by 128 · see citing works
Gohlke, H., & Klebe, G. (2002) · Journal of Medicinal Chemistry · doi:10.1021/jm020808p · Cited by 122 · see citing works
Gavaghan, C. L., Arnby, C. H., Blomberg, N., Strandlund, G., & Boyer, S. K. (2007) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-006-9095-6 · Cited by 101 · see citing works
Development and Evaluation of an in Silico Model for hERG Binding
Song, M., & Clark, M. (2005) · Journal of Chemical Information and Modeling · doi:10.1021/ci050308f · Cited by 101 · see citing works
On the interpretation and interpretability of quantitative structure–activity relationship models
Guha, R. (2008) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-008-9240-5 · Cited by 98 · see citing works
jCompoundMapper: An open source Java library and command-line tool for chemical fingerprints
Hinselmann, G., Rosenbaum, L., Jahn, A., Fechner, N., & Zell, A. (2011) · Journal of Cheminformatics · doi:10.1186/1758-2946-3-3 · Cited by 95 · see citing works
Interpretation of QSAR Models Based on Random Forest Methods
Kuz’min, V. Е., Полищук, П. Г., Artemenko, A. G., & Andronati, S. A. (2011) · Molecular Informatics · doi:10.1002/minf.201000173 · Cited by 84 · see citing works
Universal Approach for Structural Interpretation of QSAR/QSPR Models
Polishchuk, P. G., Kuz'min, V. E., Artemenko, A. G., & Muratov, E. N. (2013) · Molecular Informatics · doi:10.1002/minf.201300029 · Cited by 76 · see citing works
Robust Cross-Validation of Linear Regression QSAR Models
Konovalov, D. A., Llewellyn, L. E., Heyden, Y. V., & Coomans, D. H. (2008) · Journal of Chemical Information and Modeling · doi:10.1021/ci800209k · Cited by 74 · see citing works
Gaussian Processes for Classification: QSAR Modeling of ADMET and Target Activity
Obrezanova, O., & Segall, M. (2010) · Journal of Chemical Information and Modeling · doi:10.1021/ci900406x · Cited by 71 · see citing works
Kernel-Based Partial Least Squares: Application to Fingerprint-Based QSAR with Model Visualization
An, Y., Sherman, W., & Dixon, S. L. (2013) · Journal of Chemical Information and Modeling · doi:10.1021/ci400250c · Cited by 68 · see citing works
Interpretation of Nonlinear QSAR Models Applied to Ames Mutagenicity Data
Carlsson, L., Helgee, E. A., & Boyer, S. K. (2009) · Journal of Chemical Information and Modeling · doi:10.1021/ci9002206 · Cited by 65 · see citing works
Forest Floor Visualizations of Random ForestsPreprint
Welling, S. H., Refsgaard, H. H. F., Brockhoff, P. B., & Clemmensen, L. K. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09196 · Cited by 65 · see citing works
Interpreting linear support vector machine models with heat map molecule coloring
Rosenbaum, L., Hinselmann, G., Jahn, A., & Zell, A. (2011) · Journal of Cheminformatics · doi:10.1186/1758-2946-3-11 · Cited by 57 · see citing works
An approach to the interpretation of backpropagation neural network models in QSAR studies
Baskin, I. I., Айт, А. О., Halberstam, N. M., Palyulin, V. A., & Зефиров, Н. С. (2002) · SAR and QSAR in environmental research · doi:10.1080/10629360290002073 · Cited by 55 · see citing works
Visualization and Interpretation of Support Vector Machine Activity Predictions
Balfer, J., & Bajorath, J. (2015) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5b00175 · Cited by 51 · see citing works
Полищук, П. Г., Tinkov, O. V., Khristova, T., Ognichenko, L. N., Kosinskaya, A. P., Varnek, A. A., & Kuz’min, V. Е. (2016) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6b00371 · Cited by 51 · see citing works
Assessing toxicological data quality: basic principles, existing schemes and current limitations
Przybylak, K. R., Madden, J. C., Cronin, M., et al. (2012) · SAR and QSAR in environmental research · doi:10.1080/1062936x.2012.664825 · Cited by 48 · see citing works
Robinson, R. M., Glen, R. C., & Mitchell, J. B. O. (2011) · Molecular Informatics · doi:10.1002/minf.201000159 · Cited by 46 · see citing works
Webb, S. J., Hanser, T., Howlin, B. J., Krause, P., & Vessey, J. D. (2014) · Journal of Cheminformatics · doi:10.1186/1758-2946-6-8 · Cited by 39 · see citing works
Johnson, S. R., Chen, X., Murphy, D. K., et al. (2007) · Molecular Pharmaceutics · doi:10.1021/mp070030+ · Cited by 37 · see citing works
A novel hybrid ultrafast shape descriptor method for use in virtual screening
Cannon, E. O., Nigsch, F., & Mitchell, J. B. O. (2008) · Chemistry Central Journal · doi:10.1186/1752-153x-2-3 · Cited by 36 · see citing works
Cassano, A., Robinson, R. M., Palczewska, A., et al. (2016) · Alternatives to Laboratory Animals · doi:10.1177/026119291604400603 · Cited by 35 · see citing works
Self-citationHinselmann, G., Rosenbaum, L., Jahn, A., et al. (2011) · Journal of Chemical Information and Modeling · doi:10.1021/ci100073w · Cited by 32 · see citing works
Welling, S. H., Clemmensen, L. K. H., Buckley, S. T., et al. (2015) · European Journal of Pharmaceutics and Biopharmaceutics · doi:10.1016/j.ejpb.2015.05.012 · Cited by 28 · see citing works
A Maximum Common Subgraph Kernel Method for Predicting the Chromosome Aberration Test
Mohr, J., Jain, B. J., Sutter, A. P., ter Laak, A., Steger‐Hartmann, T., Heinrich, N., & Obermayer, K. (2010) · Journal of Chemical Information and Modeling · doi:10.1021/ci900367j · Cited by 22 · see citing works
CAUTION: Popular “Benchmark” Data Sets Do Not Distinguish the Merits of 3D QSAR Methods
Manchester, J. I., & Czermiński, R. (2009) · Journal of Chemical Information and Modeling · doi:10.1021/ci9000508 · Cited by 10 · see citing works
Generalized Workflow for Generating Highly Predictive in Silico Off-Target Activity Models
Anger, L. T., Wolf, A., Schleifer, K.-J., et al. (2014) · Journal of Chemical Information and Modeling · doi:10.1021/ci500342q · Cited by 8 · see citing works
Stålring, J., Almeida, P. R., Carlsson, L., et al. (2013) · Journal of Chemical Information and Modeling · doi:10.1021/ci400281y · Cited by 7 · see citing works
Robinson, R. M. (2017) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.495163
Flagged first, then by citations
Research works printed but not linked: 2
- [8](8) Jensen, F. Chapter 17: Statistics and QSAR. In 4 % ,"% , ; John Wiley & Sons Ltd, 2007; pp 547-561. -
- [36](36) Agresti, A. Generalized Linear Models for Binary Data. In 4 % ) 8 ; Wiley Series in Probability and Statistics; WILEY6INTERSCIENCE, 2007; pp 68-72. -
Cited by
Sorted by citations
Drug discovery with explainable artificial intelligence
Jiménez-Luna, J., Grisoni, F., & Schneider, G. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00236-4 · Cited by 1,063 · see citing works
Exploiting machine learning for end-to-end drug discovery and development
Ekins, S., Puhl, A. C., Zorn, K. M., Lane, T. R., Russo, D. P., Klein, J. J., Hickey, A. J., & Clark, A. M. (2019) · Nature Materials · doi:10.1038/s41563-019-0338-z · Cited by 578 · see citing works
Vatansever, S., Schlessinger, A., Wacker, D., Kaniskan, H. Ü., Jin, J., Zhou, M., & Zhang, B. (2021) · Medicinal Research Reviews · doi:10.1002/med.21764 · Cited by 429 · see citing works
Interpretation of Quantitative Structure-Activity Relationship Models: Past, Present, and Future
Polishchuk, P. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.7b00274 · Cited by 236 · see citing works
Coloring Molecules with Explainable Artificial Intelligence for Preclinical Relevance Assessment
Jiménez-Luna, J., Skalic, M., Weskamp, N., & Schneider, G. (2021) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c01344 · Cited by 96 · see citing works
Sheridan, R. P. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.8b00825 · Cited by 84 · see citing works
Decay radius of climate decision for solar panels in the city of Fresno, USA
Barton-Henry, K., Wenz, L., & Levermann, A. (2021) · Scientific Reports · doi:10.1038/s41598-021-87714-w · Cited by 28 · see citing works
- Authors
- Marchese Robinson, R. L., Palczewska, A., Palczewski, J., & Kidley, N.
- Affiliations
- Syngenta Ltd., Jealott’s Hill International Research Centre, Bracknell, Berkshire RG42 6EY, United Kingdom
- University of Bradford (GB)
- University of Leeds (GB)
- Syngenta (United Kingdom) (GB)
- Venue
- Journal of Chemical Information and Modeling
- Publisher
- American Chemical Society (ACS)
- Published
- 2017-08-02
- Volume
- 57
- Issue
- 8
- Pages
- 1773-1792
- ISSN
- 1549-9596 (ISSN-L)
- 1549-960X
- OpenAlex
- W2736101782
- Cited by
- 167
- References
- 68
- Citations by year
- 2026: 19 · 2025: 28 · 2024: 28 · 2023: 21 · 2022: 21 · 2021: 14 · 2020: 11 · 2019: 15 · 2018: 8 · 2017: 2
- FWCI
- 7.40
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Chemical Information and Modeling (publishedVersion, public-domain)
- PubMed (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
- Keywords
- random forest, support vector regression, partial least squares, predictive performance, hit identification, toxicology, benchmark dataset
- MeSH terms
- Hot Temperature, Models, Molecular, Molecular Conformation, Linear Models, Least-Squares Analysis, Benchmarking, Quantitative Structure-Activity Relationship, Informatics, Support Vector Machine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
