Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them)

The meaning and use of the area under a receiver operating characteristic (ROC) curve.

Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works

A Simple Sequentially Rejective Multiple Test Procedure

(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526 · see citing works

The WEKA data mining software: an update

Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · see citing works

Gene expression patterns of breast carcinomas distinguish tumor subclasses with clinical implications

Sørlie, T., Perou, C., Tibshirani, R., Aas, T., Geisler, S., Johnsen, H., Hastie, T., Eisen, M., van de Rijn, M., Jeffrey, S., et al. (2001) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.191367098 · Cited by 11,695 · see citing works

Gene expression profiling predicts clinical outcome of breast cancer

van 't Veer, L. J., Dai, H., van de Vijver, M. J., He, Y. D., Hart, A. A. M., Mao, M., Peterse, H. L., van der Kooy, K., Marton, M. J., Witteveen, A. T., et al. (2002) · Nature · doi:10.1038/415530a · Cited by 9,536 · see citing works

Encyclopedia of Machine Learning

Sammut, C., & Webb, G. I. (2010) · Springer eBooks · doi:10.1007/978-0-387-30164-8 · Cited by 3,499 · see citing works

AUC: A misleading measure of the performance of predictive distribution models

Lobo, J. M., Jiménez‐Valverde, A., & Real, R. (2008) · Global Ecology and Biogeography · doi:10.1111/j.1466-8238.2007.00358.x · Cited by 3,499 · see citing works

Better Bootstrap Confidence Intervals

Efron, B. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478410 · Cited by 3,462 · see citing works

ROCR: visualizing classifier performance in R

Sing, T., Sander, O., Beerenwinkel, N., & Lengauer, T. (2005) · Computer applications in the biosciences · doi:10.1093/bioinformatics/bti623 · Cited by 3,172 · see citing works

Gene-expression profiles to predict distant metastasis of lymph-node-negative primary breast cancer

Wang, Y., Klijn, J. G. M., Zhang, Y., Sieuwerts, A. M., Look, M. P., Yang, F., Talantov, D., Timmermans, M. A. M., Gelder, M. E. M., Yu, J. X., et al. (2005) · The Lancet · doi:10.1016/s0140-6736(05)17947-1 · Cited by 2,848 · see citing works

Gene Expression Profiling in Breast Cancer: Understanding the Molecular Basis of Histologic Grade To Improve Prognosis

Sotiriou, C. P., Wirapati, P., Loi, S., Harris, A. L., Fox, S., Smeds, J., Nordgren, H., Farmer, P. J., Praz, V., Haibe‐Kains, B., et al. (2006) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djj052 · Cited by 2,060 · see citing works

Confidence intervals rather than P values: estimation rather than hypothesis testing.

Gardner, M. J., & Altman, D. G. (1986) · BMJ · doi:10.1136/bmj.292.6522.746 · Cited by 2,014 · see citing works

What's In A Name? Malay Seals As Onomastic Sources

Gallop, A. T. (2018) · Malay Literature · doi:10.37052/ml.31(1)no1 · Cited by 1,495 · see citing works

Gene-expression profiles to predict distant metastasis of lymph-node-negative primary breast cancer

Wang, Y., Klijn, J. G. M., ZHANG, Y., et al. (2005) · The Lancet · doi:10.1016/s0140-6736(05)70933-8 · Cited by 1,233 · see citing works

Measuring classifier performance: a coherent alternative to the area under the ROC curve

Hand, D. J. (2009) · Machine Learning · doi:10.1007/s10994-009-5119-5 · Cited by 1,037 · see citing works

Prediction of cancer outcome with microarrays: a multiple random validation strategy

Michiels, S., Koscielny, S., & Hill, C. (2005) · The Lancet · doi:10.1016/s0140-6736(05)17866-0 · Cited by 1,029 · see citing works

The Prognostic Role of a Gene Signature from Tumorigenic Breast-Cancer Cells

Liu, R., Wang, X., Chen, G., Dalerba, P. D., Gurney, A. L., Hoey, T. C., Sherlock, G. J., Lewicki, J. A., Shedden, K., & Clarke, M. F. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa063994 · Cited by 984 · see citing works

Robustness, scalability, and integration of a wound-response gene expression signature in predicting breast cancer survival

Chang, H. Y., Nuyten, D. S. A., Sneddon, J. B., Hastie, T., Tibshirani, R., Sørlie, T., Dai, H., He, Y. D., van't Veer, L. J., Bartelink, H., et al. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0409462102 · Cited by 981 · see citing works

Gene expression profiling for the prediction of therapeutic response to docetaxel in patients with breast cancer

Chang, J. C., Wooten, E. C., Tsimelzon, A. I., Hilsenbeck, S. G., Gutierrez, M. C., Elledge, R. M., Mohsin, S. K., Osborne, C. K., Chamness, G. C., Allred, D. C., et al. (2003) · The Lancet · doi:10.1016/s0140-6736(03)14023-8 · Cited by 875 · see citing works

Classifier Technology and the Illusion of Progress

Hand, D. J. (2006) · Statistical Science · doi:10.1214/088342306000000060 · Cited by 723 · see citing works

Pharmacogenomic Predictor of Sensitivity to Preoperative Chemotherapy With Paclitaxel and Fluorouracil, Doxorubicin, and Cyclophosphamide in Breast Cancer

Hess, K. R., Anderson, K., Symmans, W. F., Valero, V., Ibrahim, N. K., Mejia, J. A., Booser, D. J., Theriault, R. L., Buzdar, A. U., Dempsey, P. J., et al. (2006) · Journal of Clinical Oncology · doi:10.1200/jco.2006.05.6861 · Cited by 720 · see citing works

Better Bootstrap Confidence Intervals

Efron, B. (1987) · Journal of the American Statistical Association · doi:10.2307/2289144 · Cited by 718 · see citing works

Analyzing a Portion of the ROC Curve

McClish, D. K. (1989) · Medical Decision Making · doi:10.1177/0272989x8900900307 · Cited by 709 · see citing works

Gene Expression Programs in Response to Hypoxia: Cell Type Specificity and Prognostic Significance in Human Cancers

Chi, J., Wang, Z., Nuyten, D. S. A., Rodriguez, E. H., Schaner, M. E., Salim, A., Wang, Y., Kristensen, G. B., Helland, Å., Børresen‐Dale, A., et al. (2006) · PLoS Medicine · doi:10.1371/journal.pmed.0030047 · Cited by 622 · see citing works

Gene Expression Profiles Predict Complete Pathologic Response to Neoadjuvant Paclitaxel and Fluorouracil, Doxorubicin, and Cyclophosphamide Chemotherapy in Breast Cancer

Ayers, M., Symmans, W. F., Stec, J., Damokosh, A. I., Clark, E., Hess, K., Lecocke, M., Metivier, J., Booser, D., Ibrahim, N., et al. (2004) · Journal of Clinical Oncology · doi:10.1200/jco.2004.05.166 · Cited by 566 · see citing works

AUC: a statistically consistent and more discriminating measure than accuracy

Ling, C. X., Huang, J., & Zhang, H. (2003) · Cited by 501 · see citing works

Bias as a threat to the validity of cancer molecular-marker research

Ransohoff, D. F. (2005) · Nature reviews. Cancer · doi:10.1038/nrc1550 · Cited by 480 · see citing works

Roadmap for Developing and Validating Therapeutically Relevant Genomic Classifiers

Simon, R. M. (2005) · Journal of Clinical Oncology · doi:10.1200/jco.2005.02.8712 · Cited by 382 · see citing works

Low P-Values or Narrow Confidence Intervals: Which Are More Durable?

Poole, C. (2001) · Epidemiology · doi:10.1097/00001648-200105000-00005 · Cited by 363 · see citing works

ROC Curves in Clinical Chemistry: Uses, Misuses, and Possible Solutions

Obuchowski, N. A., Lieber, M. L., & Wians, F. H. (2004) · Clinical Chemistry · doi:10.1373/clinchem.2004.031823 · Cited by 358 · see citing works

Molecular Classification of Tamoxifen-Resistant Breast Carcinomas by Gene Expression Profiling

Jansen, M. P. H. M., Foekens, J. A., van Staveren, I. L., Dirkzwager-Kiel, M. M., Ruigrok-Ritstier, K., Look, M. P., Gelder, M. E. M., Sieuwerts, A. M., Portengen, H., Dorssers, L. C. J., et al. (2005) · Journal of Clinical Oncology · doi:10.1200/jco.2005.05.145 · Cited by 340 · see citing works

Cost curves: An improved method for visualizing classifier performance

Drummond, C., & Holte, R. C. (2006) · Machine Learning · doi:10.1007/s10994-006-8199-5 · Cited by 328 · see citing works

Small-sample precision of ROC-related estimates

Hanczar, B., Hua, J., Sima, C., Weinstein, J., Bittner, M., & Dougherty, E. R. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq037 · Cited by 323 · see citing works

Deriving chemosensitivity from cell lines: Forensic bioinformatics and reproducible research in high-throughput biology

Baggerly, K., & Coombes, K. R. (2009) · The Annals of Applied Statistics · doi:10.1214/09-aoas291 · Cited by 311 · see citing works

Prediction of Docetaxel Response in Human Breast Cancer by Gene Expression Profiling

Iwao‐Koizumi, K., Matoba, R., Ueno, N., Kim, S. J., Ando, A., Miyoshi, Y., Maeda, E., Noguchi, S., & Kato, K. (2005) · Journal of Clinical Oncology · doi:10.1200/jco.2005.09.078 · Cited by 309 · see citing works

Receiver-operating characteristic curve analysis in diagnostic, prognostic and predictive biomarker research

Søreide, K. (2008) · Journal of Clinical Pathology · doi:10.1136/jcp.2008.061010 · Cited by 301 · see citing works

A show of confidence.

Rothman, K. J. (1978) · New England Journal of Medicine · doi:10.1056/nejm197812142992410 · Cited by 281 · see citing works

Use of 70-gene signature to predict prognosis of patients with node-negative breast cancer: a prospective community-based feasibility study (RASTER)

Bueno-de-Mesquita, J. M., van Harten, W. H., Retèl, V. P., Veer, L. V. ‘., van Dam, F. S. A. M., Karsenberg, K., Douma, K. F. L., van Tinteren, H., Peterse, J. L., Wesseling, J., et al. (2007) · The Lancet Oncology · doi:10.1016/s1470-2045(07)70346-7 · Cited by 274 · see citing works

StAR: a simple tool for the statistical comparison of ROC curves

Vergara, I. A., Norambuena, T., Ferrada, E., Slater, A. W., & Melo, F. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-265 · Cited by 196 · see citing works

The Area under the ROC Curve and Its Competitors

Hilden, J. (1991) · Medical Decision Making · doi:10.1177/0272989x9101100204 · Cited by 183 · see citing works

Comparing classifiers when the misallocation costs are uncertain

Adams, N. M., & Hand, D. J. (1999) · Pattern Recognition · doi:10.1016/s0031-3203(98)00154-x · Cited by 171 · see citing works

Confidence Intervals for the Area Under the ROC Curve

Cortes, C., & Mohri, M. (2004) · neural information processing systems · Cited by 158 · see citing works

A CROC stronger than ROC: measuring, visualizing and optimizing early retrieval

Swamidass, S. J., Azencott, C., Daily, K., & Baldi, P. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq140 · Cited by 115 · see citing works

Managing bias in ROC curves

Clark, R. D., & Webster-Clark, D. J. (2008) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-008-9181-z · Cited by 110 · see citing works

Diagnostic Accuracy and Receiver-Operating Characteristics Curve Analysis in Surgical Research and Decision Making

Søreide, K., Kørner, H., & Søreide, J. A. (2011) · Annals of Surgery · doi:10.1097/sla.0b013e318204a892 · Cited by 109 · see citing works

Writing for Epidemiology

Rothman, K. J. (1998) · Epidemiology · doi:10.1097/00001648-199805000-00019 · Cited by 102 · see citing works

Genomics, Prior Probability, and Statistical Tests of Multiple HypothesesPreprint

Manly, K. F., Nettleton, D. S., & Hwang, J. T. G. (2004) · Genome Research · doi:10.1101/gr.2156804 · Cited by 86 · see citing works

Run Batch Effects Potentially Compromise the Usefulness of Genomic Signatures for Ovarian Cancer

Baggerly, K., Coombes, K. R., & Neeley, E. S. (2008) · Journal of Clinical Oncology · doi:10.1200/jco.2007.15.1951 · Cited by 72 · see citing works

A critical analysis of variants of the AUC

Vanderlooy, S., & Hüllermeier, E. (2008) · Machine Learning · doi:10.1007/s10994-008-5070-x · Cited by 59 · see citing works

Supervised analysis when the number of candidate features (p) greatly exceeds the number of cases (n)

Simon, R. M. (2003) · ACM SIGKDD Explorations Newsletter · doi:10.1145/980972.980978 · Cited by 54 · see citing works

Statistical Issues in Translational Cancer Research

George, S. L. (2008) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-07-4537 · Cited by 43 · see citing works

Gene expression variation to predict 10-year survival in lymph-node-negative breast cancer

Karlsson, E. A., Delle, U., Danielsson, A., Olsson, B. E., Abel, F., Karlsson, P., & Helou, K. (2008) · BMC Cancer · doi:10.1186/1471-2407-8-254 · Cited by 39 · see citing works

Microarrays and breast cancer clinical studies: forgetting what we have not yet learnt

Ahmed, A. A., & Brenton, J. D. (2005) · Breast Cancer Research · doi:10.1186/bcr1017 · Cited by 29 · see citing works

ON THE DISTRIBUTION OF THE UNPAIRED t-STATISTIC WITH PAIRED DATA

Proschan, M. A. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19960530)15:10 · Cited by 23 · see citing works

What Properties Should an Overall Measure of Test Performance Possess?

Hilden, J. (2005) · Clinical Chemistry · doi:10.1373/clinchem.2004.041376 · Cited by 12 · see citing works

Improving Text Classifier Performance based on AUC

Wong, A., Lee, J., & Yeung, D. S. (2006) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2006.705 · Cited by 4 · see citing works

The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets

Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309 · see citing works

Translational biomarker discovery in clinical metabolomics: an introductory tutorial

Xia, J., Broadhurst, D., Wilson, M., & Wishart, D. S. (2012) · Metabolomics · doi:10.1007/s11306-012-0482-9 · Cited by 958 · see citing works

The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification

Chicco, D., & Jurman, G. (2023) · BioData Mining · doi:10.1186/s13040-023-00322-4 · Cited by 667 · see citing works

Patient clustering improves efficiency of federated machine learning to predict mortality and hospital stay time using distributed electronic medical records

Huang, L., Shea, A. L., Qian, H., Masurkar, A., Deng, H., & Liu, D. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103291 · Cited by 437 · see citing works

Bayes’ Theorem and Naive Bayes Classifier

Berrar, D. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-809633-8.20473-1 · Cited by 442 · see citing works

Redox-sensitive trace metals as paleoredox proxies: A review and analysis of data from modern sediments

Bennett, W. W., & Canfield, D. E. (2020) · Earth-Science Reviews · doi:10.1016/j.earscirev.2020.103175 · Cited by 374 · see citing works

The receiver operating characteristic (ROC) curve

Yang, S., & Berdine, G. G. (2017) · The Southwest Respiratory and Critical Care Chronicles · doi:10.12746/swrccc.v5i19.391 · Cited by 250 · see citing works

Chronic Stress Induces Activity, Synaptic, and Transcriptional Remodeling of the Lateral Habenula Associated with Deficits in Motivated Behaviors

Cerniauskas, I., Winterer, J., de Jong, J. W., Lukacsovich, D., Yang, H., Khan, F., Peck, J. R., Obayashi, S. K., Lilascharoen, V., Lim, B. K., et al. (2019) · Neuron · doi:10.1016/j.neuron.2019.09.005 · Cited by 187 · see citing works

Somatic Mutations and Neoepitope Homology in Melanomas Treated with CTLA-4 Blockade

Nathanson, T., Ahuja, A., Rubinsteyn, A., Aksoy, B. A., Hellmann, M. D., Miao, D., Allen, E. M. V., Merghoub, T., Wolchok, J. D., Snyder, A., et al. (2016) · Cancer Immunology Research · doi:10.1158/2326-6066.cir-16-0019 · Cited by 208 · see citing works

Comparison of the Predictive Performance and Interpretability of Random Forest and Linear Models on Benchmark Data Sets

Marchese Robinson, R. L., Palczewska, A., Palczewski, J., & Kidley, N. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6b00753 · Cited by 167 · see citing works

Computational models for predicting drug responses in cancer research

Azuaje, F. J. (2016) · Briefings in Bioinformatics · doi:10.1093/bib/bbw065 · Cited by 162 · see citing works

Performance Evaluation in Machine Learning: The Good, the Bad, the Ugly, and the Way Forward

Flach, P. A. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019808 · Cited by 159 · see citing works

Rethinking Drug Repositioning and Development with Artificial Intelligence, Machine Learning, and Omics

Koromina, M., Pandi, M.-T., & Patrinos, G. P. (2019) · OMICS A Journal of Integrative Biology · doi:10.1089/omi.2019.0151 · Cited by 140 · see citing works

Assessing technical performance in differential gene expression experiments with external spike-in RNA control ratio mixtures

Munro, S. A., Lund, S. P., Pine, P. S., Binder, H., Clevert, D.-A., Conesa, A., Dopazo, J., Fasold, M., Hochreiter, S., Hong, H., et al. (2014) · Nature Communications · doi:10.1038/ncomms6125 · Cited by 126 · see citing works

Area under the ROC Curve has the most consistent evaluation for binary classification

Li, J. (2024) · PLoS ONE · doi:10.1371/journal.pone.0316019 · Cited by 117 · see citing works

HE4, CA125 and risk of ovarian malignancy algorithm (ROMA) as diagnostic tools for ovarian cancer in patients with a pelvic mass: An Italian multicenter study

Romagnolo, C., Leon, A. E., Fabricio, A. S. C., Taborelli, M., Polesel, J., Pup, L. D., Steffan, A., Cervo, S., Ravaggi, A., Zanotti, L., et al. (2016) · Gynecologic Oncology · doi:10.1016/j.ygyno.2016.01.016 · Cited by 113 · see citing works

Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance

Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112 · see citing works

Three-dimensional hierarchical plasmonic nano-architecture based label-free surface-enhanced Raman spectroscopy detection of urinary exosomal miRNA for clinical diagnosis of prostate cancer

Kim, W. H., Lee, J. U., Jeon, M. J., Park, K. H., & Sim, S. J. (2022) · Biosensors and Bioelectronics · doi:10.1016/j.bios.2022.114116 · Cited by 96 · see citing works

Performance Measures for Binary Classification

Berrar, D. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-809633-8.20351-8 · Cited by 96 · see citing works

Model Evaluation Guidelines for Geomagnetic Index Predictions

Liemohn, M. W., McCollough, J. P., Jordanova, V. K., Ngwira, C. M., Morley, S. K., Cid, C., Tobiska, W. K., Wintoft, P., Ganushkina, N., Welling, D. T., et al. (2018) · Space Weather · doi:10.1029/2018sw002067 · Cited by 86 · see citing works

Genetic-based prediction of disease traits: prediction is very difficult, especially about the future†

Schrodi, S. J., Mukherjee, S., Shan, Y., Tromp, G. C., Sninsky, J. J., Callear, A. P., Carter, T. C., Ye, Z., Haines, J. L., Brilliant, M. H., et al. (2014) · Frontiers in Genetics · doi:10.3389/fgene.2014.00162 · Cited by 71 · see citing works

Effects of Study Population, Labeling and Training on Glaucoma Detection Using Deep Learning Algorithms

Christopher, M. A., Nakahara, K., Bowd, C., Proudfoot, J. A., Belghith, A., Goldbaum, M. H., Rezapour, J., Weinreb, R. N., Fazio, M. A., Girkin, C. A., et al. (2020) · Translational Vision Science & Technology · doi:10.1167/tvst.9.2.27 · Cited by 65 · see citing works

Limitations of receiver operating characteristic curve on imbalanced data: Assist device mortality risk scores

Movahedi, F., Padman, R., & Antaki, J. F. (2021) · Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2021.07.041 · Cited by 63 · see citing works

PToPI: A Comprehensive Review, Analysis, and Knowledge Representation of Binary Classification Performance Measures/Metrics

Canbek, G., Temizel, T. T., & Sağıroğlu, Ş. (2022) · SN Computer Science · doi:10.1007/s42979-022-01409-1 · Cited by 60 · see citing works

Navigating the metric maze: a taxonomy of evaluation metrics for anomaly detection in time series

Sørbø, S., & Ruocco, M. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00988-8 · Cited by 58 · see citing works

Deep Learning-Based Damage Mapping With InSAR Coherence Time Series

Stephenson, O. L., Köhne, T., Zhan, E., Cahill, B. E., Yun, S., Ross, Z. E., & Simons, M. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3084209 · Cited by 54 · see citing works

A urinary microRNA panel that is an early predictive biomarker of delayed graft function following kidney transplantation

Khalid, U., Newbury, L. J., Simpson, K., Jenkins, R. L., Bowen, T., Bates, L., Sheerin, N., Chavez, R., & Fraser, D. J. (2019) · Scientific Reports · doi:10.1038/s41598-019-38642-3 · Cited by 51 · see citing works

Optimizing area under the ROC curve using semi-supervised learning

Wang, S., Li, D., Petrick, N., Sahiner, B., Linguraru, M. G., & Summers, R. M. (2014) · Pattern Recognition · doi:10.1016/j.patcog.2014.07.025 · Cited by 48 · see citing works

ROC Curves for the Statistical Analysis of Microarray Data

Cao, R., & López‐de‐Ullibarri, I. (2019) · Methods in molecular biology · doi:10.1007/978-1-4939-9442-7_11 · Cited by 48 · see citing works

DeepGAMI: deep biologically guided auxiliary learning for multimodal integration and imputation to improve genotype-phenotype prediction

Chandrashekar, P. B., Alatkar, S., Wang, J., Hoffman, G. E., He, C., Jin, T., Khullar, S., Bendl, J., Fullard, J. F., Roussos, P., et al. (2023) · Genome Medicine · doi:10.1186/s13073-023-01248-6 · Cited by 37 · see citing works

Non-obese lipedema patients show a distinctly altered quantitative sensory testing profile with high diagnostic potential

Dinnendahl, R., Tschimmel, D., Löw, V., Cornely, M. E., & Hucho, T. B. (2024) · PAIN Reports · doi:10.1097/pr9.0000000000001155 · Cited by 38 · see citing works

Impact of Bioinformatic Procedures in the Development and Translation of High-Throughput Molecular Classifiers in Oncology

Ferté, C., Trister, A. D., Huang, E. S., Bot, B. M., Guinney, J., Commo, F., Sieberts, S. K., André, F., Besse, B., Soria, J., et al. (2013) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-12-3937 · Cited by 36 · see citing works

Sighting acute myocardial infarction through platelet gene expression

Gobbi, G., Carubbi, C., Tagliazucchi, G. M., Masselli, E., Mirandola, P., Pigazzani, F., Crocamo, A., Notarangelo, M. F., Suma, S., Paraboschi, E. M., et al. (2019) · Scientific Reports · doi:10.1038/s41598-019-56047-0 · Cited by 36 · see citing works

Unraveling networks of co-regulated genes on the sole basis of genome sequences

Brohée, S., Janky, R., Abdel‐Sater, F., Vanderstocken, G., André, B., & van Helden, J. (2011) · Nucleic Acids Research · doi:10.1093/nar/gkr264 · Cited by 36 · see citing works

1H NMR spectroscopy-based metabolomics analysis for the diagnosis of symptomatic E. coli-associated urinary tract infection (UTI)

Lussu, M., Camboni, T., Piras, C., Serra, C., Carratore, F. D., Griffin, J. L., Atzori, L., & Manzin, A. (2017) · BMC Microbiology · doi:10.1186/s12866-017-1108-1 · Cited by 33 · see citing works

ROC curve equivalence using the Kolmogorov–Smirnov test

Bradley, A. P. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2012.12.021 · Cited by 31 · see citing works

Combination of conventional culture, vial culture, and broad-range PCR of sonication fluid for the diagnosis of prosthetic joint infection

Stylianakis, A., Schinas, G., Thomaidis, P. C., Papaparaskevas, J., Ziogas, D. C., Gamaletsou, M. N., Daikos, G. L., Pneumaticos, S. G., & Sipsas, N. V. (2018) · Diagnostic Microbiology and Infectious Disease · doi:10.1016/j.diagmicrobio.2018.04.008 · Cited by 31 · see citing works

MicroRNAs in porcine uterus and serum are affected by zearalenone and represent a new target for mycotoxin biomarker discovery

Grenier, B., Hackl, M., Skalicky, S., Thamhesl, M., Moll, W.-D., Berrios, R., Schatzmayr, G., & Nagl, V. (2019) · Scientific Reports · doi:10.1038/s41598-019-45784-x · Cited by 29 · see citing works

Drug2ways: Reasoning over causal paths in biological networks for drug discovery

Rivas-Barragan, D., Mubeen, S., Guim, F., Hofmann‐Apitius, M., & Domingo‐Fernándéz, D. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008464 · Cited by 28 · see citing works

A Model-Agnostic Interpretability Approach to Predicting Customer Churn in the Telecommunications Industry

Noviandy, T. R., Idroes, G. M., Hardi, I., Afjal, M., & Ray, S. (2024) · Infolitika Journal of Data Science · doi:10.60084/ijds.v2i1.199 · Cited by 27 · see citing works

A Somatic Coliphage Threshold Approach To Improve the Management of Activated Sludge Wastewater Treatment Plant Effluents in Resource-Limited Regions

Chacón, L., Barrantes, K., Santamaría‐Ulloa, C., Barquero, M. S., Reyes, L., Taylor, L., Valiente, C., Symonds, E. M., & Achí, R. (2020) · Applied and Environmental Microbiology · doi:10.1128/aem.00616-20 · Cited by 26 · see citing works

Machine learning-based predictive models for perioperative major adverse cardiovascular events in patients with stable coronary artery disease undergoing noncardiac surgery

Shen, L., Jin, Y., Pan, A., Wang, K., Ye, R., Lin, Y., Anwar, S., Xia, W., Zhou, M., & Guo, X. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108561 · Cited by 25 · see citing works

Pathogenic potential assessment of the Shiga toxin–producing Escherichia coli by a source attribution–considered machine learning model

Im, H., Hwang, S.-H., Kim, B., & Choi, S.-H. (2021) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.2018877118 · Cited by 16 · see citing works

BioInfra.Prot: A comprehensive proteomics workflow including data standardization, protein inference, expression analysis and data publication

Turewicz, M., Kohl, M., Ahrens, M., Mayer, G., Uszkoreit, J., Naboulsi, W., Bracht, T., Megger, D. A., Sitek, B., Marcus, K., et al. (2017) · Journal of Biotechnology · doi:10.1016/j.jbiotec.2017.06.005 · Cited by 24 · see citing works

In Vitro Functional Characterization and in Silico Prediction of Rare Genetic Variation in the Bile Acid and Drug Transporter, Na+-Taurocholate Cotransporting Polypeptide (NTCP, SLC10A1)

Russell, L. E., Zhou, Y., Lauschke, V. M., & Kim, R. B. (2020) · Molecular Pharmaceutics · doi:10.1021/acs.molpharmaceut.9b01200 · Cited by 24 · see citing works

Using predictive machine learning models for drug response simulation by calibrating patient-specific pathway signatures

Khatami, S. G., Mubeen, S., Bharadhwaj, V. S., Kodamullil, A. T., Hofmann‐Apitius, M., & Domingo‐Fernándéz, D. (2021) · npj Systems Biology and Applications · doi:10.1038/s41540-021-00199-1 · Cited by 24 · see citing works

ROCS: Receiver Operating Characteristic Surface for Class-Skewed High-Throughput Data

Yu, T. (2012) · PLoS ONE · doi:10.1371/journal.pone.0040598 · Cited by 23 · see citing works

IoT-based expert system for fault detection in Japanese Plum leaf-turgor pressure WSN

Barriga, A., Barriga, J. A., Espino, M. J. M., & Clemente, P. J. (2023) · Internet of Things · doi:10.1016/j.iot.2023.100829 · Cited by 22 · see citing works

Bayes’ Theorem and Naive Bayes Classifier

Berrar, D. (2024) · Elsevier eBooks · doi:10.1016/b978-0-323-95502-7.00118-4 · Cited by 22 · see citing works

Identification of 14-3-3 epsilon as a regulator of the neural apoptotic pathway for chronic-stress-induced depression

Zhao, Y., Coulson, E. J., Su, X., Zhang, J., Sha, B., Xu, H., Deng, Y., Chen, Y., Cao, J. J., Wang, Y., et al. (2021) · iScience · doi:10.1016/j.isci.2021.102043 · Cited by 20 · see citing works

Chronic stress causes striatal disinhibition mediated by SOM-interneurons in male mice

Rodrigues, D., Jacinto, L., Falcão, M., Castro, A. C., Cruz, A., Santa, C., Manadas, B. J., Marques, F., Sousa, N., & Monteiro, P. (2022) · Nature Communications · doi:10.1038/s41467-022-35028-4 · Cited by 19 · see citing works

Evaluating Probabilistic Forecasts with Bayesian Signal Detection Models

Steyvers, M., Wallsten, T. S., Merkle, E. C., & Turner, B. M. (2013) · Risk Analysis · doi:10.1111/risa.12127 · Cited by 19 · see citing works

A real use case of semi-supervised learning for mammogram classification in a local clinic of Costa Rica

Calderon-Ramirez, S., Murillo-Hernandez, D., Rojas-Salazar, K., Elizondo, D. A., Yang, S., Moemeni, A., & Molina‐Cabello, M. A. (2022) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-021-02497-6 · Cited by 19 · see citing works

Pediatric contact allergy: A comparative study with adults

Boonchai, W., Chaiyabutr, C., Charoenpipatsin, N., & Sukakul, T. (2020) · Contact Dermatitis · doi:10.1111/cod.13672 · Cited by 18 · see citing works

The Role of the Multi-Inflammatory Index as a Novel Predictor of Hospital Mortality in Acute Ischemic Stroke

Demirel, M. E., & Türel, C. A. (2023) · Cureus · doi:10.7759/cureus.43258 · Cited by 18 · see citing works

Screening for intellectual disabilities: a validation of the Hayes Ability Screening Index for in-patients with substance use disorder

Braatveit, K. J., Torsheim, T., & Hove, O. (2018) · Nordic Journal of Psychiatry · doi:10.1080/08039488.2018.1468923 · Cited by 18 · see citing works

On the validity of time-dependent AUC estimators

Schmid, M. C., Kestler, H. A., & Potapov, S. (2013) · Briefings in Bioinformatics · doi:10.1093/bib/bbt059 · Cited by 18 · see citing works

Improving Performance Prediction on Education Data with Noise and Class Imbalance

Radwan, A. M., & Cataltepe, Z. (2017) · Intelligent Automation & Soft Computing · doi:10.1080/10798587.2017.1337673 · Cited by 16 · see citing works

Pathway-Dependent Effectiveness of Network Algorithms for Gene Prioritization

Shim, J. E., Hwang, S., & Lee, I. (2015) · PLoS ONE · doi:10.1371/journal.pone.0130589 · Cited by 17 · see citing works

Causal reasoning over knowledge graphs leveraging drug-perturbed and disease-specific transcriptomic signatures for drug discovery

Domingo‐Fernándéz, D., Gadiya, Y., Patel, A., Mubeen, S., Rivas-Barragan, D., Diana, C. W., Misra, B. B., Healey, D., Rokicki, J., & Colluru, V. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009909 · Cited by 15 · see citing works

Explainable machine learning for real-time deterioration alert prediction to guide pre-emptive treatment

Brankovic, A., Hassanzadeh, H., Good, N. M., Mann, K. D., Khanna, S., Abdel‐Hafez, A., & Cook, D. (2022) · Scientific Reports · doi:10.1038/s41598-022-15877-1 · Cited by 15 · see citing works

Early identification of bovine pregnancy status and embryonic mortality

Bishop, J. V., Güzeloğlu, A., Scheller, T., Docheff, J. J., Gonzalez-Berrios, C. L., Campen, H. V., Nett, T. M., Zezeski, A. L., Geary, T. W., Thatcher, W. W., et al. (2025) · Biology of Reproduction · doi:10.1093/biolre/ioaf066 · Cited by 14 · see citing works

Structure-based discovery and bio-evaluation of a cyclopenta[4,5]thieno[2,3- d ]pyrimidin-4-one as a phosphodiesterase 10A inhibitor

Al-Nema, M. Y., Gaurav, A., Lee, V. S., Gunasekaran, B., Lee, M. T., Okechukwu, P. N., & Nimmanpipug, P. (2022) · RSC Advances · doi:10.1039/d1ra07649c · Cited by 14 · see citing works

Discovery of natural product inhibitors of phosphodiesterase 10A as novel therapeutic drug for schizophrenia using a multistep virtual screening

Al-Nema, M. Y., Gaurav, A., & Akowuah, G. A. (2018) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2018.09.001 · Cited by 12 · see citing works

Rainfall Predictive Approach for La Trinidad, Benguet using Machine Learning Classification

Macabiog, R. E. N., & Cruz, J. C. D. (2019) · IEEE International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management (HNICEM) · doi:10.1109/hnicem48295.2019.9072761 · Cited by 14 · see citing works

Pressure Injury Prediction in Intensive Care Units Using Artificial Intelligence: A Scoping Review

Alves, J., Azevedo, R., Marques, A., Encarnação, R., & Alves, P. J. P. (2025) · Nursing Reports · doi:10.3390/nursrep15040126 · Cited by 12 · see citing works

Enhancing prediction on imbalance data by thresholding technique with noise filtering

Radwan, A. M. (2017) · doi:10.1109/icitech.2017.8080033 · Cited by 13 · see citing works

Novel nomogram for predicting paradoxical chest wall movement in patients with flail segment of traumatic rib fracture: a retrospective cohort study

Seok, J., Jeong, S. T., Yoon, S. Y., Lee, J. Y., Kim, S., Cho, H., & Kang, W. S. (2023) · Scientific Reports · doi:10.1038/s41598-023-47700-w · Cited by 12 · see citing works

Usefulness of postural sway spectral analysis in the diagnostic route and clinical integration of cervicogenic and vestibular sources of dizziness: A cross-sectional preliminary study

Micarelli, A., Viziano, A., Micarelli, B., Fulvio, G. D., & Alessandrini, M. (2021) · Journal of Vestibular Research · doi:10.3233/ves-190729 · Cited by 12 · see citing works

Stress induces behavioral disorders by promoting microglial phagocytosis via decreasing neuronal Dkk3

Chen, X., Zhang, K., Li, Y., Wang, C., Lou, H., & Yu, S. Y. (2025) · Molecular Psychiatry · doi:10.1038/s41380-025-03314-4 · Cited by 14 · see citing works

Machine learning model with output correction: Towards reliable bradycardia detection in neonates

Rahman, J. S., Brankovic, A., & Khanna, S. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108658 · Cited by 11 · see citing works

A Comparison of T 2 Relaxation-Based MRI Stroke Timing Methods in Hyperacute Ischemic Stroke Patients: A Pilot Study

McGarry, B. L., Damion, R. A., Chew, I., Knight, M. J., Harston, G., Carone, D., Jezzard, P., Sitaram, A., Muir, K. W., Clatworthy, P. L., et al. (2020) · Journal of Central Nervous System Disease · doi:10.1177/1179573520943314 · Cited by 11 · see citing works

Deep learning as a predictive tool for fetal heart pregnancy following time-lapse incubation and blastocyst transfer

Chávez-Badiola, A., Mendizabal‐Ruiz, G., Farias, A. F.-S., Garcia-Sánchez, R., & Drakeley, A. (2019) · Human Reproduction · doi:10.1093/humrep/dez263 · Cited by 10 · see citing works

Performance metric curve analysis framework to assess impact of the decision variable threshold, disease prevalence, and dataset variability in two-class classification

Whitney, H. M., Drukker, K., & Giger, M. L. (2022) · Journal of medical imaging · doi:10.1117/1.jmi.9.3.035502 · Cited by 9 · see citing works

A Comparative Analysis of Novel Biomarkers in Sepsis and Cardiovascular Disease

Jirak, P., Haertel, F., Mirna, M., Rezar, R., Lichtenauer, M., Paar, V., Motloch, L. J., Topf, A., Yılmaz, A., Hoppe, U. C., et al. (2022) · Applied Sciences · doi:10.3390/app12031419 · Cited by 9 · see citing works

Half-AUC for the evaluation of sensitive or specific classifiers

Bradley, A. P. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2013.11.015 · Cited by 9 · see citing works

An Empirical Evaluation of Ranking Measures With Respect to Robustness to Noise

Berrar, D. (2014) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4136 · Cited by 9 · see citing works

Identifying diverse metal oxide nanomaterials with lethal effects on embryonic zebrafish using machine learning

Robinson, R. M., Sarimveis, H., Doganis, P., Jia, X., Kotzabasaki, M. I., Gousiadou, C., Harper, S. L., & Wilkins, T. (2021) · Beilstein Journal of Nanotechnology · doi:10.3762/bjnano.12.97 · Cited by 7 · see citing works

What (not) to expect when classifying rare events

Blagus, R., & Goeman, J. J. (2016) · Briefings in Bioinformatics · doi:10.1093/bib/bbw107 · Cited by 8 · see citing works

N6-Methyladenosine-Related Long Non-Coding RNAs Are Identified as a Potential Prognostic Biomarker for Lung Squamous Cell Carcinoma and Validated by Real-Time PCR

Zhang, W., Zhang, Q., Xie, Z., Che, L., Xia, T., Cai, X., & Liu, S. (2022) · Frontiers in Genetics · doi:10.3389/fgene.2022.839957 · Cited by 8 · see citing works

Changes in serum amyloid A, plasma high-density lipoprotein cholesterol and apolipoprotein A-I as useful biomarkers for Mycobacterium tuberculosis infection

Fontes, C. F., Bidu, N. S., Freitas, F. R., Maranhão, R. C., Monteiro, A. D. S. S., Couto, R. D., & Netto, E. M. (2023) · Journal of Medical Microbiology · doi:10.1099/jmm.0.001726 · Cited by 8 · see citing works

Development and Characterization of a Multiplex Assay to Quantify Complement-Fixing Antibodies against Dengue Virus

Nascimento, E. J. M., Norwood, B., Parker, A., Braun, R. P., Kpamegan, E. P., & Dean, H. J. (2021) · International Journal of Molecular Sciences · doi:10.3390/ijms222112004 · Cited by 7 · see citing works

Identifying patients at risk of unplanned re-hospitalisation using statewide electronic health records

Brankovic, A., Rolls, D. A., Boyle, J., Niven, P., & Khanna, S. (2022) · Scientific Reports · doi:10.1038/s41598-022-20907-z · Cited by 7 · see citing works

Partner‐specific prediction of RNA‐binding residues in proteins: A critical assessment

Jung, Y. G., EL‐Manzalawy, Y., Dobbs, D. L., & Honavar, V. (2018) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25639 · Cited by 7 · see citing works

Informational analysis: a Shannon theoretic approach to measure the performance of a diagnostic test

Girometti, R., & Fabris, F. (2015) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-015-1294-7 · Cited by 7 · see citing works

A critical review of the hemodynamics in assessment of volume responsiveness by using passive leg raising (PLR)

Söndergaard, S. (2023) · Trends in Anaesthesia and Critical Care · doi:10.1016/j.tacc.2023.101292 · Cited by 6 · see citing works

Acute myocardial injury secondary to severe acute liver failure: A retrospective analysis supported by animal data

Uhlig, M., Hein, M., Habigt, M., Tolba, R. H., Braunschweig, T. A., Helmedag, M. J., Klinge, U., Koch, A. H., Trautwein, C., & Mechelinck, M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0256790 · Cited by 6 · see citing works

Supervised Machine Learning Thyroid Carcinoma Diagnosis Using Wide-Field SHG Microscopy

Padrez, Y., Hristu, R., Timoshchenko, I. A., Eftimie, L. G., Rutkauskas, D., & Golubewa, L. (2025) · IEEE Access · doi:10.1109/access.2025.3583435 · Cited by 5 · see citing works

EEG microstate analysis in children with prolonged disorders of consciousness

Zhang, Y., Hui, Z., Su, Y., Qi, W., Zhang, G., Zhou, L. F., Zhang, J., Shi, K., Yang, Y., Yang, L., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-11038-2 · Cited by 5 · see citing works

External validation of the German Registry for Acute Aortic Dissection Type A score in patients undergoing surgery for acute type A aortic dissection

Ahmad, D., Serna-Gallegos, D. R., Jackson, A., Kaczorowski, D. J., Bonatti, J. O., West, D. M., Yoon, P., Chu, D., Squire, J., Thoma, F. W., et al. (2024) · JTCVS Open · doi:10.1016/j.xjon.2024.12.007 · Cited by 5 · see citing works

Direct detection of 4-dimensions of SARS-CoV-2: infection (vRNA), infectivity (antigen), binding antibody, and functional neutralizing antibody in saliva

Mohammadi, A., Chiang, S. H., Li, F., Wei, F., Lau, C. S., Aziz, M. A., Ibarrondo, F. J., Fulcher, J. A., Yang, O. O., Chia, D., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-81019-4 · Cited by 5 · see citing works

Prediction of protein-DNA interactions of transcription factors linking proteomics and transcriptomics data

Kondrakhin, Y. V., Valeev, T., Sharipov, R., Yevshin, I., Kolpakov, F., & Kel, A. E. (2016) · EuPA Open Proteomics · doi:10.1016/j.euprot.2016.09.001 · Cited by 4 · see citing works

Multiplex Serology for Sensitive and Specific Flavivirus IgG Detection: Addition of Envelope Protein Domain III to NS1 Increases Sensitivity for Tick-Borne Encephalitis Virus IgG Detection

Valle, C., Shrestha, S., Godeke, G.-J., Hoogerwerf, M. N., Reimerink, J. H. J., Eggink, D., & Reusken, C. B. E. M. (2024) · Viruses · doi:10.3390/v16020286 · Cited by 4 · see citing works

Anomaly Detection in Spacecraft Telemetry: Forecasting vs. Classification

Lakey, D. T., & Schlippe, T. (2024) · IEEE Space Computing Conference (SCC) · doi:10.1109/scc61854.2024.00009 · Cited by 4 · see citing works

Rate-Oriented Point-Wise Confidence Bounds for ROC Curves

Millard, L. A. C., Kull, M., & Flach, P. A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44851-9_26 · Cited by 4 · see citing works

Evaluation of an optimal cutoff of parathyroid venous sampling gradient for localizing primary hyperparathyroidism

Lee, J., Hong, N., Kim, B. M., Kim, D. J., Yun, M., Jeong, J. J., & Rhee, Y. (2020) · Journal of Bone and Mineral Metabolism · doi:10.1007/s00774-020-01085-2 · Cited by 4 · see citing works

External validation of the ARCH score in patients undergoing aortic arch reconstruction under circulatory arrest

Ahmad, D., Sá, M. P., Brown, J. A., Yousef, S., Wang, Y., Thoma, F. W., Chu, D., Kaczorowski, D. J., West, D. M., Bonatti, J. O., et al. (2024) · Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2024.05.004 · Cited by 4 · see citing works

A Comparative Exploration of SHAP and LIME for Enhancing the Interpretability of Machine Learning Models in Obesity Prediction

Awe, O. O., Salako, J., Rodrigues, P. C., Dukhi, N., & Dias, R. (2024) · STEAM-H: Science, Technology, Engineering, Agriculture, Mathematics & Health · doi:10.1007/978-3-031-72215-8_11 · Cited by 3 · see citing works

Immune variations throughout the course of tuberculosis treatment and its relationship with adrenal hormone changes in HIV-1 patients co-infected with Mycobacterium tuberculosis

Vecchione, M. B., Angerami, M. T., Suárez, G., Turk, G., Laufer, N., Ben, G., Ameri, D., González, D., Parodi, L. M., Giavedoni, L. D., et al. (2021) · Tuberculosis · doi:10.1016/j.tube.2020.102045 · Cited by 3 · see citing works

More accurate simulation for insurance data based on a modified SVM polynomial method1

Nurhidayat, I., Pimpunchat, B., & Klomsungcharoen, W. (2023) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-222879 · Cited by 3 · see citing works

ROC analysis with multiple classes and multiple tests: methodology and its application in microarray studies

Li, J., & Fine, J. P. (2008) · Biostatistics · doi:10.1093/biostatistics/kxm050 · Cited by 167 · see citing works

ROC Curves for the Statistical Analysis of Microarray Data

Cao, R., & López‐de‐Ullibarri, I. (2019) · Methods in molecular biology · doi:10.1007/978-1-4939-9442-7_11 · Cited by 48 · see citing works

Caveats and pitfalls of ROC analysis

Berrar, D., & Flach, P. (2010)

Discussion about ROC curves and others figures used to compare microarray statistical analyses

Gaigneaux, A., Hertogh, B. D., Berger, F., Pierre, M., Bareke, E., & Depiereux, E. (2008) · Repository of the University of Namur · Cited by 1 · see citing works

Measuring classifier . . . to the area under the ROC curve

Hand, D. J. (2009)