The MicroArray Quality Control (MAQC)-II study of common practices for the development and validation of microarray-based predictive models

MAQC Consortium

doi:10.1038/nbt.1665

Cited by 894Gene expression and cancer classificationBreast Cancer Treatment StudiesComputational Drug Discovery Methods

The MicroArray Quality Control (MAQC)-II study of common practices for the development and validation of microarray-based predictive models

Open access

MAQC Consortium·2010·Nature Biotechnology

DOI 10.1038/nbt.1665PMC PMC3315840

Open this paper in CoChat — free account

Abstract

Gene expression data from microarrays are being applied to predict preclinical and clinical endpoints, but the reliability of these predictions has not been established. In the MAQC-II project, 36 independent teams analyzed six microarray data sets to generate predictive models for classifying a sample with respect to one of 13 endpoints indicative of lung or liver toxicity in rodents, or of breast cancer, multiple myeloma or neuroblastoma in humans. In total, >30,000 models were built using many combinations of analytical methods. The teams generated predictive models without knowing the biological meaning of some of the endpoints and, to mimic clinical reality, tested the models on data that had not been used for training. We found that model performance depended largely on the endpoint and team proficiency and that different approaches generated models of similar performance. The conclusions and recommendations from MAQC-II should be useful for regulatory agencies, study committees and independent investigators that evaluate methods for global gene expression analysis.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1038/nbt.1665

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

PEER yielded higher accuracy than the original limma-voom method when validated against common intra-site DEGs, indicating that global normalization methods such as PEER can improve RNA-seq data across Illumina and PGM platforms.
“Using the common intra-site DEGs to validate the called inter-site DEGs, the MCC41, 42 showed that PEER also yields a higher accuracy than the original limmavoom method (Supplementary Fig. 22d,e), indicating that global data normalization analysis methods such as PEER can also be used to improve RNA-seq data across both Illumina and the PGM platforms.”
From Detecting and correcting systematic variation in large-scale RNA sequencing data (2014) · cites this paper as [42]
The FDA employed MCC as the main evaluation measure in the MAQC/SEQC projects.
“for instance, the Food and Drug Administration (FDA) agency of the USA employed the MCC as the main evaluation measure in the MicroArray II / Sequencing Quality Control (MAQC/SEQC) projects [41, 42].”
From The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation (2020) · cites this paper as [41]
In the MAQC-II study, performance of gene expression models was largely dependent on the respective clinical endpoint.
“Within the MicroArray Quality Control (MAQC)-II study [7], we observed, however, that the performance of gene expression models in predicting clinical outcome was limited and largely dependent on the respective clinical endpoint.”
From Comparison of RNA-seq and microarray-based models for clinical endpoint prediction (2015) · cites this paper as [7]
In the MAQC-II study, the performance of gene expression models in predicting clinical outcome was limited.
“Within the MicroArray Quality Control (MAQC)-II study [7], we observed, however, that the performance of gene expression models in predicting clinical outcome was limited and largely dependent on the respective clinical endpoint.”
From Comparison of RNA-seq and microarray-based models for clinical endpoint prediction (2015) · cites this paper as [7]
Endpoints were included that cover a broad range of predictive difficulties.
“Considering the fact that the nature of the clinical endpoint strongly affects a classifier’s performance [7, 18], we included endpoints that are known to cover a broad range of predictive difficulties (low difficulty, SEX, and CLASS LABEL; intermediate difficulty, EFS ALL, and OS ALL; high difficulty, EFS HR, and OS HR).”
From Comparison of RNA-seq and microarray-based models for clinical endpoint prediction (2015) · cites this paper as [7]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Gene expression profiling predicts clinical outcome of breast cancer
van 't Veer, L. J., Dai, H., van de Vijver, M. J., He, Y. D., Hart, A. A. M., Mao, M., Peterse, H. L., van der Kooy, K., Marton, M. J., Witteveen, A. T., et al. (2002) · Nature · doi:10.1038/415530a · Cited by 9,536 · see citing works
Unbiased Recursive Partitioning: A Conditional Inference Framework
Hothorn, T., Hornik, K., & Zeileis, A. (2006) · Journal of Computational and Graphical Statistics · doi:10.1198/106186006x133933 · Cited by 4,021 · see citing works
Comparison of Discrimination Methods for the Classification of Tumors Using Gene Expression Data
Dudoit, S., Fridlyand, J., & Speed, T. P. (2002) · Journal of the American Statistical Association · doi:10.1198/016214502753479248 · Cited by 2,673 · cited 1× · see citing works
“Although previous studies have compared and benchmarked individual steps in the model development proces”
Result
The MicroArray Quality Control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements
MAQC Consortium, Shi, L., Shi, L., Reid, L. H., Jones, W. D., Shippy, R., Warrington, J. A., Baker, S. C., Collins, P. J., de Longueville, F., et al. (2006) · Nature Biotechnology · doi:10.1038/nbt1239 · Cited by 2,126 · see citing works
That BLUP is a Good Thing: The Estimation of Random Effects
Robinson, G. K. (1991) · Statistical Science · doi:10.1214/ss/1177011926 · Cited by 1,694 · see citing works
Selection bias in gene extraction on the basis of microarray gene-expression data
Ambroise, C., & McLachlan, G. J. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.102102699 · Cited by 1,487 · cited 1× · see citing works
“, poor data quality and/or insufficient cross-validation of all model development step”
Background
Prediction of Survival in Follicular Lymphoma Based on Molecular Features of Tumor-Infiltrating Immune Cells
Davé, S. S., Wright, G. W., Tan, B. K., Rosenwald, A., Gascoyne, R. D., Chan, W. C., Fisher, R. I., Braziel, R. M., Rimsza, L. M., Grogan, T. M., et al. (2004) · New England Journal of Medicine · doi:10.1056/nejmoa041869 · Cited by 1,417 · cited 1× · see citing works
“Previously, reanalysis of a widely cited single stud”
Background
Microarray data analysis: from disarray to consolidation and consensus
Allison, D. B., Cui, X., Page, G. P., & Sabripour, M. (2006) · Nature Reviews Genetics · doi:10.1038/nrg1749 · Cited by 1,339 · see citing works
Validation and Clinical Utility of a 70-Gene Prognostic Signature for Women With Node-Negative Breast Cancer
Buyse, M. E., Loi, S., Veer, L. V., Viale, G., Delorenzi, M., Glas, A. M., Saghatchian, M., Bergh, J. C. S., Lidereau, R., Ellis, P., et al. (2006) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djj329 · Cited by 1,235 · see citing works
Gene expression–based survival prediction in lung adenocarcinoma: a multi-site, blinded validation study
Shedden, K., Taylor, J. M. G., Misek, D. E., Chang, A. C., Hanash, S., Kuick, R. D., Beer, D. G., Enkemann, S. A., Yeatman, T. J., Eschrich, S. A., et al. (2008) · Nature Medicine · doi:10.1038/nm.1790 · Cited by 1,194 · see citing works
The molecular classification of multiple myeloma
Zhan, F., Huang, Y., Colla, S., Stewart, P., Hanamura, I., Gupta, S. K., Epstein, J., Yaccoby, S., Sawyer, J. R., Burington, B. E., et al. (2006) · Blood · doi:10.1182/blood-2005-11-013458 · Cited by 1,137 · see citing works
Prediction of cancer outcome with microarrays: a multiple random validation strategy
Michiels, S., Koscielny, S., & Hill, C. (2005) · The Lancet · doi:10.1016/s0140-6736(05)17866-0 · Cited by 1,029 · see citing works
A validated gene expression model of high-risk multiple myeloma is defined by deregulated expression of genes mapping to chromosome 1
Shaughnessy, J. D., Zhan, F., Burington, B. E., Huang, Y., Colla, S., Hanamura, I., Stewart, P., Kordsmeier, B., Randolph, C. E., Williams, D. R., et al. (2006) · Blood · doi:10.1182/blood-2006-07-038430 · Cited by 916 · cited 1× · see citing works
“Gene expression profiling of highly purified bone marrow plasma cells was performed in newly diagnosed patients with M”
Background
Multiple-laboratory comparison of microarray platforms
Irizarry, R. A., Warren, D., Spencer, F., Kim, I. F., Biswal, S., Frank, B. C., Gabrielson, E., Garcia, J. G. N., Geoghegan, J., Germino, G., et al. (2005) · Nature Methods · doi:10.1038/nmeth756 · Cited by 862 · cited 1× · see citing works
“and confirmed that microarray technology is able to reliably identify differentially expressed genes between sample classes or population”
Background
Outcome signature genes in breast cancer: is there a unique set?
Ein‐Dor, L., Kela, I., Getz, G. A., Givol, D., & Domany, E. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth469 · Cited by 792 · see citing works
Thalidomide and Hematopoietic-Cell Transplantation for Multiple Myeloma
Barlogie, B., Tricot, G. J. K., Anaissie, E. J., Shaughnessy, J. D., Rasmussen, E., van Rhee, F., Fassas, A., Zangari, M., Hollmig, K. A., Pineda‐Roman, M., et al. (2006) · New England Journal of Medicine · doi:10.1056/nejmoa053583 · Cited by 736 · cited 1× · see citing works
“Gene expression profiling of highly purified bone marrow plasma cells was performed in newly diagnosed patients with M”
Background
Pharmacogenomic Predictor of Sensitivity to Preoperative Chemotherapy With Paclitaxel and Fluorouracil, Doxorubicin, and Cyclophosphamide in Breast Cancer
Hess, K. R., Anderson, K., Symmans, W. F., Valero, V., Ibrahim, N. K., Mejia, J. A., Booser, D. J., Theriault, R. L., Buzdar, A. U., Dempsey, P. J., et al. (2006) · Journal of Clinical Oncology · doi:10.1200/jco.2006.05.6861 · Cited by 720 · cited 1× · see citing works
“Genomic analysis of a subset of this sequentially accrued patient population were reported previousl”
Method
Thousands of samples are needed to generate a robust gene list for predicting outcome in cancer
Ein‐Dor, L., Zuk, O., & Domany, E. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0601231103 · Cited by 691 · see citing works
Critical Review of Published Microarray Studies for Cancer Outcome and Guidelines on Statistical Analysis and Reporting
Dupuy, A., & Simon, R. M. (2007) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djk018 · Cited by 632 · cited 1× · see citing works
“, findings consistent with what was observed from a recent literature surve”
Result
Evaluation of DNA microarray results with quantitative gene expression platforms
Canales, R. D., Luo, Y., Willey, J. C., Austermiller, B., Bárbácioru, C., Boysen, C., Hunkapiller, K., Jensen, R. V., Knight, C. R., Lee, K. Y., et al. (2006) · Nature Biotechnology · doi:10.1038/nbt1236 · Cited by 615 · see citing works
Repeatability of published microarray gene expression analyses
Ioannidis, J. P. A., Allison, D. B., Ball, C. A., Coulibaly, I., Cui, X., Culhane, A. C., Falchi, M., Furlanello, C., Game, L., Jurman, G., et al. (2009) · Nature Genetics · doi:10.1038/ng.295 · Cited by 551 · see citing works
Noninvasive Discrimination of Rejection in Cardiac Allograft Recipients Using Gene Expression Profiling
Deng, M. C.-H. K., Eisen, H. J., Mehra, M. R., Billingham, M. E., Marboe, C. C., Berry, G. J., KOBASHIGAWA, J. A., Johnson, F. L., Starling, R. C., Murali, S., et al. (2005) · American Journal of Transplantation · doi:10.1111/j.1600-6143.2005.01175.x · Cited by 525 · cited 1× · see citing works
“Gene expression data have also been the basis for the development of PCR-based diagnostic assays, including the xDx Allomap test for detection of rejection of heart transplant”
Method
Performance comparison of one-color and two-color platforms within the Microarray Quality Control (MAQC) project
Patterson, T. A., Lobenhofer, E. K., Fulmer-Smentek, S., Collins, P. J., Chu, T., Bao, W., Fang, H., Kawasaki, E. S., Hager, J., Tikhonova, I. R., et al. (2006) · Nature Biotechnology · doi:10.1038/nbt1242 · Cited by 492 · see citing works
Rat toxicogenomic study reveals analytical consistency across microarray platforms
Guo, L., Lobenhofer, E. K., Wang, C., Shippy, R., Harris, S., Zhang, L., Mei, N., Chen, T., Herman, D., Goodsaid, F. M., et al. (2006) · Nature Biotechnology · doi:10.1038/nbt1238 · Cited by 406 · see citing works
Prediction of Survival in Multiple Myeloma Based on Gene Expression Profiles Reveals Cell Cycle and Chromosomal Instability Signatures in High-Risk Patients and Hyperdiploid Signatures in Low-Risk Patients: A Study of the Intergroupe Francophone du Myélome
Decaux, O., Lodé, L., Magrangeas, F., Guérin‐Charbonnel, C., Gouraud, W., Jézéquel, P., Attal, M., Harousseau, J., Moreau, P., Bataille, R., et al. (2008) · Journal of Clinical Oncology · doi:10.1200/jco.2007.13.8545 · Cited by 372 · cited 1× · see citing works
“and later on validated in three additional independent data set”
Background
A comparison of batch effect removal methods for enhancement of prediction performance using MAQC-II microarray gene expression data
Luo, J., Schumacher, M., Scherer, A., Sanoudou, D., Megherbi, D. B., Davison, T. S., Shi, T., Tong, W., Shi, L. M., Hong, H., et al. (2010) · The Pharmacogenomics Journal · doi:10.1038/tpj.2010.57 · Cited by 330 · see citing works
Predictive ability of DNA microarrays for cancer outcomes and correlates: an empirical assessment
Ntzani, E., & Ioannidis, J. P. A. (2003) · The Lancet · doi:10.1016/s0140-6736(03)14686-7 · Cited by 323 · see citing works
Getting the Noise Out of Gene Arrays
Marshall, E. (2004) · Science · doi:10.1126/science.306.5696.630 · Cited by 302 · see citing works
Customized Oligonucleotide Microarray Gene Expression–Based Classification of Neuroblastoma Patients Outperforms Current Clinical Risk Stratification
Oberthuer, A., Berthold, F., Warnat, P., Hero, B., Kahlert, Y., Spitz, R., Ernestus, K., König, R., Haas, S., Eils, R., et al. (2006) · Journal of Clinical Oncology · doi:10.1200/jco.2006.06.1879 · Cited by 291 · cited 1× · see citing works
“First, 502 preexisting 11 K Agilent dye-flipped, dual-color replicate profiles for 251 patients were provide”
Background
Development of a large-scale chemogenomics database to improve drug candidate selection and to understand mechanisms of chemical toxicity and action
Ganter, B., Tugendreich, S., Pearson, C. I., Ayanoĝlu, E., Baumhueter, S., Bostian, K. A., Brady, L., Browne, L. J., Calvin, J. T., Day, G.-J., et al. (2005) · Journal of Biotechnology · doi:10.1016/j.jbiotec.2005.03.022 · Cited by 286 · cited 1× · see citing works
“Gene expression data were generated using the Amersham Codelink Uniset Rat 1 Bioarray (GE HealthCare”
Method
Microarrays and molecular research: noise discovery?
Ioannidis, J. P. A. (2005) · The Lancet · doi:10.1016/s0140-6736(05)17878-7 · Cited by 271 · see citing works
Effect of training-sample size and classification difficulty on the accuracy of genomic predictors
Popovici, V. C., Chen, W., Gallas, B. D., Hatzis, C., Shi, W., Samuelson, F. W., Nikolsky, Y., Tsyganova, M., Ishkin, A., Nikolskaya, T., et al. (2010) · Breast Cancer Research · doi:10.1186/bcr2468 · Cited by 219 · see citing works
A Gene Expression Biomarker Provides Early Prediction and Mechanistic Assessment of Hepatic Tumor Induction by Nongenotoxic Chemicals
Fielden, M. R., Brennan, R. J., & Gollub, J. (2007) · Toxicological Sciences · doi:10.1093/toxsci/kfm156 · Cited by 206 · cited 1× · see citing works
“The study objective was to assess, upon short-term exposure, hepatic tumor induction by nongenotoxic chemical”
Background
Prognostic gene signatures for non-small-cell lung cancer
Boutros, P. C., Lau, S. K., Pintilie, M., Liu, G., Shepherd, F. A., Der, S. D., Tsao, M., Penn, L. Z., & Jurišica, I. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0809444106 · Cited by 205 · see citing works
Cross-platform comparability of microarray technology: Intra-platform consistency and appropriate data analysis procedures are essential
Shi, L. M., Tong, W., Fang, H., Scherf, U., Han, J., Puri, R. K., Frueh, F. W., Goodsaid, F. M., Guo, L., Su, Z., et al. (2005) · BMC Bioinformatics · doi:10.1186/1471-2105-6-s2-s12 · Cited by 201 · see citing works
Determination of oestrogen-receptor status and ERBB2 status of breast carcinoma: a gene-expression profiling study
Gong, Y., Yan, K., Lin, F., Anderson, K., Sotiriou, C. P., André, F., Holmes, F. A., Valero, V., Booser, D. J., Pippen, J., et al. (2007) · The Lancet Oncology · doi:10.1016/s1470-2045(07)70042-6 · Cited by 194 · see citing works
Interlaboratory Comparability Study of Cancer Gene Expression Analysis Using Oligonucleotide Microarrays
Dobbin, K. K., Beer, D. G., Meyerson, M. M., Yeatman, T. J., Gerald, W. L., Jacobson, J. W., Conley, B. A., Buetow, K. H., Heiskanen, M. A., Simon, R. M., et al. (2005) · Clinical Cancer Research · doi:10.1158/1078-0432.565.11.2 · Cited by 179 · see citing works
Total RNA yield and microarray gene expression profiles from fine‐needle aspiration biopsy and core‐needle biopsy samples of breast carcinoma
Symmans, W. F., Ayers, M., Clark, E. A., Stec, J. J., Hess, K. R., Sneige, N., Buchholz, T. A., Krishnamurthy, S., Ibrahim, N. K., Buzdar, A. U., et al. (2003) · Cancer · doi:10.1002/cncr.11435 · Cited by 174 · cited 1× · see citing works
“These specimens represent 70–90% pure neoplastic cells with minimal stromal contaminatio”
Background
Using RNA sample titrations to assess microarray platform performance and normalization techniques
Shippy, R., Fulmer-Smentek, S., Jensen, R. V., Jones, W. D., Wolber, P. K., Johnson, C. D., Pine, P. S., Boysen, C., Guo, X., Chudin, E., et al. (2006) · Nature Biotechnology · doi:10.1038/nbt1241 · Cited by 173 · see citing works
Self-citation
Microarrays: retracing steps
Coombes, K. R., Wang, J., & Baggerly, K. (2007) · Nature Medicine · doi:10.1038/nm1107-1276b · Cited by 148 · cited 1× · see citing works
“A lack of consistency in generating classifiers from publicly available data is problematic and may be due to any number of factors including insufficient annotation, incomplete clinical identifiers, coding errors and/or inappropriate use of methodolog”
Background
Reproducible and reliable microarray results through quality control: good laboratory proficiency and appropriate data analysis practices are essential
Shi, L. M., Perkins, R., Fang, H., & Tong, W. (2008) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2007.11.003 · Cited by 139 · cited 1× · see citing works
“Therefore, the second phase of the project (MAQC-II) has focused on these predictive model”
Background
k-Nearest neighbor models for microarray gene expression analysis and clinical outcome prediction
Parry, R. M., Jones, W. D., Stokes, T. H., Phan, J. H., Moffitt, A. B., Fang, H., Shi, L. M., Oberthuer, A., Fischer, M., Tong, W., et al. (2010) · The Pharmacogenomics Journal · doi:10.1038/tpj.2010.56 · Cited by 136 · see citing works
Evaluation of external RNA controls for the assessment of microarray performance
Tong, W., Lucas, A. B., Shippy, R., Fan, X., Fang, H., Hong, H., Orr, M. S., Chu, T., Guo, X., Collins, P. J., et al. (2006) · Nature Biotechnology · doi:10.1038/nbt1237 · Cited by 108 · see citing works
QA/QC: challenges and pitfalls facing the microarray community and regulatory agencies
Shi, L. M., Tong, W., Goodsaid, F. M., Frueh, F. W., Fang, H., Han, T., Fuscoe, J. C., & Casciano, D. A. (2004) · Expert Review of Molecular Diagnostics · doi:10.1586/14737159.4.6.761 · Cited by 103 · see citing works
Interlaboratory Performance of a Microarray-Based Gene Expression Test to Determine Tissue of Origin in Poorly Differentiated and Undifferentiated Cancers
Dumur, C. I., Lyons‐Weiler, M. A., Sciulli, C., Garrett, C. T., Schrijver, I., Holley, T., Rodriguez‐Paris, J., Pollack, J. R., Zehnder, J. L., Price, M., et al. (2007) · Journal of Molecular Diagnostics · doi:10.2353/jmoldx.2008.070099 · Cited by 95 · cited 1× · see citing works
“and the Pathwork Tissue of Origin Test to assess the degree of similarity of the RNA expression pattern in a patient's tumor to that in a database of tumor samples for which the origin of the tumor is know”
Method
Voluntary exploratory data submissions to the US FDA and the EMA: experience and impact
Goodsaid, F. M., Amur, S., Aubrecht, J., Burczynski, M. E., Carl, K., Catalano, J. G., Charlab, R., Close, S. L., Cornu-Artis, C., Essioux, L., et al. (2010) · Nature Reviews Drug Discovery · doi:10.1038/nrd3116 · Cited by 88 · cited 1× · see citing works
“Microarray-based gene expression data and prediction models are increasingly being submitted by the regulated industry to the FDA to support medical product development and testing application”
Method
Classification based upon gene expression data: bias and precision of error rates
Wood, I., Visscher, P. M., & Mengersen, K. L. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btm117 · Cited by 76 · see citing works
An array of problems
Frantz, S. (2005) · Nature Reviews Drug Discovery · doi:10.1038/nrd1746 · Cited by 71 · see citing works
High-risk myeloma: a gene expression–based risk-stratification model for newly diagnosed multiple myeloma treated with high-dose therapy is predictive of outcome in relapsed disease treated with single-agent bortezomib or high-dose dexamethasone
Zhan, F., Barlogie, B., Mulligan, G., Shaughnessy, J. D., & Bryant, B. M. (2008) · Blood · doi:10.1182/blood-2007-10-119321 · Cited by 67 · see citing works
Consistency of predictive signature genes and classifiers generated using different microarray platforms
Fan, X., Lobenhofer, E. K., Chen, M., Shi, W., Huang, J., Luo, J., Zhang, J. H., Walker, S. J., Chu, T.-M., Li, L., et al. (2010) · The Pharmacogenomics Journal · doi:10.1038/tpj.2010.34 · Cited by 67 · see citing works
Comparison of performance of one-color and two-color gene-expression analyses in predicting clinical endpoints of neuroblastoma patients
Oberthuer, A., Juraeva, D., Li, L., Kahlert, Y., Westermann, F., Eils, R., Berthold, F., Shi, L. M., Wolfinger, R. D., Fischer, M., et al. (2010) · The Pharmacogenomics Journal · doi:10.1038/tpj.2010.53 · Cited by 67 · see citing works
Application of Genomic Biomarkers to Predict Increased Lung Tumor Incidence in 2-Year Rodent Cancer Bioassays
Thomas, R. S., Pluta, L., Yang, L., & Halsey, T. A. (2007) · Toxicological Sciences · doi:10.1093/toxsci/kfm023 · Cited by 65 · cited 1× · see citing works
“The study objective was to apply microarray gene expression data from the lung of female B6C3F1 mice exposed to a 13-week treatment of chemicals to predict increased lung tumor incidence in the 2-year rodent cancer bioassays of the National Toxicology Progra”
Method
High-resolution serum proteomic patterns for ovarian cancer detection
Baggerly, K., Edmonson, S. R., Morris, J. S., & Coombes, K. R. (2004) · Endocrine Related Cancer · doi:10.1677/erc.1.00868 · Cited by 64 · see citing works
Genomic indicators in the blood predict drug-induced liver injury
Huang, J., Shi, W., Zhang, J., Chou, J., Paules, R. S., Gerrish, K., Li, J., Luo, J., Wolfinger, R. D., Bao, W., et al. (2010) · The Pharmacogenomics Journal · doi:10.1038/tpj.2010.33 · Cited by 63 · see citing works
Gene expression response in target organ and whole blood varies as a function of target organ injury phenotype
Lobenhofer, E. K., Auman, J. T., Blackshear, P. E., Boorman, G. A., Bushel, P. R., Cunningham, M. L., Fostel, J. M., Gerrish, K., Heinloth, A. N., Irwin, R. D., et al. (2008) · Genome biology · doi:10.1186/gb-2008-9-6-r100 · Cited by 51 · see citing works
Assessing Classifiers from Two Independent Data Sets Using ROC Analysis: A Nonparametric Approach
Yousef, W. A., Wagner, R. F., & Loew, M. H. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.218 · Cited by 41 · see citing works
Functional analysis of multiple genomic signatures demonstrates that classification algorithms choose phenotype-related genes
Shi, W., Bessarabova, M., Dosymbekov, D., Dezső, Z., Nikolskaya, T., Dudoladova, M., Serebryiskaya, T., Bugrim, A. E., Guryanov, A., Brennan, R., et al. (2010) · The Pharmacogenomics Journal · doi:10.1038/tpj.2010.35 · Cited by 41 · see citing works
Arrays of Hope
Strauß, E. (2006) · Cell · doi:10.1016/j.cell.2006.11.005 · Cited by 34 · cited 1× · see citing works
“and confirmed that microarray technology is able to reliably identify differentially expressed genes between sample classes or population”
Background
Translocation t(4;14) retains prognostic significance even in the setting of high-risk molecular signature
Chng, W. J., Kuehl, W. M., Bergsagel, P. L., & Fonseca, R. (2007) · Leukemia · doi:10.1038/sj.leu.2404934 · Cited by 34 · see citing works
Using DNA microarrays for diagnostic and prognostic prediction
Simon, R. M. (2003) · Expert Review of Molecular Diagnostics · doi:10.1586/14737159.3.5.587 · Cited by 33 · cited 1× · see citing works
“, poor data quality and/or insufficient cross-validation of all model development step”
Background
Immune Signatures in Follicular Lymphoma
Hong, W., Warnke, R. A., & Chu, G. (2005) · New England Journal of Medicine · doi:10.1056/nejm200504073521422 · Cited by 30 · see citing works
Assessing sources of inconsistencies in genotypes and their effects on genome-wide association studies with HapMap samples
Hong, H., Shi, L. M., Su, Z., Ge, W., Jones, W. D., Czika, W., Miclaus, K. J., Lambert, C. G., Vega, S. C., Zhang, J., et al. (2010) · The Pharmacogenomics Journal · doi:10.1038/tpj.2010.24 · Cited by 23 · see citing works
On the repeated use of databases for testing incremental improvement of computer-aided detection schemes
Gur, D., Wagner, R. F., & Chan, H. (2003) · Academic Radiology · doi:10.1016/s1076-6332(03)00511-7 · Cited by 11 · see citing works

Flagged first, then by citations

Sorted by citations

The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143 · see citing works
Applications of machine learning in drug discovery and development
Vamathevan, J., Clark, D., Czodrowski, P., Dunham, I., Ferran, E., Lee, G., Li, B., Madabhushi, A., Shah, P., Spitzer, M., et al. (2019) · Nature Reviews Drug Discovery · doi:10.1038/s41573-019-0024-5 · Cited by 3,228 · see citing works
Virtual microdissection identifies distinct tumor- and stroma-specific subtypes of pancreatic ductal adenocarcinoma
Moffitt, R. A., Marayati, R., Flate, E. L., Volmar, K. E., Loeza, S. G. H., Hoadley, K. A., Rashid, N. U., Williams, L. A., Eaton, S. C., Chung, A. H., et al. (2015) · Nature Genetics · doi:10.1038/ng.3398 · Cited by 2,194 · see citing works
Self-citation
Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric
Boughorbel, S., Jarray, F., & El-Anbari, M. (2017) · PLOS ONE · doi:10.1371/journal.pone.0177678 · Cited by 1,222 · see citing works
The genetic landscape of high-risk neuroblastoma
Pugh, T. J., Morozova, O., Attiyeh, E. F., Asgharzadeh, S., Wei, J. S., Auclair, D., Carter, S. L., Cibulskis, K., Hanna, M., Kieżun, A., et al. (2013) · Nature Genetics · doi:10.1038/ng.2529 · Cited by 1,170 · see citing works
Clinical drug response can be predicted using baseline gene expression levels and in vitro drug sensitivity in cell lines
Geeleher, P., Cox, N. J., & Huang, R. S. (2014) · Genome biology · doi:10.1186/gb-2014-15-3-r47 · Cited by 889 · see citing works
Comprehensive evaluation of differential gene expression analysis methods for RNA-seq data
Rapaport, F., Khanin, R., Liang, Y., Pirun, M., Krek, A., Zumbo, P., Mason, C. E., Socci, N. D., & Betel, D. (2013) · Genome Biology · doi:10.1186/gb-2013-14-9-r95 · Cited by 733 · see citing works
Machine Learning in Drug Discovery: A Review
Dara, S., Dhamercherla, S., Jadav, S. S., Babu, C. M., & Ahsan, M. J. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-10058-4 · Cited by 843 · see citing works
The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification
Chicco, D., & Jurman, G. (2023) · BioData Mining · doi:10.1186/s13040-023-00322-4 · Cited by 667 · see citing works
“Actually, as shown by the MACQ Consortium [57], if we introduce the prevalence p = TP + FN TP + TN + FP + FN as the ratio of the actual positive samples over the total number of samples, by definition specificity (TNR) and sensitivity (TPR) do not depend on p , while MCC does.”
Background
IMWG consensus on risk stratification in multiple myeloma
Chng, W. J., Dispenzieri, A. A., Chim, C. S., Fonseca, R., Goldschmidt, H., Lentzsch, S., Munshi, N. C., Palumbo, A. P., Miguel, J. F. S., Sonneveld, P., et al. (2013) · Leukemia · doi:10.1038/leu.2013.247 · Cited by 600 · see citing works
Microarrays, deep sequencing and the true measure of the transcriptome
Malone, J. H., & Oliver, B. (2011) · BMC Biology · doi:10.1186/1741-7007-9-34 · Cited by 546 · see citing works
Removing Batch Effects in Analysis of Expression Microarray Data: An Evaluation of Six Batch Adjustment Methods
Chen, C., Grennan, K. S., Badner, J. A., Zhang, D., Gershon, E. S., Li, J., & Liu, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0017238 · Cited by 544 · see citing works
The concordance between RNA-seq and microarray data depends on chemical treatment and transcript abundance
Wang, C., Gong, B., Bushel, P. R., Thierry-Mieg, J., Thierry-Mieg, D., Xu, J., Fang, H., Hong, H., Shen, J., Su, Z., et al. (2014) · Nature Biotechnology · doi:10.1038/nbt.3001 · Cited by 512 · see citing works
Data-Driven Strategies for Accelerated Materials Design
Pollice, R., Gomes, G., Aldeghi, M., Hickman, R. J., Krenn, M., Lavigne, C., Lindner-D’Addario, M., Nigam, A., Ser, C. T., Yao, Z., et al. (2021) · Accounts of Chemical Research · doi:10.1021/acs.accounts.0c00785 · Cited by 477 · see citing works
The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment
Chicco, D., Warrens, M., & Jurman, G. (2021) · IEEE Access · doi:10.1109/access.2021.3084050 · Cited by 365 · see citing works
Characterizing the replicability of cell types defined by single cell RNA-sequencing data using MetaNeighbor
Crow, M., Paul, A., Ballouz, S., Huang, Z. J., & Gillis, J. (2018) · Nature Communications · doi:10.1038/s41467-018-03282-0 · Cited by 365 · see citing works
A rat RNA-Seq transcriptomic BodyMap across 11 organs and 4 developmental stages
Yu, Y., Fuscoe, J. C., Zhao, C., Guo, C., Jia, M., Qing, T., Bannon, D. I., Lancashire, L., Bao, W., Du, T., et al. (2014) · Nature Communications · doi:10.1038/ncomms4230 · Cited by 395 · see citing works
RETRACTED ARTICLE: Artificial intelligence and machine learning in precision and genomic medicine
Quazi, S. (2022) · Medical Oncology · doi:10.1007/s12032-022-01711-1 · Cited by 455 · see citing works
Retracted
A Comparison of MCC and CEN Error Measures in Multi-Class Prediction
Jurman, G., Riccadonna, S., & Furlanello, C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0041882 · Cited by 406 · see citing works
Self-citation
Comparison of RNA-seq and microarray-based models for clinical endpoint prediction
Zhang, W., Yu, Y., Hertwig, F., Thierry-Mieg, J., Zhang, W., Thierry-Mieg, D., Wang, J., Furlanello, C., Devanarayan, V., Cheng, J., et al. (2015) · Genome Biology · doi:10.1186/s13059-015-0694-1 · Cited by 393 · see citing works
Toxicity testing in the 21st century: progress in the past decade and future perspectives
Krewski, D. R., Andersen, M. E., Tyshenko, M. G., Krishnan, K., Härtung, T., Boekelheide, K., Wambaugh, J. F., Jones, D. P., Whelan, M., Thomas, R. S., et al. (2019) · Archives of Toxicology · doi:10.1007/s00204-019-02613-4 · Cited by 384 · see citing works
Next-generation sequencing in the clinic: Promises and challenges
Xuan, J., Yu, Y., Qing, T., Guo, L., & Shi, L. M. (2012) · Cancer Letters · doi:10.1016/j.canlet.2012.11.025 · Cited by 347 · see citing works
Trials and tribulations of ‘omics data analysis: assessing quality of SIMCA-based multivariate models using examples from pulmonary medicine
Wheelock, Å. M., & Wheelock, C. E. (2013) · Molecular BioSystems · doi:10.1039/c3mb70194h · Cited by 338 · see citing works
“16 The power of supervised methods, particularly OPLS, can however be a double-edged sword.”
Upregulation of Trop-2 quantitatively stimulates human cancer growth
Trerotola, M., Cantanelli, P., Guerra, E., Tripaldi, R., Aloisi, A. L., Bonasera, V., Lattanzio, R., de Lange, R., Weidle, U. H., Piantelli, M., et al. (2012) · Oncogene · doi:10.1038/onc.2012.36 · Cited by 333 · see citing works
Transcriptomic and metabolomic data integration
Cavill, R., Jennen, D. G. J., Kleinjans, J. C. S., & Briedé, J. J. (2015) · Briefings in Bioinformatics · doi:10.1093/bib/bbv090 · Cited by 331 · see citing works
A comparison of batch effect removal methods for enhancement of prediction performance using MAQC-II microarray gene expression data
Luo, J., Schumacher, M., Scherer, A., Sanoudou, D., Megherbi, D. B., Davison, T. S., Shi, T., Tong, W., Shi, L. M., Hong, H., et al. (2010) · The Pharmacogenomics Journal · doi:10.1038/tpj.2010.57 · Cited by 330 · see citing works
A comprehensive comparison of RNA-Seq-based transcriptome analysis from reads to differential gene expression and cross-comparison with microarrays: a case study in Saccharomyces cerevisiae
Nookaew, I., Papini, M., Pornputtapong, N., Scalcinati, G., Fagerberg, L., Uhlén, M., & Nielsen, J. (2012) · Nucleic Acids Research · doi:10.1093/nar/gks804 · Cited by 309 · see citing works
Whole Blood Gene Expression Profiles to Assess Pathogenesis and Disease Severity in Infants with Respiratory Syncytial Virus Infection
Mejias, A., Dimo, B., Suarez, N. M., Garcia, C., Suarez-Arrabal, M. C., Jartti, T., Blankenship, D., Jordan-Villegas, A., Ardura, M. I., Xu, Z., et al. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001549 · Cited by 313 · see citing works
Variable influence on projection (VIP) for orthogonal projections to latent structures (OPLS)
Galindo‐Prieto, B., Eriksson, L., & Trygg, J. (2014) · Journal of Chemometrics · doi:10.1002/cem.2627 · Cited by 312 · see citing works
The accuracy and reproducibility of the endometrial receptivity array is superior to histology as a diagnostic method for endometrial receptivity
Díaz-Gimeno, P., Ruíz-Alonso, M., Blesa, D., Bosch, N. F., Martínez-Conejero, J. A., Alamá, P., Garrido, N., Pellicer, A., & Simón, C. (2012) · Fertility and Sterility · doi:10.1016/j.fertnstert.2012.09.046 · Cited by 300 · see citing works
Molecular Diagnosis of Antibody-Mediated Rejection in Human Kidney Transplants
Sellarés, J., REEVE, J. P., Loupy, A., Mengel, M., Sis, B., Skene, A., de Freitas, D. G., Kreepala, C., Hidalgo, L. G., Famulski, K. S., et al. (2013) · American Journal of Transplantation · doi:10.1111/ajt.12150 · Cited by 280 · see citing works
Assessing the clinical utility of cancer genomic and proteomic data across tumor types
Yuan, Y., Allen, E. M. V., Omberg, L., Wagle, N., Amin‐Mansour, A., Sokolov, A., Byers, L. A., Xu, Y., Hess, K. R., Diao, L., et al. (2014) · Nature Biotechnology · doi:10.1038/nbt.2940 · Cited by 283 · see citing works
Radiomics in nuclear medicine: robustness, reproducibility, standardization, and how to avoid data analysis traps and replication crisis
Zwanenburg, A. (2019) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-019-04391-8 · Cited by 306 · see citing works
Multi-platform assessment of transcriptome profiling using RNA-seq in the ABRF next-generation sequencing study
Li, S., Tighe, S. W., Nicolet, C. M., Grove, D., Levy, S., Farmerie, W., Viale, A., Wright, C., Schweitzer, P. A., Gao, Y., et al. (2014) · Nature Biotechnology · doi:10.1038/nbt.2972 · Cited by 240 · see citing works
Computational deconvolution of transcriptomics data from mixed cell populations
Cobos, F. A., Vandesompele, J., Mestdagh, P., & Preter, K. D. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty019 · Cited by 246 · see citing works
Translation reprogramming is an evolutionarily conserved driver of phenotypic plasticity and therapeutic resistance in melanomaPreprint
Falletta, P., Sánchez‐del‐Campo, L., Chauhan, J. S., Effern, M., Kenyon, A., Kershaw, C. J., Siddaway, R., Lisle, R., Freter, R., Daniels, M. J., et al. (2017) · Genes & Development · doi:10.1101/gad.290940.116 · Cited by 239 · see citing works
“Average probe intensities were determined in Genome Studio (Illumina) and analyzed in a Bioconductor using the limma package (Smyth 2004; Shi et al. 2010).”
Identification of a Poor-PrognosisBRAF-Mutant–Like Population of Patients With Colon Cancer
Popovici, V. C., Budínská, E., Tejpar, S., Weinrich, S. L., Estrella, H., Hodgson, G., Cutsem, E. V., Xie, T., Bosman, F. T., Roth, A. D., et al. (2012) · Journal of Clinical Oncology · doi:10.1200/jco.2011.39.5814 · Cited by 232 · see citing works
Cognitive Behavioral Therapy and Tai Chi Reverse Cellular and Genomic Markers of Inflammation in Late-Life Insomnia: A Randomized Controlled Trial
Irwin, M. R., Olmstead, R., Breen, E. C., Witarama, T., Carrillo, C., Sadeghi, N., Arevalo, J. M. G., Ma, J., Nicassio, P., Bootzin, R., et al. (2015) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.01.010 · Cited by 227 · see citing works
Gene expression profiling in human neurodegenerative disease
Cooper‐Knock, J., Kirby, J., Ferraiuolo, L., Heath, P. R., Rattray, M., & Shaw, P. J. (2012) · Nature Reviews Neurology · doi:10.1038/nrneurol.2012.156 · Cited by 218 · see citing works
Gene Modules and Response to Neoadjuvant Chemotherapy in Breast Cancer Subtypes: A Pooled Analysis
Ignatiadis, M., Singhal, S. K., Desmedt, C., Haibe‐Kains, B., Criscitiello, C., André, F., Loi, S., Piccart, M. J., Michiels, S., & Sotiriou, C. P. (2012) · Journal of Clinical Oncology · doi:10.1200/jco.2011.39.5624 · Cited by 218 · see citing works
Gene Pathways Associated With Prognosis and Chemotherapy Sensitivity in Molecular Subtypes of Breast Cancer
Iwamoto, T., Bianchini, G., Booser, D. J., Qi, Y., Coutant, C., Shiang, C. Y., Santarpia, L., Matsuoka, J., Hortobágyi, G. N., Symmans, W. F., et al. (2010) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djq524 · Cited by 220 · see citing works
CD8 T Cell Score as a Prognostic Biomarker for Triple Negative Breast Cancer
Oshi, M., Asaoka, M., Tokumaru, Y., Yan, L., Matsuyama, R., Ishikawa, T., Endo, I., & Takabe, K. (2020) · International Journal of Molecular Sciences · doi:10.3390/ijms21186968 · Cited by 212 · see citing works
Overfitting, underfitting and general model overconfidence and under-performance pitfalls and best practices in machine learning and AI
Aliferis, C., & Simon, G. (2024) · Health Informatics · doi:10.1007/978-3-031-39355-6_10 · Cited by 212 · see citing works
Assessing and mitigating batch effects in large-scale omics studies
Yu, Y., Mai, Y., Zheng, Y., & Shi, L. M. (2024) · Genome biology · doi:10.1186/s13059-024-03401-9 · Cited by 231 · see citing works
Comparison of clustering tools in R for medium-sized 10x Genomics single-cell RNA-sequencing data
Freytag, S., Tian, L., Lönnstedt, I., Ng, M., & Bahlo, M. (2018) · F1000Research · doi:10.12688/f1000research.15809.1 · Cited by 220 · see citing works
Use cases, best practice and reporting standards for metabolomics in regulatory toxicology
Viant, M. R., Ebbels, T. M. D., Beger, R. D., Ekman, D. R., Epps, D. J. T., Kamp, H., Leonards, P. E. G., Loizou, G. D., MacRae, J. I., van Ravenzwaay, B., et al. (2019) · Nature Communications · doi:10.1038/s41467-019-10900-y · Cited by 207 · see citing works
DRAR-CPI: a server for identifying drug repositioning potential and adverse drug reactions via the chemical–protein interactome
Luo, H., Chen, J., Shi, L., Mikailov, M., Zhu, H., Wang, K., He, L., & Yang, L. (2011) · Nucleic Acids Research · doi:10.1093/nar/gkr299 · Cited by 206 · see citing works
Self-citation
Pre-vaccination inflammation and B-cell signalling predict age-related hyporesponse to hepatitis B vaccination
Fourati, S., Cristescu, R., Loboda, A., Talla, A., Filali, A., Railkar, R., Schaeffer, A. K., Favre, D., Gagnon, D., Peretz, Y., et al. (2016) · Nature Communications · doi:10.1038/ncomms10369 · Cited by 183 · see citing works
Detecting and correcting systematic variation in large-scale RNA sequencing data
Li, S., Łabaj, P. P., Zumbo, P., Sykacek, P., Shi, W., Shi, L., Phan, J., Wu, P.-Y., Wang, M., Wang, C., et al. (2014) · Nature Biotechnology · doi:10.1038/nbt.3000 · Cited by 175 · see citing works
Self-citation
Depressive Symptoms and Cortisol Rhythmicity Predict Survival in Patients with Renal Cell Carcinoma: Role of Inflammatory Signaling
Cohen, L., Cole, S. W., Sood, A. K., Prinsloo, S., Kirschbaum, C., Arevalo, J. M. G., Jennings, N. B., Scott, S., Vence, L., Wei, Q., et al. (2012) · PLoS ONE · doi:10.1371/journal.pone.0042324 · Cited by 197 · see citing works
Development of a clinical polygenic risk score assay and reporting workflow
Hao, L., Kraft, P., Berriz, G. F., Hynes, E. D., Koch, C., Kumar, P. K. V., Parpattedar, S. S., Steeves, M. A., Yu, W., Antwi, A. A., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01767-6 · Cited by 180 · see citing works
Essential guidelines for computational method benchmarking
Weber, L. M., Saelens, W., Cannoodt, R., Soneson, C., Hapfelmeier, A., Gardner, P. P., Boulesteix, A.-L., Saeys, Y., & Robinson, M. D. (2019) · Genome Biology · doi:10.1186/s13059-019-1738-8 · Cited by 21 · see citing works
Blood molecular markers associated with COVID‐19 immunopathology and multi‐organ damage
Chen, Y., Zheng, Y., Yu, Y., Wang, Y., Huang, Q., Qian, F., Sun, L., Song, Z., Chen, Z., Feng, J., et al. (2020) · The EMBO Journal · doi:10.15252/embj.2020105896 · Cited by 183 · see citing works
A 5-MicroRNA Signature for Lung Squamous Cell Carcinoma Diagnosis and hsa-miR-31 for Prognosis
Tan, X., Qin, W., Zhang, L., Hang, J., Li, B., Zhang, C., Wan, J., Zhou, F., Shao, K., Sun, Y., et al. (2011) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-11-0419 · Cited by 181 · see citing works
The Gene Expression Barcode: leveraging public data repositories to begin cataloging the human and murine transcriptomes
McCall, M. N., Uppal, K., Jaffee, H. A., Zilliox, M. J., & Irizarry, R. A. (2010) · Nucleic Acids Research · doi:10.1093/nar/gkq1259 · Cited by 177 · see citing works
An investigation of biomarkers derived from legacy microarray data for their utility in the RNA-seq era
Su, Z., Fang, H., Hong, H., Shi, L. M., Zhang, W., Zhang, W., Zhang, Y., Dong, Z., Lancashire, L., Bessarabova, M., et al. (2014) · Genome biology · doi:10.1186/s13059-014-0523-y · Cited by 178 · see citing works
Identification and validation of potential prognostic lncRNA biomarkers for predicting survival in patients with multiple myeloma
Zhou, M., Zhao, H., Wang, Z., Cheng, L., Yang, L., Shi, H., Yang, H., & Sun, J. (2015) · Journal of Experimental & Clinical Cancer Research · doi:10.1186/s13046-015-0219-5 · Cited by 180 · see citing works
Next-generation sequencing and its applications in molecular diagnostics
Su, Z., Ning, B., Fang, H., Hong, H., Perkins, R., Tong, W., & Shi, L. M. (2011) · Expert Review of Molecular Diagnostics · doi:10.1586/erm.11.3 · Cited by 181 · see citing works
Cancer Systems Biology: a peek into the future of patient care?
Werner, H. M. J., Mills, G. B., & Ram, P. T. (2014) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2014.6 · Cited by 175 · see citing works
Comparison of normalization and differential expression analyses using RNA-Seq data from 726 individual Drosophila melanogaster
Lin, Y., Golovnina, K. A., Chen, Z., Lee, H., Negron, Y. L. S., Sultana, H., Oliver, B., & Harbison, S. T. (2016) · BMC Genomics · doi:10.1186/s12864-015-2353-z · Cited by 170 · see citing works
Systematic assessment of analytical methods for drug sensitivity prediction from cancer cell line data.
Jang, I. S., Neto, E. C., Guinney, J., Friend, S. H., & Margolin, A. A. (2014) · PubMed · Cited by 170 · see citing works
Integrating Feature Selection and Feature Extraction Methods With Deep Learning to Predict Clinical Outcome of Breast Cancer
Zhang, D., Zou, L., Zhou, X., & He, F. (2018) · IEEE Access · doi:10.1109/access.2018.2837654 · Cited by 168 · see citing works
Characteristics and Predictive Value of Blood Transcriptome Signature in Males with Autism Spectrum Disorders
Kong, S. W., Collins, C. D., Shimizu‐Motohashi, Y., Holm, I. A., Campbell, M. G., Lee, I., Brewster, S. J., Hanson, E. M., Harris, H. K., Lowe, K. R., et al. (2012) · PLoS ONE · doi:10.1371/journal.pone.0049475 · Cited by 165 · see citing works
“Multivariate gene expression–based prediction models developed from cases and non-cases have been widely used for diagnosis, screening, prediction of treatment response, and prognosis [13,14].”
Cancer gene expression signatures – The rise and fall?
Chibon, F. (2013) · European Journal of Cancer · doi:10.1016/j.ejca.2013.02.021 · Cited by 164 · see citing works
Divergent Whole-Genome Methylation Maps of Human and Chimpanzee Brains Reveal Epigenetic Basis of Human Regulatory Evolution
Zeng, J., Konopka, G., Hunt, B. G., Preuss, T. M., Geschwind, D. H., & Yi, S. V. (2012) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2012.07.024 · Cited by 163 · see citing works
Synthetic lethality-mediated precision oncology via the tumor transcriptome
Lee, J. S., Nair, N. U., Dinstag, G., Chapman, L. M., Chung, Y., Wang, K., Sinha, S., Cha, H., Kim, D., Schperberg, A. V., et al. (2021) · Cell · doi:10.1016/j.cell.2021.03.030 · Cited by 154 · see citing works
Two gene co-expression modules differentiate psychotics and controls
Chen, C., Cheng, L., Grennan, K., Pibiri, F., Zhang, C., Badner, J. A., Members of the Bipolar Disorder Genome Study (BiGS) Consortium, Gershon, E. S., & Liu, C. (2013) · Molecular Psychiatry · doi:10.1038/mp.2012.146 · Cited by 157 · see citing works
G2M Cell Cycle Pathway Score as a Prognostic Biomarker of Metastasis in Estrogen Receptor (ER)-Positive Breast Cancer
Oshi, M., Takahashi, H., Tokumaru, Y., Yan, L., Rashid, O. M., Matsuyama, R., Endo, I., & Takabe, K. (2020) · International Journal of Molecular Sciences · doi:10.3390/ijms21082921 · Cited by 157 · see citing works
“(accession number GSE20194; n = 278) [51], Symmans et al.”
Background
The Evolution of Bioinformatics in Toxicology: Advancing Toxicogenomics
Afshari, C. A., Hamadeh, H. K., & Bushel, P. R. (2010) · Toxicological Sciences · doi:10.1093/toxsci/kfq373 · Cited by 152 · see citing works
A Decade of Toxicogenomic Research and Its Contribution to Toxicological Science
Chen, M., Zhang, M., Borlak, J. T., & Tong, W. (2012) · Toxicological Sciences · doi:10.1093/toxsci/kfs223 · Cited by 152 · see citing works
Empirical Comparison of Area under ROC curve (AUC) and Mathew Correlation Coefficient (MCC) for Evaluating Machine Learning Algorithms on Imbalanced Datasets for Binary Classification
Chongomweru, H., Kasem, A., & Newaz, S. H. S. (2019) · International Conference on Machine Learning and Soft Computing (ICMLSC) · doi:10.1145/3310986.3311023 · Cited by 150 · see citing works
“Today MCC is becoming widely used metric in biomedical research [9,15,20].”
Background
Criteria for the use of omics-based predictors in clinical trials: explanation and elaboration
McShane, L. M., Cavenagh, M. M., Lively, T., Eberhard, D. A., Bigbee, W. L., Williams, P. M., Mesirov, J. P., Polley, M. C., Kim, K. Y., Tricoli, J. V., et al. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-220 · Cited by 140 · see citing works
“evaluate whether the models built using each subset have similar performance on the other subset [42].”
Method
BERMUDA: a novel deep transfer learning method for single-cell RNA sequencing batch correction reveals hidden high-resolution cellular subtypes
Wang, T., Johnson, T. S., Shao, W., Lu, Z., Helm, B. R., Zhang, J., & Huang, K. (2019) · Genome biology · doi:10.1186/s13059-019-1764-6 · Cited by 140 · see citing works
Performance assessment of DNA sequencing platforms in the ABRF Next-Generation Sequencing Study
Foox, J., Tighe, S. W., Nicolet, C. M., Zook, J. M., Byrska‐Bishop, M. S., Clarke, W. E., Khayat, M. M., Mahmoud, M., Laaguiby, P. K., Herbert, Z. T., et al. (2021) · Nature Biotechnology · doi:10.1038/s41587-021-01049-5 · Cited by 139 · see citing works
Multi-omics integration accurately predicts cellular state in unexplored conditions for Escherichia coli
Kim, M., Rai, N., Zorraquino, V., & Tagkopoulos, I. (2016) · Nature Communications · doi:10.1038/ncomms13090 · Cited by 137 · see citing works
In silico Pathway Activation Network Decomposition Analysis (iPANDA) as a method for biomarker development
Ozerov, I. V., Lezhnina, K. V., Izumchenko, E., Artemov, A. V., Medintsev, S., Vanhaelen, Q., Aliper, A. M., Vijg, J., Osipov, A. N., Labat, I., et al. (2016) · Nature Communications · doi:10.1038/ncomms13427 · Cited by 136 · see citing works
k-Nearest neighbor models for microarray gene expression analysis and clinical outcome prediction
Parry, R. M., Jones, W. D., Stokes, T. H., Phan, J. H., Moffitt, A. B., Fang, H., Shi, L. M., Oberthuer, A., Fischer, M., Tong, W., et al. (2010) · The Pharmacogenomics Journal · doi:10.1038/tpj.2010.56 · Cited by 136 · see citing works
Textural differences between renal cell carcinoma subtypes: Machine learning-based quantitative computed tomography texture analysis with independent external validation
Koçak, B., Yardımcı, A. H., Bektaş, C. T., Türkcanoğlu, M. H., Erdim, Ç., Yücetaş, U., Koca, S. B., & Kılıçkesmez, Ö. (2018) · European Journal of Radiology · doi:10.1016/j.ejrad.2018.08.014 · Cited by 135 · see citing works
Grading of soft tissue sarcomas: from histological to molecular assessment
Neuville, A., Chibon, F., & Coindre, J. (2013) · Pathology · doi:10.1097/pat.0000000000000048 · Cited by 134 · see citing works
The challenge of the application of 'omics technologies in chemicals risk assessment: Background and outlook
Sauer, U. G., Deferme, L., Gribaldo, L., Hackermüller, J., Tralau, T., van Ravenzwaay, B., Yauk, C. L., Poole, A., Tong, W., & Gant, T. W. (2017) · Regulatory Toxicology and Pharmacology · doi:10.1016/j.yrtph.2017.09.020 · Cited by 134 · see citing works
The E2F Pathway Score as a Predictive Biomarker of Response to Neoadjuvant Therapy in ER+/HER2− Breast Cancer
Oshi, M., Takahashi, H., Tokumaru, Y., Yan, L., Rashid, O. M., Nagahashi, M., Matsuyama, R., Endo, I., & Takabe, K. (2020) · Cells · doi:10.3390/cells9071643 · Cited by 128 · see citing works
Developments in toxicogenomics: understanding and predicting compound-induced toxicity from gene expression data
Alexander-Dann, B., Pruteanu, L.-L., Oerton, E., Sharma, N., Berindan‐Neagoe, I., Módos, D., & Bender, A. (2018) · Molecular Omics · doi:10.1039/c8mo00042e · Cited by 130 · see citing works
Ultrasensitive detection of rare mutations using next-generation targeted resequencing
Flaherty, P., Natsoulis, G., Muralidharan, O., Winters, M. A., Buenrostro, J. D., Bell, J., Brown, S. T., Holodniy, M., Zhang, N. R., & Ji, H. P. (2011) · Nucleic Acids Research · doi:10.1093/nar/gkr861 · Cited by 127 · see citing works
“Given the issue of the true positives relative to the true negatives being highly skewed (18), we report a balanced measure for the classification accuracy using the Matthews Correlation coefficient (MCC) as was done for the MAQCII analysis (19).”
Method
Centromere protein-A, an essential centromere protein, is a prognostic marker for relapse in estrogen receptor-positive breast cancer
McGovern, S. L., Qi, Y., Pusztai, L., Symmans, W. F., & Buchholz, T. A. (2012) · Breast Cancer Research · doi:10.1186/bcr3181 · Cited by 125 · see citing works
Toxicogenomics: A 2020 Vision
Liu, Z., Huang, R., Roberts, R. A., & Tong, W. (2018) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2018.12.001 · Cited by 127 · see citing works
Strengths and limitations of microarray-based phenotype prediction: lessons learned from the IMPROVER Diagnostic Signature Challenge
Tarca, A. L., Lauria, M., Unger, M., Bilal, E., Boué, S., Dey, K. K., Hoeng, J., Koeppl, H., Martin, F. P., Meyer, P., et al. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt492 · Cited by 121 · see citing works
Anticipatory estrogen activation of the unfolded protein response is linked to cell proliferation and poor survival in estrogen receptor α-positive breast cancer
Andruska, N. D., Zheng, X., Yang, X., Helferich, W. G., & Shapiro, D. J. (2014) · Oncogene · doi:10.1038/onc.2014.292 · Cited by 122 · see citing works
Systematic Analysis of Challenge-Driven Improvements in Molecular Prognostic Models for Breast Cancer
Margolin, A. A., Bilal, E., Huang, E. S., Norman, T., Ottestad, L., Mecham, B. H., Sauerwine, B., Kellen, M., Mangravite, L. M., Furia, M. D., et al. (2013) · Science Translational Medicine · doi:10.1126/scitranslmed.3006112 · Cited by 122 · see citing works
Challenges translating breast cancer gene signatures into the clinic
Weigelt, B., Pusztai, L., Ashworth, A., & Reis-Filho, J. S. (2011) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2011.125 · Cited by 119 · see citing works
The Prognostic Role of Androgen Receptor in Patients with Early-Stage Breast Cancer: A Meta-analysis of Clinical and Gene Expression Data
Božović‐Spasojević, I., Zardavas, D., Brohée, S., Ameye, L., Fumagalli, D., Ades, F., de Azambuja, E., Barèche, Y., Piccart, M. J., Paesmans, M., et al. (2016) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-16-0979 · Cited by 117 · see citing works
Practices and Trends of Machine Learning Application in Nanotoxicology
Furxhi, I., Murphy, F., Mullins, M., Arvanitis, A., & Poland, C. A. (2020) · Nanomaterials · doi:10.3390/nano10010116 · Cited by 118 · see citing works
“MCC is equivalent to the Pearson Correlation Coefficient for binary variables [147] and has been selected as an evaluation metric for micro-array-based predictive models by the MicroArray Quality Control (MAQC) Consortium [148].”
Method
Asynchronous and pathological windows of implantation: two causes of recurrent implantation failure†
Sebastián-León, P., Garrido, N., Remohi, J., Sr., Pellicer, A., & Díaz-Gimeno, P. (2018) · Human Reproduction · doi:10.1093/humrep/dey023 · Cited by 117 · see citing works
Microarray Meta-Analysis and Cross-Platform Normalization: Integrative Genomics for Robust Biomarker Discovery
Walsh, C. J., Hu, P., Batt, J. A. E., & dos Santos, C. C. (2015) · Microarrays · doi:10.3390/microarrays4030389 · Cited by 116 · see citing works
Correcting batch effects in large-scale multiomics studies using a reference-material-based ratio method
Yu, Y., Zhang, N., Mai, Y., Ren, L., Chen, Q., Cao, Z., Chen, Q., Liu, Y., Hou, W., Yang, J., et al. (2023) · Genome biology · doi:10.1186/s13059-023-03047-z · Cited by 116 · see citing works
“Prediction models were developed and validated using a two-layer validation strategy [3].”
Method
Enabling transparent and collaborative computational analysis of 12 tumor types within The Cancer Genome Atlas
Omberg, L., Ellrott, K., Yuan, Y., Kandoth, C., Wong, C., Kellen, M., Friend, S. H., Stuart, J. M., Liang, H., & Margolin, A. A. (2013) · Nature Genetics · doi:10.1038/ng.2761 · Cited by 113 · see citing works
Improving Breast Cancer Survival Analysis through Competition-Based Multidimensional Modeling
Bilal, E., Dutkowski, J., Guinney, J., Jang, I. S., Logsdon, B. A., Pandey, G. S., Sauerwine, B., Shimoni, Y., Vollan, H. K. M., Mecham, B. H., et al. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003047 · Cited by 112 · see citing works
Systematic integration of RNA-Seq statistical algorithms for accurate detection of differential gene expression patterns
Moulos, P., & Hatzis, P. (2014) · Nucleic Acids Research · doi:10.1093/nar/gku1273 · Cited by 107 · see citing works
Systems biology approaches to new vaccine development
Oberg, A. L., Kennedy, R. B., Li, P., Ovsyannikova, I. G., & Poland, G. A. (2011) · Current Opinion in Immunology · doi:10.1016/j.coi.2011.04.005 · Cited by 111 · see citing works
Systems vaccinology: learning to compute the behavior of vaccine induced immunity
Nakaya, H. I., Li, S., & Pulendran, B. (2011) · WIREs Systems Biology and Medicine · doi:10.1002/wsbm.163 · Cited by 106 · see citing works
Immune cell profiling in cancer: molecular approaches to cell-specific identification
Lyons, Y. A., Wu, S. Y., Overwijk, W. W., Baggerly, K., & Sood, A. K. (2017) · npj Precision Oncology · doi:10.1038/s41698-017-0031-0 · Cited by 104 · see citing works

Microarrays in the clinic
Tillinghast, G. W. (2010) · Nature Biotechnology · doi:10.1038/nbt0810-810 · Cited by 19 · see citing works
Selecting a single model or combining multiple models for microarray-based classifier development? – A comparative analysis based on large and diverse datasets generated from the MAQC-II project
Chen, M., Shi, L. M., Kelly, R. J., Perkins, R., Fang, H., & Tong, W. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-s10-s3 · Cited by 19 · see citing works
The MAQC-II Project: A comprehensive study of common practices for the development and validation of microarray-based predictive models.
Shi, L. M., Campbell, G. R., Jones, W. D., Campagne, F., Wen, Z., Walker, S. J., Su, Z., Chu, T., Goodsaid, F. M., Pusztai, L., et al. (2010) · Cited by 37 · see citing works
Reproducible and reliable microarray results through quality control: good laboratory proficiency and appropriate data analysis practices are essential
Shi, L. M., Perkins, R., Fang, H., & Tong, W. (2008) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2007.11.003 · Cited by 139 · see citing works
Consistency of predictive signature genes and classifiers generated using different microarray platforms
Fan, X., Lobenhofer, E. K., Chen, M., Shi, W., Huang, J., Luo, J., Zhang, J. H., Walker, S. J., Chu, T.-M., Li, L., et al. (2010) · The Pharmacogenomics Journal · doi:10.1038/tpj.2010.34 · Cited by 67 · see citing works

Authors: MAQC Consortium
Venue: Nature Biotechnology
Publisher: Springer Science and Business Media LLC
Published: 2010-07-30
Volume: 28
Issue: 8
Pages: 827-838
ISSN: 0733-222X (ISSN-L)
1087-0156
1546-1696
DOI: 10.1038/nbt.1665
PMC: PMC3315840
OpenAlex: W2021560851
Cited by: 889
References: 63
Citations by year: 2026: 13 · 2025: 33 · 2024: 39 · 2023: 35 · 2022: 40 · 2021: 58 · 2020: 61 · 2019: 58 · 2018: 46 · 2017: 35 · 2016: 57 · 2015: 54 · 2014: 87 · 2013: 85 · 2012: 79
FWCI: 34.94
Type: journal article
Language: en
Open access: green
www.ncbi.nlm.nih.gov
Licence: www.springer.com/tdm (tdm)
www.springer.com/tdm (vor)
Locations: Nature Biotechnology (publishedVersion)
PubMed (publishedVersion)
Europe PMC (PubMed Central) (acceptedVersion)
PubMed Central (acceptedVersion)
Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles) (submittedVersion)
Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
Institute of Process Engineering, CAS Institutional Repository (submittedVersion, cc-by-nc-sa)
IRIS (submittedVersion)
Repository of Futwangen University of Applied Sciences (Furtwangen University) (submittedVersion)
SERVAL (Université de Lausanne) (submittedVersion)
Funders: National Institutes of Health (Z99 LM999999, Z99 HL999999, Z99 MH999999, Z99CL999999)
National Cancer Institute (CA97513-01, CA55819-01A1, R00CA126184, P01CA055819, R33CA097513)
National Institute of General Medical Sciences (T32GM074906, R01GM083084, R01GM083226, R01GM072611)
National Center for Research Resources (R01RR021967)
Topics: Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
Life Sciences › Biochemistry, Genetics and Molecular Biology › Cancer Research › Breast Cancer Treatment Studies
Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
Keywords: microarray quality control, gene expression profiling, prediction model, genomic classifier, model validation, pulmonary toxicity, hepatotoxicity, breast cancer, multiple myeloma, neuroblastoma, rodent toxicity, method comparison, reproducibility
MeSH terms: Animals, Breast Neoplasms, Disease Models, Animal, Female, Humans, Liver Diseases, Lung Diseases, Multiple Myeloma, Neoplasms, Neuroblastoma, Predictive Value of Tests, Quality Control, Survival Analysis, Guidelines as Topic, Oligonucleotide Array Sequence Analysis, Gene Expression Profiling, Rats

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

The MicroArray Quality Control (MAQC)-II study of common practices for the development and validation of microarray-based predictive models

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Self-check

Cites

Cited by

Related

Record

About these checks