Pitfalls in the Use of DNA Microarray Data for Diagnostic and Prognostic Classification
The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
DNA microarrays have made it possible to estimate the level of expression of thousands of genes for a sample of cells. Although biomedical investigators have been quick to adopt this powerful new research tool, accurate analysis and interpretation of the data have provided unique challenges. Indeed, many investigators are not experienced in the analytical steps needed to convert tens of thousands of noisy data points into reliable and interpretable biologic information. Although some investigators recognize the importance of collaborating with experienced biostatisticians to analyze microarray data, the number and availability of experienced biostatisticians is inadequate. Consequently, investigators are using available software to analyze their data, many seemingly without knowledge of potential pitfalls. Because of serious problems associated with the analysis and reporting of some DNA microarray studies, there is great interest in guidance on valid and effective methods for analysis of DNA microarray data. The design and analysis strategy for a DNA microarray experiment should be determined in light of the overall objectives of the study. Because DNA microarrays are used for a wide variety of objectives, it is not feasible to address the entire range of design and analysis issues in this commentary. Here, we address statistical issues that arise from the use of DNA microarrays for an important group of objectives that has been called “class prediction” (1). Class prediction includes derivation of predictors of prognosis, response to therapy, or any phenotype or genotype defined independently of the gene expression profile. Good DNA microarray experiments, although not based on gene-specific mechanistic hypotheses, should be planned and conducted with clear objectives. Three commonly encountered types of study objectives are “class comparison,” “class prediction,” and “class discovery” (1). Class comparison is the comparison of gene expression in different groups of specimens. The major characteristic of class comparison studies is that the classes being compared are defined independently of the expression profiles. The specific objectives of such a study are to determine whether the expression profiles are different between the classes and, if so, to identify the differentially expressed genes. One example of a class comparison study is the comparison of gene expression profiles of stage I breast cancer patients who are long-term survivors with the gene expression profiles of those who have recurrent disease. Another example is the comparison between gene expression profiles in breast cancer patients with and without germline BRCA1 mutations (2). Class prediction studies are similar to class comparison studies in that the classes are predefined. In class prediction studies, however, the emphasis is on developing a gene expression-based multivariate function (referred to as the predictor) that accurately predicts the class membership of a new sample on the basis of the expression levels of key genes. Such predictors can be used for many types of clinical management decisions, including risk assessment, diagnostic testing, prognostic stratification, and treatment selection. Many studies include both class comparison and class prediction objectives. Class discovery is fundamentally different from class comparison or class prediction in that no classes are predefined. Usually the purpose of class discovery in cancer studies is to determine whether discrete subsets of a disease entity can be defined on the basis of gene expression profiles. This purpose is different from determining whether the gene expression profiles correlate with some already known diagnostic classification. Examples of class discovery are the studies by Bittner et al. (3) that examined gene expression profiles for advanced melanomas and by Alizadeh et al. (4) that examined the gene expression profiles of patients with diffuse large B-cell lymphoma. Often the purpose of class discovery is to identify clues regarding the heterogeneity of disease pathogenesis. One of the most common errors in the analysis of DNA microarray data is the use of cluster analysis and simple fold change statistics for problems of class comparison and class prediction. Although cluster analysis is appropriate for class discovery, it is often not effective for class comparison or class prediction. Cluster analysis refers to an extensive set of methods for partitioning samples into groups on the basis of the similarities and differences (referred to as distances) among their gene expression profiles. Because there are many ways of measuring distances among gene expression profiles involving thousands of genes and because there are many algorithms for partitioning, cluster analysis is a very subjective analysis strategy. Cluster analysis is considered an unsupervised method of analysis because no information about sample grouping is used. The distance measures are ge
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 20,825
Golub, T. R., Slonim, D. K., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J. P., Coller, H., Loh, M. L., Downing, J. R., Caligiuri, M. A., et al. (1999) · Science · doi:10.1126/science.286.5439.531 · Cited by 11,516
Distinct types of diffuse large B-cell lymphoma identified by gene expression profiling
Alizadeh, A. A., Eisen, M. B., Davis, R. E., Ma, C., Lossos, I. S., Rosenwald, A., Boldrick, J. C., Sabet, H., Tran, T., Yu, X., et al. (2000) · Nature · doi:10.1038/35000501 · Cited by 9,925
Gene expression profiling predicts clinical outcome of breast cancer
van 't Veer, L. J., Dai, H., van de Vijver, M. J., He, Y. D., Hart, A. A. M., Mao, M., Peterse, H. L., van der Kooy, K., Marton, M. J., Witteveen, A. T., et al. (2002) · Nature · doi:10.1038/415530a · Cited by 9,536 · cited 4×
“However, returning to the study of clinical outcome in breast cancer by van’t Veer et al. ( 14 ), a validation set of nineteen specimens was examined, which contained seven specimens from patients who were disease-free at 5 years.”
InfluentialPattern Recognition and Neural Networks
Ripley, B. D. (1996) · Cambridge University Press eBooks · doi:10.1017/cbo9780511812651 · Cited by 6,459
Rosenwald, A., Wright, G., Chan, W. C., Connors, J. M., Campo, E., Fisher, R. I., Gascoyne, R. D., Muller-Hermelink, H. K., Smeland, E. B., Giltnane, J. M., et al. (2002) · New England Journal of Medicine · doi:10.1056/nejmoa012914 · Cited by 3,829
Self-citationComparison of Discrimination Methods for the Classification of Tumors Using Gene Expression Data
Dudoit, S., Fridlyand, J., & Speed, T. P. (2002) · Journal of the American Statistical Association · doi:10.1198/016214502753479248 · Cited by 2,672
Khan, J., Wei, J. S., Ringnér, M., Saal, L. H., Ladanyi, M., Westermann, F., Berthold, F., Schwab, M., Antonescu, C. R., Peterson, C., et al. (2001) · Nature Medicine · doi:10.1038/89044 · Cited by 2,644
Bhattacharjee, A., Richards, W. G., Staunton, J., Li, C., Monti, S., Vasa, P., Ladd, C., Beheshti, J., Bueno, R., Gillette, M., et al. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.191502998 · Cited by 2,576
Shipp, M. A., Ross, K. N., Tamayo, P., Weng, A. P., Kutok, J. L., Aguiar, R. C. T., Gaasenbeek, M., Angelo, M., Reich, M., Pinkus, G. S., et al. (2002) · Nature Medicine · doi:10.1038/nm0102-68 · Cited by 2,428
Molecular classification of cutaneous malignant melanoma by gene expression profiling
Bittner, M., Meltzer, P., Chen, Y., Jiang, Y., Seftor, E., Hendrix, M., Radmacher, M., Simon, R., Yakhini, Z., Ben-Dor, A., et al. (2000) · Nature · doi:10.1038/35020115 · Cited by 1,990
Self-citationGene-Expression Profiles in Hereditary Breast Cancer
Hedenfalk, I., Duggan, D., Chen, Y., Radmacher, M., Bittner, M., Simon, R., Meltzer, P., Gusterson, B., Esteller, M., Raffeld, M., et al. (2001) · New England Journal of Medicine · doi:10.1056/nejm200102223440801 · Cited by 1,733
Self-citationImprovements on Cross-Validation: The 632+ Bootstrap Method
Efron, B., & Tibshirani, R. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474007 · Cited by 1,636
Estimation of Error Rates in Discriminant Analysis
Lachenbruch, P. A., & Mickey, M. R. (1968) · Technometrics · doi:10.1080/00401706.1968.10490530 · Cited by 1,563
Predicting the clinical status of human breast cancer by using gene expression profiles
West, M., Blanchette, C., Dressman, H., Huang, E., Ishida, S., Spang, R., Zuzan, H., Olson, J. A., Marks, J. R., & Nevins, J. R. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.201162998 · Cited by 1,318
Diversity of gene expression in adenocarcinoma of the lung
Garber, M. E., Troyanskaya, O. G., Schluens, K., Petersen, S., Thaesler, Z., Pacyna-Gengelbach, M., van de Rijn, M., Rosen, G. D., Perou, C. M., Whyte, R. I., et al. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.241500798 · Cited by 1,298
Tumor classification by partial least squares using microarray gene expression data
Nguyen, D. V., & Rocke, D. M. (2002) · Bioinformatics · doi:10.1093/bioinformatics/18.1.39 · Cited by 816
MacDonald, T. J., Brown, K. M., LaFleur, B., Peterson, K., Lawlor, C., Chen, Y., Packer, R. J., Cogen, P., & Stephan, D. A. (2001) · Nature Genetics · doi:10.1038/ng731 · Cited by 455
A Paradigm for Class Prediction Using Gene Expression Profiles
Radmacher, M. D., McShane, L. M., & Simon, R. (2002) · Journal of Computational Biology · doi:10.1089/106652702760138592 · Cited by 313
Self-citationAllocation Rules and Their Error Rates
Hills, M. (1966) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1966.tb00614.x · Cited by 215
Caldas, C., & Aparicio, S. A. J. R. (2002) · Nature · doi:10.1038/415484a · Cited by 92
Cited by
Gene selection and classification of microarray data using random forest
Díaz-Uriarte, R., & Alvarez de Andrés, S. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-3 · Cited by 2,929
Bias in error estimation when using cross-validation for model selection
Varma, S., & Simon, R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-91 · Cited by 2,485
“[2], Ambroise and McLachlan [3] and Reunanen [4], this gives a very biased estimate of the true error; not much better than the resubstitution estimate.”
Self-citationThe molecular portraits of breast tumors are conserved across microarray platforms
Hu, Z., Fan, C., Oh, D. S., Marron, J., He, X., Qaqish, B. F., Livasy, C., Carey, L. A., Reynolds, E., Dressler, L., et al. (2006) · BMC Genomics · doi:10.1186/1471-2164-7-96 · Cited by 1,491
Prediction error estimation: a comparison of resampling methods
Molinaro, A. M., Simon, R., & Pfeiffer, R. M. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti499 · Cited by 1,339
Self-citationBuyse, M., Loi, S., van't Veer, L., Viale, G., Delorenzi, M., Glas, A. M., Saghatchian d'Assignies, M., Bergh, J., Lidereau, R., Ellis, P., et al. (2006) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djj329 · Cited by 1,238
Metadata
- Authors
- Simon, R., Radmacher, M. D., Dobbin, K., & McShane, L. M.
- Venue
- JNCI Journal of the National Cancer Institute
- Publisher
- Oxford University Press (OUP)
- Published
- 2003-01-01
- Volume
- 95
- Issue
- 1
- Pages
- 14-18
- ISSN
- 0027-8874 (ISSN-L)
- 1460-2105
- OpenAlex
- W2043235003
- Cited by
- 1,006
- References
- 21
- FWCI
- 28.20
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.