Repeatability and Reproducibility in Proteomic Identifications by Liquid Chromatography−Tandem Mass Spectrometry
Open accessAbstract
The complexity of proteomic instrumentation for LC-MS/MS introduces many possible sources of variability. Data-dependent sampling of peptides constitutes a stochastic element at the heart of discovery proteomics. Although this variation impacts the identification of peptides, proteomic identifications are far from completely random. In this study, we analyzed interlaboratory data sets from the NCI Clinical Proteomic Technology Assessment for Cancer to examine repeatability and reproducibility in peptide and protein identifications. Included data spanned 144 LC-MS/MS experiments on four Thermo LTQ and four Orbitrap instruments. Samples included yeast lysate, the NCI-20 defined dynamic range protein mix, and the Sigma UPS 1 defined equimolar protein mix. Some of our findings reinforced conventional wisdom, such as repeatability and reproducibility being higher for proteins than for peptides. Most lessons from the data, however, were more subtle. Orbitraps proved capable of higher repeatability and reproducibility, but aberrant performance occasionally erased these gains. Even the simplest protein digestions yielded more peptide ions than LC-MS/MS could identify during a single experiment. We observed that peptide lists from pairs of technical replicates overlapped by 35-60%, giving a range for peptide-level repeatability in these experiments. Sample complexity did not appear to affect peptide identification repeatability, even as numbers of identified spectra changed by an order of magnitude. Statistical analysis of protein spectral counts revealed greater stability across technical replicates for Orbitraps, making them superior to LTQ instruments for biomarker candidate discovery. The most repeatable peptides were those corresponding to conventional tryptic cleavage sites, those that produced intense MS signals, and those that resulted from proteins generating many distinct peptides. Reproducibility among different instruments of the same type lagged behind repeatability of technical replicates on a single instrument by several percent. These findings reinforce the importance of evaluating repeatability as a fundamental characteristic of analytical technologies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In DDA MS, if the number of precursor ions exceeds the number of precursor selection cycles, precursor selection becomes stochastic and the peptides detected in repeat analyses become irreproducible.
“If the number of precursor ions exceeds the number of precursor selection cycles6, precursor selection becomes stochastic and the peptides detected in repeat analyses become irreproducible.”
From Multi-laboratory assessment of reproducibility, qualitative and quantitative performance of SWATH-mass spectrometry (2017) · cites this paper as [7]
A DDA-based study could not achieve such a high level of completeness across labs due to stochastic MS2 sampling and would likely experience difficulty aligning MS1 signals across labs where chromatography varies.
“We expect that a DDA-based study could not achieve such a high level of completeness across labs due to stochastic MS2 sampling7 and such a study is likely to experience difficulty aligning MS1 signals arising from different labs where chromatography will inevitably vary (Supplementary Fig. 4).”
From Multi-laboratory assessment of reproducibility, qualitative and quantitative performance of SWATH-mass spectrometry (2017) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965 · see citing works
Beckett, A. H., Rowland, M., & TURNER, P. (1965) · The Lancet · doi:10.1016/s0140-6736(65)91037-8 · Cited by 16,917 · see citing works
Keller, A., Nesvizhskii, A., Kolker, E., & Aebersold, R. (2002) · Analytical Chemistry · doi:10.1021/ac025747h · Cited by 4,970 · see citing works
A Model for Random Sampling and Estimation of Relative Protein Abundance in Shotgun Proteomics
Liu, H., Sadygov, R. G., & Yates, J. R. (2004) · Analytical Chemistry · doi:10.1021/ac0498563 · Cited by 2,486 · see citing works
ProteoWizard: open source software for rapid proteomics tools development
Kessner, D., Chambers, M., Burke, R., Agus, D., & Mallick, P. (2008) · Bioinformatics · doi:10.1093/bioinformatics/btn323 · Cited by 2,024 · see citing works
Protein biomarker discovery and validation: the long and uncertain path to clinical utility
Rifai, N., Gillette, M. A., & Carr, S. A. (2006) · Nature Biotechnology · doi:10.1038/nbt1235 · Cited by 1,867 · see citing works
Self-citationThe abc's (and xyz's) of peptide sequencing
Steen, H., & Mann, M. (2004) · Nature Reviews Molecular Cell Biology · doi:10.1038/nrm1468 · Cited by 1,139 · see citing works
Addona, T. A., Abbatiello, S. E., Schilling, B., Skates, S. J., Mani, D. R., Bunk, D. M., Spiegelman, C. H., Zimmerman, L. J., Ham, A.-J. L., Keshishian, H., et al. (2009) · Nature Biotechnology · doi:10.1038/nbt.1546 · Cited by 1,037 · see citing works
Self-citationOmenn, G. S., States, D. J., Adamski, M., Blackwell, T. W., Menon, R., Hermjakob, H., Apweiler, R., Haab, B. B., Simpson, R. J., Eddes, J. S., et al. (2005) · PROTEOMICS · doi:10.1002/pmic.200500358 · Cited by 787 · see citing works
Comparative evaluation of mass spectrometry platforms used in large-scale proteomics investigations
Elias, J. E., Haas, W., Faherty, B. K., & Gygi, S. P. (2005) · Nature Methods · doi:10.1038/nmeth785 · Cited by 703 · see citing works
Tabb, D. L., Fernando, C., & Chambers, M. (2007) · Journal of Proteome Research · doi:10.1021/pr0604054 · Cited by 548 · see citing works
Self-citationA method for reducing the time required to match protein sequences with tandem mass spectra
Craig, R., & Beavis, R. C. (2003) · Rapid Communications in Mass Spectrometry · doi:10.1002/rcm.1198 · Cited by 507 · see citing works
Standardizing global gene expression analysis between laboratories and across platforms
Weis, B. K. (2005) · Nature Methods · doi:10.1038/nmeth754 · Cited by 465 · see citing works
Proteomic Parsimony through Bipartite Graph Analysis Improves Accuracy and Transparency
Zhang, B., Chambers, M., & Tabb, D. L. (2007) · Journal of Proteome Research · doi:10.1021/pr070230d · Cited by 374 · see citing works
Self-citationA HUPO test sample study reveals common problems in mass spectrometry–based proteomics
HUPO Test Sample Working Group, Bell, A. W., Deutsch, E. W., Au, C. E., Kearney, R. E., Beavis, R., Sechi, S., Nilsson, T., & Bergeron, J. J. M. (2009) · Nature Methods · doi:10.1038/nmeth.1333 · Cited by 351 · see citing works
Kapp, E. A., Schütz, F., Connolly, L., Chakel, J. A., Meza, J. E., Miller, C. A., Fenyö, D., Eng, J. K., Adkins, J., Omenn, G. S., et al. (2005) · PROTEOMICS · doi:10.1002/pmic.200500126 · Cited by 344 · see citing works
IDPicker 2.0: Improved Protein Assembly with High Discrimination Peptide Identification Filtering
Ma, Z.-Q., Dasari, S., Chambers, M., Litton, M. D., Sobecki, S. M., Zimmerman, L. J., Halvey, P. J., Schilling, B., Drake, P. M., Gibson, B. W., et al. (2009) · Journal of Proteome Research · doi:10.1021/pr900360j · Cited by 328 · see citing works
Self-citationStatus of complete proteome analysis by mass spectrometry: SILAC labeled yeast as a model system
de Godoy, L. M., Olsen, J. V., de Souza, G. A., Li, G., Mortensen, P., & Mann, M. (2006) · Genome biology · doi:10.1186/gb-2006-7-6-r50 · Cited by 275 · see citing works
Improving Reproducibility and Sensitivity in Identifying Human Proteins by Shotgun Proteomics
Resing, K. A., Meyer-Arendt, K., Mendoza, A. M., Aveline-Wolf, L. D., Jonscher, K. R., Pierce, K. G., Old, W. M., Cheung, T. H., Russell, S. A., Wattawa, J. L., et al. (2004) · Analytical Chemistry · doi:10.1021/ac035229m · Cited by 228 · see citing works
Rudnick, P. A., Clauser, K. R., Kilpatrick, L. E., Tchekhovskoi, D. V., Neta, P., Blonder, N., Billheimer, D. D., Blackman, R. K., Bunk, D. M., Cardasis, H. L., et al. (2010) · Molecular & Cellular Proteomics · doi:10.1074/mcp.m900223-mcp200 · Cited by 183 · see citing works
Self-citationTabb, D. L., MacCoss, M. J., Wu, C. C., Anderson, S., & Yates, J. R. (2003) · Analytical Chemistry · doi:10.1021/ac026424o · Cited by 163 · see citing works
Self-citationPaulovich, A. G., Billheimer, D. D., Ham, A. L., Vega‐Montoto, L., Rudnick, P. A., Tabb, D. L., Wang, P., Blackman, R. K., Bunk, D. M., Cardasis, H. L., et al. (2009) · Molecular & Cellular Proteomics · doi:10.1074/mcp.m900222-mcp200 · Cited by 159 · see citing works
Self-citationKislinger, T., Gramolini, A. O., MacLennan, D. H., & Emili, A. (2005) · Journal of the American Society for Mass Spectrometry · doi:10.1016/j.jasms.2005.02.015 · Cited by 139 · see citing works
Washburn, M. P., Ulaszek, R. R., & Yates, J. R. (2003) · Analytical Chemistry · doi:10.1021/ac034120b · Cited by 120 · see citing works
Signal Maps for Mass Spectrometry-based Comparative Proteomics
Prakash, A., Mallick, P., Whiteaker, J. R., Zhang, H., Paulovich, A. G., Flory, M. R., Lee, H., Aebersold, R. H., & Schwikowski, B. (2005) · Molecular & Cellular Proteomics · doi:10.1074/mcp.m500133-mcp200 · Cited by 111 · see citing works
Self-citationvan Midwoud, P. M., Rieux, L., Bischoff, R., Verpoorte, E. M. J., & Niederländer, H. A. G. (2007) · Journal of Proteome Research · doi:10.1021/pr0604099 · Cited by 102 · see citing works
Slebos, R. J. C., Brock, J. W. C., Winters, N. F., Stuart, S. R., Martinez, M. A., Li, M., Chambers, M. C., Zimmerman, L. J., Ham, A. L., Tabb, D. L., et al. (2008) · Journal of Proteome Research · doi:10.1021/pr8004666 · Cited by 91 · see citing works
Self-citationPost Analysis Data Acquisition for the Iterative MS/MS Sampling of Proteomics Mixtures
Hoopmann, M. R., Merrihew, G. E., von Haller, P. D., & MacCoss, M. J. (2009) · Journal of Proteome Research · doi:10.1021/pr800828p · Cited by 59 · see citing works
Rudomin, E. L., Carr, S. A., & Jaffe, J. D. (2009) · Journal of Proteome Research · doi:10.1021/pr801017a · Cited by 37 · see citing works
Self-citationValid data from large-scale proteomics studies
Chamrad, D. C., & Meyer, H. E. (2005) · Nature Methods · doi:10.1038/nmeth0905-647 · Cited by 31 · see citing works
Delmotte, N., Lasaosa, M., Tholey, A., Heinzle, E., DORSSELAER, A. V., & Huber, C. G. (2009) · Journal of Separation Science · doi:10.1002/jssc.200800615 · Cited by 31 · see citing works
Balgley, B. M., Wang, W., Song, T., Fang, X., Li, Y., & Lee, C. S. (2008) · Electrophoresis · doi:10.1002/elps.200800050 · Cited by 26 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [20]McNaught AD Wilkinson A Compendium of Chemical Terminology (the "Gold Book") 1997 2.02 ed. Oxford Blackwell Scientific Publications
Cited by
Sorted by citations
Bruderer, R., Bernhardt, O. M., Gandhi, T., Miladinović, S. M., Cheng, L.-Y., Messner, S., Ehrenberger, T., Zanotelli, V., Butscheid, Y., Escher, C., et al. (2015) · Molecular & Cellular Proteomics · doi:10.1074/mcp.m114.044305 · Cited by 1,238 · see citing works
Selected reaction monitoring–based proteomics: workflows, potential, pitfalls and future directions
Picotti, P., & Aebersold, R. (2012) · Nature Methods · doi:10.1038/nmeth.2015 · Cited by 1,302 · see citing works
Peterson, A. C., Russell, J. D., Bailey, D. J., Westphall, M. S., & Coon, J. J. (2012) · Molecular & Cellular Proteomics · doi:10.1074/mcp.o112.020131 · Cited by 1,288 · see citing works
Systems biology and multi-omics integration: viewpoints from the metabolomics research community
Pinu, F. R., Beale, D. J., Paten, A. M., Kouremenos, K., Swarup, S., Schirra, H. J., & Wishart, D. (2019) · Metabolites · doi:10.3390/metabo9040076 · Cited by 723 · see citing works
Collins, B. C., Hunter, C. L., Liu, Y., Schilling, B., Rosenberger, G., Bader, S. L., Chan, D. W., Gibson, B. W., Gingras, A.-C., Held, J. M., et al. (2017) · Nature Communications · doi:10.1038/s41467-017-00249-5 · Cited by 576 · see citing works
Self-citationMass spectrometry in high-throughput proteomics: ready for the big time
Nilsson, T., Mann, M., Aebersold, R., Yates, J. R., Bairoch, A., & Bergeron, J. J. M. (2010) · Nature Methods · doi:10.1038/nmeth0910-681 · Cited by 470 · see citing works
A repository of assays to quantify 10,000 human proteins by SWATH-MS
Rosenberger, G., Koh, C. C., Guo, T., Röst, H. L., Kouvonen, P., Collins, B. C., Heusel, M., Liu, Y., Caron, E., Vichalkovski, A., et al. (2014) · Scientific Data · doi:10.1038/sdata.2014.31 · Cited by 413 · see citing works
A systematic evaluation of normalization methods in quantitative label-free proteomics
Välikangas, T., Suomi, T., & Elo, L. L. (2016) · Briefings in Bioinformatics · doi:10.1093/bib/bbw095 · Cited by 336 · see citing works
Lambert, J.-P., Ivosev, G., Couzens, A. L., Larsen, B., Taipale, M., Lin, Z.-Y., Zhong, Q., Lindquist, S., Vidal, M., Aebersold, R., et al. (2013) · Nature Methods · doi:10.1038/nmeth.2702 · Cited by 318 · see citing works
A high-stringency blueprint of the human proteome
Adhikari, S., Nice, E. C., Deutsch, E. W., Lane, L., Omenn, G. S., Pennington, S. R., Paik, Y.-K., Overall, C. M., Corrales, F. J., Cristea, I. M., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-19045-9 · Cited by 210 · see citing works
Self-citationIdentification of Conserved Proteomic Networks in Neurodegenerative Dementia
Swarup, V., Chang, T. S., Duong, D. M., Dammer, E. B., Dai, J., Lah, J. J., Johnson, E. C. B., Seyfried, N. T., Levey, A. I., & Geschwind, D. H. (2020) · Cell Reports · doi:10.1016/j.celrep.2020.107807 · Cited by 78 · see citing works
Identification of conserved proteomic networks in neurodegenerative dementia
Swarup, V., Chang, T. S., Duong, D. M., Dammer, E. B., Lah, J. J., Johnson, E. E. C. B., Seyfried, N. T., Levey, A. I., & Geschwind, D. H. (2019) · doi:10.1101/825802 · Cited by 78 · see citing works
A size-exclusion-based approach for purifying extracellular vesicles from human plasma
Vanderboom, P. M., Dasari, S., Ruegsegger, G. N., Pataky, M. W., Lucien, F., Heppelmann, C. J., Lanza, I. R., & Nair, K. S. (2021) · Cell Reports Methods · doi:10.1016/j.crmeth.2021.100055 · Cited by 75 · see citing works
“54; Table S1), which is within the acceptable limits of typical label-free proteomics experiments (Al Shweiki et al., 2017; Shen et al., 2018; Tabb et al., 2010).”
BackgroundStem cell secretome, regeneration, and clinical translation: a narrative review
Gwam, C., Mohammed, N., & Ma, X. (2021) · Annals of Translational Medicine · doi:10.21037/atm-20-5030 · Cited by 72 · see citing works
Tian, S., Zhan, D., Yu, Y., Wang, Y., Liu, M., Tan, S., Li, Y., Song, L., Qin, Z., Li, X., et al. (2023) · Genome Biology · doi:10.1186/s13059-023-03048-y · Cited by 43 · see citing works
- Authors
- Tabb, D. L., Vega-Montoto, L., Rudnick, P. A., Variyath, A. M., Ham, A.-J. L., Bunk, D. M., Kilpatrick, L. E., Billheimer, D. D., Blackman, R. K., Cardasis, H. L., Carr, S. A., Clauser, K. R., Jaffe, J. D., Kowalski, K. A., Neubert, T. A., Regnier, F. E., Schilling, B., Tegeler, T. J., Wang, M., Wang, P., Whiteaker, J. R., Zimmerman, L. J., Fisher, S. J., Gibson, B. W., Kinsinger, C. R., Mesri, M., Rodriguez, H., Stein, S. E., Tempst, P., Paulovich, A. G., Liebler, D. C., & Spiegelman, C.
- Affiliations
- Broad Institute (US)
- Vanderbilt University Medical Center, Nashville, Tennessee 37232, Texas A&M University, College Station, Texas 77843, National Institute of Standards and Technology, Gaithersburg, Maryland 20899, NIST, Hollings Marine Laboratory, Charleston, South Carolina 29412, University of Arizona, Tucson, Arizona 85721, Broad Institute of MIT and Harvard, Cambridge, Massachusetts 02142, New York University School of Medicine, Skirball Institute, New York, New York 10016, Purdue University, Bindley Bioscience Center,...
- Corresponding author
- Tabb, David L.
- Liebler, Daniel C.
- ORCID
- Tabb, David L.
- Rudnick, Paul A.
- Variyath, Asokan Mulayath
- Ham, Amy-Joan L.
- Bunk, David M.
- Kilpatrick, Lisa E.
- Billheimer, Dean D.
- Carr, Steven A.
- Clauser, Karl R.
- Jaffe, Jacob D.
- Neubert, Thomas A.
- Regnier, Fred E.
- Schilling, Birgit
- Wang, Mu
- Wang, Pei
- Whiteaker, Jeffrey R.
- Zimmerman, Lisa J.
- Fisher, Susan J.
- Gibson, Bradford W.
- Kinsinger, Christopher R.
- Mesri, Mehdi
- Rodriguez, Henry
- Stein, Stephen E.
- Tempst, Paul
- Paulovich, Amanda G.
- Liebler, Daniel C.
- Venue
- Journal of Proteome Research
- Publisher
- American Chemical Society (ACS)
- Published
- 2009-12-15
- Volume
- 9
- Issue
- 2
- Pages
- 761-776
- ISSN
- 1535-3893 (ISSN-L)
- 1535-3907
- PMC
- PMC2818771
- OpenAlex
- W1968063944
- Cited by
- 551
- References
- 33
- Citations by year
- 2026: 16 · 2025: 20 · 2024: 27 · 2023: 49 · 2022: 33 · 2021: 41 · 2020: 33 · 2019: 44 · 2018: 28 · 2017: 37 · 2016: 31 · 2015: 38 · 2014: 34 · 2013: 34 · 2012: 30
- FWCI
- 12.18
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Journal of Proteome Research (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- U.S. Public Health Service (U24 126480, U24 126479, U24 126477)
- National Cancer Institute (U24CA126480, U24CA126485, U24CA126477, U24CA126479, U24CA126476)
- Topics
- Physical Sciences › Chemistry › Spectroscopy › Advanced Proteomics Techniques and Applications
- Physical Sciences › Chemistry › Spectroscopy › Mass Spectrometry Techniques and Applications
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Advanced Biosensing Techniques and Applications
- Keywords
- repeatability, reproducibility, peptide identification, protein identification, data-dependent acquisition, technical replicates, interlaboratory comparison, Orbitrap, spectral counts, tryptic cleavage sites, sample complexity, biomarker discovery, yeast lysate
- MeSH terms
- Chromatography, Liquid, Reproducibility of Results, Proteome, Tandem Mass Spectrometry
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
