Big Data: Astronomical or Genomical?
Open accessAbstract
Genomics is a Big Data science and is going to get much bigger, very soon, but it is not known whether the needs of genomics will exceed other Big Data domains. Projecting to the year 2025, we compared genomics with three other major generators of Big Data: astronomy, YouTube, and Twitter. Our estimates show that genomics is a "four-headed beast"--it is either on par with or the most demanding of the domains analyzed here in terms of data acquisition, storage, distribution, and analysis. We discuss aspects of new technologies that will need to be developed to rise up and meet the computational challenges that genomics poses for the near future. Now is the time for concerted, community-wide planning for the "genomical" challenges of the next decade.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Stephens et al. (2015) forecasts growth rates of between 2-40 million terabytes of genomics data every year by 2025.
“For illustration, (Stephens et al., 2015) forecasts growth rates of between 2-40 million terabytes of genomics data every year by 2025.”
From Will we run out of data? Limits of LLM scaling based on human-generated data (2022) · cites this paper as Stephens et al., 2015
The Data Storage section projects 2–40 exabytes—equivalent to 2–40 million terabytes—of storage capacity by 2025 for human genomes. It does not describe that figure as genomics data generated every year or as an annual growth rate. (LLM, conf=0.90)
Experimental — accuracy not yet measuredStephens et al. (2015) estimated that astronomy and genomics produce several million terabytes of compressed data per year, corresponding at face value to a stock of roughly 1e18 tokens, a thousand times larger than estimates of the stock for text, images and video.
“Stephens et al. (2015) estimated that astronomy and genomics produce several million terabytes of compressed data per year. At face value, this corresponds to a stock of roughly 1e18 tokens, a thousand times larger than our estimates of the stock for text, images and video.”
From Will we run out of data? Limits of LLM scaling based on human-generated data (2022) · cites this paper as Stephens et al. (2015)
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredThere are other sources that can provide orders of magnitude more data of various types, such as financial market data and scientific databases.
“But there are other sources that can provide orders of magnitude more data of various types (e.g. financial market data, scientific databases, etc.).”
From Will we run out of data? Limits of LLM scaling based on human-generated data (2022) · cites this paper as Stephens et al., 2015
The Data Acquisition section discusses growth in astronomy, YouTube, Twitter, and genomics, including a projected increase of up to two orders of magnitude in data acquisition within those domains. Searches for financial market data, scientific databases, and other sources producing orders of magnitude more data found no passage addressing that comparison. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
An integrated map of genetic variation from 1,092 human genomes
The 1000 Genomes Project Consortium (2012) · Nature · doi:10.1038/nature11632 · Cited by 7,711 · see citing works
How Many Species Are There on Earth and in the Ocean?
Mora, C., Tittensor, D. P., Adl, S., Simpson, A. G. B., & Worm, B. (2011) · PLoS Biology · doi:10.1371/journal.pbio.1001127 · Cited by 3,045 · see citing works
Personal omics profiling reveals dynamic molecular and medical phenotypes
Chen, R., Mias, G. I., Li-Pook-Than, J., Jiang, L., Lam, H. Y. K., Chen, R., Miriami, E., Karczewski, K. J., Hariharan, M., Dewey, F. E., et al. (2012) · Cell · doi:10.1016/j.cell.2012.02.009 · Cited by 1,291 · see citing works
Identification of a large set of rare complete human knockouts
Sulem, P., Helgason, H., Oddson, A., Stefansson, H., Gudjonsson, S. A., Zink, F., Hjartarson, E., Sigurdsson, G. T., Jonasdottir, A., Jonasdottir, A., et al. (2015) · Nature Genetics · doi:10.1038/ng.3243 · Cited by 220 · see citing works
Flagged first, then by citations
Research works printed but not linked: 27
- [2]Robinson G.E., et al, Creating a Buzz About Insect Genomes. Science, 2011 18: 1386. 21415334
- [4]Chin L., Andersen J.N., and Futreal P.A., Cancer genomics: from discovery science to personalized medicine. Nature medicine, 2011 17(3): 297– 303. 10.1038/nm.2323 21383744 · DOI
- [6]Giles J., Computational social science: Making the links. Nature, 2012 488(7412): 448– 50. 10.1038/488448a 22914149 · DOI
- [7]Pak, A. and P. Paroubek, Twitter as a Corpus for Sentiment Analysis and Opinion Mining, in LREC. 2010. p. 1320–1326.
- [16]Li J.Y., Wang J., and Zeigler R.S., The 3,000 rice genomes project: new opportunities and challenges for future rice research. GigaScience, 2014 3: 8 10.1186/2047-217X-3-8 24872878 PMC4035671 · DOI
- [17]Zhu J., A year of great leaps in genome research. Genome medicine, 2012 4(1): 4 10.1186/gm303 22293069 PMC3334552 · DOI
- [18]Eisen J.A., Environmental shotgun sequencing: its potential and challenges for studying the hidden world of microbes. PLoS Biology, 2007 5(3): e82 17355177 PMC1821061
- [19]Gevers D., et al, The Human Microbiome Project: a community resource for the healthy human microbiome. PLoS Biology, 2012 10(8): e1001377 10.1371/journal.pbio.1001377 22904687 PMC3419203 · DOI · Paper page
- [25]Soon W.W., Hariharan M., and Snyder M.P., High-throughput sequencing for biology and medicine. Molecular systems biology, 2013 9: 640 10.1038/msb.2012.61 23340846 PMC3564260 · DOI
- [27]Hsi-Yang Fritz M., et al, Efficient storage of high throughput DNA sequencing data using reference-based compression. Genome research, 2011 21(5): 734– 40. 10.1101/gr.114819.110 21245279 PMC3083090 · DOI
- [28]Loh P.-R., Baym M., and Berger B., Compressive genomics. Nature Biotechnology, 2012 30: 627– 630. 10.1038/nbt.2241 22781691 · DOI
- [29]Koren S. and Phillippy A.M., One chromosome, one contig: complete microbial genomes from long-read sequencing and assembly. Current opinion in microbiology, 2015 23C: 110– 120. 25461581
- [30]Christley S., et al, Human genomes as email attachments. Bioinformatics, 2009 25(2): 274– 5. 10.1093/bioinformatics/btn582 18996942 · DOI
- [31]Patro R., Mount S.M., and Kingsford C., Sailfish enables alignment-free isoform quantification from RNA-seq reads using lightweight algorithms. Nature biotechnology, 2014 32(5): 462– 4. 10.1038/nbt.2862 24752080 PMC4077321 · DOI · Paper page
- [32]Golden A., Djorgovski S., and Greally J., Astrogenomics: big data, old problems, old solutions? Genome Biology, 2013 8: 129. PMC4053852 23953643
- [34]Smith, B., Data Transport for the Square Kilometre Array, in UbuntuNet Alliance Annual Conference. 2012. 15–22.
- [36]Sboner A., et al, The real cost of sequencing: higher than you think! Genome Biology, 2011 12: 125 10.1186/gb-2011-12-8-125 21867570 PMC3245608 · DOI
- [37]Marx, V., Drilling into big cancer-genome data. Nature Methods, 2013. 10: p. 293–297. 23538863
- [39]Erlich Y. and Narayanan A., Routes for breaching and protecting genetic privacy. Nature reviews. Genetics, 2014 15(6): 409– 21. 10.1038/nrg3723 24805122 PMC4151119 · DOI
- [41]Kumar S., Morstatter F., and Liu H., Twitter Data Analytics 2014: Springer.
- [42]Langmead B., et al, Searching for SNPs with cloud computing. Genome Biol, 2009 10(11): R134 10.1186/gb-2009-10-11-r134 19930550 PMC3091327 · DOI
- [43]Kurtz S., et al, Versatile and open software for comparing large genomes. Genome Biol, 2004 5(2): R12 14759262 PMC395750
- [44]Trelles O., et al, Big data, but are we ready? Nature reviews. Genetics, 2011 12(3): 224. 21301471
- [45]Loh, G.H. 3D-stacked memory architectures for multi-core processors. in ACM SIGARCH. 2008.
- [46]Chan, V.W., et al. Optical flow switching: An end-to-end “UltraFlow” architecture. in 15th International Conference on Transparent Optical Networks (ICTON). 2013. IEEE. 10.1109/ICTON.2013.6602704 · DOI
- [47]Schatz M.C., Langmead B., and Salzberg S.L., Cloud computing and the DNA data race. Nature biotechnology, 2010 28(7): 691– 3. 10.1038/nbt0710-691 20622843 PMC2904649 · DOI
- [49]Schatz, M.C., Computational thinking in the era of big data biology. Genome Biology, 2012. 13(11): 177. PMC3580488 23194371
Cited by
Sorted by citations
Deep Learning: Methods and Applications
Deng, L., & Yu, D. (2014) · Foundations and Trends® in Signal Processing · doi:10.1561/2000000039 · Cited by 3,480 · see citing works
Opportunities and obstacles for deep learning in biology and medicine
Ching, T., Himmelstein, D. S., Beaulieu-Jones, B. K., Kalinin, A. A., Do, B. T., Way, G. P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M. M., et al. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0387 · Cited by 2,310 · see citing works
Big data in healthcare: management, analysis and future prospects
Dash, S., Shakyawar, S. K., Sharma, M., & Kaushik, S. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0217-0 · Cited by 1,747 · see citing works
Ashley, E. A. (2016) · Nature Reviews Genetics · doi:10.1038/nrg.2016.86 · Cited by 1,116 · see citing works
Vellido, A. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04051-w · Cited by 635 · see citing works
Opening the black box: interpretable machine learning for geneticists
Azodi, C. B., Tang, J., & Shiu, S.-H. (2020) · Trends in Genetics · doi:10.1016/j.tig.2020.03.005 · Cited by 473 · see citing works
Mamoshina, P., Ojomoko, L., Yanovich, Y., Ostrovski, A., Botezatu, A., Prikhodko, P., Izumchenko, E., Aliper, A., Romantsov, K., Zhebrak, A., et al. (2018) · Oncotarget · doi:10.18632/oncotarget.22345 · Cited by 471 · see citing works
A complete reference genome improves analysis of human genetic variation
Aganezov, S., Yan, S. M., Soto, D. C., Kirsche, M., Zarate, S., Avdeyev, P., Taylor, D. J., Shafin, K., Shumate, A., Xiao, C., et al. (2022) · Science · doi:10.1126/science.abl3533 · Cited by 441 · see citing works
Self-citationSituating Ecology as a Big-Data Science: Current Advances, Challenges, and Solutions
Farley, S. S., Dawson, A., Goring, S. J., & Williams, J. W. (2018) · BioScience · doi:10.1093/biosci/biy068 · Cited by 331 · see citing works
Cloud computing for genomic data analysis and collaboration
Langmead, B., & Nellore, A. (2018) · Nature Reviews Genetics · doi:10.1038/nrg.2017.113 · Cited by 310 · see citing works
Innovations in Genomics and Big Data Analytics for Personalized Medicine and Health Care: A Review
Hassan, M., Awan, F. M., Naz, A., deAndrés-Galiana, E. J., Alvarez, O., Cernea, A., Fernández-Brillet, L., Fernández-Martínez, J. L., & Kloczkowski, A. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms23094645 · Cited by 317 · see citing works
“The big data paradigm shift is significantly transforming health care and biomedical research [109,136–139], having the potential to better process clinical and biomolecular information that spans the four dimensions of volume, velocity, variety, and veracity, referring to scale, rate, forms, and…”
BackgroundGenomes OnLine database (GOLD) v.7: updates and new features
Mukherjee, S., Stamatis, D., Bertsch, J., Ovchinnikova, G., Katta, H. Y., Mojica, A., Chen, I.-M. A., Kyrpides, N. C., & Reddy, T. (2019) · Nucleic Acids Research · doi:10.1093/nar/gky977 · Cited by 230 · see citing works
Big data in digital healthcare: lessons learnt and recommendations for general practice
Agrawal, R., & Prabakaran, S. (2020) · Heredity · doi:10.1038/s41437-020-0303-2 · Cited by 218 · see citing works
LinearFold: linear-time approximate RNA folding by 5'-to-3' dynamic programming and beam search
Huang, L., Zhang, H., Deng, D., Zhao, K., Liu, K., Hendrix, D. A., & Mathews, D. H. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz375 · Cited by 207 · see citing works
“Since experimentally determining the structure is expensive and time comsuming, and given the overwhelming increase in genomic data (about 1021 base-pairs per year) (Stephens et al., 2015), computational methods have been widely used as an alternative to automatically predict the structure.”
MethodSimultaneous cross-evaluation of heterogeneous <i>E. coli</i> datasets via mechanistic simulation
Macklin, D. N., Ahn-Horst, T. A., Choi, H., Ruggero, N. A., Carrera, J., Mason, J. C., Sun, G., Agmon, E., DeFelice, M. M., Maayan, I., et al. (2020) · Science · doi:10.1126/science.aav3751 · Cited by 151 · see citing works
Realizing the potential of blockchain technologies in genomics
Ozercan, H. I., Ileri, A. M., Ayday, E., & Alkan, C. (2018) · Genome Research · doi:10.1101/gr.207464.116 · Cited by 143 · see citing works
Mangul, S., Mosqueiro, T., Abdill, R. J., Duong, D., Mitchell, K., Sarwal, V., Hill, B., Brito, J., Littman, R. J., Statz, B., et al. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000333 · Cited by 90 · see citing works
Will we run out of data? Limits of LLM scaling based on human-generated data
Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2022) · arXiv · doi:10.48550/arxiv.2211.04325 · Cited by 80 · see citing works
“For illustration, (Stephens et al., 2015) forecasts growth rates of between 2-40 million terabytes of genomics data every year by 2025.”
BackgroundGenome assembly using quantum and quantum-inspired annealing
Boev, A. S., Rakitko, A. S., Usmanov, S. R., Kobzeva, A. N., Popov, I. V., Ilinsky, V. V., Kiktenko, E. O., & Fedorov, A. K. (2021) · Scientific Reports · doi:10.1038/s41598-021-88321-5 · Cited by 63 · see citing works
The Pathway to Patient Data Ownership and Better Health
Mikk, K. A., Sleeper, H. A., & Topol, E. J. (2017) · JAMA · doi:10.1001/jama.2017.12145 · Cited by 56 · see citing works
The Molecular Basis of Human Brain Evolution
Enard, W. (2016) · Current Biology · doi:10.1016/j.cub.2016.09.030 · Cited by 55 · see citing works
Protein sequence‐to‐structure learning: Is this the end(‐to‐end revolution)?
Laine, E., Eismann, S., Elofsson, A., & Grudinin, S. (2021) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26235 · Cited by 44 · see citing works
“Genomics has become pre-eminent in terms of data scale, with an exponential growth [64, 65].”
BackgroundHow to Design AI-Driven Clinical Trials in Nuclear Medicine
Delso, G., Cirillo, D., Kaggie, J. D., Valencia, A., Metser, U., & Veit-Haibach, P. (2021) · Seminars in Nuclear Medicine · doi:10.1053/j.semnuclmed.2020.09.003 · Cited by 36 · see citing works
Krissaane, I., De Niz, C., Gutiérrez-Sacristán, A., Korodi, G., Ede, N., Kumar, R., Lyons, J., Manrai, A., Patel, C., Kohane, I., et al. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa068 · Cited by 28 · see citing works
Formatting biological big data for modern machine learning in drug discovery
Duran‐Frigola, M., Fernández‐Torras, A., Bertoni, M., & Aloy, P. (2019) · WIREs Computational Molecular Science · doi:10.1002/wcms.1408 · Cited by 25 · see citing works
The Roles of Cloud-Based Systems on the Cancer-Related Studies: A Systematic Literature Review
Xu, B., & Zhou, F. (2022) · IEEE Access · doi:10.1109/access.2022.3181147 · Cited by 13 · see citing works
- Authors
- Stephens, Z. D., Lee, S. Y., Faghri, F., Campbell, R. H., Zhai, C., Efron, M. J., Iyer, R., Schatz, M. C., Sinha, S., & Robinson, G. E.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Cold Spring Harbor Laboratory (US)
- Corresponding author
- Schatz, Michael C.
- Sinha, Saurabh
- Robinson, Gene E.
- ORCID
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-07-07
- Volume
- 13
- Issue
- 7
- Pages
- e1002195
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4494865
- OpenAlex
- W2104846587
- Cited by
- 1,340
- References
- 4
- Citations by year
- 2026: 64 · 2025: 83 · 2024: 137 · 2023: 91 · 2022: 117 · 2021: 132 · 2020: 140 · 2019: 156 · 2018: 166 · 2017: 129 · 2016: 95 · 2015: 29 · 2014: 1
- FWCI
- 274.61
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Cold Spring Harbor Laboratory Institutional Repository (Cold Spring Harbor Laboratory) (acceptedVersion)
- Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- National Science Foundation (R01-HG006677)
- National Institutes of Health (5u54gm114838-04, 1U54GM114838, HG006677)
- National Human Genome Research Institute (R01HG006677)
- National Institute of General Medical Sciences (1U54GM114838, grant 1U54GM114838, U54 GM114838)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genetics, Bioinformatics, and Biomedical Research
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- genomics, big data, data storage requirements, data acquisition, data distribution, data analysis, computational challenges, astronomical data, YouTube, Twitter
- MeSH terms
- Astronomy, Statistics as Topic, Information Storage and Retrieval, Genomics, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
