Shedding Light on the Dark Data in the Long Tail of Science
Open accessAbstract
One of the primary outputs of the scientific enterprise is data, but many institutions such as libraries that are charged with preserving and disseminating scholarly output have largely ignored this form of documentation of scholarly activity. This paper focuses on a particularly troublesome class of data, termed dark data . “Dark data” is not carefully indexed and stored so it becomes nearly invisible to scientists and other potential users and therefore is more likely to remain underutilized and eventually lost. The article discusses how the concepts from long-tail economics can be used to understand potential solutions for better curation of this data. The paper describes why this data is critical to scientific progress, some of the properties of this data, as well as some social and technical barriers to proper management of this class of data. Many potentially useful institutional, social, and technical solutions are under development and are introduced in the last sections of the paper, but these solutions are largely unproven and require additional research and development.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Much of publicly funded research remains relatively hidden and at risk for loss unless submitted to repositories, disproportionately affecting the "long tail" of science and technology.
“Consequently, much of publicly funded research remains relatively hidden and at risk for loss unless submitted to repositories, disproportionately affecting the “long tail” of science and technology (P. Bryan Heidorn, 2008).”
From Sysrev: A FAIR Platform for Data Curation and Systematic Evidence Review (2021) · cites this paper as P. Bryan Heidorn, 2008
The Background and Abstract describe data from small projects in the long tail as hard to find and at risk of loss. “The Properties of Dark Data in the Tail” says such data often do not reach repositories and are less likely to be maintained when they do not. The paper addresses research data, however, rather than establishing that much of publicly funded research as a whole remains hidden; it also notes that public funders often require data-sharing plans. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 14 · Cited by 470
Research works printed but not linked: 4
- Big Data. (2008). Nature, 455(7209), pp. 1-136.
- Goetz, T. (2007). Freeing the dark data of failed scientific experiment. Wired Magazine, 15(10). Retrieved October 31, 2008, from http://www.wired.com/science/discoveries/magazine/ 15-10/st_essay Google Data API. (n.d.). Retrieved October 31, 2008, from http://code.google.com/apis/ gdata/overview.html International Data curation Education Action (IDEA) Working Group. (n.d.). Retrieved October 31, 2008, from http://www.dcc.ac.uk/events/idea-2008-edinburgh/ International Digital Curation Conference. (n.d.). Retrieved October 31, 2008, from http:// www.dcc.ac.uk/events/dcc-2008/ INTEROP. (n.d.). Retrieved October 31, 2008, from http://www.nsf.gov/publications/pub _summ.jsp?ods_key=nsf07565
- Primack, R., Miller-Rushing, A., Primack, D., & Mukunda, S. (2007). Using photographs to show the effects of climate change on flowing time. Arnoldia, 65(1), pp. 2-9.
- Treloar, A. (2007). DART: Building the new collaborative e-research infrastructure. Proceedings of Educause Australasia 2007, Melbourne, April. Retrieved October 31, 2008, from http:// www.caudit.edu.au/educauseaustralasia07/authors_papers/Treloar-183.pdf
- Authors
- Heidorn, P. B.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Corresponding author
- Heidorn, P. Bryan
- ORCID
- Heidorn, P. Bryan
- Venue
- Library trends
- Publisher
- Project MUSE
- Published
- 2008-09
- Volume
- 57
- Issue
- 2
- Pages
- 280-299
- ISSN
- 0024-2594 (ISSN-L)
- 1559-0682
- OpenAlex
- W2001306005
- Cited by
- 470
- References
- 14
- Citations by year
- 2026: 10 · 2025: 17 · 2024: 21 · 2023: 24 · 2022: 31 · 2021: 51 · 2020: 32 · 2019: 32 · 2018: 35 · 2017: 35 · 2016: 36 · 2015: 46 · 2014: 30 · 2013: 23 · 2012: 17
- FWCI
- 30.00
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- other-oa
- Locations
- Library trends (publishedVersion)
- Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) (submittedVersion, other-oa)
- Project Muse (Johns Hopkins University) (submittedVersion)
- Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Business, Management and Accounting › Management Information Systems › Optics and Image Analysis
- Social Sciences › Sociology and Political Science › Data Analysis and Archiving
- Keywords
- dark data, data curation, data preservation, data indexing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
