A Five-Star Guide for Achieving Replicability and Reproducibility When Working with GIS Software and Algorithms
The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
The availability and use of geographic information technologies and data for describing the patterns and processes operating on or near the Earth’s surface have grown substantially during the past fifty years. The number of geographic information systems software packages and algorithms has also grown quickly during this period, fueled by rapid advances in computing and the explosive growth in the availability of digital data describing specific phenomena. Geographic information scientists therefore increasingly find themselves choosing between multiple software suites and algorithms to execute specific analysis, modeling, and visualization tasks in environmental applications today. This is a major challenge because it is often difficult to assess the efficacy of the candidate software platforms and algorithms when used in specific applications and study areas, which often generate different results. The subtleties and issues that characterize the field of geomorphometry are used here to document the need for (1) theoretically based software and algorithms; (2) new methods for the collection of provenance information about the data and code along with application context knowledge; and (3) new protocols for distributing this information and knowledge along with the data and code. This article discusses the progress and enduring challenges connected with these outcomes.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This is attributed to the model's pretraining on huge amounts of data to achieve good generalizability and, therefore, transferability compared to models learned from and (over)fitted on a small dataset.”
From Assessment of a new GeoAI foundation model for flood inundation mapping (2023) · cites this paper as [36]
Citations
Cites
The Fourth Paradigm – Data-Intensive Scientific Discovery
Hey, T. (2012) · Communications in Computer and Information Science · doi:10.1007/978-3-642-33299-9_1 · Cited by 1,585
Terrain Analysis: Principles and Applications
Meyer, T. H. (2001) · Ecological Engineering · doi:10.1016/s0925-8574(00)00178-6 · Cited by 1,462
The Fourth Paradigm: Data-Intensive Scientific Discovery
Buchan, I. E., Winn, J. M., & Bishop, C. (2009) · Cited by 1,439
Workflows and e-Science: An overview of workflow system features and capabilities
Deelman, E., Gannon, D., Shields, M., & Taylor, I. (2009) · Future Generation Computer Systems · doi:10.1016/j.future.2008.06.012 · Cited by 831 · cited 3×
“…if not an expected, component of geospatial research, rising to the level of a “ third branch ” of science along with theory and experimentation (Deelman et al. 2009), or part of a “ fourth paradigm ” of scientific discovery beyond the existing paradigms of empiricism, laboratory analysis,…”
BackgroundInfluentialAnnals of the American association of geographers
Paterson, G. (2016) · interactions · Cited by 762
Atkins, D. E. (2003) · UA Campus Repository (The University of Arizona) · Cited by 658
Water Resources Research in 2013
Montanari, A. (2014) · Water Resources Research · doi:10.1002/2014wr015648 · Cited by 534
myExperiment: a repository and social network for the sharing of bioinformatics workflows
Goble, C. A., Bhagat, J., Aleksejevs, S., Cruickshank, D., Michaelides, D., Newman, D., Borkum, M., Bechhofer, S., Roos, M., Li, P., et al. (2010) · Nucleic Acids Research · doi:10.1093/nar/gkq429 · Cited by 299 · cited 1×
“For example, myexperiment (https:// www.myexperiment.org/) is a repository of nearly 4,000 workflows from a variety of workflow management systems (Goble et al. 2010).”
BackgroundA CyberGIS Framework for the Synthesis of Cyberinfrastructure, GIS, and Spatial Analysis
Wang, S. (2010) · Annals of the Association of American Geographers · doi:10.1080/00045601003791243 · Cited by 296 · cited 1×
“In the geo-spatial domain, researchers have been investigating new ways to capture, cache, and execute data processing workflows to improve reproducibility (Wang 2010; Li et al. 2019).”
Biodiversity data should be published, cited, and peer reviewed
Costello, M. J., Michener, W. K., Gahegan, M., Zhang, Z.-Q., & Bourne, P. E. (2013) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2013.05.002 · Cited by 271 · cited 1×
“Current metadata and provenance have been limited to the modeling of data, which is only one element in a scientific workflow (cf. Costello et al. 2013; Tullis and Kar 2020).”
Method
Research works printed but not linked: 1
- Goodchild, M. F. 2007. Beyond metadata: Towards user-centric description of data quality. Paper presented at the 5th International Symposium on Spatial Data Quality, Enschede, The Netherlands, June 13–15.
Cited by
Global Building Morphology Indicators
Biljecki, F., & Chow, Y. S. (2022) · Computers, Environment and Urban Systems · doi:10.1016/j.compenvurbsys.2022.101809 · Cited by 170
Understanding urban perception with visual data: A systematic review
Ito, K., Kang, Y., Zhang, Y., Zhang, F., & Biljecki, F. (2024) · Cities · doi:10.1016/j.cities.2024.105169 · Cited by 207
“Evaluating the data availability in urban visual perception research is a critical measure of the field ’ s commitment to transparency, reproducibility, and collaborative potential (Wilson et al., 2021).”
Biljecki, F., Chow, Y. S., & Lee, K. (2023) · Building and Environment · doi:10.1016/j.buildenv.2023.110295 · Cited by 198
Big data in Earth science: Emerging practice and promise
Vance, T. C., Huang, T., & Butler, K. A. (2024) · Science · doi:10.1126/science.adh9607 · Cited by 143
“Researchers proposed a five-star guide ( 47 ) to enhance replicability and reproducibility in geospatial research.”
BackgroundSelf-citationKang, Y., Gao, S., & Roth, R. E. (2024) · Cartography and Geographic Information Science · doi:10.1080/15230406.2023.2295943 · Cited by 143
“On the other hand, researchers prefer to endorse open-access datasets and models that empower the replicability and reproducibility of academic studies (Kitchin & Lauriault, 2014; Wilson et al., 2021).”
Self-citation
Metadata
- Authors
- Wilson, J. P., Butler, K., Gao, S., Hu, Y., Li, W., & Wright, D. J.
- Affiliations
- University of Southern California (US)
- Economic and Social Research Institute (IE)
- Department of Geography, University of Wisconsin
- University at Buffalo, State University of New York (US)
- Arizona State University (US)
- Venue
- Annals of the American Association of Geographers
- Publisher
- Informa UK Limited
- Published
- 2020-10-13
- Volume
- 111
- Issue
- 5
- Pages
- 1311-1317
- ISSN
- 2469-4452 (ISSN-L)
- 2469-4460
- OpenAlex
- W3094386498
- Semantic Scholar
- c11f2fb553e489efc9071ff3cfb39091414565e2
- Cited by
- 54
- References
- 28
- Citations by year
- 2026: 8 · 2025: 9 · 2024: 8 · 2023: 13 · 2022: 9 · 2021: 4 · 2020: 3
- FWCI
- 7.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Annals of the American Association of Geographers (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Geography, Planning and Development › Geographic Information Systems Studies
- Keywords
- replicability, reproducibility, terrain analysis, data provenance, code provenance, software selection, environmental modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Five-Star Guide for Achieving Replicability and Reproducibility When Working with GIS Software and Algorithms. https://app.cochat.ai/papers/10.1080/24694452.2020.1806026
- The paper
- Wilson, J. P., Butler, K., Gao, S., Hu, Y., Li, W., & Wright, D. J. (2021). A Five-Star Guide for Achieving Replicability and Reproducibility When Working with GIS Software and Algorithms. Annals of the American Association of Geographers. https://doi.org/10.1080/24694452.2020.1806026
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.