Replication across space and time must be weak in the social and environmental sciences
Open accessThe DOI matches its Crossref record; 63 of the 64 references it cites were found in a registry or an index.
Abstract
Replicability takes on special meaning when researching phenomena that are embedded in space and time, including phenomena distributed on the surface and near surface of the Earth. Two principles, spatial dependence and spatial heterogeneity, are generally characteristic of such phenomena. Various practices have evolved in dealing with spatial heterogeneity, including the use of place-based models. We review the rapidly emerging applications of artificial intelligence to phenomena distributed in space and time and speculate on how the principle of spatial heterogeneity might be addressed. We introduce a concept of weak replicability and discuss possible approaches to its measurement.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This is attributed to the model's pretraining on huge amounts of data to achieve good generalizability and, therefore, transferability compared to models learned from and (over)fitted on a small dataset.”
From Assessment of a new GeoAI foundation model for flood inundation mapping (2023) · cites this paper as [11]
Citations
Cites
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
A Computer Movie Simulating Urban Growth in the Detroit Region
Tobler, W. R. (1970) · Economic Geography · doi:10.2307/143141 · Cited by 9,719
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Random forest in remote sensing: A review of applications and future directions
Belgiu, M., & Drăguţ, L. (2016) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2016.01.011 · Cited by 7,142
Deep learning and process understanding for data-driven Earth system science
Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., & Prabhat (2019) · Nature · doi:10.1038/s41586-019-0912-1 · Cited by 5,673 · cited 1×
“AI has a longstanding history of notable successes in the geographic domain (42 – 45).”
BackgroundGeographically Weighted Regression: The Analysis of Spatially Varying Relationships
Fotheringham, A. S., Brunsdon, C., & Charlton, M. E. (2002) · doi:10.25405/ncl.34562610 · Cited by 5,551
Geostatistics for Natural Resources Evaluation
Goovaerts, P. (1997) · doi:10.1093/oso/9780195115383.001.0001 · Cited by 4,449
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
Dynamic Routing Between Capsules
Sabour, S., Frosst, N., & Hinton, G. E. (2017) · arXiv · doi:10.48550/arxiv.1710.09829 · Cited by 4,167
Geostatistics for Natural Resources Evaluation
Coburn, T. C. (2000) · Technometrics · doi:10.1080/00401706.2000.10485733 · Cited by 4,131
The Book of Why: The New Science of Cause and Effect
Pearl, J., & Mackenzie, D. N. (2018) · Cited by 3,781
A diagnostic approach for going beyond panaceas
Ostrom, E. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0702288104 · Cited by 2,728 · cited 1×
“Ostrom (17) argues that in SES studies it is unlikely that a simple, universally applicable, predictive model exists due to the complexity of places (see also refs.”
BackgroundKates, R. W., Clark, W. C., Corell, R., Hall, J. M., Jaeger, C. C., Lowe, I., McCarthy, J. J., Schellnhuber, H. J., Bolin, B., Dickson, N. M., et al. (2001) · Science · doi:10.1126/science.1059386 · Cited by 2,551
Hierarchical Modeling and Analysis for Spatial Data
Banerjee, S., Carlin, B. P., & Gelfand, A. E. (2014) · Chapman & Hall/CRC eBooks · doi:10.1201/b17115 · Cited by 2,500
Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data
Karpatne, A., Atluri, G., Faghmous, J. H., Steinbach, M., Banerjee, A., Ganguly, A., Shekhar, S., Samatova, N., & Kumar, V. (2017) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2017.2720168 · Cited by 1,752
The Book of Why: The New Science of Cause and Effect
Goldberg, L. R. (2019) · Notices of the American Mathematical Society · doi:10.1090/noti1912 · Cited by 1,655
A Self-Modifying Cellular Automaton Model of Historical Urbanization in the San Francisco Bay Area
Clarke, K. C., Hoppen, S., & Gaydos, L. (1997) · Environment and Planning B: Planning and Design · doi:10.1068/b240247 · Cited by 1,449
The Uncertain Geographic Context Problem
Kwan, M.-P. (2012) · Annals of the Association of American Geographers · doi:10.1080/00045608.2012.687349 · Cited by 1,413 · cited 1×
“This practice of frame-dependent geographic measurement (22) creates contextual uncertainty, causing results to differ from location to location, and this will negatively affect a study ’ s replicability even when all other experimental settings remain the same (23).”
BackgroundMultiscale Geographically Weighted Regression (MGWR)
Fotheringham, A. S., Yang, W., & Kang, W. (2017) · Annals of the American Association of Geographers · doi:10.1080/24694452.2017.1352480 · Cited by 1,352
OpenML: networked science in machine learning
Vanschoren, J., van Rijn, J. N., Bischl, B., & Torgo, L. (2014) · ACM SIGKDD Explorations Newsletter · doi:10.1145/2641190.2641198 · Cited by 1,141
Reproducibility and Replicability in Science
Committee on Reproducibility and Replicability in Science, Board on Behavioral, C., Committee on National Statistics, Education, D. O. B. A. S. S. A., Nuclear and Radiation Studies Board, Studies, D. O. E. A. L., Board on Mathematical Sciences and Analytics, Committee on Applied and Theoretical Statistics, Sciences, D. O. E. A. P., Board on Research Data and Information, et al. (2019) · doi:10.17226/25303 · Cited by 924
Ostrom, E., Janssen, M. A., & Anderies, J. M. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0701886104 · Cited by 897
Robinson, A. H. (1953) · Cited by 843
The Accuracy Of Spatial Databases
(1989) · CRC Press eBooks · doi:10.1201/b12612 · Cited by 770
Foster, E. D., & Deardorff, A. (2017) · Journal of the Medical Library Association · doi:10.5195/jmla.2017.88 · Cited by 715
Janowicz, K., Gao, S., McKenzie, G., Hu, Y., & Bhaduri, B. (2020) · International Journal of Geographical Information Science · doi:10.1080/13658816.2019.1684500 · Cited by 600
Artificial intelligence faces reproducibility crisis
Hutson, M. (2018) · Science · doi:10.1126/science.359.6377.725 · Cited by 564
Georganos, S., Grippa, T., Niang Gadiaga, A., Linard, C., Lennert, M., Vanhuysse, S., Mboga, N., Wolff, E., & Kalogirou, S. (2021) · Geocarto International · doi:10.1080/10106049.2019.1595177 · Cited by 557
Insights on linking forests, trees, and people from the air, on the ground, and in the laboratory
Ostrom, E., & Nagendra, H. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0607962103 · Cited by 551
Exceptionalism in Geography: A Methodological Examination
Schaefer, F. K. (1953) · Annals of the Association of American Geographers · doi:10.1080/00045605309352114 · Cited by 483
McNutt, M. K., Bradford, M., Drazen, J. M., Hanson, B., Howard, B., Jamieson, K. H., Kiermer, V., Marcus, E., Pope, B. K., Schekman, R., et al. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1715374115 · Cited by 392
The Nature of Geography: A Critical Survey of Current Thought in the Light of the Past
Hartshorne, R. (1939) · Annals of the Association of American Geographers · doi:10.2307/2561063 · Cited by 386
GEOGRAPHICAL INFORMATION SYSTEMS: A tool for planning
PULLAR, D., & MCDONALD, G. (1999) · Australian Planner · doi:10.1080/07293682.1999.9665763 · Cited by 377
The unreasonable effectiveness of deep learning in artificial intelligence
Sejnowski, T. J. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907373117 · Cited by 352 · cited 1×
“These models have shown good performance in specialized tasks, such as image classification and speech recognition (54).”
BackgroundGeographic Information Science and Systems
Longley, P. A., & Frank Goodchild, M. (2020) · International Encyclopedia of Human Geography · doi:10.1016/b978-0-08-102295-5.10557-8 · Cited by 314
Self-citationArtificial Intelligence in Geography
Mather, P. M., Openshaw, S., & Openshaw, C. (1998) · The Geographical Journal · doi:10.2307/3060633 · Cited by 304
Tobler's First Law of Geography: A Big Idea for a Small World?
Sui, D. Z. (2004) · Annals of the Association of American Geographers · doi:10.1111/j.1467-8306.2004.09402003.x · Cited by 281
Business Site Selection, Location Analysis and GIS
Church, R. L., & Murray, A. T. (2008) · Wiley eBooks · doi:10.1002/9780470432761 · Cited by 276
Spatial-Filtering-Based Contributions to a Critique of Geographically Weighted Regression (GWR)
Griffith, D. A. (2008) · Environment and Planning A: Economy and Space · doi:10.1068/a38218 · Cited by 264
Agents, Cells, and Cities: New Representational Models for Simulating Multiscale Urban Dynamics
Batty, M. (2005) · Environment and Planning A: Economy and Space · doi:10.1068/a3784 · Cited by 228
Open science, reproducibility, and transparency in ecology
Powers, S. M., & Hampton, S. E. (2019) · Ecological Applications · doi:10.1002/eap.1822 · Cited by 224
Real-time GIS for smart cities
Li, W., Batty, M., & Goodchild, M. F. (2020) · International Journal of Geographical Information Science · doi:10.1080/13658816.2019.1673397 · Cited by 203
Self-citationTelecoupling: A new frontier for global sustainability
Hull, V., & Liu, J. (2018) · Ecology and Society · doi:10.5751/es-10494-230441 · Cited by 199
What is Special About Spatial Data? Alternative Perspectives on Spatial Data Analysis (89-4)
Anselin, L. (1989) · eScholarship (California Digital Library) · Cited by 191
An overview of GeoAI applications in health and healthcare
Kamel Boulos, M. N., Peng, G., & VoPham, T. (2019) · International Journal of Health Geographics · doi:10.1186/s12942-019-0171-2 · Cited by 184
Wheeler, D. C. (2009) · Environment and Planning A: Economy and Space · doi:10.1068/a40256 · Cited by 165
GeoAI: Where machine learning and big data converge in GIScience
Li, W. (2020) · Journal of Spatial Information Science · doi:10.5311/josis.2020.20.658 · Cited by 163
Self-citationA geographically weighted artificial neural network
Hagenauer, J., & Helbich, M. (2021) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2021.1871618 · Cited by 162
Discrete space location-allocation solutions from genetic algorithms
Hosage, C. M., & Goodchild, M. F. (1986) · Annals of Operations Research · doi:10.1007/bf02027381 · Cited by 145
Self-citationWaller, L. A., Zhu, L., Gotway, C. A., Gorman, D. M., & Gruenewald, P. J. (2007) · Stochastic Environmental Research and Risk Assessment · doi:10.1007/s00477-007-0139-9 · Cited by 137
Single and Multiscale Models of Process Spatial Heterogeneity
Wolf, L. J., Oshan, T. M., & Fotheringham, A. S. (2017) · Geographical Analysis · doi:10.1111/gean.12147 · Cited by 113
Reproducibility and replicability: opportunities and challenges for geospatial research
Kedron, P., Li, W., Fotheringham, S., & Goodchild, M. (2021) · International Journal of Geographical Information Science · doi:10.1080/13658816.2020.1802032 · Cited by 108 · cited 1×
“Methods of spatial analysis are today implemented in geographic information systems (GISs) and in many packages of cartographic or statistical software, and issues of replicability and reproducibility in spatial analysis have been addressed in recent papers (14, 15).”
MethodSelf-citationFrame independent spatial analysis
Tobler, W. (1989) · CRC Press eBooks · doi:10.1201/b12612-22 · Cited by 108
The Expansion Method, Mathematical Modeling, and Spatial Econometrics
Casetti, E. (1997) · International Regional Science Review · doi:10.1177/016001769702000102 · Cited by 100
Reproducibility and Replicability in Geographical Analysis
Kedron, P., Frazier, A. E., Trgovac, A. B., Nelson, T., & Fotheringham, A. S. (2021) · Geographical Analysis · doi:10.1111/gean.12221 · Cited by 94
Fletcher, L., & Carruthers, P. (2012) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2011.0413 · Cited by 83
Agent-based modeling of environment-migration linkages: a review
Thober, J., Schwarz, N., & Hermans, K. (2018) · Ecology and Society · doi:10.5751/es-10200-230241 · Cited by 79
Oshan, T. M., & Fotheringham, A. S. (2017) · Geographical Analysis · doi:10.1111/gean.12133 · Cited by 73
A better approach for dealing with reproducibility and replicability in science
Nichols, J. D., Oli, M. K., Kendall, W. L., & Boomer, G. S. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2100769118 · Cited by 73
Newton, The Newtonians, and theGeographia Generalis Varenii∗
Warntz, W. (1989) · Annals of the Association of American Geographers · doi:10.1111/j.1467-8306.1989.tb00257.x · Cited by 27
NASA image-based geological expert system development project for hyperspectral image analysis
Chiou, W. C. (1985) · Applied Optics · doi:10.1364/ao.24.002085 · Cited by 27
Local Analysis of Spatial Relationships: A Comparison of GWR and the Expansion Method
Páez, A. (2005) · Lecture notes in computer science · doi:10.1007/11424857_18 · Cited by 26
The Quality of Geospatial Context
Goodchild, M. F. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04559-2_2 · Cited by 19
Self-citation
Research works printed but not linked: 3
- R. L. Church, “Location modelling and GIS” in Geographical Information Systems, P. A. Longley, M. F. Goodchild, D. J. Maguire, D. W. Rhind, Eds. (Wiley, New York, 1999), pp. 293–303.
- B. Varenius, Geographia Generalis (Elsevier, Amsterdam, 1650).
- W. Bunge, Theoretical Geography. Second Edition. Lund Studies in Geography Series C: General and Mathematical Geography, No. 1. (Gleerup, Lund, Sweden, 1966).
Cited by
Li, W., & Hsu, C.-Y. (2022) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi11070385 · Cited by 147
Self-citationOn the Opportunities and Challenges of Foundation Models for GeoAI (Vision Paper)
Mai, G., Huang, W., Sun, J., Song, S., Mishra, D. R., Liu, N., Gao, S., Liu, T., Cong, G., Hu, Y., et al. (2024) · ACM Transactions on Spatial Algorithms and Systems · doi:10.1145/3653070 · Cited by 444
“Similar to geographic bias, FMs also suffer from temporal bias, which also can be attributed to four causes: temporal representation bias, temporal aggregation bias, algorithm bias, and evaluation bias.”
BackgroundInfluentialUrban Visual Intelligence: Studying Cities with Artificial Intelligence and Street-Level Imagery
Zhang, F., Miranda, A. S., Duarte, F., Vale, L. J., Hack, G. A., Chen, M., Liu, Y., Batty, M., & Ratti, C. (2024) · Annals of the American Association of Geographers · doi:10.1080/24694452.2024.2313515 · Cited by 193
“Such spatial variability often limits the ability of models to make consistent predictions across different locations (Goodchild et al. 2020; Goodchild and Li 2021; Kedron et al. 2021; W. Li, Hsu, and Hu 2021).”
Towards the next generation of Geospatial Artificial Intelligence
Mai, G., Xie, Y., Jia, X., Lao, N., Rao, J., Zhu, Q., Liu, Z., Chiang, Y., & Jiao, J. (2025) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2025.104368 · Cited by 219
Hsu, C.-Y., & Li, W. (2023) · International Journal of Geographical Information Science · doi:10.1080/13658816.2023.2191256 · Cited by 75
“The lack of model explainability may further hinder the replicability of research using GeoAI (Goodchild and Li 2021).”
MethodInfluentialSelf-citation
Metadata
- Authors
- Goodchild, M. F., & Li, W.
- Affiliations
- University of California, Santa Barbara (US)
- Arizona State University (US)
- Corresponding author
- Goodchild, Michael F.
- Li, Wenwen
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2021-08-20
- Volume
- 118
- Issue
- 35
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC8536326
- OpenAlex
- W3194059001
- Semantic Scholar
- 8d95d9e41dc13ce437bbeb80c3e06f19fdb71e5e
- Cited by
- 157
- References
- 64
- Citations by year
- 2026: 36 · 2025: 38 · 2024: 32 · 2023: 19 · 2022: 29 · 2021: 3
- FWCI
- 15.84
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
- Life Sciences › Agricultural and Biological Sciences › Soil Science › Soil Geostatistics and Mapping
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- spatial heterogeneity, spatial dependence, artificial intelligence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.