Exploring Limits to Prediction in Complex Social Systems
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
How predictable is success in complex social systems? In spite of a recent profusion of prediction studies that exploit online social and information network data, this question remains unanswered, in part because it has not been adequately specified. In this paper we attempt to clarify the question by presenting a simple stylized model of success that attributes prediction error to one of two generic sources: insufficiency of available data and/or models on the one hand; and inherent unpredictability of complex social systems on the other. We then use this model to motivate an illustrative empirical study of information cascade size prediction on Twitter. Despite an unprecedented volume of information about users, content, and past performance, our best performing models can explain less than half of the variance in cascade sizes. In turn, this result suggests that even with unlimited data predictive performance would be bounded well below deterministic accuracy. Finally, we explore this potential bound theoretically using simulations of a diffusion process on a random scale free network similar to Twitter. We show that although higher predictive power is possible in theory, such performance requires a homogeneous system and perfect ex-ante knowledge of it: even a small degree of uncertainty in estimating product quality or slight variation in quality across products leads to substantially more restrictive bounds on predictability. We conclude that realistic bounds on predictive accuracy are not dissimilar from those we have obtained empirically, and that such bounds for other complex social systems for which data is more difficult to obtain are likely even lower.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
The quiet revolution of numerical weather prediction
Bauer, P., Thorpe, A., & Brunet, G. (2015) · Nature · doi:10.1038/nature14956 · Cited by 2,729 · cited 1×
“More recently, impressive gains have been made in predicting short-term weather patterns [5], demonstrating that under some conditions and with appropriate effort useful predictions can be obtained even for extremely complex and stochastic systems.”
BackgroundThe Parable of Google Flu: Traps in Big Data Analysis
Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014) · Science · doi:10.1126/science.1248506 · Cited by 2,523
Predicting the Present with Google Trends
CHOI, H., & VARIAN, H. (2012) · Economic Record · doi:10.1111/j.1475-4932.2012.00809.x · Cited by 2,354
Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market
Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006) · Science · doi:10.1126/science.1121066 · Cited by 2,231 · cited 4×
“…evaluation criteria have obscured a question of fundamental importance: To the extent that predictions are less accurate than desired, is it simply that the existing combination of methods and data is insufficient; or is it that the phenomenon itself is to some extent inherently unpredictable [46]?”
BackgroundInfluentialSelf-citationPredicting the future with social media
Asur, S., & Huberman, B. A. (2010) · 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology · doi:10.1109/wi-iat.2010.63 · Cited by 2,081
Everyone's an influencer: quantifying influence on twitter
Bakshy, E., Hofman, J. M., Mason, W. A., & Watts, D. J. (2011) · Proceedings of the fourth ACM international conference on Web search and data mining · doi:10.1145/1935826.1935845 · Cited by 1,689
Self-citationExpert political judgment: how good is it? how can we know?
(2006) · Choice Reviews Online · doi:10.5860/choice.43-3634 · Cited by 1,260 · cited 2×
“For example, it is well understood in theory that predictive accuracy cannot be reliably estimated from isolated predictions, especially when selected ex-post [55].”
BackgroundRomero, D. M., Meeder, B., & Kleinberg, J. (2011) · Proceedings of the 20th international conference on World wide web · doi:10.1145/1963405.1963503 · Cited by 1,082
Predicting consumer behavior with web search
Goel, S., Hofman, J. M., Lahaie, S., Pennock, D. M., & Watts, D. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1005962107 · Cited by 636 · cited 3×
“Although some of the more dramatic claims arising from this early work have subsequently been criticized [23, 34] the general idea of correlating online activity to offline outcomes continues to attract interest [11].”
BackgroundSelf-citation
Cited by
Prediction and explanation in social systems
Hofman, J. M., Sharma, A., & Watts, D. J. (2017) · Science · doi:10.1126/science.aal3856 · Cited by 429
Self-citationIntegrating explanation and prediction in computational social science
Hofman, J. M., Watts, D. J., Athey, S., Garip, F., Griffiths, T. L., Kleinberg, J., Margetts, H., Mullainathan, S., Salganik, M. J., Vazire, S., et al. (2021) · Nature · doi:10.1038/s41586-021-03659-0 · Cited by 371
“Alternatively, such a search may reveal the presence of a fundamental limit to predictive accuracy that results from the presence of system complexity or intrinsic randomness 52 , in which case the conclusion may be that we can explain less than we would like, even in principle 53 .”
BackgroundSelf-citationMeasuring the predictability of life outcomes with a scientific mass collaboration
Salganik, M. J., Lundberg, I., Kindel, A. T., Ahearn, C. E., Al-Ghoneim, K., Almaatouq, A., Altschul, D. M., Brand, J. E., Carnegie, N. B., Compton, R. J., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915006117 · Cited by 322
Self-citationA Survey of Information Cascade Analysis: Models, Predictions, and Recent Advances
Zhou, F., Xu, X., Trajcevski, G., & Zhang, K. (2022) · ACM Computing Surveys · doi:10.1145/3433000 · Cited by 259
Combining interventions to reduce the spread of viral misinformation
Bak-Coleman, J. B., Kennedy, I., Wack, M., Beers, A., Schafer, J. S., Spiro, E. S., Starbird, K., & West, J. D. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01388-6 · Cited by 145
Metadata
- Authors
- Martin, T., Hofman, J. M., Sharma, A., Anderson, A., & Watts, D. J.
- Venue
- Proceedings of the 25th International Conference on World Wide Web
- Published
- 2016-04-11
- Pages
- 683-694
- OpenAlex
- W2265595858
- Cited by
- 136
- References
- 12
- FWCI
- 25.45
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Exploring Limits to Prediction in Complex Social Systems. https://app.cochat.ai/papers/10.1145/2872427.2883001
- The paper
- Martin, T., Hofman, J. M., Sharma, A., Anderson, A., & Watts, D. J. (2016). Exploring Limits to Prediction in Complex Social Systems. Proceedings of the 25th International Conference on World Wide Web. https://doi.org/10.1145/2872427.2883001
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.