Comparing Random Forest with Logistic Regression for Predicting Class-Imbalanced Civil War Onset Data: A Comment
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In Wang, the difference between the AUC of complex ML models and LR models drops from 0.14 to 0.01 when errors are corrected.
“where the difference between the area under the curve (AUC) of the complex ML models and LR models drops from 0.14 to 0.01.”
From Leakage and the reproducibility crisis in machine-learning-based science (2023) · cites this paper as [77]
Wang’s Introduction gives AUCs of 0.91 for random forest and 0.77 for one logistic-regression model, a difference of 0.14. But after discussing the correction, §4 (“Better Predictions”; page unavailable) says the improvement from logistic models (0.82) to random forest (0.91) remains dramatic—a difference of 0.09, not 0.01. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredWhen the identified errors are corrected, complex ML models perform no better than baseline LR models for civil war prediction in each case, including Wang where the AUC difference drops from 0.14 to 0.01.
“When the errors are corrected, complex ML models perform no better than baseline LR models in each case except Wang,77 where the difference between the area under the curve (AUC) of the complex ML models and LR models drops from 0.14 to 0.01.”
From Leakage and the reproducibility crisis in machine-learning-based science (2023) · cites this paper as [77]
The citation to Wang is attached to the claimed AUC difference of 0.01 after correction. Wang instead says the improvement remains dramatic: Section 4, “Better Predictions” (page unavailable), reports AUCs of 0.91 for random forest and 0.82 for logistic models, a difference of 0.09. Section 1 reports a 0.14 difference only against one of three logistic models. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredOver the last few years, the field of civil war prediction has switched to predictive modeling using complex ML models such as Random Forests and Adaboost instead of LR.
“Over the last few years, this field has switched to predictive modeling using complex ML models such as Random Forests and Adaboost instead of LR (see Figure 2), with several papers claiming near-perfect performance of these models for civil war prediction.75,76,77,78”
From Leakage and the reproducibility crisis in machine-learning-based science (2023) · cites this paper as [77]
The Introduction describes a prominent civil-war prediction study comparing random forests with logistic regression, and §4 reports Wang’s use of AdaBoosted trees. These passages support the use of those methods, but not the broader claim that the field had switched from logistic regression to them. Section 3 also reports—and challenges—a claim that random forests predicted nearly every civil-war onset. (LLM, conf=0.84)
Experimental — accuracy not yet measuredSeveral papers claim near-perfect performance of complex ML models for civil war prediction.
“Over the last few years, this field has switched to predictive modeling using complex ML models such as Random Forests and Adaboost instead of LR (see Figure 2), with several papers claiming near-perfect performance of these models for civil war prediction.75,76,77,78”
From Leakage and the reproducibility crisis in machine-learning-based science (2023) · cites this paper as [77]
In §3, Wang reports that Muchlinski et al. claimed random forests predicted nearly every civil-war onset, but shows that cross-validation reveals substantially more missed conflicts. In §4, Wang reports AUCs of 0.94 for boosted trees. The paper supports the existence of high-performance claims, but does not establish that several papers claimed near-perfect performance. (LLM, conf=0.83)
Experimental — accuracy not yet measuredColaresi and Mahmood and Wang incorrectly reuse an imputed dataset.
“Colaresi and Mahmood76 and Wang77 incorrectly reuse an imputed dataset”
From Leakage and the reproducibility crisis in machine-learning-based science (2023) · cites this paper as [77]
The Introduction and §2, “Spot the Error” (page unavailable), say Wang identifies an evaluation error in Muchlinski et al.; §3 describes Wang’s cross-validated correction. Searches for imputation, an imputed dataset, and Colaresi and Mahmood found no passage saying that Wang incorrectly reused an imputed dataset. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162 · see citing works
Ethnicity, Insurgency, and Civil War
FEARON, J. D., & LAITIN, D. D. (2003) · American Political Science Review · doi:10.1017/s0003055403000534 · Cited by 7,366 · see citing works
Greed and grievance in civil war
Collier, P. (2004) · Oxford Economic Papers · doi:10.1093/oep/gpf064 · Cited by 6,107 · see citing works
The Elements of Statistical Learning
Hastie, T. J., Tibshirani, R. J., & Friedman, J. H. (2009) · Springer series in statistics · doi:10.1007/b94608 · Cited by 4,111 · see citing works
Sensitivity Analysis of Empirical Results on Civil War Onset
Hegre, H., & Sambanis, N. (2006) · Journal of Conflict Resolution · doi:10.1177/0022002706289303 · Cited by 1,286 · see citing works
Muchlinski, D., Siroky, D., He, J., & Kocher, M. (2016) · Political Analysis · doi:10.1093/pan/mpv024 · Cited by 234 · see citing works
Predicting armed conflict: Time to adjust our expectations?
Cederman, L.-E., & Weidmann, N. B. (2017) · Science · doi:10.1126/science.aal4483 · Cited by 156 · see citing works
The Separation Plot: A New Visual Method for Evaluating the Fit of Binary Models
Greenhill, B., Ward, M., & Sacks, A. (2011) · American Journal of Political Science · doi:10.1111/j.1540-5907.2011.00525.x · Cited by 145 · see citing works
What Can We Learn from Predictive Modeling?
Cranmer, S., & Desmarais, B. A. (2017) · Political Analysis · doi:10.1017/pan.2017.3 · Cited by 102 · see citing works
Wang, Y. (2018) · Harvard Dataverse · doi:10.7910/dvn/uiuygy · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Leakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387 · see citing works
- Authors
- Wang, Y.
- Affiliations
- University of Rochester (US)
- Corresponding author
- Wang, Yu
- ORCID
- Wang, Yu
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2018-12-31
- Volume
- 27
- Issue
- 1
- Pages
- 107-110
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W2906730591
- Cited by
- 12
- References
- 10
- Citations by year
- 2025: 3 · 2024: 2 · 2023: 3 · 2022: 2 · 2021: 1 · 2018: 1
- FWCI
- 1.14
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Political Analysis (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Political Conflict and Governance
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Social Sciences › Political Science and International Relations › Electoral Systems and Political Participation
- Keywords
- random forest, logistic regression, class imbalance, civil war onset, prediction model, model comparison, political violence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
