The DOI matches its Crossref record; 77 of the 77 references it cites were found in a registry or an index.
Evidence that psychopathology symptom networks have limited replicability.
Open accessAbstract
Network analysis is quickly gaining popularity in psychopathology research as a method that aims to reveal causal relationships among individual symptoms. To date, 4 main types of psychopathology networks have been proposed: (a) association networks, (b) regularized concentration networks, (c) relative importance networks, and (d) directed acyclic graphs. The authors examined the replicability of these analyses based on symptoms of major depression and generalized anxiety between and within 2 highly similar epidemiological samples (i.e., the National Comorbidity Survey-Replication [n = 9282] and the National Survey of Mental Health and Wellbeing [n = 8841]). Although association networks were stable, the 3 other types of network analysis (i.e., the conditional independence networks) had poor replicability between and within methods and samples. The detailed aspects of the models-such as the estimation of specific edges and the centrality of individual nodes-were particularly unstable. For example, 44% of the symptoms were estimated as the "most influential" on at least 1 centrality index across the 6 conditional independence networks in the full samples, and only 13-21% of the edges were consistently estimated across these networks. One of the likely reasons for the instability of the networks is the predominance of measurement error in the assessment of individual symptoms. The authors discuss the implications of these findings for the growing field of psychopathology network research, and conclude that novel results originating from psychopathology networks should be held to higher standards of evidence before they are ready for dissemination or implementation in the field. (PsycINFO Database Record
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Sparse inverse covariance estimation with the graphical lasso
Friedman, J., Hastie, T., & Tibshirani, R. (2008) · Biostatistics · doi:10.1093/biostatistics/kxm045 · Cited by 6,355
Graph drawing by force‐directed placement
Fruchterman, T. M. J., & Reingold, E. M. (1991) · Software: Practice and Experience · doi:10.1002/spe.4380211102 · Cited by 6,352
Estimating psychological networks and their accuracy: A tutorial paper
Epskamp, S., Borsboom, D., & Fried, E. I. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0862-1 · Cited by 5,715
qgraph: Network Visualizations of Relationships in Psychometric Data
Epskamp, S., Cramer, A. O. J., Waldorp, L. J., Schmittmann, V. D., & Borsboom, D. (2012) · Journal of Statistical Software · doi:10.18637/jss.v048.i04 · Cited by 4,856
Kessler, R. C., & Üstün, T. B. (2004) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.168 · Cited by 4,771 · cited 1×
“Both surveys were based on the World Mental Health Survey Initiative version of the World Health Organization’s Composite International Diagnostic Interview (WMH-CIDI; Kessler & Ustun, 2004).”
MethodNetwork analysis: An integrative approach to the structure of psychopathology
Borsboom, D., & Cramer, A. O. J. (2013) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-050212-185608 · Cited by 4,359
Borgatti, S. P. (2005) · Social Networks · doi:10.1016/j.socnet.2004.11.008 · Cited by 3,191
Relative Importance for Linear Regression in R : The Package relaimpo
Grömping, U. (2006) · Journal of Statistical Software · doi:10.18637/jss.v017.i01 · Cited by 2,615
Molenaar, P. C. M. (2004) · Measurement: Interdisciplinary Research & Perspective · doi:10.1207/s15366359mea0204_1 · Cited by 1,943
Identifying highly influential nodes in the complicated grief network.
Robinaugh, D. J., Millner, A. J., & McNally, R. J. (2016) · Journal of Abnormal Psychology · doi:10.1037/abn0000181 · Cited by 1,787
Borgatti, S. P. (2005) · Cited by 1,734
Learning Bayesian Networks with the bnlearnR Package
Scutari, M. (2010) · Journal of Statistical Software · doi:10.18637/jss.v035.i03 · Cited by 1,733
Comorbidity: a network perspective
Cramer, A. O. J., Waldorp, L. J., van der Maas, H. L. J., & Borsboom, D. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x09991567 · Cited by 1,575 · cited 12×
“…not detract from the attractive idea of analyzing symptom-level relationships that might allow us to carve psychopathology at its finer joints, thereby deepening our understanding the dynamic mechanisms of disorder onset, maintenance, and treatment (Cramer et al., 2010a; Goekoop & Goekoop, 2014).”
BackgroundMethodInfluentialThe theoretical status of latent variables.
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2003) · Psychological Review · doi:10.1037/0033-295x.110.2.203 · Cited by 1,566
Bayesian structural equation modeling: A more flexible representation of substantive theory
Muthén, B., & Asparouhov, T. (2012) · Psychological Methods · doi:10.1037/a0026802 · Cited by 1,562
Statistical significance in psychological research.
Lykken, D. T. (1968) · Psychological Bulletin · doi:10.1037/h0026141 · Cited by 1,350
Can network analysis transform psychopathology?
McNally, R. J. (2016) · Behaviour Research and Therapy · doi:10.1016/j.brat.2016.06.006 · Cited by 1,072
Mental disorders as networks of problems: a review of recent insights
Fried, E. I., van Borkulo, C. D., Cramer, A. O. J., Boschloo, L., Schoevers, R. A., & Borsboom, D. (2017) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-016-1319-z · Cited by 1,040
State of the aRt personality research: A tutorial on network analysis of personality data in R
Costantini, G., Epskamp, S., Borsboom, D., Perugini, M., Mõttus, R., Waldorp, L. J., & Cramer, A. O. J. (2015) · Journal of Research in Personality · doi:10.1016/j.jrp.2014.07.003 · Cited by 982
Depression sum-scores don’t add up: why analyzing specific depression symptoms is essential
Fried, E. I., & Nesse, R. M. (2015) · BMC Medicine · doi:10.1186/s12916-015-0325-4 · Cited by 981 · cited 1×
“One road to this change would be to use multiple items to measure each symptom, and/or to use multiple methods, such as self-report, others’ reports, daily diaries, observation, or physiological measures (cf. Fried & Nesse, 2015).”
MethodHistory and Use of Relative Importance Indices in Organizational Research
Johnson, J. W., & Lebreton, J. M. (2004) · Organizational Research Methods · doi:10.1177/1094428104266510 · Cited by 940
Network analysis of depression and anxiety symptom relationships in a psychiatric sample
Beard, C., Millner, A. J., Forgeard, M. J. C., Fried, E. I., Hsu, K. J., Treadway, M. T., Leonard, C. V., Kertz, S. J., & Björgvinsson, T. (2016) · Psychological Medicine · doi:10.1017/s0033291716002300 · Cited by 929
Association of Symptom Network Structure With the Course of Depression
van Borkulo, C., Boschloo, L., Borsboom, D., Penninx, B. W. J. H., Waldorp, L. J., & Schoevers, R. A. (2015) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2015.2079 · Cited by 846
The US National Comorbidity Survey Replication (NCS‐R): design and field procedures
Kessler, R. C., Berglund, P., Chiu, W. T., Demler, O., Heeringa, S., Hiripi, E., Jin, R., Pennell, B., Walters, E. E., Zaslavsky, A., et al. (2004) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.167 · Cited by 833
2007 National Survey of Mental Health and Wellbeing: Methods and Key Findings
Slade, T., Johnston, A., Oakley Browne, M. A., Andrews, G., & Whiteford, H. (2009) · Australian & New Zealand Journal of Psychiatry · doi:10.1080/00048670902970882 · Cited by 832
A new method for constructing networks from binary data
van Borkulo, C. D., Borsboom, D., Epskamp, S., Blanken, T. F., Boschloo, L., Schoevers, R. A., & Waldorp, L. J. (2014) · Scientific Reports · doi:10.1038/srep05918 · Cited by 757
Fried, E. I., Epskamp, S., Nesse, R. M., Tuerlinckx, F., & Borsboom, D. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2015.09.005 · Cited by 751
Deconstructing the construct: a network perspective on psychological phenomena
Schmittmann, V. D., Cramer, A. O. J., Waldorp, L. J., Epskamp, S., Kievit, R. A., & Borsboom, D. (2013) · New Ideas in Psychology · doi:10.1016/j.newideapsych.2011.02.007 · Cited by 740
The Replication Recipe: What makes for a convincing replication?
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & van 't Veer, A. (2014) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.10.005 · Cited by 705
A Network Approach to Psychopathology: New Insights into Clinical Longitudinal Data
Bringmann, L. F., Vissers, N., Wichers, M., Geschwind, N., Kuppens, P., Peeters, F., Borsboom, D., & Tuerlinckx, F. (2013) · PLoS ONE · doi:10.1371/journal.pone.0060188 · Cited by 680
Mental Disorders as Causal Systems: A Network Approach to Posttraumatic Stress Disorder
McNally, R. J., Robinaugh, D. J., Wu, G. W. Y., Wang, L., Deserno, M. K., & Borsboom, D. (2015) · Clinical Psychological Science · doi:10.1177/2167702614553230 · Cited by 592
The Small World of Psychopathology
Borsboom, D., Cramer, A. O. J., Schmittmann, V. D., Epskamp, S., & Waldorp, L. J. (2011) · PLoS ONE · doi:10.1371/journal.pone.0027407 · Cited by 591
Causal Inference Using Graphical Models with the R Package pcalg
Kalisch, M., Mächler, M., Colombo, D., Maathuis, M. H., & Bühlmann, P. (2012) · Journal of Statistical Software · doi:10.18637/jss.v047.i11 · Cited by 520
Critical Slowing Down as a Personalized Early Warning Signal for Depression
Wichers, M., Groot, P. C., & Psychosystems, E. G. (2016) · Psychotherapy and Psychosomatics · doi:10.1159/000441458 · Cited by 399
Revealing the dynamic network structure of the Beck Depression Inventory-II
Bringmann, L. F., Lemmens, L. H. J. M., Huibers, M. J. H., Borsboom, D., & Tuerlinckx, F. (2015) · Psychological Medicine · doi:10.1017/s0033291714001809 · Cited by 371
Estimating psychopathological networks: Be careful what you wish for
Epskamp, S., Kruis, J., & Marsman, M. (2017) · PLOS ONE · doi:10.1371/journal.pone.0179891 · Cited by 315
From loss to loneliness: The relationship between bereavement and depressive symptoms.
Fried, E. I., Bockting, C., Arjadi, R., Borsboom, D., Amshoff, M., Cramer, A. O. J., Epskamp, S., Tuerlinckx, F., Carr, D., & Stroebe, M. (2015) · Journal of Abnormal Psychology · doi:10.1037/abn0000028 · Cited by 314
Co-morbid obsessive–compulsive disorder and depression: A bayesian network approach
McNally, R. J., Mair, P., Mugno, B. L., & Riemann, B. C. (2017) · Psychological Medicine · doi:10.1017/s0033291716003287 · Cited by 308 · cited 1×
“McNally et al. (2017) describe these sparse networks as the most likely to estimate genuine edges, compared to alternative methods for computing DAGs.”
MethodBridging the Nomothetic and Idiographic Approaches to the Analysis of Clinical Data
Beltz, A. M., Wright, A. G. C., Sprague, B. N., & Molenaar, P. C. M. (2016) · Assessment · doi:10.1177/1073191116648209 · Cited by 293
Self-citationThe Network Structure of Symptoms of the Diagnostic and Statistical Manual of Mental Disorders
Boschloo, L., van Borkulo, C. D., Rhemtulla, M., Keyes, K. M., Borsboom, D., & Schoevers, R. A. (2015) · PLOS ONE · doi:10.1371/journal.pone.0137621 · Cited by 285
Emotion-Network Density in Major Depressive Disorder
Pe, M. L., Kircanski, K., Thompson, R. J., Bringmann, L. F., Tuerlinckx, F., Mestdagh, M., Mata, J., Jaeggi, S. M., Buschkuehl, M., Jonides, J., et al. (2015) · Clinical Psychological Science · doi:10.1177/2167702614540645 · Cited by 261
Winer, E. S., Cervone, D., Bryant, J., McKinney, C., Liu, R. T., & Nadorff, M. R. (2016) · Journal of Clinical Psychology · doi:10.1002/jclp.22298 · Cited by 260 · cited 2×
“For example, by relying on atemporal, unrandomized, nonexperimental data, there is limited causal information in the data to start with (Dawid, 2008; Winer et al., 2016).”
MethodTackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617690042 · Cited by 251
Self-citationNetwork analysis of persistent complex bereavement disorder in conjugally bereaved adults.
Robinaugh, D. J., LeBlanc, N. J., Vuletich, H. A., & McNally, R. J. (2014) · Journal of Abnormal Psychology · doi:10.1037/abn0000002 · Cited by 219 · cited 4×
“When X ¡ Y is stronger than Y ¡ X it is inferred that X directly predicts Y (Hoorelbeke, Marchetti, De Schryver, & Koster, 2016; McNally, 2016; McNally et al., 2015; Robinaugh et al., 2014).”
BackgroundInfluentialNetwork analysis of substance abuse and dependence symptoms
Rhemtulla, M., Fried, E. I., Aggen, S. H., Tuerlinckx, F., Kendler, K. S., & Borsboom, D. (2016) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2016.02.005 · Cited by 218
A prospective study on how symptoms in a network predict the onset of depression
Boschloo, L., van Borkulo, C. D., Borsboom, D., & Schoevers, R. A. (2016) · Psychotherapy and Psychosomatics · doi:10.1159/000442001 · Cited by 213
Reversing Arrows in Mediation Models Does Not Distinguish Plausible Models
Thoemmes, F. (2015) · Basic and Applied Social Psychology · doi:10.1080/01973533.2015.1049351 · Cited by 208 · cited 2×
“…assumptions, but because reversing the direction of an edge results in a model in the same equivalence class (i.e., with the same implied covariance matrix), which is typically indistinguishable on the basis of statistical evidence, including the size of the directed effects (Thoemmes, 2015).”
BackgroundBorsboom, D., Rhemtulla, M., Cramer, A. O. J., van der Maas, H. L. J., Scheffer, M., & Dolan, C. V. (2016) · Psychological Medicine · doi:10.1017/s0033291715001944 · Cited by 183
Answering Ordinal Questions with Ordinal Data Using Ordinal Statistics
Cliff, N. (1996) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3103_4 · Cited by 181
Consensus, Self‐Other Agreement, and Accuracy in Personality Judgment: An Introduction
Funder, D. C., & West, S. G. (1993) · Journal of Personality · doi:10.1111/j.1467-6494.1993.tb00778.x · Cited by 174
Cramer, A. O. J., Waldorp, L. J., van der Maas, H. L. J., & Borsboom, D. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x10000920 · Cited by 168
There’s more than one way to conduct a replication study: Beyond statistical significance.
Anderson, S. F., & Maxwell, S. E. (2016) · Psychological Methods · doi:10.1037/met0000051 · Cited by 167
A trifactor model for integrating ratings across multiple informants.
Bauer, D. J., Howard, A. L., Baldasaro, R. E., Curran, P. J., Hussong, A. M., Chassin, L., & Zucker, R. A. (2013) · Psychological Methods · doi:10.1037/a0032475 · Cited by 139
Hoorelbeke, K., Marchetti, I., Schryver, M. D., & Koster, E. H. W. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.02.001 · Cited by 113
Goekoop, R., & Goekoop, J. G. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112734 · Cited by 110
The network structure of psychopathology in a community sample of preadolescents.
Boschloo, L., Schoevers, R. A., van Borkulo, C. D., Borsboom, D., & Oldehinkel, A. J. (2016) · Journal of Abnormal Psychology · doi:10.1037/abn0000150 · Cited by 105
Wright, A. G. C., Beltz, A. M., Gates, K. M., Molenaar, P. C. M., & Simms, L. J. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01914 · Cited by 91
Self-citationHandley, T. E., Inder, K., Kelly, B. J., Attia, J., Lewin, T. J., Fitzgerald, M. N., & KAY‐LAMBKIN, F. J. (2011) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-011-0436-y · Cited by 81
Schryver, M. D., Vindevogel, S., Rasmussen, A., & Cramer, A. O. J. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01896 · Cited by 75
A network approach to modeling comorbid internalizing and alcohol use disorders.
Anker, J. J., Forbes, M. K., Almquist, Z. W., Menk, J. S., THURAS, P. D., Unruh, A. S., & Kushner, M. G. (2017) · Journal of Abnormal Psychology · doi:10.1037/abn0000257 · Cited by 72
Self-citationTransdiagnostic Networks: Commentary on Nolen-Hoeksema and Watkins (2011)
Borsboom, D., Epskamp, S., Kievit, R., Cramer, A. O. J., & Schmittmann, V. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611425012 · Cited by 69
Latent variable and network models of comorbidity: toward an empirically derived nosology
Eaton, N. R. (2015) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-015-1012-7 · Cited by 65
Theories of Data Analysis: From Magical Thinking Through Classical Statistics
Diaconis, P. W. (2006) · Wiley series in probability and statistics · doi:10.1002/9781118150702.ch1 · Cited by 57
Problems Attract Problems: A Network Perspective on Mental Disorders
Cramer, A. O. J., & Borsboom, D. (2015) · Wiley eBooks · doi:10.1002/9781118900772.etrds0264 · Cited by 57
Causality in Psychiatry: A Hybrid Symptom Network Construct Model
Young, G. C. (2015) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2015.00164 · Cited by 55
Curtiss, J. E., & Klemanski, D. H. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.04.007 · Cited by 55
Wiedermann, W., & von Eye, A. (2015) · International Journal of Behavioral Development · doi:10.1177/0165025415582056 · Cited by 42
Moffitt, T. E., Harrington, H., Caspi, A., Kim-Cohen, J., Goldberg, D., Gregory, A. M., & Poulton, R. (2007) · Archives of General Psychiatry · doi:10.1001/archpsyc.64.6.651 · Cited by 41 · cited 1×
“In the context of the MDE and GAD literature, which highlights the remarkable overlap between the disorders (e.g., Moffitt et al., 2007), this result illustrates the importance of the shared variance between the symptoms for understanding the relationship between disorders.”
BackgroundDirection of Effects in Multiple Linear Regression Models
Wiedermann, W., & von Eye, A. (2015) · Multivariate Behavioral Research · doi:10.1080/00273171.2014.958429 · Cited by 40
Estimating Psychological Networks and their Stability: a Tutorial PaperPreprint
Epskamp, S., Borsboom, D., & Fried, E. I. (2016) · arXiv (Cornell University) · Cited by 38
Dawid, A. P. (2008) · Cited by 27
Krueger, R. F., DeYoung, C. G., & Markon, K. E. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x10000646 · Cited by 21
Self-citationCausality, Conditional Independence, and Graphical Separation in Settable Systems
Chalak, K., & White, H. (2012) · Neural Computation · doi:10.1162/neco_a_00295 · Cited by 20
Questions about networks, measurement, and causation
Markus, K. A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x10000658 · Cited by 11
Markon, K. E., & Jonas, K. G. (2016) · Journal of Abnormal Psychology · doi:10.1037/abn0000206 · Cited by 9
Self-citation
Cited by
A tutorial on regularized partial correlation networks
Epskamp, S., & Fried, E. I. (2018) · Psychological Methods · doi:10.1037/met0000167 · Cited by 2,918
Bridge Centrality: A Network Approach to Understanding Comorbidity
Jones, P. J., Ma, R., & McNally, R. J. (2021) · Multivariate Behavioral Research · doi:10.1080/00273171.2019.1614898 · Cited by 1,741
Network analysis: a brief overview and tutorial
Hevey, D. (2018) · Health Psychology and Behavioral Medicine · doi:10.1080/21642850.2018.1521283 · Cited by 1,058
All for One and One for All: Mental Disorders in One Dimension
Caspi, A., & Moffitt, T. E. (2018) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2018.17121383 · Cited by 918
Robinaugh, D. J., Hoekstra, R. H. A., Toner, E. R., & Borsboom, D. (2020) · Psychological Medicine · doi:10.1017/s0033291719003404 · Cited by 861
Metadata
- Authors
- Forbes, M. K., Wright, A. G. C., Markon, K. E., & Krueger, R. F.
- Venue
- Journal of Abnormal Psychology
- Published
- 2017-10
- Volume
- 126
- Issue
- 7
- Pages
- 969-988
- ISSN
- 0021-843X (ISSN-L)
- 1939-1846
- OpenAlex
- W2767417220
- Cited by
- 354
- References
- 77
- FWCI
- 49.60
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.