Simulation-based model selection for dynamical systems in systems and population biology
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index; 3 of the 9 sources its claims cite were checked for support.
Abstract
Motivation: Computer simulations have become an important tool across the biomedical sciences and beyond. For many important problems several different models or hypotheses exist and choosing which one best describes reality or observed data is not straightforward. We therefore require suitable statistical tools that allow us to choose rationally between different mechanistic models of, e.g. signal transduction or gene regulation networks. This is particularly challenging in systems biology where only a small number of molecular species can be assayed at any given time and all measurements are subject to measurement uncertainty. Results: Here, we develop such a model selection framework based on approximate Bayesian computation and employing sequential Monte Carlo sampling. We show that our approach can be applied across a wide range of biological scenarios, and we illustrate its use on real data describing influenza dynamics and the JAK-STAT signalling pathway. Bayesian model selection strikes a balance between the complexity of the simulation models and their ability to describe observed data. The present approach enables us to employ the whole formal apparatus to any system that can be (efficiently) simulated, even when exact likelihoods are computationally intractable. Contact: ttoni@imperial.ac.uk; m.stumpf@imperial.ac.uk Supplementary information: Supplementary data are available at Bioinformatics online.
Overview
CoChat Summary
TL;DR
The paper develops an approximate Bayesian computation (ABC) model selection framework using sequential Monte Carlo (SMC) for dynamical systems where exact likelihoods are intractable. It demonstrates the method on stochastic reaction kinetics, Gibbs random fields, influenza outbreak data, and JAK-STAT signalling models.
Key findings
- ABC SMC model selection identified the correct model with high confidence in a synthetic stochastic reaction kinetics example.
- For Gibbs random fields, ABC SMC estimated posterior model distributions correctly and yielded approximately 50-fold speed-up on average compared with ABC rejection.
- For influenza outbreaks of the same strain, the posterior probability of the two-parameter shared-parameter model was 0.98 (median over 10 runs), supporting shared epidemiological characteristics.
- For JAK-STAT signalling, Bayes factors indicated strong evidence in favour of model 3 over model 1, positive evidence for model 3 over model 2, and positive evidence for model 2 over model 1, favouring the cycling model without a time delay.
Methods
- The authors introduce ABC SMC model selection on the joint space of model indicators and parameters, propagating particles through intermediate distributions with decreasing tolerances.
- The algorithm requires user-defined priors, distance functions, tolerance schedules, and perturbation kernels; in the examples, uniform priors were used for parameters and models.
- Applications included stochastic chemical reaction models, Gibbs random fields with known posteriors, influenza A/B outbreak datasets, and non-nested ordinary and delay differential equation models of STAT5 signalling.
Limitations
- The paper states that routine application to complex models with hundreds or thousands of parameters will require further numerical developments due to the high computational cost of repeated simulations.
- The paper states that SMC-based ABC methods are highly parallelizable and suggests future work exploit this, along with regression adjustment, automatic tolerance schedules, and more sophisticated perturbation kernels.
- Evident from the text: performance depends on choices of distance function, tolerance schedule, and perturbation kernels, and the examples use uniform priors and relatively small particle numbers (e.g. N=500 or N=1000).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
3 Valid · 0 Not valid · 3 Not checked · 28 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“If ϵ is sufficiently small then this distribution will be a good approximation for the ‘true’ posterior distribution, P(θ|D0).”
§ Simulation-based model selection for dynamical systems in systems and population biology > 1 INTRODUCTION
What the sources say
- Not checked yetcites Beaumont et al., 2002 matched to “Approximate Bayesian computation in population genetics” (2002) source
- Partly supportedcites Marjoram et al., 2003 matched to “Markov chain Monte Carlo without likelihoods” (2003) sourceExperimental — accuracy not yet measured
The Abstract paragraph following algorithms C1–C4 supports the ε-limit half of the claim: as ε → ∞ the accepted draws come from the prior, if ε = 0 they come from the posterior density f(θ|𝒟), and 'the choice of ε therefore reflects a tension between computability and accuracy'. However, the paper reserves the label 'approximate Bayesian computation' for the summary-statistic scheme D1–D4, and there it conditions agreement with the true posterior on S being sufficient, noting that 'in practice it will be hard, if not impossible, to identify a suitable set of sufficient statistics'. Its Results make the omission concrete: with S = V the posterior mean of T was ≈1.75 against the 'true' 0.68, and even at ε = 0 with S = (V, H) (Table 4) the mean was 0.59, with the authors sounding 'a note of caution' that summarizing the data can have unexpected effects on the posterior. So the unqualified assertion that a small ε alone makes the ABC output a good approximation to P(θ|D0) is only partly what this source states. (LLM, conf=0.70)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Both approaches have been applied under the ABC rejection scheme, which is computationally prohibitive for models with even an only moderate number of parameters (Grelaud et al., 2009; Wilkinson, 2007).”
§ Simulation-based model selection for dynamical systems in systems and population biology > 2 ABC FOR MODEL SELECTION
What the sources say
- Partly supportedcites Grelaud et al., 2009 matched to “ABC likelihood-free methods for model choice in Gibbs random fields” (2009) sourceExperimental — accuracy not yet measured
Our copy supports the computational burden of the ABC rejection scheme but not the stated cause: in the 'Exact rejection algorithm' section the authors say the exact (ε=0) version has an 'acceptance probability [that] may be too low to be practical', and in §3.1 (Toy example) 4×10^6 loops yielded as few as 12 acceptances in the worst case; §1.2 adds that the normalising-constant sum 'involves too many terms to be manageable'. However, the paper attributes the difficulty to the dimension of the data/summary statistics and the intractable normalising constant, never to 'even a moderate number of parameters'; it also hedges where the claim states prohibitive as fact, and its overall thrust (§2.1, §4 Discussion) is that ABC is 'naturally tuned' for Gibbs-random-field model choice and implementable on realistic datasets. (LLM, conf=0.70)
- Not checked yetcites Wilkinson, 2007
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The algorithm presented in Toni et al. (2009) is a special case of the above algorithm for discrete uniform KM kernel and uniform prior distribution of the model P(m).”
§ Simulation-based model selection for dynamical systems in systems and population biology > 2 ABC FOR MODEL SELECTION > 2.1 ABC SMC model selection algorithm on the joint space
What the sources say
- Partly supportedcites Toni et al. (2009) matched to “Approximate Bayesian computation scheme for parameter inference and model selection in dynamical systems” (2009) sourceExperimental — accuracy not yet measured
Our copy confirms the structural half of the characterisation: the model-selection scheme treats m ∈ {1,…,M} as an extra discrete parameter and draws m* afresh from the prior π(m) at every population (2.2 Model selection, p. 5, step MS2.1), with uniform perturbation kernels and a discrete uniform model prior in the worked examples (3.1.1, p. 7; Fig. 6 caption 'Population 0 (discrete uniform prior distribution)', 3.3, p. 15). What the source does not support is the subsumption relation itself: Toni et al. nowhere state that their algorithm is a special case of any more general ABC SMC formulation — the only special case they assert is T = 1 reducing ABC SMC to the ABC rejection sampler (2.1, p. 4), and their Appendix B contrasts (rather than nests) their SIS-based scheme with Sisson et al.'s ABC PRC (pp. 21–22). The 'special case of the above algorithm' is therefore the citing authors' own derivation relative to their algorithm, and Toni et al. also present π(m) and K_t generally (uniform or Gaussian), not restricted to the discrete-uniform/uniform-prior setting the claim names. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This was confirmed by applying our model selection algorithm; the inferred posterior marginal model probability of a two-parameter model was negligible (data not shown).”
§ Simulation-based model selection for dynamical systems in systems and population biology > 3 RESULTS > 3.3 Infuenza infection outbreaks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 3b shows the marginal posterior distribution of the model; the posterior probability of model 1 is 0.98 (median over 10 runs), which gives unambiguous support to model 1, meaning that outbreaks of the same strain share the same dynamics.”
§ Simulation-based model selection for dynamical systems in systems and population biology > 3 RESULTS > 3.3 Infuenza infection outbreaks
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
17 more claims
Create a free account to see all 34 claims and to verify the 6 references still open.
- Probably not a substantial claim. 7
- It may state two separate claims as one. 7
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In addition to parameter estimation, ABC approaches can also be used for model ranking and selection34.”
From A framework for parameter estimation and model selection from experimental data in systems biology using approximate Bayesian computation (2014) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Simulation-based model selection for dynamical systems in systems and population biology > 3 RESULTS > 3.3 Infuenza infection outbreaks
“find some evidence that model (1) explains the data better than model (1)”
§ Simulation-based model selection for dynamical systems in systems and population biology > 3 RESULTS > 3.3 Infuenza infection outbreaks
“Such a model can be written as (O'Neill et al., 2000) (2)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Simulation-based model selection for dynamical systems in systems and population biology > 3 RESULTS > 3.3 Infuenza infection outbreaks
“We thus explore a further model with three parameters, qc1, qc2, qh (model 1), where the two outbreaks share the same within-household characteristics (qh), and compare it against and the four-parameter model (model 2).”
§ Simulation-based model selection for dynamical systems in systems and population biology > 3 RESULTS > 3.3 Infuenza infection outbreaks
“(d) Estimation of a posterior marginal distribution. Model 1 is a two-parameter and model 2 a three-parameter model (1).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Simulation-based model selection for dynamical systems in systems and population biology > 3 RESULTS > 3.2 Gibbs random fields
“P(xi+1=xi|xi)=θ1/(1+exp(θ1)) for i=2,…, xn.”
§ Simulation-based model selection for dynamical systems in systems and population biology > 3 RESULTS > 3.2 Gibbs random fields
“Both models, m0 and m1, are defined on a sequence of n binary random variables, x=(x1,…, xn), xi∈{0, 1}”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Simulation-based model selection for dynamical systems in systems and population biology > 3 RESULTS > 3.2 Gibbs random fields
“m0 is a collection of n i.i.d. Bernoulli random variables with probability θ0/(1+exp(θ0));”
§ Simulation-based model selection for dynamical systems in systems and population biology > 3 RESULTS > 3.2 Gibbs random fields
“θ0 ∼ U(−5, 5)”
Citations
Cites
Model selection and multimodel inference: a practical information-theoretic approach.
Guthery, F. S., Burnham, K. P., & Anderson, D. R. (2003) · The Journal of Wildlife Management · doi:10.2307/3802723 · Cited by 41,495 · cited 2×
“The present approach enables us to employ the whole formal apparatus to any system that can be (efficiently) simulated, even when exact likelihoods are computationally intractable.”
BackgroundMethodGelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
Exact stochastic simulation of coupled chemical reactions
Gillespie, D. T. (1977) · The Journal of Physical Chemistry · doi:10.1021/j100540a008 · Cited by 10,136 · cited 2×
“We simulate synthetic datasets of Y measured at 20 time points using Gillespie algorithm (Gillespie, 1977) from model 2 with parameter k2 =30 and initial conditions X0 =40, Y0”
MethodSequential Monte Carlo Methods in Practice
(2001) · Springer eBooks · doi:10.1007/978-1-4757-3437-9 · Cited by 7,366
Novel Proteinaceous Infectious Particles Cause Scrapie
Prusiner, S. B. (1982) · Science · doi:10.1126/science.6801762 · Cited by 5,118 · cited 3×
“We use data from influenza A (H3N2) outbreaks that occurred in 1977–1978 and 1980–1981 in Tecomseh, Michigan (Addy et al., 1991, Supplementary Table 2), and a second dataset of an influenza B infection outbreak in 1975–1976 and influenza A (H1N1) infection outbreak in 1978–1979 in Seattle, Washington (Longini and Koop…”
BackgroundDarnell, J. E. (1997) · Science · doi:10.1126/science.277.5332.1630 · Cited by 3,979
Approximate Bayesian computation in population genetics
Beaumont, M. A., Zhang, W., & Balding, D. J. (2002) · Genetics · doi:10.1093/genetics/162.4.2025 · Cited by 2,932
Computing Bayes Factors by Combining Simulation and Asymptotic Approximations
Diciccio, T. J., Kass, R. E., Raftery, A., & Wasserman, L. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474045 · Cited by 1,943
Bayesian Data Analysis, 2nd edn
Brix, A. (2005) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2004.00347_4.x · Cited by 1,767
Research works printed but not linked: 2
- Jeffreys, 1939, Theory of Probability
- Wilkinson, 2007, Bayesian inference of primate divergence times
Cited by
Approximate Bayesian Computation (ABC) in practice
Csilléry, K., Blum, M. G. B., Gaggiotti, O. E., & François, O. (2010) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2010.04.001 · Cited by 1,302
Approximate Bayesian Computation in Evolution and Ecology
Beaumont, M. A. (2010) · Annual Review of Ecology, Evolution, and Systematics · doi:10.1146/annurev-ecolsys-102209-144621 · Cited by 1,114
Approximate Bayesian computational methods
Marin, J.-M., Pudlo, P., Robert, C. P., & Ryder, R. J. (2012) · Statistics and Computing · doi:10.1007/s11222-011-9288-2 · Cited by 818
“Fearnhead and Prangle (2010) rely on a randomised version of the summary statistics from which they derive a well-calibrated version of ABC, i.e. an algorithm that gives proper predictions of given quantities.”
BackgroundApproximate Bayesian Computation
Sunnåker, M., Busetto, A. G., Numminen, E., Corander, J., Foll, M., & Dessimoz, C. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002803 · Cited by 623
“Again, computational improvements for ABC in the space of models have been proposed, such as constructing a particle filter in the joint space of models and parameters [17].”
MethodStatistical inference for stochastic simulation models - theory and application
Hartig, F., Calabrese, J. M., Reineking, B., Wiegand, T., & Huth, A. (2011) · Ecology Letters · doi:10.1111/j.1461-0248.2011.01640.x · Cited by 384
“We have concentrated our discussion mainly on parameter estimation, but once appropriate likelihood approximations are established, model selection and uncertainty estimation can, in principle, be done in the same way as in other statistical applications (e.g. Beaumont 2010; Toni & Stumpf 2010).”
ResultDi Valentino, E., Said, J. L., Riess, A., Pollo, A., Poulin, V., Gómez-Valent, A., Weltman, A., Palmese, A., Huang, C. D., Bruck, C. V. D., et al. (2025) · Physics of the Dark Universe · doi:10.1016/j.dark.2025.101965 · Cited by 319
Lack of confidence in approximate Bayesian computation model choice
Robert, C. P., Cornuet, J.-M., Marin, J.-M., & Pillai, N. S. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1102900108 · Cited by 317
Quantification of subclonal selection in cancer from bulk sequencing data
Williams, M. J., Werner, B., Heide, T., Curtis, C., Barnes, C. P., Sottoriva, A., & Graham, T. A. (2018) · Nature Genetics · doi:10.1038/s41588-018-0128-6 · Cited by 261
“Return to 1 We used an extension of the simple ABC rejection algorithm, called Approximate Bayesian Computation Sequential Monte-Carlo (ABC SMC)22,52.”
Reverse engineering and identification in systems biology: strategies, perspectives and challenges
Villaverde, A. F., & Banga, J. R. (2014) · Journal of The Royal Society Interface · doi:10.1098/rsif.2013.0505 · Cited by 229
Joly, N., Engl, C., Jovanovic, G., Huvet, M., Toni, T., Sheng, X., Stumpf, M. P. H., & Buck, M. (2010) · FEMS Microbiology Reviews · doi:10.1111/j.1574-6976.2010.00240.x · Cited by 216
Self-citationLiepe, J., Kirk, P., Filippi, S., Toni, T., Barnes, C. P., & Stumpf, M. P. H. (2014) · Nature Protocols · doi:10.1038/nprot.2014.025 · Cited by 217
“In addition to parameter estimation, ABC approaches can also be used for model ranking and selection 34 .”
MethodSelf-citationMutant clones in normal epithelium outcompete and eliminate emerging tumours
Colom, B., Herms, A., Hall, M. W. J., Dentro, S. C., King, C., Sood, R. K., Alcolea, M. P., Piedrafita, G., Fernandez-Antoran, D., Ong, S. H., et al. (2021) · Nature · doi:10.1038/s41586-021-03965-7 · Cited by 192
Maximizing the information content of experiments in systems biology
Liepe, J., Filippi, S., Komorowski, M., & Stumpf, M. P. H. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002888 · Cited by 170
Self-citationAlsing, J., Wandelt, B., & Feeney, S. (2018) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/sty819 · Cited by 159
ABC-SysBio approximate Bayesian computation in Python with GPU support
Liepe, J., Barnes, C., Cule, E., Erguler, K., Kirk, P., Toni, T., & Stumpf, M. P. H. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq278 · Cited by 136
“An algorithm to select between several deterministic or stochastic dynamical models for a given dataset has been implemented; here the model identifiers/labels are treated as an additional parameter, as described in Toni and Stumpf, 2010.”
MethodSelf-citationGene regulatory network inference using fused LASSO on multiple data sets
Omranian, N., Eloundou-Mbebi, J. M. O., Mueller‐Roeber, B., & Nikoloski, Z. (2016) · Scientific Reports · doi:10.1038/srep20533 · Cited by 158
Model selection in systems and synthetic biology
Kirk, P., Thorne, T., & Stumpf, M. P. (2013) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2013.03.012 · Cited by 118
Self-citationTopham, A. T., Taylor, R. E., Yan, D., Nambara, E., Johnston, I. G., & Bassel, G. W. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1704745114 · Cited by 131
“Effective parameter sets for this model were identified harnessing a range of biological observations, and using approximate Bayesian computation with the criteria driving parameter selection was based on the similarity between simulated model behavior and observed biological behaviors (19) (SI Appendix, Supplementary…”
MethodCameron, E., & Pettitt, A. N. (2012) · Monthly Notices of the Royal Astronomical Society · doi:10.1111/j.1365-2966.2012.21371.x · Cited by 131
“…considerations when compiling such summary data for the 16 Application of ABC to the problem of Bayesian model choice (cf. Grelaud et al. 2009; Toni & Stumpf 2010) is far from straightforward as an unfortunate summary statistic selection can lead to disastrously incorrect Bayes factors, even…”
BackgroundLikelihood-free estimation of model evidence
Didelot, X., Everitt, R. G., Johansen, A. M., & Lawson, D. J. (2011) · Bayesian Analysis · doi:10.1214/11-ba602 · Cited by 109
Sinha, J. K. (2014) · CRC Press eBooks · doi:10.1201/b17938-11 · Cited by 98
Topological sensitivity analysis for systems biology
Babtie, A. C., Kirk, P., & Stumpf, M. P. H. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1414026112 · Cited by 97
“t=t1 b_ xnðtÞ− fnðx̂ðtÞ; t; θÞ 2 : [4]”
Self-citationEnvironment-sensitivity functions for gross primary productivity in light use efficiency models
Bao, S., Wutzler, T., Koirala, S., Cuntz, M., Ibrom, A., Besnard, S., Walther, S., Šigut, L., Moreno‐Martínez, Á., Weber, U., et al. (2021) · Agricultural and Forest Meteorology · doi:10.1016/j.agrformet.2021.108708 · Cited by 105
Bayesian design of synthetic biological systems
Barnes, C. P., Silk, D., Sheng, X., & Stumpf, M. P. H. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1017972108 · Cited by 90
“This framework can also be used to perform Bayesian model selection (18) and has been implemented in the software package ABC-SysBio (19).”
MethodSelf-citationA Sequential Monte Carlo Algorithm to Incorporate Model Uncertainty in Bayesian Sequential Design
Drovandi, C., McGree, J. M., & Pettitt, A. N. (2014) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2012.730083 · Cited by 110
“In this way, the posterior probability of a model, m , can be estimated via the weights of particles whose value for the model indicator is m (see Toni and Stumpf (2010) for example).”
MethodBen Abdessalem, A., Dervilis, N., Wagg, D., & Worden, K. (2018) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2017.06.017 · Cited by 83
“During the last decade, the ABC algorithm has been applied in many areas for both levels of inference (parameter and model): genetics [19], biology [20,21] and psychology [22].”
MethodEstrogen‐dependent control and cell‐to‐cell variability of transcriptional bursting
Fritzsch, C., Baumgärtner, S., Kuban, M., Steinshorn, D., Reid, G., & Legewie, S. (2018) · Molecular Systems Biology · doi:10.15252/msb.20177678 · Cited by 82
Inference for Stochastic Chemical Kinetics Using Moment Equations and System Size Expansion
Fröhlich, F., Thomas, P., Kazeroonian, A., Theis, F. J., Grima, R., & Hasenauer, J. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005030 · Cited by 79
“These methods rely on exhaustive stochastic simulations and accept parameter values if the differences between simulation and experimental data is sufficiently small [7, 13, 19].”
MethodSilk, D., Kirk, P. D. W., Barnes, C. P., Toni, T., Rose, A., Moon, S., Dallman, M. J., & Stumpf, M. P. H. (2011) · Nature Communications · doi:10.1038/ncomms1496 · Cited by 76
“Parameter estimation [2], model inference [3] and model selection [4, 5] all address aspects of this problem.”
Self-citationPost-GWAS: where next? More samples, more SNPs or more biology?
Marjoram, P., Zubair, A., & Nuzhdin, S. V. (2014) · Heredity · doi:10.1038/hdy.2013.52 · Cited by 74
An Overview of Bayesian Methods for Neural Spike Train Analysis
Chen, Z. (2013) · Computational Intelligence and Neuroscience · doi:10.1155/2013/251905 · Cited by 68
“In practice, the BF is often computed based on numerical approximation, such as the Laplace-Metropolis Estimator [51] or sequential Monte Carlo meth-ods [52].”
MethodAn information theoretic criterion for empirical validation of simulation models
Lamperti, F. (2017) · Econometrics and Statistics · doi:10.1016/j.ecosta.2017.01.006 · Cited by 95
Considerate approaches to constructing summary statistics for ABC model selection
Barnes, C. P., Filippi, S., Stumpf, M. P. H., & Thorne, T. (2012) · Statistics and Computing · doi:10.1007/s11222-012-9335-7 · Cited by 61
“Here we show how a natural extension of the methodology introduced above can also be employed in order to construct sets of statistics that are suitable for ABC-based model selection, when it is impractical to use the raw data (Toni and Stumpf 2010; Thorne and Stumpf 2012).”
MethodSelf-citationAbdessalem, A. B., Dervilis, N., Wagg, D., & Worden, K. (2017) · Frontiers in Built Environment · doi:10.3389/fbuil.2017.00052 · Cited by 69
“The main objective consists of comparing the simulated data,u, with observed datau*, and accepting simulations if a suitable distance measure between them, ∆(u, u*), is less than a specified threshold defined by the user, ε (for more information check Toni and Stumpf (2010) and Ben Abdessalem et al.”
BackgroundMethodInfluentialAccelerating ABC methods using Gaussian processesPreprint
Wilkinson, R. D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.1436 · Cited by 84
A Statistical Approach Reveals Designs for the Most Robust Stochastic Gene Oscillators
Woods, M. L., Leon, M., Perez‐Carrasco, R., & Barnes, C. P. (2016) · ACS Synthetic Biology · doi:10.1021/acssynbio.5b00179 · Cited by 68
Interpreting scratch assays using pair density dynamics and approximate Bayesian computation
Johnston, S. T., Simpson, M. J., McElwain, D. L. S., Binder, B. J., & Ross, J. V. (2014) · Open Biology · doi:10.1098/rsob.140097 · Cited by 57
Sequential Monte Carlo with Adaptive Weights for Approximate Bayesian Computation
Bonassi, F. V., & West, M. (2015) · Bayesian Analysis · doi:10.1214/14-ba891 · Cited by 65
“Total acceptance rates can be significantly higher than with vanilla ABC scheme (Toni et al., 2009; Toni and Stumpf, 2010).”
BackgroundMethodInfluentialSemi-automatic selection of summary statistics for ABC model choice
Prangle, D., Fearnhead, P., Cox, M. P., Biggs, P. J., & French, N. P. (2013) · Statistical Applications in Genetics and Molecular Biology · doi:10.1515/sagmb-2013-0012 · Cited by 77
“More efficient ABC model choice algorithms have been proposed, mainly based on sequential Monte Carlo (SMC) (e.g. Toni and Stumpf, 2010; Del Moral et al., 2012).”
MethodJagiella, N., Rickert, D., Theis, F. J., & Hasenauer, J. (2017) · Cell Systems · doi:10.1016/j.cels.2016.12.002 · Cited by 70
Bayesian structural inference for hidden processes
Strelioff, C. C., & Crutchfield, J. P. (2014) · Physical Review E · doi:10.1103/physreve.89.042119 · Cited by 49
“Examples include employing information-theoretic measures to infer nonlinear ordinary differential equations (ODEs) [1] and particle filtering to infer ODEs and continuous-time Markov processes [2], as well as a variety of hybrid Bayesian techniques [3, 4].”
Spatial mutation patterns as markers of early colorectal tumor cell mobility
Ryser, M. D., Min, B., Siegmund, K. D., & Shibata, D. K. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1716552115 · Cited by 48
“Bayesian model selection through rejection sampling on the joint model–parameter space (29) was performed to compare the early cell mobility model against alternative modes with selection, delayed cell mobility, and self-seeding, respectively.”
MethodpyABC: Efficient and robust easy-to-use approximate Bayesian computation
Schälte, Y., Klinger, E., Alamoudi, E., & Hasenauer, J. (2022) · The Journal of Open Source Software · doi:10.21105/joss.04304 · Cited by 69
“Its practical performance relies on a number of factors. pyABC implements at its core an ABC-SMC scheme based on Toni & Stumpf (2010) and facilicates robust and efficient inference for a broad spectrum of applications via robust methods and self-tuned choices of hyperparameters, reducing the need…”
MethodMathematical Modelling of the MAP Kinase Pathway Using Proteomic Datasets
Tian, T., & Song, J. (2012) · PLoS ONE · doi:10.1371/journal.pone.0042230 · Cited by 49
Optimal experiment design for model selection in biochemical networks
Vanlier, J., Tiemann, C. A., Hilbers, P. A. J., & van Riel, N. A. W. (2014) · BMC Systems Biology · doi:10.1186/1752-0509-8-20 · Cited by 55
“Fortunately, recent advances in both MCMC [19,62] as well as approximate sampling techniques [39,48,63,64] allow sampling parameter distributions of increasingly complex models [14,34-38].”
Aitken, S., & Akman, O. E. (2013) · BMC Systems Biology · doi:10.1186/1752-0509-7-72 · Cited by 48
“Let B be an array of l + 1 n-dimensional points such that B[1]= D[ti] and B[l + 1]= D[ti+1].”
BackgroundMethodInfluentialImpact of individual behaviour change on the spread of emerging infectious diseases
Yan, Q., Tang, S. Y., & Xiao, Y. (2017) · Statistics in Medicine · doi:10.1002/sim.7548 · Cited by 44
Deviance Information Criteria for Model Selection in Approximate Bayesian Computation
François, O., & Laval, G. (2011) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1678 · Cited by 45
Schulte, M. B., Draghi, J. A., Plotkin, J. B., & Andino, R. (2015) · eLife · doi:10.7554/elife.03753 · Cited by 40
Accuracy of parameter estimation for auto-regulatory transcriptional feedback loops from noisy data
Cao, Z., & Grima, R. (2019) · Journal of The Royal Society Interface · doi:10.1098/rsif.2018.0967 · Cited by 42
“Approximate Bayesian computing approaches perform exhaustive stochastic simulations using the stochastic simulation algorithm (SSA) [23] and accept parameter values if the differences between simulation and experimental data are sufficiently small [19,24,25].”
Dokoohaki, H., Miguez, F. E., Archontoulis, S. V., & Laird, D. A. (2018) · Agricultural Water Management · doi:10.1016/j.agwat.2018.06.034 · Cited by 42
Bayesian analysis of simulation-based models
Turner, B. M., Sederberg, P. B., & McClelland, J. L. (2014) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2014.10.001 · Cited by 51
A flexible Bayesian framework for unbiased estimation of timescales
Zeraati, R., Engel, T. A., & Levina, A. (2022) · Nature Computational Science · doi:10.1038/s43588-022-00214-3 · Cited by 51
Data-Driven Modelling of Biological Multi-Scale Processes
Hasenauer, J., Jagiella, N., Hroß, S., & Theis, F. J. (2015) · Journal of Coupled Systems and Multiscale Dynamics · doi:10.1166/jcsmd.2015.1069 · Cited by 47
“SES To ensure eciency and reliability of ABC methods, a variety of sophisticated sampling schemes have been developed. Popular are ABC Markov chain Monte-Carlo and ABC sequential Monte-Carlo methods [235]. For models with time-consuming simulations also surrogate-based approaches have been developed, i.e., approximate…”
MethodInfluentialHazra, I., Pandey, M. D., & Manzana, N. (2020) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2019.106780 · Cited by 44
Statistical inference for dynamical systems: A review
McGoff, K. A., Mukherjee, S., & Pillai, N. S. (2015) · Statistics Surveys · doi:10.1214/15-ss111 · Cited by 71
Dynamic noise, chaos and parameter estimation in population biology
Stollenwerk, N., Aguiar, M., Ballesteros, S. P., Boto, J. P., Kooi, B. W., & Mateus, L. (2012) · Interface Focus · doi:10.1098/rsfs.2011.0103 · Cited by 39
Carrignon, S., Brughmans, T., & Romanowska, I. (2020) · PLoS ONE · doi:10.1371/journal.pone.0240414 · Cited by 42
Providing data science support for systems pharmacology and its implications to drug discovery
Hart, T. D., & Xie, L. (2015) · Expert Opinion on Drug Discovery · doi:10.1517/17460441.2016.1135126 · Cited by 37
“In addition to briefly reviewing the state-of-the-art techniques in addressing these problems, we will analyze unsolved problems and propose future directions.”
BackgroundQuantification of subclonal selection in cancer from bulk sequencing data
Williams, M. J., Werner, B., Curtis, C., Barnes, C. P., Sottoriva, A., & Graham, T. A. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/096305 · Cited by 36
Caley, P. A., Ramsey, D. S. L., & Barry, S. C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0116631 · Cited by 38
Tipping point analysis of atmospheric oxygen concentration
Livina, V., Martins, T. M. V., & Forbes, A. (2015) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.4907185 · Cited by 34
Proteolytic dynamics of human 20S thymoproteasome
Kuckelkorn, U., Stübler, S., Textoris‐Taube, K., Kilian, C., Niewienda, A., Henklein, P., Janek, K., Stumpf, M. P. H., Mishto, M., & Liepe, J. (2019) · Journal of Biological Chemistry · doi:10.1074/jbc.ra118.007347 · Cited by 34
Self-citationLiepe, J., Taylor, H., Barnes, C. P., Huvet, M., Bugeon, L., Thorne, T., Lamb, J. R., Dallman, M. J., & Stumpf, M. P. H. (2012) · Integrative Biology · doi:10.1039/c2ib00175f · Cited by 33
Self-citationMacLean, A. L., Celso, C. L., & Stumpf, M. P. H. (2013) · Journal of The Royal Society Interface · doi:10.1098/rsif.2012.0968 · Cited by 33
Self-citationFisher, P., Spencer, H., THOMAS‐OATES, J. E., Wood, A. J., & Ungár, D. (2019) · Cell Reports · doi:10.1016/j.celrep.2019.03.107 · Cited by 38
Moon, S. A., Cohnstaedt, L. W., McVey, D. S., & Scoglio, C. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006875 · Cited by 36
Bayesian Learning from Marginal Data in Bionetwork Models
Bonassi, F. V., You, L., & West, M. (2011) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1684 · Cited by 35
“They are of increasing interest in biological system applications, in particular (e.g. Toni and Stumpf, 2010, Wilkinson, 2011).”
Efficient likelihood-free Bayesian Computation for household epidemics
Neal, P. J. (2010) · Statistics and Computing · doi:10.1007/s11222-010-9216-x · Cited by 33
“ABC has been successfully applied to questions of model choice and model criticism, see for example, Ratmann et al. (2007), Grelaud et al. (2009) and Toni and Stumpf (2010).”
MethodDark energy by natural evolution: Constraining dark energy using Approximate Bayesian Computation
Bernardo, R. C., Grandón, D., Said, J. L., & Cárdenas, V. H. (2023) · Physics of the Dark Universe · doi:10.1016/j.dark.2023.101213 · Cited by 37
“For more details, we refer the reader to [61] for an outstanding introduction to ABC.”
An approximate likelihood perspective on ABC methods
Karabatsos, G., & Leisen, F. (2018) · Statistics Surveys · doi:10.1214/18-ss120 · Cited by 42
Computational models of the JAK1/2-STAT1 signaling
Gambin, A., Charzyńska, A., Ellert‐Miklaszewska, A., & Rybiński, M. (2013) · JAK-STAT · doi:10.4161/jkst.24672 · Cited by 36
Graph spectral analysis of protein interaction network evolution
Thorne, T., & Stumpf, M. P. H. (2012) · Journal of The Royal Society Interface · doi:10.1098/rsif.2012.0220 · Cited by 30
Self-citationChapman, L. A. C., Kushel, M., Cox, S. N., Scarborough, A. P., Cawley, C. L., Nguyen, T. Q., Rodríguez-Barraquer, I., Greenhouse, B., Imbert, E., & Lo, N. C. (2021) · BMC Medicine · doi:10.1186/s12916-021-01965-y · Cited by 32
Quantifying cell transitions in C. elegans with data-fitted landscape models
Camacho-Aguilar, E., Warmflash, A., & Rand, D. A. J. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009034 · Cited by 31
“…likelihood problems over the past twenty years (Marin et al., 2012) including population genetics (Beaumont et al., 2009), pathogen transmission (Tanaka et al., 2006), reaction networks models (Toni and Stumpf, 2010; Liepe et al., 2014) and epidemic modelling (Mckinley et al., 2009), among others.”
BackgroundSelecting Summary Statistics in Approximate Bayesian Computation for Calibrating Stochastic Models
Burr, T. L., & Skurikhin, A. N. (2013) · BioMed Research International · doi:10.1155/2013/210646 · Cited by 35
“For examples of ABC applied to calibrate SMs, see [11–27] and the many references cited by [11–27].”
MethodToni, T., Ozaki, Y.-I., Kirk, P., Kuroda, S., & Stumpf, M. P. H. (2012) · Molecular BioSystems · doi:10.1039/c2mb05493k · Cited by 28
Self-citationSeasonal dynamics of snail populations in coastal Kenya: Model calibration and snail control
Gurarie, D., King, C. H., Yoon, N., Wang, X., & Alsallaq, R. A. (2016) · Advances in Water Resources · doi:10.1016/j.advwatres.2016.11.008 · Cited by 28
Adversarial Variational Optimization of Non-Differentiable SimulatorsPreprint
Louppe, G., Hermans, J., & Cranmer, K. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.07113 · Cited by 43
“We compare our algorithm against ABC-SMC (Toni and Stumpf, 2009) and BOLFI (Gutmann and Corander, 2016) as baselines.”
Toni, T., Jovanovic, G., Huvet, M., Buck, M., & Stumpf, M. P. (2011) · BMC Systems Biology · doi:10.1186/1752-0509-5-69 · Cited by 23
“Then a model selection approach [30] can be employed in order to determine which of the proposed models reproduces the desired behaviour most reliably and most robustly.”
BackgroundMethodInfluentialSelf-citationShin, S., Venturelli, O. S., & Zavala, V. M. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006828 · Cited by 28
Bayesian design strategies for synthetic biology
Barnes, C. P., Silk, D., & Stumpf, M. P. H. (2011) · Interface Focus · doi:10.1098/rsfs.2011.0056 · Cited by 33
Self-citationKacprzak, T., Herbel, J., Amara, A., & Refregier, A. R. (2018) · Journal of Cosmology and Astroparticle Physics · doi:10.1088/1475-7516/2018/02/042 · Cited by 27
“It has been implemented within the Monte Carlo Markov Chain (MCMC) framework [1, 10], using Sequential Monte Carlo [11, 12] and Population Monte Carlo [8].”
MethodSchälte, Y., & Hasenauer, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa397 · Cited by 29
“The second approach (A2) is to modify the forward model simulation to account for measurement noise, i.e. to randomize the model output y → ¯ y ∼ π (¯ y | y, θ ) (e.g. in Toni and Stumpf (2010)), thus replacing p by q .”
BackgroundMethodAlahmadi, A. A., Flegg, J. A., Cochrane, D. G., Drovandi, C., & Keith, J. M. (2020) · Royal Society Open Science · doi:10.1098/rsos.191315 · Cited by 34
Open problems in mathematical biology
Vittadello, S. T., & Stumpf, M. P. H. (2022) · Mathematical Biosciences · doi:10.1016/j.mbs.2022.108926 · Cited by 30
Self-citationCarrignon, S., Bentley, R. A., & Ruck, D. J. (2019) · Humanities and Social Sciences Communications · doi:10.1057/s41599-019-0295-9 · Cited by 32
pomp: Statistical Inference for Partially Observed Markov Processes
King, A. A., Ionides, E. L., & Bretó, C. (2007) · doi:10.32614/cran.package.pomp · Cited by 27
Albert, C., Ulzega, S., & Stoop, R. (2016) · Physical review. E · doi:10.1103/physreve.93.043313 · Cited by 25
“at are based on simulating model realizations and comparing them to the data. Popular methods are particle lters [6, 15], Metropolis-within-Gibbs algorithms [28, 20] or Approximate Bayes Computations [17, 1, 30, 29]. A major problem with these simulation-based methods is, however, their ineciency in the presence of ma…”
MethodBayesian uncertainty quantification for data-driven equation learning
Martina-Perez, S. F., Simpson, M. J., & Baker, R. E. (2021) · Proceedings of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rspa.2021.0426 · Cited by 31
“This need comes from the fact that the, often significant, noise in real-world data can impact the models predicted by EQL methods, and hence the predictive capability of the models for unseen data or scenarios [20,21].”
BackgroundZhang, H., Wang, Y.-C., Zhang, T.-J., & Zhang, T. (2023) · The Astrophysical Journal Supplement Series · doi:10.3847/1538-4365/accb92 · Cited by 31
“The following are the steps to use the ABC Rejection method(Toni & Stumpf 2009): • To ensure uniformity across all kernels studied, we assign the same prior probability, p(Mi), to each kernel.”
MethodInfluentialda Costa, J. M. J., Orlande, H. R. B., & da Silva, W. B. (2017) · Computational and Applied Mathematics · doi:10.1007/s40314-017-0479-0 · Cited by 22
“The concepts outlined in the above algorithm can be extended to incorporate model selection in addition to parameter estimation (Toni et al. 2009; Toni and Stumpf 2010a,b).”
BackgroundMethodInfluentialLonergan, M. E., Thompson, D., Thomas, L., & Duck, C. (2011) · Journal of Marine Biology · doi:10.1155/2011/597424 · Cited by 21
“, [19]), where explicit calculation of the likelihood is avoided by the use of easier-to-compute summary statistics.”
Engelhardt, B., Kschischo, M., & Fröhlich, H. (2017) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0332 · Cited by 26
Robert, C. P. (2013) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-022513-115543 · Cited by 22
“…setting, the formal simplicity of computing posterior probabilities can be mimicked by an ABC-MC (for model choice) algorithm as the following one (Toni and Stumpf, 2010): where M denotes the unknown model index, m being one of the possible values, with π m the corresponding prior on the…”
MethodSimulation of Territorial Development Based on Fiscal Policy Tools
Brumnik, R., Семеновна, К. Т., Guryanova, L. S., Кавун, С., & Trydid, O. (2014) · Mathematical Problems in Engineering · doi:10.1155/2014/843976 · Cited by 19
An evolutionary perspective on anti-tumor immunity
Klinke, D. J. (2013) · Frontiers in Oncology · doi:10.3389/fonc.2012.00202 · Cited by 20
“While genetic sequencing technology has revealed the complexity and heterogeneity of oncogenic transformation, understanding the dynamics of oncogenesis and how cancer cells alter the selective fitness landscape remain unclear.”
BackgroundDifferential equations in data analysis
Dattner, I. (2020) · Wiley Interdisciplinary Reviews Computational Statistics · doi:10.1002/wics.1534 · Cited by 22
“…selection has most commonly been performed using likelihood ratio tests in the case of nested models, or the Akaike information criterion (AIC), proposed by Akaike (1974), in the case of non-nested models; see also Miao et al. (2009), Toni and Stumpf (2010), and Zhang, Cao, and Carroll (2015).”
MethodHuang, L., Liao, L., & Wu, C. H. (2014) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2014.2366748 · Cited by 19
“To deal with this problem, most recently, Thorne and Stumpf [18] developed approxi L. Huang and L. Liao are with the Department of Computer and mate Bayesian computation and sequential Monte Carlo Information Sciences, University of Delaware, Newark, DE 19716. method (ABC-SMC) to do graph spectral analysis, which E-m…”
MethodInfluentialP38 and JNK have opposing effects on persistence of in vivo leukocyte migration in zebrafish
Taylor, H. B., Liepe, J., Barthen, C., Bugeon, L., Huvet, M., Kirk, P., Brown, S. B., Lamb, J. R., Stumpf, M. P. H., & Dallman, M. J. (2012) · Immunology and Cell Biology · doi:10.1038/icb.2012.57 · Cited by 18
Self-citation
Metadata
- Authors
- Toni, T., & Stumpf, M. P. H.
- Affiliations
- Imperial College London (GB)
- Corresponding author
- Toni, Tina
- Stumpf, Michael P. H.
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2009-10-29
- Volume
- 26
- Issue
- 1
- Pages
- 104-110
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- PMC
- PMC2796821
- OpenAlex
- W2126351705
- Cited by
- 260
- References
- 43
- Citations by year
- 2026: 7 · 2025: 12 · 2024: 11 · 2023: 13 · 2022: 6 · 2021: 19 · 2020: 16 · 2019: 15 · 2018: 19 · 2017: 16 · 2016: 15 · 2015: 20 · 2014: 22 · 2013: 19 · 2012: 18
- FWCI
- 6.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Bioinformatics (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by-nc)
- arXiv (Cornell University)
- Funders
- Directorate for Biological Sciences
- Medical Research Council
- Biotechnology and Biological Sciences Research Council (BB/F005210/2, BB/F005210/1, BB/C519670/1, BB/G009374/1)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Physical Sciences › Mathematics › Statistics and Probability › Markov Chains and Monte Carlo Methods
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Philosophy of Biology and Scientific Explanation
- Keywords
- model selection, dynamical systems, approximate Bayesian computation, sequential Monte Carlo, signal transduction, gene regulatory networks, JAK/STAT signaling, population biology, measurement uncertainty, mechanistic model
- MeSH terms
- Algorithms, Animals, Bayes Theorem, Computer Simulation, Data Interpretation, Statistical, Humans, Models, Biological, Population Dynamics, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.