Strengthening the reporting of empirical simulation studies: Introducing the STRESS guidelines
Open accessAbstract
This study develops a standardised checklist approach to improve the reporting of discrete-event simulation, system dynamics and agent-based simulation models within the field of Operational Research and Management Science. Incomplete or ambiguous reporting means that many simulation studies are not reproducible, leaving other modellers with an incomplete picture of what has been done and unable to judge the reliability of the results. Crucially, unclear reporting makes it difficult to reproduce or reuse findings. In this paper, we review the evidence on the quality of model reporting and consolidate previous work. We derive general good practice principles and three 20-item checklists aimed at Strengthening The Reporting of Empirical Simulation Studies (STRESS): STRESS-DES, STRESS-ABS and STRESS-SD for discrete-event simulation, agent-based simulation and system dynamics, respectively. Given the variety of simulation projects, we provide usage and troubleshooting advice to cover a wide range of situations.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Each entity and activity in the final set has already been mentioned for provenance, modeling or documentation purposes, or experiment design of simulation studies, but they have not all been used together.
“Each entity and activity has already been mentioned for provenance, modeling or documentation purposes, or experiment design of simulation studies [1, 3–5, 8, 20, 21, 60–63], but they have not all been used together.”
From Relating simulation studies by provenance—Developing a family of Wnt signaling models (2021) · cites this paper as [4]
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414 · see citing works
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Relating simulation studies by provenance—Developing a family of Wnt signaling models
Budde, K., Smith, J., Wilsdorf, P., Haack, F., & Uhrmacher, A. M. (2021) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1009227 · Cited by 23 · see citing works
Conceptual Models in Simulation Studies: Making it Explicit
Wilsdorf, P., Haack, F., & Uhrmacher, A. M. (2020) · 2020 Winter Simulation Conference (WSC) · doi:10.1109/wsc48552.2020.9383984 · Cited by 15 · see citing works
- Authors
- Monks, T., Currie, C. S. M., Onggo, B. S., Robinson, S., Kunc, M., & Taylor, S. J. E.
- Affiliations
- University of Southampton (GB)
- Trinity College Dublin (IE)
- Loughborough University (GB)
- University of Warwick (GB)
- Brunel University of London (GB)
- Corresponding author
- Monks, Thomas
- ORCID
- Venue
- Journal of Simulation
- Publisher
- Informa UK Limited
- Published
- 2018-03-06
- Volume
- 13
- Issue
- 1
- Pages
- 55-67
- ISSN
- 1747-7778 (ISSN-L)
- 1747-7786
- OpenAlex
- W2792314241
- Cited by
- 248
- References
- 2
- Citations by year
- 2026: 38 · 2025: 39 · 2024: 32 · 2023: 37 · 2022: 44 · 2021: 26 · 2020: 21 · 2019: 8 · 2018: 2
- FWCI
- 11.36
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Journal of Simulation (publishedVersion, cc-by)
- ePrints Soton (University of Southampton) (acceptedVersion, cc-by)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- Loughborough University Institutional Repository (Loughborough University) (submittedVersion, cc-by)
- Figshare (submittedVersion, cc-by)
- Loughborough University Research Repository (Loughborough University) (submittedVersion, cc-by)
- Brunel University Research Archive (BURA) (Brunel University London) (submittedVersion)
- Open Research Exeter (University of Exeter) (submittedVersion, cc-by)
- Newcastle University ePrints (Newcastle Univesity)
- Funders
- National Institute for Health and Care Research
- Department of Health and Social Care
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Simulation Techniques and Applications
- Social Sciences › Decision Sciences › Management Science and Operations Research › Complex Systems and Decision Making
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Modeling and Simulation Systems
- Keywords
- reporting guidelines, discrete event simulation, agent-based modeling, system dynamics, reproducibility, checklists, research transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
