Provenance in modeling and simulation studies — Bridging gaps
Abstract
Simulation studies are intricate processes that require interweaving model refinement and executing diverse experiments. Simulation models and data are the result of complex and interactive model and data generating processes. Information about these processes are required to assess the quality of simulation products. Capturing provenance, i.e., information about how a product has been generated, is a major concern both for assessing and reproducing scientific experiments. For parts of a simulation study, support for capturing and managing provenance is available. However, still gaps exist, e.g., how simulation models have been generated and to look therefore beyond individual simulation experiments and even simulation studies. To bridge those gaps it will be central to exploit, refine, and combine diverse methods effectively, as we demonstrate on a concrete case study and its provenance model.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Each entity and activity in the final set has already been mentioned for provenance, modeling or documentation purposes, or experiment design of simulation studies, but they have not all been used together.
“Each entity and activity has already been mentioned for provenance, modeling or documentation purposes, or experiment design of simulation studies [1, 3–5, 8, 20, 21, 60–63], but they have not all been used together.”
From Relating simulation studies by provenance—Developing a family of Wnt signaling models (2021) · cites this paper as [21]
Cites
Hucka, M., Finney, A., Sauro, H. M., Bolouri, H., Doyle, J. C., Kitano, H., Arkin, A. P., Bornstein, B. J., Bray, D., Cornish-Bowden, A., et al. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btg015 · Cited by 3,060 · see citing works
The ODD protocol: A review and first update
Grimm, V., Berger, U., Deangelis, D. L., Polhill, G. J., Giske, J., & Railsback, S. F. (2010) · Ecological Modelling · doi:10.1016/j.ecolmodel.2010.08.019 · Cited by 2,471 · see citing works
A survey of data provenance in e-science
Simmhan, Y. L., Plale, B., & Gannon, D. B. (2005) · ACM SIGMOD Record · doi:10.1145/1084805.1084812 · Cited by 1,155 · see citing works
Adeptflex—Supporting Dynamic Changes of Workflows Without Losing Control
Reichert, M., & Dadam, P. (1998) · Journal of Intelligent Information Systems · doi:10.1023/a:1008604709862 · Cited by 883 · see citing works
Novel mechanism of Wnt signalling inhibition mediated by Dickkopf-1 interaction with LRP6/Arrow
Bafico, A., Liu, G., Yaniv, A., Gazit, A., & Aaronson, S. A. (2001) · Nature Cell Biology · doi:10.1038/35083081 · Cited by 772 · see citing works
The Roles of APC and Axin Derived from Experimental and Theoretical Analysis of the Wnt Pathway
Lee, E., Salic, A., Krüger, R., Heinrich, R., & Kirschner, M. W. (2003) · PLoS Biology · doi:10.1371/journal.pbio.0000010 · Cited by 708 · see citing works
The Open Provenance Model core specification (v1.1)
Moreau, L., Clifford, B., Freire, J., Futrelle, J., Gil, Y., Groth, P., Kwaśnikowska, N., Miles, S., Paolo Missier, B., Myers, J., et al. (2010) · Future Generation Computer Systems · doi:10.1016/j.future.2010.07.005 · Cited by 706 · see citing works
Wolstencroft, K., Haines, R., Fellows, D. K., Williams, A. R., Withers, D., Owen, S., Soiland‐Reyes, S., Dunlop, I., Nenadić, A., Fisher, P., et al. (2013) · Nucleic Acids Research · doi:10.1093/nar/gkt328 · Cited by 671 · see citing works
Waltemath, D., Adams, R., Bergmann, F. T., Hucka, M., Kolpakov, F., Miller, A. K., Moraru, I., Nickerson, D., Sahle, S., Snoep, J., et al. (2011) · BMC Systems Biology · doi:10.1186/1752-0509-5-198 · Cited by 269 · see citing works
The Physiome Model Repository 2
Yu, T., Lloyd, C. M., Nickerson, D. P., Cooling, M. T., Miller, A. K., Garny, A., Terkildsen, J. R., Lawson, J., Britten, R. D., Hunter, P. J., et al. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btq723 · Cited by 169 · see citing works
Minimum Information About a Simulation Experiment (MIASE)
Waltemath, D., Adams, R., Beard, D. A., Bergmann, F. T., Bhalla, U. S., Britten, R., Chelliah, V., Cooling, M. T., Cooper, J., Crampin, E. J., et al. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001122 · Cited by 168 · see citing works
A life cycle for modeling and simulation
Balci, O. (2012) · SIMULATION · doi:10.1177/0037549712438469 · Cited by 159 · see citing works
PROV-Overview. An Overview of the PROV Family of Documents
Groth, P. T., & Moreau, L. (2013) · Cited by 113 · see citing works
Scientific Workflows: Business as Usual?
Ludäscher, B., Weske, M., McPhillips, T., & Bowers, S. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-03848-8_4 · Cited by 109 · see citing works
Rharass, T., Lemcke, H., Lantow, M., Kuznetsov, S. A., Weiss, D. G., & Panáková, D. (2014) · Journal of Biological Chemistry · doi:10.1074/jbc.m114.573519 · Cited by 108 · see citing works
Investigating ontologies for simulation modeling
Miller, J. A., Baramidze, G. T., Sheth, A., & Fishwick, P. A. (2004) · Annual Simulation Symposium (ANSS) · doi:10.1109/simsym.2004.1299465 · Cited by 106 · see citing works
Research Challenges in Modeling and Simulation for Engineering Complex Systems
Fujimoto, R., Bock, C., Chen, W., Page, E., & Panchal, J. (2017) · Simulation Foundations, Methods and Applications · doi:10.1007/978-3-319-58544-4 · Cited by 88 · see citing works
Ewald, R., & Uhrmacher, A. (2014) · ACM Transactions on Modeling and Computer Simulation · doi:10.1145/2567895 · Cited by 75 · see citing works
Self-citationFrom Repeatability to Reproducibility and Corroboration
Feitelson, D. G. (2015) · ACM SIGOPS Operating Systems Review · doi:10.1145/2723872.2723875 · Cited by 75 · see citing works
Tackling the Provenance Challenge one layer at a time
Scheidegger, C., Koop, D., Santos, E., Vo, H. T., Callahan, S. P., Freire, J., & Silva, C. (2007) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.1237 · Cited by 71 · see citing works
Haack, F., Lemcke, H., Ewald, R., Rharass, T., & Uhrmacher, A. M. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004106 · Cited by 66 · see citing works
Self-citationGit2PROV: exposing version control system content as W3C PROV
Nies, T. D., Magliacane, S., Verborgh, R., Coppens, S., Groth, P., Mannens, E., & Rik Van De Walle, B. (2013) · Cited by 45 · see citing works
Kinetics of Wnt-driven β-catenin stabilization revealed by quantitative and temporal imaging
Hannoush, R. N. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003498 · Cited by 42 · see citing works
Mazemondet, O., Hübner, R., Frahm, J., Koczan, D., Bader, B., Weiss, D. G., Uhrmacher, A. M., Frech, M. J., Rolfs, A., & Luo, J. (2011) · Cellular & Molecular Biology Letters · doi:10.2478/s11658-011-0021-0 · Cited by 40 · see citing works
Self-citationThe FTG+PM framework for multi-paradigm modelling
Mustafiz, S., Denil, J., Lúcio, L., & Vangheluwe, H. L. (2012) · doi:10.1145/2508443.2508446 · Cited by 38 · see citing works
ON REPRODUCIBILITY AND TRACEABILITY OF SIMULATIONS
Laroque, C., Himmelspach, J., Pasupathy, R., Rose, O., Uhrmacher, A. M., Dalle, O., & Antipolis, S. (2014) · Cited by 32 · see citing works
Self-citationElucidating the Sources of β-Catenin Dynamics in Human Neural Progenitor Cells
Mazemondet, O., John, M., Leye, S., Rolfs, A., & Uhrmacher, A. M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0042792 · Cited by 25 · see citing works
Self-citationImproving the reuse of computational models through version control
Waltemath, D., Henkel, R., Hälke, R., Scharm, M., & Olaf Wolkenhauer, A. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt018 · Cited by 23 · see citing works
Panel — Reproducible research in discrete event simulation — A must or rather a maybe?
Uhrmacher, A. M., Brailsford, S., Liu, J., Rabe, M., & Tolk, A. (2016) · Winter Simulation Conference · doi:10.1109/wsc.2016.7822185 · Cited by 23 · see citing works
Self-citationReusing simulation experiment specifications to support developing models by successive extension
Peng, D.-H., Warnke, T., Haack, F., & Uhrmacher, A. (2016) · Simulation modelling practice and theory · doi:10.1016/j.simpat.2016.07.006 · Cited by 22 · see citing works
Self-citationA model-driven engineering approach to simulation experiment design and execution
Teran-Somohano, A., Smith, A. E., Ledet, J. W., Yılmaz, L., & Oğuztüzün, H. (2015) · Winter Simulation Conference · doi:10.1109/wsc.2015.7408371 · Cited by 22 · see citing works
Model-as-you-go: An Approach for an Advanced Infrastructure for Scientific Workflows
Sonntag, M., & Karastoyanova, D. (2013) · Journal of Grid Computing · doi:10.1007/s10723-013-9268-1 · Cited by 22 · see citing works
OPQL: A First OPM-Level Query Language for Scientific Workflow Provenance
Lim, C., Lu, S., Artem Chebotko, B., & Fotouhi, F. A. (2011) · IEEE International Conference on Services Computing · doi:10.1109/scc.2011.60 · Cited by 19 · see citing works
Principles of Provenance (Dagstuhl Seminar 12091)
Cheney, J., Finkelstein, A., Ludaescher, B., & Vansummeren, S. (2012) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/dagrep.2.2.84 · Cited by 14 · see citing works
Peng, D., Warnke, T., Haack, F., & Uhrmacher, A. M. (2017) · SIMULATION · doi:10.1177/0037549717704314 · Cited by 14 · see citing works
Self-citationRybacki, S., Haack, F., Wolf, K., & Uhrmacher, A. M. (2014) · International ICST Conference on Simulation Tools and Techniques (SIMUTools) · doi:10.4108/icst.simutools.2014.254650 · Cited by 7 · see citing works
Self-citation
Flagged first, then by citations
1 of the 37 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Harmonizing semantic annotations for computational models in biology
Neal, M. L., König, M., Nickerson, D., Mısırlı, G., Kalbasi, R., Dräger, A., Atalag, K., Chelliah, V., Cooling, M. T., Cook, D. L., et al. (2019) · Briefings in Bioinformatics · doi:10.1093/bib/bby087 · Cited by 74 · see citing works
Towards Bayesian Model-Based Demography
Bijak, J. (2021) · Methodos series · doi:10.1007/978-3-030-83039-7 · Cited by 33 · see citing works
Relating simulation studies by provenance—Developing a family of Wnt signaling models
Budde, K., Smith, J., Wilsdorf, P., Haack, F., & Uhrmacher, A. M. (2021) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1009227 · Cited by 23 · see citing works
Self-citationArtifact-Based Workflows for Supporting Simulation Studies
Ruscheinski, A., Warnke, T., & Uhrmacher, A. M. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2899840 · Cited by 23 · see citing works
Self-citationAutomatic Reuse, Adaption, and Execution of Simulation Experiments via Provenance Patterns
Wilsdorf, P., Wolpers, A., Hilton, J., Haack, F., & Uhrmacher, A. M. (2022) · ACM Transactions on Modeling and Computer Simulation · doi:10.1145/3564928 · Cited by 21 · see citing works
Self-citationContext, Composition, Automation, and Communication: The C 2 AC Roadmap for Modeling and Simulation
Uhrmacher, A. M., Frazier, P. I., Hähnle, R., Klügl, F., Lorig, F., Ludäscher, B., Nenzi, L., Ruiz-Martín, C., Rumpe⋆, B., Szabo, C., et al. (2024) · ACM Transactions on Modeling and Computer Simulation · doi:10.1145/3673226 · Cited by 21 · see citing works
Self-citationODD+P: COMPLEMENTING THE ODD PROTOCOL WITH PROVENANCE INFORMATION
Reinhardt, O., Rucheinski, A., & Uhrmacher, A. M. (2018) · Winter Simulation Conference · doi:10.1109/wsc.2018.8632481 · Cited by 20 · see citing works
Self-citationCRISIS, WHAT CRISIS – DOES REPRODUCIBILITY IN MODELING & SIMULATION REALLY MATTER?
Taylor, S. J. E., Eldabi, T., Monks, T., Rabe, M., & Uhrmacher, A. M. (2018) · Winter Simulation Conference · doi:10.1109/wsc.2018.8632232 · Cited by 19 · see citing works
Self-citationExploring the mechanistic and temporal regulation of LRP6 endocytosis in canonical WNT signaling
Haack, F., Budde, K., & Uhrmacher, A. M. (2020) · Journal of Cell Science · doi:10.1242/jcs.243675 · Cited by 18 · see citing works
Self-citationConceptual Models in Simulation Studies: Making it Explicit
Wilsdorf, P., Haack, F., & Uhrmacher, A. M. (2020) · 2020 Winter Simulation Conference (WSC) · doi:10.1109/wsc48552.2020.9383984 · Cited by 15 · see citing works
Self-citationTowards a PROV Ontology for Simulation Models
Ruscheinski, A., Gjorgevikj, D., Dombrowsky, M., Budde, K., & Uhrmacher, A. M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-98379-0_17 · Cited by 13 · see citing works
Self-citationAgent-Based Simulation Models in Fisheries Science
Haase, K., Reinhardt, O., Lewin, W., Weltersbach, M. S., Strehlow, H. V., & Uhrmacher, A. M. (2023) · Reviews in Fisheries Science & Aquaculture · doi:10.1080/23308249.2023.2201635 · Cited by 11 · see citing works
Self-citationGENERATING SIMULATION EXPERIMENTS BASED ON MODEL DOCUMENTATIONS AND TEMPLATES
Ruscheinski, A., Budde, K., Warnke, T., Wilsdorf, P., Hiller, B. C., Dombrowsky, M., & Uhrmacher, A. M. (2018) · Winter Simulation Conference · doi:10.1109/wsc.2018.8632515 · Cited by 11 · see citing works
Self-citationTagungsband ASIM 2018 – 24. Symposium Simulationstechnik
Sundermeier, J., Gehlhoff, F., & Fay, A. (2018) · doi:10.11128/arep.56 · Cited by 10 · see citing works
Open Science, Replicability, and Transparency in Modelling
Prike, T. (2021) · Methodos series · doi:10.1007/978-3-030-83039-7_10 · Cited by 9 · see citing works
Modelling Migration: Decisions, Processes and Outcomes
Bijak, J., Higham, P. A., Hilton, J., Hinsch, M., Nurse, S., Prike, T., Smith, P. W. F., Reinhardt, O., & Uhrmacher, A. M. (2020) · Winter Simulation Conference · doi:10.1109/wsc48552.2020.9384072 · Cited by 8 · see citing works
Self-citationCoupled Traffic Simulation by Detached Translation Federates: An HLA-Based Approach
Gütlein, M., & Djanatliev, A. (2019) · Winter Simulation Conference · doi:10.1109/wsc40007.2019.9004809 · Cited by 7 · see citing works
Validation Without Data - Formalizing Stylized Facts Of Time Series
Wilsdorf, P., Zuska, M., Andelfinger, P., Uhrmacher, A. M., & Peters, F. (2023) · Winter Simulation Conference · doi:10.1109/wsc60868.2023.10408388 · Cited by 5 · see citing works
Self-citationWilsdorf, P., Fischer, N., Haack, F., & Uhrmacher, A. M. (2021) · Winter Simulation Conference · doi:10.1109/wsc52266.2021.9715362 · Cited by 4 · see citing works
Self-citationSimulation-Supported Engineering of Self-Adaptive Software Systems
Meyer, T., Ruscheinski, A., Wilsdorf, P., & Uhrmacher, A. M. (2021) · Winter Simulation Conference · doi:10.1109/wsc52266.2021.9715324 · Cited by 3 · see citing works
Self-citationSite Choice in Recreational Fisheries - Towards an Agent-Based Approach
Haase, K., Strehlow, H. V., Lewin, W., Reinhardt, O., & Uhrmacher, A. M. (2022) · Winter Simulation Conference · doi:10.1109/wsc57314.2022.10015240 · Cited by 3 · see citing works
Self-citationValid and Reproducible Simulation Studies—Making It Explicit
Reinhardt, O., Warnke, T., Ruscheinski, A., & Uhrmacher, A. M. (2019) · Simulation foundations, methods and applications · doi:10.1007/978-3-319-70766-2_25 · Cited by 2 · see citing works
Self-citationPractical Tie-Breaking for Parallel/Distributed Simulations
Piccione, A., & Pellegrini, A. (2023) · International Symposium on Distributed Simulation and Real Time Applications (DS-RT) · doi:10.1109/ds-rt58998.2023.00018 · Cited by 1 · see citing works
SIMPROV: Provenance capturing for simulation studies
Ruscheinski, A., Wolpers, A., Henning, P., Wilsdorf, P., & Uhrmacher, A. M. (2025) · PLoS ONE · doi:10.1371/journal.pone.0327607 · Cited by 1 · see citing works
Self-citationSchröder, M., Raben, H., Krüger, F., Ruscheinski, A., van Rienen, U., Uhrmacher, A. M., & Spors, S. (2019) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2019.8856841 · Cited by 1 · see citing works
Self-citationTeaching a Modeling Process: Reflections From An Online Course
Bijak, J., Hilton, J., Hinsch, M., Lipscombe, K., Nurse, S., Prike, T., Smith, P. W. F., Reinhardt, O., Uhrmacher, A. M., Higham, P. A., et al. (2021) · Winter Simulation Conference · doi:10.1109/wsc52266.2021.9715415 · Cited by 1 · see citing works
Self-citationNBSIMGEN: Jupyter Notebook Extension for Generating Simulation Experiments
Wilsdorf, P., Kirchhübel, A. W., & Uhrmacher, A. M. (2023) · Winter Simulation Conference · doi:10.1109/wsc60868.2023.10408537 · Cited by 1 · see citing works
Self-citationAgent-Based Modelling and Simulation with Domain-Specific Languages
Reinhardt, O., Warnke, T., & Uhrmacher, A. M. (2021) · Methodos series · doi:10.1007/978-3-030-83039-7_7 · Cited by 1 · see citing works
Self-citationInhalt Tagungsband ASIM 2018 – 24. Symposium Simulationstechnik
Deatcu, C. (2018) · doi:10.11128/arep.56.a56000 · Cited by 1 · see citing works
Creating PROV-DM Graphs from Model Databases
Wilsdorf, P., & Uhrmacher, A. M. (2022) · Winter Simulation Conference · doi:10.1109/wsc57314.2022.10015331
Self-citationEmpowering Cover Crop Decision Support with Visualization and Provenance Enhancement
Shrestha, S., Sun, J., Koehler‐Cole, K., Basche, A., & Yu, H. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386794
Self-Adaptive Simulation Models: A Case Study in Cell Biology
Wilsdorf, P., Henning, P., Kreikemeyer, J. N., Kliefoth, M., Baltrusch, S., & Uhrmacher, A. M. (2025) · International Symposium on Distributed Simulation and Real Time Applications (DS-RT) · doi:10.1109/ds-rt68115.2025.11185996
Self-citationDialectic Models for Documenting and Conducting Simulation Studies: Exploring Feasibility
Zschaler, S., Wilsdorf, P., Godfrey, T., & Uhrmacher, A. M. (2025) · Winter Simulation Conference · doi:10.1109/wsc68292.2025.11339048
Self-citation
- Authors
- Ruscheinski, A., & Uhrmacher, A.
- Affiliations
- University of Rostock (DE)
- Venue
- 2017 Winter Simulation Conference (WSC)
- Publisher
- IEEE
- Published
- 2017-12
- Pages
- 872-883
- ISSN
- 0891-7736 (ISSN-L)
- OpenAlex
- W4230563465
- Cited by
- 31
- References
- 37
- Citations by year
- 2025: 3 · 2024: 1 · 2023: 5 · 2022: 3 · 2021: 7 · 2020: 3 · 2019: 4 · 2018: 5
- FWCI
- 7.00
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Winter Simulation Conference (publishedVersion)
- Winter Simulation Conference
- Funders
- Deutsche Forschungsgemeinschaft
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Decision Sciences › Management Science and Operations Research › Simulation Techniques and Applications
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- provenance, simulation study, simulation model generation, model refinement, reproducibility, provenance management, provenance model
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
