Boba: Authoring and Visualizing Multiverse Analyses
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Multiverse analysis is an approach to data analysis in which all "reasonable" analytic decisions are evaluated in parallel and interpreted collectively, in order to foster robustness and transparency. However, specifying a multiverse is demanding because analysts must manage myriad variants from a cross-product of analytic decisions, and the results require nuanced interpretation. We contribute Baba: an integrated domain-specific language (DSL) and visual analysis system for authoring and reviewing multiverse analyses. With the Boba DSL, analysts write the shared portion of analysis code only once, alongside local variations defining alternative decisions, from which the compiler generates a multiplex of scripts representing all possible analysis paths. The Boba Visualizer provides linked views of model results and the multiverse decision space to enable rapid, systematic assessment of consequential decisions and robustness, including sampling uncertainty and model fit. We demonstrate Boba's utility through two data analysis case studies, and reflect on challenges and design opportunities for multiverse analysis software.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
(2015) · doi:10.1007/s11222-016-9696-4 · Cited by 5,611
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“The last decade saw widespread failure to replicate findings in published literature across multiple scientific fields [2, 6, 35, 41].”
BackgroundThe association between adolescent well-being and digital technology use
Orben, A., & Przybylski, A. K. (2019) · Nature Human Behaviour · doi:10.1038/s41562-018-0506-1 · Cited by 1,500 · cited 1×
“, [6,9,21,36,43]), few tools currently exist to aid authoring.”
BackgroundThe Statistical Crisis in Science
Gelman, A., & Loken, E. (2014) · American Scientist · doi:10.1511/2014.111.460 · Cited by 963
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
Border, R., Johnson, E. C., Evans, L. M., Smolen, A., Berley, N., Sullivan, P. F., & Keller, M. C. (2019) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2018.18070881 · Cited by 629
Cited by
The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
MENKVELD, A. J., DREBER, A., HOLZMEISTER, F., HUBER, J., JOHANNESSON, M., KIRCHLER, M., NEUSÜß, S., RAZEN, M., WEITZEL, U., ABAD‐DÍAZ, D., et al. (2024) · The Journal of Finance · doi:10.1111/jofi.13337 · Cited by 113
A Critical Reflection on Visualization Research: Where Do Decision Making Tasks Hide?
Dimara, E., & Stasko, J. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114813 · Cited by 60
How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study
Gu, K., Grunde-McLaughlin, M., McNutt, A., Heer, J., & Althoff, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641891 · Cited by 40
“Furthermore, these assistants could be seamlessly integrated with existing multiverse analysis systems [48, 81, 104] to help analysts conduct multiverse analyses.”
InfluentialSelf-citation
Metadata
- Authors
- Liu, Y., Kale, A., Althoff, T., & Heer, J.
- Venue
- IEEE Transactions on Visualization and Computer Graphics
- Published
- 2021-02
- Volume
- 27
- Issue
- 2
- Pages
- 1753-1763
- ISSN
- 1077-2626 (ISSN-L)
- 1941-0506
- 2160-9306
- OpenAlex
- W3041984699
- Cited by
- 73
- References
- 12
- FWCI
- 3.89
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Boba: Authoring and Visualizing Multiverse Analyses. https://app.cochat.ai/papers/10.1109/tvcg.2020.3028985
- The paper
- Liu, Y., Kale, A., Althoff, T., & Heer, J. (2021). Boba: Authoring and Visualizing Multiverse Analyses. IEEE Transactions on Visualization and Computer Graphics. https://doi.org/10.1109/tvcg.2020.3028985
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.