Online tools supporting the conduct and reporting of systematic reviews and systematic maps: a case study on CADIMA and review of existing tools
Open accessThe DOI matches its Crossref record; 35 of the 36 references it cites were found in a registry or an index; 3 of the 18 sources its claims cite were checked for support.
Abstract
Systematic reviews and systematic maps represent powerful tools to identify, collect, evaluate and summarise primary research pertinent to a specific research question or topic in a highly standardised and reproducible manner. Even though they are seen as the “gold standard” when synthesising primary research, systematic reviews and maps are typically resource-intensive and complex activities. Thus, managing the conduct and reporting of such reviews can become a time consuming and challenging task. This paper introduces the open access online tool CADIMA, which was developed through a collaboration between the Julius Kühn-Institut and the Collaboration for Environmental Evidence, in order to increase the efficiency of the evidence synthesis process and facilitate reporting of all activities to maximise methodological rigour. Furthermore, we analyse how CADIMA compares with other available tools by providing a comprehensive summary of existing software designed for the purposes of systematic review management. We show that CADIMA is the only available open access tool that is designed to: (1) assist throughout the systematic review/map process; (2) be suited to reviews broader than medical sciences; (3) allow for offline data extraction; and, (4) support working as a review team.
Overview
CoChat Summary
TL;DR
CADIMA is an open access online tool developed by JKI and CEE to support the conduct and reporting of systematic reviews and systematic maps across disciplines, including offline data extraction and team workflows. A review of 22 existing software packages found CADIMA was the only open access tool designed to assist throughout the review/map process, suit non-medical fields, allow offline data extraction, and support review teams.
Key findings
- The review identified 24 systematic review software packages; two were excluded (Slrtool discontinued; DRAGON ONLINE developers did not respond), leaving 22 packages analysed. Of these, nine were advertised for any field, nine for healthcare/medical science, three for software engineering, and one for experimental animal studies.
- Among the 22 packages, 16 were freely available for non-commercial use, six were open source, 15 had web-based functions, seven involved downloadable applications, and 10 supported or planned to support machine learning/text mining features.
- CADIMA supports protocol development, search-result management, duplicate identification, study selection including a kappa consistency check, on- and offline data extraction, critical appraisal, and documentation/outputs; it excludes built-in literature searching and quantitative synthesis.
- The paper concludes CADIMA is the only available open access tool designed to assist throughout the systematic review/map process, be suited to reviews broader than medical sciences, allow offline data extraction, and support working as a review team.
Methods
- Review of existing tools: four search approaches were used—online bibliographic database searches, snowballing via general web searches, screening targeted websites, and backwards/forwards citation searches. Twenty-four packages were identified; two were excluded, and the remaining 22 were researched and trialed by EJM where free access or free trials were available, then characterised by features including stages supported, team suitability, and cost.
- CADIMA description: CADIMA is a client-server web application coded in PHP V5.5 using the Yii V1.1 framework with Bootstrap CSS, running on Apache 2.4/Linux Ubuntu Server V14.04 with a MySQL 5.5 database, SSL encryption, and daily backups stored for 6 months; it is hosted and maintained by JKI.
- CADIMA’s menu structure mirrors systematic review/map steps, with two user roles: “review coordinator” and “review team members.”
Limitations
- Stated: CADIMA does not provide built-in literature searching or quantitative synthesis; duplicates are detectable but cannot be automatically removed in the current version; the same reviewers must participate during title, abstract, and full-text selection stages; the linear structure has so far precluded adjustments to review team membership between screening stages; and selection criteria cannot be changed without de novo performance of the consistency check and loss of previously extracted information.
- Stated in the review: two identified packages were excluded because one was discontinued and the other’s developers did not respond; where no further information was available, characteristics were marked “Unavailable”; and trialing occurred only where free access or free trials were available.
- Evident from the text: the review may become outdated quickly, as the paper notes existing reviews of software become out of date and three new or upgraded packages were expected to become live in 2017; comparison also relies partly on advertised or developer-provided information, so feature coverage may be incomplete.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 17 Not valid · 12 Not checked · 94 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“CADIMA is also suitable for assisting in the process of conducting other forms of evidence synthesis, including systematic maps [8, 9] and rapid reviews [26]”
What the sources say
- Supportedcites [8] James KL, Randall NP, Haddaway NR. A methodology for systematic mapping in environmental sciences. Environ Evid. 2016;5:7. source
The citation [8,9] is attached only to the exemplar 'systematic maps', so what is attributed to this source is that systematic mapping is a form of evidence synthesis with a defined conduct process. Our copy states exactly that: the Summary (p. 12) calls systematic maps 'a common method in evidence synthesis', the Abstract (p. 1) sets out the stages of the mapping process and guidance on 'which type of synthesis is most suitable', and p. 3 notes 'Systematic maps play an important role in evidence syntheses'. The paper nowhere mentions CADIMA itself, but it is not cited for that — it identifies the synthesis type, which it demonstrably is. (LLM, conf=0.85)
- Supportedcites [9] Bragge P, Clavisi O, Turner T, Tavender E, Collie A, Gruen RL. The global evidence mapping initiative: scoping research in broad topic areas. BMC Med Res Methodol. 2011;11:92. source
The citation here identifies 'systematic maps' as a form of evidence synthesis, and our copy is exactly such a methodology paper: Table 1 (Background) defines Evidence Mapping as 'The systematic organisation and illustration of a broad field of research evidence', and the Methods section sets out its core tasks (boundaries/context, search and selection, reporting on yield and study characteristics). The Discussion (p. 2) further frames evidence maps as paralleling 'all steps of a traditional systematic review except in-depth quality appraisal and synthesis', i.e. a secondary-research/synthesis method. The paper also credits Bates et al.'s 'Systematic maps to support the evidence base in social care' (References, item 4), confirming the terminology overlap. The CADIMA-suitability conclusion and the rapid-review citation [26] are the citing authors' own claims and not attributed to this source. (LLM, conf=0.85)
- Not checked yetcites [26] Collins A, Coughlin D, Miller J, Kirk S. The production of quick scoping reviews and rapid evidence assessments: a how to guide. London: Joint Water Evidence Group; 2015.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“software tools have been developed by a diverse set of providers to support review teams during the evidence synthesis process”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Following the completion of the searches, 24 systematic review software packages were identified from across a wide range of disciplines (Table 1).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, two packages did not provide this functionality.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(2) management of search results (including the identification of duplicates)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
21 more claims
Create a free account to see all 124 claims and to verify the 15 references still open.
- Probably not a substantial claim. 82
- It may state two separate claims as one. 38
- The quote only partly states this claim. 7
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Around 160 tools assist the reviewer in either one specific step (during record search [43], screening [39, 44, 45], data extraction [45], risk of bias assessment/ critical appraisal, etc.), or guide the user through several steps or the whole SR process [46–48].”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [48]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“A strength of this review was that it was carried out according to the PRISMA guidelines [32] and using the open access online tool CADIMA [33].”
From Mental health-related telemedicine interventions for pregnant women and new mothers: a systematic literature review (2023) · cites this paper as [33]
Citations
Cites
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Evidence-Based Medicine: A New Approach to Teaching the Practice of Medicine
Guyatt, G. (1992) · JAMA · doi:10.1001/jama.1992.03490170092032 · Cited by 4,202 · cited 1×
“Systematic reviews were first established in the field of healthcare to support evidence-based decision making [1].”
BackgroundEvidence summaries: the evolution of a rapid review approach
Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-10 · Cited by 1,232 · cited 1×
“…can be involved during the respective stages; and – Due to the limitations associated with the conduct of a full systematic review, further evidence synthesis approaches, such as rapid reviews, are evolving in order to save resources and to provide a timely answer to a posed question [26, 30].”
BackgroundA methodology for systematic mapping in environmental sciences
James, K. L., Randall, N. P., & Haddaway, N. R. (2016) · Environmental Evidence · doi:10.1186/s13750-016-0059-6 · Cited by 769
Self-citationAuthority, E. G. F. T. C. O. S. R. E. F. S. (2010) · EFSA Journal · doi:10.2903/j.efsa.2010.1637 · Cited by 693 · cited 1×
“…expanding into other disciplines, including social welfare, international development, education, crime and justice, 1 environmental management 2 (including the impact assessment of crop genetic improvement technologies [2–4]), software engineering [5] and food/feed safety assessment [6].”
BackgroundO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 2×
“As a general observation, many developers appear to have developed these tools without an awareness of the full range of similar tools available (a point also noted in a recent systematic review [27]).”
BackgroundDeploying an interactive machine learning system in an evidence-based practice center: abstrackr
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012) · Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium · doi:10.1145/2110363.2110464 · Cited by 532 · cited 1×
“…of, the systematic review process (e.g. reference management in endnote, duplicate checking using the systematic review accelerator [19, 20], screening in Abstrackr [21], meta-analysis in comprehensive meta-analysis (CMA), or data extraction and quantitative synthesis in RobotReviewer [22]).”
MethodO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Facilitating systematic reviews, data extraction and meta‐analysis with the metagear package for r
Lajeunesse, M. J. (2016) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12472 · Cited by 341
RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317 · cited 1×
“…of, the systematic review process (e.g. reference management in endnote, duplicate checking using the systematic review accelerator [19, 20], screening in Abstrackr [21], meta-analysis in comprehensive meta-analysis (CMA), or data extraction and quantitative synthesis in RobotReviewer [22]).”
Method
Research works printed but not linked: 5
- Practice CfRiEB. The systematic review accelerator. Centre for Research in Evidence Based Practice; 2017.
- Covidence systematic review software. Melbourne, Australia: Veritas Health Innovation.
- Thomas J, Brunton J, Graziosi S. EPPI-reviewer 4: software for research synthesis. EPPI-Centre Software. London: Social Science Research Unit, Institute of Education; 2010.
- Review Manager (RevMan) Version 5.3. Copenhagen: The Nordic Cochrane Centre, The Cochrane Collaboration; 2014.
- Systematic Review and Meta-Analysis Facility (Syrf). Edinburgh, UK: CAMARADES-NC3Rs; 2017.
Cited by
Trends in Educational Research about e-Learning: A Systematic Literature Review (2009–2018)
Valverde-Berrocoso, J., Garrido-Arroyo, M. D. C., Burgos-Videla, C., & Morales-Cevallos, M. B. (2020) · Sustainability · doi:10.3390/su12125153 · Cited by 467
Harrison, H., Griffin, S. J., Kuhn, I., & Usher-Smith, J. A. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-0897-3 · Cited by 449
Artificial intelligence and the conduct of literature reviews
Wagner, G., Lukyanenko, R., & Paré, G. (2022) · Journal of Information Technology · doi:10.1177/02683962211048201 · Cited by 296
“…evidence by surveying previous literature reviews of AI-based tools (e.g., Al-Zubidy et al., 2017; Harrison et al., 2020; Jonnalagadda et al., 2015; Kohl et al., 2018; Marshall and Wallace, 2019; Tsafnat et al., 2014; Van Dinter et al., 2021) and online registries (i.e., www.…”
MethodA systematic review of the energy and climate impacts of teleworking
Hook, A., Court, V., Sovacool, B. K., & Sorrell, S. (2020) · Environmental Research Letters · doi:10.1088/1748-9326/ab8a84 · Cited by 297
“To formulate our search and screening protocols, we followed the guidelines of the Collaboration for Environmental Evidence (Haddaway et al. 2018) and used the free online platform CADIMA to perform the screening phase (Kohl et al. 2018).”
MethodBayesian Networks in Environmental Risk Assessment: A Review
Kaikkonen, L., Parviainen, T., Rahikainen, M., Uusitalo, L., & Lehikoinen, A. (2020) · Integrated Environmental Assessment and Management · doi:10.1002/ieam.4332 · Cited by 265
Metadata
- Authors
- Kohl, C., McIntosh, E. J., Unger, S., Haddaway, N. R., Kecke, S., Schiemann, J., & Wilhelm, R.
- Affiliations
- Julius Kühn-Institut (DE)
- University of Oxford (GB)
- Stockholm Environment Institute (SE)
- Corresponding author
- Kohl, Christian
- Venue
- Environmental Evidence
- Published
- 2018-02-01
- Volume
- 7
- Issue
- 1
- ISSN
- 2047-2382 (ISSN-L)
- OpenAlex
- W2794350945
- Cited by
- 471
- References
- 36
- Citations by year
- 2026: 50 · 2025: 104 · 2024: 98 · 2023: 76 · 2022: 53 · 2021: 45 · 2020: 25 · 2019: 10 · 2018: 10
- FWCI
- 4.78
- Type
- journal article
- Language
- en
- Update notices
- correction (2018-03-27)
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Environmental Evidence (publishedVersion, cc-by)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (acceptedVersion, cc-by-sa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- OpenAgrar (submittedVersion, cc-by)
- Funders
- Seventh Framework Programme (Grant Agreement KBBE-2011-6-311957)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- systematic reviews, systematic map, web-based tool, evidence synthesis, open access software, methodological quality, software comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Online tools supporting the conduct and reporting of systematic reviews and systematic maps: a case study on CADIMA and review of existing tools. https://app.cochat.ai/papers/10.1186/s13750-018-0115-5
- The paper
- Kohl, C., McIntosh, E. J., Unger, S., Haddaway, N. R., Kecke, S., Schiemann, J., & Wilhelm, R. (2018). Online tools supporting the conduct and reporting of systematic reviews and systematic maps: a case study on CADIMA and review of existing tools. Environmental Evidence. https://doi.org/10.1186/s13750-018-0115-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.