Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index; 21 of the 45 sources its claims cite were checked for support.

Frequency of data extraction errors and methods to increase data extraction quality: a methodological review

Open access
Mathes, Tim, Klaßen, Pauline, Pieper, Dawid · 2017 · BMC Medical Research Methodology

Abstract

BACKGROUND: Our objective was to assess the frequency of data extraction errors and its potential impact on results in systematic reviews. Furthermore, we evaluated the effect of different extraction methods, reviewer characteristics and reviewer training on error rates and results. METHODS: We performed a systematic review of methodological literature in PubMed, Cochrane methodological registry, and by manual searches (12/2016). Studies were selected by two reviewers independently. Data were extracted in standardized tables by one reviewer and verified by a second. RESULTS: The analysis included six studies; four studies on extraction error frequency, one study comparing different reviewer extraction methods and two studies comparing different reviewer characteristics. We did not find a study on reviewer training. There was a high rate of extraction errors (up to 50%). Errors often had an influence on effect estimates. Different data extraction methods and reviewer characteristics had moderate effect on extraction error rates and effect estimates. CONCLUSION: The evidence base for established standards of data extraction seems weak despite the high prevalence of extraction errors. More comparative studies are needed to get deeper insights into the influence of different extraction methods.

Overview

CoChat Summary

TL;DR

This methodological review found a high prevalence of data extraction errors in systematic reviews, but these errors often had only moderate impact on pooled results or conclusions. Evidence comparing extraction methods and reviewer characteristics is sparse, and no studies on reviewer training were identified.

Key findings

  • Across four frequency studies, extraction errors were common: Carroll et al. reported error rates of 8–42% depending on outcome/review, with small pooled effect differences (relative risk range 0.01–0.05); Gøtzsche et al. found at least one erroneous trial in 63% of 27 meta-analyses, and 70% of these also had an error in the pooled effect estimate; Jones et al. found errors in half of 42 Cochrane reviews, approximately 23% of which were misinterpretations, with all data-handling errors changing summary results but none affecting review conclusions.
  • Tendal et al. found reviewer agreement of 53% at the trial level and 31% at the meta-analysis level; at the meta-analysis level, the difference in standardized mean difference was at least 1 in one of ten meta-analyses.
  • Buscemi et al. compared double independent extraction with one reviewer plus verification: agreement was 28%; the total error rate risk difference was 21.7% (p = 0.019) favoring double extraction; inaccuracy risk difference was 52.9% (p = 0.308); double extraction took 49 min longer; pooled effect estimates did not differ statistically significantly from the reference standard in any of 6 meta-analyses.
  • Horton et al. found similar error rates across experience levels (median total errors 24.3–26.4%, inaccuracy 13.6–15.7%, omission 10.0–12.1%; p-values >0.2), though inexperienced reviewers required more time (range 173–211); Tendal et al. found experienced methodologists agreed more than PhD students (61% vs 46% of trials; 33% vs 27% of meta-analyses). No studies on reviewer training were identified.

Methods

  • Design: methodological review searching PubMed and the Cochrane Methodology Register (12/2016), Cochrane Colloquium abstracts since 2009, Cochrane oral/poster/workshop presentations since 1994, reference lists, and PubMed related articles.
  • Eligibility: studies on frequency of data extraction errors or comparisons of extraction methods, reviewer characteristics, or reviewer training; required a quantitative error measure against a reference standard; intervention reviews in English or German; technical tools were excluded.
  • Analysis: screening and data extraction used a semi-structured template; one experienced statistician extracted data and a second experienced epidemiologist verified accuracy; synthesis was narrative. Six studies were included: four on error frequency, one comparing extraction methods, and two comparing reviewer characteristics (Tendal et al. included in both); no training studies were identified.

Limitations

  • Stated: no formal risk-of-bias assessment was performed because no established tool exists for methodological studies; reference standards in included studies were imperfect (e.g., Jones et al. used only one statistician; Carroll et al. used their own systematic review as reference); most studies addressed only a certain patient/intervention/comparison question, so generalizability is unclear; unspecific search terms with field limitations may have missed relevant articles; computer-aided extraction approaches were not investigated.
  • Evident from the text: only six studies were included, comparative evidence on extraction methods and reviewer characteristics is sparse, and no studies on reviewer training were identified, limiting conclusions about training and method effects.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s12874-017-0431-4

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Flow-diagram of study selection
Fig. 1 Flow-diagram of study selectionCC BY · PubMed Central

Verification

Claims the paper makes

11 Valid · 8 Not valid · 11 Not checked · 24 Own results and interpretations

27 more claims · 9 with their cited sources checked

Create a free account to see all 54 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 8
  • Probably not a substantial claim. 18

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

4 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 24 references

Research works printed but not linked: 2

  1. Joanna Briggs Institute. Reviewers’ manual: 2011 edition. Adelaide: JBI; 2014.
  2. Morton S, Berg A, Levit L, Eden J. Finding what works in health care: standards for systematic reviews.National Academies Press; 2011.

Cited by

See all 240

Metadata

Authors
Mathes, T., Klaßen, P., & Pieper, D.
Affiliations
Witten/Herdecke University (DE)
Corresponding author
Mathes, Tim
Venue
BMC Medical Research Methodology
Published
2017-11-28
Volume
17
Issue
1
Pages
152-152
ISSN
1471-2288 (ISSN-L)
OpenAlex
W2770117783
Cited by
240
References
24
Citations by year
2026: 40 · 2025: 32 · 2024: 23 · 2023: 27 · 2022: 24 · 2021: 41 · 2020: 18 · 2019: 14 · 2018: 5
FWCI
7.84
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • BMC Medical Research Methodology (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (submittedVersion, cc-by)
  • PubMed Central (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • GoeScholar The Publication Server of the Georg-August-Universität Göttingen (Georg-August-Universität Göttingen) (submittedVersion, cc-by)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Psychology › Social Psychology › Mental Health via Writing
  • Social Sciences › Education › Academic integrity and plagiarism
Keywords
data extraction errors, extraction methods, reviewer characteristics, reviewer training, error rate, effect estimates
MeSH terms
Data Accuracy, Humans, Research Design, Review Literature as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. https://app.cochat.ai/papers/10.1186/s12874-017-0431-4
The paper
Mathes, T., Klaßen, P., & Pieper, D. (2017). Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-017-0431-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.