The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index; 21 of the 45 sources its claims cite were checked for support.
Frequency of data extraction errors and methods to increase data extraction quality: a methodological review
Open accessAbstract
BACKGROUND: Our objective was to assess the frequency of data extraction errors and its potential impact on results in systematic reviews. Furthermore, we evaluated the effect of different extraction methods, reviewer characteristics and reviewer training on error rates and results. METHODS: We performed a systematic review of methodological literature in PubMed, Cochrane methodological registry, and by manual searches (12/2016). Studies were selected by two reviewers independently. Data were extracted in standardized tables by one reviewer and verified by a second. RESULTS: The analysis included six studies; four studies on extraction error frequency, one study comparing different reviewer extraction methods and two studies comparing different reviewer characteristics. We did not find a study on reviewer training. There was a high rate of extraction errors (up to 50%). Errors often had an influence on effect estimates. Different data extraction methods and reviewer characteristics had moderate effect on extraction error rates and effect estimates. CONCLUSION: The evidence base for established standards of data extraction seems weak despite the high prevalence of extraction errors. More comparative studies are needed to get deeper insights into the influence of different extraction methods.
Overview
CoChat Summary
TL;DR
This methodological review found a high prevalence of data extraction errors in systematic reviews, but these errors often had only moderate impact on pooled results or conclusions. Evidence comparing extraction methods and reviewer characteristics is sparse, and no studies on reviewer training were identified.
Key findings
- Across four frequency studies, extraction errors were common: Carroll et al. reported error rates of 8–42% depending on outcome/review, with small pooled effect differences (relative risk range 0.01–0.05); Gøtzsche et al. found at least one erroneous trial in 63% of 27 meta-analyses, and 70% of these also had an error in the pooled effect estimate; Jones et al. found errors in half of 42 Cochrane reviews, approximately 23% of which were misinterpretations, with all data-handling errors changing summary results but none affecting review conclusions.
- Tendal et al. found reviewer agreement of 53% at the trial level and 31% at the meta-analysis level; at the meta-analysis level, the difference in standardized mean difference was at least 1 in one of ten meta-analyses.
- Buscemi et al. compared double independent extraction with one reviewer plus verification: agreement was 28%; the total error rate risk difference was 21.7% (p = 0.019) favoring double extraction; inaccuracy risk difference was 52.9% (p = 0.308); double extraction took 49 min longer; pooled effect estimates did not differ statistically significantly from the reference standard in any of 6 meta-analyses.
- Horton et al. found similar error rates across experience levels (median total errors 24.3–26.4%, inaccuracy 13.6–15.7%, omission 10.0–12.1%; p-values >0.2), though inexperienced reviewers required more time (range 173–211); Tendal et al. found experienced methodologists agreed more than PhD students (61% vs 46% of trials; 33% vs 27% of meta-analyses). No studies on reviewer training were identified.
Methods
- Design: methodological review searching PubMed and the Cochrane Methodology Register (12/2016), Cochrane Colloquium abstracts since 2009, Cochrane oral/poster/workshop presentations since 1994, reference lists, and PubMed related articles.
- Eligibility: studies on frequency of data extraction errors or comparisons of extraction methods, reviewer characteristics, or reviewer training; required a quantitative error measure against a reference standard; intervention reviews in English or German; technical tools were excluded.
- Analysis: screening and data extraction used a semi-structured template; one experienced statistician extracted data and a second experienced epidemiologist verified accuracy; synthesis was narrative. Six studies were included: four on error frequency, one comparing extraction methods, and two comparing reviewer characteristics (Tendal et al. included in both); no training studies were identified.
Limitations
- Stated: no formal risk-of-bias assessment was performed because no established tool exists for methodological studies; reference standards in included studies were imperfect (e.g., Jones et al. used only one statistician; Carroll et al. used their own systematic review as reference); most studies addressed only a certain patient/intervention/comparison question, so generalizability is unclear; unspecific search terms with field limitations may have missed relevant articles; computer-aided extraction approaches were not investigated.
- Evident from the text: only six studies were included, comparative evidence on extraction methods and reviewer characteristics is sparse, and no studies on reviewer training were identified, limiting conclusions about training and method effects.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 8 Not valid · 11 Not checked · 24 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The rate of data extraction errors ranged between 8% and 42% (depending on outcome and review).”
What the sources say
- Disputedcites [16] Carroll C, Scope A, Kaltenthaler E: A case study of binary outcome data extraction across three systematic reviews of hip arthroplasty: errors and differences of selection. BMC research notes. 2013;6:539. sourceExperimental — accuracy not yet measured
The Results distinguish extraction errors from differences caused by data selection. Error rates ranged from 8% to 17% across the outcomes and reviews; 42% was a data-selection figure for the 1-year mortality outcome, not an error rate. (LLM, conf=0.94)
Checked by a reasoning model — may be wrong.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“There is a high prevalence of extraction errors [ 8 , 13 , 15 , 16 ].”
- Also stated in: § Conclusion · plain
“There is a high prevalence of extraction errors.”
What the sources say
- Not checked — no full text of the sourcecites [8] Gøtzsche PC, Hróbjartsson A, Marić K, Tendal B. Data extraction errors in meta-analyses that use standardized mean differences. JAMA. 2007;298(4):430–7. source
- Not checked — no full text of the sourcecites [13] Haywood KL, Hargreaves J, White R, Lamb SE. Reviewing measures of outcome: reliability of data extraction. J Eval Clin Pract. 2004;10:329–37. source
- Disputedcites [15] Tendal B, Higgins JP, Juni P, Hrobjartsson A, Trelle S, Nuesch E, Wandel S, Jorgensen AW, Gesser K, Ilsoe-Kristensen S et al: Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study. In: Bmj. vol. 339; 2009: b3128. sourceExperimental — accuracy not yet measured
The Results (Agreement at trial level; page unavailable) say data extraction errors were less common causes of disagreement than differing choices or trial exclusions. The paper reports substantial disagreement between observers, but does not equate that disagreement with a high prevalence of extraction errors. (LLM, conf=0.84)
Checked by a reasoning model — may be wrong.
- Partly supportedcites [16] Carroll C, Scope A, Kaltenthaler E: A case study of binary outcome data extraction across three systematic reviews of hip arthroplasty: errors and differences of selection. BMC research notes. 2013;6:539. sourceExperimental — accuracy not yet measured
The Background reports that earlier studies found extraction errors could occur frequently, including errors in 20 of 34 reviews. But the Discussion says the incidence was low in this paper’s small case study—usually no more than 8% of the data—and cautions that its findings are not generalisable. The unqualified claim of high prevalence therefore overstates what this source establishes. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Conclusion · plain
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“For example, a study on the influence of searching multiple databases found only a weak impact of reducing the number of searched databases (because of not searching additional databases) on the results of meta-analysis and review conclusions [ 18 , 25 ].”
What the sources say
- Supportedcites [18] Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Med Res Methodol. 2016;16(1):127. source
The Discussion (page not provided) reports that restricting searches to a limited number of databases left most meta-analysis results unchanged; changes in statistical significance were very rare. This supports the claim of a generally small impact on results and conclusions, within the database combinations the study examined. (LLM, conf=0.86)
- Not checked — no full text of the sourcecites [25] Halladay CW, Trikalinos TA, Schmid IT, Schmid CH, Dahabreh IJ. Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions. J Clin Epidemiol. 2015;68(9):1076–84. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“At the level of meta-analysis the difference in standardized mean difference (SMD) was at least 1 in one of ten meta-analyses.”
What the sources say
- Partly supportedcites [15] Tendal B, Higgins JP, Juni P, Hrobjartsson A, Trelle S, Nuesch E, Wandel S, Jorgensen AW, Gesser K, Ilsoe-Kristensen S et al: Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study. In: Bmj. vol. 339; 2009: b3128. sourceExperimental — accuracy not yet measured
The Results section (‘Agreement at meta-analysis level’; page unavailable) reports SMD disagreements of at least 1 in 10% of observer-pair comparisons across the meta-analyses. The claim correctly identifies the size of the disagreement but incorrectly treats that percentage of comparisons as one of the ten meta-analyses. (LLM, conf=0.89)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“These large differences might cause significant difference in effect estimates [ 15 ].”
What the sources say
- Supportedcites [15] Tendal B, Higgins JP, Juni P, Hrobjartsson A, Trelle S, Nuesch E, Wandel S, Jorgensen AW, Gesser K, Ilsoe-Kristensen S et al: Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study. In: Bmj. vol. 339; 2009: b3128. source
The Results (Agreement at meta-analysis level) report that different observers’ calculations changed some pooled effect estimates and, in some cases, whether results were statistically significant. The Discussion describes the disagreements as common and often large. (LLM, conf=0.92)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
27 more claims · 9 with their cited sources checked
Create a free account to see all 54 claims and every source verdict.
- It may state two separate claims as one. 8
- Probably not a substantial claim. 18
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The error rate varied depending on the type and complexity of the data.[3, 4]”
From Artificial Intelligence–Assisted Data Extraction With a Large Language Model: A Study Within Reviews (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A methodology review has revealed a high rate of data extraction errors (up to 63%) in systematic reviews.[3]”
From Artificial Intelligence–Assisted Data Extraction With a Large Language Model: A Study Within Reviews (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite these quality-control measures, the processes can still be prone to error [10].”
From Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models (2024) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This level of performance exceeds the performance seen for human data extraction, where between 8 and 42% of data extraction errors have been observed [10].”
From Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models (2024) · cites this paper as [10]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Empirical evidence suggests that the effect of data extraction error seems to be minor. 3 5”
From Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study (2022) · cites this paper as [3]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Buscemi et al. [ 18 ] compared data extraction of 30 randomized controlled trials by two independent reviewers with data extraction by one reviewer and verification by a second.”
“one study comparing different reviewer extraction methods [ 9 ]”
Citations
Cites
Guidance for conducting systematic scoping reviews
Peters, M. D. J., Godfrey, C. M., Khalil, H., McInerney, P., Parker, D., & Soares, C. B. (2015) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000050 · Cited by 7,472
The Joanna Briggs Institute reviewers' manual 2015: methodology for JBI scoping reviews
Peters, M. D. J., Godfrey, C. M., McInerney, P. A., Soares, C. B., Khalil, H., & Parker, D. (2015) · Cited by 1,135
Expediting systematic reviews: methods and implications of rapid reviews
Ganann, R., Ciliska, D., & Thomas, H. (2010) · Implementation Science · doi:10.1186/1748-5908-5-56 · Cited by 752 · cited 2×
“As the timely preparation and publication of SR is an important goal to support decision making considerations on the balance between resources/time and validity play an important role [7].”
BackgroundSingle data extraction generated more errors than double data extraction in systematic reviews
Buscemi, N., Hartling, L., Vandermeer, B., Tjosvold, L., & Klassen, T. P. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.11.010 · Cited by 464 · cited 5×
“Buscemi 2006 [9] One reviewer verification by a second Two reviewers independently Extraction by one reviewer and verification by an experienced statistician N = 30 (6 meta-analysis)”
BackgroundMethodResultInfluentialZaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464
Automating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287 · cited 1×
“Biomedical natural language processing techniques will be developed further in the near future [27].”
BackgroundData Extraction Errors in Meta-analyses That Use Standardized Mean Differences
Gøtzsche, P. C., Hróbjartsson, A., Marić, K., & Tendal, B. (2007) · JAMA · doi:10.1001/jama.298.4.430 · Cited by 238 · cited 8×
“[8] replicated the results of 27 metaanalyses.”
BackgroundInfluentialBias in meta-analytic research
Felson, D. T. (1992) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(92)90072-u · Cited by 231 · cited 1×
“unclear inclusion criteria), in the data collection process and in the validity assessment of included studies [2].”
BackgroundPublishing Protocols of Systematic Reviews: Comparing What Was Done to What Was Planned
Silagy, C. A., Middleton, P., & Hopewell, S. (2002) · JAMA · doi:10.1001/jama.287.21.2831 · Cited by 188 · cited 1×
“time points, measures) and omission would be reduced [15, 22].”
BackgroundShemilt, I., Khan, N., Park, S., & Thomas, J. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0315-4 · Cited by 168 · cited 1×
“liberal acceleration) are more cost-effective than independent study selection [6].”
Background
Research works printed but not linked: 2
- Joanna Briggs Institute. Reviewers’ manual: 2011 edition. Adelaide: JBI; 2014.
- Morton S, Berg A, Levit L, Eden J. Finding what works in health care: standards for systematic reviews.National Academies Press; 2011.
Cited by
Gates, M., Gates, A., Pieper, D., Fernandes, R. M., Tricco, A. C., Moher, D., Brennan, S. E., Li, T., Pollock, M., Lunny, C., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070849 · Cited by 856
Self-citationWaffenschmidt, S., Knelangen, M., Sieben, W., Bühn, S., & Pieper, D. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0782-0 · Cited by 626
Self-citationLunny, C., Pieper, D., Thabet, P., & Kanji, S. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01269-y · Cited by 252
Self-citationKusmaryono, I., Wijayanti, D., & Maharani, H. R. (2022) · International Journal of Educational Methodology · doi:10.12973/ijem.8.4.625 · Cited by 283
“Then, the researcher changed the data obtained into new data by filtering the data into several categories (Mathes et al., 2017; Munn et al., 2014; Pedder et al., 2016; Schmidt et al., 2021).”
InfluentialMenopausal hormone therapy and women’s health: An umbrella review
Zhang, G.-Q., Chen, J.-L., Luo, Y., Mathur, M. B., Anagnostis, P., Nurmatov, U., Talibov, M., Zhang, J., Hawrylowicz, C. M., Lumsden, M. A., et al. (2021) · PLOS Medicine · doi:10.1371/journal.pmed.1003731 · Cited by 208
“…(e.g., incomplete data, poor data quality) in a sample of included meta-analyses [25–28] as well as the high prevalence of data extraction errors from empirical evaluations [29], we devoted extensive efforts to obtaining and validating the data for the individual studies in each meta-analysis.”
Method
Metadata
- Authors
- Mathes, T., Klaßen, P., & Pieper, D.
- Affiliations
- Witten/Herdecke University (DE)
- Corresponding author
- Mathes, Tim
- Venue
- BMC Medical Research Methodology
- Published
- 2017-11-28
- Volume
- 17
- Issue
- 1
- Pages
- 152-152
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC5704562
- OpenAlex
- W2770117783
- Semantic Scholar
- 2e568c5ee173d547bfe6467fd01213750559649e
- Cited by
- 240
- References
- 24
- Citations by year
- 2026: 40 · 2025: 32 · 2024: 23 · 2023: 27 · 2022: 24 · 2021: 41 · 2020: 18 · 2019: 14 · 2018: 5
- FWCI
- 7.84
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (submittedVersion, cc-by)
- PubMed Central (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- GoeScholar The Publication Server of the Georg-August-Universität Göttingen (Georg-August-Universität Göttingen) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Social Sciences › Education › Academic integrity and plagiarism
- Keywords
- data extraction errors, extraction methods, reviewer characteristics, reviewer training, error rate, effect estimates
- MeSH terms
- Data Accuracy, Humans, Research Design, Review Literature as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. https://app.cochat.ai/papers/10.1186/s12874-017-0431-4
- The paper
- Mathes, T., Klaßen, P., & Pieper, D. (2017). Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-017-0431-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
