Interpreting systematic reviews: are we ready to make our own conclusions? A cross-sectional study
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Independent evaluation of clinical evidence is advocated in evidence-based medicine (EBM). However, authors' conclusions are often appealing for readers who look for quick messages. We assessed how well a group of Malaysian hospital practitioners and medical students derived their own conclusions from systematic reviews (SRs) and to what extent these were influenced by their prior beliefs and the direction of the study results. METHODS: We conducted two cross-sectional studies: one with hospital practitioners (n = 150) attending an EBM course in June 2008 in a tertiary hospital and one with final-year medical students (n = 35) in November 2008. We showed our participants four Cochrane SR abstracts without the authors' conclusions. For each article, the participants chose a conclusion from among six options comprising different combinations of the direction of effect and the strength of the evidence. We predetermined the single option that best reflected the actual authors' conclusions and labelled this as our best conclusion. We compared the participants' choices with our predetermined best conclusions. Two chosen reviews demonstrated that the intervention was beneficial ("positive"), and two others did not ("negative"). We also asked the participants their prior beliefs about the intervention. RESULTS: Overall, 60.3% correctly identified the direction of effect, and 30.1% chose the best conclusions, having identified both the direction of effect and the strength of evidence. More students (48.2%) than practitioners (22.2%) chose the best conclusions (P < 0.001). Fewer than one-half (47%) correctly identified the direction of effect against their prior beliefs. "Positive" SRs were more likely than "negative" SRs to change the participants' beliefs about the effect of the intervention (relative risk (RR) 1.8, 95% confidence interval 1.3 to 2.6) and "convert" those who were previously unsure by making them choose the appropriate direction of effect (RR 1.9, 95% confidence interval 1.3 to 2.8). CONCLUSIONS: The majority of our participants could not generate appropriate conclusions from SRs independently. Judicious direction from the authors' conclusions still appears crucial to guiding our health care practitioners in identifying appropriate messages from research. Authors, editors and reviewers should ensure that the conclusions of a paper accurately reflect the results. Similar studies should be conducted in other settings where awareness and application of EBM are different.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Publication bias in clinical trials due to statistical significance or direction of trial results
Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000006.pub3 · Cited by 898 · cited 2×
“recognised that positive studies (that is, studies that show a beneficial effect of the intervention) are more likely to be published [6,7], cited [8,9] and publicised by the media [10].”
BackgroundResultJadad, A. R., Cook, D. J., Jones, A., Klassen, T. P., Tugwell, P., Moher, M., & Moher, D. (1998) · JAMA · doi:10.1001/jama.280.3.278 · Cited by 534
Song, F., Parekh-Bhurke, S., Hooper, L., Loke, Y. K., Ryder, J. J., Sutton, A. J., Hing, C. B., & Harvey, I. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-79 · Cited by 175
Users' guide to detecting misleading claims in clinical research reports
Montori, V. M., Jaeschke, R., Schünemann, H. J., Bhandari, M., Brozek, J. L., Devereaux, P. J., & Guyatt, G. H. (2004) · BMJ · doi:10.1136/bmj.329.7474.1093 · Cited by 106 · cited 1×
“This attribute is essential to avoid following recommendations by authors who make claims that are either unsupported by their findings or are out of context [3,4].”
Background
Cited by
Yavchitz, A., Ravaud, P., Altman, D. G., Moher, D., Hrobjartsson, A., Lasserson, T., & Boutron, I. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.01.020 · Cited by 224
“[52], participants (hospital practitioners attending an evidence-based medicine course and final-year medical students) were asked to choose an appropriate conclusion for the abstracts of four Cochrane reviews.”
BackgroundPollock, M., Fernandes, R. M., Newton, A. S., Scott, S. D., & Hartling, L. (2019) · Systematic Reviews · doi:10.1186/s13643-018-0768-8 · Cited by 132
Pollock, M., Fernandes, R. M., Newton, A. S., Scott, S. D., & Hartling, L. (2019) · Systematic Reviews · doi:10.1186/s13643-018-0914-3 · Cited by 52
Reporting quality of systematic review abstracts in leading oral implantology journals
Kiriakou, J., Pandis, N., Fleming, P. S., Madianos, P., & Polychronopoulou, A. (2013) · Journal of Dentistry · doi:10.1016/j.jdent.2013.09.006 · Cited by 49
Faggion, C. M., Liu, J., Huda, F., & Atieh, M. (2014) · Journal of Periodontal Research · doi:10.1111/jre.12092 · Cited by 31
Metadata
- Authors
- Lai, N. M., Teng, C. L., & Lee, M. L.
- Venue
- BMC Medicine
- Published
- 2011-03-30
- Volume
- 9
- Issue
- 1
- Pages
- 30-30
- ISSN
- 1741-7015 (ISSN-L)
- OpenAlex
- W2059131063
- Cited by
- 39
- References
- 5
- FWCI
- 0.92
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Interpreting systematic reviews: are we ready to make our own conclusions? A cross-sectional study. https://app.cochat.ai/papers/10.1186/1741-7015-9-30
- The paper
- Lai, N. M., Teng, C. L., & Lee, M. L. (2011). Interpreting systematic reviews: are we ready to make our own conclusions? A cross-sectional study. BMC Medicine. https://doi.org/10.1186/1741-7015-9-30
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.