Cited by 74Meta-analysis and systematic reviewsStatistical Methods in EpidemiologyReliability and Agreement in Measurement

A newly developed tool for classifying study designs in systematic reviews of interventions and exposures showed substantial reliability and validity

Seo, Hyun-Ju, Kim, Soo Young, Lee, Yoon Jae, Jang, Bo-Hyoung, Park, Ji-Eun, Sheen, Seung-Soo, Hahn, Seo Kyung·2016·Journal of Clinical Epidemiology

Abstract

Objective To develop a study Design Algorithm for Medical Literature on Intervention (DAMI) and test its interrater reliability, construct validity, and ease of use.Study design and setting We developed and then revised the DAMI to include detailed instructions. To test the DAMI's reliability, we used a purposive sample of 134 primary, mainly nonrandomized studies. We then compared the study designs as classified by the original authors and through the DAMI. Unweighted kappa statistics were computed to test interrater reliability and construct validity based on the level of agreement between the original and DAMI classifications. Assessment time was also recorded to evaluate ease of use.Results The DAMI includes 13 study designs, including experimental and observational studies of interventions and exposure. Both the interrater reliability (unweighted kappa = 0.67; 95% CI [0.64-0.75]) and construct validity (unweighted kappa = 0.63, 95% CI [0.52-0.67]) were substantial. Mean classification time using the DAMI was 4.08 ± 2.44 minutes (range, 0.51-10.92).Conclusions The DAMI showed substantial interrater reliability and construct validity. Furthermore, given its ease of use, it could be used to accurately classify medical literature for systematic reviews of interventions although minimizing disagreement between authors of such reviews.

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2015.09.013

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Cites

1 retracted

Flagged first, then by citations

3 of the 33 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Seo, H.-J., Kim, S. Y., Lee, Y. J., Jang, B.-H., Park, J.-E., Sheen, S.-S., & Hahn, S. K.
Affiliations
  • Chosun University (KR)
  • Hallym University (KR)
  • CHA University Bundang Medical Center (KR)
  • Kyung Hee University (KR)
  • Seoul National University (KR)
  • Ajou University Hospital (KR)
Corresponding author
Kim, Soo Young
Venue
Journal of Clinical Epidemiology
Publisher
Elsevier BV
Published
2016-02
Volume
70
Pages
200-205
ISSN
0895-4356 (ISSN-L)
OpenAlex
W2100365148
Cited by
74
References
33
Citations by year
2026: 3 · 2025: 4 · 2024: 6 · 2023: 6 · 2022: 9 · 2021: 5 · 2020: 16 · 2019: 10 · 2018: 8 · 2017: 5 · 2016: 2
FWCI
0.82
Type
journal article
Language
en
Open access
closed
Locations
  • Journal of Clinical Epidemiology (publishedVersion)
  • PubMed (publishedVersion)
  • AJOU Open Repository (Ajou University) (submittedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Keywords
study design classification, inter-rater reliability, construct validity, non-randomized studies, classification time
MeSH terms
Algorithms, Humans, Quality Control, Research Design, Review Literature as Topic, Reproducibility of Results, Peer Review, Research, Evidence-Based Medicine, Consensus

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review