External Validity and Model Validity: A Conceptual Approach for Systematic Review Methodology
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Background. Evidence rankings do not consider equally internal (IV), external (EV), and model validity (MV) for clinical studies including complementary and alternative medicine/integrative health care (CAM/IHC) research. This paper describe this model and offers an EV assessment tool (EVAT©) for weighing studies according to EV and MV in addition to IV. Methods. An abbreviated systematic review methodology was employed to search, assemble, and evaluate the literature that has been published on EV/MV criteria. Standard databases were searched for keywords relating to EV, MV, and bias‐scoring from inception to Jan 2013. Tools identified and concepts described were pooled to assemble a robust tool for evaluating these quality criteria. Results. This study assembled a streamlined, objective tool to incorporate for the evaluation of quality of EV/MV research that is more sensitive to CAM/IHC research. Conclusion. Improved reporting on EV can help produce and provide information that will help guide policy makers, public health researchers, and other scientists in their selection, development, and improvement in their research‐tested intervention. Overall, clinical studies with high EV have the potential to provide the most useful information about “real‐world” consequences of health interventions. It is hoped that this novel tool which considers IV, EV, and MV on equal footing will better guide clinical decision making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-2 · Cited by 2,755
External validity of randomised controlled trials: “To whom do the results of this trial apply?”
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(04)17670-8 · Cited by 2,544 · cited 1×
“With that said, it is staggering how often external validity is neglected in the methodological considerations of health care research [11, 13, 14].”
BackgroundMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2012) · International Journal of Surgery · doi:10.1016/j.ijsu.2011.10.001 · Cited by 2,345
Rates of hyperkalemia after publication of the Randomized Aldactone Evaluation Study
Juurlink, D. N., Mamdani, M. M., Lee, D. S., Kopp, A., Austin, P. C., Laupacis, A., & Redelmeier, D. A. (2004) · New England Journal of Medicine · doi:10.1056/nejmoa040135 · Cited by 1,761
Armijo‐Olivo, S., Stiles, C. R., Hagen, N. A., Biondo, P. D., & Cummings, G. G. (2012) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2010.01516.x · Cited by 1,539
Moher, D., Jadad, A. R., Nichol, G., Penman, M., Tugwell, P., & Walsh, S. (1995) · Controlled Clinical Trials · doi:10.1016/0197-2456(94)00031-w · Cited by 1,308 · cited 1×
“Because external validity depends on a source population (a.k.a. target population), the first step in the assessment of the external validity is to define this source population [11].”
BackgroundAssessing the Quality of Randomized Controlled Trials:Current Issues and Future Directions
Moher, D., Jadad, A. R., & Tugwell, P. (1996) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300009570 · Cited by 436
MacLehose, Reeves, Harvey, Sheldon, Russell, & Black (2000) · Health Technology Assessment · doi:10.3310/hta4340 · Cited by 311
How to assess the external validity of therapeutic trials: a conceptual approach
Dekkers, O. M., Elm, E. V., Algra, A., Romijn, J. A., & Vandenbroucke, J. P. (2010) · International Journal of Epidemiology · doi:10.1093/ije/dyp174 · Cited by 219
Cited by
Hong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107
Scale development: ten main limitations and recommendations to improve future research practices
Morgado, F. F. R., Meireles, J. F. F., Neves, C. M., Amaral, A. C. S., & Ferreira, M. E. C. (2018) · Psicologia: Reflexão e Crítica · doi:10.1186/s41155-016-0057-1 · Cited by 780
Crawford, C., Boyd, C., Paat, C. F., Price, A., Xenakis, L., Yang, E., Zhang, W., & the Evidence for Massage Therapy (EMT) Working Group (2016) · Pain Medicine · doi:10.1093/pm/pnw099 · Cited by 178
Self-citationA systematic literature review of Chinese music education studies during 2007 to 2019
Yang, Y., & Welch, G. (2023) · International Journal of Music Education · doi:10.1177/02557614221096150 · Cited by 108
“It would be a risky practice to form a comprehensive review without further consideration of external contexts (Khorsan & Crawford, 2014), such as education system design, values and developmental phases.”
BackgroundRe‐inventing the randomized controlled trial in medical oncology: The registry‐based trial
Foroughi, S., Wong, H., Gately, L., Lee, M., Simons, K., Tie, J., Burgess, A. W., & Gibbs, P. (2018) · Asia-Pacific Journal of Clinical Oncology · doi:10.1111/ajco.12992 · Cited by 43
Metadata
- Authors
- Khorsan, R., & Crawford, C.
- Venue
- Evidence-Based Complementary and Alternative Medicine
- Publisher
- Wiley
- Published
- 2014-01
- Volume
- 2014
- Issue
- 1
- Pages
- 694804-694804
- ISSN
- 1741-427X (ISSN-L)
- 1741-4288
- OpenAlex
- W2091267779
- Cited by
- 95
- References
- 11
- FWCI
- 1.62
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.