Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: To contribute to making searching for Technology Assessment Reports (TARs) more cost-effective by suggesting an optimum literature retrieval strategy. DATA SOURCES: A sample of 20 recent TARs. REVIEW METHODS: All sources used to search for clinical and cost-effectiveness studies were recorded. In addition, all studies that were included in the clinical and cost-effectiveness sections of the TARs were identified, and their characteristics recorded, including author, journal, year, study design, study size and quality score. Each was also classified by publication type, and then checked to see whether it was indexed in the following databases: MEDLINE, EMBASE, and then either the Cochrane Controlled Trials Register (CCTR) for clinical effectiveness studies or the NHS Economic Evaluation Database (NHS EED) for the cost-effectiveness studies. Any study not found in at least one of these databases was checked to see whether it was indexed in the Science Citation Index (SCI) and BIOSIS, and the American Society of Clinical Oncology (ASCO) Online if a cancer review. Any studies still not found were checked to see whether they were in a number of additional databases. RESULTS: The median number of sources searched per TAR was 20, and the range was from 13 to 33 sources. Six sources (CCTR, DARE, EMBASE, MEDLINE, NHS EED and sponsor/industry submissions to National Institute for Clinical Excellence) were used in all reviews. After searching the MEDLINE, EMBASE and NHS EED databases, 87.3% of the clinical effectiveness studies and 94.8% of the cost-effectiveness studies were found, rising to 98.2% when SCI, BIOSIS and ASCO Online and 97.9% when SCI and ASCO Online, respectively, were added. The median number of sources searched for the 14 TARs that included an economic model was 9.0 per TAR. A sensitive search filter for identifying non-randomised controlled trials (RCT), constructed for MEDLINE and using the search terms from the bibliographic records in the included studies, retrieved only 85% of the known sample. Therefore, it is recommended that when searching for non-RCT studies a search is done for the intervention alone, and records are then scanned manually for those that look relevant. CONCLUSIONS: Searching additional databases beyond the Cochrane Library (which includes CCTR, NHS EED and the HTA database), MEDLINE, EMBASE and SCI, plus BIOSIS limited to meeting abstracts only, was seldom found to be effective in retrieving additional studies for inclusion in the clinical and cost-effectiveness sections of TARs (apart from reviews of cancer therapies, where a search of the ASCO database is recommended). A more selective approach to database searching would suffice in most cases and would save resources, thereby making the TAR process more efficient. However, searching non-database sources (including submissions from manufacturers, recent meeting abstracts, contact with experts and checking reference lists) does appear to be a productive way of identifying further studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Some researchers agree that when timeliness is of importance, hand searching of reference lists and contacting experts can be more effective than comprehensive bibliographic database searches [45, 46].”
From How to conduct systematic reviews more expeditiously? (2015) · cites this paper as [45]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
Consensus development methods, and their use in clinical guideline development.
Murphy, Black, Lamping, McKee, Sanderson, Askham, & Marteau (1998) · Health Technology Assessment · doi:10.3310/hta2030 · Cited by 2,051
Guidelines for authors and peer reviewers of economic submissions to the BMJ
Drummond, M. F., & Jefferson, T. O. (1996) · BMJ · doi:10.1136/bmj.313.7052.275 · Cited by 1,827
Evaluating patient-based outcome measures for use in clinical trials.
Fitzpatrick, Davey, Buxton, & Jones (1998) · Health Technology Assessment · doi:10.3310/hta2140 · Cited by 1,715
Egger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181
Full publication of results initially presented in abstracts.
Scherer, R. W., Langenberg, P., & von Elm, E. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000005.pub3 · Cited by 997
Direction and impact of language bias in meta-analyses of controlled trials: empirical study
Jüni, P., Holenstein, F., Sterne, J., Bartlett, C., & Egger, M. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.115 · Cited by 892
The Medical Review Article: State of the Science
MULROW, C. D. (1987) · Annals of Internal Medicine · doi:10.7326/0003-4819-106-3-485 · Cited by 856
Royle, P., & Milne, R. (2003) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462303000552 · Cited by 693
Self-citationJadad, A. R., Cook, D. J., Jones, A., Klassen, T. P., Tugwell, P., Moher, M., & Moher, D. (1998) · JAMA · doi:10.1001/jama.280.3.278 · Cited by 534
Publication and Related Biases
Song, F., Eastwood, A., Gilbody, S., Duley, L., & Sutton, A. (2001) · The Advanced Handbook of Methods in Evidence Based Healthcare · doi:10.4135/9781848608344.n21 · Cited by 455
Antimicrobial prophylaxis in colorectal surgery: a systematic review of randomized controlled trials
Song, F., & Glenny, A.-M. (1998) · Journal of British Surgery · doi:10.1046/j.1365-2168.1998.00883.x · Cited by 372
Should Unpublished Data Be Included in Meta-analyses?: Current Convictions and Controversies
Cook, D. J. (1993) · JAMA · doi:10.1001/jama.1993.03500210049030 · Cited by 259
MacLeod, Grant, Donaldson, Khan, Campbell, Daly, Lawrence, Wallace, Vale, Cody, et al. (1998) · Health Technology Assessment · doi:10.3310/hta2050 · Cited by 254
Newborn screening for inborn errors of metabolism: a systematic review.
Seymour, Thomason, Chalmers, Addison, Bain, Cockburn, Littlejohns, Lord, & Wilcox (1997) · Health Technology Assessment · doi:10.3310/hta1110 · Cited by 234
Systematic reviews in health care a practical guide
Glasziou, P. (2001) · Cited by 230
Problems and Limitations in Conducting Systematic Reviews
Egger, M., Dickersin, K., & Smith, G. D. (2001) · Systematic Reviews in Health Care · doi:10.1002/9780470693926.ch3 · Cited by 202
Location bias in controlled clinical trials of complementary/alternative therapies
Pittler, M. H., Abbot, N. C., Harkness, E. F., & Ernst, E. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00220-6 · Cited by 186
Identifying Clinical Trials in the Medical Literature with Electronic Databases
Suarez-Almazor, M. E., Belseck, E., Homik, J., Dorgan, M., & Ramos-Remus, C. (2000) · Controlled Clinical Trials · doi:10.1016/s0197-2456(00)00067-2 · Cited by 180
Egger, M., Ebrahim, S., & Smith, G. D. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.1 · Cited by 160
McQuay, & Moore (1998) · Health Technology Assessment · doi:10.3310/hta2120 · Cited by 131
Systematic review of endoscopic ultrasound in gastro-oesophageal cancer.
Harris, Kelly, Berry, Hutton, Roderick, Cullingworth, Gathercole, O'Connor, Boyce, & Smith (1998) · Health Technology Assessment · doi:10.3310/hta2180 · Cited by 121
Mowatt, Bower, Brebner, Cairns, Grant, & McKee (1997) · Health Technology Assessment · doi:10.3310/hta1140 · Cited by 112
Lefebvre, C., & Clarke, M. J. (2001) · Systematic Reviews in Health Care · doi:10.1002/9780470693926.ch4 · Cited by 109
BEYOND MEDLINE: Reducing Bias Through Extended Systematic Review Search
Savoie, I., Helmer, D., Green, C. J., & Kazanjian, A. (2003) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462303000163 · Cited by 105
Consolation for the Scientist: Sometimes it is Hard to Publish Papers that are Later Highly-Cited
Campanario, M. A. G. (1993) · Social Studies of Science · doi:10.1177/030631293023002005 · Cited by 94
Avenell, A., Handoll, H. H., & Grant, A. (2001) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/73.3.505 · Cited by 89
Sassi, F., Archard, L., & McDaid, D. (2002) · Medical Care · doi:10.1097/00005650-200205000-00004 · Cited by 73
Sampson, M., Campbell, K., Ajiferuke, I., & Moher, D. (2003) · BMC Pediatrics · doi:10.1186/1471-2431-3-1 · Cited by 69
Brandon/Hill selected list of books and journals for the small medical library.
Hill, D. R., & Stickell, H. N. (1999) · Europe PMC (PubMed Central) · Cited by 69
Song, & Glenny (1998) · Health Technology Assessment · doi:10.3310/hta2070 · Cited by 67
Topfer, L.-A., Parada, A., Menon, D., Noorani, H., Perras, C., & Serra-Prat, M. (1999) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462399015226 · Cited by 59
Efficiency of Searching the Grey Literature in Palliative Care
Cook, A. M., Finlay, I. G., Edwards, A. G. K., Hood, K., Higginson, I. J., Goodwin, D. M., Normand, C. E., & Douglas, H.-R. (2001) · Journal of Pain and Symptom Management · doi:10.1016/s0885-3924(01)00315-3 · Cited by 54
How useful are unpublished data from the Food and Drug Administration in meta-analysis?
MacLean, C. H., Morton, S. C., Ofman, J. J., Röth, E., & Shekelle, P. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(02)00520-6 · Cited by 54
Spier, R. E. (2002) · Science and Engineering Ethics · doi:10.1007/s11948-002-0035-0 · Cited by 47
Best, L., Stevens, A., & Colin‐Jones, D. (1997) · Journal of Clinical Effectiveness · doi:10.1108/eb020865 · Cited by 35
The Cochrane Collaboration in the 20th Century
Antes, G., & Oxman, A. D. (2001) · Systematic Reviews in Health Care · doi:10.1002/9780470693926.ch25 · Cited by 20
Research works printed but not linked: 7
- [16]16. Lee KP , Schotland M, Bacchetti P , Bero LA. Association of journal quality indicators with methodological quality of clinical research articles. JAMA 2002; 287 :2805-8.
- [18]18. Hopewell S, McDonald S, Clarke M, Egger M. Grey literature in meta-analyses of randomized trials of health care interventions (Cochrane Methodology Review). In The Cochrane Library (Issue 1). Oxford: Update Software; 2003.
- [22]22. Jefferson TO, Alderson P , Davidoff F, Wager E. Editorial peer-review for improving the quality of reports of biomedical studies (Cochrane Methodology Review). In The Cochrane Library (Issue 1). Oxford: Update Software; 2003.
- [36]36. Eady A. Secondary publications: economic analyses. In McKibbon A, editor. PDQ evidence-based principles and practice. Hamilton: Decker; 1999. pp. 173-86.
- [38]38. Clarke M, Oxman AD, editors. Locating and selecting studies. Cochrane Reviewers Handbook 4.1.6 (updated January 2003) Section 5. In The Cochrane Library (Issue 2). Oxford: Update Software; 2003.
- [40]40. Royle P . Literature searching for randomised controlled trials used in Cochrane reviews: rapid versus exhaustive searches [dissertation]. Southampton: University of Southampton; 2001.
- [41]41. Hopewell S, Clarke M, Stewart L, Tierney J. Time to publication for results of clinical trials (Cochrane Methodology Review). In The Cochrane Library (Issue 1). Oxford: Update Software; 2003.
Cited by
Eddy, D. M., Hollingworth, W., Caro, J. J., Tsevat, J., McDonald, K. M., & Wong, J. B. (2012) · Medical Decision Making · doi:10.1177/0272989x12454579 · Cited by 799
Expediting systematic reviews: methods and implications of rapid reviews
Ganann, R., Ciliska, D., & Thomas, H. (2010) · Implementation Science · doi:10.1186/1748-5908-5-56 · Cited by 752
Philips, Z., Ginnelly, L., Sculpher, M., Claxton, K., Golder, S., Riemsma, R., Woolacoot, N., & Glanville, J. (2004) · Health Technology Assessment · doi:10.3310/hta8360 · Cited by 697
Hobart, J., & Cano, S. (2009) · Health Technology Assessment · doi:10.3310/hta13120 · Cited by 621
Moher, D., Pham, B., Lawson, M., & Klassen, T. (2003) · Health Technology Assessment · doi:10.3310/hta7410 · Cited by 500
Metadata
- Authors
- Royle, P., & Waugh, N.
- Affiliations
- University of Aberdeen (GB)
- ORCID
- Waugh, N
- Venue
- Health Technology Assessment
- Published
- 2003-11
- Volume
- 7
- Issue
- 34
- Pages
- iii, ix-x, 1
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W2119297205
- Cited by
- 194
- References
- 37
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 1 · 2023: 1 · 2022: 4 · 2021: 6 · 2020: 4 · 2019: 3 · 2018: 4 · 2017: 8 · 2016: 12 · 2015: 9 · 2014: 3 · 2013: 13 · 2012: 5
- FWCI
- 13.24
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- University of Warwick (cc-by)
- Funders
- Health Technology Assessment programme (02/30/01)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- literature searching, National Institute for Clinical Excellence, database indexing, MEDLINE, Embase, search filters, economic models, bibliographic databases, BIOSIS
- MeSH terms
- Abstracting and Indexing, Cost-Benefit Analysis, United Kingdom, Medical Informatics, Research Design, Technology Assessment, Biomedical, Randomized Controlled Trials as Topic, Databases, Bibliographic, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.