Issues in methodological research: perspectives from researchers and commissioners
Open accessAbstract
OBJECTIVES: (1) Methodological research has few well-defined tools and processes analogous to those available for reviews and data collection in substantive health technology assessment. (2) This project was set up to obtain researchers' and others' views on the innovative projects on research methodology under the NHS Health Technology Assessment Programme and the usefulness of the research. (3) The study was intended to span both epistemological and management issues. (4) The following issues were explored: (a) the degree to which researchers would feel constrained by the "Cochrane" approach to systematic reviews when undertaking reviews of a methodological nature; (b) whether methodological projects may require exceptional design and management arrangements, in view of their novelty, subjectivity and complexity; (c) whether researchers would seek out other methods, in addition to undertaking reviews of argument, as a means of extending their understanding of methodological issues (there may be three categories of research methods in methodology: reviews of methodological argument, studies that use the literature as a source of data, and research that collects new primary data); (d) whether the Methodology Programme overall can be considered a "success". METHODS: (1) Telephone interviews were carried out on researchers (one senior and one junior per project), resulting in 35 interviews from 19 of the 20 target projects. (2) A qualitative postal survey was sent to 12 people who had played a key role in the development of the Methodology Programme; replies were received from six of them. (3) Analysis was undertaken of the hit rates for 29 projects on the NCCHTA website by the end of February and the end of May 1999, comparing those concerned with methodology (n = 10) and those concerned with other issues (n = 19). RESULTS: UNDERTAKING METHODOLOGICAL RESEARCH: VIEWS OF RESEARCHERS: This section summarises the views of 35 researchers who were interviewed by telephone. RESULTS: UNDERTAKING METHODOLOGICAL RESEARCH: VIEWS OF RESEARCHERS: (THE NATURE OF METHODOLOGICAL REVIEWS): (1) There was a reluctance among researchers to use the term "systematic review" in the methodological context. (2) Practical problems in undertaking methodological reviews were found at every stage of the research process. (a) In the initial search stage, preplanned strategies were difficult to maintain, owing to the need to respond to the problems of too few or too many references. (b) At the analysis stage, most studies were not formally weighted, but there was implicit weighting in researchers' views of their merits or relevance. (c) It was often only at the synthesis stage that researchers could see clearly what their study was able to do; iteration was frequently necessary at this point. (d) It was difficult to form simple conclusions and recommendations beyond summaries of what was known in the field. (e) Dissemination activities were most often directed to other health service researchers, with some attention to NHS policy makers and research commissioners. RESULTS: UNDERTAKING METHODOLOGICAL RESEARCH: VIEWS OF RESEARCHERS (THE NEED FOR FLEXIBILITY): (1) Few researchers had amended their topic or methods once their research was under way, although some had made minor changes to their original plan, generally to refine the topic to fit the time or data available. (2) Changing a topic was seen as inappropriate unless checked with funders, but changes in research methods were viewed as reasonable because questions might be refined in the light of information gained or early thinking. RESULTS: UNDERTAKING METHODOLOGICAL RESEARCH: VIEWS OF RESEARCHERS (THE QUESTION OF BIAS): (1) Few researchers considered that this kind of research could be undertaken or presented in a wholly unbiased way because of the need to assess the research studied. (2) Objectivity was nonetheless seen as something that researchers should strive towards. Efforts to do so included presenting data clearly, separating findings from discussion, covering all points of view, setting out their own assumptions and values, and testing their ideas on others known to have differing views. (3) The formal peer-review process was not seen to have made a difference here, primarily because of the stage at which referees become involved. RESULTS: UNDERTAKING METHODOLOGICAL RESEARCH: VIEWS OF RESEARCHERS (PROJECT MANAGEMENT--TIMING AND TIME MANAGEMENT): (1) A majority of projects were completed within 3 months of their due date. Those studies completed roughly on time were considered to have efficient junior researchers and good project management, including clear deadlines for different stages of the research. (2) Some studies had severe problems of time management. Too much time tended to be spent on collecting and reading the literature and the writing stage was not always well planned. Referees' comments were also slow in coming. (ABSTRACT TRUNCATED)
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The literature search was based on the principle of theoretical saturation.
“We based our literature search on the principle of theoretical saturation [17, 18], with the aim of identifying all relevant recommendations, rather than all relevant articles.”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [17]
Theoretical saturation suggests that identifying every article is not required in methodological research.
“Theoretical saturation suggests that identifying every article is not required in methodological research.”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [17]
Once a set of methodological articles covering all relevant methodological aspects is identified, additional searches add little to the ideas already present in the initial set of articles.
“Once a set of methodological articles covering all relevant methodological aspects is identified, additional searches add little to the ideas already present in the initial set of articles.”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [17]
The search was guided by the methodological principle of theoretical saturation.
“However, our search was guided by the methodological principle of theoretical saturation [17, 18].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Politics of Evidence-Based Medicine in the United Kingdom
Harrison, S. (1998) · Policy & Politics · doi:10.1332/030557398782018293 · Cited by 155 · see citing works
The Advanced Handbook of Methods in Evidence Based Healthcare
Stevens, A. J., Abrams, K., Brazier, J. E., Fitzpatrick, R. M., & Lilford, R. J. (2001) · SAGE Publications Ltd eBooks · doi:10.4135/9781848608344 · Cited by 88 · see citing works
Self-citationHealth services research methods : a guide to best practice
Black, N. A. (1998) · Cited by 85 · see citing works
Some limits to evidence-based medicine: a case study from elective orthopaedics.
Ferlie, E. B., Wood, M., & Fitzgerald, L. (1999) · BMJ Quality & Safety · doi:10.1136/qshc.8.2.99 · Cited by 80 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [1]1. Tannenbaum S. Knowing and acting in medical practice: outcomes research. J Health Polit Policy Law 1994; 19 :27-44.
Cited by
Sorted by citations
Jünger, S., Payne, S. A., Brine, J., Radbruch, L., & Brearley, S. G. (2017) · Palliative Medicine · doi:10.1177/0269216317690685 · Cited by 2,210 · see citing works
“or eer Rview therapeutic effects of medical interventions.(25) Its purpose is to examine the quality of the study”
BackgroundHobart, J., & Cano, S. (2009) · Health Technology Assessment · doi:10.3310/hta13120 · Cited by 621 · see citing works
Leening, M. J. G., Vedder, M. M., Witteman, J. C. M., Pencina, M. J., & Steyerberg, E. W. (2014) · Annals of Internal Medicine · doi:10.7326/m13-1522 · Cited by 594 · see citing works
Evaluation of diagnostic tests when there is no gold standard. A review of methods
Rutjes, A., Reitsma, J., Coomarasamy, A., Khan, K., & Bossuyt, P. (2007) · Health Technology Assessment · doi:10.3310/hta11500 · Cited by 524 · see citing works
Moher, D., Pham, B., Lawson, M., & Klassen, T. (2003) · Health Technology Assessment · doi:10.3310/hta7410 · Cited by 500 · see citing works
Social isolation in mental health: a conceptual and methodological review
Wang, J., Lloyd-Evans, B., Giacco, D., Forsyth, R., Nebo, C., Mann, F., & Johnson, S. (2017) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-017-1446-1 · Cited by 462 · see citing works
“We followed Lilford and colleagues’ recommendations for conducting methodological reviews [25] and used an iterative and consultative process to achieve a clear understanding of social isolation and related concepts.”
MethodMueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400 · see citing works
“methodological principle of theoretical saturation [17, 18].”
BackgroundJones, J., Shepherd, J., Baxter, L., Gospodarevskaya, E., Hartwell, D., Harris, P. S., & Price, A. (2009) · Health Technology Assessment · doi:10.3310/hta13350 · Cited by 357 · see citing works
Get real in individual participant data (IPD) meta‐analysis: a review of the methodology
Debray, T. P. A., Moons, K. G. M., van Valkenhoef, G., Efthimiou, O., Hummel, N., Groenwold, R. H. H., Reitsma, J. B., & on behalf of the GetReal methods review group (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1160 · Cited by 348 · see citing works
“This effect is known as theoretical saturation (Lilford et al., 2001).”
BackgroundSocial exclusion and mental health
Morgan, C., Burns, T., Fitzpatrick, R. M., Pinfold, V., & Priebe, S. (2007) · The British Journal of Psychiatry · doi:10.1192/bjp.bp.106.034942 · Cited by 320 · see citing works
“Conceptual and methodological literatures are less easily subjected to fully systematic reviews than discrete bodies of research evidence (Lilford et al, 2001).”
BackgroundInfluentialSelf-citationSelf-monitoring of blood glucose in type 2 diabetes: systematic review
Clar, C., Barnard, K., Cummins, E., Royle, P., & Waugh, N. (2010) · Health Technology Assessment · doi:10.3310/hta14120 · Cited by 274 · see citing works
A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine
Khan, K. S., & Coomarasamy, A. (2006) · BMC Medical Education · doi:10.1186/1472-6920-6-59 · Cited by 263 · see citing works
“This is known as theoretical saturation [8], a principle that guided our search and selection for papers on theories and principles.”
BackgroundIsmail, K., Maissi, E., Thomas, S., Chalder, T., Schmidt, U. H., Bartlett, J. W., Patel, A. K., Dickens, C. M., Creed, F. H., & Treasure, J. L. (2010) · Health Technology Assessment · doi:10.3310/hta14220 · Cited by 244 · see citing works
Turner, D., Wailoo, A., Nicholson, K. G., Cooper, N. J., Sutton, A. J., & Abrams, K. R. (2003) · Health Technology Assessment · doi:10.3310/hta7350 · Cited by 245 · see citing works
Breastfeeding promotion for infants in neonatal units: a systematic review and economic analysis
Renfrew, M. J., Craig, D., Dyson, L., McCormick, F. M., Rice, S. J. C., King, S., Misso, K., Stenhouse, E., & Williams, A. F. (2009) · Health Technology Assessment · doi:10.3310/hta13400 · Cited by 244 · see citing works
McDaid, C., Griffin, S. C., Weatherly, H. L. A., Durée, K. H., van der Burgt, M., Hout, S. V., Akers, J., Davies, R., Sculpher, M. J., & Westwood, M. E. (2009) · Health Technology Assessment · doi:10.3310/hta13040 · Cited by 239 · see citing works
Lourenço, T., Armstrong, N., N’Dow, J., Nabi, G., Deverill, M., Pickard, R. S., Vale, L. D., MacLennan, G., Fraser, C. M., McClinton, S., et al. (2008) · Health Technology Assessment · doi:10.3310/hta12350 · Cited by 222 · see citing works
Jolly, K. B., Taylor, R. S., Lip, G. Y. H., Greenfield, S., Raftery, J. P., Mant, J., Lane, D. A., Jones, M., Lee, K., & Stevens, A. J. (2007) · Health Technology Assessment · doi:10.3310/hta11350 · Cited by 216 · see citing works
Self-citationHow to develop cost-conscious guidelines
Eccles, M., & Mason, J. M. (2001) · Health Technology Assessment · doi:10.3310/hta5160 · Cited by 212 · see citing works
Shaw, L. C., Rodgers, H., Price, C., van Wijck, F., Shackley, P., Steen, N., Barnes, M., Ford, G. A., & Graham, L. (2010) · Health Technology Assessment · doi:10.3310/hta14260 · Cited by 186 · see citing works
Campbell, M. K., Skea, Z., Sutherland, A. G., Cuthbertson, B. H., Entwistle, V. A., McDonald, A., Norrie, J. D., Carlson, R., & Bridgman, S. (2010) · PubMed · doi:10.3310/hta14040 · Cited by 187 · see citing works
Understanding psychiatric institutionalization: a conceptual review
Chow, W. S., & Priebe, S. (2013) · BMC Psychiatry · doi:10.1186/1471-244x-13-169 · Cited by 186 · see citing works
“[14] were followed.”
Blood glucose self-monitoring in type 2 diabetes: a randomised controlled trial
Farmer, A. A., Wade, A. N., French, D., Simon, J., Yudkin, P. L., Gray, A., Craven, A., Goyder, L., Holman, R. R., Mant, D., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13150 · Cited by 178 · see citing works
Ginis, K. A. M., Evans, M. B., Mortenson, W. B., & Noreau, L. (2016) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2016.04.017 · Cited by 178 · see citing works
Wang, D., Cummins, C., Bayliss, S. E., Sandercock, J., & Burls, A. (2008) · Health Technology Assessment · doi:10.3310/hta12360 · Cited by 176 · see citing works
Morrell, C., Warner, R., Slade, P., Dixon, S., Walters, S. J., Paley, G., & Brugha, T. S. (2009) · Health Technology Assessment · doi:10.3310/hta13300 · Cited by 174 · see citing works
Fortnum, H. M., O'Neill, C., Taylor, R. S., Lenthall, R. K., Nikolopoulos, T. P., Lightfoot, G. R., O’Donoghue, G. M., Mason, S. M., Baguley, D., Jones, H., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13180 · Cited by 169 · see citing works
Shifting our conceptualization of social inclusion
Cobigo, V., Ouellette-Kuntz, H. M. J., Lysaght, R. M., & Martin, L. (2012) · Stigma Research and Action · doi:10.5463/sra.v1i3.45 · Cited by 169 · see citing works
“Performing a review of the literature to analyze a concept and identify points of agreement and disagreement regarding its meaning is not easily subjected to a systematic approach (Lilford et al., 2001).”
BackgroundBertens, L. C. M., Broekhuizen, B. D., Naaktgeboren, C. A., Rutten, F. H., Hoes, A. W., van Mourik, Y., Moons, K. G. M., & Reitsma, J. B. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001531 · Cited by 158 · see citing works
“Because of theoretical saturation [9], meaning that additional searches will only add papers without adding information, we only performed the search in the largest electronic medical database (PubMed) and did not update the search beyond May 2012.”
Abubakar, I., Irvine, L., Aldus, C. F., Wyatt, G., Fordham, R. J., Schelenz, S., Shepstone, L., Howe, A., Peck, M. W., & Hunter, P. R. (2007) · Health Technology Assessment · doi:10.3310/hta11360 · Cited by 159 · see citing works
Chen, Y., Jowett, S. M., Barton, P. M., Malottki, K., Hyde, C., Gibbs, J. S. R., Pepke-Zaba, J., Fry-Smith, A., Roberts, J., & Moore, D. (2009) · Health Technology Assessment · doi:10.3310/hta13490 · Cited by 147 · see citing works
Andronis, L., Barton, P. M., & Bryan, S. (2009) · Health Technology Assessment · doi:10.3310/hta13290 · Cited by 139 · see citing works
Resource-oriented therapeutic models in psychiatry: conceptual review
Priebe, S., Omer, S., Giacco, D., & Slade, M. (2014) · The British Journal of Psychiatry · doi:10.1192/bjp.bp.113.135038 · Cited by 137 · see citing works
The use of economic evaluations in NHS decision-making: a review and empirical investigation
Williams, I., McIver, S., Moore, D., & Bryan, S. (2008) · Health Technology Assessment · doi:10.3310/hta12070 · Cited by 129 · see citing works
Smith, E., Ross, F., Donovan, S., Manthorpe, J., Brearley, S., Sitzia, J., & Beresford, P. (2008) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2006.09.010 · Cited by 113 · see citing works
Good communication in psychiatry – a conceptual review
Priebe, S., Dimić, S., Wildgrube, C., Janković, J., Cushing, A. M., & McCabe, R. (2011) · European Psychiatry · doi:10.1016/j.eurpsy.2010.07.010 · Cited by 113 · see citing works
“[32] have made a number of recommendations on how to carry out conceptual reviews.”
BackgroundA systematic review of presumed consent systems for deceased organ donation
Rithalia, A. D., McDaid, C., Suekarran, S., Norman, G., Myers, L., & Sowden, A. J. (2009) · Health Technology Assessment · doi:10.3310/hta13260 · Cited by 106 · see citing works
Gowers, S., Clark, A., Roberts, C., Byford, S., Barrett, B. M., Griffiths, A., Edwards, V., Bryan, C., Smethurst, N., Rowlands, L., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14150 · Cited by 105 · see citing works
Performance of screening tests for child physical abuse in accident and emergency departments
Woodman, J., Pitt, M., Wentz, R., Taylor, B., Hodes, D., & Gilbert, R. E. (2008) · Health Technology Assessment · doi:10.3310/hta12330 · Cited by 103 · see citing works
VenUS II: a randomised controlled trial of larval therapy in the management of leg ulcers
Dumville, J. C., Worthy, G., Soares, M. O., Bland, J., Cullum, N. A., Dowson, C. G., Iglesias, C., McCaughan, D., Mitchell, J., Nelson, E. A., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13550 · Cited by 100 · see citing works
Use of the Delphi method to generate guidance in emergency nursing practice: A systematic review
Varndell, W., Fry, M., Lutze, M., & Elliott, D. (2020) · International Emergency Nursing · doi:10.1016/j.ienj.2020.100867 · Cited by 98 · see citing works
Why involve families in acute mental healthcare? A collaborative conceptual review
Dirik, A., Sandhu, S., Giacco, D., Barrett, K., Bennison, G., Collinson, S., & Priebe, S. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-017680 · Cited by 95 · see citing works
“Data analysis A multidisciplinary review team was formed to minimise biases in the searches and analyses.(15) This composed of the lead researcher (AD, a doctoral researcher), a research psychologist (SS), two clinical/academic psychiatrists (DG and SP), three individuals with lived experience of acute mental health t…”
MethodMowatt, G., Houston, J. G., Hernández, R. A., de Verteuil, R., Fraser, C. M., Cuthbertson, B. H., & Vale, L. D. (2009) · Health Technology Assessment · doi:10.3310/hta13070 · Cited by 94 · see citing works
Lefebvre, J. S., Evans, M. B., Turnnidge, J., Gainforth, H. L., & Côté, J. (2016) · International Journal of Sports Science & Coaching · doi:10.1177/1747954116676116 · Cited by 96 · see citing works
“Accordingly, we adopted a broader search process by using what Lilford et al.31 described as a synthesis review protocol as a means of integrating multiple data sources.”
MethodBond, M., Pitt, M., Akoh, J. A., Moxham, T., Hoyle, M., & Anderson, R. (2009) · Health Technology Assessment · doi:10.3310/hta13380 · Cited by 91 · see citing works
Vitamin K to prevent fractures in older women: systematic review and economic evaluation
Stevenson, M., Lloyd-Jones, M., & Papaioannou, D. (2009) · Health Technology Assessment · doi:10.3310/hta13450 · Cited by 90 · see citing works
Cooke, M. R., Marsh, J. L., Clark, M., Nakash, R., Jarvis, R., Hutton, J. L., Szczepura, A. K., Wilson, S., & Lamb, S. E. (2009) · Health Technology Assessment · doi:10.3310/hta13130 · Cited by 91 · see citing works
Self-citationChase, D., Rosten, C., Turner, S., Hicks, N., & Milne, R. (2009) · Health Technology Assessment · doi:10.3310/hta13590 · Cited by 89 · see citing works
Charlesworth, G. M., Shepstone, L., Wilson, E., Thalanany, M., Mugford, M., & Poland, F. M. (2008) · Health Technology Assessment · doi:10.3310/hta12040 · Cited by 89 · see citing works
Hummel, S., Simpson, E., Hemingway, P., Stevenson, M., & Rees, A. (2010) · Health Technology Assessment · doi:10.3310/hta14470 · Cited by 86 · see citing works
Pilgrim, H., Lloyd-Jones, M., & Rees, A. (2009) · Health Technology Assessment · doi:10.3310/hta13100 · Cited by 85 · see citing works
Sandercock, P. A., Berge, E., Dennis, M. S., Forbes, J., Hand, P. J., Kwan, J. S. K., Lewis, S. C., Lindley, R. I., Neilson, A. R., Thomas, B., et al. (2002) · Health Technology Assessment · doi:10.3310/hta6260 · Cited by 85 · see citing works
Colbourn, T., Asseburg, C., Bojke, L., Philips, Z., Claxton, K., Ades, A. E., & Gilbert, R. E. (2007) · Health Technology Assessment · doi:10.3310/hta11290 · Cited by 82 · see citing works
Clegg, A. J., Loveman, E., Gospodarevskaya, E., Harris, P. S., Bird, A., Bryant, J., Scott, D. A., Davidson, P., Little, P. S., & Coppin, R. (2010) · Health Technology Assessment · doi:10.3310/hta14280 · Cited by 82 · see citing works
Little, P., Turner, S., Rumsby, K., Warner, G., Moore, M., Lowes, J., Smith, H., Hawke, C., Turner, D., Leydon, G., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13190 · Cited by 137 · see citing works
Measuring patient-reported outcomes in psychosis: conceptual and methodological review
Reininghaus, U., & Priebe, S. (2012) · The British Journal of Psychiatry · doi:10.1192/bjp.bp.111.107615 · Cited by 79 · see citing works
Young, A. E., Brookes, S. T., Avery, K. N. L., Davies, A., Metcalfe, C., & Blazeby, J. M. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.06.016 · Cited by 77 · see citing works
“A limitation of the study is that we did not undertake a full systematic review but rather focused our attention on studies identified through the COMET database from September 2015, with related articles identified through two other search engines, as per the published advice on methodological systematic reviews [49,…”
MethodConcepts of Resilience in Adolescent Mental Health Research
Anderson, K., & Priebe, S. (2021) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2021.03.035 · Cited by 70 · see citing works
Bankhead, C. R., Brett, J., Bukach, C., Webster, P., Stewart-Brown, S. L., Munafo, M. R., & Austoker, J. (2003) · Health Technology Assessment · doi:10.3310/hta7420 · Cited by 68 · see citing works
Williamson, I. G., Benge, S., Barton, S. J., Petrou, S., Letley, L. H., Fasey, N., Abangma, G., Dakin, H., & Little, P. S. (2009) · Health Technology Assessment · doi:10.3310/hta13370 · Cited by 66 · see citing works
Kai, J., Ulph, F., Cullinan, T., & Qureshi, N. (2009) · Health Technology Assessment · doi:10.3310/hta13570 · Cited by 68 · see citing works
Michaels, J. A., Campbell, W., King, B., MacIntyre, J., Palfreyman, S. J., Shackley, P., & Stevenson, M. (2009) · Health Technology Assessment · doi:10.3310/hta13560 · Cited by 66 · see citing works
Fleeman, N., McLeod, C., Bagust, A., Beale, S., Boland, A., Dundar, Y., Jorgensen, A., Payne, K., Pirmohamed, M., Pushpakom, S. P., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14030 · Cited by 65 · see citing works
Hay, A. D., Redmond, N. M., Costelloe, C., Montgomery, A. A., Fletcher, M. E., Hollinghurst, S. P., & Peters, T. (2009) · Health Technology Assessment · doi:10.3310/hta13270 · Cited by 66 · see citing works
Shepherd, J., Rogers, G., Anderson, R., Main, C., Coon, J. T., Hartwell, D., Liu, Z., Loveman, E., Green, C., Pitt, M., et al. (2008) · Health Technology Assessment · doi:10.3310/hta12190 · Cited by 65 · see citing works
DANIELS, J. P., Gray, J. W., Pattison, H. M., Roberts, T. E., Edwards, E. A., Milner, P., Spicer, L., King, E., Hills, R. K., Gray, R. G., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13420 · Cited by 63 · see citing works
Comparison of case note review methods for evaluating quality and safety in health care
Lock, C. A., Wilson, J. A., Steen, N., Eccles, M., Mason, H., Carrie, S., Clarke, R. W., Kubba, H., Raine, C. H., Zarod, A., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14130 · Cited by 62 · see citing works
Methodological Review of Intimate Partner Violence Prevention Research
Murray, C. E., & Graybeal, J. (2007) · Journal of Interpersonal Violence · doi:10.1177/0886260507304293 · Cited by 62 · see citing works
“Lilford et al. (2001) suggested that systematic methodological reviews should be based on broad-based searches of literature from a wide range of sources.”
BackgroundRobson, S. C., Kelly, T., Howel, D., Deverill, M., Hewison, J., Lie, M., Stamp, E., Armstrong, N., & May, C. R. (2009) · Health Technology Assessment · doi:10.3310/hta13530 · Cited by 64 · see citing works
Coleman, T. J., Agboola, S., Leonardi‐Bee, J., Taylor, M. J., McEwen, A., & McNeill, A. D. (2010) · Health Technology Assessment · doi:10.3310/hta14490 · Cited by 61 · see citing works
Non-occupational postexposure prophylaxis for HIV: a systematic review
Bryant, J., Baxter, L., & Hird, S. (2009) · Health Technology Assessment · doi:10.3310/hta13140 · Cited by 60 · see citing works
Taylor, R. S., & Elston, J. (2009) · Health Technology Assessment · doi:10.3310/hta13080 · Cited by 59 · see citing works
Screening for hyperglycaemia in pregnancy: a rapid update for the National Screening Committee
Waugh, N. R., Royle, P., Clar, C., Henderson, R., Cummins, E., Hadden, D. R., Lindsay, R. S., & Pearson, D. (2010) · Health Technology Assessment · doi:10.3310/hta14450 · Cited by 59 · see citing works
McKenna, C., McDaid, C., Suekarran, S., Hawkins, N., Claxton, K., Light, K., Chester, M., Cleland, J. G. F., Woolacott, N. F., & Sculpher, M. J. (2009) · Health Technology Assessment · doi:10.3310/hta13240 · Cited by 58 · see citing works
Cahill, J., Barkham, M., Hardy, G. E., Gilbody, S. M., Richards, D. A., Bower, P., Audin, K., & Connell, J. (2008) · Health Technology Assessment · doi:10.3310/hta12240 · Cited by 58 · see citing works
POTTER, J. F., Mistri, A. K., Brodie, F. G., Tchernova, J., Wilson, E., Jagger, C., James, M. A., Ford, G. A., & Robinson, T. G. (2009) · Health Technology Assessment · doi:10.3310/hta13090 · Cited by 55 · see citing works
Tyrer, P. J., Oliver, P., Romeo, R., Knapp, M. R. J., Huline‐Dickens, S., Bouras, N., Ahmed, M. Z., Cooray, S., Deb, S. S., Murphy, D., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13210 · Cited by 55 · see citing works
Hemingway, H., Henriksson, M., Chen, R., Damant, J., Fitzpatrick, N. K., Abrams, K. R., Hingorani, A. D., Janzon, M., Shipley, M. J., Feder, G. S., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14090 · Cited by 51 · see citing works
Priebe, S., Conneely, M., McCabe, R., & Bird, V. J. (2019) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796019000428 · Cited by 46 · see citing works
“We, therefore, followed the approach suggested for conceptual reviews with (a) a wide search of disparate databases and sources, (b) forward and backward citation tracking, (c) safeguards against potential biases by using a team of researchers with different backgrounds and (d) some overlap of the searching, analysing…”
BackgroundMethodThe Follow-Up of Eating Disorders from Adolescence to Early Adulthood: A Systematic Review
Filipponi, C., Visentini, C., Filippini, T., Cutino, A., Ferri, P., Rovesti, S., Latella, E., & Lorenzo, R. D. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192316237 · Cited by 45 · see citing works
“To avoid bias, a multiprofessional team cooperated in selecting articles and analyzed data independently, discussing with each other in case of discrepancies, in accordance with review authors’ indications [63].”
MethodSpeed, C., Heaven, B., Adamson, A., Bond, J. B., Corbett, S. S., Lake, A. A., May, C. R., Vanoli, A., McMeekin, P. J., Moynihan, P. J., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14520 · Cited by 43 · see citing works
The use of hospital admission data as a measure of outcome in clinical studies of heart failure
Metcalfe, C. D. (2003) · European Heart Journal · doi:10.1016/s0195-668x(02)00384-6 · Cited by 43 · see citing works
“Proportion of patients with any admissions 4 (3) Proportion of patients with stated number of admissions 6 (5) Mean number of admissions 4 ( 3 )”
BackgroundMethodInfluentialAssessing variability in results in systematic reviews of diagnostic studies
Naaktgeboren, C. A., Ochodo, E., van Enst, W. A., de Groot, J. A. H., Hooft, L., Leeflang, M., Bossuyt, P. M., Moons, K. G. M., & Reitsma, J. B. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0108-4 · Cited by 40 · see citing works
“In other words, that including more reviews would not have significantly changed our conclusions [17].”
BackgroundAra, R., Pandor, A., Stevens, J., Rees, A., & Rafia, R. (2009) · Health Technology Assessment · doi:10.3310/hta13340 · Cited by 39 · see citing works
Bond, M., Wyatt, K. M., Lloyd, J., Welch, K., & Taylor, R. S. (2009) · Health Technology Assessment · doi:10.3310/hta13610 · Cited by 39 · see citing works
Understanding global mental health: a conceptual review
Rajabzadeh, V., Burn, E., Sajun, S., Suzuki, M., Bird, V. J., & Priebe, S. (2021) · BMJ Global Health · doi:10.1136/bmjgh-2020-004631 · Cited by 38 · see citing works
“All databases were searched from inception to 6 May 2020, using the term ‘global mental health’, identified from the title, abstract and keywords.”
Solution-Focused Approaches in the Practice of UK Educational Psychologists
Stobie, I., Boyle, J. M. E., & Woolfson, L. M. (2005) · School Psychology International · doi:10.1177/0143034305050890 · Cited by 37 · see citing works
“Systematic reviews of the outcomes from intervention generally give high weightings to randomized control trials, based upon the pre- and post-treatment follow-up of participants who are randomly allocated to a control group and to a group who receives a specific intervention (Lilford et al., 2001).”
BackgroundNaaktgeboren, C. A., van Enst, W. A., Ochodo, E., de Groot, J. A. H., Hooft, L., Leeflang, M., Bossuyt, P. M., Moons, K. G. M., & Reitsma, J. B. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.05.018 · Cited by 36 · see citing works
“We think that our sample size was large enough because our goal was to get an idea of the range of approaches used rather than to precisely estimate how many studies took each approach to investigating sources of heterogeneity [11].”
BackgroundPalm, R., Jünger, S., Reuther, S., Schwab, C. G. G., Dichter, M., Holle, B., & Halek, M. (2016) · BMC Geriatrics · doi:10.1186/s12877-016-0249-7 · Cited by 35 · see citing works
“Unlike a traditional Cochrane-style review, Lilford and colleagues [36] recommend methodological reviewers not to obtain a thorough collection of studies, but to ensure a wide search that provides an overview of the topic.”
BackgroundInfluentialDormandy, E., Bryan, S., Gulliford, M. C., Roberts, T. E., Ades, A. E., Calnan, M., Atkin, K. M., Karnon, J. D., Barton, P., Logan, J., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14200 · Cited by 34 · see citing works
Jerome, L., McNamee, P., Abdel-Halim, N., Elliot, K., & Woods, J. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1068006 · Cited by 34 · see citing works
“(2) Vote counting was then performed to produce an initial description of patterns across the included publications, the findings of which were then further explored (9).”
InfluentialSimpson, E. L., Stevenson, M., Rawdin, A. C., & Papaioannou, D. E. (2009) · Health Technology Assessment · doi:10.3310/hta13020 · Cited by 33 · see citing works
Schick, A., Rauschenberg, C., Ader, L., Daemen, M., Wieland, L. M., Paetzold, I., Postma, M. R., Schulte‐Strathaus, J. C. C., & Reininghaus, U. (2022) · Psychological Medicine · doi:10.1017/s0033291722003336 · Cited by 32 · see citing works
Elicitation and Use of Patients’ Preferences in the Treatment of Psoriasis: A Systematic Review
Umar, N., Yamamoto, S. S., Loerbroks, A., & Terris, D. (2012) · Acta Dermato Venereologica · doi:10.2340/00015555-1304 · Cited by 33 · see citing works
Parker, S. G., Oliver, P., Pennington, M., Bond, J. B., Jagger, C., Enderby, P., Curless, R., Chater, T., Vanoli, A., Fryer, K., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13390 · Cited by 29 · see citing works
Cross, J., Elender, F., Barton, G., Clark, A., Shepstone, L., Blyth, A., Bachmann, M. O., & Harvey, I. M. (2010) · Health Technology Assessment · doi:10.3310/hta14230 · Cited by 27 · see citing works
Maden, M., Cunliffe, A., McMahon, N. E., Booth, A., Carey, G. M., Paisley, S., Dickson, R., & Gabbay, M. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0638-9 · Cited by 30 · see citing works
“Methods for undertaking methodological reviews are undefined, but the focus of the search should aim for a systematic rather than exhaustive approach [26].”
BackgroundNewman, S. P., Cooke, D. D., Casbard, A. C., Walker, S., Meredith, S. K., Nunn, A., Steed, L., Manca, A., Sculpher, M. J., Barnard, M., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13280 · Cited by 23 · see citing works
Panayidou, K., Gsteiger, S., Egger, M., Kilcher, G., Carreras, M., Efthimiou, O., Debray, T. P. A., Trelle, S., Hummel, N., & on behalf of the GetReal methods review group (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1202 · Cited by 24 · see citing works
“In other words, we are con fi dent that our search reached the stage of theoretical saturation (Lilford et al ., 2001).”
Are adverse effects incorporated in economic models? An initial review of current practice
Craig, D., McDaid, C., Fonseca, T., Stock, C., Duffy, S., & Woolacott, N. F. (2009) · Health Technology Assessment · doi:10.3310/hta13620 · Cited by 23 · see citing works
- Authors
- Lilford, R. J., Richardson, A., Stevens, A., Fitzpatrick, R., Edwards, S., Rock, F., & Hutton, J. L.
- Affiliations
- University of Birmingham (GB)
- Research Consultant, London, UK
- University of Oxford (GB)
- University of Bristol (GB)
- University of Warwick (GB)
- Venue
- Health Technology Assessment
- Published
- 2001
- Volume
- 5
- Issue
- 8
- Pages
- 1-57
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W1994394357
- Cited by
- 132
- References
- 4
- Citations by year
- 2026: 1 · 2025: 1 · 2023: 3 · 2022: 2 · 2021: 5 · 2020: 4 · 2019: 2 · 2018: 3 · 2017: 5 · 2016: 6 · 2015: 4 · 2014: 5 · 2013: 2 · 2012: 3
- FWCI
- 4.31
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Health Technology Assessment (publishedVersion, public-domain)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Funders
- Health Technology Assessment programme (96/71/01)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Health Sciences › Health Professions › Health Information Management › Methodologies in Health Research and Practice
- Keywords
- methodological research, research methodology, research management, researcher perspectives, literature search strategies
- MeSH terms
- Attitude, United Kingdom, Humans, Information Services, Interviews as Topic, Research Design, Research Personnel, Technology Assessment, Biomedical, Meta-Analysis as Topic, Bias, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
