The DOI matches its Crossref record; 95 of the 95 references it cites were found in a registry or an index.
Evaluating impact from research: A methodological framework
Open accessAbstract
Interest in impact evaluation has grown rapidly as research funders increasingly demand evidence that their investments lead to public benefits. This paper analyses literature to provide a new definition of research impact and impact evaluation, develops a typology of research impact evaluation designs, and proposes a methodological framework to guide evaluations of the significance and reach of impact that can be attributed to research. An adapted Grounded Theory Analysis of research impact evaluation frameworks drawn from cross-disciplinary peer-reviewed and grey literature. Recognizing the subjective nature of impacts as they are perceived by different groups in different times, places and cultures, we define research impact evaluation as the process of assessing the significance and reach of both positive and negative effects of research. Five types of impact evaluation design are identified encompassing a range of evaluation methods and approaches: i) experimental and statistical methods; ii) textual, oral and arts-based methods; iii) systems analysis methods; iv) indicator-based approaches; and v) evidence synthesis approaches. Our guidance enables impact evaluation design to be tailored to the aims and context of the evaluation, for example choosing a design to establish a body of research as a necessary (e.g. a significant contributing factor amongst many) or sufficient (e.g. sole, direct) cause of impact, and choosing the most appropriate evaluation design for the type of impact being evaluated. Using the proposed definitions, typology and methodological framework, researchers, funders and other stakeholders working across multiple disciplines can select a suitable evaluation design and methods to evidence the impact of research from any discipline.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses
Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. (2008) · The FASEB Journal · doi:10.1096/fj.07-9492lsf · Cited by 5,039
Talking about pictures: A case for photo elicitation
Harper, D. (2002) · Visual Studies · doi:10.1080/14725860220137345 · Cited by 3,966
Core outcome domains for chronic pain clinical trials: IMMPACT recommendations
Turk, D. C., Dworkin, R. H., Allen, R. R., Bellamy, N., Brandenburg, N., Carr, D. B., Cleeland, C., Dionne, R., Farrar, J. T., Galer, B. S., et al. (2003) · Pain · doi:10.1016/j.pain.2003.08.001 · Cited by 3,510 · cited 1×
“This is similar to approaches to evidence in the medical research community (e.g. the use of common outcome measures for chronic pain clinical trials enabling findings to be synthesised across studies in meta-analyses to inform evidence-based medicine policy and practice; Turk et al., 2003) and could in theory be appl…”
MethodThe answer is 17 years, what is the question: understanding time lags in translational research
Morris, Z. S., Wooding, S., & Grant, J. (2011) · Journal of the Royal Society of Medicine · doi:10.1258/jrsm.2011.110180 · Cited by 3,340
Configurational Comparative Methods: Qualitative Comparative Analysis (QCA) and Related Techniques
Rihoux, B., & Ragin, C. (2009) · SAGE Publications · doi:10.4135/9781452226569 · Cited by 3,089
Mattley, C., Strauss, A., & Corbin, J. (1999) · Contemporary Sociology · doi:10.2307/2655359 · Cited by 2,408
Dixon-Woods, M., Cavers, D., Agarwal, S., Annandale, E., Arthur, A., Harvey, J., Hsu, R., Katbamna, S., Olsen, R., Smith, L., et al. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-35 · Cited by 2,289
Methods for the synthesis of qualitative research: a critical review
Barnett-Page, E., & Thomas, J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-59 · Cited by 2,042
Martín-Martín, A., Orduna-Malea, E., Thelwall, M., & Delgado López-Cózar, E. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.09.002 · Cited by 1,990
The need for evidence-based conservation
Sutherland, W. J., Pullin, A. S., Dolman, P. M., & Knight, T. M. (2004) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2004.03.018 · Cited by 1,822
Reed, M. S., Evely, A. C., Cundill, G., Fazey, I., Glass, J., Laing, A., Newig, J., Parrish, B., Prell, C., Raymond, C., et al. (2010) · Ecology and Society · doi:10.5751/es-03564-1504r01 · Cited by 1,554
Self-citationWriting Narrative Literature Reviews
Baumeister, R. F., & Leary, M. R. (1997) · Review of General Psychology · doi:10.1037/1089-2680.1.3.311 · Cited by 1,482
Using evidenceHow research can inform public services
Nutley, S. M., Walter, I., & Davies, H. T. O. (2007) · Policy Press eBooks · doi:10.1332/policypress/9781861346650.001.0001 · Cited by 1,335
Time to challenge the spurious hierarchy of systematic over narrative reviews?
Greenhalgh, T., Thorne, S., & Malterud, K. (2018) · European Journal of Clinical Investigation · doi:10.1111/eci.12931 · Cited by 1,267
Soft systems methodology: a thirty year retrospective
Checkland, P. (2000) · Systems Research and Behavioral Science · doi:10.1002/1099-1743(200011)17:1+<::aid-sres374>3.0.co;2-o · Cited by 1,266
An Introduction to Systematic Reviews
Gough, D., Oliver, S., & Thomas, J. (2017) · doi:10.4135/9781036234942 · Cited by 1,239
Methods for Evaluating Changes in Health Care Policy: The Difference-in-Differences Approach
Dimick, J. B., & Ryan, A. M. (2014) · JAMA · doi:10.1001/jama.2014.16153 · Cited by 1,193
Handbook of Meta-analysis in Ecology and Evolution
(2013) · doi:10.1515/9781400846184 · Cited by 1,183
Egger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181
Best evidence synthesis: An intelligent alternative to meta-analysis
Slavin, R. E. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)00097-a · Cited by 949
Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review
Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., Kyriakidou, O., & Peacock, R. (2005) · Social Science & Medicine · doi:10.1016/j.socscimed.2004.12.001 · Cited by 913
Purposeful Program Theory: Effective Use of Theories of Change and Logic Models
Funnell, S., & Rogers, P. (2011) · RMIT Research Repository (RMIT University Library) · Cited by 897
Reed, M. S., Fraser, E. D. G., & Dougill, A. J. (2006) · Ecological Economics · doi:10.1016/j.ecolecon.2005.11.008 · Cited by 785
Self-citationThe cement of the universe, a study of causation
Rota, G.-C. (1978) · Advances in Mathematics · doi:10.1016/0001-8708(78)90103-2 · Cited by 688
Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. J. (2010) · World Bank, Washington, DC eBooks · doi:10.1596/978-0-8213-8541-8 · Cited by 683
A Guide to Understanding Social Science Research for Natural Scientists
MOON, K., & BLACKMAN, D. (2014) · Conservation Biology · doi:10.1111/cobi.12326 · Cited by 678
Photovoice as a Participatory Health Promotion Strategy
Wang, C. C., Yi, W. K., Tao, Z. W., & Carovano, K. (1998) · Health Promotion International · doi:10.1093/heapro/13.1.75 · Cited by 661
The Cement of the Universe: A Study of Causation.
Berofsky, B., & Mackie, J. L. (1977) · The Journal of Philosophy · doi:10.2307/2025574 · Cited by 631
Assessment, evaluations, and definitions of research impact: A review
Penfield, T., Baker, M. J., Scoble, R., & Wykes, M. C. (2014) · Research Evaluation · doi:10.1093/reseval/rvt021 · Cited by 573
The Benefits and Challenges of Using Systematic Reviews in International Development Research
Mallett, R., Hagen-Zanker, J., Slater, R., & Duvendack, M. (2012) · Journal of Development Effectiveness · doi:10.1080/19439342.2012.711342 · Cited by 565
Monitoring and evaluation in conservation: a review of trends and approaches
STEM, C., MARGOLUIS, R., SALAFSKY, N., & BROWN, M. (2005) · Conservation Biology · doi:10.1111/j.1523-1739.2005.00594.x · Cited by 542
Introducing 'productive interactions' in social impact assessment
Spaapen, J., & van Drooge, L. (2011) · Research Evaluation · doi:10.3152/095820211x12941371876742 · Cited by 518
Handbook on Impact Evaluation: Quantitative Methods and Practices
Khandker, S., B. Koolwal, G., & Samad, H. (2009) · RePEc: Research Papers in Economics · doi:10.1596/978-0-8213-8028-4 · Cited by 485
Effectiveness in Conservation Practice: Pointers from Medicine and Public Health
Pullin, A. S., & Knight, T. M. (2001) · Conservation Biology · doi:10.1046/j.1523-1739.2001.99499.x · Cited by 481
Evidence-based Policy: In Search of a Method
Pawson, R. (2002) · Evaluation · doi:10.1177/1358902002008002512 · Cited by 403 · cited 2×
“These often use broad inclusion criteria for evidence (qualitative and quantitative) to facilitate comparison of impact evaluation methods, develop a transferable theory, and attempt to provide policy-makers with knowledge in response to time sensitive and emerging issues (Victora et al., 2011; Saul et al., 2013; Paws…”
BackgroundSHETRAN: Distributed River Basin Flow and Transport Modeling System
Ewen, J., Parkin, G., & O'Connell, P. E. (2000) · Journal of Hydrologic Engineering · doi:10.1061/(asce)1084-0699(2000)5:3(250 · Cited by 383
Chambers, R. (2013) · Routledge eBooks · doi:10.4324/9781849771665 · Cited by 325
Public Value Mapping and Science Policy Evaluation
Bozeman, B., & Sarewitz, D. (2011) · Minerva · doi:10.1007/s11024-011-9161-7 · Cited by 308
Impact pathway evaluation: an approach for achieving and attributing impact in complex systems
Douthwaite, B., Kuby, T., van de Fliert, E., & Schulz, S. (2003) · Agricultural Systems · doi:10.1016/s0308-521x(03)00128-8 · Cited by 291
PARTICIPATORY INDICATOR DEVELOPMENT: WHAT CAN ECOLOGISTS AND LOCAL COMMUNITIES LEARN FROM EACH OTHER
Reed, M. S., Dougill, A. J., & Baker, T. R. (2008) · Ecological Applications · doi:10.1890/07-0519.1 · Cited by 289
Self-citationA time-responsive tool for informing policy making: rapid realist review
Saul, J. E., Willis, C. D., Bitz, J., & Best, A. (2013) · Implementation Science · doi:10.1186/1748-5908-8-103 · Cited by 283
Causation in epidemiology: Table 1
Parascandola, M., & Weed, D. L. (2001) · Journal of Epidemiology and Community Health · doi:10.1136/jech.55.12.905 · Cited by 276
Introduction to Systematic Reviews
Clarke, M. (2017) · Veterinary Evidence · doi:10.18849/ve.v2i2.131 · Cited by 276
Arts-based methods in health research: A systematic review of the literature
Fraser, K. D., & al Sayah, F. (2011) · Arts & Health · doi:10.1080/17533015.2011.561357 · Cited by 272
The Production and Dissemination of Knowledge: A Scoping Review of Arts-Based Health Research
Boydell, K., Gladstone, B., Volpe, T., Allemang, B. A., & Stasiulis, E. (2011) · Forum qualitative Sozialforschung · doi:10.17169/fqs-13.1.1711 · Cited by 264
Sanjari, M., Bahramnezhad, F., Fomani, F. K., Shoghi, M., & Cheraghi, M. A. (2014) · PubMed Central · Cited by 247
Evaluation and Education: At Quarter Century
Smith, N. L., McLaughlin, M. W., & Phillips, D. C. (1992) · Educational Evaluation and Policy Analysis · doi:10.2307/1164531 · Cited by 217
Sheppard, C. R. C., Davy, S. K., & Pilling, G. M. (2009) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780198566359.001.0001 · Cited by 215
Victora, C. G., Black, R. E., Boerma, J. T., & Bryce, J. (2011) · The Lancet · doi:10.1016/s0140-6736(10)60810-0 · Cited by 207
Beyond Formative and Summative Evaluation
Scriven, M. S. (1991) · Teachers College Record The Voice of Scholarship in Education · Cited by 197
Martín-Martín, A., Orduna-Malea, E., & Delgado López-Cózar, E. (2018) · Scientometrics · doi:10.1007/s11192-018-2820-9 · Cited by 195
Probabilities of causation: Bounds and identification
Tian, J., & Pearl, J. (2000) · Annals of Mathematics and Artificial Intelligence · doi:10.1023/a:1018912507879 · Cited by 179
Rau, H., Goggins, G., & Fahy, F. (2018) · Research Policy · doi:10.1016/j.respol.2017.11.005 · Cited by 171
Using case studies to explore the external validity of ‘complex’ development interventions
Woolcock, M. (2013) · Evaluation · doi:10.1177/1356389013495210 · Cited by 168
The 'Payback Framework' explained
Donovan, C., & Hanney, S. R. (2011) · Research Evaluation · doi:10.3152/095820211x13118583635756 · Cited by 160
Joly, P.-B., Gaunand, A., Colinet, L., Larédo, P., Lemarié, S., & Matt, M. (2015) · Research Evaluation · doi:10.1093/reseval/rvv015 · Cited by 159
Sundin, A., Andersson, K., & Watt, R. (2018) · Environmental Evidence · doi:10.1186/s13750-018-0116-4 · Cited by 152
Collage Portraits as a Method of Analysis in Qualitative Research
Gerstenblatt, P. (2013) · International Journal of Qualitative Methods · doi:10.1177/160940691301200114 · Cited by 151
Research governance and the future(s) of research assessment
Oancea, A. (2019) · Humanities and Social Sciences Communications · doi:10.1057/s41599-018-0213-6 · Cited by 121
van der Vaart, G., van Hoven, B., & Huigen, P. P. P. (2017) · Forum qualitative Sozialforschung · doi:10.17169/fqs-19.2.2961 · Cited by 119
The UK and Italian research assessment exercises face to face
Rebora, G., & Turri, M. (2013) · Research Policy · doi:10.1016/j.respol.2013.06.009 · Cited by 113
Chubb, J., & Reed, M. S. (2018) · British Politics · doi:10.1057/s41293-018-0077-9 · Cited by 112
Self-citationGeuna, A., & Piolatto, M. (2015) · Research Policy · doi:10.1016/j.respol.2015.09.004 · Cited by 112
Samuel, G. N., & Derrick, G. (2015) · Research Evaluation · doi:10.1093/reseval/rvv007 · Cited by 108
Participatory Impact Pathways Analysis: a practical method for project planning and evaluation
Alvarez, S., Douthwaite, B., Thiele, G., Mackay, R., Córdoba, D., & Tehelen, K. (2010) · Development in Practice · doi:10.1080/09614524.2010.513723 · Cited by 106
Matt, M., Gaunand, A., Joly, P., & Colinet, L. (2016) · Research Policy · doi:10.1016/j.respol.2016.09.016 · Cited by 105
Open systems evaluation and the logic model: Program planning and evaluation tools
Julian, D. A., Jones, A., & Deyo, D. (1995) · Evaluation and Program Planning · doi:10.1016/0149-7189(95)00034-8 · Cited by 103
INTRODUCTION TO MIXED METHODS IN IMPACT EVALUATION
Bamberger, M. (2012) · Cited by 103
Bozeman, B. L., & Youtie, J. (2017) · Research Policy · doi:10.1016/j.respol.2017.06.003 · Cited by 101
Gibbs, L. F. (2017) · Educational Action Research · doi:10.1080/09650792.2017.1358493 · Cited by 101
Probabilities Of Causation: Three Counterfactual Interpretations And Their Identification
Pearl, J. (1999) · Synthese · doi:10.1023/a:1005233831499 · Cited by 100
Chubb, J., Watermeyer, R., & Wakeling, P. (2017) · Higher Education Research & Development · doi:10.1080/07294360.2017.1288709 · Cited by 94
Braham, M., & van Hees, M. (2009) · Erkenntnis · doi:10.1007/s10670-009-9184-8 · Cited by 89
Coemans, S., Wang, Q., Leysen, J., & Hannes, K. (2015) · International Journal of Educational Research · doi:10.1016/j.ijer.2015.02.008 · Cited by 85
The creative turn in evidence for public health: community and arts-based methodologies
Byrne, E., Elliott, E., Saltus, R., & Angharad, J. (2017) · Journal of Public Health · doi:10.1093/pubmed/fdx151 · Cited by 84
Impact Evaluation in Practice, First Edition [La evaluación de impacto en la práctica]
Gertler, P., Martínez, S., Prémand, P., Rawlings, L. B., & Vermeersch, C. M. J. (2011) · World Bank Publications · Cited by 84
How do we define the policy impact of public health research? A systematic review
Alla, K., Hall, W., Whiteford, H., Head, B. W., & Meurk, C. (2017) · Health Research Policy and Systems · doi:10.1186/s12961-017-0247-z · Cited by 79
A change of view: arts-based research and psychology
Chamberlain, K., McGuigan, K., Anstiss, D., & Marshall, K. (2018) · Qualitative Research in Psychology · doi:10.1080/14780887.2018.1456590 · Cited by 76
Process Tracing and Bayesian Updating for impact evaluation
Befani, B., & Stedman-Bryce, G. (2016) · Evaluation · doi:10.1177/1356389016654584 · Cited by 73
Cross-discipline evidence principles for sustainability policy
Game, E. T., Tallis, H. M., Olander, L. P., Alexander, S. M., Busch, J., Cartwright, N. D., Kalies, E. L., Masuda, Y. J., Mupepele, A., Qiu, J., et al. (2018) · Nature Sustainability · doi:10.1038/s41893-018-0141-x · Cited by 67
The photo‐survey research method: capturing life in the city
Moore, G., Croxford, B., Adams, M. D., Refaee, M., Cox, T. J., & Sharples, S. (2008) · Visual Studies · doi:10.1080/14725860801908536 · Cited by 61
Building the concept of research impact literacy
Bayley, J. E., & Phipps, D. J. (2017) · Evidence & Policy · doi:10.1332/174426417x15034894876108 · Cited by 61
Daykin, N., Gray, K., McCree, M., & Willis, J. (2016) · Arts & Health · doi:10.1080/17533015.2016.1206948 · Cited by 57
Genetics, epigenetics, and causation
Moss, M. L. (1981) · American Journal of Orthodontics · doi:10.1016/0002-9416(81)90172-x · Cited by 56
Noyes, J. (2010) · Journal of research in nursing · doi:10.1177/1744987110381696 · Cited by 53
Participatory photography in qualitative research: a methodological review
Byrne, E., Daykin, N., & Coad, J. E. (2016) · ORCA Online Research @Cardiff (Cardiff University) · doi:10.7331/vm.v4i2.66 · Cited by 48
Arts-based research: Weaving magic and meaning.
Greenwood, J. (2012) · Cited by 47
Deep impact: re-conceptualising university research impact using human cultural accumulation theory
Woolcott, G., Keast, R. L., & Pickernell, D. G. (2019) · Studies in Higher Education · doi:10.1080/03075079.2019.1594179 · Cited by 46
Brewer, J. D. (2011) · Research Evaluation · doi:10.3152/095820211x12941371876869 · Cited by 43
Engaging with arts-based research: a story in three parts
Douglas, K., & Carless, D. (2018) · Qualitative Research in Psychology · doi:10.1080/14780887.2018.1429843 · Cited by 36
Effects of necessary, sufficient, and indirect causation on judgments of criminal liability.
Greene, E. J., & Darley, J. M. (1998) · Law and Human Behavior · doi:10.1023/a:1025723010408 · Cited by 27
Mayne, J. W. (2012) · CGSPace A Repository of Agricultural Research Outputs (Consultative Group for International Agricultural Research) · Cited by 23
A proposed framework and tool for non-economic research impact measurement
Mitchell, V. (2019) · Higher Education Research & Development · doi:10.1080/07294360.2019.1590319 · Cited by 17
Building evaluation capacity in small/medium-sized conservation projects
Dickson, I., & Merriman, J. (2018) · Proceedings of the 5th European Congress of Conservation Biology · doi:10.17011/conference/eccb2018/107856 · Cited by 9
Metadata
- Authors
- Reed, M. S., Ferré, M., Martin-Ortega, J., Blanche, R., Lawford-Rolfe, R., Dallimer, M., & Holden, J.
- Venue
- Research Policy
- Publisher
- Elsevier BV
- Published
- 2021-05
- Volume
- 50
- Issue
- 4
- Pages
- 104147
- ISSN
- 0048-7333 (ISSN-L)
- OpenAlex
- W3088866792
- Cited by
- 319
- References
- 95
- FWCI
- 24.44
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.