The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
What is societal impact of research and how can it be assessed? a literature survey
Abstract
Since the 1990s, the scope of research evaluations becomes broader as the societal products (outputs), societal use (societal references), and societal benefits (changes in society) of research come into scope. Society can reap the benefits of successful research studies only if the results are converted into marketable and consumable products (e.g., medicaments, diagnostic tools, machines, and devices) or services. A series of different names have been introduced which refer to the societal impact of research: third stream activities, societal benefits, societal quality, usefulness, public values, knowledge transfer, and societal relevance. What most of these names are concerned with is the assessment of social, cultural, environmental, and economic returns (impact and effects) from results (research output) or products (research outcome) of publicly funded research. This review intends to present existing research on and practices employed in the assessment of societal impact in the form of a literature survey. The objective is for this review to serve as a basis for the development of robust and reliable methods of societal impact measurement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The New production of knowledge: the dynamics of science and research in contemporary societies
(1995) · Choice Reviews Online · doi:10.5860/choice.32-4463 · Cited by 7,046
The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies.
Baber, Z., Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1995) · Contemporary Sociology · doi:10.2307/2076669 · Cited by 4,406
The economic benefits of publicly funded basic research: a critical review
Salter, A. J., & Martin, B. R. (2001) · Research Policy · doi:10.1016/s0048-7333(00)00091-3 · Cited by 1,358
The increasing linkage between U.S. technology and public science
Narin, F., Hamilton, K. S., & Olivastro, D. (1997) · Research Policy · doi:10.1016/s0048-7333(97)00013-9 · Cited by 1,281
Academic research and industrial innovation
Mansfield, E. (1991) · Research Policy · doi:10.1016/0048-7333(91)90080-a · Cited by 1,262
Stephan, P. (2012) · Harvard University Press eBooks · doi:10.4159/harvard.9780674062757 · Cited by 934
Expert Opinions in Forecasting: The Role of the Delphi Technique
Rowe, G., & Wright, G. (2001) · International Series in Operations Research & Management Science · doi:10.1007/978-0-306-47630-3_7 · Cited by 777
Real Science: What it Is and What it Means
Ziman, J. (2000) · doi:10.1017/cbo9780511541391 · Cited by 760
Academic research and industrial innovation: An update of empirical findings
Mansfield, E. (1998) · Research Policy · doi:10.1016/s0048-7333(97)00043-7 · Cited by 643
Bornmann, L. (2011) · Annual Review of Information Science and Technology · doi:10.1002/aris.2011.1440450112 · Cited by 567
Self-citationIntroducing 'productive interactions' in social impact assessment
Spaapen, J., & van Drooge, L. (2011) · Research Evaluation · doi:10.3152/095820211x12941371876742 · Cited by 518
The Millennium Development Goals
(2014) · Maternal and Infant Deaths: Chasing Millennium Development Goals 4 and 5 · doi:10.1017/cbo9781107784758.003 · Cited by 489
Public research and industrial innovations in Germany
Beise, M., & Stahl, H. (1999) · Research Policy · doi:10.1016/s0048-7333(98)00126-7 · Cited by 476
The Research Excellence Framework and the 'impact agenda': are we creating a Frankenstein monster?
Martin, B. R. (2011) · Research Evaluation · doi:10.3152/095820211x13118583635693 · Cited by 471
How Can Payback from Health Services Research Be Assessed?
Buxton, M., & Hanney, S. (1996) · Journal of Health Services Research & Policy · doi:10.1177/135581969600100107 · Cited by 372
Knowledge, networks and nations: global scientific collaboration in the 21st century
Wilsdon, J. (2011) · Figshare · Cited by 364
Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, H. (2008) · Ethics in Science and Environmental Politics · doi:10.3354/esep00084 · Cited by 359
Self-citationTriple Helix of University-Industry-Government Relations
Leydesdorff, L. (2013) · Encyclopedia of Creativity, Invention, Innovation and Entrepreneurship · doi:10.1007/978-1-4614-3858-8_452 · Cited by 324
Walter, A. I., Helgenberger, S., Wiek, A., & Scholz, R. W. (2007) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2007.08.002 · Cited by 323
Public Value Mapping and Science Policy Evaluation
Bozeman, B., & Sarewitz, D. (2011) · Minerva · doi:10.1007/s11024-011-9161-7 · Cited by 308
Abbott, A., Cyranoski, D., Jones, N., Maher, B., Schiermeier, Q., & Van Noorden, R. (2010) · Nature · doi:10.1038/465860a · Cited by 261
Bornmann, L. (2012) · EMBO reports · doi:10.1038/embor.2012.99 · Cited by 252
Self-citationRising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future
Committee on Prospering in the Global Economy of the 21st Century: An Agenda for American Science and Technology, Committee on Science, E., National Academy of Sciences, National Academy of Engineering, & Institute of Medicine (2007) · National Academies Press eBooks · doi:10.17226/11463 · Cited by 234
Toole, A. A. (2012) · Research Policy · doi:10.1016/j.respol.2011.06.004 · Cited by 212
Describing the impact of health research: a Research Impact Framework
Kuruvilla, S., Mays, N., Pleasant, A., & Walt, G. (2006) · BMC Health Services Research · doi:10.1186/1472-6963-6-134 · Cited by 204
Tracing 'productive interactions' to identify social impacts: an example from the social sciences
Molas-Gallart, J., & Tang, P. (2011) · Research Evaluation · doi:10.3152/095820211x12941371876706 · Cited by 180
State of the art in assessing research impact: introduction to a special issue
Donovan, C. (2011) · Research Evaluation · doi:10.3152/095820211x13118583635918 · Cited by 170
The 'Payback Framework' explained
Donovan, C., & Hanney, S. R. (2011) · Research Evaluation · doi:10.3152/095820211x13118583635756 · Cited by 160
Measuring the social impact of research
Smith, R. W. (2001) · BMJ · doi:10.1136/bmj.323.7312.528 · Cited by 154
Peer review and the ex ante assessment of societal impacts
Holbrook, J. B., & Frodeman, R. (2011) · Research Evaluation · doi:10.3152/095820211x12941371876788 · Cited by 135
Donovan, C. (2008) · New Directions for Evaluation · doi:10.1002/ev.260 · Cited by 122
Peer review and the relevance gap: ten suggestions for policy-makers
Nightingale, P., & Scott, A. (2007) · Science and Public Policy · doi:10.3152/030234207x254396 · Cited by 120
Medical Research: What's It Worth? Estimating the Economic Benefits from Medical Research in the UK
Buxton, I. L. O., Hanney, S. R., Stephen Morris, A., Sundmacher, L., Mestre‐Ferrandiz, J., Garau, M., Sussex, J., Grant, J., Ismail, S., Nason, E., et al. (2008) · Cited by 114
Science economics: What science is really worth
Macilwain, C. (2010) · Nature · doi:10.1038/465682a · Cited by 107
Real-world approaches to assessing the impact of environmental research on policy
Bell, S., Shaw, B. G., & Boaz, A. (2011) · Research Evaluation · doi:10.3152/095820211x13118583635792 · Cited by 79
The Scientific Investments of Nations
May, R. M. (1998) · Science · doi:10.1126/science.281.5373.49 · Cited by 74
Roberts, M. R. (2009) · Social Epistemology · doi:10.1080/02691720903364035 · Cited by 68
Health research: measuring the social, health and economic benefits
Frank, C. B., & Nason, E. (2009) · Canadian Medical Association Journal · doi:10.1503/cmaj.090016 · Cited by 66
Measuring research performance during a changing relationship between science and society
Ernø-Kjølhede, E., & Hansson, F. (2011) · Research Evaluation · doi:10.3152/095820211x12941371876544 · Cited by 62
The impact of cancer research: how publications influence UK cancer clinical guidelines
Lewison, G., & Sullivan, R. J. (2008) · British Journal of Cancer · doi:10.1038/sj.bjc.6604405 · Cited by 59
Hanney, S. R., Packwood, T., & Buxton, M. J. (2000) · Evaluation · doi:10.1177/13563890022209181 · Cited by 57
Klautzer, L., Hanney, S. R., Nason, E., Rubin, J., Grant, J., & Wooding, S. (2011) · Research Evaluation · doi:10.3152/095820211x13118583635675 · Cited by 57
The Triple Helix of University-Industry-Government Relations (February 2012)Preprint
Loet Leydesdorff, A. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.1996760 · Cited by 48
Bornmann, L. (2010) · Scientometrics · doi:10.1007/s11192-010-0222-8 · Cited by 45
Self-citationvan der Weijden, I., Verbree, M., & van den Besselaar, P. (2012) · Science and Public Policy · doi:10.1093/scipol/scr003 · Cited by 44
Societal output and use of research performed by health research groups
Mostert, S. P., Ellenbroek, S. P., Meijer, I., van Ark, G., & Klasen, E. C. (2010) · Health Research Policy and Systems · doi:10.1186/1478-4505-8-30 · Cited by 43
The theory and practice of public-sector R&D economic impact analysisPreprint
Link, A. N., & Scott, J. T. (2013) · RePEc: Research Papers in Economics · Cited by 37
Mardis, M. A., Hoffman, E., & McMartin, F. P. (2012) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22693 · Cited by 35
Kamenetzky, J. R. (2012) · Science and Public Policy · doi:10.1093/scipol/scs059 · Cited by 32
Evaluating the societal relevance of academic research: A guide
van der Meulen, B. (2010) · Data Archiving and Networked Services (DANS) · Cited by 32
Niederkrotenthaler, T., Dorner, T. E., & Maier, M. (2011) · BMC Public Health · doi:10.1186/1471-2458-11-588 · Cited by 31
The Dedisciplining of Peer Review
Frodeman, R., & Briggle, A. (2012) · Minerva · doi:10.1007/s11024-012-9192-8 · Cited by 31
Scott, J. E., Blasinsky, M., Dufour, M., Mandai, R. J., & Philogene, G. S. (2011) · Research Evaluation · doi:10.3152/095820211x12941371876661 · Cited by 24
White, H., & Black, R. (2010) · Taylor & Francis eBooks · doi:10.4324/9780203403235_chapter_1 · Cited by 23
S&T Indicators for Policy Making in a Changing Science-Society Relationship
Barrè, R. (2006) · Kluwer Academic Publishers eBooks · doi:10.1007/1-4020-2755-9_6 · Cited by 20
Palfrey, C., Thomas, P., & Phillips, C. (2012) · Policy Press eBooks · doi:10.56687/9781847429162-006 · Cited by 10
Cited by
Bornmann, L., & Mutz, R. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23329 · Cited by 1,576
Self-citationFortunato, S., Bergstrom, C. T., Börner, K., Evans, J. A., Helbing, D., Milojević, S., Petersen, A. M., Radicchi, F., Sinatra, R., Uzzi, B., et al. (2018) · Science · doi:10.1126/science.aao0185 · Cited by 1,497
“The development of alternative metrics covering web (88) and social media (89) activity and societal impact (90) is critical in this regard.”
Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories
Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019) · Sage Open · doi:10.1177/2158244019829575 · Cited by 1,257
Do Altmetrics Work? Twitter and Ten Other Social Web Services
Thelwall, M., Haustein, S., Larivière, V., & Sugimoto, C. R. (2013) · PLoS ONE · doi:10.1371/journal.pone.0064841 · Cited by 907
Bornmann, L. (2014) · Journal of Informetrics · doi:10.1016/j.joi.2014.09.005 · Cited by 578
Self-citation
Metadata
- Authors
- Bornmann, L.
- Venue
- Journal of the American Society for Information Science and Technology
- Publisher
- Wiley
- Published
- 2012-12-17
- Volume
- 64
- Issue
- 2
- Pages
- 217-233
- ISSN
- 1532-2882 (ISSN-L)
- 1532-2890
- OpenAlex
- W1979116104
- Cited by
- 731
- References
- 56
- FWCI
- 22.64
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.