The DOI matches its Crossref record.
The policy challenges of peer review: managing bias, conflict of interests and interdisciplinary assessments
Abstract
Characteristics and challenges of peer review are elucidated from three different perspectives: a social dynamics perspective defining peer review as an important control mechanism in the research community; an uncertainty perspective focusing on the inherent uncertainty in judging research quality: and an organisational perspective focusing on the effects of different ways of organising peer review. Findings from a broad set of empirical studies are used. Peer review often has some conservative and risk-minimising aspects, which may disfavour interdisciplinary and non-conventional research. When the aim is to actively promote interdisciplinary or other kinds of non-conventional research, the involved peer-review system consequently needs to be adjusted to a more risk-taking mode.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Reviewers weigh criteria such as originality, feasibility, investigator track record, and potential impact differently, even when operating under shared guidelines (Langfeldt, 2006; Fogelholm et al., 2012).”
From Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data (2026) · cites this paper as Langfeldt, 2006
In the citing paper
“More recent evidence suggests that low inter-reviewer agreement is not primarily a consequence of reviewer incompetence but rather reflects the inherently multidimensional and judgement-based nature of grant assessment.”
From Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data (2026) · cites this paper as Langfeldt, 2006
Citations
Cites
Still being checked.
Cited by
Sorted by citations
Items 101–179 of 179
The effect of novelty on the future impact of scientific grantsPreprint
Zhuang, H., & Acuna, D. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02712 · Cited by 5
Jong, L., Franssen, T., & Pinfield, S. (2022) · doi:10.31235/osf.io/nduxf · Cited by 5
Zhang, L., Tu, Z., Yu, Y., Shang, Y., & Huang, Y. (2024) · Scientometrics · doi:10.1007/s11192-024-05129-6 · Cited by 4
Hansson, F., & Mønsted, M. (2012) · Higher Education Policy · doi:10.1057/hep.2012.17 · Cited by 5
2.6 Relationship between Peer Review and Bibliometrics
Ochsner, M. (2020) · De Gruyter eBooks · doi:10.1515/9783110646610-014 · Cited by 4
Interdisciplinary Research Collaborations: Evaluation of a Funding Program
Rons, N. (2011) · Collnet Journal of Scientometrics and Information Management · doi:10.1080/09737766.2011.10700900 · Cited by 4
Ricerca interdisciplinare e ICAR17: una proposta per la definizione di un modello condiviso
Luigini, A. (2020) · FrancoAngeli srl eBooks · doi:10.3280/oa-548.33 · Cited by 4
Kumar, K. (2012) · The Sydney eScholarship Repository (The University of Sydney) · Cited by 4
A Science of Science and Innovation Policy Research Agenda
Feller, I., & Cozzens, S. E. (2007) · SMARTech Repository (Georgia Institute of Technology) · Cited by 4
Rutter, R. N. (2013) · UEA Digital Repository (University of East Anglia) · Cited by 4
Meetings that matter: the dual benefits of panel peer review
Oxley, K. (2024) · Research Evaluation · doi:10.1093/reseval/rvaf047 · Cited by 3
Are reviewer scores consistent with citations?
Xie, W., Jia, P., Zhang, G., & Wang, X. (2024) · Scientometrics · doi:10.1007/s11192-024-05103-2 · Cited by 3
Partial and particularistic: grant peer review panels evaluating interdisciplinarity
Oxley, K. (2024) · Research Evaluation · doi:10.1093/reseval/rvaf037 · Cited by 3
Social Sciences for the Prevention of Blindness
Goldschmidt, P. L. (2015) · Tropical Medicine and Health · doi:10.2149/tmh.2014-32 · Cited by 3
A System Dynamics Analysis of National R&D Performance Measurement System in Korea
You, T., & Jung, W. (2018) · Industrial Engineering & Management Systems · doi:10.7232/iems.2018.17.4.833 · Cited by 3
Universalism and particularism in the recommendations of the nobel prize for science
Ko, B.-K., Jang, Y., & Yang, J. (2024) · Scientometrics · doi:10.1007/s11192-023-04921-0 · Cited by 3
Edlund, P., & Lammi, I. (2022) · Minerva · doi:10.1007/s11024-022-09466-9 · Cited by 4
Comprehending and Regulating Financial Crises: An Interdisciplinary Approach
Bandelj, N., Elyachar, J., Richardson, G., & Weatherall, J. O. (2016) · Perspectives on Science · doi:10.1162/posc_a_00216 · Cited by 3
Improving the efficiency of research proposals evaluation: A two-stage procedure
Seeber, M., Svege, I. C., & Hesselberg, J. (2024) · Research Evaluation · doi:10.1093/reseval/rvae020 · Cited by 2
Fatherazi, R., Balch, A., Hutchison, K., Lasonder, J., Dibie, I. S., Asquith, W., Templer, I., Glasson, O., Johnstone, O., Tecca, V., et al. (2025) · Journal of Human Trafficking · doi:10.1080/23322705.2025.2582453 · Cited by 2
Digitale Zukunft: ein inter‐ und transdisziplinäres Thema
Simon, D., & Steffen, E. (2020) · Springer eBooks · doi:10.1007/978-3-662-52979-9_7 · Cited by 2
Interdisciplinary research by accounting scholars: An exploratory study
Avallone, F. G., Quagli, A., & Ramassa, P. (2022) · FINANCIAL REPORTING · doi:10.3280/fr2022-002001 · Cited by 2
Policy variation in the external evaluation of research for tenure at U.S. universities
Hannon, L., & Bergey, M. R. (2024) · Research Evaluation · doi:10.1093/reseval/rvae018 · Cited by 2
The Many Dilemmas of Grant Peer Review
Langfeldt, L. (2024) · SpringerBriefs in political science · doi:10.1007/978-3-031-69580-3_5 · Cited by 3
Self-citationReale, E., & Zinilli, A. (2017) · RIV Rassegna Italiana di Valutazione · doi:10.3280/riv2016-064003 · Cited by 2
Designing argumentation in peer review: Patterns of evaluation in the AltCHI open peer review forum
Rughinis, R. V. (2013) · Iberian Conference on Information Systems and Technologies (CISTI) · Cited by 2
Della‐Piana, C. K., & Della‐Piana, G. M. (2007) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v4i8.33 · Cited by 2
Morillo, F., & Aparicio, J. (2011) · Revista española de Documentación Científica · doi:10.3989/redc.2011.4.837 · Cited by 2
Resolving disputes over science in natural resource agency decisionmaking
Ruell, E. W., Burkardt, N., & Clark, D. R. (2010) · Cited by 2
Identifying and Facilitating High-Quality Research
Langfeldt, L., Aagaard, K., Borlaug, S. B., & Sivertsen, G. (2024) · SpringerBriefs in political science · doi:10.1007/978-3-031-69580-3_2 · Cited by 3
Self-citationFunding transformative research - rationale and impacts
Luukkonen, T. (2022) · Edward Elgar Publishing eBooks · doi:10.4337/9781788970815.00011 · Cited by 2
Coordination of European transnational research in organic food and farming
Jespersen, L. M., & Niels Halberg, A. (2012) · Organic Eprints (International Centre for Research in Organic Food Systems, and Research Institute of Organic Agriculture) · Cited by 2
Identification of Factors to Promote Interdisciplinary Research: A Trial at COINS
Katoh, S., Lauto, G., Anzai, T., & Sengoku, S. (2018) · Portland International Conference on Management of Engineering and Technology (PICMET) · doi:10.23919/picmet.2018.8481881 · Cited by 2
Creating what is necessary for optimizing psychotherapy
Krause, M. S. (2018) · Psychotherapy Research · doi:10.1080/10503307.2018.1534019 · Cited by 2
Coryn, C. L. S., & Scriven, M. S. (2008) · New Directions for Evaluation · doi:10.1002/ev.256 · Cited by 2
Sandström, U., & Thelwall, M. (2026) · Journal of Data and Information Science · doi:10.1515/jdis-2026-0048 · Cited by 1
Ex-ante evaluation of research project funding: what is the importance of scientific quality?
Reale, E., & Zinilli, A. (2020) · WELFARE E ERGONOMIA · doi:10.3280/we2019-001004 · Cited by 1
Peterson, H., & Husu, L. (2024) · Research Evaluation · doi:10.1093/reseval/rvaf030 · Cited by 1
Equality is a quality! The assessment of sex and gender perspectives as a quality indicator
Lundgren, J., Carlsson, V., & Söderberg, J. A. (2025) · Science and Public Policy · doi:10.1093/scipol/scaf042 · Cited by 2
Multidimensional bibliometric assessment of science funding effectiveness
Huang, T.-Y., & Xiong, W. (2025) · Journal of Informetrics · doi:10.1016/j.joi.2025.101731 · Cited by 1
Shaw, J., Pinto, M. F., & Wilholt, T. (2026) · Perspectives on Science · doi:10.1162/posc.a.584 · Cited by 2
Assessment of uncertainty in the quality of knowledge in the research publication review process
Sabetzadeh, F., Tsui, E., & Lee, W. B. (2013) · International Conference on Fuzzy Systems and Knowledge Discovery (FSKD) · doi:10.1109/fskd.2013.6816331 · Cited by 1
Alemanne, N. D. (2014) · Cited by 1
Impact from the Evaluators’ Eye
Derrick, G. (2018) · Springer eBooks · doi:10.1007/978-3-319-63627-6_1 · Cited by 1
Five key issues in inter- and transdisciplinary research grant: A literature review
Wang, Z., Gong, Y., Wu, F., & Li, Y. (2016) · International Conference on Management Science and Engineering (ICMSE) · doi:10.1109/icmse.2016.8365525 · Cited by 1
Peer review in a pickle: Policy approaches for academic peer review
Byers, B. S., Dougherty, T. R., & Horo, U. (2023) · MIT Science Policy Review · doi:10.38105/spr.dx30fr4j7c · Cited by 1
An exploratory analysis of the operational restrictions of virtual peer review panels
Derrick, G. (2024) · Research Evaluation · doi:10.1093/reseval/rvae065 · Cited by 1
Fiorentin, F. A., & Suárez, D. (2024) · Research Evaluation · doi:10.1093/reseval/rvaf048 · Cited by 1
16 Messung von Forschungsleistungen? Was gemessen wird und was gemessen werden will
Ochsner, M. (2020) · De Gruyter eBooks · doi:10.1515/9783110689884-017 · Cited by 1
Burkardt, N., & Ruell, E. W. (2012) · Antarctica A Keystone in a Changing World · doi:10.3133/ofr20121186 · Cited by 1
Peer Review of Impact: Could It Work?
Derrick, G. (2018) · Springer eBooks · doi:10.1007/978-3-319-63627-6_2 · Cited by 1
Interacting online: Examining behaviour in a crowdfunding setting
Moy, N. (2020) · Queensland University of Technology · doi:10.5204/thesis.eprints.204263 · Cited by 1
How competition for funding impacts scientific practicePreprint
Meirmans, S. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.07.30.502158 · Cited by 1
Odprta vprašanja in dileme kvantitativnega in kvalitativnega ocenjevanja v znanosti
Mali, F., & Jug, J. (2006) · Organizacija znanja · doi:10.3359/oz0604152
Mortuary science programs : examination of the external evaluation team
Reinhard, D. L. (2010) · doi:10.54014/e1r4-qsgb
Della‐Piana, C. K., & Della‐Piana, G. M. (2010)
: (Sociology Academic World: The Institutionalization of Relations of Power and Knowledge)Preprint
Vakhshtayn, V. S., & Nazarenko, A. P. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2831386
DEPLOYMENT TO RURAL COMMUNITIES IN KWAZULU-NATAL
Jere, N. (2011)
An analysis of rater effects in reviews of scientific manuscripts
Turner, D. P. (2017) · NC Digital Online Collection of Knowledge and Scholarship (The University of North Carolina at Greensboro)
Evaluation practices in the selection of ground-breaking research proposals, Working Paper
WWTF, Luukkonen, Stampfer, & Strassnig (2015) · doi:10.22163/fteval.2018.292
Kulczycki, E., Rozkosz, E. A., Engels, T., Guns, R., Hołowiecki, M., Pölönen, J., & Ciereszko, K. (2019) · Nauka i Szkolnictwo Wyższe · doi:10.14746/nisw.2019.1-2.6
Interdisciplinarity in academic research? Evidences from competitive project funding
Reale, E., & Zinilli, A. (2020) · WELFARE E ERGONOMIA · doi:10.3280/we2019-001008
(2018) · Elsevier eBooks · doi:10.1016/b978-0-08-102474-4.00023-6
Edlund, P. (2020) · Edward Elgar Publishing eBooks · doi:10.4337/9781839106095.00019
Taylor, M., Marrone, M., Chang, F., & Rooney, D. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4851219
How transport journals weed out qualitative research and why it matters
Klein, N. J., Linovski, O., Lowe, K., & Smart, M. (2026) · Transportation · doi:10.1007/s11116-026-10777-z
Rakap, S., Balıkcı, Ş., & Gülboy, E. (2025) · European Journal of Special Needs Education · doi:10.1080/08856257.2025.2533555
Yan, Z. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009381338.010
Yan, Z. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009381338.005
Possibilities for the peer review of interdisciplinary research: a small-scale qualitative study
Vesma, G. (2026) · Research Integrity and Peer Review · doi:10.1186/s41073-026-00204-3
Design e metodi di ricerca pedagogica. Uno sguardo alle riviste di settore
Pastore, S. (2011) · CINECA IRIS Institutional Research Information System (University of Bari Aldo Moro)
Gichuru, E. N., & Rukooko, A. B. (2026) · African Identities · doi:10.1080/14725843.2026.2648035
The time has come for big changes to improve research funding
Kolarz, P. (2026) · PLoS Biology · doi:10.1371/journal.pbio.3003754
Reviewing the reviewers: how social and traditional media repurpose open promotion reviews
Remisiewicz, Ł. (2026) · Science and Public Policy · doi:10.1093/scipol/scag023
Stanley, A. E., Koley, M., Kumar, A., & Dutta, D. (2026) · Research Evaluation · doi:10.1093/reseval/rvag029
Krumsvik, R. J., & Johansen, M. O. (2026) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2026.1874986
Evaluating automated refereeing with heterogeneous LLM panels
Douven, I. (2026) · Scientometrics · doi:10.1007/s11192-026-05783-y · Cited by 1
Overcoming animal methods bias and its impacts on nonanimal research assessments
Trunnell, E. R., & Krebs, C. E. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-26514-3.00012-6
Funding mechanisms as incentive systems: a role for science philanthropy
Parr, L., Sarfraz, N., Raver, S. M., & Penrod, N. M. (2026) · Frontiers in Systems Biology · doi:10.3389/fsysb.2026.1939099
Metadata
- Authors
- Langfeldt, L.
- Corresponding author
- Langfeldt, Liv
- ORCID
- Langfeldt, Liv
- Venue
- Research Evaluation
- Publisher
- Oxford University Press (OUP)
- Published
- 2006-04-01
- Volume
- 15
- Issue
- 1
- Pages
- 31-41
- ISSN
- 0958-2029 (ISSN-L)
- 1471-5449
- OpenAlex
- W2078257123
- Cited by
- 179
- References
- 0
- Citations by year
- 2026: 11 · 2025: 6 · 2024: 18 · 2023: 7 · 2022: 12 · 2021: 6 · 2020: 13 · 2019: 9 · 2018: 11 · 2017: 9 · 2016: 12 · 2015: 14 · 2014: 4 · 2013: 7 · 2012: 12
- FWCI
- 29.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Research Evaluation (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Business, Management and Accounting › Management of Technology and Innovation › Research, Science, and Academia
- Keywords
- peer review, research evaluation policy, peer review bias, conflicts of interest, interdisciplinary research, research quality assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review