CoChat
Sign Up
Cited by 14scientometrics and bibliometrics researchResearch, Science, and AcademiaEfficiency Analysis Using DEA

Machine learning in scientific grant review: algorithmically predicting project efficiency in high energy physics

Open access
Sikimić, Vlasta, Radovanović, Sandro·2022·European Journal for Philosophy of Science
DOI 10.1007/s13194-022-00478-6PMC PMC9307966
Open this paper in CoChat — free accountOpen PDF

Abstract

As more objections have been raised against grant peer-review for being costly and time-consuming, the legitimate question arises whether machine learning algorithms could help assess the epistemic efficiency of the proposed projects. As a case study, we investigated whether project efficiency in high energy physics (HEP) can be algorithmically predicted based on the data from the proposal. To analyze the potential of algorithmic prediction in HEP, we conducted a study on data about the structure (project duration, team number, and team size) and outcomes (citations per paper) of HEP experiments with the goal of predicting their efficiency. In the first step, we assessed the project efficiency using Data Envelopment Analysis (DEA) of 67 experiments conducted in the HEP laboratory Fermilab. In the second step, we employed predictive algorithms to detect which team structures maximize the epistemic performance of an expert group. For this purpose, we used the efficiency scores obtained by DEA and applied predictive algorithms – lasso and ridge linear regression, neural network, and gradient boosted trees – on them. The results of the predictive analyses show moderately high accuracy (mean absolute error equal to 0.123), indicating that they can be beneficial as one of the steps in grant review. Still, their applicability in practice should be approached with caution. Some of the limitations of the algorithmic approach are the unreliability of citation patterns, unobservable variables that influence scientific success, and the potential predictability of the model.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s13194-022-00478-6

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Some previous studies have used AI to assess funding applications, but none seem to have produced results good enough to be adopted by funders.

    “While some previous studies have used AI to assess funding applications (e.g., Sikimić, & Radovanović, 2022 ), none seem to have produced results that were good enough to be adopted by funders”

    From Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data (2026) · cites this paper as Sikimić, & Radovanović, 2022

    The Introduction describes a study using machine learning to predict project efficiency from data available in proposals, so the paper is an example of AI research relevant to funding-application assessment. But it analyzes records of completed Fermilab experiments, rather than reporting that AI assessed submitted funding applications. The sentence’s broader conclusion about adoption by funders is not specifically attributed to this citation. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 52 · Cited by 14

Research works printed but not linked: 52

  1. Abu-Elyounes, D. (2020). Contextual fairness: A legal and policy analysis of algorithmic fairness. Illinois Journal of Law, Technology & Policy, 1, 1–54.
  2. Adam D Science funders gamble on grant lotteries Nature 2019 575 7785 574 575 10.1038/d41586-019-03572-7 31772357 · DOI
  3. Aysolmaz, B., Nemeth, M., & Iren, D. (2021, June). A method for objective performance benchmarking of teams with process mining and DEA. In 29th European Conference on Information Systems (ECIS 2021): Human values crisis in a digitizing world [1773]. AIS Electronic Library.
  4. Benicio J de Mello JCS Productivity analysis and variable returns of scale: DEA efficiency frontier interpretation Procedia Computer Science 2015 55 341 349 10.1016/j.procs.2015.07.059 · DOI
  5. Boisot M Nordberg M Yami S Nicquevert B Collisions and collaboration: The Organization of Learning in the ATLAS experiment at the LHC 2011 OUP Oxford
  6. Carillo MR Papagni E Sapio A Do collaborations enhance the high-quality output of scientific institutions? Evidence from the Italian research assessment exercise The Journal of Socio-Economics 2013 47 25 36 10.1016/j.socec.2013.08.005 · DOI
  7. Charnes A Cooper WW Lewin AY Seiford LM Data envelopment analysis: Theory, methodology, and applications 2013 Springer Science & Business Media
  8. Chen C Cook WD Imanirad R Zhu J Balancing fairness and efficiency: Performance evaluation with disadvantaged units in non-homogeneous environments European Journal of Operational Research 2020 287 3 1003 1013 10.1016/j.ejor.2020.05.015 · DOI
  9. Coelli TJ Rao DSP O'Donnell CJ Battese GE An introduction to efficiency and productivity analysis 2005 Springer Science & Business Media
  10. Contopoulos-Ioannidis DG Alexiou GA Gouvias TC Ioannidis JP Life cycle of translational research for medical interventions Science 2008 321 5894 1298 1299 10.1126/science.1160622 18772421 · DOI · Paper page
  11. Cooper WW Seiford LM Zhu J Handbook on data envelopment analysis (Vol. 164) 2011 Springer Science & Business Media
  12. Cyranoski D Artificial intelligence is selecting grant reviewers in China Nature 2019 569 7756 316 318 10.1038/d41586-019-01517-8 31089232 · DOI
  13. Ding, F., Hardt, M., Miller, J., & Schmidt, L. (2021). Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34, 6478–6490.
  14. Doyle JM Quinn K Bodenstein YA Wu CO Danthi N Lauer MS Association of percentile ranking with citation impact and productivity in a large cohort of de novo NIMH-funded R01 grants Molecular Psychiatry 2015 20 9 1030 1036 10.1038/mp.2015.71 26033238 PMC5526589 · DOI
  15. Emrouznejad A Banker R Lopes ALM de Almeida MR Data envelopment analysis in the public sector Socio-Economic Planning Sciences 2014 48 1 2 3 10.1016/j.seps.2013.12.005 · DOI
  16. Fang FC Bowen A Casadevall A NIH peer review percentile scores are poorly predictive of grant productivity Elife 2016 5 e13323 10.7554/eLife.13323 26880623 PMC4769156 · DOI
  17. Fister I Jr Fister I Perc M Toward the discovery of citation cartels in citation networks Frontiers in Physics 2016 4 49 10.3389/fphy.2016.00049 · DOI · Paper page
  18. Friedman JH Greedy function approximation: A gradient boosting machine Annals of Statistics 2001 29 1189 1232 10.1214/aos/1013203451 · DOI · Paper page
  19. Guthrie, S., Rincon, D. R., McInroy, G., Ioppolo, B., & Gunashekar, S. (2019). Measuring bias, burden and conservatism in research funding processes. In F1000Research (Vol. 8, p. 851). 10.12688/f1000research.19156.1 · DOI
  20. Halkos GE Nickolaos GT Productivity efficiency and firm size: An empirical analysis of foreign owned companies International Business Review 2007 16 6 713 731 10.1016/j.ibusrev.2007.06.002 · DOI
  21. Heesen, R., & Bright, L. K. (2020). Is peer review a good idea? The British Journal for the Philosophy of Science, 72(3), 635–663.
  22. Herbert DL Barnett AG Clarke P Graves N On the time spent preparing grant proposals: An observational study of Australian researchers BMJ Open 2013 3 5 e002800 10.1136/bmjopen-2013-002800 PMC3664356 23793700 · DOI
  23. Hoddeson L Kolb AW Westfall C Fermilab: Physics, the frontier, and megascience 2008 University of Chicago Press
  24. Huang MH Chang YW Characteristics of research output in social sciences and humanities: From a research evaluation perspective Journal of the American Society for Information Science and Technology 2008 59 11 1819 1828 10.1002/asi.20885 · DOI
  25. Irvine J Martin BR CERN: Past performance and future prospects: II. The scientific performance of the CERN accelerators Research Policy 1984 13 4 247 284 10.1016/0048-7333(84)90011-8 · DOI
  26. Kao C Lin PH Qualitative factors in data envelopment analysis: A fuzzy number approach European Journal of Operational Research 2011 211 3 586 593 10.1016/j.ejor.2010.12.004 · DOI
  27. Katz R The effects of group longevity on project communication and performance Administrative Science Quarterly 1982 27 81 104 10.2307/2392547 · DOI
  28. Kelly, K., & Mayo-Wilson, C. (2010). Causal conclusions that Flip repeatedly and their justification. In P. Gruenewald & P. Spirtes (Eds.), Proceedings of the 26th conference on uncertainty and artificial intelligence (pp. 277–286). AUAI Press.
  29. Mansoury, M., Abdollahpouri, H., Pechenizkiy, M., Mobasher, B., & Burke, R. (2020, October). Feedback loop and bias amplification in recommender systems. In Proceedings of the 29th ACM international conference on Information & Knowledge Management (pp. 2145–2148). 10.1145/3340531.3412152 · DOI
  30. Martin BR Irvine J CERN: Past performance and future prospects: I. CERN’s position in world high-energy physics Research Policy 1984 13 4 183 210 10.1016/0048-7333(84)90014-3 · DOI
  31. Martin BR Irvine J CERN: Past performance and future prospects: III. CERN and the future of world high-energy physics Research Policy 1984 13 4 311 342 10.1016/0048-7333(84)90017-9 · DOI
  32. Murphy KP Machine learning: A probabilistic perspective 2012 MIT press
  33. Mutz, R., Bornmann, L., & Daniel, H. D. (2017). Are there any frontiers of research performance? Efficiency measurement of funded research projects with the Bayesian stochastic frontier analysis for count data. Journal of Informetrics, 11(3), 613–628.
  34. Peñalvo FJG Benito JC González MM Ingelmo AV Prieto JCS Sánchez RT Proposing a machine learning approach to analyze and predict employment and its factors IJIMAI 2018 5 2 39 45 10.9781/ijimai.2018.02.002 · DOI
  35. Perović S Radovanović S Sikimić V Berber A Optimal research team composition: Data envelopment analysis of Fermilab experiments Scientometrics 2016 108 1 83 111 10.1007/s11192-016-1947-9 · DOI
  36. Perović, S., & Sikimić, V. (2019). How theories of induction can streamline measurements of scientific performance. Journal for General Philosophy of Science, 51, 267–291.
  37. Pusztai L Hatzis C Andre F Reproducibility of research and preclinical validation: Problems and solutions Nature Reviews. Clinical Oncology 2013 10 720 724 10.1038/nrclinonc.2013.171 24080600 · DOI
  38. Radovanović S Savić G Delibašić B FairDEA - removing disparate impact from efficiency scores European Journal of Operational Research 2021 301 3 1088 1098 10.1016/j.ejor.2021.12.001 · DOI
  39. Radovanović, S., Petrović, A., Delibašić, B., & Suknović, M. (2021). A fair classifier chain for multi-label bank marketing strategy classification. International Transactions in Operational Research. 10.1111/itor.13059 · DOI
  40. Roger S de Regt HW Hartmann S Okasha S Stopping rules and data monitoring in clinical trials EPSA philosophy of science: Amsterdam 2009, The European Philosophy of Science Association Proceedings 2012 Springer 375 386
  41. Schulte O Inferring conservation Laws in particle physics: A case study in the problem of induction The British Journal for the Philosophy of Science 2000 51 4 771 806 10.1093/bjps/51.4.771 · DOI
  42. Sikimić, V., & Herud-Sikimić, O. (2022). Modelling efficient team structures in biology. Journal of Logic and Computation. 10.1093/logcom/exac021 · DOI
  43. Simko I Analysis of bibliometric indicators to determine citation bias Palgrave Commun 2015 1 15011 10.1057/palcomms.2015.11 · DOI
  44. Steele K Persistent experimenters, stopping rules, and statistical inference Erkenntnis 2013 78 4 937 961 10.1007/s10670-012-9388-1 · DOI
  45. Tibshirani R Regression shrinkage and selection via the lasso Journal of the Royal Statistical Society: Series B: Methodological 1996 58 267 288
  46. Van den Besselaar P Sandström U Early career grants, performance, and careers: A study on predictive validity of grant decisions Journal of Informetrics 2015 9 4 826 838 10.1016/j.joi.2015.07.011 · DOI
  47. Von Tunzelmann, N., Ranga, M., Martin, B., & Geuna, A. (2003). The effects of size on research performance: A SPRU review. Report prepared for the Office of Science and Technology, Department of Trade and Industry, SPRU. University of Sussex.
  48. Zafar MB Valera I Gomez-Rodriguez M Gummadi KP Fairness constraints: A flexible approach for fair classification The Journal of Machine Learning Research 2019 20 1 2737 2778
  49. Žliobaitė, I. (2010). Learning under concept drift: An overview. arXiv preprint arXiv:1010.4784. 10.48550/arxiv.1010.4784 · DOI
  50. Zollman KJ The communication structure of epistemic communities Philosophy of Science 2007 74 5 574 587 10.1086/525605 · DOI
  51. Zollman KJ The epistemic benefit of transient diversity Erkenntnis 2010 72 1 17 35 10.1007/s10670-009-9194-6 · DOI
  52. Woodall T This AI could predict 10 years of scientific priorities—If we let it 2021 MIT Technology Review

Authors
Sikimić, V., & Radovanović, S.
Affiliations
  • University of Tübingen (DE)
  • University of Belgrade (RS)
Corresponding author
Sikimić, Vlasta
ORCID
Venue
European Journal for Philosophy of Science
Publisher
Springer Science and Business Media LLC
Published
2022-07-23
Volume
12
Issue
3
Pages
50-50
ISSN
  • 1879-4912 (ISSN-L)
  • 1879-4920
DOI
10.1007/s13194-022-00478-6
PMC
PMC9307966
OpenAlex
W4286685077
Cited by
14
References
52
Citations by year
2026: 3 · 2025: 3 · 2024: 6 · 2023: 1 · 2022: 1
FWCI
1.66
Type
journal article
Language
en
Open access
Licence
Locations
  • European Journal for Philosophy of Science (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PhilPapers (PhilPapers Foundation) (submittedVersion, other-oa)
  • Repository of the University of Belgrade Faculty of Organizational Sciences (RFOS) (submittedVersion, cc-by)
  • PhilPapers (PhilPapers Foundation) (submittedVersion)
Funders
Eberhard Karls Universität Tübingen
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Social Sciences › Decision Sciences › Information Systems and Management › Research, Science, and Academia
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Efficiency Analysis Using DEA
Keywords
grant peer review, algorithmic prediction, project efficiency, particle physics, team size, citation analysis, data envelopment analysis, Fermilab, research funding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review